Delivery Assurance
Weekly Update 1 Apr 2010
Post Office Ltd - HNGX
Gavin Bounds
Deliverables:
FUJ00095073
FUJ00095073
Mileston
e
Deliverable
Date due
Forecast
date
Owner
Acceptance criteria
HNGX_L2_10
30
Counter Roll Out Commence
25/02/10
12/04/10
Geoff Butts
This date will slip
due to High Volume
bring suspended on
26/03/10 as a result
of live issues. The
revised date will be
confirmed as soon
as possible once
High Volume Pilot
resumes.
HNGX_L2_71
9
Branch Router Roll Out Complete
13/04/10
13/04/10
Geoff Butts
100% of branch
routers installed
(excluding 53
branches requiring
satellite comms,
which will be
implemented in
May 2010).
HNGX_L2_76
oO
Counter Roll Out Complete
30/04/10
11/06/10
Geoff Butts
96% of branches
migrated to HNGX.
11/06/10 date will
slip following
FUJ00095073
FUJ00095073
delays to pilot and
start of rollout.
Higher beat rates
will be tested early
in rollout to allow
time to be saved.
Commentary:
There are now 622 HNG-X branches trading on HNG-X, including 8 Model Offices. The High
Volume pilot was suspended on 26/03/10 due to issues with live service which have impacted
on the HNG-X branches:-
™ Branch database hanging incidents occurred 26/03/10, 27/03/10, 29/03/10 — affecting
HNG-X estate.
® Datacentre power supply issue overnight on 30/03/10.
™ On 01/04/10, there was a problem with the processing of banking and debit card
transactions on Horizon, and a limited impact on some HNG-X branches.
As a result of these issues, the HNG-X Service has been put on Red Alert.
Intensive monitoring of HNG-X has continued since last weekend, and an Oracle Patch Set is
being tested that is known to address the Oracle error messages seen in live when the
Branch database hangs. Subject to completion of testing, this will be implemented into live,
together with a number of other fixes on 04/04/10.
The start of the full HNG-X rollout will be delayed from 12/04/10.
9,958 branches (87%) of the Post Office branches have now had a Branch Router installed.
Next Actions:
Complete Testing of Oracle Patch Set, to implement by Sunday 04/04/10 latest.
Continue to monitor live service with all technical teams to pick up any component failures as
quickly as possible.
Telecons planned with Customer over the weekend to monitor progress.
Action/Support Required:
Support currently provided through an Amber Alert Reference: 1053
ANNEX A - Top ten risks from Risk Pro
Client
Ref
HNG-
xRO003
HNG-
xRO006
Project
HNG-x Rt
Development
HNG-x
Migration
Owner
Wright
Roger
Butts Geoff
I
Description——--
The Development
requires more time
for incident
clearance than
planned causing
delay to go live
(POL owner KT)
We may not be
compliant from a
Security
perspective before
we go into pilot
(POL owner
Glan Trundell)
Caused by-
Cause: Compressed testing
timescales, incident
clearance / software stability
not achieved
Cause: Processes not in
place to support $2700
compliance
I-----Containment Strategy-~
Containment: Micro Manage delivery teams
Model expected workload to forecast effort required
Weekly Reporting to POL
‘Ongoing Manage Incident backlog into available release slots
31/7 - Additional release 14.16 introduced. Additional triage measures
introduced to limit the number of fixes to be incorporated pre live.
Maintenance release introduced to include highly desirable fixes required prior to
rollout.
15/9 Prob% increased. Although, application specific must have peaks fixes
have reduced to a manageable level assisted by replan.
30/9 - Current Peak load manageable, albeit higher than desired
29/10 - Post Weekend B/C PEAKs to support Model Office high. Manageable
but stretching teams - contingency used to extend a number of key staff
24/11 - Peaks remain high (see above)
42/1/10 - Resolving peaks as they arise and keeping on top of them. Do not
anticipate any impact to small volume pilot
8/2/10 - Closed. New issue raised.
Containment: Detailed plan produced;
Weekly review of progress against plan,
Close engagement with Security ops team,
Close engagement with delivery projects to ensure compliance to policy
Next Steps = Ongoing
09/03 Internal 1527001 audit completed and recommendations incorporated into
the remedial action plan, Plan progressing to plan,
Additional Pen Test, plans and respond to audit
23/06 - IS027001 audit observations & rectifications progressing well. Risk
Management deficiencies escalated to account level and corrective action
anticipated early July
31/7 - Corrective action plan for Risk Management deficiencies in place and are
being agreed with POL.
30/9 - 18027001 observations should be closed by 9 Oct 09. Evidence of risk
management process to be in place by mid Oct 09
29/10 - 1S27001 Risk Management process and supporting toolset approved by
POL. Target 18/12 to demonstrate that process is fully operational and
managed
24/11 no change
12/01 - 1027001 compliance due end of Jan 10.
8/2/2010 - Closed. Latest date for achieving 18027001 compliance
estimated as end of Feb 2010. New Risk opened.
FUJ00095073
FUJ00095073
Impact: Exit criteria not
achieved from test cycles
Impact: Acceptance Gateway 3
will not be achieved
Client
Ref
HNG-
xR0008
HNG-
xROO18
Project ‘Owner I
Caused by
I
Description——- —
HNG-x R1
Development
Richardson
Debbie DB
During Post Live
VOL testing, the
solution dees not
operate in
accordance with
the required
performance
targets (i.e.
Transactions per
second BAL/OSR)
(POL owner IT)
Cause: Complex transaction
system, with multiple
application layers
HNG-x
Migration
Butts Geoff HNG-X Rollout
fails in Pilot
leading to
Pilot/Rollout stop
(POL owner WR)
Cause: significant defects
are found during pilot,
I Containment Strategy-
Containment: Extensive performance modelling and testing prior to delivery to
test
Next Step = Monitor through testing
JW comment - We are now in testing - so should either kill this off, or re-word
around risk that we are not going to exit with required performance
7/7 Performance tests have been positive Incidents raised dealt with to date
However extended tests into the VOL cycle means this risk will persist until after
migration and VOL execution
31/7 - Horizon performance tested positive, residual risk around HNG-x
performance to be validated during Vol Testing
30/9 - No Change to Risk
29/10 - No change
24/11 Replan of Pilot has resulted in cycle 2 VOL testing completing at the end
of large volume pilot. Therefore an early decision will be needed during cycle 2
to give approval for start of large VOL pilot.
12/01 — No change
8/2/2010 - Volume test started 1/2/2010. Cycle 1 estimated completion is 17/2
1/3/10 - Proven that system can handle a high level of CAPO transactions
(design limit reached for 1 Hour with no problem) Scope of cycle one has been
completed. Some issues found in Pilot need to resolved before rollout
Containment:
Completion of test cycles,
Continuous review of technical approach
War room during pilot
09/03 - SV&l test reports showing positive results
23/06 - Risk may reduced following RV Mig cycle 3 (completes 23/09)
01/07 JW - Comment above is less relevant than positive results out of other
testing strands, notably V+I and SV+1 and acceptance through the RABs. The
containment is about ensuring residual risks at go live are understood, with
mmitigations/ work-around in place at that stage
29 - Pilot is reduced by 3 calendar weeks to accommodate replan and not
impact rollout end date. This reduces time fix must have defects prior to counter
rollout.
30/9 - CP to change migration schedule being impacted and to be released for
formal impacting within FS 30 Sep 09. PO reluctant to release rollout schedule,
need Branch Router details required by mid Oct 09.
29/10 - CP rejected. POL have released schedule. However, will be impacted
due to Post Weekend BIC planning
24/11 - Post wie BC planning has resulted in revised pilot and rollout schedule.
Revised plan already impacted by delays in branch router pilot and putting
revised schedule at risk
12101 — No change
8/2/2010 - All 12 Low Volume Pilot branches trading successfully. Next tranche
of external branches successfully migrated 4/2. Further 10 planned 11/2.
1/3/10 Key issue impacting on live estate (of 152 branches) is screen pauses
and loss of connection to DC. Several causes identified including, incorrect VPX
Server Cache setting (should be 25k actual value 272), possible linkage to
branch router config level, Further fixes to address other causes are included
FUJ00095073
FUJ00095073
Impact Description-—-
Impact: Development rework
impacting with extension of
rollout time scale
Impact: Software rollout is
stopped, customer
dissatisfaction
Client
Ref
HNG-
xRO059
HNG-
xRO062
Project
HNG-x
Owner
Cochrane
Infrastructure Vince
HNG-x
Migration
Butts Geoff
I Caused by
I Containment Strategy----------I
I
Description.
Following
migration of Data
Centre critical
elements of
Horizon fail. (POL
Owner WR)
Any further delays
to the HNGX
rollout result in
Fujitsu missing the
revised exit date
from the Wigan
and Bootle
Datacentres of
31/07/2010. (POL
Cause: Critical requirement
not met. Insufficient and/or
inadequate testing does not
pick up issue prior to go live
Cause: The current planned
rollout completion date of
21/04/10 slips.
within 01.08 release, these include BAL timeout and premature withdrawal of
Credit card to cause screen freeze. The VPX server change is being
implemented 1 M
Containment: Revised plan includes additional testing to ensure full coverage
and stability of solution
23/06: WR to speak to Vince Cochrane
30/6 - FJ to review areas that may benefit from additional validation to mitigate
risk. POL to support
2/9 - Additional BC weekend planned in RV schedule, which will mitigate this
isk
ris
29/10 - Twenty Four/Seven Data Centre support arranged for first three weeks of
live operation
24/11 - Continue monitoring live service.
1211 - No major issue encountered so far.
Containment: Site visits conducted 27/08/09 with Group Property to confirm
what decommissioning activities are required. FJ will look to minimise the
removal of networks/routers from the sites. Up to 3 week slippage in the plan
can be tolerated without an impact on the exit date.
29/10 - No Change
24/11 - Decommissioning PM engaged with estate management team. Scoping
of decommissioning activities confirmed and can be completed by end of July,
even when allowing for a 3 week delay to overall plan
12/1 - Moved start of small volume pilot by 8 days. Still gives enough time to
1
FUJ00095073
FUJ00095073
Impact Description---
Impact: Major service issue
Impact: Substantial additional
costs for Fujitsu in having to
keep additional Datacentres
open to run Horizon.
FUJ00095073
FUJ00095073
Client Project ‘Owner I I Caused by I Containment Strategy-------—I I---—-Impact Description
Ref Description I ol