FUJ00093028
FUJ00093028
To:
From: Allen Graham (BRA01)
Sent: Tue 16/02/2010 2:37:10 PM (UTC)
Subject: RE: Amber Alert Ref 1053 - Post Office (HNG-X) - Update Report
No problem. Will do.
Graham Allen
Application Services ~ Post Office Account
FUJITSU
RG12 8SN.
ss Please consider the environment - do you really need to print this email?
Fujitsu Services Limited, Registered in England no 96056, Registered Office: 22 Baker Street, London, W1U 3BW.
This e-mail is only for the use of its intended recipient. Its contents are subject to a duty of confidence and may be privileged. Fujitsu Services does not guarantee
that this e-mail has not been intercepted and amended or that it is virus-free.
From: Goldsmith Steve
Sent: 16 February 2010 14:36
To: Allen Graham (BRAO1)
Subject: FW: Amber Alert Ref 1053 - Post Office (HNG-X) - Update Report
Graham
Would really like an update on the BTS fix at the end of the week for a meeting I have with Roger Gilbert on Monday.
Ta
Steve
Steve Goldsmith
Director, Application Services, UK & Ireland
From: Amber Alerts
Sent: 16 February 2010 12:00
To: Amber Alerts; Armstrong Sue; Bounds Gavin; Butts Geoff; Cochrane Vince; Connor Darren; D'Alvarez Alan; Dunmore Phil;
Goldsmith Steve; Hughes Martyn; Jeram Peter; Keeling David; Kostuch Maz; Perek Barbara; Provoost Martin; Swain Caroline; Tait
Duncan; Wood Mike; Young Lester
Subject: Amber Alert Ref 1053 - Post Office (HNG-X) - Update Report
Fujitsu Services Limited, Registered in England no 96056, Registered Office: 22 Baker Street, London W1U 3BW. This email is only for the use of its intended recipient.
Its contents are subject to a duty of confidence and may be privileged. Fujitsu does not guarantee that this email has not been intercepted and amended or that it is virus-
free.
Amber Alert
There are now 30 customer-facing Post Office branches trading on HNG-X Release 1. A further 8 branches were successfully
migrated on 12/02/10. Release Authorisation Board approval was given on 11/02/10 to proceed to Medium Volume Pilot, dependent
FUJ00093028
FUJ00093028
on the implementation of the fix into live for the Branch Trading Statement issue. The fix has been signed-off in test and will be
implemented into live by 16/02/10 as a pre-requisite for the Medium Volume Pilot migrations to start 17/02/10. The first (Datacentre)
component of the 01.08 Maintenance Release was installed onto LST on 14/02/10.
Key challenges remaining are getting through the Medium Volume Pilot, and delays to Volume Testing cycle 1 due to software
issues. Connectivity problems were resolved over the weekend, but the latest forecast for completion of Cycle 1 tests is now
26/02/10. Additional issues from live, including another issue with the Branch Trading Statement (BTS) where printing the Trials
Balance corrupts the Final Balance Report, need to be accommodated within the 01.08 maintenance release if possible. This may
impact on the release timescales.
The additional technical resource identified for the Branch connectivity 3rd line support team has agreed to join the team this week.
Next Actions:
Completion of Medium Volume Pilot (migrations complete by 24/02/10).
Continued testing of first tranche of 01.08 maintenance release fixes.
Customer's current perception: Committed to continuing with the pilot and rollout as quickly as possible. There is concern about the
BTS problems given the high visibility of perceived “issues” with Horizon data integrity.
; Private Sector Division
Migration to HNG-x Solution
Project delay and associated costs.
Successful completion of small, Medium and High Volume HNG-X pilots...
FUJ00093028
FUJ00093028
4 Forecast Closure: 2010-03-09.
5 Progress/Current Actions:
2010-02-16
There are now 30 customer-facing Post Office branches trading on HNG-X Release 1. A further 8 branches were
successfully migrated on 12/02/10. Release Authorisation Board approval was given on 11/02/10 to proceed to Medium
Volume Pilot, dependent on the implementation of the fix into live for the Branch Trading Statement issue. The fix has been
signed-off in test and will be implemented into live by 16/02/10 as a pre-requisite for the Medium Volume Pilot migrations
to start 17/02/10. The first (Datacentre) component of the 01.08 Maintenance Release was installed onto LST on 14/02/10.
Key challenges remaining are getting through the Medium Volume Pilot, and delays to Volume Testing cycle 1 due to
software issues. Connectivity problems were resolved over the weekend, but the latest forecast for completion of Cycle 1
tests is now 26/02/10. Additional issues from live, including another issue with the Branch Trading Statement (BTS) where
printing the Trials Balance corrupts the Final Balance Report, need to be accommodated within the 01.08 maintenance
release if possible. This may impact on the release timescales.
The additional technical resource identified for the Branch connectivity 3rd line support team has agreed to join the team
this week.
Next Actions:
Completion of Medium Volume Pilot (migrations complete by 24/02/10).
Continued testing of first tranche of 01.08 maintenance release fixes.
Customer’s current perception: Committed to continuing with the pilot and rollout as quickly as possible. There is concern
about the BTS problems given the high visibility of perceived “issues” with Horizon data integrity.
2010-02-09
A further 10 branches were successfully migrated to HNG-X on 05/02/10. There are now 27 branches trading on HNG-X,
including 6 Model Offices. One branch failed to migrate on 05/02/10 due to the Migration button being locked. Work is
progressing to develop a fix for the Branch Trading Statement issues where Horizon transaction data is incorrectly
summarised. This needs to be rectified for larger numbers of branches to be migrated. The fix, together with monitoring
routines is scheduled to be developed and available to test by 12/02/10. It will then require 2 days of testing. The schedule
for Medium Volume Pilot has slipped and has been agreed with Post Office this afternoon, and will consist of :-
1. 10 branches migrating 11/02/10 to prove that the Fujitsu migration reporting database works correctly. Fixes for
reporting issues are in test and are scheduled to be signed-off by end of play on 11/02.
2. 60 branches migrated over 3 days from 17/02/10 to 19/02/10 (20 branches migrating each night) following
implementation of the Branch Trading Statement fix. Small volumes will be used to enable spot checks to be made in live on
the fix working correctly and to give an opportunity (if there are issues) to manually correct them before larger branch
volumes are migrated.
3. 80 branches migrated on 22/02/10.
4. 100 branches migrated on 23/02/10.
The large volume pilot is now scheduled to commence on 03/03/10 subject to satisfactory completion of the medium
volume pilot and will comprise 3 days of branch migrations. Two days of 275 branches to prove the maximum migration
rate during the rollout and 1 day of 110 branches on a Friday to prove weekend processes and support.
The success rate for installation of Branch Routers had improved to 86% by the end of last week, largely due to a fix being
implemented for the Slave/LAN issue. Work is in progress to schedule increased numbers of branch router installations to
make up time for delays to the rollout from January. This has entailed additional scheduling staff and engineers, and
working with Post Office to ensure that scheduling calls are not wasted.
Overall, the delays to the start of Medium Volume pilot have impacted the planned completion of rollout.
Account level discussions are in progress to determine the best way to proceed
2010-02-02
The 12 branches from the small volume HNG-X pilot continue to operate on the new application. The medium volume pilot
start has been deferred to start on 04/02/10 with the next 10 branches due to two sets of issues being found. Firstly, Coton
and Warwick branches had issues with producing their trading statements as a result of incorrect migration of Horizon
branch summary data. Secondly, a duplicate transaction issue was found at the Derby branch. Fixes for these issues need to
be in place for the next tranche of branches to migrate. The revised schedule for the medium volume pilot in the week
commencing 08/02/10 will be confirmed with Post Office by 04/02/10. A key success factor for the medium volume pilot
FUJ00093028
FUJ00093028
will be the ability to migrate 100 branches per night ahead of the high volume pilot.
The Vol rig build has now completed after a number of technical challenges relating to the database builds were overcome.
This has resulted in Cycle 1 of Volume Testing starting 2 days later than planned on 01/02/10.
The branch router rollout is continuing, and the installation success rate has improved to a daily average of 80%. The fix for
the Slave/LAN connectivity issue has been tested successfully and is starting implementation this week. Other technical
issues that have prevented successful branch router installations are being worked on by the 3rd line team to prevent a
significant backlog of branches building up. Challenges remain with scheduling installations. To address this — additional
schedulers have been recruited by 2E2, and statistics on the number of calls to branches to achieve a successful appointment
are being collected.
Next Action:
Develop fixes for live issues at Warwick, Coton and Derby and complete testing by 04/02/10.
Confirmation by 04/02/10 to progress with next 10 branches migrating to HNG-X.
Confirm content of first Maintenance Release (0108).
Continue with Branch Router rollout.
Customer’s current perception: Customer confidence has been affected by the live issues. It is important that the next
tranche of branches migrate successfully to rebuild confidence in the solution.
2010-01-26
The 12 HNG-X branches for Small Volume Pilot are now operational on HNG-X, the remaining 5 being migrated overnight
on 21/01/10. The Branch Router installation success rate improved last week, with 85% being achieved on I day. On
average for the week - the success rate was 76%, and over 1,000 branches have now had a Branch Router installed.
Over the weekend there was an issue that caused a maximum of 138 branches to be unable to trade. It was caused by an
error in the Tivoli software distribution mechanism from a change made to support Weekend D. It was exposed because a
decision was taken to send and commit the BRTR-Migration package required prior to branch router installs to all the
remaining 10,173 branches. It was successfully committed overnight to 9,165. The commit process should have stopped at
8 a.m. ona trading day but didn't. The functionality was tested, but this issue was not found (it would only have been
discovered from doing a volume test of software distribution). A root cause summary has been provided to Post Office but
further questions have been raised about what the change was, and why the decision was taken without the programme's
involvement to target all branches rather then the next week of installs. A further response will be provided to the customer
later today (26/01/10). At close of play last night, 2 branches were still reported as having problems.
A further 10 branches are targeted for migration on 28/01/10 subject to some issues being resolved. The remainder of the
medium volume pilot branches are scheduled to migrate from 01/02/10, dependent on a number of fixes being tested and
implemented into live by 31/01/10.
The impact of the delays on the overall rollout plan is being assessed.
There is continued focus on the Branch router rollout, which ramps up significantly this week. A series of fixes to address
support issues for branch router was implemented into 50 pilot branches on 17/01/10.
There is a high focus on ensuring that key issues are resolved for the start of Medium Volume Pilot.
Next Action:
RAB Checkpoint Medium Volume Pilot 27/01/10.
Start of Medium Volume Pilot 28/01/10.
Customer’s current perception: Very concerned about the impact of the weekend incident where a number of branches
could not trade, and on the rollout schedule slipping.
2010-01-19
Status 18/01/10 at 2100hrs:
Last week, the programme started the Small Volume pilot for HNG-X at live Post Office branches. Of the 2 planned for
13/01/10, 1 (Coton) successfully migrated to HNG-X and has been trading since with no problems. The other branch
FUJ00093028
FUJ00093028
successfully auto regressed due to an issue with the NT Security Policy on one of the counters. This issue has now been
corrected for all the small volume pilot branches and a fix is being worked on to apply to the Horizon estate prior to the
Medium Volume Pilot. On 14/01/10, 3 branches successfully migrated to HNG-X. On 15/01/10, 3 out of 8 branches
migrated. Several issues have been identified for resolution by 19/01/10 to enable a decision to be made by 19/01/10 on
whether or not to attempt the migrations again on 20/01/10. These include making some changes to the Branch Database to
overcome PinPad issues, reviewing the Friday overnight schedule between Reference data and Help Data, incident
management and escalations processes not working effectively and ensuring that the backup process for counters to
download reference data functions. Key to this was the fact that there was a delay between the decision to regress 2 of the
branches manually and the completion of the regression, resulting in trading time being lost. Steps are being taken to a)
ensure management is involved in any decisions on regression due to technical issues and b) confirming the regression time
per branch and counter.
There is continued focus on the Branch router rollout, which ramps up significantly this week. A series of fixes to address
support issues for branch router was implemented into 50 pilot branches on 17/01/10.
There is a high focus on ensuring that key issues are resolved for the start of Medium Volume Pilot.
Next Action:
Decision to proceed with migration for remaining 5 Small volume pilot branches — 19/01/10.
Release Authorisation Board (RAB) checkpoint 21/01/10 for proceeding to Medium Volume Pilot.
Rollout of Branch Router support fixes following successful pilot (Branch router only branches) — 22/01/10.
Start of Medium Volume Pilot on 25/01/10, subject to confirmation by RAB.
Customer’s current perception: Confidence was impacted by the issues over the weekend.