RE: HNGX Programme Delays and Operational Issues
From
To:Bounds Gavin
Thanks Gavin.
Roberts David (LON22
I will speak to Pete as soon as he comes out of his meeting
FUJ00174290
FUJ00174290
Sent: Wed 07/04/2010 9:47:18 AM (UTC)
Ihave also spoken to Duncan about a comms plan for the customer and we are due to speak again in a while.
David
From: Bounds Gavin
Sent: 07 April 2010 10:24
To: Roberts David (LON22)
Subject: FW: HNGX Programme Delays and Operational Issues
As discussed.
Kind regards,
Gavin
From: Bounds Gavin
Sent: 06 April 2010 23:34
To: Tait Duncan
Cc: Wood Mike
Subject: HNGX Programme Delays and Operational Issues
Duncan,
The new POL CEO, Dave Smith, started today. Mike Young mentioned before he went away that if/when the new CEO
arrived he would likely be reaching out to Richard in his first few days in the role.
By way of a briefing for Richard, please review the following:
HNGX pilot was progressing well and moved into day 4 of high volume pilot (to 215 branches) on Thursday 25 March. 208
branches were successfully rolled out ~ a good outcome. Then began a catalogue of outages/issues as follows:
I BALIBDB issues Fri 26/3/10
ing 2:30
pxperienced 14:55
Fesulting in
hinavailability
across HNG
Fri 26/3/10
3:45,
pact, comments
Two HNG-X service outages affecting
all branches. The first of these
occurred at 12.30 p.m., when
branches reported loss of connection
to the datacentre. This was caused byI
the Branch Application Layer (BAL)
hanging. The BAL servers were
recycled to restore service by 1.45
p.m. At 2.55 p.m. there was a second
loss of service for 30 minutes when
there were problems with Node 1 of
the Branch Database.
POL verbally stated that the outages
were causing them considerable
problems with the NFSP (National FedI
of Sub-Post masters) who were
threatening to contact the press.
Current Status
configuration changes were made on the
latabase components to alleviate the
jssue and ensure a more rapid recovery
rom the outage. Oracle were engaged on
pp Priority 1 Incident and 247 diagnostics
as enabled.
JA recommendation was made by Oracle to
ipgrade to Patch set 3
1/4/10 that they would not allow High
Volume Pilot to move ahead unti
a. They had seen the findingsI
of an independent report on.
2 BAUBDB issues Bat27/a/10 _ Bat27/a10 1107 ING ‘Same issues experienced as on yacle Patch set 3 upgrade was completed]
ing 4:01 Friday 26/03, n Sunday 4" of April and the stability of
bxperienced fhe BALIBDB is being monitored. No Major
fesulting in Incidents in BAL/BDB have occurred since
unavailability he upgrade. Both BAL and BDB are
across HNG Josely monitored by all component
mners/support teams to ensure any
jssues are quickly identified. Further
nalysis is on-going with Oracle and other
improvements are being considered
Diient Meeting [on 2973/10 _ Mon 2973/10 The client stated at a meeting on
we now have this underway
and led by Rachel Daka, PSD
Ops Director of Retail,
FUJ00174290
FUJ00174290
‘our CS operations - giving supported by Business
them assurance that we were ‘Assurance
correctly resourced and had Ib. fixes were deployed on Sun
our events reporting under 414/10 — no recurrences of the
control Oracle issues, we continue to
», Fixes were deployed for monitor
the issues seen between 26/3
and 29/3 and followed by a
period of stability
Belfast Data___ fue 301/10 Ifhur 1/4/1002:08 Horizon, David Silitoe's team have been io update yet from David.
Centres suffered 20:54 HNG, investigating but don’t have a firm root
b power surge, POLFS and cause as of yet - our position for now
is impacted the POLMI is that we are working with the
Post Office National Grid to get to the bottom of
state. the cause.
‘The power surge resulted in
intermittent hardware failures which
resulted in Network Banking issues
being experienced on Thursday
morning (see below)
Account requested HNGX issues and CS stability’ be moved to red alert,
B Major IncidentsIThur 1/4/10 Ilhur 1/4/10
ere
pxperienced; 09:40
4:35 Horizon 1) Our overall conclusion is that 4) Following the remedial work to
) Horizon following the power issues on the change the C Blade and
letwork banking 14:41 30” March there were subsequent removing the P Blade 24
(CAPO) issues 5:12 issues identified as a hardware failure. (Hardware failures) and
This issue directly contributed to analysis of the data collected
2) MoneyGram Horizon and Horizon PCI transactions failures above we can conclude that we}
jssues NG Which affected Network Banking have resolved all known
(CAPO) across the Horizon estate. contributing factors to the
Horizon Network Banking
2)1 MoneyGram Agent out of 2 had issues experienced on the
received a corrupt Mapping File which 4 April. As a result of the
resulted in 50% of MoneyGram issue we have also been able
transactions failing this impacted both to identify a service
Horizon and HNGx. improvement which has been
implemented on the Firewall
which will aid our ability to
retum to service in a shorter
timeframe should we
experience issues in the future.
2) This was resolved by
restarting the service on
Ipppmws001 which forced a
reload of the FAD to
MoneyGram agent mapping.
Major IncidentsITue 6/4/10 __ffue 6/4/10,
ere
experienced
8:58 9:51 Horizon and 1) Debit Card Authorisation 1) Memory issue identified,
) Debit Card HNG Agents reported one of a known defect awaiting fix. The
uuthorisation resilient pair was down. workaround consists of a daily
lAgent failed in a Both agents required a re- reboot out of trading hours.
fesilient pair p9:17 9:58 Horizon boot. Business Impact wasI
that debit card and e-pay I2) RCA in progress and being
») Horizon transactions failed both in progressed with Networks
jetwork Banking}! 1:15 n-going Horizon Horizon and HNG
Issue
2) Horizon Network Banking I3) Incident still open and being
3) Proxy Radius issue affected about 20% investigated with Fujitsu
jerver issue of transactions failing networks and Cable and
bffecting ISDN Wireless
ites 3) Upto 10 ISDN out of 685
are failing to authenticate
which impacts their ability
to do online transactions in
Horizon
The above reflects the significant issues we have experienced over a short time span. Relationships with the POL leadership
are tense and we are clearly on the back-foot. Dave Smith (as in the CIO that has just retired from POL) takes with him a
maturity of approach and first hand experience of the original roll-out of Horizon, his counsel to Mike Young and the POL ET
was that there were many large issues back then and that these should be expected now ~ this is a loss to our cause.
However, we do still have a strong supporter in Andy Mclean, the POL Service Director ~ whilst not pleased with where we
are he advocates a logical cooperative ethic to overcome issues and is concerned by some of the more emotional reactions
he is seeing within POL.
Dave Smith (from Parcelforce) started today, and has met with the HNG-X POL Team. Our contacts tell us that he is likely to
take a more cautious stance towards re-starting the pilot. The National Federation of Sub-postmasters will not allow any more
of their members’ branches to be migrated to HNG- for the time being. The likely way forward is that POL will look to deploy
HNG-X to all their Crown offices first, to prove that the system works and to persuade the Federation to continue.
We are working to agree the criteria upon which we can move forward with the pilot, in the mean time we continue to
implement patches/fixes and monitor system performance.
hope this is useful, apologies for the length, but wanted to make sure you had the context/detail.
Kind regards,
Gavin
Gavin Bounds
Business Unit Director
Private Sector Division
Fujitsu
22 Baker Street, London W1U 3BW
FUJ00174290
FUJ00174290