POL00241349
POL00241349
From: Alisdair Cameroni__ GRO.
Sent: Mon 09/05/2016 7:09:40 PM (UTC)
To: Angela Van-Den-Bogerg,
Subject: Fwd: Horizon issue - urgent
Angela, see below. this happened at a weekend a couple of months ago and at much greater volume today. I
have promised that we will ensure postmasters don't lose out through stuck or missed transactions. Can you
revert tonight with your knowledge of whether any issues happened last time, whether we have had this sort of
problem before and how we will approach delivering on my promise. Paula needs a brief before she stands up at
the Fed conference tomorrow and may well look at you for answers! Thanks Al
Sent from my iPad
Begin forwarded message:
From: <alisdair.cameron:
_I Martin George
_} <david.hussey
ane MacLeod <jane.macleodi
, Chris Broe <Chris. Bro:
Mark Davies <mark.r.davie:
<Sharon.Gil
evin.gilliland
Subject: Horizon issue
Alwen, please send this note straight onto the Board, as discussed. Thanks Al
We are receiving media coverage over a failure of the Horizon system that caused queues and
service disruption across many branches this morning. It will be carried in the newspapers
tomorrow.
From 9am today a significant number of branches were unable to enter transactions on Horizon or
had the transactions rejected.
A recovery exercise - re-booting the system - was carried out at 9.45am and branches started to
return to service from 10.10am, with full service restored from 10.30am. No further service
disruption has been experienced and none is expected.
Horizon has been operating on its primary server for the last year without issue.
We were concerned, earlier in the year, around the robustness and availability of the back-up,
secondary server capability supporting Horizon. This has been improved and required testing as
part of our business continuity procedures.
Such tests are carried out after significant planning, out of standard hours and with the full
awareness of the business.
POL00241349
POL00241349
On Saturday 30th April, Fujitsu moved the Horizon service from the primary to the secondary
server. It ran without incident for a full week. On the evening of Saturday 7th May the service
was transferred back to the primary server. It ran through Sunday successfully.
However, when transaction volumes increased this morning, the primary server became unstable
and required a re-boot. This was successful.
There are a few points to make:
- this is wholly unacceptable
- we do not fully understand the issue that led to a re-boot being required and the priority is to
identify and resolve the weakness
- we also have to learn lessons around the volume testing of change, which was demonstrably
inadequate in this case
- we will not undertake any further business continuity exercises on Horizon until we are
completely confident that this issue has been resolved
- we have escalated the issue and our disappointment with Fujitsu and will be following that up
with a formal contractual communication
- we will undertake a careful review to ensure that postmasters are not caught out with
incomplete or unbalanced transactions
I will of course keep you informed of any further developments. A formal review of lessons learnt
will be carried out by the Group Executive.
Please let me know what other information you need. Many thanks,
Al
Sent from my iPad