POL00241374
POL00241374
From:
Sent:
To: Paula Vennel
Subject: Re: Horizon issue
Thanks Paula. The important thing is that we responded speedily to the issue at hand. Good luck with the Fed
tomorrow and speak soon.
Regards.
Tim
Sent from my iPhone
On 9 May 2016, at 22:33, Paula Vennells < wrote:
Tim, thank you for this.
I'm disappointed as we are usually better than this. I'm sorry and take full responsibility.
As I'm at the Fed conference tomorrow, I've a note to go to colleagues tomorrow am (not helpful
to send tonight!), to pick up the lessons quickly.
As always, I'm grateful for your correct challenge and the helpful manner in which it is made.
Best wishes,
Paula
Paula Vennells
Chief Executive
Post Office Ltd
Sent from my iPad
On 9 May 2016, at 21:15, Tim Franklin
wrote:
Thank you for the further information Alasdair. These sound like good additional
prudent steps. Much appreciated.
Kind regards.
Tim
Tim Franklin
POL00241374
POL00241374
Sent from my iPhone
On 9 May 2016, at 21:08, Alisdair Cameron <
wrote:
Thanks Tim. I appreciate both your point and your very gentle way of
making it...
We will be quicker in future.
Many transactions will simply not have been far enough advanced and
will therefore never have been recorded or completed. Where a
transaction was in mid-flow, on recovery of the system, the agent should
be offered a recovery screen. If she or he follows the questions, the
transaction should be completed successfully.
We had a very small weekend incident a few months ago and no agent
impact was noted.
Given the scale of this incident, I have asked the relevant back office
team to review the transactions where a recovery screen was offered and
to make sure that no issues are left unresolved or unbalanced.
Does that cover your question?
Many thanks Al
Sent from my iPad
On 9 May 2016, at 20:59, Tim Franklin
wrote:
Hi Al and thank you for the update. I have deliberately not
copied the rest of the board into this note.
Although the update is very helpful, it is very late. I was
caught out earlier today when asked a question about the
issue by a KPMG partner. I was embarrassed to say that I
knew nothing about it and then got my update from the web
and media reports. I fully appreciate that there is a lot to do
when things like this happen, but please can we ensure that
someone in Comms is on point to brief the board swiftly
when coverage is as wide as it has been here.
In terms of the issue at hand, there is clearly an opportunity
for this to be linked to Sparrow. How confident are we that
reconciliation will not be an issue in this case and that SPMs
will be able to get proper timely support to ensure that
discrepancies are quickly identified and resolved?
Kind regards.
Tim
Sent from my iPad
Begin forwarded message:
Fro
Date: 9 May 2016 at 20:13:40 BST
To: Tim Parker
Alwen Lyons
Alwen Lyons
Subject: Horizon issue
Dear All
Please see below a message sent on behalf of
Al.
Thanks
Alwen
Dear All
We are receiving media coverage
over a failure of the Horizon
system that caused queues and
service disruption across many
branches this morning. It will be
carried in the newspapers
tomorrow.
From 9am today a significant
POL00241374
POL00241374
number of branches were unable to
enter transactions on Horizon or
had the transactions rejected.
A recovery exercise - re-booting
the system - was carried out at
9.45am and branches started to
return to service from 10.10am,
with full service restored from
10.30am. No further service
disruption has been experienced
and none is expected.
Horizon has been operating on its
primary server for the last year
without issue.
We were concerned, earlier in the
year, around the robustness and
availability of the back-up,
secondary server capability
supporting Horizon. This has been
improved and required testing as
part of our business continuity
procedures.
Such tests are carried out after
significant planning, out of
standard hours and with the full
awareness of the business.
On Saturday 30th April, Fujitsu
moved the Horizon service from
the primary to the secondary
server. It ran without incident for a
full week. On the evening of
Saturday 7th May the service was
transferred back to the primary
server. It ran through Sunday
successfully.
However, when transaction
volumes increased this morning,
the primary server became unstable
and required a re-boot. This was
successful.
POL00241374
POL00241374
POL00241374
POL00241374
There are a few points to make:
- this is wholly unacceptable
- we do not fully understand the
issue that led to a re-boot being
required and the priority is to
identify and resolve the weakness
- we also have to learn lessons
around the volume testing of
change, which was demonstrably
inadequate in this case
- we will not undertake any further
business continuity exercises on
Horizon until we are completely
confident that this issue has been
resolved
- we have escalated the issue and
our disappointment with Fujitsu
and will be following that up with
a formal contractual
communication
- we will undertake a careful
review to ensure that postmasters
are not caught out with incomplete
or unbalanced transactions
I will of course keep you informed
of any further developments. A
formal review of lessons learnt will
be carried out by the Group
Executive.
Please let me know what other
information you need. Many
thanks,
Al
JSG SSIS CII SI SISICICICR I IGICICICIGISICICI IR ICICI ICR A ICC oR
FSS SIS CII SISIGICICC I SIGICICICIGISICICIC I AIC I CK aR
This email and any attachments are confidential
and intended for the addressee only. If you are
not the named recipient, you must not use,
disclose, reproduce, copy or distribute the
contents of this communication. If you have
received this in error, please contact the sender
by reply email and then delete this email from
your system. Any views or opinions expressed
within this email are solely those of the sender,
unless otherwise specifically stated.
POST OFFICE LIMITED is registered in
England and Wales no 2154540. Registered
Office: Finsbury Dials, 20 Finsbury Street,
London EC2Y 9AQ.
SIO SR SISI ICICI SI SICICIOI I ICICI ICRI ICICI I A ICC OR aR CAC oR ak
ERR RR COR RR A RR aOR RR a aR
POL00241374
POL00241374