POL00248910
POL00248910
Sent:
To:
Ce:
UKGH
Alisdair Camer
Subject: RE: Horizon Outage
I don’t know — quick response would be; a) its a combination of actually low amount of transactions at 11AM ona
Sunday (I could find out how many actual transactions but I’m trying to get the team focused on root cause currently
so I’d rather wait) b) Postmasters waiting before calling as Horizon being out is unusual so they may wait to see if it
comes back.
R
From: Mcinnes, Tim - UKGI [mailt:
Sent: 02 May 2017 11:02
To: Rob Houghton
Cc: Parker, Nick - UKGI; Callard, Richard - UKGI; Martin Edwards; Alisdair Cameron
Subject: RE: Horizon Outage
Rob,
This is great and of course thanks for your time on the phone earlier also.
It all makes sense (within reason, given my IT knowledge!) but if we have any questions I’ll come back to you.
One quick query though. If >5,000 branches were supposed to be open and the system went down, why do you think
only 177 calls went in to the NBCS / Atos IT Service desk (1 would have thought many more would contact support if
their systems weren’t working)?
Thanks,
Tim
From: Rob Houghton [ma
Sent: 02 May 2017 09:58 _
To: Mcinnes, Tim - UKGI ¢
Ce: Parker, Nick - UKG!
Martin
Alisdair Cameron ¢
Context: The issue started at approx. 11:40 Sunday and was resolved approximately 55 minutes later. The business
impact was that all 5107 Branches that were due to be trading at the time of the outage lost service. This resulted in
around 120 calls to the NBSC and around 57 calls queuing on the Atos IT service desk.
Three areas that support the need for investment:
- Predictive capability — need for funding to build the Operational Command Centre and Security Operations
Centre (as an aside — we had an interesting “Hack” attempt from Prague as well on Friday — will give separate
details on that if you like); this would enable us to predict failure events rather than react to events that occur
— moving into a predictive zone is key for retail and online businesses
- Ageing Infrastructure and Software — root cause is not yet known but we are way outside of our risk appetite
on both the software (supporting old Horizon) and infrastructure on which Horizon runs — the Horizon Data
Centre Refresh, Thin Client and Cloud migration activities were core funding asks
POL00248910
POL00248910
- Resilience — the system and software should FAIL OVER to a secondary system anytime failure occurs so you
can run continuous service. Downtime of an hour in the online world is a lifetime. The Cloud Migration and
Operational Command Centre both support this ambition.
Hope this helps as per conversation — hastily put together so shout if not useful?
Rob
From: Mclnnes, Tim - UKGI [mailt:
Sent: 02 May 2017 09:33
To: Rob Houghton
Cc: Parker, Nick - UKGI; Callard, Richard - UKGI
Subject: Horizon Outage
Rob,
Conscious you’re probably snowed but when you have / someone on your team has a moment it would be really
useful to get a few bullet points (or a 5 minute conversation) to understand a bit more about the Horizon outage on
Sunday. As I’m sure you’re aware we are still having a number of conversations within Government about POL
funding and where we can add colour to some of the points we are making (e.g. need for IT investment) it’s really
useful.
Thanks,
Tim
Tim McInnes I Assistant Director
UK Government Investments
1 Victoria Street I London I SW1H OET
This email and any files transmitted with it are intended solely for the use of the individual(s) to whom they are
addressed. If you are not the intended recipient and have received this email in error, please notify the sender
and delete the email. This footnote also confirms that our email communications may be monitored to ensure
the secure and effective operation of our systems and for other lawful purposes, and that this email has been
swept for malware and viruses.
JIGS SISI CII SISIOISIOIC ISI GICICICI GI SICIGICI SICK ICICI aA ICICI ICR I IC Ik aR CC FoR I ICA RoR ACCA DOR aR CAC RoR aR CAC
This email and any attachments are confidential and intended for the addressee only. If you are not the named
recipient, you must not use, disclose, reproduce, copy or distribute the contents of this communication. If you
have received this in error, please contact the sender by reply email and then delete this email from your
system. Any views or opinions expressed within this email are solely those of the sender, unless otherwise
specifically stated.
POST OFFICE LIMITED is registered in England and Wales no 2154540. Registered Office: Finsbury Dials,
20 Finsbury Street, London EC2Y 9AQ.
FSS SISI CII SI SISIOISIOCI SI GICICIGISICISIGIOCI I SICICII GI ICICICI III I ICR I ACC ICR I a 31 1 8 a CAR aR a ACS RR aR Cao
POL00248910
POL00248910
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com