FUJ00232704
FUJ00232704
SECURUITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
re)
FUJITSU
19 October 2020 and 21
Incident Date: Fujitsu Refs: INC6469861
October 2020
Version: 3.0 Problem Record: PRBO048676
Owner: Piotr Nagajek ; Peak Record: N/A
Status: Issued Change: N/A
MAJOR INCIDENT REPORT - Fujitsu
Network Issue 19 10 2020 & 21 10 2020
INCIDENT DETAILS
Organisation Managing the Major Customer / Account Name / Name of the Major Incident
Incident Service Line Manager
Fujitsu Post Office Account, Fujitsu
NOC MIM Team Post Office Ltd. Matthew Hatch
On Monday, 19" of October, Fujitsu observed an Internet connectivity
issue, affecting the traffic since 16:17. The issue was affecting all Fujitsu
POA services using the Internet. Fujitsu NOC MIM Team investigated the
issue. Restoration of affected services started at approx. 17:30. The HSI
(High Speed Internet) router issue was confirmed to be the cause of a loss
of Fujitsu VPN, UMI and corporate Internet access. TCY02’S interfaces
shut down for an unknown reason, in addition a commissioning error with
CenturyLink meant that resilient routes were not available, resulting in the
loss of internet connectivity for some customers. This was diagnosed as
an error on the router by Juniper, which would be fixed by a planned
restart.
3 A recurrence of the router failure occurred at 11:51 BST on Wednesday,
Executive Summary 21st of October. It affected the same services as on Monday. Fujitsu NOC
MIM Team investigated the router on site and failed over the traffic, which
restored the connection at approx. 12:43. All affected services started to
recover at the same time. The team followed the recommended fix
supplied by Juniper and identified that the TCY02 chassis hardware
required a replacement.
The hardware was replaced at 23:30 BST on the 21st October and was
confirmed at 1:10 BST on the 22nd October the router was restored and
health checks performed to ensure all services were green.
The firmware upgrades and testing on the Fujitsu High Speed Internet
platform have been successfully completed on the 10° and 12" of
November.
On Monday, 19th of October since approx. 16:20, Fujitsu observed a
network connectivity issue. As a result, the entire Internet traffic was
affected which resulted in a loss of some POL services, which use Fujitsu
Internet. Fujitsu POA support and service teams also lost access to the
secure Fujitsu VPN (remote access) as a direct result.
ieldent Summary For the detailed impact of the incident, please refer to Services Impacted
and Overview of Service Impact 19.10.2020 sections of this report.
Fujitsu POA raised multiple high priority incidents for the affected services
SECURITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
Page 1 of 20
FUJ00232704
FUJ00232704
SECURUITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
re)
FUJITSU
19 October 2020 and 21
Incident Date: Fujitsu Refs: INC6469861
October 2020
Version: 3.0 Problem Record: PRBO048676
Owner: Piotr Nagajek ; Peak Record: N/A
Status: Issued Change: N/A
from 16:28. For details of raised incidents, please refer to the Incident
Timeline. Investigation with POA technical resources was initiated via
bridge call at approx. 16:50, but with difficulties as most Fujitsu users are
working from home due to COVID-19 and were affected by no access to
the VPN due to the network issue.
During the Fujitsu POA bridge call, it was confirmed with POA Networks
Team, that we were facing a major network connectivity issue and
suspected onward Internet connectivity traffic has not failed over.
Acknowledged that the PODG file transfer is also impacted - POA UNIX
Team started to investigate this.
During the POA internal tech bridge call, it was confirmed that Banking,
Card Payments, EPOSS, Self-Service Kiosk service remain unaffected.
Some services failed over as expected.
At approx. 17:30, the POA users observed that the connection began to
be restored and affected services started to recover. Fujitsu were able to
resume the uploading of the PODG files at 17:31 following a manual
intervention by technical support.
At 17:37, Fujitsu Core Network and NOC ISP&DSL Teams confirmed to
Fujitsu POA that at 16:17, the HSI router in Fujitsu TCYO2 location
suffered an event causing it to stop responding, which affected multiple
Fujitsu customers. Fujitsu POA were informed that Fujitsu NOC Major
Incident Management Team is managing and investigating the issue since
observing the events on the router.
POA monitoring teams confirmed Track & Trace was the last of the
services to recover; this went green in HORIce (Fujitsu Monitoring Tool) at
19:20. This was owing to a major incident in Royal Mail Group domain
related to the business interface gateway that started at approx. 16:00 and
was confirmed as not related to the Fujitsu Network Incident.
A recurrence of the router failure occurred at 11:51 BST on Wednesday,
21st of October. It affected the same services as on Monday. Please see
details in Overview of Service Impact 21.10.2020 section of this
document. Fujitsu NOC MIM Team investigated the router on site and
failed over the traffic, which restored the connection at approx. 12:43. All
affected services started to recover at the same time.
The team followed the recommended fix supplied by Juniper and
identified that the TCY02 chassis hardware required a replacement.
The hardware was replaced at 23:30 BST on the 21st October and was
confirmed at 1:10 BST on the 22nd October the router was restored and
health checks performed to ensure all services were green.
The following actions were taken for resolution:
1. Correct the commissioning error with the CenturyLink connection
2. Source replacement Hardware for failing component (router
chassis)
SECURITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
Page 2 of 20
re)
FUJITSU
Incident Date:
19 October 2020 and 21
FUJ00232704
FUJ00232704
SECURUITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
POL Refs: INC0296097
Fujitsu Refs: INC6469861
October 2020
Version: 3.0 Problem Record: PRB0048676
Owner: Piotr Nagajek Peak Record: N/A
Status: Issued ‘Change: N/A
3. Install replacement chassis and re-establish internet connections
All the above actions were completed by 01:23 BST on 22nd October.
The firmware upgrades and testing on the Fujitsu High Speed Internet
platform have been successfully completed on the 10" and 12" of
November.
Generic FCN Internet access:
« DXI Internet access
* Bank of Ireland (PODG)
«Fujitsu VPN access
e British Gas/Payzone
« = Smart card
« Smart Meter
e Drop & Go transactions
e — Ingenico (Card transactions unaffected)
« GWS
o DVLA,
o FMCC/FMCV
Services Impacted o Health Lottery
o HPBB
o National Express
« cGl
« BWS
o Postal Guaranteed Delivery Date service (PGDD)
o Post Office Card Account service (PCAI)
o Bank Account Checker (BKAC)
« PODG
« CapitalOne
* CDP transactions
e HORIce monitoring tool access
ase ee ond Hseliee HSI router in Fujitsu TCYO2
items)
SECURITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
Page 3 of 20
FUJ00232704
FUJ00232704
SECURUITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
re)
FUJITSU
19 October 2020 and 21
Incident Date: Fujitsu Refs: INC6469861
October 2020
Version: 3.0 Problem Record: PRBO048676
Owner: Piotr Nagajek Peak Record: N/A
Status: Issued Change: N/A
2 problems :
Aratyale or one - A commissioning error with CenturyLink meant that resilient
routes were not available, resulting in the loss of internet
connectivity for some customers.
- A hardware issue requiring the chassis to be replaced on TCY02.
The HSI router in Fujitsu TCY02 suffered an event causing it to stop
responding, which resulted in a loss of Fujitsu VPN, UMI and corporate
Internet access.
Resilience is dependent on both CenturyLink and Cogent accepting all
Fujitsu’s public prefixes. The investigation identified that due to a
commissioning/configuration error, some of Fujitsu’s public prefixes were
Root Cause Analysis not being accepted by CenturyLink.
The result of this was that routes that should have been available via
CenturyLink to and from the internet were not.
The net result of this was that when the interfaces on TCY02 shut down,
customers who had a dependency on the subset of routes only available
via TCY02 lost Internet connectivity.
A hardware fault on TCY02 router chassis requiring a replacement.
The TCY02 router's interfaces were restored without intervention and
normal functionality resumed on Monday, 19" of October at 17:30.
POA UNIX removed the lockfiles on the PODG manually to allow the file
transfers, rather than waiting for the automated process to run.
Fujitsu NOC MIM Team expedited the planned changes and failed over
the traffic to the resilient path, which restored the connection on
Wednesday, 21* of October starting from 12:43.
The following actions were completed on the 21% October to restore and
Corrective Actions resolve the root cause.
- Correct the commissioning error with the CenturyLink connection
- Source replacement Hardware for failing component (router
chassis)
- Install replacement chassis and re-establish internet connections
All the above actions were completed by 01:23 on 22nd October.
The firmware upgrades and testing on the Fujitsu High Speed Internet
platform have been successfully completed on the 10° and 12" of
November.
SECURITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
Page 4 of 20
re)
FUJITSU
19 October 2020 and 21
October 2020
Incident Date
Version: 3.0
Owner: Piotr Nagajek
Status: Issued
SECURUITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
FUJ00232704
FUJ00232704
POL Refs: INC0296097
Fujitsu Refs: INC6469861
Problem Record: PRB0048676
Peak Record: N/A
Change: N/A
Start - End Date Of Incident
Time of Service Outage
Time Service Restored
19.10.2020 — 16:17-17:30
21.10.2020 -— 11:51-12:43
19.10.2020 - 1 hr 13 min
21.10.2020 - 52 min
19.10.2020 — 17:30
21.10.2020 — 12:43
RELATED REFERENCES
Master Incident Problem Record No of Linked Incidents
19.10.2020 - INC6469861 19.10.2020 - 12
PRB0048676
21.10.2020 - INC6485885 21.10.2020 — 4
Previous Occurrence MTTR Action required to Close
N/A N/A Completion of the MIR Actions
Complaints & Alerts Reference ICT Continuity Plan
Number(s) Ratersnce Change Record Number
N/A N/A N/A
TEAM MEMBER DETAILS
Role Team Members
Sonia Hussain — POA Head of Online Services
Service Line Lead(s)
. Matthew Hatch, Rajaramana Ravula
Major Incident Manager - Mandatory
Fujitsu NOC MIM Team
Resolver - Mandatory
SECURITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
Page 5 of 20
FUJ00232704
FUJ00232704
SECURUITY CLASSIFICATION
foe) COMMERCIAL IN CONFIDENCE
FUJITSU
POL Refs: INC0296097
19 October 2020 and 21
Incident Date: Fujitsu Refs: INC6469861
October 2020
Version: 3.0 Problem Record: PRBO048676
Owner: Piotr Nagajek ; Peak Record: N/A
Status: Issued ‘Change: N/A
Problem Manager — Case by Case basis
Matthew Hatch
Service Delivery Manager — Case by
Case basis Steve Bansal — Senior Service Delivery Manager
Others — Case by Case basis Piotr Nagajek
INCIDENT TIMELINE
Date Time Activity
16:17 I The HSI router in Fujitsu TCY02 suffered an event causing it to stop responding
19.10.2020 where POA is affected
16:17 I Fujitsu POA lost access to the VPN and all the Internet traffic was impacted which
result in loss of some POL services which use FJ Internet
16:23 I Fujitsu NOC ISP&DSL Team raised a P1 incident INC6469650 for Loss of Fujitsu
VPN
16:28 I Fujitsu POA raised a P1 incident INC6469656 for GWS errors for the National
Express service
16:29 I Fujitsu NOC ISP&DSL assessed initial impact - Fujitsu staff working remotely
have lost VPN access, staff in the office have lost all internet connectivity. The
issue was affecting multiple accounts
16:32 I Fujitsu POA raised a P1 incident INC6469675 for the CDP Interface down
16:39 I Fujitsu POA raised a P1 incident INC6469708 for load balancers issue - associate
services with pit 1nlb202 and pit1nlb201 are inaccessible
16:50 I POA DM Team started an internal bridge call and getting hold of the technical
resources
17:00 I Fujitsu POA investigation with technical resources continued with difficulties as
most users are working from home due to COVID-19 and are affected by the VPN
issue.
17:08 I Fujitsu POA raised incident INC6469806 for connection issues affecting Postal
Guaranteed Delivery Date service (PGDD), Post Office Card Account service
(PCAI) and Bank Account Checker (BKAC) service.
SECURITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
Page 6 of 20
FUJ00232704
FUJ00232704
SECURUITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
re)
FUJITSU
19 October 2020 and 21
Incident Date: Fujitsu Refs: INC6469861
October 2020
Version: 3.0 Problem Record: PRBO048676
Owner: Piotr Nagajek ; Peak Record: N/A
Status: Issued ‘Change: N/A
17:11 I Fujitsu POA raised a P3 incident INC6469817 for Branch Access and Collect and
return showing critical on HORIce
17:12 I Fujitsu POA investigation continued, discussed with POA Networks Team possible
network connectivity issue and suspected onward Internet connectivity traffic has
not failed over. Acknowledged that the PODG file transfer is also impacted — UNIX
Team investigating
17:14 I Fujitsu POA raised a P3 incident INC6469826 for Track and trace Showing Critical
in HORIce
17:15- I Fujitsu POA investigation with technical resources continued. Fujitsu POA SMC.
17:30 I tried reaching Fujitsu NOC ISP&DSL team to get the details, but all NOC.
resources were engaged in the MIM bridge, trying to resolve the issue
17:17 I Fujitsu POA OOH Duty Manager Rajaramana Ravula joined the POA internal
bridge call
17:19 I Following a manual removal of lockfiles by the UNIX Team, PODG file transfer
commenced
17:23 I Fujitsu POA SMC raised a master incident for multiple services affected by the
network issue — P1 INC6469847
17:32 I Fujitsu POA SSC support confirmed VPN access restored
17:33 I Fujitsu POA logged incident INC0296097 with POL IT DSD for Internet Traffic
Usage (GWS, CDP & HBS services affected)
17:37 I POA OOH DM reached NOC resource who informed Fujitsu POA that NOC MIM
is still investigating the network issue under Fujitsu NOC major incident
INC6469650, however, connectivity is recovering
17:42 I On the POA bridge call, it was confirmed that no Card or Banking transactions
were affected
17:49 I POA SMC confirmed Branch Access and Collect and Returns turned to Green but
still, Track & Trace is showing critical
17:58 I Fujitsu POA SSC resource confirmed the Track & Trace alerts were occurring
BEFORE the network incident. The first alert was observed at 14:29:50. Agreed
this need to be investigated separately
18:05 I Update from Fujitsu NOC MIM: The issue has now been mitigated and services
should now be restored. Bridge call is going on with technical to understand the
cause, further updates will be supplied by the MIM team and RFO will be provided
by MIM team
18:12 I POA SMC informed the Track & Trace alerting amber in HORIce
18:15 I Update from Fujitsu NOC Operations Manager: UK - Live Services - P1 -
INC6469650 x2 18:00 BST. Loss of Fujitsu VPN, UMI and corporate internet
access. Confirmation has been received that services are restoring and customers
are able to access external websites. Further investigations into the root cause are
ongoing by the FJ Networks team. Next update when more information is available
SECURITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
Page 7 of 20
re)
FUJITSU
Incident Date:
Version:
Owner:
Status’
19 October 2020 and 21
October 2020
3.0
FUJ00232704
FUJ00232704
SECURUITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
POL Refs: INC0296097
Fujitsu Refs: INC6469861
Problem Record: PRB0048676
Piotr Nagajek ; Peak Record: N/A
Issued
Change: N/A
18:28
POA SSC resource confirmed regarding Track & Trace issue: Track & Trace
reporting is near real time - no impact on counter, the messages are queued and
sent to Royal Mail / Parcel Force. So if people check the barcodes - would be
delayed. The TT barcodes will be processed eventually
19:13
POA Unix confirmed that BIG (PODG) issue was sorted around 19:00. Related
TfSNow incident: INC6470271. Summary of the issue: After the network problems
in the afternoon, we had issues with the BAP_FILE_BIG PODG route. The route
was running very slowly with a single file taking 20-30 mins to transfer and be
renamed. This seemed to recover at approx. 19:00 with the backlog of files all
transferred at that point. We believe this was an issue at the BIG end
19:19
POA SMC confirmed Track & Trace returned to green in HORIce
19:50
Update from NOC Ops Manager: HS/ router at TCY02 had an event issue at
router 16:17 causing it to stop responding and caused a loss of service. When the
event occurred, some customer traffic rerouted via LD6 link instead but not all
services rerouted correctly. Services began to restore at 17:29 without intervention
from the Fujitsu third line engineers. Investigations are ongoing with Fujitsu third
line engineers and hardware supplier Juniper to understand the cause of the
outage. Juniper case has been logged and they have been provided with the event
logs, all technical information. The NOC will perform additional monitoring on the
HSI routers and escalate any potential issues immediately to the third line
engineers.
22:00
NOC update: AMCS third line support have confirmed that the case with 3rd party
Juniper has been accepted as a Severity 1. Juniper have performed initial
investigation and requested additional information from the device, which is being
supplied by AMCS third line support. Extended monitoring is now in place and will
continue overnight with AMCS third line support being contacted immediately any
performance issues are detected and a restart of the TCY02 router will be
undertaken. Further logs files would also be collected at this point to assist with
Juniper investigations. Further hourly status updates will be provided by AMCS
monitoring team.
20.10.2020
09:20
NOC update: UK — AMCS - P1 - INC6469650 x5 09:10 BST — Services have been
stable all night, Fujitsu are actively working with vendor to understand root cause.
Vendor has recommended restarting some services to aid diagnostics (Daemon
restart) which is being discussed — likely time 0100 Wednesday morning (this will
be confirmed)
There are Indications of a hardware issue but more to be done to confirm or
otherwise hence the further diagnostics that are required from the restart.
Fujitsu continue to work with juniper to understand the root cause and develop an
action plan
Full active monitoring continues, next update when more information available.
21.10.2020
11:51
A recurrence of the HSI router failure occurred
11:54
Fujitsu NOC MIM Team informed: Loss of Corp VPN affecting user's in the NOC -
contacting MIM and escalating to ISP/DSL team
12:00
Fujitsu POA SMC raised a P11 incident INC6485885 for CDP critical alerts
12:00
Fujitsu POA Service Management initiated an internal bridge call
SECURITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
Page 8 of 20
FUJ00232704
FUJ00232704
SECURUITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
re)
FUJITSU
19 October 2020 and 21
POL Refs: INC0296097
Incident Date: Fujitsu Refs: INC6469861
October 2020
Version: 3.0 Problem Record: PRBO048676
Owner: Piotr Nagajek Peak Record: N/A
Status: Issued ‘Change: N/A
12:06 I Fujitsu POA raised a P11 incident INC6485949 for Health Lottery GWS critical
errors
12:06- I POA Tech bridge call continued. Impact initially assessed, confirmed that
12:20 I monitoring teams in India did not lose access to monitoring tools and were raising
incidents
12:13. I POA SMC raised a P1 incident INC6486044 for Branch Access and Collect &
Return monitors showing critical in HORIce monitoring tool
12:20 I POA MAC raised a P1 incident INC6486129, which was bonded with POL IT DSD
incident INCO296694
12:28 I Fujitsu NOC MIM Team confirmed engineers performing device troubleshooting
on site
12:43 I On the POA tech bridge call a restoration of VPN access and other services was
observed
12:57 I POA Infrastructure Manager confirmed with Fujitsu NOC MIM Team that the
traffic was failed over to the secondary device and all services were recovered
13:00 I Enhanced monitoring by Fujitsu POA was in place
13:05 I Fujitsu POA confirmed that First Rate successfully logged into the PODG and
pulled the files. Automated transfer of other PODG files clearing the backlog was
ongoing
13:40 I Fujitsu NOC MIM Team confirmed that a technical bridge call is underway with
Networks teams and the vendor in order to restore the primary device interfaces
List of all incidents related to the issue raised by Fujitsu POA on 19.10.2020:
Number Opened Short description Parent Incident
INC6470271 I 19.10.2020 16:28 PODG route issues on BAP_FILE_BIG INC6469861
INC6470193 19.10.2020 16:32 Spectrum AlarmNotifier (Check Spectrum Console) I INC6469861
: Mon 19 Oct, 2020 - 17:11:07
INC6470133 I 19.10.2020 16:39 SYSMGT_LOG_AKS_HIH_BAL_ERROR_ID0003 I INC6469861
WoodSN15243
INC6469973 I 19.10.2020 17:08 Observed Multiple PODG Alerts INC6469861
INC6469847 I 19.10.2020 17:11 VPN Issues - ITGI1084969 INC6469861
INC6469826 19.10.2020 17:14 Track and trace Showing Critical in HORIce INC6469861
INC6469841 19.10.2020 17:21 INC0296048 - Unable to perform Drop and Go I INC6469861
transactions in all branches
INC6469817 19.10.2020 17:23 Branch Access and Collect and return showing I INC6469861
critical on HORIce
SECURITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
Page 9 of 20
re)
FUJITSU
Incident Date:
19 October 2020 and 21
October 2020
SECURUITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
FUJ00232704
FUJ00232704
POL Refs: INC0296097
Fujitsu Refs: INC6469861
Version: 3.0 Problem Record: PRB0048676
Owner: Piotr Nagajek Peak Record: N/A
Status: Issued ‘Change: N/A
INC6469806 I 19.10.2020.17:46 syslog. TCOMLoadBalancer INC6469861
TCOM_MONSERV:MONID:PCAI.SERVICE
MONSEV:B DIAG 559 19
INC6469708 I 19.10.202018:11 IT TRRELEVANT I LIVE Service - I INC6469861
NEOPOST-L¢ - State Changer
INC6469675 I 19.10.2020 18:19 ALERTID-4005-00000-0] CDP Interface is down-I INC6469861
@e
INC6469656 I 19.10.2020 18:43 MASTER INTERNAL Ref - I INC6469861
GWS::ERR:0204:MONID:GWS.NEXP.SVC
MONSEV:B The threshold for third-party failures
Parent Incident
INC6469861 19/10/2020 17:29:16 I Internet Traffic Usage (GWS, CDP & HBS
services)
List of all incidents related to the issue raised by Fujitsu POA on 21.10.2020:
INC6485949 I 21/10/2020 12:06 GWS::ERR:0210:MONID:GWS.HLTY.ERROR I INC6485885
MONSEV:B 22 requests failed in the last
monit
INC6486044 I 21/10/2020 12:13 Branch Access and Collect and return showing I INC6485885
critical on HORIce
INC6486129 I 21/10/2020 12:20 Internet Connectivity issues INC6485885
INC6486171 21/10/2020 12:24 Critical 21-Oct-2020 11:54:45 BST NEOPOST-I INC6485885
L4LB (IRE11) SERVICE IS DOWN No 21-Oct
INC6486268 I 21/10/2020 12:31 yslog. TCOMLoadBalancer INC6485885
TCOM_MONSERV:MONID:PGDD.SERVICE
MONSEV:B DIAG 559
Parent Incident
INC6485885 21.10.2020 12:00:41 I Internet Traffic Usage (GWS, CDP & HBS
services)
SECURITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
Page 10 of 20
SECURUITY CLASSIFICATION
foo) COMMERCIAL IN CONFIDENCE
FUJITSU
Incident Date:
19 October 2020 and 21
FUJ00232704
FUJ00232704
POL Refs: INC0296097
Fujitsu Refs: INC6469861
October 2020
Version: 3.0 ~~ : Problem Record: PRBO048676
Owner: Piotr Nagajek . Peak Record: N/A
Status: Issued ‘Change: N/A
IMPACT OF INCIDENT
Track and Trace (includes impact of simultaneous external incident in RMG domain):
¢ SOAP faults 16:08 — 19:35
e T&T “success” stats comparison Mon 12 v Mon 19 16:00-20:00:
Track and Trace Success
Mon 12 v Mon 19
QBSSAARSRSARASRASARREE
SASSRSRS5BRRSRSRRRBARAAATAA
me 12 Oct TT Succ mmmw 19 Oct TT Succ
Overview
of Service
Impact I CDP responses using the SLT_stats:
19.10.2020 e Recovery at 17:30:
CDP Responses
raw_slt_stats for CDPAPADCGateway
600
500
400
300
200
100
0
SSSSASRAARRIESRRSSRAARARFTLAS
SSS8S8S8S8S8SSReSS8RESRRSSSSSR8S85 558
sme GOOD_RESPONSE me BAD_RESPONSE
SECURITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
Page 11 of 20
FUJ00232704
FUJ00232704
FUJITSU COMMERGIAL IN CONFIDENCE
POL Refs: INC0296097
Incident Date 2 Cetober 2020 and 21 Fujitsu Refs: INC6469861
Version: 3.0 Problem Record: PRB0048676
Owner: Piotr Nagajek Peak Record: N/A
Status: Issued ‘Change: N/A
I RCode I Meaning I Counts
) Success 10000
38 Error sending request (timeout) 182
80 Could not connect unavailable 16512
82 Time out 122
98 CDP technical error 4
99 CDP business error 208
27039
Reqs Resps I Success I Error I Resps
Service I received received I resps resps I returned
DVLA 1812 2 0 4 1809 1510 299 I 1812
FMCC /
FMCV 1208 0 945 40 253 486 4 I 1208
HLTY 786 7 4 0 30 16 T I 186
HPBB 6 0) cn 0 1 6 o] 26
NEXP 52 a i 0 34 34 0 I 52
TOTAL 3284 7 4020 rl 2444 2052 I 310 I 3284
Page 12 of 20
re)
FUJITSU
Incident Date:
Version:
Owner:
Status’
SECURUITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
19 October 2020 and 21
October 2020
3.0
FUJ00232704
FUJ00232704
POL Refs: INC0296097
Fujitsu Refs: INC6469861
Problem Record: PRB0048676
Piotr Nagajek
Issued
Peak Record: N/A
Change: N/A
Overview
of Service
Impact
19.10.2020
Collect & Return comparison 19.10.2020
Collect & Return
PODG:
PCAI
PGDD
Fujitsu observed alerts from multiple PODG routes between 17:15 and 17:39. Please see
details in attached spreadsheet
PODG routes
affected.xlsx
BKAC time outs: 72
First Response Code 82: 16:17
First request post outage: 17:30
No requests failed as service unavailable as h/c failing.
No impact — requests processed throughout
3 Request time outs (Not a busy service)
SECURITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
Page 13 of 20
FUJ00232704
FUJ00232704
SECURUITY CLASSIFICATION
foe) COMMERCIAL IN CONFIDENCE
FUJITSU
Incident Date:
Version:
Owner:
Status’
POL Refs: INC0296097
19 October 2020 and 21
Gotober 2020 Fujitsu Refs: INC6469861
3.0 Problem Record: PRBO048676
Piotr Nagajek - s Peak Record: N/A
Issued ‘ ‘Change: N/A
Overview
of Service
Impact
21.10.2020
No impact to the counter estate directly, or Banking, EPoSS, Card payments, or Track and
Trace (impact to T&T on the 19" of October was a result of subsequent issue in RMG domain).
e CDP service degraded (comparison):
GRR Tocays vee MERE Last Weeks vane
o Stat Counts for CDP between 11:00 — 13:00
RCODE I Meaning Total
t
7 3
tee His 4
I &2 I timeout I
84 invalid resp 4
98 CDP technical error 5
SECURITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
Page 14 of 20
oO
FUJITSU
Incident Date:
Version:
Owner:
Status’
19 October 2020 and 21
October 2020
3.0
Piotr Nagajek
Issued
SECURUITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
POL Refs: INC0296097
Fujitsu Refs: INC6469861
Problem Record: PRB0048676
Peak Record: N/A
Change: N/A
FUJ00232704
FUJ00232704
Overview
of Service
Impact
21.10.2020
. Collect & Return service degraded (comparison):
« GWS services:
WEGR Tocoys \oiue (REI La tects Valve
o Summary counts between 11:50 — 12:55
Regs Resps I Success I Error
SERVICE I received received I resps resps
DVLA 1415 3 1 0 1411 1222 I 189
FMCC 1210 0 7020 0 I Bam 190 378 1
FMCV 126 0 64 39 23 23 0
HLTY 554 383 709 0 I Beem 62 38 2
HPBB. 24 3 14 0 6 0 6
NEXP, a 6 73 0 20 20 2
TOTAL 3370 395 T221 39 1712 Test I 200
SECURITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
Page 15 of 20
FUJ00232704
FUJ00232704
SECURUITY CLASSIFICATION
foo) COMMERCIAL IN CONFIDENCE
FUJITSU
POL Refs: INC0296097
19 October 2020 and 21
Incident Date: Groner 2000 Cs™, * Fujitsu Refs: INC6469861
Version: 3.0 & \ 7 Problem Record: PRBO048676
Owner: Piotr Nagajek m\ Peak Record: N/A
Status: Issued . e ‘Change: N/A
« , Payzone comparison with previous week
o, British Gas & Electric vends
ayrone comparison
(epess payment)
Overview
of Service i i adi
Impact
21.10.2020 « Smart Metering (SMS)
co This shows a spike in retries during the incident
BREE eebys Vove BRET Lest eas Velve
“KS
300
‘Smart Metering (SMS) Success Curtest.207 Prev:22
SECURITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
Page 16 of 20
oO
FUJITSU
FUJ00232704
FUJ00232704
SECURUITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
POL Refs: INC0296097
19 October 2020 and 21
Incident Date: Getoper 2020 Fujitsu Refs: INC6469861
Version: 3.0 Problem Record: PRBO048676
Owner: Piotr Nagajek Peak Record: N/A
Status: Issued ‘Change: N/A
. (Pops transfers
a) P . i
_) All PODG file transfers completed successfully following restoration of the service
~~ (spike at 13:10 represents that)
RRB Teansters
40.
Overview I 006 transfers 5
of Service
Impact e Banking, Card Payments, APOP and Track & Trace remained unaffected:
21.10.2020
WRRRBER Todays Vaiue RIB Last Weeks Vaiue WEBER Todays Valve BRMBNN Last Weeks Value
15,000
10,000
5,000
0
> > PHP SHOE ©
2 SPS EPEEES S
Track & Trace Current:18,591 Prev:17,199 Banking Transactions Current:5,552 Prev:6,085
(BIR Todays Value (RRR Last Weeks Value (BRRB Todays Value RRMERE Last Weeks Value
10,000
8,000
6,000
4,000
2,000
SESS PM SESS
FS § § cS FS FP LS §
¢ os ‘ ¢ PEEL FFF LEVI LOLS S
Card Settlements Current 7,848 Prev-7,906 AP out payments Current 660 Prev 598
‘SECURITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
Page 17 of 20
re)
FUJITSU
Incident Date: 19 October 2020 and 21
FUJ00232704
FUJ00232704
SECURUITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
POL Refs: INC0296097
Fujitsu Refs: INC6469861
October 2020
Version: 3.0 Problem Record: PRB0048676
Owner: Piotr Nagajek Peak Record: N/A
Status: Issued Change: N/A
« , BWS.
o BKAC — Bank Acct Checker
. KAC. TIMEOUT MONSEV:B 18% (28 Timeouts in 154 requests)
. BKAC.TIMEOUT MONSEV:B 11% (19 Timeouts in 162 requests)
Overview = No requests rec'd between: 11:55 and 12:45
of Service
Impact
21.10.2020 o PGDD (guaranteed delivery)
_o-"""PGDD.TIMEOUT MONSEV:B 100% (1 Timeouts in 1 requests)
I IRRELEVANT I
i. __J?GDD.TIMEOUT MONSEV:B 100% (1 Timeouts in 1 requests)
o PCAI-no impact
Overview
of ti
Business I Please see statistics above.
Impact
COMMUNICATION METHOD
Email: Yes Voice Calls: Yes
SMS: No Bulletin: N/A
ACTIONS
Actions P
No Owner Bene Comments / Updates ea
jate k
Fujitsu NOC Team continue to I Eyitcy Completed NI
work with Juniper to understand Noe 21.10.2020 - Further tests scheduled, I A
4 the root cause and develop an Shared I 23.10.2020 investigation continues.
p Service 22.10.2020 — RCA for both incidents
s Team expected shortly
SECURITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
Page 18 of 20
re)
FUJITSU
Incident Date, 19 October 2020 and 21
October 2020
Version: 3.0
Owner: Piotr Nagajek
Status: Issued
FUJ00232704
FUJ00232704
SECURUITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
POL Refs: INC0296097
Fujitsu Refs: INC6469861
Problem Record: PRB0048676
Peak Record: N/A
Change: N/A
23.10.2020 — The following actions were
completed on the 21% October to restore
and resolve the root cause.
- Correct the commissioning error
with the CenturyLink connection
- Source replacement Hardware for
failing component (router chassis)
- Install replacement chassis and re-
establish internet connections
Fujitsu technical SMEs to be physically on a
Fujitsu Site for two weeks as a Hyper care
measure.
See Action 7 for further action plan.
Fujitsu High Speed Internet platform are
Fujitsu POA to assess a full ” Completed N/
impact on Post Office Account Fujitsu I Completed Please refer to Overview of Service Impact I A
POA 20.10.2020
services section of this document.
Fujitsu POA to conduct a Post Completed N/
Incident Review to identify 21/10 — Awaiting outcomes of Fujitsu NOC I A
improvements investigation with the vendor (See Action 1)
. 06/11 - A new process has been put in place
Fujitsu I ¢ 44.2020 I for changes involving Fujitsu Critical
POA Infrastructure. Senior Management and
account teams will be informed and
engaged, so that the Major Incident team
have confirmed points of escalation during
such changes.
POA Problem Manager to raise a POA Completed N/
problem record to track the Proble Completed Problem record PRB0048676 raised to track A
actions m I 37 16-2000 I actions.
Manag
er
POA UNIX removed the lockfiles Completed N/
on the PODG manually to allow I Boa I completed I Completed during the POA bridge call A
the file transfers, rather than
UNIX 19.10.2020
waiting for the automated process
to run.
Fujitsu to confirm to POL Sonia Completed Completed N/
regarding the POL traffics being I Hussai I 35 10 2020 Sonia Hussain provided evidence to POL to I A
observed on both routers n o_o demonstrate traffic on both prefixes.
Fujitsu to confirm the disaster Completed N/
recovery plan 26/10 - Fujitsu are planning a change on I A
Fujitsu the 10" and 12" November on the HSI
NOC routers where the resilience of the service
ce leted
Shared I $37 Poq99 I will be tested. HS! Firmware Update
Service — CHG0201209 — LDC06, CHG0201213 -
s Team LDC08
06/11 - The firmware upgrades on the
SECURITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
Page 19 of 20
re)
FUJITSU
Incident Date, 19 October 2020 and 21
October 2020
Version: 3.0
Owner: Piotr Nagajek
Status: Issued
FUJ00232704
FUJ00232704
SECURUITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
POL Refs: INC0296097
Fujitsu Refs: INC6469861
Problem Record: PRB0048676
Peak Record: N/A
Change: N/A
Planned for the nights of Monday 9*
November (through to early hours on
Tuesday 10° November) and Wednesday
11" November (through to early hours on
Thursday 12" November).
10/14 - CHG0201213 completed
successfully with no impact to the
service.
12/14 - CHG0201209 completed
successfully with no impact to the
service.
POL shared an observation . Completed
around speed of incidents being I Sonia Completed I For any Major Incident, POA DM will make a
8 I raised and escalated to POL MIM. I Hussai I 57 49 9920 I telephone call to POL MIM team
n immediately to notify them regardless of the
incident ticket being raised.
CLOSURE
Date Closed Fujitsu Approval POL Approval
30.11.2020 Steve Bansal Lorna Owens
For individual Major Incidents a Post Incident Review shall be held at which
the Root Cause Analysis will be conducted and improvements considered
from a lessons learnt perspective.
SECURITY CLASSIFICATION
COMMERCIAL IN CONFIDENCE
Page 20 of 20