POL00002717
POL00002717
FUJITSU
shaping tomorrow with you
Meeting Sharon Gilkes 16/5/16
Pete Thompson
F/1473/1
POL00002717
POL00002717
Major Incident 9/5/16 — Branch Database
@ Background
Between Saturday 30th April and Saturday 7th May a business continuity test was conducted, to fail-over from Branch Primary
Database (BDB) to Branch Standby Database (BDS) under MSC 043J0444003-06, On Saturday 7th May the fall-back to Branch
Primary Database was successfully concluded and service was provided from the BDB throughout Sunday 8th May. On the
morning of the Sth the system was running on the primary database servers with the last known good configuration which had
been running successfully for the previous year.
@ Incident Summary
(On Monday 9th May 2016 from 08:55 to approximately 10:15 the HNG-X Branch Database Primary (BDB) incurred performance
degradation
At 09.20 a Fujitsu internal technical Bridge was convened to investigate the incident and implement corrective actions.
Following analysis it was recommended by the Fujitsu technical recovery manager that the Oracle Linux configuration item
“MEMlock” should be set to the same configuration parameters as the BDS server instances. In order to stabilise the service
and minimise any further impact it was proposed to cycle through the BDB nodes changing the setting rather than restarting all
nodes at the same time, as this would be less disruptive to service.
By 10:15 on the Sth May all 4 BDB nodes were configured with the new settings and service normalised
Root cause is still under investigation with Oracle. This is not a legacy issue as the Oracle 11 database, operating system and
hardware are still currently under vendor support.
Fujitsu and Oracle directors are engaged in daily conference calls to progress the resolution of this incident.
® Service Impact
The Horizon service was impacted from approximately 08:55 to approximately 10:15 as a result of the branch database
performance degradation.
The branch database experienced performance problems resulting in 65% of transactions failing over an 80 minute period
No liquidated damages have occurred ,4 SLA’s will be missed within this current period as a result of this incident.
F/1473/2
POL00002717
POL00002717
POA Internal Communication Timeline ——_rujisu
® 08:55 Enterprise Monitoring detects Swapping on Branch Database
Servers
® 09:10 Horice picks up transaction failure rates
® 09:12 — 09:43 Atos called via Mac team of incident —Atos lines down
unrelated issue — email also sent to advise Atos of engaged tone at desk.
@ 09:20 Fujitsu (Muhunda S) informed lan Thomas/Atos team whilst attending
NFSP
® 09:22 Call raised to PT informing of service incident
@ 09:22 Internal SMS informing stakeholders
® 09:20 Tech Bridge established
@ 09:27 Gavin Bell informed Chris Broe of incident
@ 09:30 Fujitsu (Steve Bansal) informed lan Humphries and Gary Blackburn
at 0930
@ 0930 Fujitsu (Pete Thompson) spoke to Andy Garner
® Start of incident to tech bridge initiated 25 minutes
3 FUJITSU RESTRICTED -UK IRELAND EYES ONLY
F/1473/3
POL00002717
POL00002717
Branch and Counter Availability SLT
300.98
ea
, sit Counter Secon
Nov 2028 Dec 2025 fan 2036 Feb 2026 Mar 2046 Any 2038
SLA Performance
Moy hun 25 Jul XB Aug 38 Sep 18 Ost 15 Nov 26 Dec 75 fan 36 Fed I Mar 36g 36
a
HNG-X Production Service Dashboard — Apr 16
Major Incidents
Nov-15 Dec-15 Jan-16 Feb-16
Mar-16 y
Jan 28th - POLSAP Material Movement
Feb 13th - BDS Server Memory Leak
Problem Management
All Operational Problems not yet CLOSED at EOM
Reconciliation file transfer false alerting.
POLSAP Period End incorrectly executed resulting in processing issues.
'BDS Server Memory Leak leading to failed Horizon Transactions.
British Smart Gas Metering interface marked as unavailable.
Fujitsu unable to deliver files to the POADMIN Server via PODG.
PODG routes failing following POL/RMG separation.
ComputaCenter Engineers are unable to install ingenico PinPads.
F/1473/4
POL00002717
POL00002717
HNG-X Production Service Dashboard — Apr 16
Patch Management Total no. of ARQs queried
z
Jon tS Jul 35 Aug Sep Ose 35 Nov Bee Ina 36
5 comes 515
F/1473/5
POL00002717
POL00002717
HNG-X Production Service Dashboard — Apr 16
Open High's - Summary
Q1 PCI Risks - as of 12/05/2016
ech
801 Count
Fistral competing Cont
Seed rd
‘Grace Applcsion Sever Werkiow
Carsdge Component Unspectted devs
Apache Tomest Malle Vorbis For Window
apn patch Prt Low Unikely LOM peck accepted
Vanessa me ‘
ace option Sever Woon cane, 7
Component Mlle Uwe wnt ra Low Untkaly Low Test Compenaing cont
‘Serer are not Inemet accessible and me
protected by frewals hl ony perm
ques securty domain acess,
Low Unikely Low
Fepet-Compenatingconat
feet Comonatg conta
Fisk previously accepted by Post acute
Low Ome - To be adaressed under HOCR
POL
[Risk previously accepted by Post Discuss
tow Gace Tobe asswced uncer HocR
tow Rsk previously accepted by Post DUNS
‘once - To be adoressed under HOCR Mt
POL
+ The substantial majority of Medium issues have previously been accepted by POL and will be
addressed under HDCR
F/1473/6
POL00002717
POL00002717
HNG-X Production Service Dashboard — Ap! 16 /
OBC Changes v OBC21’s
ApS May3S unt5 235
Cost on OBC Service
S$ QuAS Kov35 Oue35 dan 16 Fats
OBC Count of Errors (21's)
£109,000
50,000
269,000
F/1473/7
POL00002717
POL00002717
HNG-X Production Service Dashboard — May 16
freeet Ret ranch cece oe ret Aon
THOS CC peru get 00. SF aporemet sans
vonccoecess deme eae
Avteioaes atte
Alziioan a7asie_
2004 Cite te Ft speak one bors (aa) abou araning se
Tesco nthe avscho et own eu than yun eye ne schedded
pees (EE 80 ea ts Ke entre omet
we tryngto gain urbe ac
sa/own16 10:39 1swan16 16.06
swoumnie 1612 2oowanre1320 118
180042016 15.19
2UzDI6 1847 Z408-Freato BT or
irtermitency
ai fautedto BT ee ere ee ae
2a008
-rp9oa2090 19
20/04/2018 15.58 Fault msed wah TTB fr intemittency, pending futher updates
FUJITSU EYES ONLY
F/1473/8
POL00002717
POL00002717
HNG-X Production Service Dashboard — Apr 16
Ref Data Statistics
Ref Data Variable Cost
36,606.06
econ
ll
ss 38- Jul 15 Ag 35 (Oct 25 Now 15 Dac 15 Jan 36 Fa 26 Mar 26 Agr 36
Open Service Review Actions
Against Target
Ey 201
L-3Montis © 4. 6 Months
MSC Analysis
6 Months +
2/8) 28/8
January I February
7
F/1473/9
POL00002717
POL00002717
HNG-X Production Service Dashboard — May 16
CSAT Scores
‘samme Wares Measure
F/1473/10
POL00002717
POL00002717
HNG-X Forward Schedule of Change — May 16
Start Date End Date
Activity Deployment Group Title Time Time Priority
may 01/05/2016 00:00
bank Holiday
an Manager Upgrade Pan Manager Upgrade (07/05/2036 17-00 07/05/2016 23:002 - HOCR/New Change
‘Annual Failover Test 97/05/2036 20:00 07/05/2036 23:00
ne
Failback test
Compliance
a °
rag.a5 M2 Migration (Branch). (08/05/2036 02:00 08/05/202605:002 - HOCR/New Change
Security Patching Solaris, Linux & Windows _ March Manufacturer Released Patches 14/05/2016 20:00 14/05/2036 23:301
Compliance
Security Patching Solaris, Linux & Windows March Manufacturer Released Patches 15/05/2016 04:30 15/05/2026 10:301- Compliance
Annual Electrical Health check Partial PH2 Buildings Electrical Health Check 35/05/2036.09:00 15/05/2026 18:00,
Security Patching Solaris, Linux & Windows March Manufacturer Released Patches 16/05/2016 08:00 16/05/2016 22:301 - Compliance
Security Patching Solaris, Linux & Windows _ March Manufacturer Released Patches 37/05/2036 08:00 17/05/2036 23:303 - Compliance,
Security Patching Solaris, Linux & Windows March Manufacturer Released Patches(Contingency) 18/05/2016 20:00 18/05/2036 22:301 - Compliance
Security Patching Solaris, Linux & Windows March Manufacturer Released Patches 19/05/2016 20:00 19/05/2026 23:301 - Compliance
Mz Mi
a)
21/08/2016 19:30 22/05/2026 23:552- HOcR/New Change
ae a ie ae,
F/1473/11
POL00002717
POL00002717
HNG-X Branch Transaction Volumes — Apr 16.
F/1473/12
POL00002717
POL00002717
HNG-X Average Trading Branches — Apr 16
a FUJITSU EYES ONLY
F/1473/13
POL00002717
POL00002717
HNG-X Average Trading Branches — Apr 16
“ FUJITSU EYES ONLY
F/1473/14
POL00002717
POL00002717
HNG-X Continuous Service Improvements — May 16
Issue
Increasing number of branches unable to
trade due to PM's switching ISP provider
Delays in diagnosis and resolution of
branch network incidents where calls
passed back and forth between suppliers
Standard Broadband Care level equal to
consumer BB products, resulting in lengthy
outages at critical sites
Higher than expected volume of rejected
incidents
Uncommunicated changes made by third
parties impacting POL estate necessitating
FJS support in resolution
Act
‘Comms campaign with branches
to improve knowledge/education
of restrictions
Proactive notification of these
incidents to IM and process to be
agreed for requesting joint
supplier visits
Explore feasibility of enhanced
care packages at critical sites
Help Desk SIP to ensure:
- KA’s are accurate/up to date
- Agents receive additional
coaching/training to ensure KA's
are embedded
RMG — Mechanism for aligning
change processes to be agreed
between Service Directors
Benefit
ISP switching
Reduced incidents due to
Improved branch availability,
faster resolution for
customer, reduced
escalations
Improved avaitability at POL
strategic/critical sites
Reduced number of rejected
incidents and faster
diagnosis and resolution of
issues
Reduced incidents as a
result of uncommunicated
changes
csi
Ref
3820
3821
3822
3823
3893
F/1473/15
POL00002717
POL00002717
HNG-X Continuous Service Improvements — May 16
issue
File Transfer issues —
improve knowledge on Atos
Service Desk of support
routes and contact details
Branch Closures / trading
disputes
Session Correction
Approvals Process is unclear
resulting in protracted delays
in authorising corrections
Delays in diagnosing
technical issues within
mobile vans
Action
Desk SIP
FJ to conduct feasibility study to
understand opportunities for
restricting network traffic during
branch disputes
Work with FSC Team to develop
agreed process and MDS for
approval requests
Resurrect the van manual that was
previously developed by FJS, to act
as a ‘spot guide’ for PM's operating
mobile branches
lan Humphries to develop within Help
Benet
Faster diagnosis and resolution of file
transfer issues
Reduce risk of branch hardware and
systems being used inappropriately
during branch disputes
Faster resolution of session related
incidents and improved counter
availability
Faster resolution of mobile branch
issues and improved branch availability
cst
Ref
2963
3825
3836
3837
F/1473/16
POL00002717
POL00002717
HNG-X Production Programme Key — May 16 -
Programme performance POA project delivery remains positive overall, Initial HDCR network migration (Support DMZ) completed 16-17/4. Testing
{BAS Ops Summary) lunderway on HOCR Ri5 (Infrastructure Services), inital build of AD complete in test environment. HNGA continuing on track.
Project Baseline I Last Week I This Week IComment:
R&R? (Horizon I Design/ Build/ I Mayi7 ‘iW Suppor DME migration successfully completed 10/4, intemet OME migration planned 177%,
DataCentre Deploy RAB scheduled 14/4, The RMG DMZ migration scheduled fr 30/4 is going to be deferred asa
Roles) I ___.._ sult ofthe impacts ofthe POLSAP & Credence/MOM service extensions. Alo discussing the
ed to defer the Branch DMZ migration because of the potential changes to the PO Branch
Counter Refresh / Network Transformation plas.
CT fora design study to review the full impact of the POLSAP & Credence/MOM extensions is
being finalised ahead of CCB 18/4
SVBl for R15 started 11/4 (delayed from 4/4 de to PO prioritising testing for Drop & Go}, inital
{AD build tests complete
Horizon R13 the N Future plans for Branch Counter Refresh / Network Transform still unclear, including plans for end
Anywhere 13.05 — tend testing ofthe solution, n meamtime work continuing addressing number of sues raise in
© the R13.055V&I to ensure as high qualty as possible for when eZe testing ready o stat
Barcoding AllI Design/ Build I Phase 2a Project on hold awaiting approval of CT2065a for Phase 2a sothat the team can be re-engaged
Parcels (BAP) (Feb/ntarts) and work can be resumed. POLPM indicate that CT shouldbe signed shortly
Phase 20
Test Rigs Refresh Build/Deploy I Various I (I Project in closedown. Decommissioned kt removed from racks, intial quote received ready to be
I sent fordestruction
Branch Network IBuild/Deploy I Various EE migration plans established tangeted for completion by end April
I __.. Pans for Vodatone migration stil tobe agreed, meeting scheduled 25/4 to establish plan to start
Refresh following the E migration {to complete by 31/5)
Tower Transition I FOSOEUG Continuing to wait for a decision from PO onthe future drecion / plans Yor Branch Counter
‘Transitions Networks, Refresh and Networks Transformation ~ discussions have been taking place with PO/ Atos to help
I answers queries and in preparation for activity being inated quickly once decisions reached.
Back Office - engagement continuing on the plans atest view shows migration to new solutions
— by end ‘Sep. Credence service extension CT still awaiting formal approval
Royal Mail Deployment I u/s The RM Separation work s now winding down, with any outstanding issues around the separation
Group : being handled by the agreed BAU process (mainly by the Service team). PM contro wll remain
Separation until the end ofthe month to ensure follow up activities are completed
1
Discussions ongoing with PO regarding their plans to improve the project governance and demand planning, once these changes are confirmed then the current internal meetings
objectives and structure will be reviewed to try ensure alignment with the PO/ Atos.
2. Internal Delivery Board scheduled for 20/4.
F/1473/17
POL00002717
POL00002717
HDCR Release 14 & 15 Live dates
Release
R14.10 Network Core Join
Target Live migration dates
R14.15 Support DMZ Migration
R14.15 Internet DMZ Migration
R14.15 RMG DMZ Migration
R14.15 Branch DMZ Migration
R14.15 Banking DMZ Migration
21-224 May 2016
R15.10 Infrastructure Management
9-10 July 2016
R15.15 SYSMAN4 Monitoring
27-28" August 2016
R15.20 Key Management Services
1-24 October 2016
R15.30 Network Management
8-9! October 2016
Note — R16 & R17 Live migration’s are planned to continue from Nov 2016 and on in to 2017
F/1473/18
POL00002717
POL00002717
Risks
Basaipion
Imere nas been 2 recent service neler esuting fom
faower and altace of he branch deabace severe
Inconel contre Reference dala hasbeen rlensed
to NGSX: three (3) recent neigerts nar 16.
[Tare ares numberof ranch Network Sarvces
rminaing in Match 2017 Le. Vodafone Network, BT
ete Senet
Sonor Raa
areity with Oracle to erty rot cause
nau is required to review te current sis and
rowedge whine As Team an
owe fons earls wah bath
ro
ts and Als lo mga ture ince
[The containment plans for POL fo commission @
yen Maren 2017
Fama rOe
10, Serdes Decors Meeting & SUR
rogrammes & Projects Governance meetings anawih I,
ne Service Team,
Iino ve Courter VPRveys we aio ls wipro Novarip
16. CC are yet io agree the commer ems forthe
pare Gotcha suomted to thom in Ast 2018. Rss
sg propetonatl to tne
OL ad CC to agree the Commercial Terms
Co sey the gals bls vt he pare oop and
ave the ne Bul avalible for roe
ence Decors Meeting & SMR
"OL o ote te endo-end model
"OL to review resourcing inthis ares to werk.
frongsie Fu TOS aretects.
AT testing for 201418 was canceled. During
011/12 testing twas fund tat over 10 hardware Hema
Inbranches had unsate equoment requrng replacement.
or
“ytsu recommend fat elcieal testing
he requoncy recommenced by the UK
ety Execuhe,
tent siskto the exec of te Post Ofice business
endor suppor of EOSL legacy components sn longer
Necessary process and Wectnic! changes have not been
“OD and Credence HOM have nel been designed to Be
Cl compliant schsions
rou dala be included dla es ten these patfoms
Wie requtres tobe updated
Jaanagement of suppies, dependencies and conrad!
mange
-unctonaleuremet mapping sey ane
Jenin o gaps
tems Gua
ay
approach to ensure tecnica!
Renearsas and prosectve negated testing
ost Once spect spares of some old components
dressed by Horizon Data Cente Reftesh Activ.
nage eques(s) fo be ralseato meet compliance
poigatens.
OL need to ensixe 3 partes rem his data
lasieatlon whi scripts producing data
erdce Decors Meeting & SMR
01 Aug 2018
COMMERCIAL IN CONFIDENCE
F/1473/19
POL00002717
POL00002717
Risks
eee
_ a
i POT
dat to support higston acs,
imedtamping seross mule vendors de capture
Ine acionl CFS woriond wane parading te ola
OLSAP sever
6 design only requ aslgle MOM server in each
fata centre (ne restence wahin each dota cenze)
fn ik a resut of he Post Omi requ! oinplment
Needs a cocrdinated mut-supler review to assess
ual of sample evidence wih experts
fo Upgrades Ye enbie data ean'be
erie Directors Mesting & SMR.
fo10 Mostngs. Senice Drectors Mestng & SMR.
fi i ii i Ps
fae ienaea somo
Int Compute was on intended for Test worioads
ju d oer Cloud Hosing Wom ever Oala
ries
Petcteed usage
Inter Compute was oly intended for Test worhonds,
however Fufsu understand that POL have now deployed
Ive berdces
fature rates WTBF or components supporting ine
‘ure
ray be worth remeving unused counters
Jncrense spares sup.
hange Requests o be raised to nvodue resent
twa component
Test Rigs are not fully scaled versions of ive, ne volume Ah identical HAW and SAW technologies,
[There are some miner HAW variations inthe equipment P* model
sed inthe Figs 6.9. lower spec servers. Resilience paper produced and distributed further {CIO Meatings. Service Directors Meeting & SMR a
Gren ate Btat ope ayn
Recent major incident highiighted beneft of future volume [* Ststeay going Forward,
aaa spoke transaction rien to emulate Monday
recta
ar Reena! wr agaeT wa
is
tock an hhard isk spares based on f-O% 10 review win ATOSICC the meantime BewEN leavicg Orectors Meeting & SMR u
erie Dectrs Meeting & SUR.
‘Fujisu do not have tat
iat of the
[ontgured senices)
F/1473/20
POL00002717
POL00002717
Service Availability April 2016
Branch Network Service Call to Fix SLT
jargs eve’
95% within 3
1 hours 55 minutes 276 276 100.0%
100% within 5
1 hours 55 minutes 276 276 100.0%
Branch & Counter Availability SLT
99.64%
100.00% 100.00% 100.00%
100.00%
FUSITSU RESTRICTED -
F/1473/21
POL00002717
POL00002717
HNG-X Production Service Dashboard — Apr 16
CSAT Scores
F/1473/22
POL00002717
POL00002717
Quarterly Relationship Tracker
Regen Comments proved by Babu Pao
woud recommend jeu [5 Nth gree or ciaagee ins wi co charge, POLave rot ane to recommen Fults,
[would be happy to provide a reference for Fuad [3 - Neither agree or disagree ins wi rot crags, POLare rot apie wove a wiewnceor fa
IW are ikl to continue working with Fut ls Agree ie have sgn anew cont
POL where there chic to ie oer suapieae Tat Gott we Fur
ets ‘ew rots, fur areon com, matt sunsets 7 hag, OO
I would consider Fut for futher projets [s-neitne agree or sage resource residowns ne snp. Nous te to ong to spon
No, Fits require ot of work and have to be chased corsa. Exams Tine
\wecking with Fujitsu requires minimal etfrt on ou part [2 - Disagree sheeting, Asset Regstr and rai cenng
Feedback provided to Fujitsu isacted on [= Neither agree oF dsagie r sway Futures enang
Response
Tamiveiy Repay wth the save Ieee rom Fajitas [5 Neither age o isgies Coarse to
IKey milestones are always met on time. [4 - Agree: ree
Service [The seve levels we have agreed are alwajamat [4 Agree ee
Yoel that my needs asa custome ae being met By fated ot understand neds, yaa rot ncovtie Bare Buccs, er
re [s- Neither agree or disagree ronnngona ct but fut unaietofersolsionso age and 2507 POL
Response
[Nats very good reaTonsipin Hace Aways convactal as Fujau not TORT
(Overall there isan excellent retationship with Fujitsu [3 - Neither agree or disagree coming hep POL
Working, Fulsu demonstrate a deep understanding of ou needs [4 Agree Yes. Fujtsukrow been arose 20 years but
ath Fite D2 the ight level of contaet within Fujitsu = Neither agree of disagree No sige oi of contact, no iret access to dees on makes.
get access to wider Fujitsu expertise when needit [4 Aye igre
Some ae, sore are rot, Canorer ave rat Kept ores of progres onion
completion ot curing. Simple responses provid ut no further upcates ness
[Any problems that occur are handled etfectively __ [3 Nelther agree or aisagree crass.
Response
utes are proactive in anticipating our needs [3 Neither agree or disagree cannot see evidence of is behviour
‘Vetue ae [Fulss'are innovative in their thinking [3= Nether agree or csagree cannot ser evcerceof his enevour
eel that Fujitsu go above and beyond the contract to
[add value to out business L2- Degree No value dd from Fats
‘No imroversent nthe ast 6 monn, Should pt he best response [seni a af he
asiness otking with Fujitsu hasimproved ver time [s- Neither agree or disagree sme butco not
le Fujitsu nave a focus on continual improvement [3- Neither agree or disagree vo evcerce see of ary mprovenent from Fss
(Gonaistenoy within the Fujitsu account team I4= Agree gree, ro charges
Fujitsu continue to present uswith new ideas [2- Dssgres ‘No rew cess, fuss stage to Inoenen ousirgsolion
[What have been the key things that Fujitsu have done really well over the last six months?
Liaise with Post Citice ver change of staf on the account,
“ “ ““«FUSITSU EYES ONE
F/1473/23
POL00002717
POL00002717
me)
FUJITSU
shaping tomorrow with you
FUJITSU EYES ONLY
F/1473/24