POL00242846
POL00242846
Branch Network
business II Hertormance
Q4 March 206
Content :
High Severity Incidents
¢ 9 June— Branch Network
* 14" June —SSK Network 14° June
Performance Dashboard
Ingenico stock management
MoneyGram Problem Management
POL00242846
POL00242846
> Branch Network
Severity 1 — Branch Network Incident Ginwnsoe
Issue Summary
- Commenced Thursday, 9°" June 2016 at 13:06
- Layer-Two Network Switch (LNS) failed
- Service designed with 6 LNSs — These are used to connect broadband sessions from Branches to the Fujitsu Data
Centres
- Branch connections are load-balanced across the devices
- Branches impacted would reconnect to another LNS
- At 14:20 several hundred broadband sessions were dropped from one of the five remaining LNS routers — this was
due to a load balancing activity
Impact
- Majority of Branches experienced between 0 and 10 minutes outage
- Variance depends on whether the Branch router automatically switched over to the backup network, or the
Branch rebooted the router
- Approximately 17% of Branches using broadband were impacted
- Monitoring identified 1% of Branches had not reconnected following the initial issue
- Investigation discovered authentication issues and ‘stale’ sessions (which were stopping Branches from
reconnecting)
- All sessions were restored by the Service Teams by 16:10
Root Cause
- LNS failed due to a software crash.
- Device has a history of reliability
: No faults or errors identified
- Device will be tested and checked rigorously before re-introduction to service
POL00242846
POL00242846
> Branch Network
Severity 2 — Self Service Kiosk Incident Guweacoe
Issue Summary
- Commenced Wednesday, 15" June 2016 at 10:40
- SSKs not accepting card payments either via reader or contactless device
- Small number of Branches reported the issue
Impact
- Impact included Branches yet to receive contactless upgrade as well as those who had been upgraded
- Kiosks continued to operate in ‘cash mode’
- Investigations identified transactions ‘timing out’
Resolution Activity
- Investigations by NCR, WorldPay, GlobalPay and Fujitsu
- Network link utilisation threshold identified as exceeding 10 Mbit/s capacity into the NCR Data Centre
- Management Tool Application generating significant inbound traffic from transaction log requests
- Management Tool link terminated to reduce bandwidth load experienced
Root Cause Investigation
- Investigation by NCR into the Management Tool
- Network link is shared between Horizon Business Service, WorldPay and Maintenance Tool
- Root cause for bandwidth load to be determined however there is likely to be traffic prioritisation criteria applied
to protect the capacity required for the live production service
- Tool remains disconnected.
POL00242846
POL00242846
Branch Network
Horizon Business Service is available and open for business eect Pertormence
Branches - Intermittent & Unable to Trade Equivalent Number of Branches Closed for full month
Number of Branches
g
oO
=
e
5
[
5
o
2
€
5
2
No. of Branches Unable to Trade No. of Branches With Intermittent Faults Equivalent Number ef aranches Stretch Target sarvice (avel Taree
What happened to service performance ? What was done to improve it ? What is service performance doing now ?
* Overall service performance remains * The IM team proactively investigate the * Service Availability and incident
stable however the 9" May service intermittent issues. management performance has
incident materially impacted our improved in June
customers and branches * The team contact the Customer directly
to inform them that it is being looked at * Abusiness continuity plan is to be
* 0.4% of network availability was lost in and what actions the team will be reviewed with the Rob Houghton 23°
May equating to 40 branches being taking. June. This is to test the failover of the
closed for the month primary and secondary servers and
* Daily monitoring of the SLA thresholds demonstrate the resilience in the
* Intermittent branch performance is to ensure tickets which fall into this overall Horizon Service
improving category are managed accordingly
Number of Kiosks
Self Service Kiosks working time isn’t lost due to IT disruption
Self Service Kiosk - Kiosks Unavailable
‘No. of Branches with 100% navalable
No. of B with $05 Kiosks Unavailable
Overall NCR system performance is very
stable.
Overall Kiosk Service Availability
continues to exceed 98% target, 99.7%
in May 2016.
Average Response and Resolution Times
were 4.7 and 7.9 hours respectively in
May 2016.
This compares favorably with our
original 6 hour restoration SLA when
taking all other service improvements
into consideration
Self Service Kiosk - Network Availability
Overall Service Availability Service Level Target
Service levels were improved in
December 2015.
The scope of a Priority 1 incident has
been extended to include any VIP site
with a SSK down & 50% or more
unavailable in any site (previously only
branch with SINGLE kiosk being
unavailable)
Post Office now has equal access to the
NCR multi-client leveraged engineer
field team and therefore fix times are
improved.
POL00242846
POL00242846
Branch Network
business II Hertormance
Q4 March 206
Self Service Kiosk - Response & Resolution
% Availabil
Average Response Times (only 1ALT and P1)
lution Times (
What happened to service performance ? What was done to improve it ? What is service performance doing now ?
Service levels are much improved and
low volume kiosk outages.
For new P1 category, a single device
restoration within 24 hours is improved
from 91.7% to 92.2% however the
scope is significantly broader.
Branches with 50% kiosks unavailable is
slightly increased in April which is being
picked up with NCR (see Trend Analysis
on slide 7)
Trend analysis of branches with 50% of SSKs unavailable
Revisit Program
Post Office I _ SSK67 New Malden
Post Office SSK70 Broadway 175 I 5.00
Post Office] _SSK71Cityoflondon I 1.75 I 4.00
Post Office I __ SSK67 South Shields 175 I 6.00
Post Office SSK67 Camden 1.75 I 10.00
Post Office SSK68 Raynes Park 175 I 6.00
Post Office SSK68 Harborne 175 I 4.00
Post Office SSK68 Windsor 175 I 4,00
Post Office ‘SSK67 Kingsland 175 I 8.00
Post Office ‘SSK67 Maidstone 175 I 5.00
Post Office SSK70 Croydon 175 I 4.00
Post Office] _ SSK69 Houndsditch 175 I 4.00
Kettering Dundee Peterborough Trowbridge Paddington Q
[Shepherds Bush South Woodford Bolton Leighton Buzzard Dorking AS i if &
Camberley New Street Crouch End Poplar Southall Sheffield
Bexhill on Sea___ Petersfield Redditch iit
Tower Ramparts Barnstaple
Grantham Hastings Chesterfield
Frederick St Battersea Dunfermline SUTUSTISE: NANOS
Raynes Park Darley St St Johns Wood Mount Pleasant__ Kingsland High Street}
Eccleston St Aa TT Wl
South Woodford _ Camberley Chesterfield TEN (P I
Manchester MS Cirencester Haywards Heath Leighton Buzzard Battersea BRR BUNS
Manchester MS__Leighton Buzzard Dunraven Place Wigan Southall Bognor Regis Trowbridge
POL00242846
POL00242846
Branch Network
business II Hertormance
Q4 March 206
Preventative Maintenance
(PM) calls have been created as
part of individual SSK
monitoring of branches with
‘regular’ device faults
If an SSK generates 4 or more
calls per calendar month, it will
have a PM call raised for an
engineer to attend and then
the performance will be
monitored for the following 12
weeks.
If the Service Calls Measured
per Quarter (SCMQ) is <1.75
per month target, it will be
closed.
If this remains above the
threshold a further month of
monitoring will occur before
deciding whether to schedule
another visit. The tracker is
shown below up to and
including May’s results.
POL00242846
POL00242846
Branch Network
Trend analysis of branches with 50% of SSKs unavailable BusinessI! Herformance
Q4 March 206
AREA OF FAILURE: Field only * Software failures are the key
root cause of failures.
Month
sTalelels * This is being looked at in detail
oloelelalIls including any complexities
Top Areas of Failure alalalala caused by the Contactless
PRINTER-RECEIPT 51 49 51 66 40 deployment e.g. approval
SOFTWARE 62 49 68 104 95 difficulties when the device
COIN ACCEPT/RECYCL 19° 21 130 2746 goes off-line.
CORE ELECTRONICS 19. 1924 2037
DISPLAY/TOUCH 9 6 4 6 9 * These are not linked to the
SCANNER/SCAN-SCALE 3 7 5 5 recent SSK incident.
RECYCLER/GBNA/GBRU 5 9 8 10° 8
CABINETRY 9511 “18s 10° 6
PAYMENT TERMINAL 1 5 8 7 5
NOTE ACCEPTOR 7 7 6 1 4
AEI terminal working time isn’t lost due to IT disruption
AE! - Device Restoration Resolution
120%
70
100% -% of Branch Incidents. 60
resolved within 4 hours ue
80% a =
~~ Service Level Target
60% 4
40% eee % of Branch Incidents
; resolved within 12 :
ae hours 10
oe “———-Service Level Target :
bY 4 oy mh © we wow i
irdb oed Wcaoh have ERH NE GRE TN GraR Ly aa ~
What happened to service performance ?
* Overall 3M Cogent system performance
is very stable.
* 94% device incidents were resolved
within the 6 hour target.
* However the 6 and 10 hour targets of
95% and 100% respectively remain
unachievable.
ye ye
Oa oe KS
-gomeeme CBES PC
POL00242846
POL00242846
Branch Network
Business II Herformance
Q4 March 206
AE - Key IT Component Failures
o C xo
a
©
See
‘ommene Canon Camera
What was done to improve it ? What is service performance doing now ?
Student Surge planning includes
New AEI Desktop PC is tested
Pilot commences July 1* with 13
devices in branch
The new device is to be rolled out to
prioritised UKVI sites between during
August and September prior to Student
Surge in October 2016
* 3M Cogent Right First Time has
significantly improved in Q3 2015-16.
This is due to the improved competency
and quality of the new 3rd party
technical support desk (Kelway picked
up the service in July 2015). This has
seen an improvement in March
* 6 hour restoration SLA is much
improved over the last 6 months
POL00242846
POL00242846
Branch Network
business II Hertormance
Horizon Business Service is not compromised by IT maintenance on a Sunday
Q4 March 206
Branch Extended Hours
5
3
=
6
g
2
=
2
Extended Hours (10.00 - 17.00 Sunday) Target
What happened to service performance ? What was done to improve it ? What is service performance doing now ?
Post Office has extended trading hours + We have requested that Fujitsu avoid * Fujitsu continue to deliver an extended
into Sunday which impacts Fujitsu’s maintenance windows during Sunday hours service whilst managing their
contractual obligation to perform trading hours (target 25 hours per forward schedule of change (security
critical IT maintenance. month). They have exceeded this over patches, IT maintenance releases,
each of last 6 months (actual 36.8 strategic business
hours)
POL00242846
POL00242846
Branch Network
Stock levels of Ingenico Paystation terminals are not compromising our business tarsnessII Herformence
4 March
Paystatlon - Next Day Break Fix Measure 13 - Paystation Stock Management
[Month Jun-15 [I Jul-t5 I Aug-15 I Sep-15 I Oct-15 I NovelS I Dec-15 I Jan-16 I Feb-16 I Mar-16 I Apr-6 I May-16
[> Opening Stock 64 m I 19 I 6 I nun I mo I mw] a I 2 0 0 1
New Orders M4 34 51 68 49 a 34 64 cu 67 0 100
Stock Added 64 a4 2B 3 SL 36 1 66 37 67 1 151
Closing Stock
.% of Devices replaced next Working Day Service Level Target
* Paystation performance is stable with * Property Projects have had 214 * Next Day break fix is 99% against a 95%
stock unit availability being much confirmed as to be returned to target however branch failures still
improved Chesterfield need to be flagged to IT to pick up
with Ingenico
* Demand Supply process is embedded * These are being returned to Ingenico
with Network providing Ingenico with and a healthy surplus of stock has been
early visibility to ensure no backlog developed.
scenario redevelops
* Monthly stock on hand volumes being
* 249 break fixes were completed in ‘micro managed’ by Network and IT
May teams
POL00242846
POL00242846
Branch Network
MoneyGram end to end transaction processing is working effectively business! Hertormance
. Q4 March 206
MoneyGram Network Domain performance
( >)
Hours Latency
6.00 55
4.03
4.00
soo 2.08 1.92
1.23 133 T33
7 O38
o EB» i ie
0.00 mm I
z BY > > >
§ S $ S$ $ $ 5 B B N N © és
§ 3 ~ e a xv : a 0 a ; . a &
x x P > e # 8 Ny Ss N 8 8 Ss iS
8 to 8 ty tN N 5 S S S Ey Ey 2 2
x x & 2 2 2 ° & & & o a
2 2 a - a Week Starting
XN S
What happened to service performance ? What was done to improve it ? What is service performance doing now ?
* Overall 3M Cogent system performance Increased training and improved speed * 3M Cogent Right First Time has
is very stable. to competency of new Kelway technical significantly improved in Q3 2015-16.
support desk agents This is due to the improved competency
* 85% device incidents were resolved and quality of the new 3rd party
within the 6 hour target. * Improved spare stock availability technical support desk (Kelway picked
up the service in July 2015). This has
* However the 6 and 10 hour targets of seen an improvement in March
95% and 100% respectively remain
unachievable. * 6 hour restoration SLA is much
improved over the last 6 months.
POL00242846
POL00242846
& Branch Network
Problem Management — MoneyGram Trend tusinessII Hertormance
Analysis 01/04/2016 - 31/05/2016
GY:18825408
P3 01/04/16 GY:18840283 GY:18863105
Accenture Incident P3 05/04/16 P3 11/04/16
Accenture Incident I NBSC advise MoneyGram
networkadown Total
an
°
=Total
+
OFRNWEAUDNOE ©
01/04/ 04/04/ 05/04/ 07/04/ 08/04/ 11/04/ 12/04/ 14/04/ 20/04/ 26/04/ 29/04/ 03/05/ 07/05/ 09/05/ 10/05/ 12/05/ 20/05/ 26/05/
2016 2016 2016 2016 2016 2016 2016 2016 2016 2016 2016 2016 2016 2016 2016 2016 2016 2016
Graph represents total number of Incidents raised within the Atos SDM12 Toolset by Date. The
graph also highlights incidents of significance.
POL00242846
POL00242846
Branch Network
Problem Management — MoneyGram Trend & tusiessI Hertormance
Q4 March 206
Analysis 01/04/2016 — 31/05/2016
Load
1.8m ws 57 eae Resporise Time 1,356
I AL aad Vath diva banat . Adndnnndanloidindemenddeniodeals
toad
The above monitoring on this slide has been provided by Accenture and covers from 01/04/2016 to
30/04/2016. Both Load and Response time are measured.
The below graphs provide a closer view of the response time spike on the 11/04/2015 and an increase in
calls to the Atos Service Desk.
O1.SK este BP caisienn Response Time
POL00242846
POL00242846
& Branch Network
Problem Management — MoneyGram Trend tusiessI Hertormance
Analysis 01/04/2016 — 31/05/2016 °
Load LZ sais 52 sais min Response Time ( 1,208 ms average
The above monitoring on this slide has been provided by Accenture and covers from 01/05/2016 to
31/05/2016. Both Load and Response time are measured.
The below graphs provide a closer view of the response time spike on the 03/05/2015 and an increase in
calls to the Atos Service Desk.
toad O.1M cats 106 esscrmin
4
POL00242846
POL00242846
Branch Network
business II Hertormance
Q4 March 206
Volume of Severity 1 & 2 incidents — May 2016
l EI _ I 3 EISEN \ 4 A
DVLA 1 c Severity 2's recorded this month. A failure
Horizon 2 2 ue SED { 2 8 within the Horizon Service on the 9th May
4 poneyaram ‘ 1 ENR 7 4 resulted in many Branches unable to
aystation \ i ‘ ‘ complete transactions for 2 hours.
Drone bes i] [POcal fi N i °
SSID FT 1 + Anoverrun of the overnight batch process
-IBT-Telephone 1 I 4 within the Credence system resulted in Users
Lottery TEE i I i i being unable to access the system for 3 hours
GlobalPay 1 aptoess 3 on the 13th May.
POLSAP Zz
Contact HPBB Services 2 i z 1 6 + The Post Office Money website was
Centres NSBC Servi ces 1 unavailable to Customers for 4 hours on the
Home Phone & rs 23rd May. Access was still available via the
leinanee Credence/MbM FUNC NEE a 1 i 3 main Post Office website.
patties PODG - File Transfer 2 ray aT \ . Tout ie 6 + Transtrack experienced multiple failures this
cent POLSAP_ i as STUNG HENS EMU STILE uae NETES month, This was as a result of the age of
Me Local Collect 0 equipment and application causing outages.
Track & Trace 0 The upgrade to the new system successfully
Office — SharePoint, Intemet I 2 EEE f SINS UNE S 4 went live on the 28th May.
Post Office IWAVE/Top Up i 8 6 5 1 a i i 4
POLSAP NN IRIE 2 8 ig 1 + POCA experienced two issues during the
I Transtrack RIRIEL 6 he 1 TULANE 4 ain month ~ one involving the Bank of Ireland
Supply Chain potemet 1 i i and one involving HP. Both were successfully
Shared Drive MIN IRENE 3 EES resolved.
Site Network/Power I «I : PRED AUER GE i
Website CDP 2 6 2 4 2
15