POL00002091
POL00002091
Co)
FUJITSU
shaping tomorrow with you
NFSP Briefing
4 July 2012
COMMERCIAL IN CONFIDENCE © Copyright 2012 Fujitsu Services Limited
POL00002091
POL00002091
Agenda FUITSU
Introductions and Scene Setting 10:00 - 10:10 Mark Gordon
Horizon Overview 10:10-11:00 Alex Kemp
Horizon Resilience Model 11:00- 11:30 Alex Kemp
le eee eee 11:30 - 12:30 ae nae
Event Management ae
Lunch 12:30 - 13:00
Major Incident History 13:00- 14:00 Steve Bansal
Tactical/Strategic Plan Update 14:00 - 15:00 Graham Welsh
AOB 15:00 - 16:00
Close 16:00
z/SS6/4
POL00002091
POL00002091
History & Overview FUJITSU
™1996- 1998 ICL Pathway Contract
™1999-2007 Horizon
™2008- 2015 Horizon on-Line
@ Horizon On-line (Active/Passive Datacentres)
1,500 Locations
0,000 Counters / 68,000 users/logons
1 42,000,000 Transactions/week
argest ADSL network in Europe
™ Transaction processing time typically circa 0.5 second
I 11,000 Helpdesk incidents/month
900 servers in 2 Datacentres and Bracknell test location
™@ One of Fujitsu's largest UK private sector customers
e/SS6/4
2 COMMERCIAL IN CONFIDENCE
Horizon On Line - Key Deliverables
vISS6/4
m™ Data Centre Operations
® Central Network Service
m@ Engineering Service
m@ Systems Management Service
@ Horizon Service Desk
@ Branch Network Service
m@ Service Management Service
@ Application Development
@ Application Support
POL00002091
POL00002091
Fujitsu
COMMERCIAL IN CONFIDENCE
POL00002091
PoLoo002091
Agenda —Fujirsu
UE
Introductions and Scene Setting 10:00: 10:10 Mark Gordon
Horizon Overview 10:10-11:00 Alex Kemp
Horizon Resilience Model 11:00- 11:30 Alex Kemp
at ier al 11:30 - 12:30 nae rane
Event Management oe
Lunch 12:30 - 13:00
Major Incident History 13:00- 14:00 Steve Bansal
Tactical/Strategic Plan Update 14:00- 15:00 Graham Welsh
AOB 15:00 - 16:00
Close 16:00
sini
S/SS6/4
POL00002091
POL00002091
Horizon Online System Overview - Fujitsu :
DR/Test Datacentre
i Transaction NetWOrks ums)
Servers
Network Devices
- Intercampus I
Links
yt
Business Logic
Reference Data
Created
ADSL, ISDN, VPN an
Reference Data Deliver ; re ~
Tivoli. is
Horizon Base Units
9/SS6/4
COMMERCIAL
POL00002091
POL00002091
Branch Base Units Fujitsu
@ High quality Fujitsu PCs with 300
(1%) base unit breaks per month.
@ Advanced point of sale functions
@ Reference Data can dynamically
change prices and products.
® Encrypted internal hard drive(s)
@ Encrypted individual transactions.
@ Virtual Private Network (VPN)
to data centres
between branch counters
COMMERCIAL IN CONFIDENCE
POL00002091
POL00002091
Branch Router and Local Branch Networks —_—rujitsu_
@ 4 main network technologies:
mADSL - broadband
@ ISDN - digital dialup network
m VSAT - satellite dishes
I Wireless 2G / 3G - on Vodafone and EE
(Orange/T-Mobile) SIM cards. everything everywhere
6 BT@e
vodafone
@ Special locations:
Mobile Vans
i Kingston Telecoms in Hull
ural Serve Locations (e.g. Pubs)
verseas British Forces (BFPO)
vents (e.g. Party Conferences)
g/SS6/4
7 COMMERCIAL IN CONFIDENCE
POL00002091
POL00002091
Branch & Central Network FUSITSU
UK ADSL m™ Brings together BT, TalkTalk, C&W,
Internet, EE and Vodafone traffic.
® Diverse cable paths where available.
@ Enterprise quality Cisco and Juniper
devices with complex configuration
supporting multiple environments.
m@ 94 routers, 105 switches, 68 firewalls in
Post Office data centre racks.
m@ Application Control Engines (ACE) for load
balancing individual traffic paths.
@ Horizon has one of the world’s largest
network management systems.
COMMERCIAL IN CONFIDENCE
POL00002091
POL00002091
Horizon is a highly secure system Fujitsu
@ Intruder detection on inbound
network packets
@ File integrity monitoring
@ Anti virus updates
™ Operating system patching
@ Regular key material refresh
@ Regular security audits
m@ Support users:
Limited vetted UK staff have access
to transaction data
All system commands are controlled
and logged
Ikey logon protection Oe.
OL/SS6/4
COMMERCIAL IN CONFIDENCE
POL00002091
POL00002091
Horizon Data Centre Servers FUjITSU
m™ >900 servers of which 40% are live and
60% are for test and software
development.
m >120 different “types” of PO servers
running Linux, Windows and Solaris.
® Mostly virtualised on “Bladeframes”
@ High volume of change per annum:
™ 1,800 operational changes.
= 1,500 software release notes (each usually
updating multiple servers).
= >2 million overnight batch jobs per
annum (backup, file transfers, etc).
LL/SS6/4
10 COMMERCIAL IN CONFIDENCE
POL00002091
POL00002091
Horizon Data Centre Storage —-Fufitsu
® Online storage is critical for the operation of
Horizon Online servers
™ Data is automatically copied between the
data centres over 20Gbps intercampus links ir
milliseconds.
@ 5 different storage technologies:
m™ DMX - super fast online storage
® Clariion & Celerra - online storage
® Centera — write once audit arrays
= Virtual Tape Libraries - for backups
@ Post Office data centres have over 2,500
concurrently spinning hard drive spindles!
ZL/ISS6/4
1 COMMERCIAL IN CONFIDENCE
POL00002091
POL00002091
Transaction Networks - FUSITSU
eLiSsé/si
@ Post Office data centres are connected to
lots of different transaction providers
around the world.
@ Fujitsu monitor the transactions and
network connections (where allowed) to
identify failure events.
@ Failures with external providers can cause
particular transaction types to become
unavailable on Horizon.
® @ Transactions are reconciled overnight.
™ Post Office employees can access reports
to analyse transactions.
12 COMMERCIAL IN CONFIDENCE
POL00002091
PoLoo002091
Agenda - Fujitsu
CEE
Introductions and Scene Setting 10:00: 10:10 Mark Gordon
Horizon Overview 10:10-11:00 Alex Kemp
Horizon Resilience Model 11:00- 11:30 Alex Kemp
TCA NTthe Wai i
Event Management oe
Lunch 12:30 - 13:00
Major Incident History 13:00- 14:00 Steve Bansal
Tactical/Strategic Plan Update 14:00- 15:00 Graham Welsh
AOB 15:00 - 16:00
Close 16:00
SS neem
COMMERCIAL IN CONFIDENCE
v
e
POL00002091
POL00002091
Resilience Introduction FUJITSU
m ‘the ability to provide and maintain an agreed level of service in
the face of faults and challenges.”
™ Given the volume of equipment used to support the Horizon
service resilience incidents happen at predictable frequency
(e.g. hard fault on a network circuit every 6 years).
@ Resilience loss resolved by Incident Management Process.
me SS nemesis
ISS6/4
S
POL00002091
POL00002091
Horizon Resilience Levels FUJITSU
Immediate/Automatic - for
components necessary for PO
branch trading. Horizon
design target is that any
required automatic failovers
complete within 2 minutes.
m Intermediate - for components which are not used in PO
branch trading, but are still essential for daily operations.
(typical recovery time within 2 hours).
m™ Longer Term - for components with negligible impact to live
service. This might include test rigs, provisioning servers, or
cryptographic key generation workstations.
QL/SS6/4
15
POL00002091
POL00002091
3 types of Immediate/Automatic failover FUJITSU
LVISS6/4
m N+1. N servers are capable of handling the full workload, with N+1
deployed; an intelligent switch (e.g. Cisco ACE) probes each server to
confirm if service is available — then distributes requests across all
active instances.
m Active/Standby pairs. Active and warm standby servers are each
capable of handling the full workload. They exchange heartbeat
messages and the standby will only take over if it decides that the
active server has failed.
® Clustering. A cluster is configured using a chosen technology (e.g.
Oracle Database); such that should a cluster member fail, remaining
servers will automatically take over the workload.
16
POL00002091
POL00002091
Important Service Resilience Models FUsITSU
Branch Access Layer N+1 Load Balanced Server Array
Primary — 4 Node Oracle Database
I Cluster
Secondary — Oracle DataGuard
Branch Database
I (manual aie)
Network Banking & APOP Postal Order Oranle Database Cluster
Database
Main Host Database I I I I ActivelStandby Clustered Conf iguration
Debit Card and EtopUp Agents Active/Standby Clustered Configuration
Network Banking Agents I I Active/Standby Clustered Configuration
DVLA, PAF, APOP & Moneygram
N+1 Load Balanced Server Pair
Agents
SL/iSS6/4
POL00002091
POL00002091
Central Network Resilience FUJITSU
6L/ISS6/4
Design has no single points of failure.
Cisco components - excellent reputation and the Horizon
hardware reliability exceeds vendor targets.
Intercampus links ordered with full seperacy/diversity.
WAN services ordered with full seperacy/diversity.
Triangulation of some WAN external services.
Firewall clustering with configuration replication.
ACE (Load Balancing & Transaction Encryption Terminating)
active/standby Cisco switches including the ACE Modules with
configuration replication.
Ongoing failover testing as part of agreed annual test plan.
18
Data Centre Site Resilience
Data Centre environment
provides facilities resilience:
m Air Conditioning
m Multiple Mains Feeds
m UPS Batteries
i
Fuel Generator
0z/SS6/4
POL00002091
POL00002091
POL00002091
POL00002091
Agenda FUJITSU
Introductions and Scene Setting 10:00: 10:10 Mark Gordon
Horizon Overview 10:10-11:00 Alex Kemp
Horizon Resilience Model 11:00-11:30 Alex Kernp
Capacity Management Steve dsekar
Transaction Monitoring 11:30 - 12:30 St
Event Management piel
Lunch 12:30 - 13:00
Major Incident History 13:00- 14:00 Steve Bansal
Tactical/Strategic Plan Update 14:00- 15:00 Graham Welsh
AOB 15:00 - 16:00
Close 16:00
snnnntnininaanaatnnininni wn
bz/ss6/4
Z2/SS6/4
POL00002091
POL00002091
Capacity Management & Limits FUSITSU
™ Capacity Management:
Post Office estimates the business volumes
Fujitsu implement the infrastructure to support those volumes.
@ Limits:
Contractual Limit
eThe maximum volumes that Fujitsu contract to support.
Design Limit
«The volumes that the system can support without significant
failures.
POL00002091
PoLoo002091
EPOSS and Settlement Transactions - May 2012
25,000,000
20,000,000 +
EPOSS and Settlement transactions Ma
Wed 02/05
Thu 03/05
Fria fos
Sun 06/05
Mon 07/05
Tue 08/05
Wed 09/05
Priltyos
Sun 13/05
Mon 14/05
Tue 15/05
Wed 16/05
Fri 1305
Sun 20,05
Wed 23/05
Thu 24/05
Fri 25/08
Sun 27/05
Mon 28/05
Tue 29/05
Wed 30/05
Thu 31/05
POL00002091
PoLoo002091
EPOSS and Settlement - Monthly
359,000,000
Design Li
I Contracted Volume
F-09111) 1) i eee eee
I
I
i
I
300,000,000 I
I
I
i
I
I
200,000,000 q———-—~
150,000,000 -
100,000,000 -
50,000,000
EPOSS and Settlement transactions - Months
POL00002091
PoLoo002091
j
Post Office Card Account transactions - May 2012
3,000,000 ©
2,500,000
Contracted volume
2,000,000
DSO OG iii £
1,909,000
500,000
POCA transactions May 2012
a
NN ANN NN NNN NNN NNN NNN NNN NNN NN NON ON
sass e tS BSSSSSSHKESS ASS SEAS SEES ESS S
28225343 e 2852522 82425282 8225882832 88852528 825828
wa WW A RR FF A AF FP FR PAP PP Gp RP OF RP RP PR RP
Sseeces8 5858 88858 SSBB ESHE SE SESS SEERBSBESESZESESES
f2eexgegcxege eee 2222 222228222822 282 28
SHA FR SE PSSFSRPAS FR SR SASS AS SFR SR SERS SR
sss e628 S58 FSRBRR BTS SER TARR RR AR AR RR RR
*SUSER SES VP BER SEE BVP BZBERS SE eVvpezERe Ss £2RB
os _ So s 5c ue => ss od o = — [=] =
FeFge wo asFeeger © aAsFee was Feet “wage se
i
I
]
i
i
i
i
i
I
i
i
i
i
i
i
i
i
i
i
i
I
POL00002091
PoLoo002091
Post Office Card Account - monthly —
35,000,000
30,000,000
20,998,000
15,000,000
10,000,000
5,000,000
POCA transactions - Months
POL00002091
PoLoo002091
Credit and Debit Card Settlements - May 2012
§00000 ~
590000 ~
400000
300000
200000
199000
Contracted Volume
Tue 01/05
Thu 03/05
Frida /o5
n
s
=
a
S
=
E
2
Thu 10/65
Frili/o5
Credit and Debit Card Settlement- May 2012
n
s
S
S
a
3
=
2
Mon 14/05
Tue 15/05
Thu 17/05
Fri1g/o5 :
Sat 19/05
Sun 20/05 :
Mon 21/05
‘Tue 22/05
Wed 23/05
Thu 24/0!
5
5
Fri 25/0
Sun 27/05
Mon 28/05
Tue 29/05
Wed 30/05
POL00002091
PoLoo002091
Credit and Debit Card - monthly
8,000,000 ~
FQOO,000 penn
Contracted Volume
6,009,000
5,000,000
4,000,000
3,000,000 ~
2,009,000 -
1,000,000
Credit and Debit Card Settlement - Months
=
Sete PPP PPP PPP PPP SB
- &£ 5 &§ & € S B&B 4& ¢# 5 é4 5 5 &£ € = &
§ 25 Bs § 53 $8 62 6 2 S&S EF
S§ 2s 2283 245084 S&S & 5228 2
Och.
dJan-12
62/SS6/4
Governance
@ Weekly reporting
™ Conference call every two weeks
= Monthly review meetings
@ Escalation to Service and Commercial review
POL00002091
POL00002091
FUJITSU
POL00002091
POL00002091
Monthly review suMMAly FUSITS
TransactI Transact] Total Average Transaction % - :
Service Peak? IAvet TransactI Daily I Daily YTD] Monthly pucces} Comments
Day Day ions Last YTD
NBS Network Banking 1,293,506) 631138] 19,565,288)/¢7 B2[e8 16) 3.6] 99.99%
CAPO Card Account Transz gae573I 45,755] ize88.40ie Bae ore 27 VYooalink continuing to increase, monthly total at
Santander Personal & Business & 87,229) 47,091] 1,459,812) i 8.0.8 O8)e* 28 92% of contracted volume. Atleast two
Yocalink Banking Services 264,484 169,293] 5,217,075/¢t nel ee 3altt 53 proposals under discussion which mayincreare
Vocalink transactions.
; - Steady increase year on year, may need review
pcs pepitCredit Card 292,055] 163,440] 5,066,645/Sx 49178 10} 2.0] 99.98%] before peak business tow ards year end
ETu Electronic Mobile 48,905 27377] —94e,695/tt ae sole 3.0] 99.77%
‘op up
Web Services
DYLA DYLA Taz Dise 276,132 53434] atlse7I2* a7[s “28% -0.8I 100.0%
Kahala Parcelforce 541 298 g24elee w2itt 25ltt 10.7] 99.92%
Telecoms I Bank Account a x
BACS Checker 19,813) 10222] —316,886/4P ally 02Iz 18I 93.87%
Telecoms AQ] Broadband Checker 463 236 reel be 51g} 45.0] -43.3] 96.98%)
Automated Postal se Pa R ole e 5
APOP Orders & Payout 159,608 95,806] 2,969,986]: ale 2.0) 1.1} 100.0%
MoneyGram I Money Gram cash 60,133 22,363] 1,003,243] elt 73) a4] 99.94%
PAF Postal Address 54,207 32.183I -997,68iIff 21st zit 10.3] 100.0%
Finder
“Transaction Yolumes
APS Automated Payment 1057578] 555,565] I 17,222.509I4 mea gift 66
Bill Payment Service ate : ue :
BDC Bureau De Change 100,262 4ige6I 1,292,255/¢ 30.5] ato} 33.6 seasonal increase
EPOSS Electronic point of 9514293] 4,549,054] w1ozagez}q 2.91% -23)% “10
Sales & Service
New Services
App Enrolment :
a AEI identification 4.408 2ralI 86,221 ft nett 62) 8.3] 99.96%
2 Post Office Card ei
FA PCAI(PoCay 9, tL 5,152! 3002 93,063/4% 102I
ao
o
POL00002091
POL00002091
Agenda - FUSITSU
Introductions and Scene Setting 10:00: 10:10 Mark Gordon
Horizon Overview 10:10-11:00 Alex Kemp
Horizon Resilience Model 11:00-11:30 Alex Kernp
Capacity Management Steve Parker
Transaction Monitoring 11:30 - 12:30 Stays Banal
Event Management
Lunch 12:30 - 13:00
Major Incident History 13:00- 14:00 Steve Bansal
Tactical/Strategic Plan Update 14:00- 15:00 Graham Welsh
AOB 15:00 - 16:00
Close 16:00
scnnntnnnninanaatcnnnni wn
be/ssé/4
POL00002091
POL00002091
Web Transaction Monitor
CO Sr—™—C—C—C
ze/SS6/4
Details of transactions
ee/SS6/4
ank
Monitor:HNGX_NWB_ SCANNER Start Time:
Response Code:
oo YO hw NM
Description:
Authorised OK (Bank)
Declined by Bank - Impound Card
Declined by Bank - Invalid PIN
Declined by Bank - Insufficient Funds
Declined by Bank - Usage Violation{ameunt)
Declined by Bank - Trsmsaction not supported
Declined by Bank - Other
Declined By Bank - Maximum PIN tries exceeeded
Failed by FI - Retry Transaction
Fsiled by FI - Try ister, FI unavailable - Return card
Failed by Counter:Counter Timeout
Failed at Counter: Card check failure (by PO cleric) before Ack Received
Cancelled during pin entry
Transaction declined by PinPad before Ri
Cardholder Verification Error
34 End Time: 2012-06-15 1
Count:
76s0
1
139
114
13
oonoooo go GO GO OF NO
POL00002091
POL00002091
Drill down
Monitor 2 :
Response Code:3 Start Time: 2012-06-15 11:34:34 End Time: 2012-06-15 11:44:34
Bank: Count 8
JRL 4 3
Santander 2 1
Barclays Bank Pie 3 z
Lloyds TSB 18 12
Nationwide Building Society 2 i
The Co-operative Bank 8 5
CAPO 7 66
HBoS. s 3
Nerthern Bank Ltd i tL
RBS 4a 3
Yorkshire Bank Ple i 1
NS&I 2 L
Post Office Financial Services i i
POL00002091
POL00002091
velssé/4a
Se/Ssé/4
Usage
= Monitored by Systems Management Center
B Alerting notified to Fujitsu and Post Office duty mangers
POL00002091
POL00002091
FUJITSU
POL00002091
PoLoo002091
Agenda - Fujitsu
CEE
Introductions and Scene Setting 10:00: 10:10 Mark Gordon
Horizon Overview 10:10-11:00 Alex Kemp
Horizon Resilience Model 11:00- 11:30 Alex Kemp
at ier al Wa0-240 Se
Event Management Mes
Lunch 12:30 13:00
Major Incident History 13:00- 14:00 Steve Bansal
Tactical/Strategic Plan Update 14:00- 15:00 Graham Welsh
AOB 15:00 - 16:00
Close 16:00
scnnnitnnninanmaatnaninnninni wn
9e/SS6/4
POL00002091
POL00002091
Managed Environment Fujitsu
Event Management
NetCool/Reporter
Business Sysherrs
Whores
Network Devi = stat
etwork Devices Branch Estate
LeiSsé/4
se/Sse/4
POL00002091
POL00002091
Central Views FUjITSU
Tivoli Business Service Views (TBSM) view alerts in a
business’ context, correlated to the application or system
that is impacted
WebTop provides a filtered view of all events not consumed
within TBSM including counter and Security Alerts
Reporter holds Event Data consumed by TBSM and WebTop
for 1 month to allow historical reporting.
All events that reach SYSMAN are audited
6e/SS6/4
POL00002091
POL00002091
Tivoli Business Service Views _
.
-.
sees
Btncieaepnasis ROM Rawieaate cit
uea ?
ov/Ssé/4
POL00002091
POL00002091
be/ssé/4
POL00002091
POL00002091
Event Data FUJITSU
Collection of 1.6 million plus events per day
Security Events
Application events
Simple Network Management Protocol Events
External Application Health Checks
Business Application Data
POL00002091
PoLoo002091
Agenda —Fujirsu
CEE
Introductions and Scene Setting 10:00: 10:10 Mark Gordon
Horizon Overview 10:10-11:00 Alex Kemp
Horizon Resilience Model 11:00- 11:30 Alex Kemp
SAE a 11:30 - 12:30 nae cae
Event Management oe
Lunch 12:30 - 13:00
Major Incident History 13:00- 14:00 Steve Bansal
Tactical/Strategic Plan Update 14:00- 15:00 Graham Welsh
AOB 15:00 - 16:00
Close 16:00
sna wn
cv/SS6/4
POL00002091
PoLoo002091
Agenda - Fujitsu
CEE
Introductions and Scene Setting 10:00: 10:10 Mark Gordon
Horizon Overview 10:10-11:00 Alex Kemp
Horizon Resilience Model 11:00- 11:30 Alex Kemp
at ier al 11:30 - 12:30 nae rane
Event Management oe
Lunch 12:30 13:00
Major Incident History 13:00- 14:00 Steve Bansal
Tactical/Strategic Plan Update 14:00- 15:00 Graham Welsh
AOB 15:00 - 16:00
Close 16:00
ssnntnnininnntnanne
ev/SSé/4a
vviSSé/4
Major Incident Summary
# 12 December 2011
= 1 February 2012
= 1 March 2012
= 2 April 2012
= 25 April 2012
Server/Storage Disk Locking
Ref Data Token Error
Suspected Cisco ACE Blade
Security Patching
Debit Card (Streamline/TNS)
POL00002091
POL00002091
FUJITSU
Major Incident Detail
= 12 December 2011 - Server/Storage Disk Locking
« 12:54 - 14:30 degradation to on line transaction service
» Database locks caused by Bladeframe firmware issue
» 11 actions identified (all complete)
» No reoccurrence
= 1 February 2012 - Ref Data Token Error
Sv/SS6/4
* 08:00 - 11:15 loss of POCA and 1% of AP transactions
» Caused by code logic weakness
» 23 actions identified (all complete)
« No reoccurrence
POL00002091
POL00002091
FUJITSU
9v/SS6/4
POL00002091
POL00002091
Major Incident Detail FUjITSU
= 1 March 2012 - Suspected Cisco ACE Blade
» 11:07 - 14:25 degraded transaction processing
» Caused by suspected software bug on ACE Blade
* 6 actions identified (all complete)
» No reoccurrence
= 2 April 2012 - Security Patching
» 08:00 - with reducing impact until 11:27 causing performance issues at
4% of the estate
» Caused by security patch issue that did not occur during testing
* 13 actions identified (12 complete)
«1 open action to recreate fault in test environment
« No reoccurrence
LviSS6/4
Major Incident Detail
= 25 April 2012 - Debit Card (Streamline/TNS)
» 11:18 - 12:19 Debit and Credit card transactions
» Caused by suspected fault in transaction provider domain
» 9 actions identified (all complete)
» No reoccurrence
POL00002091
POL00002091
FUJITSU
sp/SSée/4
Problem Management
* Counter Transaction Processing
Affected End User I Branch Estate]
Reported by Alex Kemp
Problem Manager I Andrew Hemingway
Classification O Operational
§ Software
NumberofUsers I 40-70% ofuserbase
affected
Urgency 2
Pronty 2
Problem Summary
Counter Transaction Perfomance
Last week we analysed the milliseconds each transactiontakes
and found anissue in the recent version of the IBM Tivoli
software that has a fiected counter transaction performance
Problem
Description
. Most of the counters now have this version of the software We
are still well inside SLA butit is not as gptemum as we would
usually prefer.
The issue was not happening all the time so PMs would
perceive periods where their counter would appearto go slowly
and then speed up again.
Impact
When a counter (with the new version) would experience over
doubling of transactiontimes. The transactions were still
completing and nottiming out.
Assignee‘Group
Steve Bansal
Linked Incidents
list the ncidents which are linked to the problem}
POL00002091
POL00002091
6r/SS6/4
POL00002091
POL00002091
Problem Management rufirsu
* Counter Transaction Processing
Bansal, Steve 27/06/2012 12:32 00:01:04 Log Comment POA-Horizon Meeting has been scheduled for 29/06/12 with SDU's to
agree impacts and limitations of software actions Bansal, Steve 27/06/2012 09:17 00:10:02 Log Comment POA-Horizon A
workshop has been held on this issue and a resolver group have reviewed the issue.
MSS (JA, SBM,AC), SMG (MC), SMC, RM (JB), Test (BS, MW) and Integration (MS)
The current work around will remain in place with a clear set of working instructions.
Actions
1) MSS - To produce a work instruction for support teams to toggle notifications on and off. This will take into account MSS
work shift patterns.
2) RM - To provide MSS with Release Note instructions for all composite products, to pre cache products that need to be. This
will go through LST testing.
Bansal, Steve 27/06/2012 08:55 00:00:35 Log Comment POA-Horizon Fujitsu response back 25/06/12
The newer Tivoli version has been in distribution for the past few months and is nearly complete.
Our workaround is that we have changed the scheduling for how jobs are loaded for the overnight software distribution.
Collecting the notification of new scheduled jobs causes an unexpected performance issue in the newer version. The
scheduling of work is a human function and the counter impact happens a bit later. There are some days where no jobs are
scheduled so there is no impact. The actual jobs are running out of hours as per the older version.
Bansal, Steve 27/06/2012 08:51 00:01:32 Log Comment POA-Horizon Fujitsu response 25/06/12
Last week we analysed the milliseconds each transaction takes and found an issue (that Mark Gordon has briefed Dave
Hulbert about) in the recent version of the IBM Tivoli software that has affected counter transaction performance. Most of the
counters now have this version of the software. We are still well inside SLA but it is not as optimum as we would usually
prefer.
Bansal, Steve 27/06/2012 08:50 00:00:41 Log Comment POA-Horizon POL (Alex Todd) contacted Fujitsu on 25/06/12
We are getting feedback regarding transaction speeds.
POL00002091
POL00002091
Agenda FUJITSU
Introductions and Scene Setting 10:00: 10:10 Mark Gordon
Horizon Overview 10:10-11:00 Alex Kemp
Horizon Resilience Model 11:00-11:30 Alex Kernp
Capacity Management Sieve Bere
Transaction Monitoring 11:30 - 12:30 Sa dene
Event Management gale:
Lunch 12:30 13:00
Major Incident History 13:00- 14:00 Steve Bansal
Tactical/Strategic Plan Update 14:00- 15:00 Graham Welsh
AOB 15:00 - 16:00
Close 16:00
snnnitnnninaaimaatntnnni wn
os/SS6/4
POL00002091
POL00002091
Four Step Plan - Objectives —rufirsu
Tactical investigation and activity Independent Service Line peer review
to ensure stability in the short Employing Fujitsu and partner experts to review and make changes as needed to ensure system and monitoring
term in place to address similar incidents occurring (Oracle / Cisco / EMC etc)
Detail the four incidents and why separate and not endemic
Horizon Online ™system and ™ Clearly define the role of Horizon Online and the changed business needs of Post Office moving into the future
service review performed by Review team to include Account team CTO, Chris Furmanski, Fujitsu experts outside of account team and system
Fujitsu, Post Office Limited nd partners including Cisco and Oracle.
Independent Partners with Board
Level Exec steering board Review to cover:
™ ~~ Our current plans for the current upgrades planned
Review Design of system looking at tooling, potential points of failure, resilience, business continuity
Compare against incidents looking for systemic problems/ design constraints
How do we stop these points causing similar failures in the future ?
Map service affecting and non service affecting failure rates looking at trends
Review best practice in retail and finance markets and compare against design of the current Horizon
Online™
@ Output - document covering findings and recommendations
Presentation to Post Office Exec Presenting the findings from the review.
and Board = Covering:
Principles of the solution design
Reliability, transaction volumes and scale
Complexity, security standards
2
a
™ ~~ Benchmark with Gartner - findings
# — Level of change related incidents and volume of change on platforms
es
Recommendations, Future and implications for Extended hours and Channel Integration
ost Office exec visit to Data @ — Exec Visit to data centres / service desk, support teams across the enterprise
‘entres and monitoring in Belfast
bs/SS6/4
50 FUJITSU EYES ONLY
Approach & Scope
@ 28 key tasks identified
m 22 completed
l@ 3 ongoing
™ 3 awaiting confirmation to Close
@ Review of
i Design
™ Infrastructure & Configuration
m Testing & Validation
l@ Process & Governance
ZS/SS6/4
POL00002091
POL00002091
Major Incidents responses.
» Critical Components
> Change Management
*Preventative Maintenance
+ Resilience & Failover Testing
# Monitoring
_~
a
Agree Actions >
a
Service
Improvement Plan
Technology
Improvernent Plan
FUJITSU EYES ONLY
POL00002091
POL00002091
Specific Areas FUJITSU
= Design m Testing & Validation (Ref Data)
Resilience & Recovery position i Approach
* Platforms Relationship between Live & Test
* Storage Config
* Networks
m infrastructure & Configuration Process & Governance
Independent Subject Matter ™ Change Processes
Expert review & report ° Ref Data both Post Office & Fujitsu
Findings Published
* Some minor Service Improvements
identified & being implemented
Rigs & Testing Environments
Communications
« Day 2 Day
° Forward Schedule of Change
eS/SS6/4
52 COMMERCIAL IN CONFIDENCE
POL00002091
PoLoo002091
Agenda —Fujirsu
CEE
Introductions and Scene Setting 10:00: 10:10 Mark Gordon
Horizon Overview 10:10-11:00 Alex Kemp
Horizon Resilience Model 11:00- 11:30 Alex Kemp
SAE a 11:30 - 12:30 nae cae
Event Management oe
Lunch 12:30 - 13:00
Major Incident History 13:00- 14:00 Steve Bansal
Tactical/Strategic Plan Update 14:00- 15:00 Graham Welsh
AOB 15:00 - 16:00
Close 16:00
nnn wn
vS/SS6/4
SS/SS6/4
(oe)
FUJITSU
shaping tomorrow with you
POL00002091
nna OL00002091
POL00002091
POL00002091
Horizon Online™ Links to Post Office Ltd. Systems FUITSU I
@ Key Channel / Branch Focused Solutions
@ Paystation Ingenico Small footprint - competitive product to PayPoint
®@ Post & Go Wincor Self Service Mail (branching into retail)
m AEI Cogent Identify Enrolment
FS Web Portal(s) csc Access to Quoting tools for FS products
lm POCA HP/EDS Benefits Card
® Vocalink Vocalink Banking Network Connectivity
@ EDG csc Data Gateway for Client File Delivery
(Being replaced by PODG)
® Key Office Solutions
@ POLMI (Credence) Logica Fujitsu (Hosting) - Management Information
tj POLSAP Fujitsu Finance
ft SAPHR RMG / SAP HR
9S/SS6/4
55 COMMERCIAL IN CONFIDENCE
POL00002091
POL00002091