POL00029471
POL00029471
FUJITSU Minutes of Joint Tactical & Strategic Meeting
held at Fujitsu BRAO1 on 14/3/2012
COMMERCIAL IN CONFIDENCE
Table of Contents
ATTENDEES (ALPHABETICAL ORDER) .......cssssssesssee sesssssnessesesesesneaceesseacsnenseeeee 1
AGENDA.......cescsesseseseestsesesneneenssneesneae seeeteeaeansssneeeeasaceseneseeneateeseaeseeeansn aeeneaeeeeeeaeaes 2
MEETING OBJECTIVES .......ccscsssssssssssesnsssees sesceessseesseacsneaeeeseeacseeaeacaeeneaeentaeae anes 3
DISCUSSION POINT 1 (CRITICAL COMPONENTS) ...........:ssssssssssessssssseesnessneees 4
DISCUSSION POINT 2 (PROCESS, TESTING & MONITORING) ..
Change Management...
Preventative Maintenanc
Resilience & Failover Testing
Monitoring
DISCUSSION POINT 3 (APPLICATION) ............:s:0+0 sessssssssssssssessessseeeseseteeeneseeeeees 8
NEXT STEPS
1 Attendees (alphabetical order)
Steve Bansal Fujitsu
Graham Bevan Post Office Ltd
James Brett Post Office Ltd
Andrew Corbett Post Office Ltd
Andrew Jacques Post Office Ltd
Tim Jones Fujitsu
Alex Kemp Fujitsu
Adam Parker Fujitsu
Steve Parker Fujitsu
Peter Thompson Fujitsu
lan Trundell Post Office Ltd
Graham Welsh Fujitsu
© Copyright Fujitsu Services COMMERCIAL IN CONFIDENCE Author: Alex Kemp
Limited, Post Office Limited 2012 Version: = 0.1
UNCONTROLLED IF PRINTED Date: 16-Mar-2012
OR LOCALLY STORED Page No: 10f9
Fe)
FUJITSU
Minutes of Joint Tactical & Strategic Meeting
held at Fujitsu BRAO1 on 14/3/2012
COMMERCIAL IN CONFIDENCE
POL00029471
POL00029471
2 Agenda
Agenda
¢ Introductions
¢ Background
© 4 major Incidents — RCA — common theme all had an impact to POCA
© Review scope of tactical strategy
* Factors/considerations
© Costvrisk paradigm & tactical versus strategic
¢ Would additional footprint/segmentation /improve availability cost v benefit
*® EOSL-— bath tub scenario /risk to service could possibly increase age of kit
© Review upgrades recommended by 3” parties that need to happen between
now and end of current contract
*® New Business and service offering and impacts on Horizon™
°
eo0o00
Channel Integration
BMS
Network Transformation/extended hours
FOOG
GWS
Sales-force
* What can be done interim to benefit business functionality and reduce service
incidents.
Outputs
¢ Produce a list of recommendations activities
¢ Weekly Report on progress
* Tactical plan with timelines
e Identify any recommendations to be included into Strategic Strategy
© Copyright Fujitsu Services COMMERCIAL IN CONFIDENCE = AY"
Limited, Post Office Limited 2012 Version:
UNCONTROLLED IF PRINTED Date:
OR LOCALLY STORED Page No:
Alex Kemp
0.1
16-Mar-2012
20f9
FUJITSU Minutes of Joint Tactical & Strategic Meeting
held at Fujitsu BRAO1 on 14/3/2012
COMMERCIAL IN CONFIDENCE
POL00029471
POL00029471
3 Meeting Objectives
TACTICAL REVIEW.
Key goal:
identify changes that should be made to how we operate Horizon Online,
how the current system is monitored, and where chan ges should be made to
the key architecture components to reduce the risk of system outages
Areas to cover:
* Identification and review of all critical components that impact the
delivery of an acceptable Horizon service — where would changes to
components reduce the risk of system outage
© Operational review covering change management, preventative
maintenance, resilience and failover testing, how critical
components and monitored, whether our monitoring is correctly
aligned to business services
* Review application development and maintenance (including
reference data and system configuration data) to identify
improvements in how availability, resilience, and system integrity is
engineered and tested
‘Approach includes:
Independent input from Fujitsu service tower SME’s
Review of major incidents in terms of ‘reasons for unavailability’ and general
principles that can be applied across other domains
Mapping critical business transactions to the depen dent components
(hardware, software, services)
Review of operational and application development and maintenance
practises
© Copyright Fujitsu Services COMMERCIAL IN CONFIDENCE
Limited, Post Office Limited 2012
UNCONTROLLED IF PRINTED
OR LOCALLY STORED
Author:
Version:
Date:
Page No:
Alex Kemp
0.1
16-Mar-2012
3of9
Fe)
FUJITSU
Minutes of Joint Tactical & Strategic Meeting
held at Fujitsu BRAO1 on 14/3/2012
COMMERCIAL IN CONFIDENCE
POL00029471
POL00029471
4 Discussion Point 1 (Critical Components)
Identification and review of all critical components that impact the delivery of an acceptable Horizon
service — where would changes to components reduce the risk of system outage?
Ref I Action Tactical I Owner Start/Duration
/Strategic
A SRRC (Service Resilience and Redundancy I T Adam Parker I W/C 19/3
Catalogue) documents will be updated. These (Jason Clark & 6 weeks
cover platform, storage and central network. Mark Jarosz)
B SRRC documents will be update to coverI T Adam Parker I W/C 19/3
branch network. (Andy 3 weeks
Hemingway)
[e Once the SRRC’s are updated they will be I T/S Tim Jones & I W/C 30/5
group reviewed to identify any potential lan Trundell 1 week
solution changes - considering any new
learnings from recent incidents.
Joint action. Need to agree participants and set
a first meeting.
D SRRC documents will be update to cover I T Adam Parker I W/C 30/5
Reference Data and LST environments (Jason Clark & 4 weeks
Mark Jarosz)
E I Can the SRRC documents be better leveraged I T Tony Jamasb I W/C 30/5
in both organisations? & Adam Parker
1 day
F Perform an audit of Network Design by Fujitsu I T/S Alex Kemp I On-going Cisco
Subject Matter Experts and Cisco. (Vince RCA activity for
Cochrane) Ace Module
& Cisco Incident
G Perform an audit of Storage Area Network by I T Alex Kemp WIC 26/3 5 days
Fujitsu Subject Matter Expert and EMC & EMC
H Blade-frame Configuration audit by Fujitsu I T Alex Kemp I W/C 19/3 1 day
Subject Matter Expert from Product Group. (Cleeve
Strachan)
I Consider moving to an active/active data} S Amit Apte & I TBC
centre configuration — review, cost benefits lan Trundell
against service incidents
J Consider availability options further of N+X IS Amit Apte & I TBC
resilience. lan Trundell
© Copyright Fujitsu Services COMMERCIAL IN CONFIDENCE Author: Alex Kemp
Limited, Post Office Limited 2012 Version: = 0.1
UNCONTROLLED IF PRINTED Date: 16-Mar-2012
‘OR LOCALLY STORED PageNo: 4 0f9
POL00029471
POL00029471
FUJITSU Minutes of Joint Tactical & Strategic Meeting
held at Fujitsu BRAO1 on 14/3/2012
COMMERCIAL IN CONFIDENCE
5 Discussion Point 2 (Process, Testing & Monitoring)
Operational review covering change management, preventative maintenance, resilience and failover
testing, how critical components and monitored, whaher our monitoring is correctly aligned to business
services.
5.1 Change Management
Horizon Online™ and associated systems have >3,600 reference data, >1,800 operational changes
(MSCs) and >1,400 server release notes per annum.
Ref I Action Tactical Owner Start/Duration
/Strategic
K Review change processes and procedures T&S Graham Ongoing building
(across both organisations) considering any Welsh, Mark initially on the
new information from recent incidents and Gordon & Andy I areas identified.
the overall robustness of the process. Corbett (Post
Office Refdata)
Commenced until
Scope covers both OBC Ref Data,
Operational Change and Release Notes. 17th April
L Review of the use of model office (despite T Andrew Awaiting
limitations) in future testing activities to Jacques (Post I agreement on
minimise the risk of service impacts Office) & options to improve
Graham Welsh I validation &
(Fujitsu) verification.
M Publish the findings of the recent Fujitsu T Graham Welsh I By 29” March
Subject Matter Expert audit of the release
management process and actions arising
from the audit
N Make improvements to the information T lan Trundle & TBD
supplied as part of the change request to Graham Welsh
cover the desired business solution, criticality
of the service and impact of service
unavailability to the business.
[e) Review the impact of extended hours and Ss Graham Initial assessment
other new business offerings on maintenance Welsh, Alex and report to be
and change windows. Kemp and POL I contained within
Release the Extended
Hours Study target
Completion 29”
March.
P Review opportunities to reduce the time to TIS Graham Welsh I TBC
regress Reference Data changes. (Steve Parker)
© Copyright Fujitsu Services COMMERCIAL IN CONFIDENCE Author: Alex Kemp
Limited, Post Office Limited 2012 Version: O41
UNCONTROLLED IF PRINTED Date: 16-Mar-2012
OR LOCALLY STORED Page No: Sof9
POL00029471
POL00029471
(oe)
FU ITSU Minutes of Joint Tactical & Strategic Meeting
held at Fujitsu BRAO1 on 14/3/2012
COMMERCIAL IN CONFIDENCE
& Andy Corbett
Q Review opportunities to reduce the time to T Mark Gordon TBC
gain approval from PO Duty Manager. and Andy
Jacques
5.2 Preventative Maintenance.
Horizon Online has a Maintenance Release Manager who delivers regular preventative maintenance
releases. In addition the Fujitsu shared environments and services include their own maintenance plans.
Ref I Action Tactical Owner Start/Duration
/Strategic
R Post Office to provide Fujitsu with their key I T Andy Jacques I W/C 19/3
activities for the 18 month Forward Schedule & Graham 2 weeks
of Change. Welsh
Ss It is recognised that current and future Post I S Amit Apte TBD
Office business needs will require higher (Graham
systems availability. In turn this will need to Welsh and
have an agreed approach to Testing and Pete
Validation that balances Costs against Risk. Thompson) &
To that end there will be a need for a joint lan Trundell
review of the testing approaches deployed
today with an objective of arriving at
recommendations for test strategy revision to
fit new business requirements and priorities.
T Create a summary view of End of Service I S Alex Kemp WIC 19/3
Life Road-maps. (Jason Clark 3 weeks
and Mark
Jarosz)
© Copyright Fujitsu Services COMMERCIAL IN CONFIDENCE Author: Alex Kemp
Limited, Post Office Limited 2012 Version: 0.1
UNCONTROLLED IF PRINTED Date: 16-Mar-2012
OR LOCALLY STORED PageNo: 60f9
POL00029471
POL00029471
FUJITSU Minutes of Joint Tactical & Strategic Meeting
held at Fujitsu BRAO1 on 14/3/2012
COMMERCIAL IN CONFIDENCE
5.3 Resilience & Failover Testing
At the start of each year Fujitsu and Post Office review and agree a plan for resilience and failover testing
of Horizon Online. During 2011 there were 28 operational exercises where live systems were tested. All
observations are recorded and tracked to conclusionin the end of year report.
One of the limitations of this approach is that itis not possible to simulate all the operational failure
modes that could occur within a sub system. However the test plans are optimised to maximise the scope
of the testing.
Ref I Action Tactical Owner Start/Duration
/Strategic
U Although the 2012 test plan has been joint I T Adam May 2012
reviewed and approved there will be a further Parker/Tony
review of the test plan after the SRRC review Jamasb
cycle has completed. This will take into
consideration any new information from
recent incidents.
Vv Increase POL awareness /understand what I T Adam Parker, I May 2012 initial
level of resilience and DR agreed to and Alex Kemp & I then quarterly.
document limitations. Tony Jamasb
Information needs to be cascaded to POL
Exec and strategic Business Users and client
relationship managers within the Post Office.
W_I Provide a report to POL detailing the number I T Adam Parker I W/C 2/4
of times that the resilience in place across
the service has been initiated without causing 4 weeks
any service impacts over the last 6 months.
© Copyright Fujitsu Services COMMERCIAL IN CONFIDENCE Author: Alex Kemp
Limited, Post Office Limited 2012 Version: O41
UNCONTROLLED IF PRINTED Date: 16-Mar-2012
OR LOCALLY STORED PageNo: 7of9
POL00029471
POL00029471
FUJITSU Minutes of Joint Tactical & Strategic Meeting
held at Fujitsu BRAO1 on 14/3/2012
COMMERCIAL IN CONFIDENCE
5.4 Monitoring
The configuration of event monitoring systems is under continual improvement by Fujitsu's enterprise
management team to reflect new information from recent incidents. Fujitsu have developed extremely
detailed transaction monitoring to determine the occurrence and business impact of failures.
However recent incidents have shown the main challenge to be pinpointing the exact source of the failure
in a timely manner when there has been a subtle fault for which the vendor supplied equipment does not
generate alerts.
Ref I Action Tactical Owner Start/Duration
/Strategic
x Review monitoring design Des/sym/hld/0014 I T Saheed WIC 26/3
and identify opportunities to further improve Salawu (John 4 Weeks
the current toolsets. Bradley)
Y Note: Can counter messaging to BAL carry I S Graham Welsh I This is an
more intelligence to help monitoring and identified initiative
diagnosis through the life of a service of the Extended
incident? Hours Study
Z Can a view of transaction stats used by I T Graham Welsh I Already On-going
Fujitsu teams be shared with Post Office (Steve Parker) I Project Work
(possible change of start of day checks by & Andrew
both parties) Jacques
6 Discussion Point 3 (Application)
Review application development and maintenance (induding reference data and system
configuration data) to identify improvements in how availability, resilience, and system integrity is
engineered and tested
Ref I Action Tactical Owner Start/Duration
/Strategic
AA I Assess the viability of using the SRRC. TS G Welsh / Tim I W/C 26/3 4
documentation structure for applications. Jones to discuss I weeks
with lan Trundell
and Peter
Stanley.
AB I Review steps to minimise impact on TWS TIS G Welsh and [w/e 19" March /
changes on/from other platforms (CFD, POL Release 20" May
Credence MDM and POLSAP).
© Copyright Fujitsu Services COMMERCIAL IN CONFIDENCE Author: Alex Kemp
Limited, Post Office Limited 2012 Version: O41
UNCONTROLLED IF PRINTED Date: 16-Mar-2012
OR LOCALLY STORED PageNo: 8of9
POL00029471
POL00029471
FUJITSU Minutes of Joint Tactical & Strategic Meeting
held at Fujitsu BRAO1 on 14/3/2012
COMMERCIAL IN CONFIDENCE
7 Next Steps
¢ All parties to review actions and agree owners and estimated timelines to complete tasks
* Pete Thompson to schedule meeting in W/C 19/3 to agree planned activities
e “Firm up” the timelines of activities and associated plan
¢ Production of a report for Exec Briefing
* Feedback to Pete Thompson by close of play Monday 19/3
© Copyright Fujitsu Services COMMERCIAL IN CONFIDENCE Author: Alex Kemp
Limited, Post Office Limited 2012 Version: O41
UNCONTROLLED IF PRINTED Date: 16-Mar-2012
OR LOCALLY STORED PageNo: 90of9