FUJ00120061
FUJ00120061
oO POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
Document Title:
Document Type:
Release:
Abstract:
Document Status:
Author & Dept:
Internal Distribution:
External Distribution:
Security Risk
Assessment Confirmed
Approval Authorities:
POA Operations Major Incident Procedure
Procedure Definition
HNG-X
This document describes the POA Operations Major Incident
Management Procedure.
FOR APPROVAL
Tony Wicks— POA Operations
As listed on pages 4 and 5 for
Mandatory Review
Optional Review
Issued for information
For information
Dave Hulbert (POL), Steve Beddoe (POL), Antonio Jamasb(POL)
YES
me Role Signature Da
Steve Bansal POA Tower Lead BAS (Interim) I See Dimensions for record of approval.
©Copyright Fujitsu Services Ltd 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0001
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
PageNo: 1 of 41
FUJ00120061
FUJ00120061
oO POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
0 Document Control
0.1 Table of Contents
i} DOCUMENT CONTROL
0.1 Table of Contents
0.2 Document History
0.3 Review Details .
0.4 Acceptance by Document Review:
0.5 Associated Documents (Internal & Externa
0.6 Abbreviations
0.7 Glossary.
0.8 Changes Expected.
0.9 Accuracy
0.10 Security Risk Assessment
4 INTRODUCTION
41.4 Owner..
1.2 Rationale...
2 MANDATORY GUIDELINES.
3 DEFINITION OF A MAJOR INCIDENT.
3.
3.1 Incident Classific:
3.2 _ Influencing Factors in calling a Major Inciden'
3.3 Major Incident Triggers.
i Network Triggers...
Infrastructure Components Triggers
Data Centre Triggers ..
Online Service Triggers .
Security Triggers
SECURITY MAJOR INCIDENTS. ...........ccesessseseeee
rr
lon
CALLING THE MAJOR INCIDENT ........cseseseseeee
PROCESS FLOW...........:000
Process Description..
COMMUNICATION.........0006
Technical Bridge
Service Bridge ..
> 1
NIN
PASE
}O0 FETC FOr} 169 1 I
Major Incident Progress Template ...
Escalation Communication Protocol
Core Major Incident Management Team
Corporate Alert...
SCopyright Fujitsu Services Ltd 2014. FUJITSU RESTRICTED (COMMERCIAL IN Ref. SVMISDM/PROIO00%
CONFIDENCE) Version: 9.0
Date 15-Aug-2014
PageNo: 20f 41
FUJ00120061
FUJ00120061
POA Operations Major Incident Procedure
Pe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
8 FORMAL INCIDENT CLOSURE & POST INCIDENT REVIEW .
8.1 Calculating potential LD liability for Major Incidents ...
9 FUJITSU ROLES AND RESPONSIBILITIES DURING A MAJOR INCIDENT....37
9.1 Role of the MAC Team...
9.2 Role of the Major Incident Manager 38
9.3 Role of the Technical Recovery Manager-.. 39
9.4 Role of the Problem Manager 40
9.5 Role of the Communications Manage: 40
9.6 Role of the SDUs: (Technical Teams /SMC/MAC & Third Parties) 40
9.7 Role of the Service Delivery Manager owning the affected service
9.8 Role of the Tower Lead
10 APPENDICES 42
10.4 List of Templates...
10.2 Daytime Duty Manager Contact Details.
10.3 Out of Hours Duty Manager Contact Details
10.4 POA Service Delivery Contact Details
10.5 Special Situations
Personnel Absenc
OOH
Duty Manager Change Over.
©Copyright Fujitsu Services Ltd 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0001
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
PageNo: 3 of 41
FUJ00120061
FUJ00120061
oO POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
0.2 Document History
VersionNo. Date Summary of Changes and Reason for Issue Associated Change -
CP/PEAK/PPRR
Reference
0.4 03-Oct-06 First draft — to detail the Major Incident Escalation process.
Draft taken from Horizon Document CS/PRD/122, V1.0
1.0 11-Oct-06 Revision following comments from Reviewers
20 02-Sep-08 Changes for Acceptance by Document Review: insertion of
Section (0.4) containing table of cross references for
Acceptance by Document Review and addition of note to front
page. No other content changes
24 24-Feb-2009 I Changes made for Acceptance by Document Review by Fiona
Wootfenden including the removal of references to.
CS/PRDIO74 which has been Withdrawn and replaced by
SVM/SD/PRO/0018 and other tidying up changes.
Other changes to update Contact details.
22 14-Apr-2009 I Some Personnel Name changes and POA to POA +
Abbreviations. Security Updates to sections 5.1, 6.3, 8.2.1,
9.0,
23 3-June-2009 I Some Personnel Changes and minor changes following review
in May 2009
3.0 7-July-2009 Following security audit 2 minor changes. Alan Simpson now
on distribution and 6.3, an extra bullet entry added at end of
list
3.0 7-uly-2009 Following security audit 2 minor changes. Alan Simpson now
on distribution and 6.3, an extra bullet entry added at end of
list
34 44-Jan-2010 I Changes following director failing to sign off v3.0, plus minor
contact changes.
40 26-Mar-2010 I Approval version
44 18-May-2010 _I Following team restructure, the process has been significantly
reviewed
42 03-Jun-2010 I Updated following minor comments provided during review
cycle of version 4.1. This version will be presented for approval
at v5.0
5.0 07-Jun-2010 I Approval version
6.0 14-Sep-2010 _I Approved version following updates to personnel and table in
10.4 and section 10.8
64 15 July-2011 I Updates to personnel and changes from ‘Process’ to
Procedure’
62 05-Sept-2011 I Updates following changes requested by Bill Membery from
6.1, plus clarification of TRM role
63 14-Oct-2011 I Cosmetic changes mainly changing RMGA with POA and also
updating abbreviations
64 21-Dec-2011 I Updating of details for a Service Bridge
Also some POL requests
Despite this being an internal POA document, all external
comments that can improve the document are considered.
65 46-Jan-2012 _I Updated, following review and cosmetic changes in relation to
version 6.4
SCopyright Fujitsu Services Ltd 2014. FUJITSU RESTRICTED (COMMERCIAL IN Ref. SVMISDM/PROIO00%
CONFIDENCE) Version: 9.0
Date 15-Aug-2014
Page No: 4 of 41
FUJ00120061
FUJ00120061
co POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
Version No Summary of Changes and Reason for Issue Associated Change -
CP/PEAK/PPRR
Reference
02-Jan-2013 Changes in relation to Personnel and also Tower Leads and
other cosmetic changes
7.4 04-Feb-2013 Changes in relation to Personnel and revisions around
Communications
72 17-Sep-2013 I Major update to align with Business Assurance Management
procedures and for organisational changes.
(This version was originally identified as version 8.1)
8.0 18-Oct-2013 I Updated for minor changes from Nana Parry
8.1 10-Jun-2014 ‘Amended to replace the HSD function with the Atos Service
Desk and replaced IMT references with the MAC team.
Also updated to reflect the introduction of Atos as POL’s
Service Integrator.
9.0 14-Aug-2014 __I Implemented minor changes following 8.1 review cycle
0.3 Review Details
Review Comments by
Review Comments to Tony Wicks
Mandatory Review
Role Name
POA Tower Lead BAS (Interim) Steve Bansal
POA Acceptance Manager Steve Evans
Role Name
POA Infrastructure Operations Manager Andrew Hemingway
POA Business Continuity Manager Changdev Pawashe
POA POLSAP and Online Services SDM Gaby Reynolds
POA Credence and Sales force SDM Simon Edmondson
POA Problem Manager Stephen Gardiner
POA Lead SDM Managed Infrastructure Services Leighton Machin
POA Lead SDM Online Services Yannis Symvoulidis
POA Senior Ops Manager HNS Alex Kemp
POA SDM MAC and CMT Sandie Bothick
POA SMC Manager Catherine Obeng
POA Security Manager Kumudu Amaratunga
POA MAC Team Manager Sandie Bothick
POA Quality Compliance and Risk Manager Bill Membery
POA Network Infrastructure SDM. Roger Stearn
sued for Information — Please restrict this
distribution list to a minimum
SCopyright Fujitsu Services Ltd 2014. FUJITSU RESTRICTED (COMMERCIAL IN Ref SVMISDM/PROIO00%
CONFIDENCE) Version: 9.0
Date 15-Aug-2014
Page No: 5of 41
FUJ00120061
FUJ00120061
oO POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
Position/Role Name
POA CISO Tom Lillywhite
POA Unix Team Leader Fiona Lennox
Fujitsu NT Team Leader lan Gibson
Fujitsu DC Operations Manager John Hill
(*) = Reviewers that retumed comments
0.4 Acceptance by Document Review
The sections in this document that have been identified to POL as comprising evidence to support
Acceptance by Document review (DR) are listed below for the relevant Requirements:
POLNFRDR — IntemalFSPOL — Documeni Document Section Headin
NFR Reference Section Number
SER-2200 SER-2178 Whole Document
SER-2202 SER-2179 Whole Document
SEC-3095 SEC-3266 335 Security Triggers
SEC-3095 SEC-3266 10.5 Security Major Incidents
0.5 Associated Documents (Internal & External)
Referen: Version Date Title Source
PGMDCM/TEM/0001 Fujitsu Services Royal Mail Group Account _I Dimensions
HNG-X De it Templat
(DO NOT REMOVE) jocument Template
CS/IFS/008 POA/POL Interface Agreement for the Dimensions
Problem Management Interface
SVM/SDM/SD/0025 POA Problem Management Procedure Dimensions
PA/PRO/001 Change Control Process Dimensions
CS/QMS/001 Customer Service Policy Manual Dimensions
SVMI/SDM/PLA/0001 HNG-X Support Services Business Dimensions
Continuity Plan
SVMI/SDM/PLA/0002 HNG-X Services Business Continuity Plan I Dimensions
SVM/SDMI/PLA/0030 HNG-X Engineering Service Business Dimensions
Continuity Plan
SVMI/SDM/PLA/0031 HNG-X Security Business Continuity Plan Dimensions
SVMISDM/SD/0011 Branch Network Services Service Dimensions
Description
SVM/SDM/PRO/0018 CS Incident Management Procedure Dimensions
C-MSv1.3 Manage Incidents Process BMS
C-MSv_roles Service Management Process Roles and BMS
Responsibilities
SVM/SEC/STD/1823 LINK information security standard issued Dimensions
January 2001 (subject to such dispensations
from that standard as LINK may grant from
©Copyright Fujitsu Services Ltd 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0001
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
Page No: 6 of 41
FUJ00120061
FUJ00120061
oO POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
Reference Version Date Title Source
time to time).
!Mo02_MAJOR 5.0 23/05/2013 I Manage Major Incidents Procedure BMS
INCIDENT
MANAGEMENT
PROCEDURE
FJ- BMS- 1-AB1.6 60 24/04/2012 I Fujitsu Services Business Management BMS
Systems Process: Conduct Root Cause
Analysis
Unless a specific version is referred to above, reference should be made to the current approved
versions of the documents.
0.6 Abbreviations
Abbreviation Definition
AtG Advice & Guidance
BCP Business Continuity Plan
BMS Business Management System
Iso Intemational Standards Organisation
ITIL Information Technology Infrastructure Library
KEDB Known Error Database
KEL Known Error Log
MAC Major Account Controllers
MBCI Major Business Continuity Incident
MIM Major Incident Manager
MICcM Major Incident Communications Manager
MIR Major Incident Report
MSC Manage Service Change
MSU Management Support Unit
ocp Operational Change Proposal
OOH Out Of Hours
PCI Payment Card Industry (as per Security Standards Council)
PO Post Office
POA Fujitsu Post Office Account
POL Post Office Limited
RFC Request For Change
ScT Service Continuity Team
SDM(s) Service Delivery Manager(s)
(NB: Throughout this document SDM refers to a person responsible for the Service,
and the SDM could work in, but not limited to, the Service Delivery, Service Support,
and Release Management or Security teams).
SDU Service Delivery Unit
SLT Service Level Targets
SCopyright Fujitsu Services Lid 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref. SVM/SDM/PRO/000%
CONFIDENCE) Version: 9.0
Date 15-Aug-2014
PageNo: 7 of 41
FUJ00120061
FUJ00120061
co POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
Abbreviation Definition
SiSD Service Integrator Service Desk (Atos Service Desk)
SMC Systems Management Centre
SMS Short Message Service (as known globally within Mobile Telephone Networks)
SRRC Service Resilience & Recovery Catalogue
ssc System Support Centre
TB Technical Bridge
TP Third Party or Third Parties
TRM Technical Recovery Manager
vIP VIP Post Office, High Profile Outlet
0.7 Glossary
Term Definition
T Time of incident occurring
T+3 Time Incident Occurred plus 3 minutes
0.8 Changes Expected
Changes
Changes to reflect process and organisational changes.
This is expected to be changed for the OSR Messaging release.
0.9 Accuracy
Fujitsu Services endeavours to ensure that the information contained in this document is correct but, whilst every
effort is made to ensure the accuracy of such information, it accepts no liability for any loss (however caused)
sustained as a result of any error or omission in the same.
0.10 Security Risk Assessment
Security risks have been assessed and it is considered that there are no security risks relating specifically to this
document.
SCopyright Fujitsu Services Ltd 2014. FUJITSU RESTRICTED (COMMERCIAL IN Ref SVMISDM/PROIO00%
CONFIDENCE) Version: 9.0
Date 15-Aug-2014
PageNo: 8 of 41
FUJ00120061
FUJ00120061
oO POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
1. Introduction
1.1 Owner
The owner of the Major Incident Management process at the local POA level is the Fujitsu POA Lead
SDM, Problem and Major Incident.
Objective
The key objective of the procedure is to ensure effective and efficient management of Major Incidents,
through:
« Improvement of communication channels
« Clarification of the need to communicate awareness of potential incidents.
e Improved accuracy of reporting of incident status
e Allowing technical teams the right amount of time to diagnose and impact an incident
e Avoiding unnecessary alerting of the service integrator and/or the customer
e Demonstrating a professional approach to Atos, the Service Integrator contracted to POL, and
Post Office Limited (the customer) and their clients.
e Provision of clearly defined roles and responsibilities
« Defined reporting and updating timelines throughout a major incident.
e Improved governance
e Assessing which incidents are major and which are ‘Business as Usual’
1.2 Rationale
This document outlines the communication and management procedure and guidelines to be used for
Major Incidents impacting the live estate.
The methodology defined within this document augments the existing SMS framework procedure
presently deployed within the live estate.
The aim of the document is to provide a pre-defined procedure for future major incident communication
and management.
SCopyright Fujitsu Services Lid 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref. SVM/SDM/PRO/0001
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
PageNo: 9 of 41
FUJ00120061
FUJ00120061
oO POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
2 Mandatory Guidelines
It is important to maintain a balance between:
a) Allowing the technical teams the right amount of time to diagnose and impact an incident
b) Avoiding unnecessary alerting of the customer
c) Assessing which incidents are major
The following guidelines should be adhered to.
e During the MAC Core Hours (Monday — Friday 08:00 — 20:00, Saturday 08:00 — 17:00) and Bank
Holidays 0800 — 1400 excluding Christmas Day. The MAC should be the first point of operational
contact between Fujitsu and the Atos Service Desk. Outside these hours the Atos Service Desk
or Atos OOH Duty Manager should contact the SMC. The SMC are responsible for escalation of
incidents to the POA OOH Duty Manager. The POA OOH Duty Manager may initiate
communications with the Atos OOH Duty Manger. The SMC operate on a 24 x 7 x 365 basis.
e Any activity detailed in this document which is assigned to the MAC team is handed over to the
SMC outside the MAC Core Hours, with the exception of the above.
e The relevant technical teams who are aware of and monitoring a potential major incident must
page / call the appropriate Major Incident Manager (Duty Manager out of hours) as soon as
possible. This is not limited to major incidents alone, but applies wherever a state other than
Business as Usual has been detected. The Major Incident Manager must in turn communicate
the potential incident, to the Atos Service Desk for awareness and monitoring in Atos. This is
usually done via the MAC team in core hours or via SMC out of hours.
« The Major Incident Manager (or Duty Manager out of hours) is responsible for communicating
both up the Fujitsu organisation and across (see appendix 10.3) to their counterpart in Atos.
Where this is impractical (e.g. leave, out of hours, unavailable), the initiative should be taken to
jump up the organisation. Of prime importance is that the customer is informed in a timely
manner and at the correct touch point. This communication should be by voice or direct SMS.
The communication should include the date, time, name, nature of problem, severity, if service
affecting, likely impact, and the Fujitsu owner to contact.
e The Major Incident Manager (Duty Manager OOH, who covers Monday to Friday 17:30 to 09:00
and from 17:00 Friday though to 09:00 Monday) should also initiate communication using SMS
via the MAC team (see operational hours above.). Outside of these hours the SMS should be via
the SMC. The SMS distribution list used is titled ‘SMS Internal’ and amongst others includes the
appropriate members of the POA Operations Management Team.
SCopyright Fujitsu Services Lid 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref. SVM/SDM/PRO/0001
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
PageNo: 10 of 41
FUJ00120061
FUJ00120061
oO POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3 Definition of a Major Incident
3.1. Incident Classification
As a general rule a Major Incident will be an incident rated as a Business Critical Incident as shown in the
following
- The ‘CONTRACT’
- Sections 3.2 and 3.3 below.
- POA Operations Incident Management Procedure document (SVM/SDM/PRO/0018).
- A series of connected lower severity incidents which combine to have a significant business
impact.
However not all incidents rated as Severity 1 qualify as a Major Incident as the severity levels do not
always reflect the overall business impact to POL. For example a single counter post office which is
unable to trade, regardless of its business volumes, is rated as a Severity 1 incident.
For simplicity, incidents are classified into three impact levels: High, Medium and Low.
High — An Incident that has occurred with a significant and potentially prolonged adverse impact on POL
business. Typically these incidents will initially require a significant amount of reactive management
before they can be controlled and resolved.
Medium — An incident that has the potential to cause significant impact to POL business but can be
controlled and contained through effective management.
Low — An Incident that requires business attention but if managed effectively will not have significant
impact on POL business.
3.2. Influencing Factors in calling a Major Incident
It is important that a Major Incident is defined in accordance with section 3.3 Major Incident Triggers, as
such, because of its business impact on the day when it occurs, rather than simply being defined as a
Major Incident because it appears on a list. However the following parameters will also feed into the
consideration of whether a major incident should be called:
e Duration, i.e. how long has the vulnerability to service already existed?
« Impact across the estate, including consideration of whether a service is merely degraded or
actually stopped
«Time at which the event occurs in relation to the 24 hour business day
e Time of year —- e.g. Christmas / Easter / End of month / quarter
« Anticipated time before service can be resumed
e Impact to POL branches, customers, clients or brand image
e Business initiatives e.g. product launches
3.3. Major Incident Triggers
The following criteria could trigger a major incident, however as detailed in 3.2, the influencing factors
must also be considered. As such the list below is not exhaustive, whilst if an incident occurs which is
not detailed below, e.g., legislative, it should not necessarily be precluded from being declared a major
incident.
©Copyright Fujitsu Services Ltd 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0001
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
PageNo: 11 of 41
FUJ00120061
FUJ00120061
oO POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
It should be noted that any call trends in relation to the following, should be reported to the POA Duty
Manager as soon as the agreed threshold levels have been breached.
3.3.1 Network Triggers
Network Major Incident triggers are as follows:
e Complete or significant outage of the Central network, e.g. failure of both 3750 stack Catalyst
switches in totality for the Core layer in IRE11.
« Complete or significant outage of the Talk Talk network
« Complete or significant outage of VSAT sites
e Complete or significant outage of the ISDN network (whether C&W, BT or Kingston Comms)
3.3.2 Infrastructure Components Triggers
Infrastructure component Major Incident Triggers are as follows:
« Total loss of environments providing individual online service capability
« Breach of access to data centres
e Breach of security
° Virus outbreak
3.3.3. Data Centre Triggers
Data Centre Major Incident triggers are as follows:
e Network / LAN outage
e Loss of Data Centre, or significant loss of Data Centre Components
e Breach of security
3.3.4 Online Service Triggers
Online services Major Incident Triggers are as follows:
e Online service unavailable within the Data Centre (not counter level)
e Number of Branches without Communications Services — as defined by POL and in accordance
with Ping script thresholds.
e Third party provided service failure — e.g. DVLA, Link, Moneygram, Santander etc
N.B Once the third party service provider has been deemed to be the source of the Major
Incident; it will be managed by either POA or Atos Service Desk in accordance with whichever
organisation manages that supplier relationship.
3.3.5 Security Triggers
Security major incident triggers are as follows:
« Actual or suspected attacks on the Fujitsu Services Buildings and its resources, POA Network or
Information Systems
e Theft of IT equipment / property, and in particular PIN Pads
«= Theft of software
©Copyright Fujitsu Services Ltd 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0001
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
PageNo: 12 of 41
FUJ00120061
FUJ00120061
oO POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
e Any PIN Pad Issues that are in breach of FIPS 140-1 level 3 and ISO 9564 1st Edition 1991
section 6.3.1 as specified in the LINK information security standard issued January 2001 (subject
to such dispensations from that standard as LINK may grant from time to time). The main criteria
being as follows:
The purpose of this Standard is to protect the LINK Network, its Members and their
cardholders and ATM owners from attacks designed to compromise sensitive data or
defraud financial institutions and their cardholders. This protection takes into account not
just the direct financial losses that may be incurred but also the potential reputational
damage to the LINK ATM Scheme and its Members and its impact on customer
confidence in LINK and ATMs in general. It is intended to protect the Link Brand:
+ The interests of the Members
+ The interests of the Members‘ customers
+ The reputation and integrity of the UK cash handling infrastructure
In the event of a Security Major Incident (which may also include PCI Incidents), the POA Security
Manager MUST be alerted who will then follow the Security Incident Management processes, as detailed
in both:
SVM/SDM/PRO/0018 Appendix A
SVM/SDM/PLA/0031 HNG-X Security Business Continuity Plan (defines the actions to be taken if
security violations are identified).
In the event of a Major Incident Security trigger for Fujitsu Services Buildings and its resources, the POA
Security Manager MUST inform the Group Property Security Team who will be alerted either by
telephone on a 24/7 basis or the next working day via our Incident Reporting process and the actual or
potential impact of the incident dictates which route is followed.
The Group Property Security Team will then take responsibility for interfacing into the corporate process
by entering reports on to the corporate system.
SCopyright Fujitsu Services Lid 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref. SVM/SDM/PRO/0001
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
Page No: 13 of 41
FUJ00120061
FUJ00120061
oO POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
4 Security Major Incidents
In the event of a security major incident, the incident procedure as detailed in the POA Customer
Services Incident Management Procedure (SVM/SDM/PRO/0018 Appendix A) must also be followed.
SVM/SDM/PLA/0031 HNG-X Security Business Continuity Plan defines the actions to be taken if security
violations are identified.
©Copyright Fujitsu Services Ltd 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0001
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
PageNo: 14 of 41
FUJ00120061
FUJ00120061
oO POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
5 Calling the Major Incident
During business hours the Major Incident Manager declares and manages the Major Incident (with
handovers to the POA OOH Duty Manager where applicable.)
Where the impact of the incident is not immediately obvious, and it is not clear if a Major Incident should
be called, escalation and discussion with the POA Operations Management Team should occur, and a
collective decision made. If a Major Incident is not called, the incident should be monitored until closure,
to ensure that the impact does not increase to that of a Major Incident.
In the event that multiple services are impacted, multiple Major Incident Managers may be appointed by
any Tower Lead and will remain in their roles until incident closure.
Out of hours the POA OOH Duty Manager is responsible for declaring a Major Incident.
Section 9 of this document specifies the roles and responsibilities during a major incident. The Major
Incident Manager, see section 9.2, is referred to the Manage Major Incident Procedure and must
endeavour throughout the life of a major incident to adhere to the principles of that procedure.
SCopyright Fujitsu Services Lid 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref. SVM/SDM/PRO/0001
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
PageNo: 15 of 41
FUJ00120061
FUJ00120061
oO POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
6.Process Flow
MAJOR INCIDENT — Process, Information & Communication Overview
: Technical Recovery
High-Level Major Incident Process at ‘Manager ep ey
2
:
I snow age mats na tc wn SOU ot potas Bi Bas ad
‘eau epbopiom dapat arate Teh a
Sass]
srotae ine Teneeae
G3 Menage Technical
“tpn vere
sete soncne
2
=
2 *Sechman sc arco
é ka
I I mide
oe ‘wrengaion sca
{81 Compaen inmate aie
4 “chueanear eseem a conned
a
N
2: pep tan )
“awake nesngnch
Sra (eeteeer route rectal
ona — TT sage,
sei ce
©Copyright Fujitsu Services Ltd 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref. SVMISDM/PRO/000%
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
Page No: 16 of 41
FUJ00120061
FUJ00120061
oO POA Operations Major Incident Procedure ~
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
MAJOR INCIDENT - Process, Information & Communication Overview
MAC & ee no Major Incident Technical Recovery I Technical Teams
suc I, High-Level Mejor incident Process =! “i nager Manager Inc Third Parties
x So cas Yon be
tanner ae
es stemtendsinenet Tah Cn a
Re, nzeemenerermay “Boecclogecoen I II 893 Provide Testrica
‘ “Bienes parton mpeg
Creranon tat se
‘ncaa reson none
‘Carpe clon aoa acre aio
: ee) are completes I
—— Sade
corees cpa appopae
ets
Pert)
[pants
imps onatsI
ig] II moet
2 I
gi ON
:
-
sie
8
A Ney
a ‘Send SMS 1 update
‘opr
— barre eed
iar wl op
= -Igehusede Aes)
Scher
ad
Se fn pees
‘ Waje eR
i severe
i us
=
613Fonmal I
sede Cone
©Copyright Fujitsu Services Ltd 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref. SVMISDM/PRO/000%
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
PageNo: 17 of 41
FUJITSU
POA Operations Major Incident Procedure
FUJITSU RESTRICTED (COMMERCIAL IN CONFIDENCE)
FUJ00120061
FUJ00120061
5.1 Process Description
Bexrie Pesce 7 ‘Key timescales
61 Major Incident I Incident identified, the defini nt is “Any event which is not part of the standard operation of a
Identified? service and which causes, or may cause, an interruption to, or a reduction in, the quality of that service.”
(SVM/SDM/PRO/0018). An incident may be reported from within POL domain, a supplier domain or other route
6.2 BC Incident? The Major Incident Manager will consult with the Business Continuity Plans I T+3 POA Duty Escalation as a
(see section 0 of this document) to identify if the potential MBCI or MBCI Manager (A) MBCI or Potential
triggers have been met, and inform the POA Business Continuity Manager if MBCI is
appropriate. undertaken if
required. (O)
6.3 Major Incident I An initial impact assessment of the incident is undertaken by members of I T+3 Major Incident I Major Incident
Triggers Met? I the POA Service Team taking into account impact on: Alt Manager (A) Manager assigned
imescales (0)
Live Service, Financial Integrity, Business Image. quoted are ‘best
a i endeavours’
Refer to Section 3 of this procedure. and are
If the incident is profiled as a Major Incident, including consideration of I dependent upon
influencing factors, e.g. time, geographical coverage, business impact, I circumstances
security, public perception, duration and relevant business initiatives T+5
coinciding at POL then go to 6.3.1
If the incident does not meet the Major Incident criteria go to 6.4.
6.3.1 Initial The Atos Service Desk will be informed by the MAC or Major Incident Major Incident I Potential Major
Communicatio Manager of the incident, and this will also be escalated to POA Service Manager (A) Incident advised.
i Management / POA Service Operations team managers, if this has not (O)
already been done. (Note, in most cases of an issue impacting branches or
POL clients, e.g., unable to contact Data-centre, it is more likely that the
©Copyright Fujitsu Services Ltd 2014. FUJITSU RESTRICTED (COMMERCIAL IN _ Ref. ‘SVM/SDM/PRO/000%
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
Page No: 18 of 41
FUJITSU
POA Operations Major Incident Procedure
FUJITSU RESTRICTED (COMMERCIAL IN CONFIDENCE)
FUJ00120061
FUJ00120061
Atos Service Desk will initially inform the MAC team of the incident.) In the
event of either a potential Major Incident or a Major Incident in its own right,
the POA Major Incident Manager will escalate to the Atos Duty Manager or
Atos Live Service Manager and advise accordingly.
6.3.2 With agreement from the POA SDM for the affected service, or the Duty POA SDM or SMS sent when
Manager out of hours, an SMS will be sent to POA Management and Atos POA Duty agreed
Duty Manager alerting to the potential existence of a Major Incident. Manager(A) (0)
6.3.3 POA Service Operations Manager or POA Duty Manager to send out an POA Ops SMS sent calling
SMS calling a Tech Bridge with a brief synopsis of the MI and Tech Bridge Mgr/POA Duty I TB. (O)
phone details. Manager (A)
Go to 6.5.
6.4 BAU Incident If a Major Incident is not declared then the BAU Incident procedure is followed — the Atos I POA Duty Atos SD advised.
Procedure Service Desk will be informed that there is no Major Incident and an SMS sent to the POA I Manager (A) SMS sent to POA
Management Team. The POA SDM for the service should ensure that the Incident is re- Management(0)
impacted during its lifecycle to ensure that the impact has not increased. If, subsequently the 9
incident is declared a Major Incident, go to 6.5.
6.5 Major Incident T+5
Investigation &
Diagnosis
6.5.1 Relevant internal SDUs / Third Parties contacted to initiate investigation and Major Incident I Initial contact with
diagnosis.
Attendees at the Tech Bridge may include POA Service Management,
SDus, Third Parties, POA Operations Security
Technical teams mobilised, diagnostics requested, further clarification on
the MI and symptoms, etc.
Manager (A)
SDUs & Third
Parties. (O)
©Copyright Fujitsu Services Ltd 2014
FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0001
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
Page No: 19 of 41
FUJ00120061
FUJ00120061
oO POA Operations Major Incident Procedure ~
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN CONFIDENCE)
<a ne =< = Lae Kev un. S a
. I a. _ . _. I -
_ . _
6.6 Tech Bridge T+10
6.6.1 Tech Bridge Before commencing the first and subsequent Tech Bridge calls, the Major Major Incident I MI Laminates
Incident Manager is to pickup the Major Incident Laminates from outside the Manager (A) available for TB.
POA ‘Parcel Room’ or from the desk of the Lead SDM for Problem and (Input)
Major Incidents.
6.6.2 Tech Bridge Once confirmed as a Major Incident the Major Incident Manager must T+15 Major Incident I The information
ensure that the information required for the Major Incident Report is Manager (A) required to
captured. See section 10.1 for details of the template. progress the MI
The Tech Bridge agenda which covers: Roll call, Summary / Overview of investigation,
4 aaa ‘ : provide updates
Incident, Current Impact, Investigation / Recovery Action, Remedial and maintain
Actions, Actions to carry forward to Major Incident Review records for MI
During the Technical Bridge the MIM and TRM must consider if any of the report. (O)
following are required and invoke applicable POA local procedures
The need for a Problem Record
Potential work around activities
Anormal or emergency change.
Sufficient details to populate both the Major Incident Progress Template and
Report Template
Consider if a Problem Record is required or if the major incident could
potentially be resolved via a work around or planned change
6.6.3 & I Tech Bridge The Tech Bridge is chaired by the Major Incident Manager with assistance Technical
6.6.4 from the Technical Recovery Manager (TRM). Recovery
z . . . 7 Manager/All
The TRM is to ensure that the Technical Bridge aims are met as follows: (R)
eTo discuss and agree the recovery, investigation and resolution of the
©Copyright Fujitsu Services Ltd 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0001
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
Page No: 20 of 41
FUJITSU
POA Operations Major Incident Procedure
FUJITSU RESTRICTED (COMMERCIAL IN CONFIDENCE)
FUJ00120061
FUJ00120061
- 4 - ~ — oo
I Fiption - . oe .
I :
I I ce _ ponsible
_ I _ _
Major Incident Major Incident
eTo provide a forum for up-to-date progress reports Manager (A)
eTo aid communication and support the MIM to produce a short term
technical recovery plan and if appropriate longer term corrective
actions. These will be included in the Major Incident report.
6.6.5 Tech Bridge Where a Major Incident could be as a result of a Third Party, or require their Third Parties Technical Support
assistance in rectifying the issue, there input will be required in the Tech (As applicable) I (I)
Bridge
6.6.6 Tech Bridge If the outcome of the Tech Bridge is that the incident is determined Tech Bridge + Major Incident I Provide an SMS
Business As Usual (low) then an SMS communication will be sent stating 15 Manager (A) update. (O)
that the incident is not a Major Incident.
From this point forward SMS communication, including both timing and
delivery requests, becomes the responsibility of the Major Incident
Manager.
NB 30 minute updates should be the norm unless otherwise requested by
Atos Duty Manager or Service Management. Technical Distribute planned
The Major Incident Manager will also distribute recovery actions (provided Recovery recovery actions.
by the TRM), during the conference call. Manager (a)
At the time agreed at the first Tech Bridge, subsequent Tech Bridges are
held as required. The same agenda is followed, and progress on actions / Major Incident
recovery is provided. Manager! Decision on need
If no clear recovery path is identified, the decision is then taken on whether Technical pra Service
to escalate for Service Bridge direction. (Invoking a Service Bridge) Recovery ridge. (0)
Manager (A)
©Copyright Fujitsu Services Ltd 2014 FUJITSURESTRICTED (COMMERCIAL IN _ Ref. SVM/SDM/PRO/000%
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
Page No: 21 of 41
FUJITSU
POA Operations Major Incident Procedure
FUJITSU RESTRICTED (COMMERCIAL IN CONFIDENCE)
FUJ00120061
FUJ00120061
a
. .
Lo . _
6.6.7 Tech Bridge If during the Tech Bridge a clear recovery path is identified, this should be
discussed and agreed alternatively further diagnostics and evidence will be . . .
Technical Define diagnostic
required. i, "
Recovery evidence required.
Schedule a follow-up Technical Bridge to co-inside with either the Manager (A) (O)
completion of the recovery activities, if these are expected to be completed
within one hour, or at appropriate ‘touch-points’ agreed with the Technical
Recovery Manager, for recoveries that cover an extended duration. Major Incident I Next Technical
7 Manager (A) Bridge time
For recovery go to 6.7 and for further investigation go to 6.8. agreed, (0)
6.7 Recovery T+x
6.7.1 Recovery The Technical Recovery Manager will liaise with the SDUs and /or third Technical Co-ordinating and
parties during the recovery. Recovery Managing the
Where appropriate technical conference calls may be arranged for technical Manager (A) on Process.
discussions between SDUs and if applicable Third Parties.
The TRM is to advise the MIM if it is considered that the Recovery has been Advising the MIM
F . to call the MI
successfully completed. The MIM is to call the MI Closure Tech Bridge. Go Cl Tech
to 6.10. losure Tec
. Bridge (O).
6.7.2 Recovery After the MI has been in-progress for one hour the MIM is to consult with I >T+ 60 Major Incident I Decision on
POA Service Management to ascertain if a Service Bridge is required. See Manager (A) holding Service
6.11 for Service Bridge details. Bridge (O)
6.8 Investigation T+x
6.8.1 Investigation The Technical Recovery Manager will liaise with the SDUs and /or third Technical Diagnostic
parties during the MI investigation. Recovery information, event
Where appropriate technical conference calls may be arranged for technical Manager(A) logs, st resus
SDUs and as applicable (I)
©Copyright Fujitsu Services Ltd 2014. FUJITSURESTRICTED (COMMERCIAL IN Ref. ‘SVM/SDM/PRO/000%
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
Page No: 22 of 41
FUJ00120061
FUJ00120061
oO POA Operations Major Incident Procedure ~
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN CONFIDENCE)
- 4 - ~ — oo
I : Fiption oo . oe .
I :
I I ce . ponsible
_ I _ _ _
discussions between SDUs and if applicable Third Parties. Third Parties
The TRM is responsible for ensuring that the SDUs and Third Parties obtain )
the agreed evidence to enable subsequent Technical Bridges.
6.8.2 Investigation After the MI has been in-progress for one hour the MIM is to consult with I >T+ 60 Major Incident I Decision on
POA Service Management to ascertain if a Service Bridge is required. See Manager (A) holding Service
6.11 for Service Bridge details. Bridge (O)
6.9 Tech Bridge This procedure will be followed as per instructions, irrespective of how Major Incident I Individual Major
1+ (in the many MI's are running. Manager/ Incident Reports
evsitble MIs) After the time agreed in section 6.6.7 the next Technical Bridge is to start. penne Inaidonts (0) Major
P All SDUs investigating the MI are to take the evidence they have obtained Mana: er tA) °
following their investigations. 9
The MIM is to go to step 6.6 and ensure that they have copies of the Major
Incident Laminates to record the further details.
6.10 MI Closure T+X
Tech Bridge
6.10.1 MI Closure Once the incident is deemed to be resolved, a final Post Incident Review Major Incident I SMS sent
Tech Bridge (PIR) Technical Bridge is to be arranged to review the Major Incident. Manager (A) confirming that the
Major Incident has
been resolved and
the action taken to
resolve it. (O)
6.10.2 MI Closure The MIM is responsible for producing a Draft Major Incident Report and Major Incident I Produce the
Tech Bridge distributing this within one working day of resolution of the Major Incident. Manager (A) minutes of the
Therefore the MIM must ensure the results of this closure technical bridge Closure Technical
are documented. Bridge (O)
©Copyright Fujitsu Services Ltd 2014 FUJITSURESTRICTED (COMMERCIAL IN _ Ref. SVM/SDM/PRO/000%
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
Page No: 23 of 41
FUJ00120061
FUJ00120061
oO POA Operations Major Incident Procedure ~
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN CONFIDENCE)
I 7 7 a
(De I oo _ So : .
_ LL co co co oe
6.10.3 MI Closure The SDU and Third Parties are to provide updates on the actions taken to Major Incident I Actions Completed,
Tech Bridge restore service and confirm that all actions have been completed and that Manager (A) Service Restored
the affected end service has been restored. SDUs & TPs (I)
(R)
6.10.4 MI Closure The MIM, in conjunction with the TRM, is to confirm that service has been Major Incident I MI Resolved
Tech Bridge restored and the MI resolved. For resolved Mls go to 6.10.5 Manager (A) Decision (O)
If there is any doubt about the status of the MI it shall still be considered
Open and a further Tech Bridge is required. Go to 6.6
6.10.5 MI Closure The MIM is to send an SMS communication confirming resolution of the Major Incident I SMS sent providing
Tech Bridge incident. Manager (A) agreed resolution
The MIM is to produce the draft report which is to be sent to Atos within one detalls. (0)
working day and a formal version 1.0 of the report within seven days. Draft MI Report
Within one working
day (0)
6.11 Service Bridge I The nature of the incident determines which POA Service Team members Timescale Major Incident I Relevant decisions
and Atos Managers are involved in the Service Bridge but it would include dependant upon I Manager (A) and information
all or some of the following: impact and Atos Service from the Service
eAtos (personnel as instructed by Atos Duty Manager or Live Systems. nature of Manager Bridge(s) Is fo be
Service Mgr) Major Incident
POA Tower Lead (Chair Person) Report. (O)
ePOA Other Tower Leads
POA Lead SDM, Problem and Major Incident
POA Business Continuity Manager
©Copyright Fujitsu Services Ltd 2014 FUJITSURESTRICTED (COMMERCIAL IN _ Ref. SVM/SDM/PRO/000%
CONFIDENCE) Version: i
Date: 15-Aug-2014
Page No: 24 of 41
FUJITSU
POA Operations Major Incident Procedure
FUJITSU RESTRICTED (COMMERCIAL IN CONFIDENCE)
FUJ00120061
FUJ00120061
=
3
POA Security Manager
POA SDM owning the affected service
ePOA Technical Recovery Manager
eThird Party Executives (if appropriate)
eAppointed working group representatives as appropriate
MAC team Representative
The purpose of the Service Bridge is to:
eProvide appropriate direction on incident resolution
elmprove communications across Third Party business boundaries and
enable senior management in the respective organisations to address
any factors impeding a more timely resolution.
eProvide added impetus to restoration of service as quickly as possible
Define communication intervals to key stakeholders
eProvide focused incident management in line with the impact and severity
of the incident
6.12 Post Incident I Hold a Post Incident Review of the Major Incident. Note there is no POA Lead Finalise the Major
Review & I predefined time in which the PIR is held as it is dependant upon follow-up SDM, Service I Incident Report
formal Incident I actions including Problem Records being addressed. Refer to section 8.0 Operations (A) I (using the output of
Closure for further details and give consideration to the following: the PIR). (O)
eLessons learnt
elncident definition
©Copyright Fujitsu Services Ltd 2014
FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0001
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
Page No: 25 of 41
FUJITSU
POA Operations Major Incident Procedure
FUJITSU RESTRICTED (COMMERCIAL IN CONFIDENCE)
FUJ00120061
FUJ00120061
BHO i 7 kev ein
What went well
Timeline
eChanges required to the infrastructure
eA review of the Major Incident communications
Root Cause Analysis (if known at this point)
eBusiness impact
Action plan, including any changes requiring MSCs
Service Improvement Plan update
eReview any service risk(s) and update the Risk Register as appropriate
6.13 Formal
Closure
All remedial actions completed both short and long term. Including root
cause analysis, and also reviewing/closure with Atos all associated Problem
Records, including Atos signing off the Major Incident.
POA Lead
SDM, Service
Operations (A)
All PIR actions
completed. (O)
Note: Within ‘Key Timescales’ the reference made to T, = Time of incident occurring. hence T+3 = time incident occurred plus 3 minutes.
©Copyright Fujitsu Services Ltd 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0001
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
Page No: 26 of 41
FUJ00120061
FUJ00120061
POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN CONFIDENCE)
6 Communication
6.1 Technical Bridge
This is a Fujitsu technical conference for Technical experts and SDU's to discuss and analyse the
incident and to formulate an action plan to restore the service to POL without delay. It should enable the
Technical Recovery Manager to baseline the anticipated response, covering resolution, time and
resources required. This will also include the appropriate owning SDU of the service affected by the
Major Incident.
The Technical Bridge will be set up as required by the Major Incident Manager.
Invitations to the Technical Bridge will be via SMS, email or voice. The SMS will be sent to the distribution
list titled ‘SMS Technical Bridge’. The SMS text will be sent to all technical experts on the POA and will
include outline details of the Major Incident. Also dial in details and the start time will be provided as part
of the meeting invitation.
The Technical Bridge will be started at T + 15, and reconvened at regular intervals during the Major
Incident; the exact scheduling will be discussed and agreed at each preceding Major Incident Call.
Each Technical Bridge follows a set agenda which will be distributed with the meeting invitation where
possible. The conference call is chaired by the Major Incident Manager with the recovery managed by
the Technical Recovery Manager.
A request for a Technical Recovery Manager (TRM) will be made to the appropriate Tower Lead, who will
appoint one of his team to be the TRM.
Following each Technical Bridge, it is the responsibility of the TRM to publish any actions as follows
-Recovery / restoration actions (which should normally include associated MSC numbers),
-Service Improvement Plan recommendations
-Risk Register recommendations
-Recommendations for any improvements to KELS / Alerting / Configuration changes
The above will be documented in the Major Incident Report and stored on:
http://sites.cafevik fs. fujitsu.com/sites/00672/Service_Management/default.aspx under Service Support >
Major Incident Reports.
©Copyright Fujitsu Services Ltd 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0007
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
PageNo: 27 of 41
FUJ00120061
FUJ00120061
POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN CONFIDENCE)
6.2 Service Bridge
This is a service focussed call for Service Management (including the Technical Recovery Manager if
appropriate) and POL to discuss the service impact of the Major Incident and to receive updates on the
progress towards resolution.
The purpose of the Service Bridge is to provide a focussed area from which strategic decisions can be
made regarding a Major Incident.
Attendance is made up of the following or their designated representative:
eAtos (Personnel as instructed by Atos Duty Manager or Live Systems Service Manager)
POA Tower Lead (Chair Person)
ePOA other Tower Leads
POA Lead SDM, Problem and Major Incident
ePOA Business Continuity Manager
ePOA Security Manager
POA SDM owning the affected service
POA Technical Recovery Manager
eThird Party Executives (if appropriate)
eAppointed working group representatives as appropriate
eMAC team Representative
Service Bridge responsibilities include:
e Agreement of a containment plan
« Documentation of all agreed actions and timescales with owners.
e Consistent management of the Major Incident across all the locations involved
e Management of potential Major Business Continuity Incidents (MBCI's) within Atos and the POA
e Co-ordinate meeting times and locations
In the event of a Major Incident requiring a Service Bridge, it is envisaged that this will be in place at T+60
(or earlier if required by Atos). Participants required in the Service Bridge will be contacted via SMS as
appropriate.
A POA Tower Lead will send out a text via the MAC team in order to organise a Service Bridge.
Invitations to the Service Bridge will be via SMS, email or voice. The SMS will be sent to the distribution
list titled ‘SMS HNGX External’
The SMS text should state such details as;
oAn outline of the ongoing incident,
oDial in details
Start time.
e.g. ‘Your attendance is required at a Service Bridge to discuss the current Major Incident in relation to
Online Services. Please cal _Participant code: ##HHHH at 11.00 hrs.’
©Copyright Fujitsu Services Ltd 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0007
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
PageNo: 28 of 41
FUJ00120061
FUJ00120061
POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN CONFIDENCE)
The chairperson’s code is held by the POA Tower Leads and the Problem and Major Incident Managers.
The chairperson, normally the POA Tower Lead will initiate the call.
The TRM will attend meetings as required and provide appropriate root cause analysis and corrective
action detail.
6.3. Communication Process Flow
On suspicion or confirmation of a Major Incident, the MIM will escalate to the Lead SDM for the
area, Problem and Major Incident Management SDM, and to the POA Tower Leads.
The MIM will inform the Atos Service Desk, via the MAC team, of the start of the service incident
alerting of potential issues — including date, time, nature of problem, severity and impact if
known and then directly inform the Atos Live Service Manager
All updates to the Atos Service Desk are via the MAC team, within agreed timescales controlled
by the MICM
The MICM will issue an SMS text to the POA via the MAC team, alerting of potential issues —
including date, time, nature of problems, severity, impact and name
APOA Tower lead will inform the following within 10 minutes of start of the service incident
o POA Delivery Executive
o POL Senior Service Delivery Managers
And will coordinate and ensure consistency of response to Atos and POA Senior Management
via The Service Bridge
Periodic (interval to be determined depending on the nature of the issue but not more than 30
minutes for Major Incidents) SMS updates to be sent to the original SMS Dist list
On final service restoration, an SMS text message must be sent to the original SMS Dist list
The POA Tower Lead, will confirm understanding of Major Incident closure with Atos
management and POA senior management, and agree next steps
©Copyright Fujitsu Services Ltd 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0007
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
PageNo: 29 of 41
FUJ00120061
FUJ00120061
POA Operations Major Incident Procedure
SU FUJITSU RESTRICTED (COMMERCIAL IN CONFIDENCE)
Fe)
FUIT
4 Major Incident Communication Flow Diagram
6
mm
as soly
ye
ayepdn
WOIW
{( peaxctreuw
ij
i !
Jadeuey I
' Aamnuguoypue — I] I+
I apiniagaarypuegs — I
I soy ovaepdn yeu PI! saSeueyy
I GSsouy aepdn anion “¢ I pea]
, ! IAS
aONlas arp a ml
! I sow Insumnd}
SpLsaysewaepdy "T I
I Repdraig ( Oi
so nnnnnn annem ') aspug aatMas
juawadeueysowy I I juawadeuepjnsyiiny
JAS ANS
IWS INS
pea] samo]
juawageuey)
juapiou Joey
spud
JEUYIaL
SVM/SDM/PRO/0007
9.0
Ref:
FUJITSU RESTRICTED (COMMERCIAL IN
‘©Copyright Fujitsu Services Ltd 2014
15-Aug-2014
30 of 41
Version
Date:
Page No:
CONFIDENCE)
FUJ00120061
FUJ00120061
POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN CONFIDENCE)
What has happened?
Where in the system has a fault occurred?
\/s this in the Fujitsu domain or third party (ie. TTB)?
6.6 Escalation Communication Protocol
The primary principle:
©Copyright Fujitsu Services Ltd 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref SVM/SDM/PROTO001
Up” CONFIDENCE) Version: 9.0
and Date: 15-Aug-2014
“Across Page No: 34 of 41
FUJ00120061
FUJ00120061
POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN CONFIDENCE)
Example:
The Major Incident Manager would escalate up to POA Lead SDM, Problem and Major Incident
Management, and across to the Atos Service Desk.
Major Business Continuity Incidents (MBCl)
For HNG-X the MBCI triggers are listed in:
e HNG-X Support Services Business Continuity Plan (SVM/SDM/PLA/0001)
e HNG-X Services Business Continuity Plan (SVM/SDM/PLA/0002)
* HNG-X Security Business Continuity Plan (SVM/SDM/PLA/0031)
e HNG-X Engineering Service Business Continuity Plan (SVM/SDM/PLA/0030).
These documents should be referred to as appropriate in the event of Major Incident to determine if
Business Continuity needs to be invoked.
6.7 Core Major Incident Management Team
The POA MICM has the task to communicate to Fujitsu Core Major Incident Management team within the
Fujitsu Services Resolution Management team when an incident meets the criteria of a Major Incident.
Monday-Friday 08:00 - 18:00 (GMT) -:
Out of Hours -
6.8 Corporate Alert
Escalation to Corporate Alerts (in line with the Manage Complaints and Alerts Corporate Business
Improvement) is to be approved by POA Business Unit.
©Copyright Fujitsu Services Ltd 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0007
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
PageNo: 32 0f 41
FUJ00120061
FUJ00120061
POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN CONFIDENCE)
7 Formal Incident Closure & Post Incident Review
The Post Incident Review is chaired by the Major Incident Manager and follows a set agenda which is
distributed with the Post Incident Review meeting invitation, along with the draft copy of the Major
Incident Report (if available).
The purpose of a Post Incident Review is:
« Tounderstand the incident that prevented a Service or Services from being delivered.
« To confirm the impact to the business during and after the Incident and agree the number of
branches impacted and duration of Major Incident.
e To confirm the end-to-end recovery process and timeline, and identify that all documented
processes were followed.
e To analyse the management of the incident and the effectiveness of the governance process.
e To identify corrective actions, including agreed Third Party actions, to:
o prevent recurrence of the incident
o minimise future business impact
o improve the procedure for the management of incidents
Output: To confirm details provided in the draft MIR provided to Atos, update with corrective actions and
redistribute. To also include any of the following as appropriate
- any activities for a Service Improvement Plan
- any Changes and associated MSC numbers
- any follow up that requires to be progressed via Problem Management
- any improvements to KELS, alerting and /or event management
The agreed impact of the Major Incident must be provided for inclusion in the Counter Availability SLT
Figures.
If this review highlights areas where improvements can be made, an agreed Service Improvement Plan
will be produced with appropriate actions, owners and timescales. It will also identify any ongoing risks to
the service, together with any changes. Service Management will track all actions to resolution. Third
party actions will be reviewed at Service Review meetings.
Itis important that the number of branches impacted and the duration of the Major Incident is agreed at
the Major Incident Review. This information is required to calculate the impact on Branch and Counter
Availability and any associated Liquidated Damages (LD) liabilities
©Copyright Fujitsu Services Ltd 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0007
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
PageNo: 33 of 41
FUJ00120061
FUJ00120061
POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN CONFIDENCE)
7.1 Calculating potential LD liability for Major Incidents
Major Incidents which qualify as Failure Events are detailed in the Branch Network Service Description
(SVM/SDM/SD0011). A Failure Event is defined in this document as an event or series of connected
events which causes one or more Counter Positions to be deemed to be Unavailable due to a Network
Wide Failure or a Local Failure. Ongoing failures will be deemed to be part of such a Failure Event until
the Failure Event is closed in accordance with the Incident closure and Major Incident Review process as
detailed in section 7.0.
For a Failure Event the Incident Closure & Major Incident Review Process will require Atos and Fujitsu to
agree the number of branches and counter positions affected and the duration of the outage (rounded to
the nearest 30 minutes as detailed in the Network Wide Rounding Table).
Network Wide Rounding Table
30 minutes or less 30 minutes
More than 30 minutes 1 hour
but less than 1 hour
1 hour or more but less 1 hour
than 1 hour 30 minutes
1 hour 30 minutes or 2 hours
more but less than 2
hours
N hours or more but N hours
less than N hours 30
minutes
N hours 30 minutes or (N+1) hours
more but less than
(N+1) hours
©Copyright Fujitsu Services Ltd 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0007
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
PageNo: 34 of 41
FUJ00120061
FUJ00120061
POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN CONFIDENCE)
8 Fujitsu Roles and Responsibilities during a Major
Incident
This section defines the roles and responsibilities individuals and teams have as part of the Major
Incident Escalation Procedure. The following roles will be laminated and available for the MIM to assign
during a Major Incident.
8.1 Role of the MAC Team
The role of the Major Account Controllers team in the event of a Major Incident is as follows:
e Receive and log calls from Atos Service Desk, and communicate the progress of investigations to
the Atos Service Desk.
e Escalation of any Call Threshold Breaches to the POA Duty Manager
« Confirming times and details to Major Incident Manager (MIM)
e Send/update service impact details from the Atos Service Desk (e.g., trend analysis, which the
MAC is dependant upon Atos supplying) to the Major Incident Manager. These details will be fed
into the Technical Bridge in real time as requested, whilst details for the overall Major Incident will
be provided to the Major Incident Manager post the incident.
e Be responsible for sending communications as provided by the Major Incident Communications
Manager for the following:-
To call and attend all Technical Bridges
-SMS to SMS Technical Bridge
To call a Major Incident and provide updates for Major Incident progress, to the following
-E Mail Atos Service Desk
- Voice Atos Service Desk
-SMS to SMS Internal — POA Internal
-SMS to SMS TOWER - Senior Management
NB
The above communications will be as per instructed by the Major Incident
Communications Manager
ALL should be identical, in order to avoid any misunderstandings.
This also of course includes notification to Atos Service Desk and POA Management of
the restoration of service.
©Copyright Fujitsu Services Ltd 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0007
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
PageNo: 35 of 41
FUJ00120061
FUJ00120061
POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN CONFIDENCE)
8.2 Role of the Major Incident Manager
Major Incident Manager (MIM). This will by default be either the Day Time Duty Manager or OOH Duty
Manager (hours shown in 10.3). However a separate member of the Service Management team may be
appointed as the MIM depending on the situation. The primary role of the MIM in a Major Incident is to
facilitate the management of the Incident through investigation and diagnosis to resolution, with the aim
of making the process as efficient and effective as possible. Upon determining that a Major Incident has
been called, a request for a Technical Recovery Manager (TRM) will be made to the appropriate POA
Tower lead who will appoint one of his team to be the TRM. The Major Incident Manager acts as the
central point for communication and non-technical information flow, allowing the TRM to focus on the
technical situation and the resolution of the Incident. The Major Incident Manager is also responsible for
creating and maintaining all the associated documentation. For the process to be effective, all updates
and information regarding the incident must be fed to the MIM to update the timelines and report.
The Major Incident Manager:
e Calls and chairs the Technical Bridge
e Has responsibility for creating the Major Incident Report, using the template defined in section
10.1 and ensuring that the applicable information is captured.
e Records the Technical Bridge attendees names so they can be documented in the Major Incident
Report.
e Identifies Business and Service impact though discussions with the users, the Atos Service Desk
and the MAC team — providing this input into the Tech Bridge.
e Distributes the Technical Bridge actions provided by the TRM (if appropriate).
e In conjunction with the TRM considers if escalation into the Corporate Alert process is desirable
and recommends this when required, see section 7.8 above.
« Assists with communication internally within the POA
¢ Track time lines
e Along with the POA Problem Manager, ensures that the TRM provides regular updates on any
longer term corrective actions.
e Following the resolution of the Incident, schedules and chairs the PIR
©Copyright Fujitsu Services Ltd 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0007
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
PageNo: 36 of 41
FUJ00120061
FUJ00120061
POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN CONFIDENCE)
8.3 Role of the Technical Recovery Manager
The primary functions of the Technical Recovery Manager are to co-ordinate and manage the restoration
of service, manage the technical teams, and act as the communication point for the technical teams and
third parties. The function will also include managing all longer term technical corrective actions, e.g.
recommendations for improvements to KELs, eventing and configuration.
The Technical Recovery Manager:
« Manages the technical recovery of the Incident — liaising with SDUs and third parties.
« Provides updates on the recovery, when technicians / representatives of technical teams are
unable to attend the Technical Bridge.
e Is the only person to liaise directly with the technical teams, including technical third parties.
e Provides summarised actions from Technical Bridge to the Major Incident Manager, including:
°
°
°
°
Current status including impact and risk
Advising on potential workarounds.
Planned recovery activities including timelines
Root Cause Analysis’, corrective actions, and their corresponding action owners and
timelines (where known)
The TRM will be responsible for attending any meetings and providing appropriate root cause analysis
and corrective action detail. This will also include managing any longer term technical corrective actions
that are documented in the Major Incident Report and will include where appropriate
- Any activities for a Service Improvement Plan
- Any Changes / MSC numbers
- Any Risks
- Any Configuration changes
- Any improvements to KELS, alerting and /or events
- Any associated Peak or TfS calls
« For Root Cause Analysis refer to section 7.0 and Fujitsu Services Business Management
Systems Process: Conduct Root Cause Analysis.
©Copyright Fujitsu Services Ltd 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0007
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
PageNo: 37 of 41
FUJ00120061
FUJ00120061
POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN CONFIDENCE)
8.4 Role of the Problem Manager
The Problem Manager ensures that corrective actions / investigations are tracked and completed
following the major incident.
Any corrective actions arising from the Major Incident Review will be added to the Major Incident Report
and also a Problem Record if appropriate, and tracked with POL through to completion. The updates will
be distributed to Atos as required, and in the case of a Security Major Incident associated with PCI
failures, the POL Security team will also receive a copy of the report.
8.5 Role of the Communications Manager
The Major Incident Communications Manager (MICM) will attend the Technical Bridge and produce each
update, where possible trying to ensure that updates are provided on time and following the agreed Major
Incident Progress Template. This will reduce any miscommunication and ensure all parties follow
process.
e Above all ensuring only one update is circulated
e Will ensure that updates are provided within the agreed times
« Updates will adhere to the agreed Major Incident Progress Template
« Update the master Tfs call with all updates
e Ensure update is provided to MAC to circulate through to Atos SD
« Supply update to Service Bridge
* Manages all communication internally within the POA
* Communicate to Fujitsu Core Major Incident Management team
e Manages via MAC, the communication with the Atos Service Desk on the progression of the
incident
8.6 Role of the SDUs: (Technical Teams /SMC/MAC & Third
Parties)
The role is to investigate the Incident, monitor the progress and feed into the Technical Bridge. Also in
the event of no pre-determined recovery options, suggest and evaluate potential recovery options to
resolve the Incident.
The technical teams should not be contacted by any party other than the Technical Recovery Manager.
The Technical Teams / SMC/ MAC team & Third Parties should send an attendee to the Tech Bridge and
the associated Major Incident Review meeting. Where attendance on the Tech Bridge is not possible, a
suitable alternative resource should attend. If neither is possible then a full update MUST be provided to
the TRM to ensure that the Bridge can be updated.
©Copyright Fujitsu Services Ltd 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0007
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
PageNo: 38 of 41
FUJ00120061
FUJ00120061
POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN CONFIDENCE)
8.7 Role of the Service Delivery Manager owning the
affected service
e Attends Technical Bridge
e Attends PIR
« — Responsible for any further action proposed by the Problem Manager that falls outside the
Major Incident closure criteria.
e — Responsible for any Service Improvement Plan actions
8.8 Role of the Tower Lead
e Appoint a Technical Recovery Manager
e POA Tower lead will inform within 10 minutes of the start of the service incident the following-
o POA Delivery Executive
o Atos Senior Service Delivery Managers
eWill coordinate and ensure consistency of response to Atos and POA Senior Management via
the Service Bridge
©Copyright Fujitsu Services Ltd 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0007
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
PageNo: 39 of 41
FUJ00120061
FUJ00120061
POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN CONFIDENCE)
9 Appendices
9.1 List of Templates
All templates are stored on the central share.
_k under Service Support >
Major Incident I The Major Incident Report contains all the I Atos
Report information about a Major Incident. This
Template document is distributed to Atos. POA Service’. Management &
Service Operations
POA All Tower Leads
Fujitsu Support Teams
9.2 Daytime Duty Manager Contact Details
e — Steve Bansal =.
e Steve Gardiner -;
9.3 Out of Hours Duty Manager Contact Details
*OOH Duty Manager Pager {~
17.30 - 09.00 each day Monday PM to Friday AM
17.00 - 09.00 throughout Friday PM and all weekend to Monday AM
Outside these times, please contact the POA Duty Manager
Note: Names and phone numbers are correct at the time of document issue and subject to change. In the
event of difficulties refer to the Fujitsu Services Global Address List for the latest details.
©Copyright Fujitsu Services Ltd 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0007
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
PageNo: 40 of 41
FUJ00120061
FUJ00120061
POA Operations Major Incident Procedure
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN CONFIDENCE)
9.4 POA Service Delivery Contact Details
The Post Office Account service delivery contact details can be found on the Post Office Account Share
Point under Operations > BCP in a folder named Post Office Account Service Delivery Contact Details
9.5 Special Situations
9.5.1 Personnel Absence
e — In the absence of a POA Tower Lead, an alternative Lead will be appointed.
e — Role cards have been produced and will be available to expedite the process.
9.5.2 OOH
e The OOH Duty Manager will act as the Major Incident Manager
9.5.3 Duty Manager Change Over
e The Duty Manager at the beginning of the incident will be by default responsible for all MIM
communications responsibilities unless a different arrangement is made between the
outgoing and incoming Duty Managers
©Copyright Fujitsu Services Ltd 2014 FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0007
CONFIDENCE) Version: 9.0
Date: 15-Aug-2014
PageNo: 41 of 41