FUJ00086267
FUJ00086267
Oo End to End Application Support Strategy
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
Document Title: End to End Application Support Strategy
Document Reference: SVM/SDM/PRO/0875
Release: Release Independent
Abstract: Defines the operational responsibilities of the units involved in the
end to end support of the HNGX solution software in relation to
each other.
Document Status: APPROVED
Author & Dept: Steve Parker RGMA SSC
External Distribution: None
Security Risk YES, none identified
Assessment Confirmed
Approval Authorities:
Name Signature
Peter Thompson Fujitsu Services: Head of See Dimensions for record
applications services
See HNG-X Reviewers/Approvers Matrix (PGM/DCM/ION/0001) for guidance on who should approve.
+, os . FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) 7
Limited 2011 Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY _ Date: 28-Jul-2011
STORED Page No: 1 of 30
POL-0122492-/93/
FUJ00086267
FUJ00086267
End to End Application Support Strategy
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
QO Document Control
Table of Contents
DOCUMENT CONTROL.
Table of Contents.
Document Histor
Review Details..
Associated Documents (Internal & External
Abbreviations.
Glossary..
Changes Expected.
Accuracy...
Security Risk Assessment
oe99990999 ©
CONODUAWNE
INTRODUCTION
1 Exclusions.
2 Support chai
3 Logging systems.
.4 Restoring normal service.
5
1
1
1
1
PReeee
Security restrictions...
1 PO counter acces:
2 Obfuscation of log:
.3 Access to and repair of user data.
4 Audit servers.
2 QUALITIES AND PRINCIPLES
2.1 Accurate call logging and updating.
2.1.1 Updates.
Filtration..
Review.
Timely transfer of incidents.
Metrics.....
Incident count:
Filtration percentage.
Incident life in team.
Evidence gathering.
2.6.1 Obfuscation...
2.7 Removal of duplication..
2.8 Trail
2.9 Knowledge base maintenanc
2.10 Support tools...
We
3 48" LINE SUPPORT..........
+, os . FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) 7
Limited 2011 Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY _ Date: 28-Jul-2011
STORED Page No: 2 of 30
POL-0122492-/93/2
FUJ00086267
FUJ00086267
End to End Application Support Strategy
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
3.1 Hardware calls.
3.2 System monitoring.
3.3 1* line obligations to 2" line support.
4 2"° LINE SUPPORT.
4.1 2" line obligations to 1* line support..
4.2 2" line obligations to 3” line suppott....
5 3®° LINE SUPPORT...
5.1 3° line obligations to 2™ line support.
5.2 3" line obligations to 4" line suppor
6 4™ LINE SUPPORT....
6.1 4" line obligations to 3“ line support.
DEFINITION OF INCIDENT PRIORITIES.....
Support priorities.
Development priorities.
NN O™
RB
DEFINITION OF INCIDENT TIMESCALES
go 99 0
Ne
INCIDENT CLOSURE CATEGORIES.
Peak closure categories......
SMW fix released to call logger.
Build fix released to call logger.
No fault in product...
Programme Approved. No fix required
Published Known Errot
Unpublished known erro!
Enhancement request.
Solicited Known Erro!
Administrative response.
Avoidance Action Supplied.
Duplicate call...
Fixed at Future release.
Reconciliation — resolved
Suspected hardware fault
Advice and Guidance given.
Advice after investigation.
Insufficient evidence.
Unspecified insufficient evidence.
: User Error...
9.1.20 Route call to TFS.
9.1.21 Call withdrawn by user.
oo
OOOOGOGOGGO GGG GGG GOGO Oia
2
10 OUTLINE CONTENTS OF A SUPPORT GUIDE
. os . FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) 7
Limited 2011 Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY _ Date: 28-Jul-2011
STORED Page No: 3 of 30
POL-0122492-/593/3
FUJ00086267
FUJ00086267
End to End Application Support Strategy
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
10.1 Overview of the facility.
10.2 Documentation.
10.3 Server definition.
10.4 Diagnostics and loggi
10.5 Errors and messages.
10.6 Code base and API:
10.7 Support route......
11. KNOWLEDGE BASE MAINTENANCE
11.1 TfS documents...
11.2
11 New KEL generation.
KEL authorisation.
KEL rejection.
, ; . FUJITSU RESTRICTED (COMMERCIAL IN _ Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) ,
Limited 2011 Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY Date: 28-Jul-2011
STORED PageNo: 4 of 30
POL-0122492-/5034
FUJ00086267
FUJ00086267
End to End Application Support Strategy
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
0.1 Document History
Version No. Date Summary of Changes and Reason for Issue Associated Change -
CP/PEAK/PPRR
Reference
04 03/09/2010 First Draft NIA
02 26/11/2010 Changes as a result of comments NIA
Title change to reflect focus on HNGX application support not
infrastructure support.
Definitions changed to use ITIL® terminology
Defined generic 1* - 4° line support qualities
Defined 2” line “virtual team” to explain the lack of a dedicated
2" line team in RMGA
Added definition of Peak response category advice after
investigation.
03 18/07/2011 Minor changes as a result of comments NA
Change of document name to reflect content
04 28-Jul-2011 Further change of document name from “Process to NIA
“Strategy”
1.0 28-Jul-2011 Approval version NIA
Review Details
See HNG-X Reviewers/Approvers Matrix (PGM/DCM/ION/0001) for guidance on completing the lists below. You
may include additional reviewers if necessary, but you should generally not exclude any of the mandatory reviewers
shown in the matrix for the document type you are authoring.
Review Comments by :
Review Comments to : parkersp¢
Mandatory Review
Role Name
Head of Application Services Peter Thompson
Role Name
‘SMC Operations Manager Saha Saptarshi
Operations SDM Saheed Salawu (*)
HSD SDM Sandie Bothick
AMO Manager lan T Turner (*)
Data Centre Development Manager ‘Adam Spurgeon
Estate Management Manager Mukesh Mehta
SMG/MSS (Nth) Manager Jerry Acton (*)
Issued for Information — Please restrict this
distribution list to a minimum
Position/Role Name
, ; . FUJITSU RESTRICTED (COMMERCIAL IN _ Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) 7
Limited 2011 Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY Date: 28-Jul-2011
STORED PageNo: 5 of 30
POL-0122492-/503/5
FUJ00086267
FUJ00086267
Oo End to End Application Support Strategy
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
Operations Director James Davidson
Head of Service Operations Tony Atkinson
(*) = Reviewers that returned comments
Associated Documents (Internal & External)
Referenc Version Date Title Source
PGM/DCM/TEM/0001 I 5.0 03 June 2009 RMGA HNG-X Generic Document Dimensions
(DO NOT REMOVE) Template
SVM/SDM/SD/0001 Service Desk Service: Service Dimensions
Description
SVM/SDM/SD/0004 End to End Application Support Dimensions
Strategy
SVM/SDM/SD/0005 Application Support Service (4" I Dimensions
line): Service Description
SVM/SDM/SD/0006 Systems Management Service: Dimensions
Service Description
SVM/SDM/OLA/0017 Operational Level Agreement Dimensions
HNGx 4" line support
POL/HNGICIS/001 Community Information Security Dimensions
Policy for Horizon & Horizon
Online
CS/FSP/006 End to End Support Process, Pvcs
Operational Level Agreement
SVM/SDM/PRO/0018 RMGA Operations Incident Dimensions
Management Procedure
DES/APP/DPR/0008 Obfuscation of Counter / BAL- Dimensions
OSR data for 4LS
Unless a specific version is referred to above, reference should be made to the current approved
versions of the documents.
Abbreviations
At Definition
AD Applications Division
BIF Business impact forum
CET Counter Eventing Team
1® line team monitoring PO counter events
CMT Communications Management Team
1® line team specifically focused on communication incidents
COTS Commercial off the shelf. COTS purchases are alternatives to in-house
developments.
DPA Data Protection Act
; FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) ;
Limited 2011 Version; 1.0
UNCONTROLLED IF PRINTED OR LOCALLY _ Date: 28-Jul-2011
STORED PageNo: 6 of 30
POL-0122492-/503/6
Fe)
FUJITSU
CONF!
FUJ00086267
FUJ00086267
End to End Application Support Strategy
FUJITSU RESTRICTED (COMMERCIAL IN
IDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
Abbreviati Definiti
HSD Horizon Service Desk
IMT Incident Management Team
ITIL® Information Technology Infrastructure Library
ISD Infrastructure Services Division
KEL Known Error Log
LST Live System Test
MO Model Office
MSS Management Systems Support
oTl Open Teleservice Interface. Interface used to transfer incidents between
different logging systems.
PCI Payment Card Industry
PPRR Period Process Review Record
Prescan Takes place prior to allocating incidents to support team members. Ensures
that any incident which can be turned round quickly (e.g. known error,
insufficient evidence) does not wait for the attention of a diagnostician who
may be working on other duties.
Qc Quality Centre. Incident logging system used by test teams within RMGA
Peak Fujitsu services incident and release management system
POSD Post Office Service Desk.
SRR System readiness review
SMC. Systems Management Centre.
1* line team providing the Systems Management Service.
Ssc Software Support Centre.
3° line application support
TS Triole for Service. Incident logging system used by first line support units
Glossary
Event storm Asystem event that is repeating itself so quickly that it causes performance issues
with the server on which it runs or the event recording infrastructure.
Incident An ‘Incident’ is any event which is not part of the standard operation of the service
and which causes, or may cause, an interruption or a reduction of the quality of the
service
Normal service operation Service operation within Service Level Agreement (SLA)
Workaround Method of avoiding an incident or problem, either from a temporary fix or through
access to an alternative service. A method to bypass a recognized problem in a
system. A workaround is typically a temporary fix that implies that a genuine solution
© Copyright Fujitsu Services
Limited 2011
FUJITSU RESTRICTED (COMMERCIAL IN _ Ref: SVM/SDM/PRO/0875
CONFIDENCE) Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY Date: 28-Jul-2011
STORED PageNo: 7 of 30
POL-0122492-/9377
FUJ00086267
FUJ00086267
End to End Application Support Strategy
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
Term Definition
to the problem is needed.
Resolution Resolution is the action taken to repair the root cause of an incident or problem, or to
implement a workaround.
Changes Expected
Definition of 2" line support will be changed when the new HSD team responsible for 2LS is well established.
Accuracy
Fujitsu Services endeavours to ensure that the information contained in this document is correct but, whilst every
effort is made to ensure the accuracy of such information, it accepts no liability for any loss (however caused)
sustained as a result of any error or omission in the same.
Security Risk Assessment
Security risks have been assessed and it is considered that there are no security risks relating specifically to this
document.
+, os . FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) 7
Limited 2011 Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY _ Date: 28-Jul-2011
STORED Page No: 8 of 30
POL-0122492-/038
FUJ00086267
FUJ00086267
End to End Application Support Strategy
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
1 Introduction
This document is a rewrite of CS/FSP/006 which was last updated in 2003. It provides an overview of the
support strategy and the interfaces between support units which provide application support for HNGX. It
also describes qualities, obligations and objectives expected from those units.
This document is effectively a high level design for application support. It does not attempt to define the
low level procedures by which each support unit will deliver the qualities described here. The specific
implementation of this design is documented in the relevant service descriptions for the support groups.
1.1 Exclusions
This document excludes any detail of the management of operational problems (which are owned by
infrastructure services) and all hardware related incidents. It also excludes details of the business impact
forum which is a governance process owned by RMGACS and not a support process.
This document also excludes any specifics of software and reference data distribution support (SMC,
MSS and SMG). This may be included at a later date.
Test teams do not interface formally with the support chain when they are testing a future release. Their
interface is with the release management process and directly with the development unit resolving faults
in the release. Details of these interfaces are excluded from this document. Live System Test are an
exception. When testing the resolution for an outstanding problem with the live release they still fall within
the release management process. In some cases they may notice a new issue within the live release
when they will generate a new QC defect which results in a Peak. For the purposes of this document they
are treated as a 3° line unit under these circumstances.
The remainder of this section describes extant situations and expectations which influence application
support.
1.2 Support chain
There are normally 4 levels of support within a convention support organisation:
1" line 2nd line 3rd line 4th line
The support strategy expects that incidents will be raised by users and then passed through the chain of
support units until a resolution can be supplied to the user. It is important that an incident starts at 1st line
and then follows each stage of the chain as appropriate. This ensures:
1. The incident is quickly defined and logged
An initial response is given
Priority is correctly evaluated
YN
The correct skills are applied such that a resolution is supplied quickly
5. The call is correctly recorded, auditable and relevant metrics can be produced.
As incidents move from left to right across the support chain they become:
¢ More difficult to resolve
. os . FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) 7
Limited 2011 Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY _ Date: 28-Jul-2011
STORED Page No: 9 of 30
POL-0122492-/03/9
FUJ00086267
FUJ00086267
End to End Application Support Strategy
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
e More time consuming to resolve
e The training level and cost of the staff resolving the incident rises
e Tooling and supporting infrastructure costs rise
Support costs and timescales for resolution increase as the incident moves to the right. Hence the effort
spent “moving support to the left”. Ensuring that the incident is resolved as early in the chain as possible
reduces the cost and increases customer satisfaction (assuming a first time fix is achieved).
1.3 Logging systems
For historic reasons RMGA uses two logging systems. 1st line groups use a system called Triole for
Service (TfS) which is primarily an incident management system. All other support, development, test
and release management teams within RMGA use a system called Peak (no it is not an acronym, it's a
name!) for incident, problem and release management. The two systems are closely coupled using an
interface called the Open Tele service Interface (OTI).
An incident is passed over the OTI as it moves between 1" and 2™ line support groups. When the
incident has been “moved” in this way it is no longer being actively progressed in the sending system.
Updates made in either system are reflected in the other, this allows additional information to be passed
between support groups while the call is not active in one of the systems.
The use of two logging systems imposes some restrictions on incident processing:
1) The OTI interface must be monitored by 1* and 2 line units to ensure that it is working correctly.
2) Attached evidence cannot be moved between the two systems. This means that evidence files can
only be passed by reference (within the information logged).
3) TfS has a limit of 4000 characters within a single update which sometimes results in lost information.
The requirement for detailed evidence sets starts at 2“ line. This is where the interface has to exist
between the two logging systems. Using Peak from this level ensures that evidence can be easily
passed between 2” to 4" line support groups and that facilities such as obfuscation and encryption
(features of the Peak system) can be applied to the evidence files.
1.4 Restoring normal service
It is incumbent on this support route to restore normal service operation as quickly as possible and
minimise the adverse effect on business operations. The restoration of service is considered to be
completed once it has been documented and communicated by a support team to the end user who
raised the incident. This does not necessarily imply that a change has been made since an incident as
perceived by the end user may not be impacting service operation. Restoration of service has targets
dependent upon the priority assigned to the incident. These priorities are defined in section 8 of this
document.
Once normal service operation has been re-established it may be necessary to resolve the root causes of
incidents (Problem management in ITIL terms). Although it is expected to resolve the problem as quickly
as possible this strategy is focused on the resolution of the problem rather than the speed of resolution
and as such no targets are defined.
1.5 Security restrictions
Access to certain parts of the system or certain support tools must be restricted in order to conform to
DPA or PCI data security standards. The application of these standards within the RMGA support
. os . FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) 7
Limited 2011 Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY _ Date: 28-Jul-2011
STORED Page No: 10 of 30
POL-012249%g03/10
FUJ00086267
FUJ00086267
End to End Application Support Strategy
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
community is defined by the RMGA CS Security team in various documents which are not reproduced
here. The following describes restrictions that exist and impact the support community.
1.5.1 PO counter access
Direct access to live Post Office counters is currently only available to the SSC. It depends on access to
the SSN servers and the use of a passphrase which allows public key authentication to a fixed user on
the counter. Data visible using this access is subject to DPA restrictions and thus this access cannot be
allowed to offshore units.
1.5.2 Obfuscation of logs
Certain log files must be processed to obscure personal details that exist within them Gee section 2.6.1
for details) before they can be passed to support teams outside the European Union. As new log files are
generated by system enhancements development units need to be aware of the DPA and ensure that
information in any new log files is either benign in DPA terms or that appropriate changes are made to the
obfuscation tool. The CS operational security team can advise on DPA issues.
1.5.3 Access to and repair of user data
Access to RMGA data has been secured using the standard security principle of separation of duties.
Separation of duties ensures that an individual can not complete a critical task by themselves. For
example: someone who submits a request for reimbursement should not also be able to authorize
payment. An applications programmer should not also be the server administrator or the database
administrator - these roles and responsibilities must be separated from one another.
In RGMA the separation of duties principle has been implemented by ensuring:
e Development units cannot have update access to any of the system data.
e Database administration functions are carried out by IS staff
e Data repair is carried out by SSC staff
The DPA requires that access to personal data remains within the European Union and PCI data security
standards mandate physical security restrictions must be applied where update access is allowed to user
data. Currently the only units which fulfil all these requirements for data access are the SSC and ISD
Unix. The responsibility for data correction is vested with the SSC although ISD sometimes act under
SSC authorisation.
1.5.4 Audit servers
The audit servers provide a full audit trail of all information on the HNGX system. In order to ensure that
this audit trail is irrefutable the teams which have the ability to change data (i.e. SSC) must not also have
the ability to change the audit trail. For this reason, Audit server 3” line support rests with the Audit
development team and not the SSC.
2 Qualities and principles
This section describes qualities and principles that apply to all support groups working within RMGA. It is
expected that each support group will define the processes and procedures that implement these
principles.
. os . FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) 7
Limited 2011 Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY _ Date: 28-Jul-2011
STORED Page No: 11 of 30
POL-012249/g03/11
FUJ00086267
FUJ00086267
End to End Application Support Strategy
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
2.1 Accurate call logging and updating
All support groups are expected to receive incidents passed from other support groups and to ensure that
any incidents so received are maintained on the relevant call management system:
e 1% line TIS
e 2" to 4" line Peak
The initial description logged should include a full explanation of the problem, the accurate recording of
any references supplied and the actions that were taken which caused the problem to occur.
2.1.1 Updates
When relevant additional information has been made available that information should immediately be
added to the incident to ensure it reaches the diagnostician currently working on the call.
The 1*' line agent making such updates (in TfS) should ensure they are made as an OTI Comment (non-
OTI updates are not transmitted over the OTI to the investigating unit in Peak).
The 2™ — 4" line diagnostician processing the incident should ensure that relevant updates are applied on
a regular basis appropriate to the priority of the call. These Peak updates should be made using a
numbered response category to ensure they go over the OTI and become available to f' line should the
end user request an update. These “OT! updates” are often repeated verbatim to Post Masters when they
ring in asking for an update so diagnosticians should ensure that any OTI updates they make are
appropriate to the audience who will receive them (i.e. the end user).
2.1.2 Closure
Ensure that every incident reported has a resolution recorded on the logging system and that the
resolution is acceptable to the end user. Where the final response has been entered on Peak it will be
returned to the 1* line call management system (TfS) for communication to the end user and final
closure. The description should include a full explanation of the resolution. It is acceptable for any level of
support to agree closure but where that agreement has been made outside f* line support it should be
recorded in the incident to ensure that 1* line do not make unnecessary calls to the end user.
2.1.3 Withdrawal
Support units have the facility to withdraw an incident, resulting in its automatic closure on the active
incident management system. This should be rarely used and any usage must be communicated in
advance to the team currently processing the incident to ensure that they stop investigation.
It is expected that a request to withdraw an incident will originate from the end user who raised it. When
this is not the case then governance should be in place at each support unit to ensure the facility is not
misused and that the end user is informed.
2.2 Filtration
All support units are expected to resolve as many incidents as possible and only pass on those that are
relevant to the next line of support, hence it being called a filtration process.
The first level of support to discuss the incident with the customer should close any incident for which the
problem and resolution is already known to the support community. When this does not happen the
incident is deemed to be a filtration failure.
. os . FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) 7
Limited 2011 Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY _ Date: 28-Jul-2011
STORED Page No: 12 of 30
POL-012249% 503/12
FUJ00086267
FUJ00086267
End to End Application Support Strategy
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
When an incident is closed the diagnostician entering the closure also applies a category for the closure.
Categories such as, insufficient evidence, published known error and user error indicate that the call
should have been filtered rather than sent on to the next level of support. Seeincident closure categories
for a list of closures categories that are considered to be filtration failures.
2.3 Review
It is essential that each support group carries out regular reviews of incidents returned as filtration failures
and reports on this OLA and the actions being taken to address failures.
2.4 Timely transfer of incidents
All support groups must ensure that any incidents that will require the attention of another level of support
are passed in a timely manner. The exact timings are detailed in section 8 of this document.
The timings vary according to the total time allowed for resolution of the incident in the contract between
Fujitsu Services RMGA and the customer. These timings will therefore be dependent on the priority of the
incident, with (for example) less time allowed for an “A” priority call than will be permitted for a “D” priority.
These timings should not be used without consideration being given to resolution. Since the requirement
is the resolution of a call (not simply its transfer within timescales) then it is acceptable for a support
group to retain the call past its normal time if it is confident that it can provide a resolution within the
maximum time allowed for the incident.
2.5 Metrics
Each support level needs to produce call metrics to describe the service they supply and the quality of
that service. These should be made available to all other support units in a shared location and serve as
a guide to the efficiency of all parts of the support chain. These metrics should form the input to regular
review meetings within each support unit to improve service.
Metrics produced should include:
2.5.1 Incident counts
1. Count of incidents input to the support unit per day (daily input)
2. Count of incidents closed by the support unit per day (daily output)
3. Incidents remaining in the support unit at the end of the day (daily work in progress)
4. Count of incidents input to the support unit in the last 34 days* (monthly input)
5. Count of incidents closed by the support unit in the last 34 days* (monthly output)
*A figure of 34 days is used to prevent the figures being affected by months which start end and / or end
with a weekend. Such months make comparisons unrepresentative.
2.5.2 Filtration percentage
Filtration is an important measure of the efficiency of a support unit. It shows the number of calls
transferred that should not have been passed onto the next level of support. This is expressed as a
percentage of the total calls the support unit closes.
For example:
, ; A FUJITSU RESTRICTED (COMMERCIAL IN _ Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) ,
Limited 2011 Version 1.0
UNCONTROLLED IF PRINTED OR LOCALLY Date: 28-Jul-2011
STORED PageNo: 13 of 30
POL-012249%503/13
FUJ00086267
FUJ00086267
End to End Application Support Strategy
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
100 calls closed by 2” line in period, of which 25 calls returned from 3° line in “black mark” categories:
2” line filtration rate to 3° line = 75%
This shows that the 3” line support group had to deal with an additional 25 calls that should have been
stopped by previous support units.
See incident closure categories for a list of closures categories in use and which constitute filtration
failures (AKA black marks).
Whilst 100% filtration is generally expected and strived for a 5% deficit (i.e. 95% filtration) on a rolling
three month basis is accepted by all parties.
Filtration metrics are required for:
1) Total number of filtration failure calls - broken down by Peak category
2) List of Peak failure reference numbers
2.5.3 Incident life in team
Ameasure of the performance against incident closure / transfer time scales:
e The average time to transfer / close an incident by priority per team. Average time an incident is
within a team split by final priority on transfer.
e The average age of all open incidents in the team by priority.
2.6 Evidence gathering
All support groups gather and evaluate evidence as part of the problem solving process. It is also a
responsibility of all support groups to ensure that relevant evidence is referenced or attached to all
incidents that are passed to another support group.
Where relevant evidence has not been supplied it is highly likely that the support group that the call is
passed to will just return the call asking for (and detailing) further evidence. Such a response counts as a
“black mark” for filtration purposes and will be reflected in the sending support group's filtration figures.
Specifically excluded from this measure are instances where:
e Although the evidence was inadequate, no documentation existed describing the relevant
evidence required.
* Occasions where the evidence required was unobtainable.
What constitutes “relevant” evidence can be determined by:
1. Examination of the support guide for the area of the system being investigated.
2. Asearch of existing knowledge entries.
3. Examination of high and low level design documentation.
It is an inconvenient truth that 1“ line support groups do not use the same incident logging system as the
rest of the support chain. When passing incidents from1st to 2nd line, evidence files can only be supplied
by quoting a reference to the file. When passing evidence via Peak any relevant evidence files are
attached directly to the incident.
2.6.1 Obfuscation
, ; . FUJITSU RESTRICTED (COMMERCIAL IN _ Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) ,
Limited 2011 Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY Date: 28-Jul-2011
STORED PageNo: 14 of 30
POL-012249% 03/14
FUJ00086267
FUJ00086267
End to End Application Support Strategy
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
There are requirements within the data protection act for the processing of personal data. It is essential
that all support groups are aware of this restriction and do not allow the transfer of any evidence files
offshore if they contain personal data. Current offshore units are:
1. 4LS for Counter and BAL-OSR
2. SMC
Areas currently identified as potentially containing personal data (see DES/APP/DPR/0008):
* Counter OSR / BAL message log file
e Counter application log file
e Database exports (e.g. CSV exports —- Message Journal Exports)
e Screen captures of live system data*
e Audit data extracts (content of message journal)*
*Not handled by obfuscation tool
In order to allow the use of such information offshore an obfuscation too! has been developed for use on
the log files that are known to contain sensitive information before passing to any external support team.
The tool has now been integrated into Peak. Information on its use can be found in the FAQ section of
Peak.
If anybody suspects that personal data is present in other log files they should raise an incident with the
CS operational security team so that it can be checked before the incident is sent to an offshore support
group. It may be necessary to process the information onshore or have the obfuscation tool enhanced.
2.7 Removal of duplication
All support groups should ensure that they do not pass to the right duplicate incidents, i.e. incidents which
are repetitions of an incident which has already been passed to the next line of support. They should
either retain the duplicate incidents within their own call logging system or close them as duplicates:
a) 1“line units retain duplicates under a “master call” and to ensure that when the resolved incident is
received from 2” line, the end user is contacted and duplicated calls incidents closed within TfS.
b) 2°¢ — 4" line support units normally immediately close the incidents as duplicates because they add no
value to the support process at these levels. This results in the incidents being returned to 1 line (TfS)
Duplicate incidents are only acceptable where the symptoms reported by the customer did not match the
symptoms recorded in the original incident, and which therefore could not reasonably have been
identified as a duplicate.
Failures will be reflected in filtration figures where the incidents are closed in the “duplicate incident”
category in Peak by subsequent support units.
2.8 Training
Training on new facilities added to the system will be provided by the architects and development units
that design and implement those facilities. A checkpoint for this will be included in the SRR for each new
release. This training takes the form of one or more of:
e Classroom training (workshops)
e One-to-one training
e Training collateral
. os . FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) 7
Limited 2011 Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY _ Date: 28-Jul-2011
STORED Page No: 15 of 30
POL-012249¢ 503/15
FUJ00086267
FUJ00086267
End to End Application Support Strategy
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
« Updated support guides
It is expected that the training provided is then relayed from right to left through the support chain. The
form and content of the training is likely to change as it is modified to ensure it is suitable for the new
audience at a different level of support.
It is the responsibility of the unit receiving the training to ensure that it is completed and fit for purpose.
Inadequate training on new facilities will be reflected in reduced filtration figures for the receiving unit so it
is in their best interest to ensure that the training material is timely and appropriate for its audience.
2.9 Knowledge base maintenance
All support units are expected to search, enter, update and maintain the information in the support
knowledge base. Criteria detailed in section 11.
Because of the time constraints applied to 1* line units there should be no requirement for them to raise a
KEL for every new incident. It is a requirement that 2” line ensure that a KEL is generated for any new
incident encountered. NOTE: because of the lack of an effective 2nd line unit within RMGA this
requirement is being partially satisfied by 1 line support units (STE04 CET and SMC).
2.10 Support tools
Support tools normally result from a requirement placed on development for a new release or are
generated by 3” and 4" line support groups when necessity demands. In either case these tools must
have the following qualities:
1. They must be tested on a test rig (normally LST) before they are deployed in the live
environment.
2. It must be possible to restrict their use on an individual basis. This is to ensure that DPA/ PCI
rules can be enforced dependent upon the person (not just the team) using the tool.
3. They must be self documenting or have suitable documentation written in the form of a work
instruction.
Such support tools will always be preferable to various diagnosticians generating manually crafted scripts
or SQL which need individual testing, are rarely documented and often represent duplication of effort.
Support tools generated outside development units are not expected to be subject to the CP process
since this represents an unacceptable overhead to this type of tool. It is expected that any support unit
generating support tools will put in place a process or work instruction to ensure:
1. No duplication of effort takes place
2. The effort required to write the tool does not exceed any benefit gained from it.
3 1% line support
4* line support units within RMGA comprise HSD, IMT, CET, CMT and SMC. From the point of view of 2°
line support they can be treated as one unit since incidents arriving at 2° line support will always have
passed through one of these units first.
1* line support log incidents by directly interacting with the user or from monitoring systems. They clearly
document incident symptoms based on customer perception or observed alerting information. Trained to
the same level as the user they should resolve all issues where the cause is user training or environment.
1st line resolve incidents by the identification of knowledge base entries and the application of defined
. os . FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) 7
Limited 2011 Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY _ Date: 28-Jul-2011
STORED Page No: 16 of 30
POL-012249¢)g93/16
FUJ00086267
FUJ00086267
End to End Application Support Strategy
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
scripts (flowcharts). 1° line support provide the “touch point” with the end user (e.g. Post Masters or in
some cases internal units such as ISD) and are responsible for ensuring the end user is kept informed of
progress of their incidents and taking any escalations from that end user. It is incumbent upon the
subsequent lines of support to provide the information and escalation routes necessary for 1" line to do
this.
3.1 Hardware calls
1* line support should filter all hardware calls and ensure they are routed to the correct engineering
group for resolution. These hardware calls are subject to strict SLAs which are not present for software /
application support. If a hardware call gets past 1" line it will inevitably fail those strict SLAs.
3.2 System monitoring
1* line support are also responsible for monitoring the live estate and taking corrective actions for all
critical events seen. This role is currently fulfilled by two units:
1. SMC: Data centre event and schedule monitoring
2. CET: Counter events
For each event seen 1" line must check for the event in documentation (knowledge base and support
guides) for the relevant system subsection and to take the documented action (if the required support
tools are available). A new incident should be raised for each critical event that is not already documented
and passed 2" line support teams for action.
1* line support will also ensure that “event storms” are correctly handled. Where an event storm is being
produced by a server, or by a subsystem in the RMGA solution AND where the cause of that event has
already been documented, then first line should, where possible, take appropriate action to prevent the
system becoming “swamped” with repeat events.
3.3 1% line obligations to 2" line support
The section describes obligations inherent in the interface between 1" and 2™ line support units. These
are in addition to the general obligations defined in section 2.
1) Filter all hardware calls and route them to the appropriate unit for hardware support. No hardware calls
should be passed to 2nd line except in circumstances where Fujitsu Services RMGA need to be made
aware of:
1. Recurrent hardware problems which need to be addressed between Fujitsu Services RMGA and
a supplier.
2. Hardware behaviour that can be influenced by application design changes
2) OTI Monitoring: Ensure that any incident which requires investigation by 2 line support is passed onto
the call management system used by Fujitsu Services RMGA (currently Peak) and assigned to the
correct 2™ line support team. This includes monitoring the OTI to ensure that the call reaches Peak
correctly.
3) Addition reports to the metrics defined in section 2.3:
1. Number of incidents passed to 2” line by priority.
2. Time taken between receipt of the incident and transfer to 2” line by priority.
3. Time between receipt of the incident by 2 line and the resolution passed back to 1* line by
priority.
; FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) 7
Limited 2011 Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY — Date: 28-Jul-2011
STORED PageNo: 17 of 30
POL-012249¢g03/17
FUJ00086267
FUJ00086267
End to End Application Support Strategy
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
4) Monitoring: Ensure that no critical events on the live system go unnoticed and ensure that appropriate
action is taken for each event. In this context, a critical event is one which is logged as critical, and
appears in red on the Tivoli screens and with a red marker in NT event logs.
No event calls should be passed to 2nd line without documentation having been consulted (Support
guides and knowledge base).
5) Monitoring: Ensure that for any incident which has been resolved and passed back to the TfS system,
the end user has been contacted and made aware of the closure
4 2" line support
2nd line support staff can be described as expert users of the system. They use the symptoms
documented by ‘st line to understand the error and then gather additional information / logs from
standard sources within the system to provide evidence of the environment and other factors present at
the time of the error.
From their understanding of the system and the use of reference equipment that simulates what is
available to the end user, 2nd line can devise procedural workarounds and apply operational changes to
alleviate incidents.
2nd line produce knowledge entries for all incidents using the symptoms collected by first line and where
possible add a definition of the root cause problem and any solution or workaround found. They also
generate scripts and procedures for 1st line, produce explanatory documentation and MI reports. 2nd line
have a level of access to the system that allows evidence gathering and operational workarounds to be
applied and the use of simple GUI support tools.
At the time of writing there is no dedicated 2” line support unit within RMGA. Some of the 2” line
responsibilities are being fulfilled by 1* line or 3“ line units providing a “virtual” 2" line function
as follows:.
Initial production of knowledge entries HSD, SMC, SSC
Evidence gathering HSD, SSC
Operational changes Ssc
Workarounds via GUI support tools HSD
Procedural workarounds HSD, SSC
OTI monitoring HSD
KEL reference applied to Peak SSC
Priority assessment and change HSD, SSC
4.1 2" line obligations to 1* line support
The section describes obligations inherent in the interface between 2” and 1“ line support units. These
are in addition to the general obligations defined in section 2.
1) OTI monitoring: Where updates are made to the calls which are of relevance to 1" line then the second
line support unit will ensure that these updates reach the first line call logging system (currently TfS).
2) To ensure that any resolutions or workarounds that are passed back to 1" line have been tested. The
exception to this rule is the case where resolutions are being passed specifically to be downloaded to the
RMG Account test rigs for testing.
+, os . FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) 7
Limited 2011 Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY _ Date: 28-Jul-2011
STORED Page No: 18 of 30
POL-012249; 03/18
FUJ00086267
FUJ00086267
End to End Application Support Strategy
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
4.2 2™ line obligations to 3" line support
The section describes obligations inherent in the interface between 2” and 3” line support units. These
are in addition to the general obligations defined in section 2.
1) To ensure that the priority of any incident is assessed and recorded correctly. No calls should be
passed to 3” line support whose priority does not conform to the specification defined in section 8.
2) No incidents should be passed to 3” line without a valid KEL reference recorded in the Peak
references.
5 3' line support
3" line support groups within RMGA include:
SSC - 3” line support for RGMA written application code.
MSS -— 3” line support for software distribution and event management
3rd line support staff apply analytical skills to the symptoms and evidence gathered by 1st and 2nd line
and undertake in-depth investigation into incidents. They have detailed knowledge of the system based
on documentation and source code inspection.
Trained on operating systems, COTS packages that underlie the application and the coding languages
used within the application. They are also expected to self train by examination of support guides, design
documentation written for the components of the end user application. They will also have access to
development and package management tools to allow the production of specialised diagnostic code,
scripts and support tools.
It is incumbent upon the 3% line support unit to produce a work around and on 4th line to produce the final
code solution to any software problem. This does not preclude the production of a workaround by other
units or negate the requirement for 4" line to provide assistance in the generation of a workaround.
The SSC are responsible for the implementation of any workarounds that require data changes to the live
system. They are the only unit with authorisation and sufficient physical security controls to perform this
function.
5.1 3 line obligations to 2"? line support
The section describes obligations inherent in the interface between 3° and 2™ line support units. These
are in addition to the general obligations defined in section 2.
1) To ensure that any workarounds have been tested and have been correctly authorised via MSC. The
exception to this rule is the case where workarounds or resolutions are being passed specifically to be
downloaded to the RMG Account test rigs for testing.
5.2 3 line obligations to 4" line support
1) To ensure that any incident which requires investigation by 4" line support is assigned to the correct
Peak team dependent on the specific product in which the incident has occurred.
2) To ensure that the priority of any incident is assessed and recorded correctly.
3) To ensure that for any incident passed to 4th line support, the exact area of the problem has been
identified, and wherever possible a workaround already produced.
4) To ensure that for any code error a probable solution is indicated prior to passing to 4th line support,
and wherever possible, the possible solution has undergone limited testing.
. os . FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) 7
Limited 2011 Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY _ Date: 28-Jul-2011
STORED Page No: 19 of 30
POL-012249¢ 03/19
FUJ00086267
FUJ00086267
End to End Application Support Strategy
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
6 4 line support
Have intimate knowledge of narrow areas of the system and are ultimately responsible for the production
of permanent fixes to repair the root cause of an incident or problem in the live application. Trained in
development languages and coding techniques there is often overlap between 4th line support and
development roles. 4" line assist 3rd line with workarounds and resolution of incidents, produce test
scripts for testing of code fixes and unit test those fixes.
4" line support within RMGA is supplied by various AD and offshore units. Since 4” line own the interface
with development (and the functions are often vested in the same people) they are also tasked with
ensuring that various development obligations to the support groups are met.
6.1 4" line obligations to 3" line support
The section describes obligations inherent in the interface between 4th and 3° line support units. These
are in addition to the general obligations defined in section 2.
1) To ensure that the incident reported is correctly resolved and the resolution recorded on the Peak
system and the incident and resolution passed back to 3% line. Where appropriate this should also
contain the method of recreation of the problem.
2) To ensure that the incident is resolved within the total time allowed by the contract between the
customer and Fujitsu Services RMG Account. Specific targets for timescales are documented in
section 8_of this document. However, in most cases the provision of the fix is at the discretion of the
Release Management Forum, and the target for the provision of any fix therefore is as specified by that
forum.
3) To ensure that any resolutions or workarounds that are passed to 3° line have been tested.
Where they are being recommended for application to the live system they must have been correctly
authorised via the MSC process.
4) To ensure that when a resolution is produced the baseline reference is added to the relevant KEL entry
that describes the problem.
5) To ensure that 3” line is supplied with training and documentation relating to new releases of
the RMG Account solution in sufficient time to enable 3° line staff to become familiar with the product
prior to its release, and in sufficient time to enable 3” line to adequately train other support staff.
Preferably this knowledge transfer should be a continual process during the course of development but
an adequate timescale is 6 weeks prior to data centre release or model office for counter releases.
6) Ensure that support guides have been written or updated for all new facilities in HNGX. An outline
description of the contents of a support guide is given in section 10 of this document
7) To ensure that 2” and 3” line support groups are supplied with read access to all source code
developed within RMG Account development prior to the release to live of that component.
8) In addition to the metrics defined in section 2.3: to ensure that the following figures are available to
other support units on demand.
1. Total number of calls where resolution has been deferred to a future release
2. Counts of deferred calls by future release.
7 Definition of incident priorities
The definition of an incident priority changes depending upon the stage of the life cycle. Full details of
RGMA incident management process life cycle can be found in SVM/SDM/PRO/0018.
. os . FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) 7
Limited 2011 Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY _ Date: 28-Jul-2011
STORED Page No: 20 of 30
POL-012249%,503/20
FUJ00086267
FUJ00086267
End to End Application Support Strategy
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
7.1 Support priorities
Support priorities are used from the point when the customer initially logs a call until a work around has
been achieved.
Priority A A Post Office unable to trade (where engineering cover available),
Business stopped unable to process any business*
Acentral system failure which will result in a number of Post
Offices being unable to process work.
Causes significant financial loss (as agreed between POL and
RMGA Customer Services)
Results in data corruption or unrecoverable data loss.
Outage of key infrastructure
Priority B A Post Office restricted in its ability to transact business e.g. 50%
Business restricted of counters unable to trade or trading with restricted business
capability.
Has an adverse impact on the delivery of service to a number of
end users.
Causes a financial loss that impacts POL and/or RMGA reputation
(as agreed between POL and RMGA Customer Services)
If a PCI Major Incident process is invoked.
Priority C A Post Office working normally but with a known disability, e.g. an
Non critical interim solution (workaround) has been provided.
Has a minor adverse impact upon the delivery of service to a small
number of end users
Priority D Insignificant and usually cosmetic error, either a trivial
Non urgent documentation error or spelling error on the system.
Single-user affecting incidents on non key functionality
Non user affecting incidents
NOTE: This is the default priority if 1" line do not provide an
“RMGA severity” in the TfS incident.
Priority E
Internal incidents
* Post Office down A priority. 2 — 4" line application support units do not provide cover for Post Offices
down outside normal working hours (These hours are defined in SVM/SDM/SD/0004).
7.2 Development priorities
Is it envisioned that development units may want to redefine these incident priorities once a satisfactory
work around has been agreed and documented with 3° line support and RMGA customer service. This is
the stage where:
1. Acode or documentation fix is required.
. os . FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) 7
Limited 2011 Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY _ Date: 28-Jul-2011
STORED Page No: 21 of 30
POL-0122495/g03/21
FUJ00086267
FUJ00086267
End to End Application Support Strategy
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
2. The release management process cuts in (defining target timescales).
3. Support definitions of priority become meaningless.
Within development groups the sequence of addressing peaks tends to be based on the requirement for
understanding and resolving peaks (returning to standard service) and then fixing peaks:
1st priority - analysis of new peaks
2nd priority - fixing targeted peaks
3rd priority - analysis of peak backlog or deferred peaks
4th priority - preparing fixes for untargeted peaks
8 Definition of incident timescales
It is expected that although calls may enter the support chain at a high priority, in the majority of cases
support will produce a resolution for the incident, and at that stage the priority of the call will be reduced
in order to provide 3° and 4th line support with sufficient time to allow a root cause analysis and possible
code fix.
Since it is incumbent upon the 3° line to produce an incident resolution and on 4th line to produce the
final code solution to any software problem, for the majority of its “life” any incident should be with one of
those two units.
Once a resolution has been generated for an incident the root cause fix may be deferred to a later
release of the software and the targets specified below no longer apply. The RMGA release management
process takes over at this point.
8.1 Full life times
Target times to resolve software incidents are as follows:
A Priority 2 working days
B Priority 4 working days
C Priority 7 working days
D Priority 28 working days
Note that these are targets and that no formal SLA or penalties apply to these timescales
8.2 Transfer times
The target times within each line of support are show below. They represent the maximum time each
team should retain an incident before transfer to the next level of support. These times are measured
from the time the incident was logged.
1* to 2nd 2" to 3rd 3" to 4th 4" resolution
Apriority 30 mins 2 hours 1 day 2 days
B priority 1 hour 1 day 2 days 4 days
C priority 1 hour 2 days 4 days 7 days
D priority 1 hour 7 days 14 days 28 days
; A FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) ;
Limited 2011 Version; 1.0
UNCONTROLLED IF PRINTED OR LOCALLY Date: 28-Jul-2011
STORED PageNo: 22 of 30
POL-012249¢g03/22
Fe)
FUJITSU
FUJ00086267
FUJ00086267
End to End Application Support Strategy
FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
9
Incident closure categories
9.1 Peak closure categories
The KEL column is used to indicate whether a KEL should be raised (or amended). “Opt,” indicates that it
is left to the discretion of the person closing the call.
The Fail column has a “Yes” if use of this category is counted as a filtration failure (AKA black mark)
against previous support groups.
Code KEL Fail
60
61
62
63
64
Yes
Yes
Opt.
Yes
Yes
No
No
Yes
No.
Yes
Meaning/Usage
9.1.1 S/W fix released to call logger
Code fix has been tested and can be (or has been) released into the live estate or
test rigs.
9.1.2 Build fix released to call logger
Build fix — i.e. configuration, registry edit, incorrect DLL loaded etc - has been
tested and can be (or has been) released into the live estate or test rigs.
9.1.3 No fault in product.
Indicates that the product is working to specification. No changes are required in
software code, scripts, hardware, documentation, work instructions or training
plans. Really indicates that previous lines of support have completely mis-
diagnosed the problem. (See also 66, 70, 94, and 98).
9.1.4 Programme Approved. No fix required
Rarely used. Covers the case where there IS a fault in the product and this is
acknowledged by both Fujitsu and POL, but the fault is there as a result of an
agreed design specification, and Fujitsu would require POL to fund any correction.
MUST NOT be used without approval from HNGX programme manager or
authorized representative.
9.1.5 Published Known Error
Should only be used when the resolution of the problem is documented in a KEL
and does not require the call to be passed to this support group.
When the KEL is raised after the call is logged the call should be closed as
Unpublished known error.
To be used when there was already a KEL in existence when the call was passed
© Copyright Fujitsu Services CONFIDENCE)
Limited 2011
FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0875
Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY _ Date: 28-Jul-2011
STORED PageNo: 23 of 30
POL-012249¢;g03/23
Fe)
FUJITSU
FUJ00086267
FUJ00086267
End to End Application Support Strategy
FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
65
66
67
68
70
72
Yes
Yes
Yes
No
Opt.
No
No
No.
Yes
on. And the KEL fully described the problem. And no changes have been made to
the KEL as a result of diagnosing this or other later calls. (c.f. cat 65). KEL number
MUST be quoted in response text.
9.1.6 Unpublished known error
To be used instead of 64 to close those calls where this support group now knows
the problem (and optionally solution), but where no KEL was visible at the time the
call was passed on. New KEL to be raised or old KEL to be updated with every use
of category 65. KEL number MUST be quoted in response text.
9.1.7 Enhancement request
To be used when the error complained of is not a fault against the original
specification, but everyone agrees that a change is needed to avoid the error in
future. CP number or other equivalent reference should be quoted in the response.
9.1.8 Solicited Known Error
To be used to cover those cases where a call has been sent to the support group
in response to a specific request in a published KEL. Ideally it would only be used
if itis clear in the call text that the previous support group have indeed spotted the
KEL and have sent the call in as a result. If there is no such indication, then
category 64 should be used instead.
9.1.9 Administrative response
Only to be used for closing calls which cannot be closed in a legitimate category
for “administrative” reasons — e.g. incident incorrect changed by the system (Peak,
TfS or the OTI); Test calls; Miss-routes; Double escalates; Unintended escalates
etc. Not to be used as a catch-all for “unable to decide which category to use”. See
also 200 — “Withdrawn by user”.
9.1.10 Avoidance Action Supplied
To be used when there IS a fault in the product (usually a one-off), but for whatever
teason, there is no time or justification for fixing it in the current (or any future)
release. Typically this will be used for migration or build problems or when the
facility is to be withdrawn at a future release. Should include a KEL reference in
the response if there is even the slightest chance of a recurrence. MUST include a
clear avoidance action that can be taken by the user / support on this or a
subsequent occurrence.
9.1.11 Duplicate call
To be used when two calls are discovered to relate to the same incident. (E.g.
when both HSD and SMC event management report an error message). Not
recommended for use when 2 calls for separate incidents can be traced to a single
root cause (e.g. code error). In that case use a combination of appropriate code for
© Copyright Fujitsu Services CONFIDENCE)
Limited 2011
FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0875
Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY _ Date: 28-Jul-2011
STORED PageNo: 24 of 30
POL-012249% 503/24
Fe)
FUJITSU
FUJ00086267
FUJ00086267
End to End Application Support Strategy
FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
74
90
92
94
95
96
97
Yes
No
Yes
Yes
Yes
Yes
Opt.
No
Yes
Yes
No
Yes
the first incident and then published or unpublished known error for the second and
subsequent incidents.
9.1.12 Fixed at Future release
Genuine error in the product but cannot be closed in 60 or 61 because the fix will
not be available until a later release. Applies to manuals / support guides etc as
well. Release initially targeted to contain the fix should be quoted in the response
text as well as the “target release” field. (See also enhancement request for use
when target release is unknown).
9.1.13 Reconciliation — resolved
Special category for closing reconciliation calls, Only used between SSC and MSU.
incident management.
9.1.14 Suspected hardware fault
Use when the problem was caused by a fault in a piece of hardware or the
network. Record the symptoms and cure in a KEL to aid future diagnosis of similar
problems. Hardware fault calls are subject to strict SLAs and may result in
penalties if these have not been resolved at 1* line.
9.1.15 Advice and Guidance given
This code should be used as an alternative to “No fault” or “User Error” in those
cases where it should be obvious to everyone that a product is working to
specification — e.g. documented feature or symptoms described in a support
guide. Can also be used to highlight cases where the end customer or someone
in the support chain could benefit from further training in this area of the product
(e.g. if the PM does not accept that (s) he is guilty of a user error).
9.1.16 Advice after investigation
Similar to “Advice and guidance given” but used in the situations where it was not
obvious that the system was working to specification. i.e. User documentation or
support guides do not contain the information the advice given is based on.
9.1.17 Insufficient evidence
Use when it is crystal clear from the KEL or other published sources what evidence
is required to accompany a call of this class, but that information has not been
supplied. (Check that the call post-dates the latest update of the relevant KEL).
See also special case use of 62 - “No fault”
9.1.18 Unspecified insufficient evidence
To be used when the problem cannot be resolved from the evidence supplied and
© Copyright Fujitsu Services CONFIDENCE)
Limited 2011
FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0875
Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY _ Date: 28-Jul-2011
STORED PageNo: 25 of 30
POL-012249¢g03/25
FUJ00086267
FUJ00086267
End to End Application Support Strategy
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
more evidence is being requested but there was nothing to tell the other
support groups that this particular piece of data is required to investigate
this class of call.
KEL should be raised or amended to give future guidance, if this class of evidence:
may be needed in future. A support guide update may also be appropriate.
98 No Yes
9.1.19 User Error
Close in this category when it is clear from the evidence that the end user has
caused the problem by doing something incorrectly. The documentation or training
available to the user concerned should specifically cover this incident — if it does
not then consider using 94 instead. No KEL is required - documentation should
already cover the case.
100 No No
9.1.20 Route call to TfS
Special code which “closes” the call on Peak and causes it to be routed to the
system used by ISD NT / Unix (Ops) or Networks, without it registering as “closed”
or “finished with” on their systems.
200 No No
9.1.21 Call withdrawn by user
Specific case of an Administrative Closure. To be used when the user (e.g. PM) or
other support group specifically request that a call is returned to them without the
currently assigned team doing any further work on it.
10 Outline contents of a support guide
Support guides are generally written by a combination of the architect / designer for the product and the
developers of that product and are based on the DES/APP/SPG/nnnn document management template.
Although intended for the support community the support guide should be produced in time for the initial
test cycles of a new facility. This will help testers understand what they are testing and also validate the
contents of the support guide.
10.1 Overview of the facility
Often cribbed from relevant design HLDs (for which cross-references should also be provided)
e Purpose and functional overview
e The way in which it performs the function
10.2 Documentation
A list of document references, with titles, associated with the facility. This is intended give the support
community a list of reading matter that can be used to self train.
. os . FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) 7
Limited 2011 Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY _ Date: 28-Jul-2011
STORED Page No: 26 of 30
POL-012249¢ 503/26
FUJ00086267
FUJ00086267
End to End Application Support Strategy
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
¢ Table of document references and titles. This will certainly include RMGA written HLDs, LLDs
and design notes.
e If the facility includes COTS products then references to any externally produced support or user
guides (normally these will have been pre-registered in Dimensions).
10.3 Server definition
Define the servers and workstations which support will need to access in order to support the product.
e Which servers and / or workstations are involved in the delivery of the facility
e List any COTS products and version that should be installed that the facility depends on.
e Define the location of configuration files relevant to the facility or COTS product.
10.4 Diagnostics and logging
Define what log files are produced by the facility and where the support groups can find them. Can the
level of information logged be changed, if so how? How are they interpreted?
e Location, configuration and layout of log files
e What other diagnostics are written
e How can the diagnostics be configured
e Supply samples of diagnostics
10.5 Errors and messages
This is intended to provide support with guidance on common messages or errors produced by the
software. It is not helpful to simply copy and paste a list of error texts into the guide without any further
detail!
It is expected that this section can be enhanced with:
« Any messages encountered by testers that they did not understand — support will probably see
the same thing in live!
e Any errors that result due to miss-configuration of the test rigs
The information will probably take the form of a table including:
e The exact text of the error / message
« What events / activities cause the message to be produced
« What avoidance action should be taken (KEL reference where applicable)
« What impact will it have on the service the facility provides
10.6 Code base and APIs
In general support groups would expect that all code written is self documenting with comments included
to aid diagnostics. Meanwhile in the real world (again, just checking!):
e Outline of the code standards and languages used for the various parts of the facility.
. os . FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) 7
Limited 2011 Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY _ Date: 28-Jul-2011
STORED Page No: 27 of 30
POL-012249¢;g0327
FUJ00086267
FUJ00086267
End to End Application Support Strategy
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
e Where can support find the source code, which source repository. Define location and type
(CVS, Subversion etc).
e Define administrator for source repository (i.e. where can a user / password be obtained to
access).
e Details of external APIs provided (may be a cross-reference to appropriate LLDs)
10.7 Support route
Definition and contact list. Who supports the product at what levels. At 1* to 4" line this is likely to be just
a definition of which RMGA support groups are involved.
In development, define who the lead developers and architects are.
If any interface touches units external to RMGA define:
e Their role in the support process
* How to contact them
« What hours they work
« What level of service is offered
e Expected clearance timescales for problems of different severities
NOTE: This may already be provided in an interface specification document in which case a cross
reference to that document is adequate.
11 Knowledge base maintenance
Two knowledge bases are maintained within RMGA
11.1 TfS documents
Just known as “documents” within TfS and referred to as a DOC ID in incident updates. These are
maintained by the HSD in some cases as a copy of KELs. These copies are purely an HSD requirement
to allow them to update the information so that it is suitable for 1 line agents to understand.
11.2 KEL
The KEL is the master knowledge base for all incidents being progressed through the support chain. It
was developed by the SSC who also maintain the servers it runs on. Because of this history, the SSC
have also become the arbiters of the information within the KEL.
It is also the SSC’s responsibility to support the KEL system and to allow access to this register to all
other support units so that they can enter details within their area. Support for the KEL system is only
provided during normal SSC working hours.
11.2.1 New KEL generation
All support units have access to the KEL system and are able to generate new knowledge entries. Anew
KEL should be generated for each new incident that is raised on the live system. Before any new KEL is
generated it is essential that an extensive search of the KEL is done to ensure that a duplicate is not
being created.
. os . FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) 7
Limited 2011 Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY _ Date: 28-Jul-2011
STORED Page No: 28 of 30
POL-012249¢g03/28
FUJ00086267
FUJ00086267
End to End Application Support Strategy
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
It is not necessary to fill in all fields when generating a new KEL. This may happen when a 2° line
diagnostician can define the symptoms but has been unable to determine problem and solution. It can
then be passed as a skeleton KEL to the next level of support who should update with further details
when known. In these circumstances the KEL number MUST be added as a reference on the Peak.
Aminimum KEL would consist of:
1. Title
2. Summary
3. Release
4. Type
5. Peak reference
6. Symptoms
This should be considered to be a minimum standard.
11.2.2 KEL authorisation
When a KEL is created or updated is has to be authorised before it can be seen by all users of the KEL.
This is an SSC function. The KEL web site will send an automated email to request authorisation. It
selects a suitable default recipient SSC diagnostician as follows:
1. Newly generated KELS are authorised by the SSC duty Prescan diagnostician for the day.
2. Updated KELS are authorised by the owner of the KEL
It is possible to amend the authoriser in specific cases if required.
It is expected that the SSC diagnostician will complete the authorisation within one working day. If this
does not happen please follow the usual escalation process.
11.2.3 KEL rejection
Anew or updated KEL may be rejected if:
e The update adds no value
¢ — Information on the KEL is incorrect
e Minimum information has not been specified (see 11.2.1)
If a KEL is rejected a reason is given by the authoriser and an email is sent to the originator. It is then the
originators responsibility to either:
e Correct the rejected KEL based on feedback from the authoriser.
e Respond to the authoriser if they do not agree with the rejection.
e Request that the SSC delete the rejected KEL
It is not expected that a KEL will be rejected where a minor change can be made by the authoriser that
will result in a satisfactory KEL.
NOTE: If a rejection is not responded to within 2 weeks the new KEL details are deleted.
11.2.4 KEL updates
. os . FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) 7
Limited 2011 Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY _ Date: 28-Jul-2011
STORED Page No: 29 of 30
POL-012249¢;g03/29
FUJ00086267
FUJ00086267
End to End Application Support Strategy
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)FUJITSU RESTRICTED (COMMERCIAL
IN CONFIDENCE)
Updates to existing KELs can be made by anyone with access to the KEL system.
e There is no requirement to update a KEL with a new Peak number simply because another
incident has been seen for an unresolved issue. Counting this type of incident is the function of
HSD master call processes.
e KELs should be updated with workaround or resolution details as soon as they are available.
e KELs should be updated with the details of a baseline when generated by 4" line
e KELs should be updated with details of when a resolution was delivered to the live estate. KELs
should not be deleted under these circumstances since code regression may occur later.
e All KELs have version numbers, it is always possible to view a previous version and highlight the
differences between versions.
When a KEL is updated it will need to be authorised again. The old version of the KEL is still visible until
the authorisation has taken place.
11.2.5 KEL deletion
KELs should be deleted when:
e The function they refer to no longer exists in the live system
e Itis a duplicate of an existing KEL with the same problem and resolution.
e The KEL contains misleading information and no update is appropriate.
KELs can only be deleted by members of the SSC. If you think a KEL should be deleted then update the
KEL and replace the KEL summary with the text “KEL SHOULD BE DELETED: Reason” and specify
“Reason”. The SSC diagnostician you direct the update authorisation to will then check and delete the
KEL if they agree on the reason.
. os . FUJITSU RESTRICTED (COMMERCIAL IN Ref: SVM/SDM/PRO/0875
© Copyright Fujitsu Services CONFIDENCE) 7
Limited 2011 Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY _ Date: 28-Jul-2011
STORED Page No: 30 of 30
POL-012249%g03;30