POL00393981
POL00393981
ICL Pathway ICL Pathway problem management Ref:CS/PRO/063
procedure Version:1.0
Date:30/1/01
Document Title: ICL Pathway Problem Management Procedures
Document Type: Procedures
Abstract: This document describes the procedures for ICL Pathway
Duty and Problem Management to support the Customer
Service Operations Manual.
Status: Approved
Distribution: Stephen Muchow
Martin Riddell
Peter Burden
Paul Westfield
CS Problem managers
Library
Author: Evandro Manolas (amended by P.Curley)
Comments to: Author
Comments by: 15" February 2001
0 Document control
0.1 Document history
COMMERCIAL IN CONFIDENCE Page 1 of 1
© 2001 ICL Pathway Ltd
ICL Pathway
POL00393981
POL00393981
ICL Pathway problem management Ref:CS/PRO/063
procedure Version:1.0
Date:30/1/01
0.2
0.3
Version Date
0.1 01/05/99
0.2 04/02/99
0.3
1.0 30/01/01
Reason
First draft
Update following observations from walkthrough held on
13/01/99.
Update following Problem management and Business
Continuity Management Walkthroughs
Document rewritten to reflect current procedures
Approval Authorities
Name
Paul Curley
Position Signature Date
Service Manager
Associated Documents
Reference Vers Date Title Source
CS/PRD/021 2.3
CS/QMS/005 1.01
CS/PRO/110 2.0
CS/IFS/009 0.3
CS/IFS/008 0.2
CS/QMS/001
CS/QMS/002
ICL Pathway Problem Management ICL Pathway
Process
24/01/01 ICL Pathway Customer Service: ICL Pathway
Operations Services Operations
Manual
13/11/00 ICL Pathway Customer Service: ICL Pathway
Problem Management database
procedures
4/9/00 ICL Pathway / OSD Interface ICL Pathway
agreement for the problem
management interface
7/7/00 ICL Pathway / PON Interface ICL Pathway
agreement for the problem
management interface
CS Policy Manual ICL Pathway
CS Process Manual ICL Pathway
COMMERCIAL IN CONFIDENCE Page 2 of 1
POL00393981
POL00393981
ICL Pathway ICL Pathway problem management Ref:CS/PRO/063
procedure ve te:30/1/01
0.4 Abbreviations
ATP. Authorised Temporary Procedures
FSM Field Service Manager
HSH Horizon Systems Helpdesk
HSRF Horizon Service Review Forum
ICLP ICL Pathway
ISD Infrastructure Services Division (formerly OSD and division within ICL)
MBCI Major Business Continuity Event
NBSC Network Business Support Centre
PIR Post Implementation Review
PM Problem Manager
PON Post Office Network , a division of Post Office Counters Limited
TP Transaction Processing (A unit within PON)
XDPMF Cross Domain Problem Management Forum
0.5 Changes in this version
- Document re-written to reflect current problem management operation.
COMMERCIAL IN CONFIDENCE Page 3 of 1
POL00393981
POL00393981
ICL Pathway ICL Pathway problem management Ref:CS/PRO/063
procedure Version:1.0
Date:30/1/01
0.2 Table of content
0.1 DOCUMENT HISTORY.
0.2 APPROVAL AUTHORITIES.,
0.3. ASSOCIATED DOCUMENTS.......
0.4 ABBREVIATIONS.
0.2 TABLE OF CONTENT.
1 INTRODUCTION...
1.1 TERMINOLOGY..... svsennnnseen soe - cee 5
1.1.1 Incidents.
1.1.2 Problems...
SCOPE...
PROCEDURAL RULES.
3.1 COMMUNICATION .......c00 sosnnenrnnnnesenn ese arene 6
3.2 THE PROBLEM DATABASE. 6
3.3. ESCALATION. 6
3.4. HoLtDAy/ SICK LEAVE. 6
4 PROBLEM MANAGEMEN’
4.1 PROBLEM SOURCES.
4.1.1 Customer identified.
4.1.2 FSM identified...
4.1.3 Operational.
4.2 PROBLEM REPORTING.
4.2.2 ICLP reported problems
4.3 PROBLEM ACCEPTANCE.
4.4 PROBLEM MANAGEMENT.
4.4.1 Assessing the impact.
4.4.2 Closure Criteria...
4.4.3 Managing the problem.
4.4.4 Temporary Procedures.
4.4.5 Problem Closure.
4.4.6 PIR (Post Incident Review
5. I AUTHORISED TEMPORARY PROCEDURES (ATP) FOR CROSS-DOMAIN PROBLEM
6 ESCALATIO!
6.1 MBCI. 12
6.2 Poor Progress... wl?
6.3 Customer Dissatisfaction. 12
12
6.4 Significant Business impa
7. ALERT PROCESS....
8. APPENDIX 1...
COMMERCIAL IN CONFIDENCE Page 4 of 1
POL00393981
POL00393981
ICL Pathway ICL Pathway problem management Ref:CS/PRO/063
procedure Version:1.0
Date:30/1/01
1
1.1
Introduction
This document describes the procedures for problem management within
ICLP Customer Service. The procedures support the ICL Pathway Problem
Management Process, CS/PRD/021 [Ref.1].
Terminology
Incidents
Incidents are individual day to day events resulting from:
= Faults or failures in equipment, software, services or procedures;
= User error
= User requests for advice and guidance
Incidents are managed via a framework of function specific Help Desk
services. Once a solution or temporary procedure has been supplied and
accepted the individual incident will be closed. The focus of these help desks
is to restore the operation as quickly as possible. System and Technical
incidents are managed via the Horizon System Helpdesk (HSH) and business
incidents are managed via the Network Business Support Centre (NBSC).
Problems
A problem is an underlying cause that may result in incidents
A problem potentially exists when a defect in the specification, design,
production, implementation, or use of any of the service components results
in any aspect of the service not meeting expectations.
A problem will be raised when the impact of the defect is substantial enough
to warrant action to eradicate it. The problem will be closed when it has been
agreed that the underlying cause has been fixed or removed.
Scope
This document provides guidelines for Problem Managers to effectively
manage problems. The document underpins the ICL Pathway Customer
Service problem management process (CS/PRD/021) [Ref. 1] and identifies
the sources of problems, how to register and manage the problem and
responsibilities of the Problem Manager.
The scope of the document includes the following:
1. Duty Management procedures for problem reception
2. Problem Management procedures
3. Escalation and alert procedures.
COMMERCIAL IN CONFIDENCE Page 5 of 1
POL00393981
POL00393981
ICL Pathway ICL Pathway problem management Ref:CS/PRO/063
procedure Version:1.0
Date:30/1/01
Procedural Rules
3.1
3.2
3.3
When following the procedures within this document, it is important that the
rules within this section below are complied with.
Communication
Communication is vital within the problem management process. The
originators of problem calls should regularly updated and there should be
constant communication between those involved in the resolution of
problems.
When possible, communication must be pro-actively arranged through
agreed regular update intervals. However, those involved in problem
resolutions must also be prepared to re-act to changing circumstances and
unexpected ‘twists’ and communicate with others as necessary.
The Problem database
The ICL Pathway Customer Service department Service Managers and the
Duty Manager will be made aware of problems. It will be their responsibility to
ensure that problems are progressed and logged upon the department’s
Problem database.
Having been assigned a problem, it is essential that the Problem Manager
keeps the database updated with the latest events of the problem, the actions
to be taken and deadlines. Therefore, anyone else needing to know the
details of the problem can easily find them out by looking up the details on
the database.
Details the Database logging procedures can be found in ICL Pathway
Customer Service: Problem Management database procedures document
(CS/PRO/110) [Ref. 3]
Escalation
It is important to note that ‘Escalation’ (Section 6) has the potential to occur at
any stage of the process. The procedures detailed within this proposal may at
times actually point specifically towards escalation at certain steps of the
process since these areas are the most likely to invoke the escalation
process.
However, the Problem Manager can escalate at any time that he/she feels it
necessary to do so, in order to inform the escalated authority of events.
Unresolvable disagreements or decisions that need to be made at a higher
level can also occur at any stage of the process which again might require
escalation in order to resolve them.
COMMERCIAL IN CONFIDENCE Page 6 of 1
POL00393981
POL00393981
ICL Pathway ICL Pathway problem management Ref:CS/PRO/063
procedure Version:1.0
Date:30/1/01
3.4
41
4.1.1
4.1.2
Holiday/ Sick Leave
The PM should ensure that during any planned leave a suitable stand in is
found and where necessary progress the problem in his or her absence.
During sick leave the line manager responsible should provide progress
updates where necessary.
Problem Management
The aim of problem management is to identify and remove the root causes of
issues that cause incidents to be raised from within the ICL Pathway estate.
The role of the Problem Manager is to drive any identified problem to a
conclusion by ensuring that resources are available to investigate and
remove the problem and to report on progress. Problem management within
ICLP is resourced from the various operating and service management units
within Customer Service. Problems are allocated to individual problem
managers depending on areas of knowledge and expertise, areas of
responsibility and availability.
Problem sources
Problems arise from various sources within the Horizon operational estate
most problems are concerning the live estate but are occasionally raised
before a product or release is live.
Customer identified
PON identify issues from feedback with various PON operational units such
as reconciliation (TIP) and NBSC helpdesk. PON raise the potential problem
via the daytime Duty Manager.
FSM identified
The Field Service Managers (FSM) interface on a daily basis with the outlets
and identify “problem” outlets from various sources such as trend analysis of
incidents raised, and feedback from outlets and PON. The FSM will arrange
visits to outlets to interview the outlet staff and ascertain the scope of the
issues within the outlet. Many of these issues are resolved within the FSM
processes but some may develop into problems.
Operational
The units managing the day to day operations and support of the horizon
estate escalate potential problems to the duty manager. The duty managers
will decide if the issue in hand is to be treated as an incident, a problem or a
major business continuity incident. This is usually decided based on the
operational impact, severity and visibility of the issue.
COMMERCIAL IN CONFIDENCE Page 7 of 1
POL00393981
POL00393981
ICL Pathway ICL Pathway problem management Ref:CS/PRO/063
procedure Version:1.0
Date:30/1/01
4.2 Problem reporting
Problems that are identified outside of ICLP for example by the operational
units such ISD, HSH or PON are reported and raised via the daytime Duty
Manager. These types of problems and responsibilities of parties involved are
documented in the problem management interface agreements between ISD
and PON [Ref. 4 &5]. Problems identified from ICLP staff such as Service
Managers or FSM’s are registered directly onto the ICLP problem database.
4.2.1 Reporting problems via Duty manager
Once a problem is identified and meets the criteria described in the problem
management interface agreement(s) the originator pages the Duty Manager
(pager numbers are documented in the respective interface agreements). The
pager message is sent requesting contact with the originator. The information
contained within the page is Contact Name, telephone number and a brief
description of the problem.
4.2.2 ICLP reported problems
Problems identified from within ICLP will normally arise from incident trend
analysis against which, underlying problems are identified. The process for
raising problems directly onto the ICLP database are explained in ICL
Pathway Customer Service: Problem Management database procedures
document (CS/PRO/110) [Ref. 3]
4.3 Problem Acceptance
Once the request to raise a problem has been received by the Duty Manager
he or she will contact the originator to gain a better understanding of the
issue and decide:
1) If the issue in hand meets the problem criteria (initial check)
2) If the issue is a Major Business Continuity Incident (MBCI) and if
necessary invoke the MBCI process.
3) Who is best placed to be problem manager for this issue (This decision is
based on departmental areas of responsibility, expertise of staff within
those areas and availability of those staff. If the Problem Manager cannot
identify a suitable problem manager he or she will escalate the problem to
the unit line manager.
The appointed Problem Manager is then responsible for contacting the
originator and accepting or rejecting the issue as a problem. If the issue is
accepted as a problem the PM needs to carry out the acceptance routine:
e Check the database to see if the issue is already recorded as a problem
« If Cross Domain record PON problem title, problem number and PON PM
COMMERCIAL IN CONFIDENCE Page 8 of 1
POL00393981
POL00393981
ICL Pathway ICL Pathway problem management Ref:CS/PRO/063
procedure Version:1.0
Date:30/1/01
44
e Record any supporting evidence (Incident numbers, FAD codes etc)
e Agree closure criteria
e Agree update schedule
Problem Management
The PM co-ordinates the resolution of the problem and acts as a single point
of contact for everyone involved with the problem.
This role is part of the end-to-end process described in the ICL Pathway
Problem Management Process (Ref. CS/PRD/021) [Ref. 1].
The stages involved in managing a problem once accepted are:
e Analysis
« Action
e Resolution
e Monitor
The PM needs to assess the impact of any problem and arrange for the
gathering of evidence or statistics to support this assessment and identify the
root cause(s). At this stage a work around or Temporary procedure may need
to be developed to circumvent the impact of the problem. The PM will develop
an action plan describing who does what and when and once the root cause
is known how to resolve the issue. Once the solution has been applied the
problem and change must be monitored to ensure that the solution has
worked
4.4.1 Assessing the impact
The problem manager should attempt to assess the impact of the problem.
The reason for assessing the impact is twofold. Firstly it is important to
attempt to understand the extent of the problem to enable effective escalation
if required. Secondly, the PM will be expected to provide regular updates on
the problem. Understanding the impact will enable a suitable update
timetable to be set.
To enable a better understanding of the impact the PM should check to see if
the issue as an affect on the daily operational schedule. Check the volumes
of incidents raised at the helpdesk(s) and if it’s a cross domain problem, the
business impact within PON.
Serious operational issues that affect large numbers of outlets or concerning
a serious hardware failure may result in a Major Business Continuity Incident
COMMERCIAL IN CONFIDENCE Page 9 of 1
POL00393981
POL00393981
ICL Pathway ICL Pathway problem management Ref:CS/PRO/063
procedure Version:1.0
Date:30/1/01
(MBCl) in these cases the problem should be escalated by the Duty Manager
to the MBCI manager.
4.4.2 Closure Criteria
When registering a problem the PM should set and agree suitable closure
criteria. If the problem is cross domain and affects PON then the closure
criteria must be agreed with the PON problem manager.
Closure criteria should be specific to the issue, achievable, measurable and
clear. It is important to ensure each problem is being driven towards a
specific point (the closure criteria) and that progress is reported towards
achieving closure.
Once closure criteria is agreed any change to this must be avoided otherwise
the problem will “drift” and closure will become difficult.
4.4.3 Managing the problem
Most problems will arise from an incident or several incidents. In these cases
the incident management process will progress the underlying incidents to
resolution. In these cases, the PM must take ownership of the resolution
timescale and ensure resolution is line with the problem requirements. .
To do this the PM should regularly review/monitor Powerhelp and PINICL to
determine progress and chase the support unit managers to ensure visibility
of the problem and support focus is maintained in correcting it. Any incident is
likely to pass through several stages:
e Initial investigation and possible deployment of a temporary procedure
e Evidence collection and identify the root cause
e Developing the solution
e Testing and Release
e Monitoring
The PM should also monitor further incidents that relate to the problem, these
may increase the impact of the problem and raise the urgency.
The PM should initially identify if a work around or temporary procedure is
required to circumvent the fault condition. Any change to the outlet or PON
business processes will have to be agreed with PON and communicated to the
outlets via normal communication channels.
The problem manager should develop an outline plan to manage
communications, resources and actions.
Many problems and incidents stall whilst evidence is collected, the PM must
drive support by monitoring the any further incidents and ensuring that they
are progressed to the relevant support unit and re-enforcing any business
impact through escalation if necessary.
COMMERCIAL IN CONFIDENCE Page 10 of 1
POL00393981
POL00393981
ICL Pathway ICL Pathway problem management Ref:CS/PRO/063
procedure Version:1.0
Date:30/1/01
4.4.4
4.4.5
4.4.6
5.
Once a solution is found the development and testing cycles can be extensive,
the PM must determine the delivery timescale on any fix and ensure that
business impact is considered.
Temporary Procedures
Discuss with the expert domain and any other relevant parties if temporary
procedures need to be put in place whilst the solution is being produced. If
they are, ensure that the process to develop and implement the temporary
procedures is carried out by the responsible domain.
Problem Closure
Problem closure must be agreed with the originator and will probably only be
achieved after a monitoring period. During the monitoring period the Problem
Manager activities will be to detect and report any further incidents that relate
to the problem.
Withdraw any Temporary Procedures. These will be withdrawn in parallel with
the release of the solution.
Agree with the appropriate parties that the problem can be closed.
PIR (Post Incident Review)
A PIR can be called by exception, and is normally called if the management
of a problem goes awry or if requested by a third party. The Problem Manager
is responsible for setting up and managing the PIR. The PIR will then be used
as the tool for authorising the closure of the problem and will be signed by
each party that is required to agree closure.
A copy of the PIR report should be attached to the problem on the Problem
database, and a copy with the signatures will be kept in a file that is
maintained by the Problem Database administrator.
If a PIR is not required, the Problem will only be closed once all parties
concerned have sent confirmation that they agree to closure.
Once confirmation for closure has been received, whether through the PIR or
not, the problem can be closed on the Problem Management database. If
necessary, close it with the HSH and inform all other impacted parties that the
problem has been closed.
Authorised Temporary Procedures (ATP) for Cross-
Domain Problems
An ATP is a temporary arrangement that may be required to allow normal
operations to continue in advance of the eventual resolution of a problem that
affects PON operation or business processes. All ATP must be developed in
conjunction with PON Business Service Management unit and communicated
to the outlet via normal communication channels.
The Problem Manager will own the ATP and will co-ordinate the development,
authorisation, implementation and withdrawal of the ATP.
COMMERCIAL IN CONFIDENCE Page 11 of 1
POL00393981
POL00393981
ICL Pathway ICL Pathway problem management Ref:CS/PRO/063
procedure Version:1.0
Date:30/1/01
6.1
6.2
6.3
6.4
Escalation
Escalation occurs in several different circumstances:
« MBCI
e Lack of support or progress with a problem
« Customer dissatisfaction
e Significant business impact
MBCI
There are several routes of escalation available to the PM that align to these
criteria. Any call to the duty manager that is deemed to be an MBCI will be
escalated by the Duty Manager. Once the MBCI is declared PON is informed
of the issue and the PM will then be required to provide frequent updates to
the MBCI distribution list. The focus at this point is recovery of the operational
failure and the PM will be the centre of communications. Once the operational
recovery is complete the problem will come off the escalation list and the PM
will run the problem as normal identifying and removing the root cause.
Poor Progress
If a PM is dissatisfied with the progress of any problem it should be escalated
to the Customer Service Weekly prayers review meeting. Attendees ate the
meeting include representatives from development, support, operations and
infrastructure. Actions are placed at the review to progress problem areas.
If progress required from PON is unsatisfactory then these problems are
escalated to the Cross-Domain Problem Review Forum (XDPMF) which is
held monthly. Problems requiring escalation to the Horizon Service Review
Forum (HSRF) are also discussed at this forum to ensure that both PON and
ICLP are able to properly brief the HRSF attendees.
Customer Dissatisfaction
The complaint procedure and Field Service Management procedures
normally handled issues surrounding customer dissatisfaction. Some of the
underlying issues in these cases become registered and managed as
problems.
Significant Business impact
COMMERCIAL IN CONFIDENCE Page 12 of 1
POL00393981
POL00393981
ICL Pathway ICL Pathway problem management Ref:CS/PRO/063
Version: 1.0
procedure
Date:30/1/01
Problems that have a significant business impact on ICLP or PON will be
escalated at the weekly and monthly reviews. They may also be placed on
divisional alert status.
Alert Process
The divisional alert process is used to identify significant problems to ICL
Pathway senior management and to gain support from other directorates in
addressing the problem. Once a problem is on divisional alert, weekly
updates are provided to ICLP directors and senior managers appraising them
of progress against plans and issues with the problem.
COMMERCIAL IN CONFIDENCE Page 13 of 1
POL00393981
POL00393981
ICL Pathway ICL Pathway problem management Ref:CS/PRO/063
procedure Version:1.0
Date:30/1/01
8. Appendix 1
Information required to log and maintain a problem on the Problem database
Logging a problem
1. (If available) incident reference numbers.
- these might come from the helpdesk (such as the HSH) or from the customer
(such as PON Problem reference number)
Where the call originated and who logged it.
Details of the problem
Actions taken to date
ak ON
A contact for the Problem Manager
- For example, if the call originated from PON, who in PON should the Problem
manager be contacting to find out more information and progress the incident.
Updating the Problem
The problem should be updated at least weekly on the problem database. Problems
available to PON are updated weekly each Friday.
Cross-Domain Problem Manager Responsibilities
The Problem manager is the person within the Service management organisation
with responsibility for:
a) accepting and logging problems
b) managing impact assessment
c) brokering prioritisation with counterparts in the other organisation
d) monitoring progress
e) reviewing solutions
f) closing problems
g) Primary channel for all communication between the organisations
h) Responsible for logging all details
COMMERCIAL IN CONFIDENCE Page 14 of 1