Fujitsu Services
Major Incident Report
Commercial-in-Confidence
POL00029277
POL00029277
Ref: CS/REP/182
Version: 0.2
Date: 01-MAR-2004
Document Title:
Document Type:
Release:
Abstract:
Document Status:
Originator & Dept:
Contributors:
Internal Distribution:
External Distribution:
Approval Authorities:
Major Incident Report for Debit Card Transaction Failures on
Saturday 14" February 2004
Report
Report on the Debit Card Service Failures on Saturday 14" of
February
Draft
Tony Wicks - POA Customer Service
Distribution for draft - as per section 0.2
Distribution for Approved version - as above plus Pete Jeram,
Liam Foley, Colin Lenton-Smith, Ian Lamb, Richard Brunskill,
Reg Barton, Bill Mitchell
Dave Hulbert
(see PA/PRO/O10 for Approval roles)
Name
Position
Signature
Date
Martin Riddell
FS POA Service Director
Commercial-in-Confidence
©Copyright Fujitsu Services Limited 2004
Page: I of 11
POL00029277
POL00029277
Fujitsu Services Major Incident Report Ref: (CS/REP/182
Version: 0.2
Commercial-in-Confidence Date: 01-MAR-2004
0.0 Document Control
0.1 Document History
Version No. I Date Reason for Issue Associated
CP/PinICL
0.1 18/2/2004 Initial Draft
0.2 01/03/04 Includes comments from Peter Burden on the initial
draft.
0.2 Review Details
Review Comments by : I 03/03/04
Review Comments to: I Tony Wicks
Mandatory Review Authority Name
See Review Role Matrix in PA/PRO/010
Fujitsu Services Post Office Account Mik Peach
Fujitsu Services Post Office Account Martin Riddell
Fujitsu Services Post Office Account Peter Burden
Fujitsu Services Post Office Account Reg Barton
Fujitsu Services Post Office Account Mike Stewart
Fujitsu Services Post Office Account Simon Fawkes
Fujitsu Services Post Office Account Dave Tanner
Fujitsu Services Post Office Account Richard Brunskill
Fujitsu Services Post Office Account Dave Law
Fujitsu Services Networks Colin Johnson
Fujitsu Services System Management Centre Tan Cooley
Optional Review / Issued for Information
( * ) = Reviewers that returned comments
Commercial-in-Confidence Page: 2 of 11
©Copyright Fujitsu Services Limited 2004
Fujitsu Services
POL00029277
POL00029277
Major Incident Report Ref: (CS/REP/182
Version: 0.2
Commercial-in-Confidence Date: 01-MAR-2004
0.3 Associated Documents
Reference
Version I Date Title Source
Unless a specific version is referred to above, reference should be made to the current
approved versions of the documents.
0.4 Abbreviations/Definitions
Abbreviation Definition
BCM Business Continuity Manager
cs Customer Service (in POA)
DCS Debit Card Service
DM Duty Manager
FAD Financial Accounts Division (Post Office)
HSH Horizon System Helpdesk
IVR Interactive Voice Response
NB Network Banking
PM Problem Manager
PMDB Problem Management Database
PO Post Office
POA Post Office Account
POL Post Office Limited
SSC System Support Centre
TNS Transaction Network Services
TSD Technical Services Desk
0.5 Changes in this Version
Version Changes
0.1 This is the first draft
0.2 Includes comments from Peter Burden on the initial draft.
Commercial-in-Confidence Page: 3 of 11
©Copyright Fujitsu Services Limited 2004
Fujitsu Services Major Incident Report
Commercial-in-Confidence
POL00029277
POL00029277
Ref: CS/REP/182
Version: 0.2
Date: 01-MAR-2004
0.6 Changes Expected
Changes
Commercial-in-Confidence
©Copyright Fujitsu Services Limited 2004
Page: 4 of 11
POL00029277
POL00029277
Fujitsu Services Major Incident Report Ref: (CS/REP/182
Version: 0.2
Commercial-in-Confidence Date: 01-MAR-2004
0.7 Table of Contents
1.0 INTRODUCTION...
2.0 SCOPE.
3.0 MANAGEMENT SUMMARY...
4.0 DESCRIPTION OF THE FAULT AND SERVICE FAILURE
4.1 SYMPTOMS AND BUSINESS IMPACT.
4.1.1 Symptoms as seen by Branches...
4.1.2 Symptoms as seen by Fujitsu Services.
4.2 DETAILED EXPLANATION OF THE INCIDEN’
5.0 INCIDENT MANAGEMENT.
6.0 PROBLEM MANAGEMENT.
7.0 CORRECTIVE ACTIONS.
Commercial-in-Confidence Page: 5 of 11
©Copyright Fujitsu Services Limited 2004
POL00029277
POL00029277
Fujitsu Services Major Incident Report Ref: (CS/REP/182
Version: 0.2
Commercial-in-Confidence Date: 01-MAR-2004
1.0 Introduction
This document reports on the issues that arose from a major incident which affected debit card
transactions on Saturday 14" of February 2004.
This report covers:
- How the problem came to light
- The impact on the branch service
- The investigation
- The resolution
- The suspected network and related infrastructure
- Actions to prevent recurrence
2.0 Scope
The scope of this report is the major incident that occurred on Saturday February 14" that
affected Debit Card transactions at some 1271 Branches.
3.0 Management Summary
On Saturday 14" February some Branches were experiencing intermittent service on
performing online debit card transactions. Evidence from the HSH indicated that the nature of
the fault was such that a subsequent attempt (i.e. retry) generally would have been successful.
The underlying cause of the problem has not been established but it is known that the failures
occurred in the infrastructure between the Wigan X25 router and Streamline, via the TNS
X25 network.
The HSH took 28 calls from Branches experiencing difficulties with debit card transactions.
Analysis of the Debit Card MIS Data indicates that the peak of failures occurred between
approximately 11.45 and 13.00. 1271 Branches experienced debit card failures on Saturday
and failures were experienced with 3016 individual debit cards.
At 12.06 Powerhelp incident E-0402 140283 was raised at ‘A’ priority when the SMC
received a top level monitoring alert informing them of the timeout situation.
This incident was reported to the TSD, who, at 12.30, raised Powerhelp incident E-
0402140303 and informed Dave Hulbert Post Office Limited at approximately 12.35 and
Streamline at approximately 12.38.
After being advised, during the Saturday afternoon, that TNS were unable to see network
traffic on logical access ports 1024 to 1047 on the Wigan X25 router, Fujitsu Services
Networks decided to close the X25 interface at Wigan in order to force the virtual circuits to
operate from the Bootle to circumvent the failure.
Commercial-in-Confidence Page: 6 of 11
©Copyright Fujitsu Services Limited 2004
POL00029277
POL00029277
Fujitsu Services Major Incident Report Ref: (CS/REP/182
Version: 0.2
Commercial-in-Confidence Date: 01-MAR-2004
4.0 Description of the fault and service failure
4.1 Symptoms and Business Impact
4.1.1 Symptoms as seen by Branches
1271 branches experienced failures when attempting to take payments for transactions using
debit cards. Evidence, from the handling of the calls to the HSH, indicates that subsequent
attempts to perform the financial transaction using the debit card would have been successful
on one or more retries.
4.1.2 Symptoms as seen by Fujitsu Services
The following top level monitoring alert was received by the SMC at 12:02 informing them of
the timeout situation.
@12:02 A bad alert was at the top level of DCA01, DCA02, DCA03 & DCA3 based on the
DCAO_1.SSE_TIMEOUT. With text: Debit Card Authorisation Agent had n(number)
timeouts in the last reporting period which exceeds the limit of 5.
Core Services Networks did not receive any events or warning for the failures on the X25
routers or network connections to TNS.
4.2 Detailed explanation of the incident
The underlying cause of the problem has not been established but it is known that the failures
occurred in the infrastructure between the Wigan X25 router and Streamline, via the TNS
X25 network.
Core Services Networks were initially called, at about 12.22, by the SMC who asked verbally
if Networks could see anything wrong with the debit card infrastructure.
Networks initial investigation showed that the X25 routers appeared to be functioning
normally and that all X25 interfaces were available.
There are two X25 routers, Wigan_DCS_1 and Bootle_DCS_1, which are used to connect to
Streamline. The service is configured to use 16 X25 virtual circuits to pass transaction traffic.
The routers are configured to use a range of logical access ports as follows,
Bootle 1024-1053 (30 ports)
Wigan 1024-1063 (40 ports)
At approximately 16.50 Fujitsu Services were advised by TNS that they had been able to see
network traffic on logical access ports 1048 to 1063 but saw no traffic on logical access ports
1024 to 1047.
Fault Circumvention.
Commercial-in-Confidence Page: 7 of 11
©Copyright Fujitsu Services Limited 2004
POL00029277
POL00029277
Fujitsu Services Major Incident Report Ref: (CS/REP/182
Version: 0.2
Commercial-in-Confidence Date: 01-MAR-2004
Core Services Networks’ investigations showed that all 16 virtual circuits were active on the
Wigan DCS router. At 17:15 Networks decided to close the X25 interface at Wigan in order
to force the virtual circuits to operate from the Bootle.
The virtual circuits duly moved onto Bootle X25 router and TNS confirmed that 16 virtual
circuits were established and appeared okay. TNS were unable to monitor for network traffic
on the logical access ports as live DCS transactions were no longer being passed to Streamline
as it was outside trading hours.
High level checks were conducted by the SSC on-call and Networks on the circuits.
The SSC and Networks undertook pro-active monitoring of the debit card service from 08.00
on the Monday morning.
On Saturday 21* of February at 07.00 the X25 link from Wigan to TNS was reactivated. The
SSC and TNS monitored the reintroduction of this circuit and no operational issues were
identified or experienced. The Wigan X25 link has remained in operation since the 21“ of
February.
5.0 Incident Management
Date & I Avoidance, mitigation and Communication and Business Impact
time _I resolution activities escalation activities
14/2/04
12.06 SMC raise Powerhelp call E- I Branches experienced failures
0402140283 for the DCA when attempting to take
timeout alert. payments for transactions
using debit cards.
By 12.22 The SMC checked with
Fujitsu Services Networks
By 12.30 The SMC also informed the
TSD
At 12.30 The TSD raised incident E-
0402140303
By 12.35 The TSD paged the Post
Office Limited Duty Manager
(Dave Hulbert)
By 12.36 The SMC spoke with SSC on-
call. and transferred the
incident into PinICL by 12.43
At 12.38 The TSD informed
Streamline. Streamline
reported they were unaware of
a problem but agreed their
Network team would
investigate.
On-going The TSD undertook their
scheduled phone calls to
Commercial-in-Confidence Page: 8 of 11
©Copyright Fujitsu Services Limited 2004
Fujitsu Services
Major Incident Report
Commercial-in-Confidence
POL00029277
POL00029277
Ref: CS/REP/182
Version: 0.2
Date: 01-MAR-2004
Streamline.
At 14.26
The SMC advised the SMC
Duty Manager of the incident
At 14.55
The SMC advised the POA
Duty Manager (Tony Wicks)
of the incident.
At 15.00
The POA Duty Manager
(Tony Wicks) attempted to
phone Dave Hulbert
At 15.05
The POA Duty Manager
(Tony Wicks) used the paging
service to attempt to contact
Dave Hulbert on the POL
BCM pager.
15.00
Onwards
The POA Duty Manager
(Tony Wicks) became the
problem owner and liased
between the SSC on-call,
Fujitsu Services Networks and
the SMC.
At 16.50
Fujitsu Services Networks
contacted Streamline who
advise that incoming channels
to Streamline on logical
access numbers 1066 to
appeared to be okay but
nothing appeared to becoming
through on channels 1047 and
below.
1048
Atl7:15
Network’s decided to close the X25
interface at Wigan in order to force
the virtual circuits to operate from
the Bootle.
At 20.20
After various discussion with
the SSC and Networks Tony
Wicks reported the afternoon
events to the CS Operations
and Support Manager (Peter
Burden). The CS Operations
and Support Manager advised
the CS Director of the
incident.
Sunday
15/02/04
AM
Tony Wicks paged Dave
Hulbert and gave an overview
of the events after 15.00.
Sunday
PM
Tony Wicks arranged with the
SSC and Networks to
undertake pro-active
monitoring of the debit card
service from 08.00 on the
Monday morning when
Branches would start to use
Commercial-in-Confidence
©Copyright Fujitsu Services Limited 2004
Page: 9 of 11
POL00029277
POL00029277
Fujitsu Services Major Incident Report Ref: (CS/REP/182
Version: 0.2
Commercial-in-Confidence Date: 01-MAR-2004
the service.
6.0 Problem Management
The failure was detected and reported via the Horizon System Management tools, i.e. from the
12:02 bad alert regarding debit card Authorisation Agent timeouts.
The SMC checked with Networks by 12.22 and spoke with SSC on-call by 12.36. The SMC
also informed the TSD by 12.30. However Incident E-0402140283, which was raised as a ‘A’
priority confirms that the SMC did not contact the POA Duty Manager (Tony Wicks) until
14.55 and indicates that the SMC Duty Manager was also unaware of this incident until 14.26.
The TSD raised incident E-0402140303 at 12.30 and paged the Post Office Limited Duty
Manager (Dave Hulbert) at 12:35. At 12.38 the TSD informed Streamline. Streamline
reported they were unaware of a problem but agreed their Network team would investigate.
The SMC continued to receive alerts from the top level monitor.
The TSD cither spoke with or attempted to contact Streamline at the predefined scheduled
intervals. At 14.30 Streamline reported that they could see no issues with their network or
servers. At 14.50 Streamline contacted the TSD to report that one of their boxes was not
receiving transactions and that Streamline officially rejected ownership of the incident. At this
stage Streamline had been investigating for 2hours and 10 minutes before reporting seeing an
issue on one machine.
At 16.30 the SSC on-call confirmed, by analysing counters at failed outlets, that the problem
appears to have started around 12:00 where Agents were timing out because they had
exceeded the waiting for A2s from the MA.
The SSC on-call advised the SMC to contact Networks and also Streamline.
At 16.50 Fujitsu Services Networks contacted Streamline who advise that incoming channels
to Streamline on logical access numbers 1066 to 1048 appeared to be okay but nothing
appeared to becoming through on channels 1047 and below.
At 17:15 Network’s decided to close the X25 interface at Wigan in order to force the virtual
circuits to operate from the Bootle.
Problem 456 has been logged on the Problem Management database for this incident.
7.0 Corrective Actions
Incident/problem Action to be taken By By When Progress made
Issue Whom
This incident Investigate if process I Tony 22/03/04
highlighted improvements or Wicks/
limitations on monitoring tools can I Fujitsu
identifying where be implemented to Services
within the DCS assist in identifying
Commercial-in-Confidence
©Copyright Fujitsu Services Limited 2004
Page: 10 of 11
Fujitsu Services
Major Incident Report
Commercial-in-Confidence
POL00029277
POL00029277
Ref: CS/REP/182
Version: 0.2
Date: 01-MAR-2004
comms ‘supply
chain’ failures were
occurring.
where, in the comms
‘supply chain’,
failures are
occurring in a timely
manner.
Streamline took two I Review with Tony 05/03/04
hours to detect that I Streamline the Wicks
transaction traffic System Management
was not getting detection and report
through to one of its I tools.
machines
The SMC did not SMC staff to be Tan Cooley I Completed I Completed
escalate the incident
to the SMC or POA
Duty Managers in a
timely manner.
reminded that ‘A’
priority incidents are
to be escalated as
per predefined
process.
Commercial-in-Confidence
©Copyright Fujitsu Services Limited 2004
Page: 11 of 11