Fujitsu Services
POL00029293
POL00029293
Major Incident Report Ref: CS/REP/189
Version: 1.0
COMMERCIAL-IN-CONFIDENCE Date: 24th-Aug-2004
Document Title:
Document Type:
Release:
Abstract:
Document Status:
Originators & Dept:
Contributors:
Internal Distribution:
External Distribution:
Approval Authorities
Major Incident Report Covering APS Stranded Transactions &
APS Reconciliation’s between the period 10° July — 29" July 04
Report
S60
Report covering APS Transactional Processing following S60
Data centre upgrade weekend.
Review
Tan Daniel - POA Customer Service
Richard Hicks, Nial Finnigan, Alan D’Alvarez Andrew Gibson,
Mic Peach, Roy Birkenshaw, Garrett Simpson
Distribution for Approval, Martin Riddell, Richard Brunskill,
Carl Marx, Andy Gibson, Reg Barton
Post Office Ltd Library plus reviewers
Name
Position
Signature
Date
Dave Baldwin
FS CS Service Director
Dave Hulbert
Operations Service Manager,
Post Office Ltd.
© 2002 Fujitsu Services
COMMERCIAL-IN-CONFIDENCE
Page: 1 of 10
POL00029293
POL00029293
Fujitsu Services Major Incident Report Ref: CS/REP/189
Version: 1.0
COMMERCIAL-IN-CONFIDENCE Date: 24th-Aug-2004
0.0 Document Control
0.1 Document History
Version No. I Date Reason for Issue Associated
CP/PinICL
0.1 ® Aug 04 Initial Draft
1.0 24 Aug 04 For Approval
0.2 Review Details
Review Comments by : I 6" September
Review Comments to: I Author
Mandatory Review Authority Name
Post Office Ltd Dave Hulbert
FS CS Director Martin Riddell
FS CS Business Support Management Manager I Richard Brunskill
FS CS Infrastructure and Availability Manager I Carl Marx
FS CS Service Introduction Manager Reg Barton
Optional Review / Issued for Information
( * ) = Reviewers that returned comments
0.3. Associated Documents
Reference Version I Date Title Source
Unless a specific version is referred to above, reference should be made to the current
approved versions of the documents.
© 2002 Fujitsu Services
COMMERCIAL-IN-CONFIDENCE
Page: 2 of 10
Fujitsu Services Major Incident Report
COMMERCIAL-IN-CONFIDENCE
POL00029293
POL00029293
Ref: CS/REP/189
Version: 1.0
Date: 24th-Aug-2004
0.4 Abbreviations/Definitions
Abbreviation Definition
AP Automated Payments
CTS Client Transaction Summary
CFM1 Core Services Unix
DM Duty Manager
APS Automated Payments Service
LST Live System Testing
PM Problem Manager
PMDB Problem Management Database
PO Post Office
POA Post Office Account
POL Post Office Limited
SMC Systems Management Centre
0.5 Changes in this Version
Version Changes
0.1 This is the first draft
1.0 For Approval
0.6 Changes Expected
Changes
© 2002 Fujitsu Services COMMERCIAL-IN-CONFIDENCE
Page: 3 of 10
POL00029293
POL00029293
Fujitsu Services Major Incident Report Ref: CS/REP/189
Version: 1.0
COMMERCIAL-IN-CONFIDENCE Date: 24th-Aug-2004
0.7 Table of Contents
1.0 I INTRODUCTION.....
2.0
3.0
3.0
4.0...
41 SYMPTOMS AND BUSINESS IMPACT.
4.1.1 Symptoms as seen by Branches & Post Office Lt
4.1.2. CTS File Impact
4.1.3 AP stranded Transaction 1p
4.2 DETAILED EXPLANATION OF THE INCIDEN
5.0 INCIDENT MANAGEMENT.
6.0 PROBLEM MANAGEMEN
7.0 CORRECTIVE ACTIONS...
© 2002 Fujitsu Services COMMERCIAL-IN-CONFIDENCE Page: 4 of 10
POL00029293
POL00029293
Fujitsu Services Major Incident Report Ref: CS/REP/189
Version: 1.0
COMMERCIAL-IN-CONFIDENCE Date: 24th-Aug-2004
1.0 Introduction
This document reports on the issues of the failure of Fujitsu Services to Deliver APS Client
data files and CTS reconciliation’s issues.
This report covers:
- How the problems came to light
- The impact on the branch service
- The investigation
- The resolution
- The root cause
- Actions and recommendations to prevent recurrence
2.0 Scope
The scope of this report covers the failures of Fujitsu Services to Deliver AP client data to a
number of AP clients, those of which do not receive files on all 7 days of the week, between
the period 10" July — 15" July 04. Attached at Annex A provides a complete list of clients
It also covers the failure to produce automated APS reconciliation reporting accurately in the
form of the daily CTS file produced, between 10" July 04 — 29" July 04. It should be noted
that whilst automated process was non operational manual reporting was being covered daily.
3.0 Management Summary
The problem was initially discovered following the weekend of the Data Centre upgrade 9" —
10th July 04. Following the overnight schedule on the 11" July 04, at approximately 09:30
Monday 12" July 04, POL informed fujitsu service’s, service Management that there was
possible errors in the CTS file. It was suggested that this file was considerably less value than
would have been normally expected the approximate value of transactions expected being
reduced by up to £30 million.
Throughout these problems the correct escalation routes — including to POL — were followed
Full services were resumed on 15" July 04, following successful development and testing of
the required scripts prior to release into the live estate for all AP client stranded data.
The fix for the CTS reconciliation’s problem was delivered to live on 29" July 04.
5.0
© 2002 Fujitsu Services COMMERCIAL-IN-CONFIDENCE Page: 5 of 10
POL00029293
POL00029293
Fujitsu Services Major Incident Report Ref: CS/REP/189
1.0
COMMERCIAL-IN-CONFIDENCE Date: 24th-Aug-2004
5.1 Symptoms and Business Impact
5.1.1 Symptoms as seen by Branches & Post Office Ltd.
5.1.2 CTS File Impact
The CTS file had been reported as being considerably under value than expected. The effect
of the reconciliation reports being incorrect would have impacted POL’s financial settlement
with their AP clients. Volumes of client enquiries to POL regarding the missing data and
subsequent client settlement, meant that POL had to settle a number of client account on
previous trend data. The effort required to settle client accounts and the subsequent
adjustments once the true figures had been obtained caused a considerably amount of extra
work involved.
4.1.3 AP stranded Transactions Impact
The potential impact of the incident where AP transactions were not migrated from Dynix to
Solaris during the S60 Data Centre upgrade was high. All clients who do not receive files 7
days a week were impacted by this, 5 -6 day a week clients did not received data harvested
Friday 9th and Saturday 10th July and 6 day a week clients did not receive data harvested
Saturday 10thJuly.
The impact is both visible to the AP Clients and their customers with no record of their
transaction being received by the AP Client, this could cause red bills, customer
disconnections for utility companies and BT, penalty charges or court action due to late
payments, cancellation of insurance for AON and would be very visible to the customer.
Branches would be receiving complaints from customers who had paid but their account had
not been debited and the Helpdesk would have been receiving calls both at Post Office and the
AP Clients.
5.2 Detailed explanation of the incident
Clients affected are those that receive files on a 5 day or 6 day basis. These clients receive a
‘mop up’ file on a Sunday, which includes Friday and or Saturdays files plus any from the
Sunday. The clients had been sent the Sunday file but this file only contained Sunday data and
did not include the data from Friday and/or Saturday.
This has been diagnosed as affecting those Post Office clients whose transaction data, put
through the system on Friday prior to the Solaris port, was scheduled to be sent on
Monday. As per normal operation, these transactions were captured in 'Pass Through’ files,
generated by Dynix. There was a migration step missed in the port to Solaris and these
transactions were not migrated into the Pass Through tables in the Solaris database. The net
effect being that any transactions generated on the Dynix database not processed that evening
for transfer to the clients are still being held in the Pass Through files.
« There were 581,481 transactions in the pass through files that were not processed.
These include Reversed/Reversal pairs that should not be sent to clients.
« There are 578,091 transactions not placed into client transmission files.
© 2002 Fujitsu Services COMMERCIAL-IN-CONFIDENCE Page: 6 of 10
POL00029293
POL00029293
Fujitsu Services Major Incident Report Ref: CS/REP/189
Version: 1.0
COMMERCIAL-IN-CONFIDENCE Date: 24th-Aug-2004
« These transactions had a value of £22,421,647.86
The difference between these figures and those supplied _on Monday 12" July 04 is explained
by the inclusion of Saturday and Sunday transactions that were sent to clients. These figures
are shown on reconciliation report APSS2133 in the reconciliation error line for processing
dates 11/07/2004 and 12/07/2004.
Note that the reconciliation reports include Reversed/Reversal pairs .
Value (£)
36,603,932.60 Monday figures as supplied
14,055,741.47 Minus Sunday recon error
126,543.27 Minus Monday recon error
22,421,647.86 Total
© 2002 Fujitsu Services COMMERCIAL-IN-CONFIDENCE Page: 7 of 10
Fujitsu Services
POL00029293
POL00029293
Major Incident Report Ref: CS/REP/189
Version: 1.0
COMMERCIAL-IN-CONFIDENCE Date: 24th-Aug-2004
6.0 Incident Management
Date & I Avoidance, mitigation and I Communication and escalation I Business Impact
time resolution activities activities
INCIDENT
12/07/04 I Alerted by POL that the CTS file I Not aware of any problem at this I POL could not settle
was potentially Light by some £30 I point investigations started toI with AP clients
09:30 : 1
million determine what had gone wrong. accurately.
11:59 Confirmed by SSC that a number of I Advised POL & FJS Senior business
TX’s had not been passed to AP Managers.
clients. Investigations still
continuing.
POL update confirming that a
number of clients had reported
lower than average TX’s.
12:10 Raised PM on PMDB PM 0000503 I Advised POL of cross domain
problem
14:26 Updated POL with confirmation I POL & FJS Business Manager made
that a number of AP clients had I aware.
not received data for either
Friday's / Saturday's TX’s
14:30 Advised POL Problem I Advised FJS business Managers
Management that Problem now
Identified, as affecting all AP
clients who do not receive files 7
days of the week.
13/07/04 I Conference call with POL & FJS, I Advised FJS business Managers
10:30 & I ivequent action Ding taken
14:00 4 s
16:00 At POL request the problem from
herein was Managed via PM.
Hitsr-Service: ECOAMEREE
E=AAN-CONFIDENCE Page—tott
POL00029293
POL00029293
Fujitsu Services Major Incident Report Ref: CS/REP/189
Version: 1.0
COMMERCIAL-IN-CONFIDENCE Date: 24th-Aug-2004
It should be noted that, additionally, senior management in POA and, onwards, senior
management in POL were advised of these issues during the day.
7.0 Problem Management
The problem (PM0000503) was raised on 12" July 04 at 12:01, the problem was well
managed and the appropriate procedures were followed. POL were kept fully informed as to
the Root cause, actions being taken & the expected fix time, daily conference calls were set-up
to monitor the progress and overall understanding of the problem, theses were then handed
back over to formal problem Management procedures on Tuesday 13" July 04 at 16:00. Daily
updates were provided to POL and Fujitsu Business Managers until the problem had been
rectified.
The script (fix) for the stranded APS client data was delivered into live the evening of the 14"
July 04, for overnight processing 15" July 04 following successful development and LST
testing. The problem is still active on the PMDB at the moment following a period of
monitoring, the forecast closure date being 4th Aug 04.
APS Client CTS file errors, the above problem reference number exist for both calls. POL
were kept fully informed as to the Root cause, actions being taken & the expected fix time
scale. The final fix for this was delivered to live on 29" July 04, after extensive development
and testing, this being more prolonged due to the extraordinarily amount of data that had to be
checked.
8.0 Corrective Actions
Incident/problem Action to be taken By By Progress made
Issue Whom When
Migration Issues as I Lessons which have
such a one off been learnt, have
operation. highlighted the need
for additional peer
reviews of code
changes & and
associated testing
© 2002 Fujitsu Services COMMERCIAL-IN-CONFIDENCE Page: 9 of 10
Fujitsu Services
Major Incident Report
COMMERCIAL-IN-CONFIDENCE
POL00029293
POL00029293
Ref: CS/REP/189
Version: 1.0
Date: 24th-Aug-2004
© 2002 Fujitsu Services
COMMERCIAL-IN-CONFIDENCE
Page: 10 of 10