POL00029460
POL00029460
Fe)
FUJITSU Ref POL:Q17472700 FJ: MI 5103991
Control Version: 0.1 Status: Draft
FUJITSU SERVICES
Owner Steve Bansal
MAJOR INCIDENT REPORT
INCIDENT DETAILS
Post Office Card Account transactions were unable to complete and a
number (approximately 1%) of AP Transactions were affected following the
Incident St
dsctel Sle ad AP Ref Data Monthly update being enabled on Wednesday 1°‘ February.
Between 08:00 and 11:15 on Wednesday 1 February all Post Office Card
Account transactions were unable to complete and a number of AP
Service Impacted u "
Transactions were matched to an incorrect client
Post Office Card Account transactions and some AP Transactions were
affected from 08:00 on Wednesday " February. Due to a problem with a
scheduled Reference Data update of monthly AP Ref Data for token IDs.
This resulted in the two token IDs being incorrectly mapped, 1980 and
1981 (Highland Council).
Pxecutive surmmary, To resolve the issue Fujitsu identified the fault, produced correcting data in
conjunction with the Post Office Reference Data team, and tested and
delivered an update to the AP Ref Data.
Corrected AP Ref Data was available to Branches from 11:00. Post Office
Card Account transactions and the affected AP transactions resumed from
approximately 11:15. By 12:30 99% of branches had downloaded the
corrected Ref Data.
Post Office Ltd provided Reference Data for the monthly AP Ref Data
update which included the two Token IDs 1980 and 1981. The Fujitsu
Reference Data Team completed a technical validation against the file
provided, which was subsequently verified by Post Office Ltd.
This was a second iteration of the data as a problem had been identified
with the initial data set that had been supplied during validation of the
token data. The CTO update was in effect a primary package with an
incremental update.
Following validation of the data by both Reference Data teams the update
was loaded on 26" January to the live environment with an enablement
date of the 1*' February.
The definition of a magnetic card or barcode within Reference Data
5S includes a number of elements. These elements are essential in defining
Aualyais ol crotlent the token mask which is used to match the set of Reference Data tokens
(cards and barcodes) against each swiped card or barcodeso as to
determine which client the token applies to, and also what counter
functionality is to be invoked.
In the case of these tokens defined for 1°' February, an element was
missing from one version of the Reference Data for an AP client. As a
consequence, any scanned magnetic card or barcode that matched with
the rogue token matched to Highland Council as the client instead of being
matched to the correct client for that token. This led to an AP transaction
being launched for the client associated with the rogue token. Both AP and
ETOP clients were affected as in both cases their transactions are
launched via a card swipe or barcode read.
Evidence to date suggests that Post Office Card Account cards were the
only banking cards affected
02/02/2012
POL00029460
POL00029460
fe)
FUJITSU Ref I POL:Q17472700 FJ: MI5103991
Control Version: 0.1 Status: Draft
FUJITSU SERVICES Owner Steve Bansal
Activity to date has been focussed on resolving the issue and minimising
the impact to Post Office Ltd and their clients.
Root Cause Analysis
However the focus of the parallel investigation is the system handling of
the incremental package. In addition as the update was validated any
variance of the verification rigs to the live environment.
Fujitsu identified the mismatch on Ref Data and produced a corrected set
of Ref Data to load into the BRDB and resolve the issue.
Corrective Actions As authorised by Post Office Ltd, Fujitsu quarantined the 6831
transactions which were affected to contain the situation and limit the
impact to POLSAP and Credence.
See below for further actions
Date Of Incident Time of Service Outage Time Service Restored
01/02/2012 08:00 — 11:15 11:15 POCA commence
11:30 45% of counters have the fix
12:00 90% of counters have the fix
12:30 99% of counters have the fix
RELATED REFERENCES
Master Incident Problem Record No of Linked Incidents
Previous Occurrence MTTR Action required to Close
INCIDENT TIMELINE
Date Time Activity
01/02/12 08:00 Start of Incident
08:06 POA DM paged
08:06 Master call TfS 5104011
08:08 Fujitsu contact POL (Antonio Jamasb)
08:13 IVR activated ( High call volumes)
02/02/2012
POL00029460
POL00029460
fe)
FUJITSU Ref I POL:Q17472700 FE: MI5103991
Control Version: 0.1 Status: Draft
FUJITSU SERVICES Owner Steve Bansal
INCIDENT TIMELINE
Time Activity
08:14 POL log call, POL — Q17472700 — Next agreed update 08:30
08:30 Tech Bridge open
08:33 Post Office Service Desk informed of Tech bridge and current call volumes
(Antonio Jamasb)
08:46 Post Office Service Desk informed of current call volumes (Maureen)
08:55 SMS sent and Post Office Service Desk informed of current call volumes mma)
09:15 Confirmed normal traffic for A&L and Link
09:20 Service Bridge running
09:21 SMS sent and Post Office Service Desk informed of current call volumes (Andrew)
09:40 Investigating potential mismatch on Ref Data
09:40 SMS sent and Post Office Service Desk informed of current call volumes (Andrew)
10:00 Identified incorrect Ref Data Tokens released on 26/01 with activation date of
01/02, correction is being progressed
10:00 SMS sent and Post Office Service Desk informed of current call volumes (Tony)
10:19 SMS sent and Post Office Service Desk informed of current call volumes (Julie)
10:20 Fix identified and awaiting POL approval for deployment to Live environment
10:30 POL approve deployment to Live environment
10:32 SMS sent and Post Office Service Desk informed of current call voumes (David)
10:59 SMS sent and Post Office Service Desk informed of current call volumes David),
next update agreed 11:30
11:00 Successful deployment of package 2951 into BRDB, Branches now need to pole
the BRDB to check for new Ref Data and then process new data at branch level
11:15 Post Office Card Account transactions commence
11:30 Post Office Card Account transactions increasing, as approximately 45% of
branches have processed new Ref Data
11:30 SMS sent and Post Office Service Desk informed of current call volumes (Julie),
next update agreed 12:00
11:39 No calls queuing
11:48 IVR removed
12:00 Post Office Card Account transactions higher then monthly average, as
approximately 90% of branches have processed new Ref Data
12:04 SMS sent and Post Office Service Desk informed of current call volumes (Andrew),
next update agreed 12:30
12:30 Post Office Card Account transactions higher then monthly average, as
approximately 99% of branches have processed new Ref Data
12:34 SMS sent and Post Office Service Desk informed of current call volumes (Tony)
16:00 Joint conference call- discuss impact on POLSAP finance and Credence
02/02/2012
Fe)
FUJITSU
POL00029460
POL00029460
Ref I POL:Q17472700 FJ: MI5103991
Control Version: 0.1 Status: Draft
FUJITSU SERVICES Owner Steve Bansal
IMPACT OF INCIDENT
There were two specific service issues that affected 3009 branches;
. : e — Inability to perform card transactions using Post Office Card Accounts:
Overview of Service across the entire branch estate
Impact ¢ ETOP and Automatic Payments transactions mismatched and posted in
error to the wrong Automatic Payment client, currently believed to impact
up to 3 ETOP and 65 AP Clients
02/02/2012
Fe)
FUJITSU
FUJITSU SERVICES
Ref
Control
Owner
POL:Q17472700
Version: 0.1
Steve Bansal
FJ: MI5103991
Status: Draft
POL00029460
POL00029460
Overview of
Business Impact
The graph below shows the commencement and steady increase to above
monthly average flow rate of the Post Office Card Account transaction levels.
NBX Statistics - Transactions
The table below shows the flow of all Network Banking transactions
Period start
07:00:00
07:15:00
07:30:00
07:45:00
08:00:00
08:15:00
08:30:00
08:45:00
09:00:00
09:15:00
09:30:00
09:45:00
10:00:00
10:15:00
10:30:00
10:45:00
11:00:00
11:15:00
11:30:00
11:45:00
12:00:00
12:15:00
12:30:00
12:45:00
13:00:00
APS
“13
33
40
53
235
370
1606
3069
19361
20509
19084
19441
19674
20537
20575,
20975
21538
22056
24547
25016
24263
24310
23189
22022
19525
BURE
AU
20
38
80
87
253
554
543,
851
948
1020
1077
1036
1010
970
974
1103
1159
1054
1044
1078
1159
1257
1558
Des
13
17
24
48
80
415
719
5181
5332
4764
4675
4850
4934
5110
5337
5537
5730
6143
6431
6873
7529
7430
7322
6606
EPOSS
971
1285
1713.
3103
5316
10438
22654
43491
127519
136748
130645,
133330
135483
141501
145445
151460
153809
163767
180032
185397
182591
188120
176901
169495
151419
eTU
17
9
44
28
31
610
558
610
624
623
637
787
837
1300
1439
1391
1284
1202
1152
944
NWB
(AL)
"]
20
19
28
75
108
216
394
1283
1508
1493
1427
1529
1536
1514
1591
1537
1688
1885
1905
1974
1734
1595
1479
1186
NWB
(CAPO)
4550
18397
22706
22815
21822
19577
17219
14302
NWB
(LINK)
35
49
93
91
176
256
805
1406
7260
6067
5255
5163
5142
5257
5387
5260
5411
5723
6983
6955
6900
6974
6504
5693
4707
02/02/2012
POL00029460
POL00029460
fe)
FUJITSU Ref I POL:Q17472700 FE: MI5103991
Control Version: 0.1 Status: Draft
FUJITSU SERVICES Owner Steve Bansal
ACTIONS
s Comments /
Actions Owner Target Date Updates Peak
Fujitsu (Pete Completed MSC
Fujitsu to load corrected Ref Data Thompson) 01/02/12 043J0332578-02
From joint meeting 01/02/12 16:00: Fuji Completed
jitsu
Fujitsu to provide Post Office Ltd with a (Graham 02/02/12
list of AP Clients, quantities of Welsh)
transactions, and values
Fujitsu to expire incomplete Ref Data to Completed MSC
ensure that all counters which are Fujitsu (Pete 04340332568
currently switched off will automatically be Thom eo 01/02/12
forced to download the new version of the Ps
Ref Data.
Fujitsu to quarantine data Fujitsu (Pete 102/12 Completed MSC
Thompson) 043J033257
From joint meeting 02/02/12 09:30: Completed
Fujitsu to provide the following to the Post
Office Service Desk .
Fujitsu
« AP Client list (Graham 02/02/12
. List of complete list of branches Welsh)
affected by this issue
« — List of ETOP branches
Resolve data matching issues Fujitsu and
Post Office Ltd
Reference 10/02/12
Data teams
Further investigation to establish the root I Fujitsu (Steve 08/02/12
cause of the incident Bansal)
CLOSURE
oo pane COUR Details of Fix Applied Comments / Updates
eferences
Date Closed Fujitsu Approval POL Approval
02/02/2012