POL00130164
POL00130164
Strictly Private and Confidential; Subject to Legal Privilege
Spot Review 1 - Response-te-Spot Report SROG4
STATUS: DRAFT
Author:-.Gareth LJenkins
Date: 02/04/201346:27:00
Version: 2
___Bond Pearce
Last edit date: 17/04/2013
APPROVED BY DATE
POL legal
Fujitsu
Simon Baker
FINAL SUBMISSION TO BE IN PDF FORMAT TO ENSURE ALL META-DATA
ABOUT PREVIOUS DRAFTS IS REMOVED
Strictly Private and Confidential; Subject to Legal Privilege
Page I of 4
POL00130164
POL00130164
Strictly Private and Confidential; Subject to Legal Privilege
Horizon Spot Review - Response
ROL; Debit Cards — Cash Withdrawals and GIRO Payment
1, Executive Summary
Thi Ri
automatic reversals of transactions when Horizon is unable to connect to the Data Centre.
The analysis below shows that Horizon does provide adequate notification.
Further, in the particular case raised in th Review, the r use of the diffi
suffered by the SPMR_ was his failure to follow the on-screen _and_printed instructions
given by Horizon. Post Office Limited is confident that the SPMR_ knew that some
transactions hi: m matically reversed
e The branch h: n_sufferins nnectivity i in the run he incident in
question.
¢ When the transactions in question first failed to be processed (because Horizon
could_not_get_a_response from the Data Centre), Horizon asked the SPMR
whether he wished to cancel or retry the transactions. The SPMR opted to retry
the transactions.
© When ran ions fail in, the SPMR he transacti
Horizon_then automatically disconnected and _printed_a_“disconnect" receipt that.
shi he_transactions that ha 0 matically rever: A_sampl
"disconnect" receipt is included the appendix to this response.
A standard customer receipt was not produced — this would tell the SPMR that
the full transaction had not proceeded,
Following the disconnect, the SPMR was required to log back on to Horizon and
duly did so,
Following .the_log_on,,and_as_part_of the standard _recoyery_process, Horizon.
printed a "recovery" receipt which again showed the transactions that had_been
reversed and those that had been recovered. A sample "recovery" receipt_is
included in the appendix to this response,
2 4-Introduction
This spot review relates to an issue raised by John Armstrong the SPMR in Lepton Branch
(FAD Code 1913204) relating specifically to transactions carried out on Horizon Online on 4"
October 2012. The issue is headed “Debit Cards — Cash Withdrawals and GIRO payments”.
investigate-the-transactions—on-31/1/13.— As_this—was—within—6—months—of_the-
transaction, then—the—data—was-still_available—in—_the-support-database—and—a-
questi ided-whick Lthis inf ion.
Strictly Private and Confidential; Subject to Legal Privilege
Page 2 of 4
POL00130164
POL00130164
Strictly Private and Confidential; Subject to Legal Privilege
This report provides information as to what was alleged by the SPMR (see section 2) and a
detailed analysis of what actually occurred as shown in the system logs (section 3). Section 4
then describes how recovery operates on Horizon and Section 5 identifies those points in the
report which are not supported by the Logs. Finally section 6 prevides—a-response—by_
addresses the FSC-of Pest-Offiee-Ltd-as-_to-why-it-was-deemed-unnecessary-to-earry-
out-the thorough investigation_that-has-now—taken-placequestion of access to raw_
transaction data.
ther °. —These-are—-not-nermally—provided_in—response—to—queries—or-
examined. -However-in-this-ease+ they: were passed to Second Sight and: } they-have-
b edt id detailed
P Lid
3. 2+The SPMR’s view of what happened
The following is an extract from the Spot Report saying what the SPMR says happened:
The SPMR reports that there were intermittent internet connectivity problems (also
reported to Chesterfield) on 4 October 2012. Online payments and withdrawal
transactions were sometimes successful but also failed on occasions. It is possible that
Horizon was partially operating through its back-up (mobile phone) connection. Some
card payments had to be attempted two or three times before being accepted. At
approximately 10:32 a customer tried to pay his £76.09 BT phone bill with his LTSB
card but was not successful. The customer then withdrew £80.00 cash and used this to
pay the phone bill. The SPMR stamped the customer’s phone bill to evidence receipt
of the cash, returning change of £3.91. Several weeks later, the customer returned from
holiday to find his phone had been cut off due to non-payment. The SPMR’s
examination of the Transaction Log showed that all components of the transaction had
been reversed. The SPMR did not initiate those reversals nor did he receive any
reversal notifications. The SPMR raised this as an issue with Chesterfield but was told
that due to cost issues Horizon transaction data could not be requested. It was implied
that the SPMR had stolen the money and he was told to make good the shortage. This
meant that 2 people had paid the phone bill (the customer, who handed cash to the
SPMR and the SPMR on instructions from Chesterfield). The SPMR was informed
that he should have a surplus of £76.09 due to the reversal of the transactions. The
SPMR disputes this conclusion, but the more important issue here is the automated,
unreported, reversal of the transactions.
From this information, the following key issues have been identified:
2 When Horizon cannot get_a response from the Data Centre, are automatic
ransaction reve notifi PI ¥4
e Why is r: i i PMRs?_It is n
n Post Office's procedures and pr.
4, 3-What the System Logs show
Note that the system logs show all times in GMT rather than local time. On 4" October 2012,
GMT was I hour behind Local Time (ie BST). The times quoted in this review relate to the
system logs. Therefore the mention of 10:32 by the SPMR above relates to 09:32 in the logs.
4h ively, Leould go-through-the description bek belt Htimes-to- BST.
i it is-simple Hess: )-to-keep-the-deseription-elos.
My feeling was thatitis simpler (and less p to-keey scrip x that]
Strictly Private and Confidential; Subject to Legal Privilege
Page 3 of 4
POL00130164
POL00130164
Strictly Private and Confidential; Subject to Legal Privilege
There do appear to be 2 cases on 4 October where the system had a forced Log Out that
resulted in a recovery Log On being required. This supports the statement above: “The SPMR
reports that there were intermittent internet connectivity problems (also reported to
Chesterfield) on 4 October 2012”. The two “Recovery Log Ons” occurred at 08:51:40 (when
no recovery was required) and at 09:37:20 when recovery was required as will be described
later in this report.
The following table looks at the number of online requests for either Banking or Credit / Debit
Card Payments that appear to have timed out:
Tot
Date al
04/10/201
2 13
05/10/201
2 4
08/10/201
2 11
10/10/201
2 2
11/10/201
2 2
16/10/201
2 1
17/10/201
2 2
18/10/201
2 2
19/10/201
2 3
22/10/201
2 1
23/10/201
2 1
25/10/201
2 2
Grand
Total 44
This supports the comment regarding intermittent connectivity problems on 4" October. I note
that there were similar problems on 8" October.
Fean-seeThere are 4 examples prior to 09:30 where either a Banking withdrawal or a Credit
/ Debit card payment initially failed and was successful on the second attempt. There was also
one example where there were two failures for a card and presumably the customer or the
SMPR gave up. This supports the statements that “Online payments and withdrawal
transactions were sometimes successful but also failed on occasions.” and_that “Some card
payments had to be attempted two or three times before being accepted”,
The raw logs do have statistics regarding times taken to connect to the Data Centre and also an
indication of the type of Comms currently in use. From these it can be seen that the Branch
normally operates using ADSL, but at the time of the failure that is being examined it appears
to be using a mixture of O2G and O3G (ie mobile networks) presumably due to a failure of the
main ADSL connection. This supports the statement_that “It is possible that Horizon was
partially operating through its back-up (mobile phone) connection-”. This may have been
visible to the user as a slower than normal response time.
Strictly Private and Confidential; Subject to Legal Privilege
Page 4 of 4
POL00130164
POL00130164
Strictly Private and Confidential; Subject to Legal Privilege
The key transactions are those described as occurring at 10:32 (ie 09:32 GMT).
This analysis starts at 09:26 and shows the sequence of baskets processed between that time
and 09:40.
1.
Nv
09:26:30: Session 537799 contained two transactions: A Card Account Withdrawal
(Withdraw Limit) for £271.54 and a corresponding Cash Settlement.
09:27:34: Session 537800 contained three transactions: A failed Card Account
Withdrawal (Withdraw Limit) immediately (09:28:13) followed by a successful Card
Account Withdrawal for £141.80 using the same card and a corresponding Cash
settlement.
09:29:27: Session 537801 contained a single transaction: A failed Visa Debit card
payment. This payment had been requested for £141 and had failed due to no response
having been received by the counter within the timeout period (33 seconds). Clearly an
attempt had been made to purchase something or pay for a service for £141, but when
the Debit card payment failed, the original transaction was voided and the basket
completed.
Th
Jue-of £14--is-only-visible-in-th k
09:31:56: Session 537802 contained 2 transactions. A Halifax Current Account
Withdrawal for £200 followed by the corresponding cash settlement. It would appear
that the card used here was the same as the one used in the previous session when the
Debit Card payment failed.
09:32:52: Session 537803 contained 3 transactions. A bill payment to BT for £76.09
followed by a Cash Withdrawal for £80 using a Lloyds TSB card and £3.91 cash for
the difference.
09:37:19: User JAROOI Logged On again
09:37:44: Session 537805 generated by the system as part of the Recovery that takes
place during Log On and contains 3 transactions. The first 2 are the Reversals for the
BT Bill Payment and Cash transactions in session 537803, and the 3“ is a Cash
balancing transaction for £80 to correspond to the £80 cash withdrawal which should
have been treated as successful at the time of failure. This is why “The SPMR’s
examination of the Transaction Log showed that all components of the transaction had
been reversed.”
It should be noted that the statement is not quite correct. The Banking Withdrawal for
£80 has not been reversed.
09:40:19: Session 537806 contained 2 transactions. A Card Account Withdrawal
(Withdraw Limit) for £229.72 and a corresponding Cash Settlement.
It should be noted that there was no Session 537804. In-particular-this-was-neted-by-the
SPMR-in-atetterte-FSC implying that this-was-afault-in-the-system.—There are a
number of circumstances under which there are gaps in Session Sequence Numbers and in
general they are not expected to be contiguous. In fact they are based on an underlying Journal
Sequence Number which are contiguous and relate to any record that has been audited.
In this case the “missing” number relates to the Journal Sequence Number used in the Log On
Request, but there are a number of other circumstances that can result in a Journal Sequence
Number being used where there is no corresponding Basket.
Looking at the statistics recorded with the Recovery basket at Point 7 above, it can be seen that
there were a number of issues during session 537803:
Th
tatisties-are-only-visible-in-th Togs.
Strictly Private and Confidential; Subject to Legal Privilege
Page 5 of 4
POL00130164
POL00130164
Strictly Private and Confidential; Subject to Legal Privilege
a. The Authorisation for the Cash Withdrawal was successful and was done on a 3G
comms Connection.
b. The subsequent attempt to update the Recovery information in the basket after
completing the Banking Transaction failed due to a timeout on a 2G comms connection
c. There are then 4 attempts (at roughly 45 second intervals) to store the completed
basket to the Data Centre. The first 2 use a connection type of 2G and the other 2 use
a 3G comms connection. ‘FheyFrom the branches records, they are all marked as
having failed. (Fhough-elearly—one-of the-attempts—had_resulted*in—data-
being successfully saved in-the Data Centre*,)
From the Data Centre's perspective, one of the attempts did result* in data being
successfully saved in the Data Centre* but, due_to_the connectivity issues, the
ranch did not receiv nfirmation from the D: Centre. The branch will
therefore record this as a failure.
Moving on to the end of the day the following Cash Declarations were made:
A. At 16:31:27 a Declaration was made for £22,160.54 followed by a variance check
which indicated a discrepancy (loss) of £1,237.16.
B. At 16:32:46 a second cash Declaration was made for £23,460.54 followed by a
variance check which indicated a discrepancy (gain) of £64.84.
I
Looking forwards, the following variance check discrepancies were recorded:
Variance Check Loss or
Date Discrepancy Gain
04/10/201
2 £62.84 Gain
05/10/201
2 £66.15 Gain
06/10/201
2 £76.98 Gain
08/10/201
2 £71.91 Gain
09/10/201
2 £69.05 Gain
10/10/201
2 £63.99 Loss
The Stock Unit was Balanced and rolled over from Balance Period 3 into Balance period 4 on
10" October and the Discrepancy committed to the accounts. (There was also a £37.75
discrepancy Gain on stamps at the same time.)
5, 4-Explanation of Recovery
The fact that a Log On (and Recovery) occurred at point 6 above indicates that there must have
been a failure just before that point and the User would have been informed of a Forced Log
Off. The fact that Recovery reversed most of the last Session recorded prior to the recovery
indicates that the following sequence of events occurred. This is confirmed by the statistics
described above at point c in section 3 above.
The user must have been aware that there was a problem in this circumstance. What they
would have observed was the following:
Strictly Private and Confidential; Subject to Legal Privilege
Page 6 of 4
POL00130164
POL00130164
Strictly Private and Confidential; Subject to Legal Privilege
1. Having completed the Bill Payment and Cash Withdrawal, the User would have either
selected “Settle” or “Fast Cash”. If Settle was selected then they would again have
selected either “Cash” (and keyed in the amount) or selected Fast Cash.
2. This would have completed the Basket and attempted to save the basket to the Data
Centre.
3. Following a failure of the first attempt, the system would automatically carry out a
retry and attempt to save the basket to the Data Centre again.
4. Following the failure of the second attempt, a message would have been displayed to
the User informing them that there was a failure to contact the Data Centre and did
they wish to Retry or Cancel.
5. The fact that there were 4 attempts to contact the Data Centre, indicates that the User
must have selected Retry and so the system would have made a 3“ attempt to save the
basket to the Data Centre.
6. Following a failure of the third attempt, the system would automatically carry out a
retry and attempt to save the basket to the Data Centre yet again.
7. Following the failure of the fourth attempt, a message would have been displayed to
the User again informing them that there was a failure to contact the Data Centre and
did they wish to Retry or Cancel.
8. The fact that there were only 4 attempts to contact the Data Centre indicates that the
user must have selected Cancel this time. This would have resulted in a Forced Log
Out. What-thisThis means-is:
a. System- would= have—reealeulated—the— basket— -and—would—have-
4 ble-transacti thi the BE_Bill
and the Cash “change”Horizan would cancel those transactions that
could. e-cancelled.—In-this-case- the BI- Bill and_the Cash “change” could_
1 ran: ni S til
basket ones and in this instance the basket had failed.
b. The cash withdrawal transaction for £80 could not be cancelled. Prior to
he_disconn Horizon_had_alr: mM he_customer's bank
confirm that_a_cash withdrawal _could_be_ made from the customer's
account, The customer's bank had_therefore already _registered_the
withdrawal from the customer's account and this transaction could not be
cancelled.
bet-would hav 1-thet indicati hat-£80
should -be-handed-to-the-eustomerHorizon would then re-calculate the.
basket showing that the customer should have £80. This is because the
only remaining transaction would hav. n_the irreversible cash
withdrawal for £80.
d. e-4+tHorizon would then have printed out 3 copies of the Disconnected
Session Receipt which would indicate this.3-eopies—as—follows:— (one for
Customer, one for Branch records and one to attach to the F##till to aid with
recovery).
e. It would not have printed out the AP receipt for the BT Bill.
f. Horizon would then have logged out and disconnected.
Strictly Private and Confidential; Subject to Legal Privilege
Page 7 of 4
POL00130164
POL00130164
Strictly Private and Confidential; Subject to Legal Privilege
hat the SPMR fail follow the instructions from Horizon
ensure that the customer had received £80.
10, 9:-The system would then display the Log On screen.
BE
40--Again the User must have been aware of this as at 09:37:19 they Logged On again
44.-As part of the Log On process, the system checks the identity of the last basket
successfully saved at the Data Centre (which appears to be 537803) and compares it
with the identity of the last Basket successfully processed by the counter (in this case
537802). As the last basket saved in the Data Centre has a higher number than that
considered to be the last successful basket processed by the counter, Reeeverythe_
recovery process in the Data Centre would then repeat the process that the counter
carried out at step 8 above. This would have generated the Recovery Basket stored at
09:37:44 as Session 537805,537805 (ie, the reversal of both the BT Bill and the
cash "change" but a valid transaction for the Cash Withdrawal). A Recovery
receipt would have been printed te-indieatethisreflecting these transactions.
6, I 5-What the Logs don’t support
There are some parts of the initial statement that are not supported by the logs. Specifically:
1. “At approximately 10:32 a customer tried to pay his £76.09 BT phone bill with his
LTSB card but was not successful. The customer then withdrew £80.00 cash and used
this to pay the phone bill.”
There is no indication of any attempt to pay this by card. Closer examination of the
Business Rules show that it is not permissible to pay for a BT Phone Bill with a Credit
Card. However the LTSB card used for the Banking withdrawal was a Debit (and not
a Credit) Card. It could be that an attempt was made to settle with a different Credit
Card and the system indicated that it was not acceptable. There is no record of any
attempt to use the LTSB card as a Payment card. Also, when checking for a failed
card transaction in an earlier basket (point 3 in section 3), the value of the failed
payment was £141 and not £76.09. Therefore this couldn’t be the failure referred to.
It would appear that the only attempt to pay this BT Bill was with the withdrawn cash.
2. “The SPMR stamped the customer’s phone bill to evidence receipt of the cash,
returning change of £3.91.”. This may be what the SPMR did. However if so he was
not following the instructions provided by Horizon as outlined in section 4. It is
recognised that the bill may well have been stamped prior to the Disconnected Session
Receipts being produced.
As explained in section 4, there were a number of indications that the transaction was
not successful, and so the Bill payment had not been recorded:
a. The fact that the SPMR was asked twice about Retrying after failed Data
Centre interactions
b. The fact that 3 copies of the Disconnected Session Receipt would have been
printed out on the counter printer
c. The fact that no AP receipt to confirm payment of the Bill was printed
d. The fact that the User had to Log On again and a Recovery Receipt was
printed.
3. “The SPMR did not initiate those reversals nor did he receive any reversal
notifications.”s The SPMR did not initiate the reversals but he would have been
Strictly Private and Confidential; Subject to Legal Privilege
Page 8 of 4
POL00130164
POL00130164
Strictly Private and Confidential; Subject to Legal Privilege
notified. When Recovery was carried out (point 7 in section 3) a Receipt would have
been printed. Also messages are displayed to the User during the recovery process.
4. “The SPMR raised this as an issue with Chesterfield but was told that due to cost
issues Horizon transaction data could not be requested. It was implied that the SPMR
had stolen the money and he was told to make good the shortage.” This is addressed in
section 6 below.
5. “This meant that 2 people had paid the phone bill (the customer, who handed cash to
the SPMR and the SPMR on instructions from Chesterfield).”> The logs show that if
the customer has paid the bill, this payment was not recorded on Horizon. This means
that the phone bill had not been paid as intended at the time of transaction. If in fact
the SPMR had received the payment and not recorded it on Horizon, then there should
be a corresponding surplus of cash in-Herizenat the branch.
It was to instigate the bill paymentthatpayment that FSC raised the Transaction
Correction.
6. “The SPMR was informed that he should have a surplus of £76.09 due to the reversal
of the transactions.”. The figures in section 3 relating to cash declaration indicate that
there was a surplus of around £63 that day.
Ldowthave-l h ious-de
wf + ise:
7. “The SPMR disputes this conclusion, but the more important issue here is the
automated, unreported, reversal of the transactions.”. The Automated Reversal is
explained in section 4. That section also explains that the SPMR should have been
aware of this for a number of reasons.
6-FSC’s Input
“The decision by P&BA not to examine the Horizon detailed transaction data on cost
grounds delayed or denied the SPMR the opportunity to process the transactions correctly
or understand what happened.”
© Under-th A ith Fujitsu-Post-Office-Ltd
itled Fuji ide—detailed_t historical
tr tional-data—Th ber-of requests that-can-be-actioned-within-a-
month -without-additional charge -is eapped-at24.
1 a led fie S ity i sos “hie ;
1 5 A isd k : Lbek flaw:
. Fi Servier C is-abl h b bud
fund-this-should-a-request-breach-the-eap-It is noted that this is not_an issue
ith_Horizon_but_rather estion_around P. ffice Ltd's pr for
investigating disputes raised by SPMRs.
Suel Id-inevitabl jelay-i idine the} 1
with-a-meaningful-reply;Horizon does retain full transaction logs. There is no
question of this information not being available or being somehow inaccurate.
° Fi Serviee-Centre-was-able-to-informally-determine-from—Fujit
hy-th -as-a-break-of-continuity_in-the transact ssi b
not however readily accessible by Post.
Strictly Private and Confidential; Subject to Legal Privilege
Page 9 of 4
POL00130164
POL00130164
Strictly Private and Confidential; Subject to Legal Privilege
fii ind_mi re from Fuji Pi ffice_therefore_onl
the logs when it is pré mM so and when an issue cannot b
resolved using other available information.
the case raised in this Spot Review. there was no need to access the transaction.
logs. First, it was possible to determine what had happened _in_the branch from
the "disconnect" and "recovery" receipts alone.
2, Atoss-of connectivity-eaused the problem.
3. Itistunderstandable-that-the branch-may-have got-confused.
4, Thelack of receipt-indicates- thatthe bill had-not-been-paids
he—response—had—net—addressed—the—eoncerns—raised.Secondly, the
nly _show the reversal of a transaction, not the meth _reasons for th:
reversal. The logs would therefore not show that the reversals were automatic
responses to a disconnect scenario.
A—transaction—correction—had—already—been—issued—to—eorreet—the-
ion-bef het he-Relationship_M.
5 “
leeisi lid " tunity—to—precessthe—transactions
eorrectly.”Thirdly, to_extract_any meaningful information from _the Horizon
records requires the "raw" transaction interr This cann
done without technical expertise.
on the information available at the time, was correct and its approach justified.
Strictly Private and Confidential; Subject to Legal Privilege
Page 10 of 4
POL00130164
POL00130164
Strictly Private and Confidential; Subject to Legal Privilege
Appendix — sample recei
Strictly Private and Confidential; Subject to Legal Privilege
Page 11 of 4
Document comparison by Workshare on 19 April 2013 11:17:27
Input:
[Document 1 ID
ifile://C:/Users/ap6/Desktop/SRO01 Response v0.3.docx
Description
ISROO1 Response v0.3
Document 2 ID
porary Internet
lpot Review 1 - Response.DOCX
ifile://C:/Users/ap6/AppData/Local/Microsoft/Windows/Tem
Files/Content.Outlook/TOYAJ407/_DOC_26684229(1)_S
Description
DOC_26684229(1) Spot Review 1 - Response
Rendering set
lbp2 - basic - colour
Legend:
*Moved fron *
*Moved to *
Format change
Inserted cell
Deleted cell
Moved cell
Split/Merged cell
Padding cell
Statistics:
Count
Insertions
Deletions
Moved from
Moved to
Style change
Format changed
[Total changes
POL00130164
POL00130164