POL00144533
POL00144533
Strictly Private and Confidential; Subject to Legal Privilege
Spot Review 1 - Response
STATUS: DRAFT
Author: Gareth Jenkins
Version: 3
Last edited by: Bond Dickinson
Last edit date: 01/05/2013
APPROVED BY DATE
POL legal
Fujitsu
Simon Baker
ALL APPROVALS REQUIRED BEFORE SUBMISSION TO SECOND SIGHT
THIS PAGE TO BE DELETED BEFORE SUBMISSION
FINAL SUBMISSION TO BE IN PDF FORMAT TO ENSURE ALL META-DATA ABOUT PREVIOUS
DRAFTS IS REMOVED
Strictly Private and Confidential; Subject to Legal Privilege
Page I of 10
POL00144533
POL00144533
Strictly Private and Confidential; Subject to Legal Privilege
Horizon Spot Review - Response
SRO1: Debit Cards ~ Cash Withdrawals and GIRO Payments
1. Executive Summary
This Spot Review does not demonstrate any failing in Horizon.
This Spot Review principally asks whether a SPMR will be properly notified about automatic
reversals of transactions when Horizon is unable to connect to the Data Centre. The analysis
below shows that Horizon does provide adequate notification.
Further, in the particular case raised in the Spot Review, the root cause of the difficulties
suffered by the SPMR was his failure to follow the on-screen and printed instructions given
by Horizon. Post Office Limited is confident that the SPMR knew that some transactions had
been automatically reversed because:
© The branch had been suffering connectivity issues in the run up to the incident in
question.
¢ When the transactions in question first failed to be processed (because Horizon
could not get a response from the Data Centre), Horizon asked the SPMR whether he
wished to cancel or retry the transactions. The SPMR opted to retry the
transactions.
e@ When the transactions failed again, the SPMR opted to cancel the transactions.
e@ Horizon then automatically disconnected and printed a "disconnect" receipt that
showed the transactions that had been automatically reversed. A sample
"disconnect" receipt is included the appendix to this response.
e A standard customer receipt was not produced — this would tell the SPMR that the
full transaction had not proceeded.
© Following the disconnect, the SPMR was required to log back on to Horizon and duly
did so.
© Following the log on, and as part of the standard recovery process, Horizon printed a
"recovery" receipt which again showed the transactions that had been reversed and
those that had been recovered. A sample "recovery" receipt is included in the
appendix to this response.
2. Introduction
This spot review relates to an issue raised by John Armstrong the SPMR in Lepton Branch
(FAD Code 1913204) relating specifically to transactions carried out on Horizon Online on 4"
October 2012. The issue is headed “Debit Cards — Cash Withdrawals and GIRO payments”.
This report provides information as to what was alleged by the SPMR (see section 2) and a
detailed analysis of what actually occurred as shown in the system logs (section 3). Section 4
then describes how recovery operates on Horizon and Section 5 identifies those points in the
report which are not supported by the Logs. Finally section6 addresses the question of
access to raw transaction data.
Strictly Private and Confidential; Subject to Legal Privilege
Page 2 of 10
POL00144533
POL00144533
Strictly Private and Confidential; Subject to Legal Privilege
3. The SPMR’s view of what happened
The following is an extract from the Spot Report saying what the SPMR says happened:
The SPMR reports that there were intermittent internet connectivity problems (also
reported to Chesterfield) on 4 October 2012. Online payments and withdrawal
transactions were sometimes successful but also failed on occasions. It is possible
that Horizon was partially operating through its back-up (mobile phone) connection.
Some card payments had to be attempted two or three times before being accepted.
At approximately 10:32 a customer tried to pay his £76.09 BT phone bill with his
LTSB card but was not successful. The customer then withdrew £80.00 cash and
used this to pay the phone bill. The SPMR stamped the customer’s phone bill to
evidence receipt of the cash, returning change of £3.91. Several weeks later, the
customer returned from holiday to find his phone had been cut off due to non-
payment. The SPMR’s examination of the Transaction Log showed that all
components of the transaction had been reversed. The SPMR did not initiate those
reversals nor did he receive any reversal notifications. The SPMR raised this as an
issue with Chesterfield but was told that due to cost issues Horizon transaction data
could not be requested. It was implied that the SPMR had stolen the money and he
was told to make good the shortage. This meant that 2 people had paid the phone
bill (the customer, who handed cash to the SPMR and the SPMR on instructions from
Chesterfield). The SPMR was informed that he should have a surplus of £76.09 due
to the reversal of the transactions. The SPMR disputes this conclusion, but the more
important issue here is the automated, unreported, reversal of the transactions.
From this information, the following key issues have been identified:
¢ When Horizon cannot get a response from the Data Centre, are automatic
transaction reversals notified to SPMRs?
*® Why is raw transaction data not provided to SPMRs? It is noted that this second
issue does not raise a question about an error in Horizon. Rather, it is focussed on
Post Office's procedures and processes.
4. What the System Logs show
Note that the system logs show all times in GMT rather than local time. On 4'* October 2012,
GMT was 1 hour behind Local Time (ie BST). The times quoted in this review relate to the
system logs. Therefore the mention of 10:32 by the SPMR above relates to 09:32 in the logs.
There do appear to be 2 cases on 4" October where the system had a forced Log Out that
resulted in a recovery Log On being required. This supports the statement above: “The
SPMR reports that there were intermittent internet connectivity problems (also reported to
Chesterfield) on 4 October 2012”. The two “Recovery Log Ons” occurred at 08:51:40 (when
no recovery was required) and at 09:37:20 when recovery was required as will be described
later in this report.
The following table looks at the number of online requests for either Banking or Credit /
Debit Card Payments that appear to have timed out:
Tota
Date I
04/10/2012 I 13
05/10/2012 I 4
08/10/2012 I 11
Strictly Private and Confidential; Subject to Legal Privilege
Page 3 of 10
POL00144533
POL00144533
Strictly Private and Confidential; Subject to Legal Privilege
10/10/2012
11/10/2012
16/10/2012
17/10/2012
18/10/2012
19/10/2012
22/10/2012
23/10/2012
25/10/2012
Grand
Total 44
NBRWNNENN
This supports the comment regarding intermittent connectivity problems on 4" October. I
note that there were similar problems on 8" October.
There are 4 examples prior to 09:30 where either a Banking withdrawal or a Credit / Debit
card payment initially failed and was successful on the second attempt. There was also one
example where there were two failures for a card and presumably the customer or the SMPR
gave up. This supports the statements that “Online payments and withdrawal transactions
were sometimes successful but also failed on occasions” and that “Some card payments had
to be attempted two or three times before being accepted”.
The raw logs do have statistics regarding times taken to connect to the Data Centre and also
an indication of the type of Comms currently in use. From these it can be seen that the
Branch normally operates using ADSL, but at the time of the failure that is being examined it
appears to be using a mixture of O2G and 03G (ie mobile networks) presumably due to a
failure of the main ADSL connection. This supports the statement that “It is possible that
Horizon was partially operating through its back-up (mobile phone) connection”. This may
have been visible to the user as a slower than normal response time.
The key transactions are those described as occurring at 10:32 (ie 09:32 GMT).
This analysis starts at 09:26 and shows the sequence of baskets (meaning the group of
individual transactions undertaken during a single customer visit) processed between that
time and 09:40.
1. 09:26:30: Session 537799 contained two transactions: A Card Account Withdrawal
(Withdraw Limit) for £271.54 and a corresponding Cash Settlement.
2. 09:27:34: Session 537800 contained three transactions: A failed Card Account
Withdrawal (Withdraw Limit) immediately (09:28:13) followed by a successful Card
Account Withdrawal for £141.80 using the same card and a corresponding Cash
settlement.
3. 09:29:27: Session 537801 contained a single transaction: A failed Visa Debit card
payment. This payment had been requested for £141 and had failed due to no
response having been received by the counter within the timeout period (33
seconds). Clearly an attempt had been made to purchase something or pay for a
service for £141, but when the Debit card payment failed, the original transaction
was voided and the basket completed.
4. 09:31:56: Session 537802 contained 2 transactions. A Halifax Current Account
Withdrawal for £200 followed by the corresponding cash settlement. It would
appear that the card used here was the same as the one used in the previous session
when the Debit Card payment failed.
Strictly Private and Confidential; Subject to Legal Privilege
Page 4 of 10
POL00144533
POL00144533
Strictly Private and Confidential; Subject to Legal Privilege
5. 09:32:52: Session 537803 contained 3 transactions. A bill payment to BT for £76.09
followed by a Cash Withdrawal for £80 using a Lloyds TSB card and £3.91 cash for
the difference.
6. 09:37:19: User JAROO1 Logged On again
7. 09:37:44: Session 537805 generated by the system as part of the Recovery that
takes place during Log On and contains 3 transactions. The first 2 are the Reversals
for the BT Bill Payment and Cash transactions in session 537803, and the 3 is a Cash
balancing transaction for £80 to correspond to the £80 cash withdrawal which
should have been treated as successful at the time of failure. This is why “The
SPMR’s examination of the Transaction Log showed that all components of the
transaction had been reversed.”
It should be noted that the above comment is not correct. The Banking Withdrawal
for £80 has not been reversed.
8. 09:40:19: Session 537806 contained 2 transactions. A Card Account Withdrawal
(Withdraw Limit) for £229.72 and a corresponding Cash Settlement.
It should be noted that there was no Session 537804. There are a number of circumstances
under which there are gaps in Session Sequence Numbers and in general they are not
expected to be contiguous. In fact they are based on an underlying Journal Sequence
Number which are contiguous and relate to any record that has been audited.
In this case the “missing” number relates to the Journal Sequence Number used in the Log
On Request, but there are a number of other circumstances that can result in a Journal
Sequence Number being used where there is no corresponding Basket.
Looking at the statistics recorded with the Recovery basket at Point 7 above, it can be seen
that there were a number of issues during session 537803:
a. The Authorisation for the Cash Withdrawal was successful and was done on a 3G
comms Connection.
b. The subsequent attempt to update the Recovery information in the basket after
completing the Banking Transaction failed due to a timeout on a 2G comms
connection
c. There are then 4 attempts (at roughly 45 second intervals) to store the completed
basket to the Data Centre. The first 2 use a connection type of 2G and the other 2
use a 3G comms connection. From the branch's records, they are all marked as
having failed.
d. From the Data Centre's perspective, one of the attempts did result in all the data in
that basket being successfully saved in the Data Centre but, due to the connectivity
issues, the branch did not receive a confirmation from the Data Centre. The branch
will therefore record this as a failure.
Moving on to the end of the day the following Cash Declarations were made:
A. At 16:31:27 a Declaration was made for £22,160.54 followed by a variance check
which indicated a discrepancy (loss) of £1,237.16.
B. At 16:32:46 a second cash Declaration was made for £23,460.54 followed by a
variance check which indicated a discrepancy (gain) of £64.84.
Looking forwards, the following variance check discrepancies were recorded:
Date Variance Check Discrepancy [Loss or Gain I
Strictly Private and Confidential; Subject to Legal Privilege
Page 5 of 10
POL00144533
POL00144533
Strictly Private and Confidential; Subject to Legal Privilege
04/10/2012 I £62.84 Gain
05/10/2012 I £66.15 Gain
06/10/2012 I £76.98 Gain
08/10/2012 I £71.91 Gain
09/10/2012 I £69.05 Gain
10/10/2012 I £63.99 Loss
The Stock Unit (ie. the cheques) was Balanced (ie. by the SPM manually correcting or making
good the discrepancy) and rolled over from Balance Period 3 into Balance period 4 on 10"
October 2012 and the Discrepancy committed to the accounts. (There was also a £37.75
discrepancy Gain on stamps at the same time.)
5. Explanation of Recovery
The fact that a Log On (and Recovery) occurred at point 6 above indicates that there must
have been a failure just before that point and the User would have been informed of a
Forced Log Off. The fact that Recovery reversed most of the last Session recorded prior to
the recovery indicates that the following sequence of events occurred. This is confirmed by
the statistics described above at point c in section 4 above.
The user must have been aware that there was a problem in this circumstance. What they
would have observed was the following:
1. Having completed the Bill Payment and Cash Withdrawal, the User would have
either selected the “Settle” or “Fast Cash” option from Horizon. If Settle was
selected then they would again have selected either “Cash” (and keyed in the
amount) or selected Fast Cash.
2. This would have completed the Basket and attempted to save the basket to the Data
Centre.
3. Following a failure of the first attempt, the system would automatically carry out a
retry and attempt to save the basket to the Data Centre again.
4. Following the failure of the second attempt, a message would have been displayed
to the User informing them that there was a failure to contact the Data Centre and
did they wish to Retry or Cancel.
5. The fact that there were 4 attempts to contact the Data Centre, indicates that the
User must have selected Retry and so the system would have made a 3 attempt to
save the basket to the Data Centre.
6. Following a failure of the third attempt, the system would automatically carry out a
retry and attempt to save the basket to the Data Centre yet again.
7. Following the failure of the fourth attempt, a message would have been displayed to
the User again informing them that there was a failure to contact the Data Centre
and did they wish to Retry or Cancel.
8. The fact that there were only 4 attempts to contact the Data Centre indicates that
the user must have selected Cancel this time. This would have resulted in a Forced
Log Out. This means:
a. Horizon would cancel those transactions that could be cancelled. In this
case, the BT Bill and the Cash “change” could be cancelled because those
Strictly Private and Confidential; Subject to Legal Privilege
Page 6 of 10
10.
11.
12.
POL00144533
POL00144533
Strictly Private and Confidential; Subject to Legal Privilege
transactions do not get processed until the basket completes and in this
instance the basket had failed.
b. The cash withdrawal transaction for £80 could not be cancelled. Prior to the
disconnect, Horizon had already contacted the customer's bank to confirm
that a cash withdrawal could be made from the customer's account. The
customer's bank had therefore already registered the withdrawal from the
customer's account and this transaction could not be cancelled.
c. Horizon would then re-calculate the basket showing that the customer
should have £80. This is because the only remaining transaction would have
been the irreversible cash withdrawal for £80.
d. Horizon would then have printed out 3 copies of the Disconnected Session
Receipt which would indicate this (one for Customer, one for Branch records
and one to attach to the till to aid with recovery).
e. It would not have printed out the customer receipt for the BT Bill.
f. Horizon would then have logged out and disconnected.
The SPMR should then have made sure that, in accordance with the Disconnect
Receipt, the Customer had been given cash to the sum of £80. It is at this point that
the SPMR failed to follow the instructions from Horizon in that he did not ensure
that the customer had received £80.
The system would then display the Log On screen.
Again the User must have been aware of this as at 09:37:19 they Logged On again
As part of the Log On process, the system checks the identity of the last basket
successfully saved at the Data Centre (which appears to be 537803) and compares it
with the identity of the last Basket successfully processed by the counter (in this case
537802). As the last basket saved in the Data Centre has a higher number than that
considered to be the last successful basket processed by the counter, the recovery
process our the Counter would then repeat the process that the counter had carried
out at the point of failure at step 8 above. This would have generated the Recovery
Basket stored at 09:37:44 as Session 537805 (ie. the reversal of both the BT Bill and
the cash "change" but a valid transaction for the Cash Withdrawal). A Recovery
receipt would have been printed reflecting these transactions.
6. What the Logs don’t support
There are some parts of the initial statement that are not supported by the logs. Specifically:
1.
“At approximately 10:32 a customer tried to pay his £76.09 BT phone bill with his
LTSB card but was not successful. The customer then withdrew £80.00 cash and
used this to pay the phone bill.”
Although the LTSB card used for the Banking withdrawal was a Debit Card, there is
no record of any attempt to use that LTSB card as a Payment card. Also, when
checking for a failed card transaction in an earlier basket (point 3 in section 4), the
value of the failed payment was £141 and not £76.09. Therefore this couldn’t be the
failure referred to.
It would appear that the only attempt to pay this BT Bill was with the withdrawn
cash.
Strictly Private and Confidential; Subject to Legal Privilege
Page 7 of 10
POL00144533
POL00144533
Strictly Private and Confidential; Subject to Legal Privilege
2. “The SPMR stamped the customer’s phone bill to evidence receipt of the cash,
returning change of £3.91.”. This may be what the SPMR did. However if so he was
not following the instructions provided by Horizon as outlined in section 5 (ie. to
ensure the customer received the total sum of £80).
As explained in section 5, there were a number of indications that the transaction
was not successful, and so the Bill payment had not been recorded:
a. The fact that the SPMR was asked twice by Horizon about Retrying after
failed Data Centre interactions
b. The fact that 3 copies of the Disconnected Session Receipt would have been
printed out on the counter printer, which should have showed the
transaction reversals.
c. The fact that no customer receipt to confirm payment of the Bill was printed
as would normally happen.
d. The fact that the User had to Log On again and a Recovery Receipt was
printed.
It is recognised that the bill may well have been stamped prior to the Disconnected Session
Receipts being produced.
3. “The SPMR did not initiate those reversals nor did he receive any reversal
notifications.” The SPMR did not initiate the reversals but he would have been
notified. When Recovery was carried out (point 7 in section 4) a Receipt would have
been printed. Also messages are displayed to the User during the recovery process.
4. “The SPMR raised this as an issue with Chesterfield but was told that due to cost
issues Horizon transaction data could not be requested. It was implied that the
SPMR had stolen the money and he was told to make good the shortage.” This is
addressed in section 7 below.
5. “This meant that 2 people had paid the phone bill (the customer, who handed cash
to the SPMR and the SPMR on instructions from Chesterfield).” The logs show that if
the customer has paid the bill, this payment was not recorded on Horizon. This
means that the phone bill had not been paid as intended at the time of transaction.
If in fact the SPMR had received the payment and not recorded it on Horizon, then
there should be a corresponding surplus of cash at the branch.
It was to instigate the bill payment that Financial Service Centre raised the
Transaction Correction.
6. “The SPMR was informed that he should have a surplus of £76.09 due to the reversal
of the transactions.” The figures in section 4 relating to cash declaration indicate
that there was a surplus of around £63 that day.
7. “The SPMR disputes this conclusion, but the more important issue here is the
automated, unreported, reversal of the transactions.” The Automated Reversal is
explained in section 5. That section also explains that the reversals would have been
notified to the SPMR.
Strictly Private and Confidential; Subject to Legal Privilege
Page 8 of 10
POL00144533
POL00144533
Strictly Private and Confidential; Subject to Legal Privilege
7. FSC’s Input
“The decision by P&BA not to examine the Horizon detailed transaction data on cost
grounds delayed or denied the SPMR the opportunity to process the transactions
correctly or understand what happened.”
e It is noted that this is not an issue with Horizon but rather a question around Post
Office Ltd's processes for investigating disputes raised by SPMRs.
© Horizon does retain full transaction logs. There is no question of this information not
being available or being somehow inaccurate.
e These logs are not however readily accessible by Post Office and must requested
from Fujitsu at a cost. Post Office therefore only accesses the logs when it is
proportionate to do so and when an issue cannot be resolved using other available
information.
e In the case raised in this Spot Review, there was no need to access the transaction
logs. First, it was possible to determine what had happened in the branch from the
"disconnect" and "recovery" receipts alone.
* Secondly, the transaction logs would not have assisted the SPMR. The transaction
logs would only show the reversal of a transaction, not the method or reasons for
that reversal. The logs would therefore not show that the reversals were automatic
responses to a disconnect scenario.
© Thirdly, to extract any meaningful information from the Horizon records requires the
"raw" transaction data to be interrogated. This cannot be done without technical
expertise therefore incurring significant cost.
e Fourthly, the above analysis proves that Post Office's assessment, based on the
information available at the time, was correct and its approach justified.
Strictly Private and Confidential; Subject to Legal Privilege
Page 9 of 10
POL00144533
POL00144533
Strictly Private and Confidential; Subject to Legal Privilege
Appendix — sample receipts
(TO BE INSERTED]
Strictly Private and Confidential; Subject to Legal Privilege
Page 10 of 10