POL00045518
POL00045518
Amsphere + Confidential and Privileged
as On instruction of Coomber Rich Solicitors
Yard House, Basingstoke, RG2] 7NX
Re: Seema Misra
Technical expert’s report to the Court prepared by Charles Alastair
McLachlan, a Director of Amsphere Consulting Ltd.
90 Fenchurch Street
London EC3M 4BY
England
This report contains 42 pages
POL00045518
POL00045518
Amsphere . Confidential and Privileged
: On instruction of Coomber Rich Solicitors
Yard House, Basingstoke, RG21 7NX
Contents
1 INTRODUCTION 1
2 SUMMARY OF FINDINGS...........csssssssecsssecssssensnesensesessnees sa seeneaensesesssoeses 3
3 > CONCLI SIONS.
4 TERMS OF REFERENCE......ccsccsssscsssssssecssssssesssssussssersusessssssecerssseensees 17
5 APPENDIX A...
6 APPENDIX B
7 APPENDIX C
8 APPENDIXD
9 APPENDIX E.
10 APPENDIX F.
11 APPENDIX G....
12. APPENDIX H.
13
14
15
16
Charles McLachlan 1
POL00045518
POL00045518
Amsphere : Confidential and Privileged
On instruction of Coomber Rich Solicitors
Yard House, Basingstoke, RG2] 7NX
1 Introduction
11
Background to to the Horizon system
Accounting systems are usually designed around a ‘double entry’ booking
keeping principle. The double entry book keeping principle means that for
every entry into the system there is an equal and opposite entry_that:should
maintain the ‘balance’ between the accounts.
So, for example, if somebody at the till sells a stamp for £1 paid in cash then
the stock account would be reduced by £1 value of stock and the cash on
hand account would be increased by £1 — overall the balance between the
accounts would be unchanged.
As part of the process of financial control, it would be normal for the value of
stamps to be physically counted and recorded (stock value) and the value of
cash on hand physically counted and recorded (cash value) and these two
values compared (‘reconciled’) to what is recorded in
The sub post office uses specialised terminals to conduct business using the
Horizon system. This activity is recorded in messages of two types —
transaction messages and event messages. The messages are transmitted to
and from the Horizon data centre managed by Fujitsu.
The Horizon system developed and managed by Fujitsu is integrated to a
ard
number of other systems controlled by Post Office Ltd (POL) and various 3
parties (for example, the Driver Vehicle Licensing Agency (DVLA),
Charles McLachlan 1
© accounting system. _
ie)
Q-
POL00045518
POL00045518
Amsphere i Confidential and Privileged
On instruction of Comber Rich Solicitors
Yard House, Basingstoke, RG21 7NX
merchants services providers for debit card services (such as Link) and E-Top
Up for mobile phone credit). This full set of systems and the operational
processes supporting them, I will refer to as the Post Office Limited
Operating Environment
1.2 Hypothetical issues with the Horizon system
1.2.1
1.2.2
1.2.3
The User Interface gives rise to incorrect data entry: poor user experience
design and inadequately user experience testing can give rise to poor data
entry quality. In cases that users are working under pressure, insufficiently
trained or are using a system presented in a language different from their first
language the problems of data entry can be exacerbated.
The Horizon system fails to properly process transactions: accounting
systems are usually carefully designed to ensure that accounts balance after
each “double entry” transaction. In particular, a database technology referred
to as ‘two-phase’ commit is used to ensure that either both entries or neither
entry is recorded on the system.
External systems across the wider Post Office Limited Operating
Environment provide incorrect externally entered information to the Horizon
accounts through system or operator error outside Horizon.
Charles McLachlan 2
POL00045518
POL00045518
Amsphere o Confidential and Privileged
On instruction of Coomber Rich Solicitors
Yard House, Basingstoke, RG21 7NX
2 Summary of findings
2.1 Introduction
My findings are subject to certain limitations in the manner in which I was able to
pursue my investigations and which may have impacted my ability to provide a
complete picture. “
2.1.1 Reliance on Fujitsu
For my understanding of the Horizon system and the manner in which it integrated
into the full Post Office Limited Operating Environment, I am completely reliant on
the information provided by Gareth Jenkins of Fujitsu. Gareth Jenkins has provided
every possible assistance subject, however, at all times to the instructions of his
+ employers and Post Office Ltd.
It should be noted that Fujitsu were originally contracted to design, build and operate
the Horizon system and continue to manage and develop the system under contract to
Post Office Ltd. In addition, I understand that Fujitsu have recently been awarded a
contract by the Post Office Ltd to operate, manage.and develop-other systems in the-
Post Office Limited Operating Environment.
2.2 Although Gareth Jenkins was able to explain the various interfaces to Horizon
vis-d-vis the Post Office Limited Operating Environment, he was not able to
» comment on its operation. See Appendix A Horizon Architecture Diagrams
Provided by Gareth Jenkins of Fujitsu.
2.2.1 Independent investigation
Charles McLachlan 3
POL00045518
POL00045518
Amsphere , Confidential and Privileged
: On instruction of Coomber Rich Solicitors
Yard House, Basingstoke, RG] 7NX
The Post Office provided no opportunity for independent investigation of the
operation of the Horizon system under test conditions or using video observation in a
live environment.
The Post Office provided no opportunity to observe and review the training of sub
post masters.
The Post Office provided no opportunity to examine the logs of defects, change
QO- ——Tequests and outstariding known issues for the Horizon system.
The Post Office provided no opportunity to understand and review the systems and
processes in the Post Office Limited Operating Environment outside Horizon that
could give rise to transactions in Horizon. In particular:
© It was not possible to examine the process for introducing Transaction
Corrections that can give rise to changes in the cash that Horizon records at
the branch
© It was not possible to examine the processes for Remittances (the movement
of cash and stock) into and out of the branch that changes the cash and stock
oo that Horizon records at the branch _ a a
© It-was not possible to examine the processes for revaluing foreign currency
which could change the value of cash held at the branch.
¢ It was not possible to examine the processes of reconciliation conducted by
the Post Office that could give rise to Transaction Corrections.
2.2.2 Opportunities for reconciliation
Charles McLachlan 4
POL00045518
POL00045518
Amsphere Confidential and Privileged
. On instruction of Coomber Rich Solicitors
Yard House, Basingstoke, RG21 7NX
In the normal process of investigating the consistent operation of a system, I would
expect to examine the end to end trail of paper vouchers, transaction files, log files
and other electronic records across the Post Office Limited Operating Environment.
The Horizon system has been designed to operate as an accounting system and
therefore is designed to ensure that for every set of credits there is a matching set of
debits. In the set of transaction files I have been provided this design constraint is
met. However, the only Way to determine whether it is-the-correct: set of credits oF
debts is to reconcile the Horizon transaction with externally held records, Ina
typical banking environment this would typically be a set of paper vouchers (debit
slips, paying in slips, cheques, etc.) recorded at thSe counter as the transaction was
conducted. These vouchers may be hand written or may be printed by out at the
counter terminal. The cashier can then ensure that their till is balanced at the end of
the shift and, if for example a cheque has been entered for the wrong value, deal with
the matter immediately.
Unfortunately, the Horizon system has not been designed to automatically provide
these vouchers and it does not appear that Post Office counter staffs are trained to
ensure they retain them. It has not been possible, therefore, to reconcile the Horizon
branch records to the actual transaction undertaken over the counter in the branch; ~
2.3. Problems of data entry at sub post office.
2.3.1 Incorrectly calibrated touch screen
’ The Horizon system provides a touch screen for data entry. If when you ‘touch’ the
screen the screen does not respond properly it may be because the screen has not
properly calibrated the position of the ‘touch’ to the representation of the button on
Charles McLachlan 5
POL00045518
POL00045518
Amsphere : Confidential and Privileged
. On instruction of Coomber Rich Solicitors
Yard House, Basingstoke, RG2] 7NX
the screen. The logs provided by Dunks show that Misra asked for guidance on how
to re-calibrate the screen on at least one occasion.
If this gave rise to incorrect entry of cash amounts then this would explain
discrepancies. The Horizon system does not record the recalibration of the touch
Screen as a system event so it is not possible to identify how frequently individual
screens were re-calibrated.
. a printed slip) to
be retained at the counter for every transaction so it is not easy for the cashier to
identify such data entry errors either immediately on entry or when seeking to
balance the till at the end of the day.
The Post Office have not provided us with an opportunity to record the use of a
system in a sub post office experi encing problems.
For all of these reasons it has not been possible to assess the impact of poor screen
calibration on data entry.
2.3.2 Poor User Interface Design
Poor user interface design can contribute to poor data entry quality and user errors.
The Post Office have not provided us with an opportunity to conduct a user I interface
design audit or record the use of a system in a sub post office experiencing ; problems.
2.3.3 Use of the FASTCASH button
One of the features of the Horizon branch terminals is that it is possible to complete a
transaction: by use of the ‘Fast Cash’ button. When the ‘Fast Cash’ button is pressed,
the value of the basket of items being transacted is added up and then any payment
Charles McLachlan 6
POL00045518
POL00045518
Amsphere , Confidential and Privileged
: On instruction of Coomber Rich Solicitors
Yard House, Basingstoke, RG21 7NX
by debit card, cheque or whatever is accounted for. The outstanding balance is
automatically calculated and treated as an over the counter cash payment and then
‘clears the basket’ (i.e. completes the transaction and makes the terminal ready for
the next transaction),
It was anticipated that counter staff would use the ‘Fast Cash’ button as a matter of
habit on every transaction in order to clear the basket, The Horizon system does not
log whether the basket is cleared using the ‘Fast Cash’ button or some other method: ~~
oO
Both Jenkins and myself recognised that there could be circumstances in which a
debit card transaction was not authorised but (either because the printer was not
working or because of force of habit) the counter still cleared the basket using the
“Fast Cash’ button. The result would be that the Horizon system recorded the receipt
of an amount of cash over the counter covering the whole outstanding balance.
The effect of this would be to create a cash discrepancy (shortfall) in the till.
I have identified a number of transactions for which the ‘Fast Cash’ button could
have given rise to such a discrepancy (see Appendix B and items in italics in the
highlighted sections).
QO ~~~“The total value of these transactions is £ 2544.09. I have considered Jenkins report
(see Appendix B) and believe that he may have overlooked the transaction of £7,000
on 11-Jan-07 at the bottom of Page 5 of the document ‘Transactions Associated
With Rejected Cards in Appendix I for which I was unable to find a reversal.
Irrespective of the facts in relation to the transaction of £7,000, in my opinion the
relevance of the ‘Fast Cash’ button in this matter is:
the ‘Fast Cash’ button is demonstrated to be a source of data entry error (the
reversals confirm this).
Charles McLachlan 7
POL00045518
POL00045518
Amsphere i Confidential and Privileged
On instruction of Coomber Rich Solicitors
Yard House, Basingstoke, RG21 7NX
© the specific circumstances relating to rejected card transactions give rise to a
relatively small value of possible discrepancies during the 13 month period
for which records have been provided compared to the total discrepancy
found in the audit.
¢ their may be other circumstances in which the use of the “Fast Cash’ button
gives rise to discrepancies which have not been identified or investigated
__ because I was unable to record the-live-operation-of the-use-of the“Horizon™
system in a sub post office.
2.3.4 Insufficient training
The activities of a sub post office more properly correspond to those of a retail bank
branch rather than a retailer of stamps and postal services. (A cursory review of
Appendix H shows a transaction throughput of £48m recorded in the Horizon
transaction logs in a 13 month period for example). The Post Office were unable to
provide a definitive set of training materials, learning goals and competence
assessments which would make it possible to understand the extent to which Misra
was adequately trained and properly understood how to conduct the operations ofa
sub post office —
However, an examination of the transaction and event logs provided by Fujitsu from
1 Dec 06 — 31 Dec 07 shows that:
* The Declared Branch position had discrepancies vis-a-vis the Horizon totals
at the end of almost every period.
¢ The Variance Checks conducted to reconcile the branch position vis-a-vis
Horizon showed a discrepancy on the vast majority of occasions ranging from
18 pence to more than £11,000.
Charles McLachlan 8
POL00045518
POL00045518
Amsphere . "Confidential and Priviteged
On instruation of Coomber Rich Solicitors
Yard House, Basingstoke, RG2] 7NX
Such a consistent and pervasive failure to ensure that the tills balanced on a daily
basis can be explained by:
a) Theft and/or fraud that the Post Office failed to investigate for at least 13
months
b) Inability of individual branch counter staff to operate properly
ra) —---..)_ Inability of the-sub-post-mistress to train/manage staff and/or conduct end of
day processes.
d) Persistent system failure
I have not been able to conduct the kind of investigation which would exclude
persistent system failure.
2.4 Problems with Horizon
2.4.1 The Calendar Square, Falkirk Problem
Q Jenkins provides a summary of the problem first identified in proceedings relating to—
a sub post office in Calendar Square, Falkirk. (see Appendix C). Unfortunately, as
Jenkins acknowledges, we have not been provided the transactions for the period
prior to March 2006 when we may have been able to independently determine
whether this was an issue for West Byfleet between 30/06/2005 and the bug fix to
Horizon provided in March 2006. Jenkins confirms that the implementation of the
Horizon system at West Byfleet, where each counter terminal is managed as a
separate “stock unit” rather than all the terminals being pooled, is precisely the
circumstances which could give rise to the “Calendar Square’ problem.
Charles McLachlan 9
POL00045518
POL00045518
Amsphere . Confidential and Privileged
: On instruction of Coomber Rich Solicitors
Yard House, Basingstoke, RG21 7NX
As far as the Calendar Square problem applies in relation to West Byfleet, in my
opinion:
¢ It demonstrates that there have been faults with the Horizon system which
give rise to discrepancies that can cause losses. It is not reasonable to
exclude the possibility of system problems when considering a case such as
Misra.
© Lam unable to determine whether the Calendar Square problem contributed to
the discrepancies at West Byfleet because the Post Office have not provided
the relevant transactions and event logs.
2.4.2 The travellers cheque stock problem
The Horizon system is used to record the stock of travellers cheques held by the sub
post office and account for the transactions when they are sold or encashed. In one
of my branch visits 1 was shown a sequence of actions that demonstrated how the
Horizon system reported on the stock of travellers cheques in a manner that was
completely confusing and misleading.
"Take the example of 10 travellers cheques of value USD 100 at the beginning of the
day. If you run a stock report it will show 10 x USD TC 100 which corresponds to a
value of USD 1,000.
A customer comes in and purchases one travellers cheque at USD100 and pays for it
using a debit card.
if you then run a stock report it will show -90 x USD TC 100 which corresponds to a
value of USD -9,000.
Charles McLachlan 10
POL00045518
POL00045518
Amsphere . Confidential and Privileged
On instruction of Coomber Rich Solicitors
Yard House, Basingstoke, RG21 7NX
In other words, the report has treated deducted the USD 100 from the travellers
cheque item count of 10 to get =90. Clearly you can’t hold a negative stock of a
physical item such as a travellers cheque so the report is both meaningless and
completely misleading.
I discussed this with Jenkins and he acknowledge that this is a known feature of
Horizon and that the Post Office have not instructed Fujitsu to change the system to
0 properly accounted for the travellers cheques in the end of day process but I had no
opportunity to test whether this was true.
In my opinion, this stock report could give rise to counter staff or sub post masters
seeking to correct the perceived problem through manual adjustments leading to real
discrepancies.
2.5 System problems from beyond Horizon
2.5.1 Transaction Corrections
@. ._....The Post Office Limited Operating-Environment encompasses a large number of
systems that use outputs from the Horizon system to conduct other processes to
support Post Office operations. It is from these systems that Transaction Corrections
arise.
Consider this hypothetical example by way of illustration:
At the Post Office counter a cheque was encashed for £50 and a cash payment to a
customer of £50 was recorded on the system, However, as a result of the clearing
Process and various reconciliation processes, the face value for the cheque is
identified as £5. This gives rise to a cash discrepancy of £45 which the branch would
Charles McLachlan 11
POL00045518
POL00045518
Amsphere : Confidential and Privileged
: On instruction of Coomber Rich Solicitors
Yard House, Basingstoke, RG2] 7NX
be expected to make up. A Transaction Correction would be issued to the branch for
them to acknowledge and the branch accounts would be updated.
Both myself and Jenkins considered whether Transaction Corrections could be a
source of the branch discrepancies. Jenkins limited his examination to Transaction
Corrections to credit/debit card and banking transactions and identified relevant
Transaction Corrections to the value of £1,840 (Appendix G). Ihave take a slightly
wider scope of transactions of interest which have_a total-absolute-valueof £ -
19,257.21 and absolute value of £ 82,918.35 (Appendix d.
Jenkins acknowledges in his e-mail that he is unable to comment on the integrity of
the processes used by Post Office Limited to create Transaction Corrections or the
Operating processes used to generate them.
Unfortunately, the Post Office failed to make anybody available to discuss the
operation of the Post Office Limited Operating Environment and the reconciliations,
error rates, controls and internal audit Processes used to ensure integrity.
In my opinion, the value of the Transaction Corrections identified by Jenkins or the
transactions of interest identified by myself is not the issue. What is clear is that
Transaction Corrections are generated from outside Horizon: We have vio evidence
“as to whether or not:
a) The Transaction Corrections are of the correct value
b) Some Transaction Corrections should be applied at all
c) Some Transaction Corrections are omitted
The Post Office seek to address Concerns (a) and (b) by providing the sub post master
with an opportunity to ‘request evidence’ (i.e. challenge) a Transaction Correction.
Charles McLachlan 12
POL00045518
POL00045518
Amsphere a Confidential and Privileged
On instruction of Coomber Rich Solicitors
Yard House, Basingstoke, RG2] NX
There is no record of Misra requesting evidence in the transactions provided between
1 Dec 06 and 31 Dec 07. .
However, as discussed above, the Horizon system does not automatically provide the
paper vouchers that would support a dispute over a transaction at the branch,
ene
Further, this process does not address the possibility (c) that some necessary
Transaction Corrections are omitted.
Jenkins appears to assume that the limited value of the Transaction Corrections he
has identified means that they cannot explain the discrepancy in the Misra case. He
overlooks the possibilities that:
© The values are incorrect
* There are missing Transaction Corrections which would reduce the cash
balance expected by the Horizon system (i.e. be in favour of Misra).
In my opinion, we have insufficient evidence to exclude in correct or missing
Transaction Corrections as an contributing factor to the discrepancies in the Misra
case.
2.5.2. Remittances
The systems supporting the movement of cash and stock to and from the sub post
office are integrated into Horizon through Remittance transactions (colloquially
referred to as Rems). There is no Suggestion in this case that there were problems
with the operation of the Remittance system. However, it should be noted that I am
currently instructed in other cases in which the defendant suggests that the system
gives rise to incorrect cash balances being recorded on the branch system. The Post
Charles McLachlan B
POL00045518
POL00045518
Amsphere - Confidential and Privileged
On instruction of Coomber Rich Solicitors
Yard House, Basingstoke, RG2] 7NX
Office have not, at this stage, provided me with an opportunity to investigate these
claims. .
It is worth noting that Jenkins has analysed the transactions and identified a patie
or remittance transactions which is consistent with Misra’s statement that she
declared cash held in remittance pouches in the safe which was not actually present.
2.5.3 If the potential source of the incorrect transaction processing can be identified
then it would be helpful to be able to reproduce the problems under controlled
test conditions in a consistent and reproducible manner. This would require
the assistance of Fujitsu in providing access to the test environments
maintained in support of the Horizon system.
Charles McLachlan 14
POL00045518
POL00045518
Amsphere : Confidential and Privileged
: On instruction of Coomber Rich Solicitors
Yard House, Basingstoke, RG2] 7NX
3 Conclusions
3.1 It is evident that trial balances (Variance Checks) and period balances (Branch
declarations) showed a continuous pattern of discrepancies through out the
period for which transactions were provided. It appears that no action was
taken by the Post Office to investigate these discrepancies or to ensure that
Misra was competent to prevent them from arising. Instead, Misra removed an __
~~ employee tinder suspicion of theft and implemented in independent stock units
for each counter. Neither action appears to have resolved the issue.
3.2 The possibility that problems with screen calibration and the use of the “fast
cash’ button contributed to the discrepancies at West Byfleet has not been
excluded by the investigations of Jenkins and myself. However, it is difficult
to demonstrate that they are of a magnitude to explain the full amount of the
discrepancy.
3.3. The Horizon system has had problems in the past as acknowledged by Jenkins
Square, Unfortunately the Post Office has not provided oe
in relation to Cal
Byfleet nor have they provided a list of known defects in Horizon. The
‘travellers cheque’ problem is an illustration of the known defects we
independently identified but Jenkins confirmed that Fujitsu maintain a full list
which has not been released.
3.4 The Horizon system is a component of the full Post Office Operating
Environment, Other elements of this environment can result in changes to the
cash balances recorded at the branch. Both Transaction Corrections and
Remittances will act in this way. Jenkins was unable to provide any opinion as
Charles McLachlan 15
POL00045518
POL00045518
Amsphere . Confidential and Privileged
. On instruction of Coomber Rich Solicitors
Yard House, Basingstoke, RG21 7NX
to the integrity of these systems and I was provided with no opportunity to
investigate them. The Post Office has provided no evidence as the integrity of
these systems and the processes used to manage them.
oOo
Charles McLachlan 16