Joint Statements
POL00028953
POL00028953
JS2 Bugs table:
Extract
Comments
Coyne XX
Bugs table — recovery
issues (row 8)
acha959T
Ifa TI recovery request times out at the counter, recovery is abandoned and no second attempt is made to
get the recovery information. This is as designed; it was decided to keep recovery simple and not have too
many error paths. The priority is to get the User working again, so in this sort of error path we just mark
the recovery as failed and leave it for SSC to sort out
Bugs table — reversals
(row 9)
PSteed2847N (2003) Remming issue but error notices, rather than TCs, at the time. Obvious
DEA failure.
(I said TC - should have said error notice)
Bugs table — data tree
build (row 10)
MScardifield2219S says ‘nothing gets lost in the end’ ‘ the declare cash problem clears itself
overnight’ (2007) — but is much later and probably not connected to the 2000 incident.
F/22 PCO033128 (Dungannon) ‘immediate impact of this week’s balance has been
addressed’; ‘seen as a one off’; error trapping introduced. Then Yate Sodbury £52K, Appleby
£9K; appeared fixed by July 2000 — so why the much later Peak?
Coyne 3.109 says other Peaks are ‘likely related’ but there is no analysis in his table to
support this.
© F/29 PC0046811 June 2000 RP mismatch £37.40 — not clear what the link is
between the two Peaks. No mention of data trees, or Riposte.
© — F/66 PCO055964 Oct 2000 RP mismatch. No mention of data trees. No Riposte
fault.
© F/76 PCO0S8161 Nov 2000 another RP mismatch, £3K. No mention of data trees.
Riposte system calls failing with no error logged.
Coyne could be asked — how are the Peaks he cites in row 10 of
the table JS2 related to data tree build? I have found no
obvious relation.
He seems just to have picked up three RP mismatch Peaks in
2000, which were fairly common in 2000-2001
The KEL from 2005-2007. Mentions a possible Riposte failure,
but no mention of data trees
POL00028953
POL00028953
Next Peak cited is 2006, PC0132133 and references the KEL. Mentions data trees, but not
data tree build problems — and nobody made a link to the earlier problem.
Conclusion: my estimate of £10K financial impact was conservative
Withdrawn stock (row
13)
Although I included this in my table of possible bugs (for impact assessment), my opinion in
the bugs table was that it had no lasting effect. This was based on my previous analysis of
the KEL (some possible impact) and FJ’s analysis of the same KEL (no impact).
I was being conservative to include it in the table.
Bureau Discrepancies
{row 14)
F/1679 PCO261541 2017 £200 shortfall from FX. Comms failure during tx.
F/1681 PCO261710 - same incident as above - Coyne padding it out.
F/1722 PC0265443 - — long and confusing Peak ~ SAP, Accenture. It is about revaluation of
bureau holdings, not a Tx that goes wrong. Took a long time to resolve — included for effect.
Concurrent Logins (row
18)
Early Riposte problem. Mr Lui left.
Coyne says FJ did not follow up with Escher. How does he
know?
Bureau de change (row
23)
Tiny financial impact
Wrong customer change
(row 24)
F/319 KEL AChambers4134R. smartpost — some clerks do it differently. Fix in ref data -on
and off. One of the KELs I though in RW1 were bugs
Lyca top up (row 25)
Isaid could cause discrepancy through wrong TC
F/698 ballantj020)
TPS (row 27)
{said possible financial impact? FJ said not. Coyne sys net impact zero? Included only in case
FJ evidence not accepted
Drop and go (row 28)
As above — included just in case FJ are wrong. (not challenged in XX, I think)
Mean impact spreadsheet:
POL00028953
POL00028953
a * © » e . ¢ Fy
IsTable eug Kes Financial Probability (SPM Loss toSPMs Probability (SPM Loss to SPMs
row Impact compensated) (RWestimate) compensated) —_(conservative)
4 (RW estimate) (conservative)
TReceipts/payments mismatch wrightm33145) 20000 09 2000 0 20000
2 Callendar Square Jsimpkins338 3000 07 300 ° 3000
3 suspense account acha2230K 14000 os 1400 ° 14000
10 Data Tree Build Mscardifield22195 105000 03 10500 ° 105000
13 Withdrawn stock pothapragadact359R 5000 o7 1500 ° 5000
I 14 Bureau Discrepancies 3000 03 2100 ° 3000
48 Concurrent logins 000 07 2700 ° 9000
I 23 Bureau de change AChambers2252R 300 03 210 ° 300
24 wrong customer change AChambers4134R 300 02 240 ° 300
25 Lyca top up ballantjoz0s 2000 03 1400 ° 2000
I artes ballantj2547« 2000 08 400 ° 2000
28 Drop and Go carde235, 2000 os 1000 ° 2000
Summed impact on SPMs 24350 165600
Mean impact/bug on SPMs 2009 13800
Previous mean estimate 1000 6000
See explanation of coleulations on second sheet.
JS2 Other:
row Extract Comments
03 KELs and Peaks together form a useful source of information about bugs in
Horizon but are a limited window on what happened. It is sometimes
necessary to use evidence from both to try to understand, but even so they
are not a comprehensive picture. It is to be expected that both KELs and
Peaks are incomplete in various respects.
0.5 agreed Peaks assessment
1.2 agreed 12 rows ‘may have financial impact’ — note it did not mention ‘lasting’ impact.
14 agreed No correlation between likelihood and impact — a weird one. 1.4a says I never
assumed it.
POL00028953
POL00028953
1.7 agreed
RP mismatches evident to SPM
Possible line for Coyne XX:
JS2 1.7 talks about Receipts payments mismatches
1.7 makes it clear that it was not just the one bug — but it
happened more frequently?
As in some of the Peaks you have cited? (examples above — bugs
table row 10)
Generally , if an RP mismatch happened, it was not the SPM’s
fault?
It was some system issue?
You have agreed in 1.7 than any RP mismatch is evident to the
SPM?
So it is evident to him , and he knows it is not his fault?
He can see the size of the discrepancy? (in his reports — trial
balance)
So in these cases, he is likely to call the help line? (which
probably leads to a Peak)
And — knowing it is not his fault — he is unlikely to stop calling
until the issue is resolved?
Until he knows the discrepancy has no adverse impact on his
accounts?
In other words, if ever there is an RP mismatch (especially a large
one), the branch is very likely to have been compensated for any
discrepancy?
So these Peaks are not really evidence of lasting discrepancies in
branch accounts?
1.9 agreed to disagree
Definition of branch impact
Coyne’s definition is pretty meaningless — could XX.
My follow ons are 1.10, 1.11
1.12-JC PEAKS and KELs are Fujitsu recording tools and a discrepancy would only Obviously wrong — could take to counterexamples (eg recovery Peaks)
appear in a PEAK or KELs if a Horizon system problem is suspected
115 agreed The number of distinct bugs, for which the experts have seen strong evidence Coyne’s problem is 29; my problem is ‘strong’ evidence — meaning P > 0.2
of the bug causing a lasting discrepancy in branch accounts, is between 12
and 29.
POL00028953
POL00028953
1.21 RW What was I not suggesting?
5.284 If Dr Worden is suggesting that there should be no consideration of the
number of bugs, then I disagree with this position
1.25 _RW Complex recoveries are not bugs. Cite many Peaks.
1.26 _RW Bugs in reports are not discrepancies
1.27_RW Bugs which top branches working
1.28 RW Remming bugs are not lasting discrepancies
1.31 RW Revision of extent estimate
Discusses data tree build bug, whether branches were compensated — factual
issue?
1.34 RW Coyne has shown no flaws in estimate.
135-139 RW Defences against Coyne attacks on extent. Could use these for Xx of Coyne
1.40_RW Raindrops to lightning strikes
1.43 RW Coyne has totally misread the graph.
144-151 RW Deep into claimant analysis. Too much:
If asked: why so much RW here? Apologise if I got it wrong. We tried to do better in later JS. The low level of agreement in JS2 arose partly from our very
different approaches to Horizon Issue 1.
(One of the purposes of a joint statement may be to ensure a level playing field - that neither expert gets a ‘last word’, giving the other expert no chance to
reply. Coyne 2 gave him a last word on RW1; I had no last word on Coyne 1. The JS were a level playing field. Coyne could have come back on the RW points.
Needs to be used carefully.)
The overall balance of RW and JC points in JS2,3,4 is not that far from even.
JS3:
row Extract Comments/ points for XX
31 Fairly useful agreement on robustness. ‘PO’ should read ‘FJ’
32
3.5 Post Office does not consult the full audit data (unfiltered ARQ Data) before * — Surely PO processes for creating TCs are out of scope?
deciding how to handle discrepancies and issuing Transaction Corrections.
* So the experts have effectively agreed that the effects of such
processes are in scope?
POL00028953
POL00028953
3.6 JC More bugs/errors and defects have been shown to impact branch accounts than Surely this is Issue 1, no issue 3?
the initial three acknowledged by Post Office. Already covered by JS2?
So why were you coming back to have another shot at issue 1?
37 JC Same point
3.15 agreed Itis difficult to measure the extent of the robustness of Horizon, apart from how if you cannot measure the extent of robustness, how can you
it might limit the extent of impact on branch accounts, as in Issue 1. comment on how it has changed?
Surely you are jus saying that various facets of robustness have
changed over time?
Various countermeasures?
But you can say little about how the net effect of
countermeasures has changed over time?
3.16 agreed The extent to which these problems were serious, or evaded countermeasures, or This disagreement goes back to 3.15?
caused discrepancies in branch accounts, is not agreed. You cannot really say anything?
So it is a kind of negative disagreement?
3.18 The designers of a system should not make unrealistic assumptions about the What unrealistic assumptions are you referring to?
users of the system.
3.22 Many software bugs can have the same effects as a user error (as illustrated, for _I To make Coyne look evasive:
instance, by the Dalmellington bug, which produced a remming error)
Can you give the court any idea how many user errors occur per
year?
You have not looked at it at all?
© E.g how often monthly accounts need to be balanced?
© E.g how many remming TCs per year?
© Other TCs? Surely they often correct user errors?
So you have no opinion about the relative frequency of user
errors and software errors?
Or their relative impact on branch accounts?
Is it fair to say that once a software error if fixed, it usually goes
away for all time — whereas user errors keep on happening?
POL00028953
POL00028953
3.23 Tightly run ship — see separate note What other IT support organisations have you worked with in
the last 10 years?
On the same scale as Fujitsu?
3.25 IC Therefore, it would be incorrect to say that Horizon was extremely unlikely to be Can you define the scope of the word ‘likely’?
the cause of shortfalls. © During the lifetime of Horizon?
© During one month's branch accounts?
Surely these are very different? (by a factor 3 million)
Surely the second sense is what matters to the claimants?
And to get at it, you would have to do some arithmetic?
Have you done any such arithmetic? Where?
44 Reference data proving Can you tell the court about reference data proving?
Take him to diagram below from TD/ARC/0039
© So you have not looked properly into the processes for
managing reference data?
© Or: does not this show that reference data was
carefully managed?
45 JC However, Peaks and KELs illustrate (by their existence) that errors in data You say ‘illustrate’ and we can agree that
recorded did occur, some with financial impact, some not. Can you define the scope of ‘did occur’?
© During the life of Horizon?
© During one month’s branch accounts?
As above, for ‘likely’ ~ 3 million times different
5.2 PO back office accounting processes Out of scope — but effects were in scope?
53 TCs — experts have agreed that the effects of TC errors were in scope ditto
5.4 3" party data ditto
6.2 JC Measures and controls that existed to reduce the risk of “c. a failure to detect, Limited = not perfect?
correct and remedy software coding errors or bugs” were limited.
Surely that is saying nothing new? Agreed in any case?
POL00028953
POL00028953
ovattaat
‘Slerern terry
There are four main areas within the Horizon Architecture:
1. POL-FS — financial accounting system based on SAP
2. Reference Data Proving — environment in which changes to reference data are proved
before releasing into live (reference data controls things such as which products are
sold, their price and where in the menu hierarchy they are displayed).
3. Branches — the branches themselves
4. Core Horizon — the central systems that support Horizon
POL00028953
POL00028953
POL00028953
POL00028953
JS4:
row Extract Comments
10.1 RW Bullet 2 - definition of remote access
10.1 RW The number of occurrences of remote access for which I have seen evidence is This was just what I had seen at the time — given the definition.
not more than 10. in RW3 I have an updated opinion
10.1 RW The Managed Service Change logs are not a useful source of evidence about Updated in RW3
remote access.
10.2 JC In both Legacy and Online, usage of the Transaction Repair Tool to fix transaction I Can you describe the TRT to the court?
data within the Horizon...
You are aware it was called the TIP TRT?
Can you describe the function of the TIP?
Can you describe the function of TPS?
© See 5.1.15 of Ds/APP/HLD/0027 below , and TRT HLD
* So TRT was used only to repair transaction to be sent on to POL
systems?
* — Sono direct effect on branch accounts?
© Only indirect effects, which we will come to?
* Have you any reason to suppose it did have effect on branch
accounts? Evidence?
POL00028953
POL00028953
10.10 JC
The modification from a TRT could provide an erroneous view of a branches’
accounting position and therefore Post Office could potentially issue a
Transaction Correction to a branch in error.
Does this not apply to any change at all made in a back end system, or
a POL system?
Was not the main purpose of the TRT to repair something that had
gone wrong?
Typically the ‘mode ‘ of a transaction - not the amount? (cite Peaks)
So the chances of that repair itself going wrong are a second-order
effect?
And the chances of that getting back into branch accounts
undetected are third-order effect?
Avery remote possibility?
And you have made no attempt yourself to estimate the frequency
with which this third-order effect might happen?
Whereas RW has at least made an estimate of the financial impact of
incorrect TCs?
Which, given what you say here, is a relevant thing to do?
And you have not contested his estimate?
Do you not think that , by dragging in such a remote possibility of a
third order impact on branch accounts:
© You are allowing yourself to draw in such a wide range of
events as to make your conclusions almost meaningless —
including almost anything that might go wrong in the back
end, or in POL?
© You are diluting the concept of ‘remote access’ out of any
real meaning?
© You are attempting to give an impression that problems
may have been more widespread than they actually were —
i.e being biased?
© Does this really assist the court?
10.12 - agreed
Uncommon types of repair —
Therefore, it is usually difficult for the experts to make categorical negative
statements of the form: ‘x or Y never happened’.
You have agreed that it is rather difficult to make a negative
statement like ‘X or Y never happened’?
So a statement of the form ‘X or Y might have happened’ is rather
empty? Obvious? Like ‘bugs could occur’?
Let us look back earlier in JS4 - 10.6 to 10.9
You have agreed that these are rather empty, obvious statements?
POL00028953
POL00028953
10.12 - agreed
However, to repair less common issues which arose from time to time, standard
tools and procedures might not have been sufficient, and evidence might not
persist of what was done at the time.
* You have agreed there were less common issues, which went beyond
the use of standard tools?
* Which required something out of the ordinary?
* — Offthe beaten track?
* Could this have been what Ann Chambers meant by ‘going off piste’?
10.14 RW If any malicious person within Fujitsu were ever to alter branch accounts for (These controls include the TripWire system — see EY service audit report
malign purposes, controls around branch account data were such that any such _I — to detect and log any unauthorised activity)
alteration would be likely leave traces behind. it.
10.15 JC The controls around branch account data do not specifically consider if the © You are aware of Payment Card Industry (PCI) standards?
monies within the transaction actually go to the correct accounts. It would be © You are aware that Horizon was subject to these?
possible through simple changes to alter the sort code and account number of © See extract below from EY service audit
the destination account and unless this was spotted by the PM or the client, Post : : So
Office system would not detect this. * And that they are designed specifically to prevent things like this,
from happening?
* Sowhy did you mention it, when it plainly cannot happen?
114 Jc There is reference to “When we go off piste we use appsup” and “making adhoc * See 10.12 above — you have agreed this may have been not what
changes” in the disclosed material which could suggest that the approved ‘Ann Chambers meant?
mechanism for modification was not always used, should this be the case then
evidence would not be available.
11.13 JC OCP and OCR records do exist, but these do not directly correspond with © What is the timeframe covered by OCP/OCR (up to about 2010) ?
Managed Service Change “MSC” entries, where it is stated by Mr Godeseth that © What is the timeframe covered by MSC ? (from about 2008)
audit records should exist. , ,
* So why would you expect them to correspond, outside possibly
2008 - 2010?
12.4 JIC In summary, the existence of multiple audit points for differing actions make it © Would it be fair to say that you have never tried to ‘accurately
difficult to accurately quantitatively assess how often the abilities and facilities quantitatively assess’ any number at all in your reports?
were used. There are no reporting matrices either expert is aware of that record * Can you point me to one?
such figures across Horizon’s whole lifespan. ° I
* So why do you specially draw attention to it under Issue 12?
13.1 JC Further, transactions within Horizon’s back end systems (POLSAP etc) are subject * Have you not said the same thing before, under 10.10?
to fixes performed via the Transaction Repair Tool. These back-end systems are
used by Post Office when determining if a TC should be issued to address branch
accounts discrepancies. Therefore, any changes made to data within these back
end systems could have a consequence on Subpostmaster branch accounts.
© So why say it again here?
* As before, are you not trying to cast your net so widely as to be
meaningless?
POL00028953
POL00028953
JS4 generally
After JS2, were the experts not instructed by the court to look
for agreements, not to take up further JS with their
disagreements?
Yet in this JS, there are vast screeds of ‘IS thinks’
Much more than the agreed parts? (maybe do a word count)
From DES/APP/HLD/0027:
5.1.15 {TIP Transaction Repair Tool
Function
‘The TIP Transaction Repair Tool is an ORACLE Forms application that runs on the TPS workstation.
SSC uses it to correct records that were rejected when the Branch Database tried to load them into the
TPS tables. The records are rejected because they fail the check constraints on the tables into which
they should be loaded. The corrected records are loaded into the 65% partition of the relevant table.
From POL-0032939.docx:
Document Title:
Document Type:
Release:
Abstract:
Document Status:
Originator & Dept:
Contributors:
TPS - EPOSS Reconciliation - TIP Transaction Repair
HLD Specification
Release 16.40
This document describes details of a maintenance tool that will
assist SSC to repair EPOSS transactions processed at the
Counter but are unable to copy from BRDB into the TPS Host.
These transactions are saved within the Host in a table holding
Harvester Exceptions. SSC require to process these transactions
and have them sent to RECIPIENT SYSTEMS as quickly as
possible. POCL require that these transactions be sent using the
current electronic interface (TPS to TIP).
APPROVED
Wing Pang
POL00028953
POL00028953
From EY service audit , about PCI standards:
4.6 Monitoring
Fujitsu util
and ethical
4 Performance Dashboards.
4. Standard Reporting Packs.
41 Audits and Health Checks
including
1 Reviews and Lessons Learned.
1 Customer Satisfaction Scores.
a variety of systems, processes and tools to help ensure that operations are efficient, effective
Audits by Fujitsu Business Assurance Teams, Fujitsu External ISO Accreditors (the POL account is itself
accredited to ISO 27001) and POL or its agents (including PCI DSS and Link Auditors) are used to
support the design, implementation and post implementation review of the controls in place and to help
ensure that the relevant governance, strategy and needs and requirements of both POL and Fujitsu are
met.
HCOUNS HH Yala LUN UPUUH UF Un CUUN CaS Uata Woo.
B Major
BUSINESS RESTRICTED, a Post Office restricted in its ability to
transact business e.g., 50% of counters unable to trade or trading with
restricted business capability.
Has an adverse impact on the delivery of service to a number of end
users.
Causes a financial loss that impacts POL and / or POA reputation (as
agreed between POL and POA Customer Services).
If a PCI Major Incident process is invoked.
Cc Medium
NON-CRITICAL, a Post Office working normally but with a known
disability, e.g., an interim solution (workaround) has been provided.
If a PCI Minor Incident process is invoked.
Has a minor adverse impact upon the delivery of service to a small
number of end users.
POL00028953
POL00028953