Witness Name: William Paul Patterson (on behalf of Fujitsu Services Limited)
Statement No.: Third
Dated: 14 September 2023
POST OFFICE HORIZON IT INQUIRY
THIRD CORPORATE STATEMENT OF FUJITSU SERVICES LIMITED
INTRODUCTION
1.
lam a director of Fujitsu Services Limited (“Fujitsu”) and am duly authorised to
make this statement on its behalf. I make this statement in response to the
Inquiry’s Rule 9 Requests, dated 16 June 2023 and 31 July 2023, for a corporate
statement addressing issues relevant to Phases 3 and 4 of the Inquiry (the
“Requests’). In particular, the Requests focus on the provision and content of
Audit Record Query (“ARQ”) data by Fujitsu to Post Office Limited (“POL”) over
time.
As noted in the First and Second Corporate Statements dated 28 September
2022 and 29 December 2022 respectively (the “First and Second Corporate
Statements’), I do not have first-hand knowledge of many of the matters which
are set out in this corporate statement. For this reason, I wish to reiterate at the
outset how the information in this statement has been compiled. As with the
responses in the First and Second Corporate Statements, responses to
Page 1 of 103
WITN06650300
WITN06650300
questions set out in this statement are generally drawn from documentary
sources. For the purposes of preparing this statement, I have been assisted by
a team of individuals within Fujitsu and Morrison Foerster. This is due to the vast
amount of documentation and sources of evidence which have had to be
reviewed for a time period stretching over 25 years. This team has provided to
me the documents which are referenced in this statement and exhibited in
accordance with the index at the back of this statement, and which are the
principal source of my knowledge of this statement’s content. In addition to
available documentary evidence, I have also been provided with the witness
statements of three Fujitsu employees (John Simpkins, Jonathan Hulme and
Gerald Barnes) who have also provided responses to the Requests based on
their own personal knowledge or recollections (the “Fujitsu Witness
Statements’).
The responses provided in this third corporate statement represent Fujitsu’s
current understanding of the information available. Given that work in relation to
Phase 4 and other phases of the Inquiry is still on-going, it may be that Fujitsu
will need to supplement this corporate statement as further material is identified
and made available to Core Participants.
Further, as noted in the First and Second Corporate Statements, I do not have a
detailed technical knowledge of the Horizon IT System (“Horizon”) and I am
reliant upon Fujitsu staff with relevant technical expertise and knowledge of such
matters.
Page 2 of 103
WITN06650300
WITN06650300
WITN06650300
WITN06650300
5. For ease of reference, in this statement:
a. ICL Pathway Limited (which initially managed the Horizon contract,
and later novating the contract to Fujitsu Services Limited) and Fujitsu
Services Limited will be referred to collectively as “Fujitsu”; and
b. Post Office Counters Limited (subsequently Post Office Limited) will
be referred to collectively as “POL”).
CONTRACTUAL BACKGROUND
6. From the outset of Horizon, Fujitsu has been required by contract to maintain an
audit trail of “all Transactions and Events...” (see for example paragraph 3.1 of
Schedule AO3 to the Codified Agreement entered into between Fujitsu and POL
on 28 July 1999 (the “1999 Codified Agreement”), and paragraph 3.1 of
Schedule D5 to version 13 of the Codified Agreement which was entered into by
Fujitsu and POL on 23 November 2020 (the “2020 Codified Agreement’)).' This
contractual obligation flowed from Requirement 699 contained at paragraph
1.102 of Schedule A15 to the 1999 Codified Agreement. In particular,
Requirement 699 notes at:
a. Paragraph 1.102.6: “The content of the audit trail shall be agreed with
POCL by a date consistent with the Project Plan.”
b. Paragraph 1.102.9: “The audit trail shall have a level of security such
that it cannot be altered or deleted.”
* FUJ00000071; FUJ00000003
Page 3 of 103
c. Paragraph 1.102.11: “The audit trail shall comply with Requirement
829", namely the prosecution support Requirement set out at
paragraph 1.133 of Schedule A15 to the 1999 Codified Agreement.
Fujitsu’s solution for Requirement 699 is set out in Schedule A16 to the 1999
Codified Agreement. During the course of 1999-2000, this solution was amended
by Change Control Notice (“CCN”) 0423a to include the following additional
wording “[t]he audit trail is specified in the Audit Trail Functional Specification.’*
CCN 0423a also introduced the Audit Trail Functional Specification document
(version 3.0 dated 1 July 1999 at the relevant time)* as a Contract Controlled
Document (“CCD”) to be agreed between Fujitsu and POL. In order to assist the
Inquiry, Fujitsu sets out at Appendix 1 to this corporate statement a schedule of
approved Audit Trail Functional Specification documents from version 3.0
onwards. In accordance with paragraph 3.3 of Schedule D5 to the 2020 Codified
Agreement, the Audit Trail Functional Specification continues to be a CCD as
agreed between Fujitsu and POL.
THE ARQ SPREADSHEET
8.
With the first of the Requests, the Inquiry enclosed a spreadsheet containing
transaction and event data for the Marine Drive Post Office branch with unique
Financial Accounts Division (“FAD”) Code 213337 (“Marine Drive”) dating from
2 February 2004 (the “ARQ Spreadsheet”,* pages 17 and 18 only). Fujitsu has
now been made aware that the ARQ Spreadsheet forms part of a larger
document. However, for the avoidance of doubt, in preparing this statement,
? See CCN 0423a at FUJ00000394, and confirmation that the CCN was approved in the document
entitled ‘Change Control Notices Applied’ and dated 23 October 2010 (FUJ00001431)
$ FUJ00001318
* LCAS0001383
Page 4 of 103
WITN06650300
WITN06650300
WITN06650300
WITN06650300
Fujitsu understands that the Inquiry requires consideration of pages 17 and 18
of LCAS0001383 only, and does not require consideration of the remaining
pages of the document. Fujitsu also understands from the Inquiry that the ARQ
Spreadsheet was provided to Mr Lee Castleton, a former postmaster of Marine
Drive, by way of disclosure in the civil case of POL v Lee Castleton.>
9. By way of background, Fujitsu has to date identified two requests for Audit
Request Queries (the “ARQ Requests”) submitted by Graham Ward (Casework
Manager, POL) to Fujitsu in relation to Marine Drive for Horizon data dating from
the period in which Mr Castleton was postmaster at the branch. These ARQ
Requests are dated 26 October 2005 and 4 November 2005,° and request
various data from the period 1 January 2004 to 31 March 2004 (the “ARQ Date
Range’). The first of the ARQ Requests includes a request for disclosure of the
following information:
“Please also conduct an analysis of all Helpdesk calls for the above period,
commenting on any calls that may indicate faults / problems with the system
Please also supply a report of all transactions and events for the office for the
relevant days, including remittances received, transfers between stock units and
error notices.
We would like the following format for logs (in Excel format with each category in
a separate column):
® Case number HQ05X02706, Judgment Citation [2007] EWHC 5 (QB) (LCAS0000649)
® ARQ 0506/405 dated 26 October 2005 (FUJ00152562) and ARQ 0506/421-423 dated 4 November
2005 (FUJ00152564)
Page 5 of 103
10.
11.
12.
Balancing Period; Cash Accounting Period; Session Type - i.e. Serve Customer,
Reversal. Rem In etc Transaction No; Session Indicator; Date; Time; Stock; User
ID; Transaction Type; Amount £p
2 columns specifying whether an OBCS (& state) of scan accompanied the
transaction
(Session Indicator is whatever way the system has of indicating that individual
transactions are linked)”.
The second of the ARQ Requests does not include any reference to Helpdesk
calls but otherwise appears to be substantially the same as the first. In each of
the ARQ Requests, Mr Ward confirms that no witness statement is required.
It appears from contemporaneous records collected by Fujitsu that (i) the
Helpdesk call logs requested were provided to POL on a CD on 2 November
2005 (see email correspondence from Brian Pinder (Security Manager, Fujitsu)
to Mr Ward on 22 November 2005),’ and (ii) the ARQ data requested was also
provided to POL by CD during the course of November 2005 (see email
correspondence from Mr Pinder to Penny Thomas (Security, Fujitsu) also on
22 November 2005).®
The ARQ extraction process that Fujitsu understands was in place in the context
of prosecution support at the time the ARQ Requests were made is set out in
version two of the ‘Network Banking Management of Prosecution Support’
procedure document dated 29 February 2005 (the “2005 Prosecution Support
7 FUJ00152574
® FUJ00152569
Page 6 of 103
WITN06650300
WITN06650300
WITN06650300
WITN06650300
Process’). The originator of the 2005 Prosecution Support Process was
recorded as being Beatrice Neneh Lowther (Security, Fujitsu), with additional
contributors listed as Bill Mitchell (Security Manager, Fujitsu), Ms Thomas, Jan
Holmes (Quality Assurance Manager, Fujitsu) and Alan Holmes (Audit, Fujitsu).
The approval authorities for the 2005 Prosecution Support Process are listed as
Mr J. Holmes, Mr Mitchell and Dave Baldwin (Customer Service Director,
Fujitsu).
13. Section 7 of the 2005 Prosecution Support Process includes a graphic setting
out the steps followed by Fujitsu when responding to ARQ requests from POL.
This graphic is replicated below for ease of reference:
° FUJ00152209
Page 7 of 103
14.
a
Search for files required to
Create Audit Trail of Request.
complete request
According to section 3.3 of the 2005 Prosecution Support Process, the initial step
for responding to an ARQ request at the time was to identify the search criteria
to be applied. These search criteria are said to generally have been limited to the
data fields below, although Fujitsu understands that these fields could be varied
I
Check Horizon system Helpdesk
Logs
I
System automatically
_I logs date and time of
each process to an
‘Audit Trail’
Analysis Non-polling reports
I
by POL upon request in the ARQ request:
a.
b.
the ID for the user logged-on;
counter position ID;
Page 8 of 103
Analysis appropriate Peak logs.
I
Complete witness statement of fact
Tr
Complete Exhibit Labels
i
Despatch to PO Ltd
WITN06650300
WITN06650300
15.
16.
c. — stock unit reference;
d. transaction ID;
e. transaction start time and date;
f. customer session ID;
g. mode (e.g., serve customer);
h. — product number;
i. product quantity; and
je sales value.
All of these fields appear to be contained within the ARQ Spreadsheet.
In order to obtain the relevant ARQ data, section 7.1 of the 2005 Prosecution
Support Process explains the steps that the relevant Fujitsu analyst then had to
take, including:
a. Searching and selecting the necessary files from audit archive and
extract them to the audit server.
b. Generating a messagestore for the files extracted.
c. Using the RQuery tool to select the fields necessary for the relevant
ARQ request and then exporting those fields to an Excel 95 document
(or native format if requested).
d. Burning the exported data to a “‘closed’ CD-W’ in addition to a Word
document providing an explanation of the format in which the data was
Page 9 of 103
WITN06650300
WITN06650300
WITN06650300
WITN06650300
provided. According to section 7.1.7 of the 2005 Prosecution Support
Process, the CD-W was to be labelled with the ARQ reference
number, the FAD Code for the relevant branch, the name of the Fujitsu
employee who compiled the data, the date on which compilation of
the data was completed, the date range requested in the relevant
ARQ request and the name of the Fujitsu employee who checked the
data on the CD-W.
e. Checking the CD-W for viruses using anti-virus software.
f. Despatching the CD-W to the POL Casework Manager by Special
Delivery.
17. If these steps were followed in relation to the ARQ Spreadsheet, it follows that
the data it contained would have been provided to POL in an Excel 95 format on
a CD-W which was delivered in hard copy by post. It does not appear from the
contemporaneous process documents available that Fujitsu would have had
sight of how the information was subsequently shared by POL (either in whole or
part) with Mr Castleton.
18. Further detail in relation to the practical steps to be taken when extracting audit
data are set out in the “ARQ Data Extraction Process” documents in place over
time.’ A schedule of the various ARQ Data Extraction Processes and
prosecution support process documents that Fujitsu has identified to date and
© The ARQ Data Extraction Process in place at the time the ARQ Requests were made can be found
at FUJ00176265
Page 10 of 103
19.
which appear to have been in place over time is set out in Appendix 2 to this
corporate statement.
The Inquiry has asked Fujitsu to confirm whether, in its view, the ARQ data
provided to POL over time was sufficient to enable a postmaster to understand
whether Horizon was operating correctly at their branch. In light of (i) the
evidence heard by the Inquiry from postmasters during the Human Impact
hearings, (ii) the evidence set out in the Fujitsu Witness Statements, and (iii) the
matters set out in this corporate statement in relation to the ARQ Spreadsheet,
Fujitsu cannot confirm that ARQ data on its own was sufficient to enable a
postmaster to understand whether Horizon was operating correctly at the
relevant branch in the time period covered by the ARQ data requested by POL.
AVAILABILITY OF ARQ DATA
20.
21.
In the first of the Requests, the Inquiry sought disclosure from Fujitsu of the
“original log files in their raw form” for the data reflected in the ARQ Spreadsheet.
The Inquiry also asked for confirmation as to whether the data had been reliably
and accurately parsed from the “original raw form” to the ARQ Spreadsheet. In
the context of the ARQ process, Fujitsu understands that “original log data” or
“raw data files” may refer either to (i) data recorded in the audit archive from
which the Riposte messagestore in Legacy Horizon and ARQs were extracted,
or (ii) the Riposte messagestore itself.
Fujitsu understands that, until CCN 1122 dated 5 January 2004,1" the retention
policy for data stored in Fujitsu's audit archive was 18 months.’ Following the
"* FUJ00000918
12 FUJ00001318
Page 11 of 103
WITN06650300
WITN06650300
22.
23.
introduction of CCN 1122, this data retention period was increased from
18 months to 7 years. It appears that this remained the policy until 2014, when
Commercial Terms (“CT”) 1542 dated 2 June 2014"? was agreed between POL
and Fujitsu. In accordance with CT 1542, from around June 2014, data purging
activities in relation to audit and call data were suspended and this arrangement
was envisaged to last for 12 months. Since then, the agreement to suspend data
purging activities relating to audit and call data has been renewed periodically"4
and remains in place at the date of this corporate statement.’® On this basis,
Fujitsu has retained in its audit archive, audit data dating from around June 2007
onwards.
Fujitsu can therefore confirm that the audit data for Marine Drive within the ARQ
Date Range is no longer available in its audit archive. The only way that audit
data from during/before 2007 would now be available to Fujitsu is if employees
at Fujitsu made copies of the relevant data and stored it separately from the audit
archive.
To identify if any copies of audit data relating to Marine Drive from the time of
Mr Castleton’s tenure as postmaster there still exist outside of the audit archive,
Fujitsu undertook searches of documents it has collected into hard copy archives
and its Relativity platform. In doing so, Fujitsu identified a CD labelled “ARQs,
Message store, Marine Drive, 12/03/04 — 02/04/04”. The data from this CD, which
consisted of two documents, had already been collected and processed into
13 FUJ00156946
‘4 FUJ00176279 ; FUJ00176274; FUJ00176278; FUJ00176275; FUJ00176276; FUJ00176277
18 FUJ00176277
Page 12 of 103
WITN06650300
WITN06650300
Fujitsu’s Relativity platform at the time of the Requests. The two documents
contained on the CD are as follows:
a. a .txt file containing the Riposte messagestore for Marine Drive for the
period 12 March 2004 to 2 April 2004;"° and
b. an Excel spreadsheet which appears to split each of the messages in
the messagestore described above into separate rows, with a column
for each attribute."”
24. Although these documents do not contain the messagestore data for the
2 February 2004 date reflected in the ARQ Spreadsheet, Fujitsu has produced
both the above-mentioned documents to the Inquiry and exhibits them to this
statement.
25. As is clear from the messagestore that has been identified, each message
relating to Marine Drive began with “<Message:<Groupld:213337>”" (213337
being the FAD Code for the branch). Accordingly, Fujitsu ran searches for this
identifier across its Relativity platform. Regrettably, no additional responsive
documents were found. It follows that Fujitsu is unable to identify and produce
Marine Drive messagestore data for 2 February 2004.
THE RELIABILITY OF ARQ DATA
26. The Inquiry has asked whether Fujitsu is aware of any cases where an ARQ log
produced for the purpose of court proceedings against postmasters (i) may not
Page 13 of 103
WITN06650300
WITN06650300
27.
have accurately matched the “original log files”, or (ii) was, or may be, unreliable
(together, “ARQ Reliability’).
In the course of Fujitsu’s investigations to date, a number of incidents that may
have impacted either the underlying audit trail from which ARQ data is extracted
or the ARQ extracts themselves have been identified. Fujitsu’s investigations
have included both document searches and discussions with relevant current
employees. The incidents identified by Fujitsu to date are described in more
detail below. Given the expansive period covered by the Inquiry’s Requests, and
the limitations of relying on and interpreting records of technical matters (some
of which may be incomplete), without the benefit of guidance or explanation from
relevant employees with contemporaneous knowledge, Fujitsu cannot be sure
that the incidents contained in this corporate statement are exhaustive. To the
extent that Fujitsu identifies further incidents relevant to the issue of ARQ
Reliability after the date of this corporate statement, Fujitsu will inform the Inquiry
as soon as possible. By way of summary, the incidents identified to date are as
follows:
Description Paragraphs
a. Broken Audit Trail 30 - 63
b. Omissions in ARQ Data Caused by Operator Error I 64-74
Cc. 2008 Incidents related to Riposte Lock Event 75-116
d. Duplicate Transactions 117 — 152
e. Historic Gaps in ARQ Data 153 - 159
f. Other Potential Issues relating to ARQ Data 160 — 161
Page 14 of 103
WITN06650300
WITN06650300
28.
29.
During its investigations to date, I understand that Fujitsu has identified
approximately 2,400 ARQ requests dating from November 2002 onwards. For
the reasons highlighted in paragraph 27 above, it has not been possible to
conduct a forensic investigation into the ARQ Reliability of the audit data supplied
to POL in response to each ARQ request.
The following summaries of incidents which Fujitsu has identified as having a
potential impact on the issue of ARQ Reliability have been prepared from
documents produced to the Inquiry. The incidents addressed in these summaries
are not within my personal knowledge or recollection.
BROKEN AUDIT TRAIL
Background
30.
31.
In or around May 2001, it was identified that there was a data loss in the audit
trail for a six-day period in August 2000 (the “Broken Audit Trail”)."® At this time,
audit data was gathered by an audit server and written to Digital Linear Tape
(‘DLT’) tapes for long-term storage, to be retrieved when needed."? There were
two “Data Centres”, located at Bootle and Wigan, which contained the main
Horizon servers.”° Audit data could be accessed at audit workstations at either
site or Fujitsu's Feltham HQ.?"
According to PinICL PC0066318, which was opened on 24 May 2001, the Broken
Audit Trail issue related to “[aJn incomplete TMS audit trail for the period 8 to
"8 FUJ00171959; FUJ00152184; FUJ00171967
19 FUJ00201501
20 FUJ00201501
21 FUJ00201501
Page 15 of 103
WITN06650300
WITN06650300
WITN06650300
WITN06650300
14" August 2000 caused by coincidental DLT failure at both datacentres” in
Bootle and Wigan.” The problem was compounded when one of the tapes from
the Wigan Data Centre was lost in transit while in the possession of a courier
company, TNT Couriers (“TNT”) on its way to Feltham for further analysis.
32. As set out below, Fujitsu took various actions to mitigate the impact of the Broken
Audit Trail, including:
a. Engaging with TNT regarding attempts to locate the tape from the
Wigan Data Centre. Ultimately, neither Fujitsu nor TNT were able to
locate the tape.”4
b. Attempting to recover the data from the Bootle tape, with the
assistance of third-party data recovery specialists, Vogon Data
Recovery (Vogon”). Vogon concluded that there was a flaw in the
DLT media, and that recovery would only be 85% successful.”
c. Obtaining data from other archived sources. This was said to have
reduced the break in the audit trail from 6 days to 1 day.
d. Introducing automated media management and a "read after write"
process to mitigate the risk of the problem re-occurring.””
22 FUJ00172093
23 FUJ00152184
24 FUJ00176297
25 FUJ00152184
28 FUJ00171967
27 FUJ00120516; FUJ00116033
Page 16 of 103
Identification and notification of the Broken Audit Trail
33.
34.
According to a letter from Jan Holmes (Quality & Audit Manager, Fujitsu) to Sue
Kinghorn (Consignia, later Royal Mail, Internal Audit) dated 23 May 2001, Fujitsu
identified the Broken Audit Trail when undertaking an audit data extraction for
Charles Leighton (Internal Crime Manager, POL), in relation to an ARQ request
(‘ARQ 8”). It appears that data for the period 8 to 14 August (the “relevant
period”) was held on four DLT tapes.”9 In order to retrieve the data, the tapes
needed to be read by dedicated “Legato” equipment and software.°° Legato
failed to read one of the tapes held at the Wigan Data Centre, and retrieval of the
data from that tape therefore failed. The tape was dispatched via TNT to Fujitsu’s
Feltham offices for further investigation and, although track and trace facilities
were employed, the tape was lost in transit.*"
Fujitsu subsequently attempted to source the outstanding data from the
corresponding DLT tape held at the Bootle Data Centre for the same period.
Recovery was attempted internally and externally using Vogon, “a recognised
industry expert in the field of data retrieval services’.6? Vogon concluded that
there was “a flaw in the DLT media and recover would only be 85% successful.
This is no more than [Fujitsu] could do ourselves...A similar scenario would have
been expected from the lost Wigan tape” .°>
28 FUJ00171959
29 FUJ00176297
8° FUJ00176297; FUJ00152184
3" FUJ00176297
82 FUJ00176297
33 FUJ00172093
Page 17 of 103
WITN06650300
WITN06650300
35. The Broken Audit Trail was initially reported to POL as follows:
a. On 9Q May 2001, Mr Leighton was notified that Fujitsu was unable to
source the evidential data requested.*4
b. Ina letter dated 23 May 2001, Mr J. Holmes informed Ms Kinghorn of
the issue and explained that “/tlhe break has arisen due to a
combination of events outside [Fujitsu's] immediate control but it does
mean that we are not able to retrieve TMS records for that 6 day
period. All other elements of the audit trail are complete.*>
c. Also on 23 May 2001, the issue was discussed by representatives of
Fujitsu and POL during a Contract Administration Meeting.°*
36. According to the minutes of the 23 May 2001 Contract Administration Meeting,
Keith Baines (Head of Commercial, POL) asked Colin Lenton-Smith
(Commercial and Finance Director, Fujitsu) “if a Security impact on the loss of
the data had been performed and what validation process for the tapes was in
place”3”
37. MrJ. Holmes and Graham Hooper (Security Manager, Fujitsu) were the assigned
Fujitsu Problem Managers for PinICL PC0066318.5° On 24 May 2001, they
prepared the following responses to the questions raised by Mr Baines at the 23
May 2001 Contract Administration Meeting:°°
%4 FUJ00172093
5 FUJ00171959
86 FUJ00176285
37 FUJ00176285
8 FUJ00172093
39 FUJ00152184
Page 18 of 103
WITN06650300
WITN06650300
WITN06650300
WITN06650300
a. In relation to the security impact of the data loss: “/n the event that a
third party obtained the necessary Legato equipment and software, it
is evident from the fact that the tape could not be read on the
dedicated equipment at the datacentres that any attempt by a third
party to do the same would not prove fruitful. Specialist data recovery
equipment is available primarily to forensic recovery experts such as
Vogon International, to whom Pathway referred the corrupted Bootle
tape for analysis. These companies operate strict controls to ensure
that data recovery is attempted only for legitimate reasons. Assuming
a third party succeeded where Fujitsu failed or managed to utilise
other specialist recovery services, the information on the tape is not
in a readily interpretable format and it is not possible to infer to what it
relates.”
b. In relation to data validation: “The validity of the data held, and
subsequently retrieved, is proven through the generation of a
ChecksumSeal at the time that the data is written to the DLT. This
value is stored on a database, separate from the audit data, and
subject to an entirely independent backup process. When data is
retrieved from the DLT the Checksum Value is re-calculated and the
result compared with the original value maintained in the database.
The results are recorded in the database and these are checked as
part of the Extraction process prior to despatch of data to PON.”
38. Mr Hooper notified Richard Benton (Problem Manager, POL) of the issue on 24
May 2001 and recorded his summary of this conversation in PinICL PC0066318
Page 19 of 103
39.
as follows: “/ stressed that back-ups were taken and the problem resulted from a
corruption of both tapes relevant to the period in question - a situation that could
not reasonably have been foreseen. It is clear that Consignia’s prime issue is in
attempting to recover the lost data — primarily in respect of evidence to support
potential prosecutions....The cause of the read error on the Wigan tape is
unknown and resulted in the decision to forward this to FELO1 for analysis (during
which it was lost by TNT)...lt was agreed that the only reasonable progression
was to try and locate the lost Wigan tape so that an analysis and data recovery
attempt could be performed. To this end I advised that I had been trying to get
TNT to undertake a search but was not content that TNT were doing all they
could to find the tape” .4°
The Broken Audit Trail issue was also reported in a Customer Service Monthly
Report dated 30 May 2001.*" This report stated that “/t]he issue has been raised
with the Customer and will be managed under normal problem management
procedures. The requirement to undertake large data extractions continues to
have a detrimental effect on the availability of the audit servers’.
Attempts by Fujitsu to recover the lost audit data
Correspondence with TNT regarding the missing tape
40.
In the first instance, as agreed during the call between Mr Hooper and Mr Benton
on 24 May 2001, Fujitsu engaged with TNT in order to attempt to locate the lost
Wigan tape. Accordingly, Mr Hooper wrote to TNT on 25 May 2011 requesting
4° FUJ00172093
#1 FUJ00176282
Page 20 of 103
WITN06650300
WITN06650300
41.
42.
43.
assistance with locating the package containing the missing tape.‘? Fujitsu has
not to date identified copies of the correspondence with TNT but is aware of their
existence from the extracts recorded within the relevant PinICL.
TNT responded to this letter on 3 June 2001, “apologising for the loss and stating
that there had been a discrepancy in the tracking system, which could have
contributed to the problem’ .*3 Mr Hooper reported in PinICL PC0066318 that he
was “convinced that they are not trying hard enough to locate this package which
must be somewhere in their system. I propose to respond and to ask that a
search of all unaccounted-for items be made”.“ In the meantime, Fujitsu advised
POL that TNT was “no longer being used for the transport of media or other
sensitive audit/security information” “>
On 20 June 2001, Mr Hooper again wrote to TNT “stressing the importance of
finding the missing item and offering assistance in attending TNT sites to identify
the package”.*° When no response was received from TNT by 26 June 2001, a
“[c]hase up letter’ was sent.4”
TNT responded on 13 August 2001, stating that “they have recently introduced
a database of missing items, which can be searched against details of package
contents.”48 Following an “extensive check’ of this database, TNT confirmed on
23 August 2001 that the package containing the missing Wigan tape was not
recorded on their system. TNT concluded that they had undertaken everything
42 FUJ00172093
43 FUJ00172093
4 FUJ00172093
45 FUJ00172093
46 FUJ00172093
47 FUJ00172093
48 FUJ00172093
Page 21 of 103
WITN06650300
WITN06650300
44.
possible to locate the lost tape and apologised for “the difficulties this had caused
Fujitsu and any other third party’ 49
Following receipt of TNT’s last letter, Mr Hooper reported in PinICL PC0066318
that he was “now satisfied that there is no possibility of finding this tape. In any
event it needs to be borne in mind that the tape in question was corrupt before
despatch so the likelihood of data recovery from it was negligible” .©°
Rebuilding the audit trail
45.
Fujitsu also explored the possibility of re-constituting the missing data in the audit
archive using back up tapes from the Correspondence Server (“CS”).5' This was
discussed at a Joint Audit & Security Panel Meeting on 18 June 2001, attended
by Mr Leighton and Gary Potts (PON Internal Audit, POL) on behalf of POL, and
Mr J. Holmes and Mr Hooper on behalf of Fujitsu.5? According to the minutes of
this meeting, Mr J. Holmes and Mr Hooper explained the background to the
Broken Audit Trial and informed Mr Leighton that:
a. Fujitsu would be introducing a “read after write” procedure that would
provide assurance that data is not corrupt when written to tape.*? The
purpose of the “read after write” activity was to protect against the
accidental use of flawed media, but it would not guarantee that similar
problems would not occur in the future as it would only provide
assurance that the tape could be read at that moment in time.5*
48 FUJ00172093
5° FUJ00152184
5' FUJ00171971
52 FUJ00171971
3 FUJ00171971
®4 FUJ00176296; FUJ00171971
Page 22 of 103
WITN06650300
WITN06650300
46.
47.
48.
b. Fujitsu was also exploring the possibility of recovering the missing
data from Correspondence Server backup tapes.®>
On 26 July 2001, Change Proposal (“CP”) 3061, entitled “Rebuild broken Audit
Trail due to Missing / Damaged tapes”, was raised by Mr J. Holmes.5° The
purpose of the CP was to reconstruct the Broken Audit Trail using some old
messagestore backup tapes from the NON-Live rig and, subsequently, store the
data in the audit archive.*’ It was highlighted in the CP that, if it was not approved,
Fujitsu would remain contractually non-compliant in the provision of the audit trail
and unable to satisfy PON’s requests for audit data in relation to the relevant
period.5®
CP 3061 was discussed by the Programme Change Control Board (“PCCB”) on
2 and 16 August 2001.59
Following this, CP 3061 was escalated to the Change Control Board (“CCB”) on
20 August 2001.° During the CCB meeting, when asked how the Broken Audit
Trail could be prevented from happening again in future, Mr Hooper informed the
CCB that a read after write mechanism had been introduced to check the data,
but “there was no way to guarantee that this won't happen again”.®' CP 3061
was approved by the CCB on 20 August 2001.° However, it was reverted back
to the PCCB on 18 October 2001 to consider an additional impact.®? On 25
55 FUJ00171971
5° FUJ00155529
57 FUJ00155530
58 FUJ00155530
58 FUJ00176286; FUJ00176287
°° FUJ00176287; FUJ00176288
®' FUJ00176288
2 FUJ00176288
®3 FUJ00176291
Page 23 of 103
WITN06650300
WITN06650300
49.
50.
October 2001, the CCB noted the PCCB’s decision to re-target CP 3061 for the
Cl4 S10R release.**
By October 2001, the CS backup tapes had also been recovered from the
relevant Data Centre and held in secure fire-safe storage, pending attempted
retrieval of the data for the relevant period.® In order to reconstitute the audit
data from the CS backup tapes, a pseudo audit server was built which the backup
tapes were to be loaded on to. As a result, by around 7 December 2001, Fujitsu
had reportedly identified that approximately 66% of the missing data for the
relevant period was available on the backup tapes.® The remaining 34% was
not present on the tapes and was deemed irretrievable.®” The gap in the audit
trail was therefore said to have been reduced from a period of 6 days (7 to 14
August 2001) to less than 24 hours (19:27pm on Sunday, 6 August 2000 until
16:09pm on Monday, 7 August 2000).°*
According to a report entitled “Audit Trail Break — Pathway Position” prepared by
Mr Hooper, following identification of the available data on the CS backup tapes,
Fujitsu's recommendation was to rebuild the audit trail using the CS backup tapes
and provide POL with the available data for the period requested in ARQ 8.® In
accordance with the agreed retention periods in place at the time, the data held
on the back up tapes was due to expire on 14 February 2002.”°
° FUJ00176292
5 FUJ00152184
© FUJ00152184
®7 FUJ00152184
8 FUJ00152184
9 FUJ00176294
7° FUJ00176294
Page 24 of 103
WITN06650300
WITN06650300
51.
By 7 January 2002, Fujitsu had briefed Consignia Internal Audit and Security
Managers on the outcome of the data recovery activities.”’ Consignia confirmed
that the information requested by ARQ 8 was not, at that stage, required in
support of a prosecution. It was agreed that Fujitsu would not need to retrieve
the data from the CS backup tapes at that time but would take steps to ensure
that the data relevant to ARQ 8 was stored and made available if requested.’
Audit of tape handling procedures
52.
53.
Although not solely relevant to the Broken Audit Trail, Fujitsu has disclosed
correspondence relating to an audit that was conducted by Fujitsu, POL and
Consignia of Fujitsu’s management and operation at the Wigan and Bootle Data
Centres. The audit also encompassed Fujitsu’s Belfast Operations Centre,
although Consignia did not participate in the audit visit to Belfast. The report of
this audit does refer to the Broken Audit Trail issue.
In summary, a planned audit into the activities and operation of the Horizon Data
Centres at Wigan and Bootle took place in October 2001 (the “Data Centre
Audit”).’> This also included an audit of the Operations Centre in Belfast, which
was where much of the work carried out in relation to Data Centres was
controlled at that time.” It had been agreed at the Joint Security Audit meeting
on 18 June 2001 that the Data Centre Audit would include tape-handling
procedures and would be observed by a representative from POL, Rashpal
7 FUJ00152184
72 FUJ00152184
73 FUJ00080514
74 FUJ00080514
Page 25 of 103
WITN06650300
WITN06650300
54.
55.
Dhesi (Internal Audit, Consignia and later POL), to verify the tape-handling
procedures in place.”
The Data Centre Audit was carried out by Mr J. Holmes, Mr Hooper, and Mark
Ascot (Test Manager, Fujitsu) from Fujitsu, accompanied at Wigan and Bootle
by Mr Dhesi. Fujitsu’s findings from the Data Centre Audit were recorded in an
internal report entitled “Audit of Horizon Data Centres and Belfast Operations
Centre” dated 21 November 2001 (the “Data Centre Audit Report’).”© This
report set out a number of recommendations for the management of the Data
Centres.”’ A copy of the Data Centre Audit Report was provided to Mr Dhesi on
27 November 2001.78
Mr Dhesi subsequently summarised the Data Centre Audit Report in the format
of a Consignia note (the “Consignia Report”) which he proposed to distribute to
Mike Hannon (Horizon Contract and Commercial Manager, POL), Dave Miller
(Managing Director Post Office Network, POL), Paul Rich (Group Managing
Director, POL), Peter Corbett (Finance Director, POL), Vince Mulholland (Head
of Corporate and Strategic Finance, POL), David Lewington (Head of Group
Internal Audit, POL), and Ernst & Young (External Auditors).’? The Consignia
Report and the extent of its distribution appears to have been a point of
contention at that time. Whilst Fujitsu has not been able to find all relevant
correspondence on this matter, Mr J. Holmes raised various concerns with Mr
75 FUJ00171970; FUJ00171971; FUJ00171972; FUJ00171973
78 FYJ00080514
77 FUJ00080514
78 FUJ00171974
78 FUJ00171979
Page 26 of 103
WITN06650300
WITN06650300
Dhesi on 6 February 2002.®° The correspondence that Fujitsu has identified to
date in this regard is exhibited to this statement.**
Proposals to prevent re-occurrence of the Broken Audit Trail
56. The following measures were put in place by Fujitsu to mitigate the risk of the
Broken Audit Trail re-occurring:
8° FUJ00171983
The “read after write’ mechanism was introduced in September
2001.82
The Audit Panel was reintroduced to PON Internal Audit during June
2001 and expanded to include Security, to allow PON and Fujitsu
security and audit staff to deal with day-to-day issues in an informal
but professional manner.®*
Automated Media Management (to automate tape labelling) was
scheduled to be introduced on 20 August 2001 “to minimise manual
tape intervention” .°4
In June 2001, Fujitsu proposed to introduce write failure checking on
archive tapes in order to provide some assurances that other Legato
tapes were not corrupted.®>
® FUJ00171978; FUJ00171980; FUJ00171983; FUJ00171981; FUJ00171982; FUJ00171970;
FUJ00171968; FUJ00171984; FUJ00176754; FUJ00176756
®2 FUJ00152184; FUJ00176290
83 FUJ00115976
®4 FUJ00152184; FUJ00176280
®5 FUJ00152184
Page 27 of 103
WITN06650300
WITN06650300
57. Notwithstanding the above-mentioned measures, in a letter dated 7 August 2001,
Mr Lenton-Smith informed Mr Baines that Fujitsu had advised PON Audit and
Security Managers that “the measures to remove altogether the risk of future
tape corruption can be achieved only by a complete re-design of the current
solution” °°
58. Mr Baines responded to this letter on 29 August 2001, raising the following
concerns:*”
a. The “read after write” process only deals with part of the problem — a
more common fault with the media being used by Fujitsu is for the
data to become corrupt after the “write” process. “Normal working
practice we believe is to keep ‘parent’, and ‘grandparent’ copies of
tapes” 88
b. Fujitsu “should have in place a process which alerts POCL of any
failure of their obligation to maintain a full audit traif’ °°
c. Mr Baines also sought confirmation of whether there were any
alternative methods of extracting the data required to assist POL’s
investigations and prosecutions.°°
59. Mr Lenton-Smith sought to respond to these concerns in a letter dated 19
September 2001, in which he confirmed that:%"
®© FUJ00176280
®7 FUJ00176289
®8 FUJ00176289
89 FUJ00176289
°° FUJ00176289
°* FUJ00176290
Page 28 of 103
WITN06650300
WITN06650300
WITN06650300
WITN06650300
a. The purpose of maintaining ‘parent’ and ‘grandparent’ copies of data
is to ensure there is always more than one copy of an archive from
which recovery can be attempted. In fact, two copies of the audit
archive were maintained by Fujitsu (one copy at each Data Centre).
b. In relation to Mr Baines’ request for a process to alert POL of any
failure in the audit trail, Fujitsu was not in a position to know whether
data could not be recovered from either of the two Data Centres until
recovery was attempted. The reason for any delay in notifying POL of
the Broken Audit Trail “was the result of our attempts to recover the
situation without raising undue concern in POCL” and “[t}he
unprecedented circumstances, including a protracted search by TNT
for the lost tape, introduced greater delay’.
c. In relation to alternative methods of extracting the relevant data, Mr
Lenton-Smith wrote that the audit archive “is fully secure, duplicated
across two locations and contains special integrity features to provide
assurances that data written to DLT, and subsequently retrieved, have
not been amended during storage. It is considered the only source
capable of being presented in Court and the only one that Pathway
would be prepared to use in those circumstances’.
Commercial resolution and closure of PinICL PC0066318
60. Issues surrounding the Broken Audit Trail gave rise to correspondence between
POL and Fujitsu regarding alleged contractual breaches.
Page 29 of 103
61.
62.
On 6 February 2002, Mr Baines replied to Mr Lenton-Smith’s 19 September 2001
letter alleging breaches by Fujitsu of “various obligations in the Codified
Agreement...which give rise to the right to terminate, and there is a risk of future
similar defaults because the root causes have not been identified and
corrected’.°* Mr Baines further wrote that “this incident and the issues arising
from lost data represent significant risks to Post Office Limited’s business. It
compromises our ability to assure end to end financial integrity of our business” .°>
Mr Lenton-Smith replied to Mr Baines on 15 February 2002. In this letter, Mr
Lenton-Smith denied that the lost data incident demonstrated “poor
implementation of the audit trail and failure by Fujitsu to exercise appropriate
management control in carrying out the procedures for servicing audit trail
requests. This is the only incident of its kind since provision of this service since
the Summer of 1999.”* Mr Lenton-Smith further wrote that he “cannot accept
that the incident and subsequent developments makes [sic] you believe that
there is a significant risk to your business which in turn makes you question the
fitness of Fujitsu’s infrastructure to deliver financial services’. In relation to the
remediation steps that had been taken to prevent the Broken Audit Trail from
reoccurring, Mr Lenton-Smith wrote that following the introduction of the “read
after write” checks and tape cloning at both Data Centres, “there are two copies
of all audit data at each centre...All audit DLTs are stored in the Data Centre
°? FUJ00176295
°8 FUJ00176295
°4 FUJ00176296
°5 FUJ00176296
Page 30 of 103
WITN06650300
WITN06650300
63.
computer rooms, in their individual plastic cases, on correct storage racking, in
line with recommendations for long-term storage of DLT media.”°>
Following further correspondence, POL and Fujitsu agreed to settle any claims
regarding possible breaches by Fujitsu of its contractual obligations in return for
a payment of £150,000.%” One of the conditions on which Fujitsu agreed to this
settlement was the approval of CCN 1019 “documenting the necessary changes
to be made to the Codified Agreement in respect of data storage”. PinICL
PC0066318 was closed on 6 November 2002 and CCN 1019 was approved on
21 November 2002.%
OMISSIONS IN ARQ DATA CAUSED BY OPERATOR ERROR
64.
65.
In 2003, it was identified that data had been omitted from ARQ reports provided
in response to three requests (ARQs 198, 199 and 200) relating to the Forest
Gate branch received by Fujitsu on 23 July 2003 (“Forest Gate ARQs”) and one
request (ARQ 320) relating to the Urmston branch received on 1 December 2003
(the “Urmston ARQ’).
In response to the Forest Gate ARQs, ARQ data was provided to POL by
Beatrice Neneh Lowther (IT Security Analyst, Fujitsu) on 29 August 2003.1 In
September 2003, Ms Lowther provided a witness statement relating to the data
provided. Ms Lowther explained that she had “access to reports that monitor
faults, polling failures, equipment failures and calls for advice and guidance
© FUJ00176296
87 FUJ00176305
°8 FUJ00176298; FUJ00176299; FUJ00176300; FUJ00176301; FUJ00176303
°° FUJ00176304
100 FUJ00121987
Page 31 of 103
WITN06650300
WITN06650300
66.
67.
68.
69.
logged by the Horizon System Helpdesk. During the 01 October 2002 to 31
January 2003, there were 13 calls from Forest Gate (FAD 100002) to the
Helpdesk. None of these calls relate to faults which would have had an effect on
the integrity of the information held on the system.”'"'
Ms Lowther also provided an explanation of the process of extracting data in
response to ARQs, including that “fijnformation is presented in exactly the same
way as the data held in the archive although it can be filtered depending upon
the type of information requested’. Ms Lowther further explained that
“Extraction’s [sic] are only made by authorised individuals” and are logged on the
audit workstation and supported by documented ARQs. 102
According to Ms Lowther’s witness statement, she undertook extractions of data
held on the Horizon system on various dates and at various times between 21
August 2003 and 26 August 2003, in response to the Forest Gate ARQs. The
resultant data was copied onto CDs and exhibited to her witness statement.1%
Ms Lowther also extracted data in response to the Urmston ARQ. This was
provided to POL on 8 December 2003.1
On 13 May 2004, Graham Ward (Casework Manager, POL) notified Fujitsu that
Ms Lowther would be required to attend trial in relation to the Forest Gate ARQs
during the week commencing 21 June 2004.'°5 However, as Ms Lowther was on
“I during this period,’°° POL’s legal services team subsequently
101 FUJO0121891
102 FUJ00121891
195 FUJ00121891
104 FUJ00121995
195 FUJO0121975
106 FUJ00121988
Page 32 of 103
WITN06650300
WITN06650300
70.
71.
requested that an alternative witness statement was provided by Penelope Anne
Thomas (IT Security Analyst, Fujitsu). Ms Thomas's witness statement was also
to include “an explanation as to how days were omitted from the original data
supplied” .1°7
Between 13 and 27 May 2004, Ms Thomas re-extracted the relevant ARQ data
in response to the Forest Gate ARQs, copied this to a CD and submitted it to
POL on 27 May 2004.'°° Ms Thomas also provided an accompanying witness
statement, dated 17 June 2004. In this witness statement, Ms Thomas stated
that the CD provided on 27 May 2004 contained “additional transaction data to
that which was originally supplied in August 2003”.'°° In March 2004, Fujitsu
received the following ARQ requests: ARQ 455, 456, 457 458, 459 & 460 of
0304. The requests noted that the witness statement prepared in response to the
request should also “refer to previous data supplied iro. of ARQ’s 320 & 321."11°
Ms Thomas therefore undertook a similar process in relation to the Urmston
ARQ.
Between 25 and 28 May 2004, William Leslie Mitchell (Security manager,
Fujitsu) performed a comparison of the data extracted by Ms Lowther in 2003
and by Ms Thomas in 2004 in response to the ARQ requests received in respect
of the Forest Gate and Urmston branches.''' According to witness statements
later provided by Mr Mitchell, the “recreation of the data and the check was due
to Beatrice Neneh Lowther’s absenc: ind the possibility of a
107 FUJ00121975
198 FUJ00121987
109 FUJO0121987
"0 FUJ00176306
"11 FUJ00121988
Page 33 of 103
WITN06650300
WITN06650300
witness being required to attend a planned court date.” The comparison revealed
a set of omissions in the data provided by Ms Lowther in 2003.112
72. \Inanemail dated 1 June 2004, Mr Mitchell provided Mr Ward with an “update as
to the root cause” of the omissions, as follows:
a.
112 FUJ00121988
ARQs 198 and 200: These requests related to the period 14/10/2002
to 13/11/2002 (totalling 31 query days) and 12/12/2002 to 08/01/2003,
(28 query days) respectively. The root cause for the omission of data
from the August 2003 retrievals was “that the retrieval was executed
as a single task’ resulting in the volume of data retrieved exceeding
the “available limit of the Message Store area on the Audit Server’.
This caused the Audit Server to “randomly drop 11,135 data entries”
in relation to ARQ 198, affecting 10 query days. One query day was
affected by the same issue in relation to ARQ 200.
ARQ 199: This request related to the period 14/11/2002 to 11/12/2002
(28 query days). “The root cause for the omission of data from this
ARQ is when an ARQ is retrieved it is necessary to add additional
days to the end of the requested date span to ensure a full and
complete capture of the data which may have been harvested at
different times. The operator should then confirm that an end of day
log off is present and extract only the required data files. In this case
the operator added two additional days to each ARQ, which is
normally sufficient, but it appears did not confirm that an end of day
log off was present, consequently an additional 235 data entries were
Page 34 of 103
WITN06650300
WITN06650300
73.
74.
not included in the data extraction. The affected date was 27
November 2002 — Partial, no end of day’.
c. ARQ 320: “[t]he root cause for the omission of data from this ARQ was
the same as ARQ 199 and 200 above”.""°
Mr Mitchell also noted in the 1 June 2004 email that Fujitsu had conducted
checks in relation to a number of other ARQs for 16 other branches “for which
statements have been requested. No discrepancies have been found”.1"4
Mr Mitchell provided a witness statement setting out the above-mentioned
explanations in relation to the Urmston branch (dated 14 June 2004) and the
Forest Gate branch (dated 17 June 2004).''5 In these witness statements, Mr
Mitchell concluded that the omissions made within the data provided by Ms
Lowther had not been repeated in the ARQ data provided by Ms Thomas and
that the latter data was complete in accordance with the original ARQ.""°
2008 INCIDENTS RELATED TO RIPOSTE LOCK EVENT
Peak PC0152376
75.
In or around December 2007, an incident was reported by a Post Office branch
to the Network Business Support Centre (“NBSC”) operated by POL. As
recorded in Peak PC0152376 (“Peak PC0152376”),'"” the call was referred by
the NBSC to Fujitsu. The NBSC informed Katrina Brooks (Helpdesk Support
Technician, Fujitsu) that a “BM stock unit had a gain of £465.73” which “did not
"3 FUJ00172020
4 FUJ00172020
"18 FYJ00121979; FUJO0121988
"16 FUJ00121979; FUJ00121988
117 FUJ00154684
Page 35 of 103
WITN06650300
WITN06650300
76.
77.
go to local suspense”. When the stock unit rolled over, the local suspense was
cleared and the gain was not accounted for. The value of the gain was shown on
the trading position line on the branch’s trading statement. According to Peak
PC0152376, the trading position line “should always show zero”.118
On 21 December 2007, David Seddon (Technical Support Product Specialist,
Fujitsu) recorded in Peak PC0152376 that the incident was found to have been
caused when two processes were run together: the rollover process on counter
1 and various end of day (“EOD”) processes running in the background. This
caused a timeout, also known as a ‘Riposte lock’ (the “Riposte EOD lock”).1°
On 2 January 2008, Gerald Barnes (Software Developer, Fujitsu) attributed the
Riposte EOD lock to the EPOSS code, describing it as “not resilient to errors”.
Mr Barnes noted that the problem was identified to be caused by a fix made to
the relevant EOD process (known as the “CABSProcess’) as part of a previous
Peak PC0140715,1° which caused CABSProcess to write out messages
‘atomically’."2' As I explain below, the Riposte EOD lock problem could affect
the completeness of ARQ data, which raised potential implications for witness
statements that Fujitsu employees had provided or would provide in relation to
prosecutions brought by POL (the “2008 ARQ Issue”). This problem was found
by the Post Office Account team (the “POA’”) investigating this matter at the time
to have existed in Horizon’s live estate between May 2007 and November
2008.12?
118 FUJ00154684
"'8 FUJ00154684
120 FUJ00155207
"21 FUJ00154684; FUJ00155396
122 FUJ00155396
Page 36 of 103
WITN06650300
WITN06650300
78.
79.
In January 2008, a fix for Peak PC0152376 was proposed; however, as noted in
Peak PC0152376, it was decided that this would not be rolled out “given the rarity
of the problem’.'2° According to an entry by Mr Seddon in Peak PC0152376, the
need for a fix would be reviewed if the problem became more prevalent. In the
meantime, a Known Error Log (“KEL”) was created in relation to the problem
(‘KEL dsed5628Q’).'*4 Mr Barnes provided input on KEL dsed5628Q, noting
that the CABSProcess only ran on “node 1” (i.e. the first counter at the branch),
and postmasters working after 7pm should use any counter other than the first
counter.'25 Peak PC0152376 was closed on 10 January 2008.
Peak PC0152421, a duplicate of Peak PC0152376, records that the branch was
not “out of pocket’ as the recorded loss was written off by POL. Further, a
Business Incident Management System (“BIMS”) report was issued to POL.126
Peaks PC0153009 and PC0152828
80.
Two other Peaks were raised in early 2008 and contain references to
PC0152376. It is unclear whether they relate to the Riposte EOD lock issue but
they have been included in this statement for completeness.'?” Peak PC0153009
was raised in January 2008 and related to a Post Office branch with FAD 226242,
where the postmaster was having difficulty with rolling over. The call was cloned
(i.e., linked) to Peak PC0152828.126
123 FUJ00154684
124 FUJ00130827
"25 FUJ00155209
126 FUJ00154686
"27 FUJO0155224
128 FUJ00155211
Page 37 of 103
WITN06650300
WITN06650300
81.
82.
The incident was discussed in an email chain by various members of POA team
members, including Mr Barnes, Michael Peach (Software Support Centre
(“SSC”) Manager, Fujitsu) and Steve Parker (SSC Support Manager, Fujitsu).'2°
Mr Barnes, who was assigned to the Peak, described the problem as the branch
being unable to rollover a stock unit. Mr Barnes analysed the problem and found
that it had been caused by one corrupt transaction. Mr Barnes proposed a
general fix so that DataServer could correct corrupt transactions, though he
noted that the frequency of the problem’s occurrence would dictate whether the
fix would be required’®°. Mr Barnes also recorded on the Peak that he was “not
sure it is worth spending time improving the EPOSS version which is shortly to
be replaced; it would be better spending the same effort making the HNGX
version better. I had already requested that this be advised to the HNGX team in
PC0152376’.
As noted in Peak PC0152376, in this instance, Mr Barnes corrected the corrupt
transaction at the affected branch by running a patched DataServer, and the
Peak was closed later that month once the affected branch confirmed that they
had rolled over successfully.'%"
Peak PC0155120
83.
Peak PC0155120 was opened on 6 March 2008 after the NBSC informed Fujitsu
that a Post Office branch with FAD 485611 had reported that the default stock
unit (which usually rolled over automatically) had failed to roll over."3? Peak
"28 FUJ00155210
130 FUJ00155210
'S1 FUJ00155211
182 FUJ00154685
Page 38 of 103
WITN06650300
WITN06650300
PC0155120 included a reference to KEL dsed5628Q and stated that it appeared
the problem related to “a rare timing issue” where a software process ran at the
same time as rolling over.'%* An Operational Change Request (“OCR”) was
approved by Fujitsu to correct the problem and the Peak was subsequently
closed.'** KEL kiangl823S was raised on 6 March 2008, which recorded the
incident as effectively being “a specific instance of KEL dsed5628Q”.'%
Incident at the Craigpark branch (PC0158102)
84.
85.
In or around July 2008, it appears that there was an issue relating to a
discrepancy at the Craigpark branch with FAD 141832. The issue was discussed
at a Product & Branch Accounting workshop attended by POL and Fujitsu on
1 August 2008.'%° According to the “Actions” report from this meeting, Mike
Stewart (Service Delivery Manager for On Line Services, Fujitsu) noted that a fix
could be introduced, but “what is being proposed for the bug fix [...] if it were to
go wrong could impact the whole estate so avoiding such a fix sounds like the
correct decision”. It appears from the Actions report that an associated BIMS
report was issued to POL.
On the same day, 1 August 2008, Anne Chambers (SSC Systems Support
Specialist, Fujitsu) emailed Gareth Jenkins (Distinguished Engineer, Fujitsu) and
Mr Stewart regarding the incident at the Craigpark branch.'%” In her email, Mrs
Chambers explained that the incident caused a Riposte lock and that it had
occurred once before at the same branch, but did not affect the branch balance
133 FUJ00154685
154 FUJ00154685
135 FUJ00130476
138 FUJ00155230
"87 FUJ00155245; FUJO0154683
Page 39 of 103
WITN06650300
WITN06650300
86.
87.
on the earlier occasion. Mrs Chambers noted the underlying problem was known,
citing Peak PC0152376. Mrs Chambers further explained that while there was a
fix available, “given the low incidence of problems (and the errors introduced on
the back of other fixes) it was decided not to implement’. However, Mrs
Chambers suggested that “since [they] have no way of monitoring for it, and
[receipts & payments] mismatches may not be reported by the [sub-postmaster],
this decision should be reviewed" .1°8
Mr Jenkins responded to Mrs Chambers on 4 August 2008 noting that he
understood similar incidents would be flagged to Fujitsu through various
processes, including generating a ‘red event’.°° Mr Jenkins suggested it might
be more practical for Fujitsu to carry out weekly manual checks for the issue, and
that this “would be significantly less risky than a bug fix given the low number of
occurrences.” In particular, Mr Jenkins noted that he understood “what is being
proposed for the bug fix and if it were to go wrong [it] could impact the whole
estate so avoiding a fix sounds like the correct decision”. However, given
discussions that had taken place on 1 August 2008, Mr Jenkins explained that
“we need to be able to re-assure POL that we would spot any further occurrences
and have a mechanism to correct the accounts (if necessary) even if we don't fix
the root cause”.'40
In a separate email chain, on 11 August 2008, Mr Jenkins notified Peter Sewell
(Security Operations Manager, Fujitsu) of the issue at the Craigpark branch,
stating “Given we only have a couple of instances, and a fix is as likely to cause
138 FUJ00155245
189 FUJ00155245
140 FUJ00155245
Page 40 of 103
WITN06650300
WITN06650300
88.
89.
further problems, then we're reluctant to make a change to Horizon. However, if
Horizon data is being used in evidence for the prosecution of a Postmaster, it is
probably wise to check to see if any such events were produced during the period
in question. Is this something that can / should be built into the ARQ process?’."41
It appears that a meeting took place on 13 August 2008 attended by Mr Jenkins,
Mr Sewell, Alan Holmes (Technical Design Authority, Fujitsu), Steven Meek
(System Developer, Fujitsu) and Penelope Thomas (Security Analyst, Customer
Services, Fujitsu) “to discuss the issue of errors produced by [R]iposte [...] in
relation to the validity and integrity of data provided to [POL] under the ARQ
agreement.” Ms Thomas prepared a note of the meeting dated 14 August 2008
(‘13 August Meeting Note”).'42
According to the 13 August Meeting Note’, Mr Jenkins explained the issue as
follows: “An EOD process (CABSProcess) was being run between 1900 and
2000 hours, and at the same time the user was performing a balancing process
on the gateway PC. During the EOD operation the CABSProcess created a ‘lock’
on the messagestore during which time (30 seconds) causing any other message
writes to wait, subject to a 10 second timeout, until the lock was released. The
balancing operation attempted to write messages to the messagestore but this
operation timed out and the messages were discarded. Due to a deficiency in
the implementation of the counter code the end user was not informed of the
failure and the transaction (the balancing operation) appeared to complete
successfully. When this type of error happens Riposte records an event to the
"41 FUJ00155233
"42 FUJ00154823; FUJ00154824
"43 FUJ00154824
Page 41 of 103
WITN06650300
WITN06650300
90.
91.
92.
event log. It was said that this type of error could happen with any type of
transaction.”
The 13 August Meeting Note recorded, inter alia, (i) a set of follow-up actions
that were agreed during the meeting to understand the types of transactions that
were subject to the 2008 ARQ Issue, and (ii) that Fujitsu “cannot provide any
further ARQs until this exercise is complete as the audit server is being fully
utilised retrieving the 5.5 years worth of Event log data. We must question
whether it is advisable to provide further ARQ data or witness statements until
we have a process in place to fully validate our returns. It was agreed that the
process of retrieving all of the available Event logs would be carried out and this
would start immediately.”'4+
On 22 August 2008, Mr Stewart emailed Mr Jenkins and Mrs Chambers providing
an update on his interactions with Shaun White (Branch Systems IT Advisor,
POL) regarding the incident. In his email, Mr Stewart noted that Mr White was
seeking further clarification of the causes of the Riposte EOD lock event at
Craigpark to pass onto the POL Fraud team. On the same day, Mr Jenkins
replied to Mr Stewart and noted that he and Roy Birkinshaw (Software and
Solutions Development Manager, Fujitsu) would provide an update to Mr Stewart
and Mrs Chambers in terms of communicating with POL.'45
On 28 August 2008, Mr Birkinshaw circulated to Mr Jenkins and David Johns
(Operations Manager, Fujitsu) a draft description of the Riposte EOD lock issue,
which was purportedly targeted at the “initial goal of defining the problem for
4 FUJ00154824
"45 FUJ00155245; FUJ00155238
Page 42 of 103
WITN06650300
WITN06650300
93.
R&RMG management’ and it was not intended for Post Office in its initial draft
form.'4° The document noted the following: “/f the counter were to mis-handle a
financial transaction, whilst simultaneously failing to write to the Riposte
Message Store, this could have potentially serious implications on the integrity
of the audit information supplied to Post Office.”
On or around 3 September 2008, members of POA, including Mr Jenkins, Mrs
Chambers, Mr Stewart and Mr Peach prepared an explanation of the incident at
the Craigpark branch as requested by Mr White.’4” The agreed draft explanation
included the following: “We have identified two occurrences of this event in the
last 2 years which have resulted in an accounting problem and the rest appear
to be benign or had a minor effect which did not impact the branch accounts...We
have a possible mitigation for the timing problem which has been identified.
However, since there had only been 2 occurrences of the accounting problem in
two years, it had been the decision of the Release Management Forum that the
cost of implementing the mitigation would outweigh the potential benefit. This
decision will be reviewed in the light of recent evidence as part of the normal
process”.
Investigation and remediation of the 2008 ARQ Issue
Fixing the Riposte EOD lock problem
94.
By reference to various email correspondence, during September 2008, it
appears it was decided that a fix should be issued in relation to the underlying
"48 FUJ00155241; FUJ00155242
"47 FUJ00155246; FUJ00155247; FUJ00155252; FUJ00155248
Page 43 of 103
WITN06650300
WITN06650300
95.
96.
97.
software bug relating to the Riposte EOD lock.'#8 Mr Barnes, Mr Birkinshaw, John
Burton (Programme Manager, Fujitsu), Mrs Chambers, Steve Evans (Systems
Integration Team Leader, Fujitsu), Mr Jenkins, Mr Peach and Mr Seddon appear
to have been involved in the decision and/or subsequent work that was
undertaken to develop, prepare, test, and release a software code fix to the live
estate (which was operating Legacy Horizon at the relevant time)."4°
As noted at paragraphs 75 to 79 of this statement, the Riposte EOD lock appears
to have been first recorded in Peak PC0152376. Due to administrative reasons,
however, related Peak PC0164429 was used as a means of delivering the
software code fix.'5°
As the software code fix involved changes to Horizon’s live estate, the creation
and implementation of the fix was decided as part of the release management
process, and authorisation to release the fix was given on 29 September 2008."51
As noted in the relevant Release Note’®?, in or around October 2008, the
software code fix was tested, and the fix was delivered to the Horizon live estate
via release “T86” (Release number reference RNT8601).1°>
Review of the audit mechanism
98.
In September 2008, the POA conducted a “review of the audit mechanism and
of the Horizon counter’s behaviour (see, for example, FUJ00155253 and
148 FUJ00155396
"49 FUJ00155258; FUJO0155259; FUJO0155261
*80 FUJ00155366; FUJ00155258; FUJ00155261
181 FUJ00155366;, FUJ00155486; FUJ00155258; FUJ00155261
182 FUJ00155486
183 FUJ00155486
Page 44 of 103
WITN06650300
WITN06650300
99.
FUJ00155257)."*4 The review involved Peter Ambrose (Technical Manager,
Fujitsu), Mr Birkinshaw, Mr Burton, Mrs Chambers, Andrew Dunks (Information
Technology Security Analyst, Fujitsu), Mr Evans, Mr A. Holmes, Mr Jenkins, Mr
Johns, Mr Meek, Mr Sewell and Ms Thomas. The review appears to have
involved (i) an analysis of some of the branches and counters with high incidents
of the Riposte EOD lock problem, (ii) identifying, inter alia, where the problems
associated with the Riposte EOD lock did and did not occur, and (iii) the SSC
checking ARQs to determine whether the Riposte EOD lock had resulted in any
financial or operational impact on the branch for the time period relating to the
ARQ.*55 A working paper, titled “Analysis of Audit timeouts” appears to have been
prepared as a result of the work undertaken as part of the review. ©
In an email from Mrs Chambers to Ms Thomas dated 16 September 2008, Mrs
Chambers explained that the Riposte EOD lock event would occur when one
process had ‘locked’ the Riposte messagestore and another process tried to
access the messagestore at the same time.'5”? The CABSProcess, which would
run at 7.00pm during the EOD procedures at the branch, was the “worst offender”
for causing ‘locking’ to occur; however, event locks could occur in other ways.'®°
Whether or not the Riposte EOD lock event could cause a financial or operational
impact depended on what the second process was attempting to do, and whether
the process handled the error situation appropriately.°° Therefore, Mrs
Chambers checked the ARQs provided by Ms Thomas to (i) identify the two
184 FUJ00155253; FUJ00155257
185 FUJ00155264
"88 FUJ00155256
187 FUJ00155264
188 FUJ00155264
189 FUJ00155264
Page 45 of 103
WITN06650300
WITN06650300
processes involved, and (ii) assess whether the second process “could
conceivably have failed to handle the situation, and if so would it have any
financial or operational impact.” \t also appears that subsequent investigations
into the Riposte EOD lock found that the “atomic transaction” that caused the
CABSProcess to lock was not attempted in any other part of the EPOSS code."
The change proposal for HNG-X
100. In or around October 2008, a change proposal was prepared in relation to HNG-
101.
X, which proposed the creation of an automated solution in HNG-X for the
Riposte event log checks that were undertaken manually when ARQ requests
were made by POL."® It is understood that these manual checks of the Riposte
event log were introduced as a result of the 2008 ARQ Issue. The change
proposal was titled “Enable analysis of Horizon Counter event messages within
the HNG-X Audit solution’, and it was later given the reference number CP0336
(also referred to as CP4867) (“CP0336”). CP0336 was also referred to as the
“Counter Audit Change Proposal” or “Audit Strengthening Change Proposal”.
CP0336 was approved and implemented in or around July 2009, which is
explained further at paragraph 116 of this statement.
CP0336 noted the “current horizon tactical solution” for retrieving archived data
to respond to ARQ requests from POL was largely a manual process which was
“error prone & time consuming”, and that it required “local & insecure storage of
event audit data, invalidating certain standards made within the current witness
statement’.'® It appears, CP0336 proposed a more automated solution be
180 FUJ00155396
*81 FUJ00155272
162 FUJ00155272
Page 46 of 103
WITN06650300
WITN06650300
102.
103.
104.
implemented in the HNG-X audit server and workstation applications to
automatically retrieve and filter Horizon counter event log data when performing
data retrievals in relation to branches operating Legacy Horizon or HNG-X. At
the time, it also appears that those individuals involved in CP0336 considered
that it was impracticable to introduce this change into the Legacy Horizon audit
server given it was being replaced by HNG-X.'®?
CP0336 cited Peak PC0152376 as an example of the potential for the audit trail
to contain instances of event messages which, in the event of an ARQ retrieval,
needed to be analysed to understand if they were significant or not. CP0336
introduced a change in HNG-X’s audit retrieval tooling.
The changes proposed in CP0336 in relation to the audit system included
amending the audit server and workstation applications to automatically retrieve
and filter event data when performing branch data retrievals (i.e. ARQs) in
relation to branches operating Legacy Horizon and HNG-X.164
Mr Birkinshaw, Mr A. Holmes, Mr Evans, Mr Jenkins, Mr Meek, Howard Pritchard
(Principal Security Consultant, Fujitsu), Mr Sewell and Ms Thomas were involved
in the initial drafting and/or discussions in relation to CP0336 (see, for example,
FUJ00155271, FUJ00155278, FUJ00155373 and FUJ00155380).1°
The December 2008 Presentation
105.
According to an email from Ms Thomas to Mr Sewell dated 1 December 2008,
CP0336 raised concerns in relation to the ARQ process and provision of witness
163 FUJ00155272
164 FUJ00155272
165 FUJ00155271; FUJ00155274; FUJ00155278; FUJ00155373; FUJ00155380
Page 47 of 103
WITN06650300
WITN06650300
106.
107.
statements provided by Fujitsu to POL as part of the prosecution support
service.'® It appears the persons involved in the initial drafting and discussions
regarding CP0336 (noted above at paragraph 104) agreed that the proposed
changes to the event checking process needed to be implemented.'®” As a result,
in December 2008, a PowerPoint presentation was prepared titled “Prosecution
Support Urgent Issue” (“December 2008 Presentation”)'®, which appears to
have been prepared in order to:
a. _ internally escalate the 2008 ARQ Issue and the concerns regarding
the completeness of the transaction data on Horizon’s audit archive
that was being provided to POL as part of Fujitsu's prosecution
support service (i.e. as part of the ARQ process); and
b. support the sponsorship within Fujitsu of budget to fund the
developments summarised in CP0336.
It appears Ms Thomas prepared the December 2008 Presentation. Graham Allen
(Application Development Manager, Fujitsu), Adam Cousins (Application
Development Manager, Fujitsu), Mr Evans, David Hinde (Project Manager,
Fujitsu), Mr A. Holmes, Mr Jenkins, Mr Pritchard and Mr Sewell also appear to
have been involved or otherwise had sight of the December 2008 Presentation
during its preparation.©
The December 2008 Presentation was presented to Steve Denham (Head of
Service Management, Royal Mail Group Account, Fujitsu) at a meeting on 17
188 FUJ00155373
167 FUJ00155373; FUJ00155378; FUJ00155380
188 FUJ00154835
169 FUJ00155387; FUJ00154834
Page 48 of 103
WITN06650300
WITN06650300
108.
109.
December 2008.'7° Mr Denham attended the meeting on behalf of Wendy
Warham (Operations Director, Royal Mail Group Account, Fujitsu).'71 Other
attendees of the meeting appear to have included Mr Allen, Mr Cousins, Mr
Evans, Mr A. Holmes, Mr Pritchard and Mr Sewell. 172
Following the meeting, it appears Mr Denham was going to speak with “Legal”
about the 2008 ARQ Issue’”’; however, to date, Fujitsu has not identified any
documents or records as to whether Mr Denham contacted Fujitsu’s in-house
legal team or external legal advisors, and if so, what was discussed.
In January 2009, the POA team prepared a revised draft of the “standard witness
statement’ that Fujitsu provided POL in relation to ARQ data. This contained
proposed amendments in relation to the 2008 ARQ Issue (“Proposed Witness
Statement”) (see, for example, FUJ00122592, FUJ00122593, FUJ00122596
and FUJ00122597).'74 Mr Allen, Mrs Chambers, Mr Denham, Mr Evans, Mr A.
Holmes, Mr Jenkins and Ms Thomas were primarily involved in drafting or
reviewing the proposed amendments, which included adding (i) an explanation
of the event checking process, and (ii) a description of the 2008 ARQ Issue.
Fujitsu’s notification to POL
110.
On 7 January 2009, Ms Warham notified Sue Lowther (Head of Information
Security, POL) and David Gray (Chief Technical Architect, POL) about the 2008
ARQ Issue via email'’5. In the email, Ms Warham provided a summary of the
17 FYJ00155392; FUJO0155394; FUJ00155395
17 FUJ00155385; FUJ00155394; FUJ00155395
172 FUJ00154834
173 FUJ00155394; FUJ00155395
174 FUJ00122592; FUJ00122593; FUJ00122596; FUJ00122597
178 FUJ00155399
Page 49 of 103
WITN06650300
WITN06650300
2008 ARQ Issue in similar terms to the description provided in the Proposed
Witness Statement, and set out various steps that should be taken by Fujitsu and
POL to address the issue, including:
a.
checking the ARQs and confirming the data integrity in the period May
2007 to November 2008;
discussing and agreeing on how they disclose the 2008 ARQ Issue in
the witness statements provided by Fujitsu to POL;
identifying witness statements Fujitsu had previously supplied and
were still awaiting a court appearance to (i) confirm whether they were
impacted by the 2008 ARQ Issue, and (ii) recall and replace impacted
witness statements;
automating the messagestore alerts on the Horizon system so that
manual intervention was not required, with apparent reference to
CP0336; and
providing education to ensure that these types of incidents were
raised as a major incident so that the communication and
management of the incident was undertaken within relevant
timescales.
111. On 7 January 2009, Ms Thomas forwarded Ms Warham’s email to Dave Posnett
(Casework Manager, Investigation Team, POL), attaching the Proposed Witness
Statement.’’® Later that day, Mr Posnett emailed Ms Thomas, forwarding an
176 FUJ00155399
Page 50 of 103
WITN06650300
WITN06650300
112.
113.
email exchange between Mr Posnett and Rob Wilson (Head of Criminal Law
Team, POL) regarding the 2008 ARQ Issue.'”” Referring Ms Thomas to his email
exchange with Mr Wilson, Mr Posnett stated: “/ would say Business As Usual re
witness statements ie don’t include the two additional paragraphs on the last
page. If any issues materialize in due course, we can address then - suggest the
ARQs for these 4 cases are assessed first.”
On 8 January 2009, Ms Thomas forwarded Mr Posnett’s email of 7 January 2009
to Ms Warham and Mr Denham (copying Mr Pritchard and Mr Sewell).178
Referring the email recipients to Mr Posnett’s email, Ms Thomas noted that “POL
clearly do not want the specific details of this incident included in the witness
statement. I will hold off providing the 4 outstanding statements until our review
is complete.”
On 8 January 2009, it appears that relevant members of the POA also held a
meeting where it was decided that POA would conduct a review to identify ARQs
that may have been affected by the 2008 ARQ Issue (see, for example,
FUJ00155402 and FUJ00155400).'79 Mr Allen, Mr Barnes, Mrs Chambers, Mr
Denham, Mr Evans, Mr A. Holmes, Mr Peach, Mr Pritchard, Ms Thomas and Ms
Warham were primarily involved in this review, which included the following
steps:
a. Fujitsu's audit team compared a list of ARQs prepared by the Security
team with relevant event data limited to events occurring between 1
May 2007 and 30 November 2008 that were logged between 7pm and
177 FUJ00155400
178 FUJ00155400
178 FUJ00155402; FUJO0155400
Page 51 of 103
WITN06650300
WITN06650300
7.10pm, which were logged by counter one at the relevant branch'®.
As a result of this work, 27 instances of the 2008 ARQ Issue were
identified'®!; and
the SSC then undertook further checks in relation to these 27
instances to confirm whether the Riposte EOD lock had impacted the
transactions or balancing carried out by the Post Office branch. To
undertake these checks, the SSC reviewed the SSC event archives,
Riposte messagestores, and Riposte event/transaction logs (where
these were available).'®? Following these checks, it appears the SSC
concluded none of the 27 instances of the 2008 ARQ Issue identified
by the audit team had impacted the transactions or balancing carried
out at the relevant branches.'®°
114. On 4 February 2009, Ms Thomas emailed Mr Posnett'®* and confirmed the
following:
180 FUJ00155418
"81 FUJ00155418;
182 FUJ00155422;
"83 FUJO0155418
184 FUJ00155420
Fujitsu's checks in relation to the ARQs for the period 1 May 2007 to
30 November 2008 had been completed, which involved checking the
event logs in relation to 195 ARQs that fell within this timeframe;
27 instances of concern had been identified, which had been fully
analysed, and Fujitsu could confirm the locking issue had been
FUJ00155419
FUJ00155418; FUJ00154839; FUJ00155409
Page 52 of 103
WITN06650300
WITN06650300
WITN06650300
WITN06650300
caused by “contention between the EOD process and a Riposte
checkpoint being written”;
c. following the analysis of the 27 instances of concern, no transactions
or balancing activities carried out at the relevant branches had been
found to have been affected;
d. a change proposal had been raised to “automate the event checking
process”, which appears to be a reference to CP0336; and
e. _ the “standard witness statement’ had been reviewed and no reference
had been made to the “/ocking issue”, but minor revisions had been
made. 85
115. Mr Denham, Mr Pritchard and Mr Sewell were involved in drafting or otherwise
had sight of Ms Thomas’ email to Mr Posnett (see, for example,
FUJ00155422).186
The implementation of CP0336
116. CP0336, which is described at paragraphs 104 to 108 above, was finalised and
passed in March 2009. Mr Denham, Mr Hinde, Mr A. Holmes, Mr Sewell, Ms
Warham and Guy Wilkerson (Commercial Director, Royal Mail Group Account,
Fujitsu) appear to have been primarily involved in the preparation and approval
of CP0336, which was raised on or around 26 February 2009 and approved
internally by Fujitsu on or around 31 March 2009"8’. The changes referred to in
185 FUJ00155421
186 FUJ00155422
"87 FUJ00155474
Page 53 of 103
CP0336 appear to have been implemented as part of HNG-X Release 2 in or
around July 20097°8, following a period of testing which involved teams at Fujitsu
and POL".
DUPLICATE TRANSACTIONS
117.
The Inquiry has asked for details of the Duplicate Transactions incident, first
identified in 2010. I also address apparent known recurrences of the incident in
2014 and 2016.
2010 Incident
118.
119.
On 21 June 2010, Penny Thomas (Security Analyst, Fujitsu) raised Peak
PC0200468 ‘Horizon Audit Spreadsheet Producing Duplicate Transactions’ as
an ‘A priority’, as she had identified duplicate transaction records in an ARQ
return during an audit retrieval for branch 072128.'°° The Peak was assigned to
Gerald Barnes (Software Engineer, Fujitsu).
On 23 June 2010, Ms Thomas shared an initial report on the Peak with Graham
Welsh (Application Services, Post Office Account, Fujitsu), Gaetan Van Achte
(Service Director, Royal Mail Group Account, Fujitsu), Donna Munro (Security
Operations Manager, Fujitsu), Peter Thompson (Head of Service Operations and
Applications, Fujitsu), Gareth Jenkins (Applications Architect, Fujitsu) and Alan
Holmes (Service Delivery Manager, Fujitsu).197 The report noted that under
Legacy Horizon, duplicate records were held in the source Transaction
"88 FUJ00155490
188 FUJ00155511
19° FUJ00097038; FUJ00097058; FUJ00172183
181 FUJ00097038; FUJ00097057; FUJ00097058
Page 54 of 103
WITN06650300
WITN06650300
120.
121.
Management Service (“TMS’) file on the Audit server under “what was thought
to be ‘exceptional circumstances”;"9? when the Audit application was in use,
Riposte would not record duplicate messages in the reconstructed message
store, and so transactions were not duplicated on the ARQ returns. However, the
HNG-X application did not identify or discard duplicated records, resulting in the
problem of duplicate transactions appearing in ARQ returns (Peak PC0200468).
Ms Thomas’ report further stated that an analysis conducted the day before (22
June 2010) had identified that one third of ARQ returns generated under the
HNG-X application had duplicate transactions.'%°
Ms Thomas’ report recorded that a discussion had taken place between Mr A.
Holmes, Mr Jenkins and Ms Thomas. The various actions agreed during that
discussion were:
a. Toreport Peaks PC0200468 and PC0194639 (see paragraph 123) to
Pat Lywood (Service Implementation Manager, Fujitsu) “to be
identified in CS Prayers as urgent’ .1%4
b. As an interim measure, to incorporate the unique identifier NUM in
ARQs to identify duplicate transactions / unique sequence
numbers. 195
c. Mr Jenkins was to draft a statement for management review detailing
the issue for transmission to POL.'%°
182 FUJ00097058
"88 FUJ00097058; FUJ00172183
194 FUJ00097058
195 FUJO0097058
196 FUJ00097058
Page 55 of 103
WITN06650300
WITN06650300
d.
197 FUJ00097058
198 FUJ00172043
199 FUJ00172043
A separate issue was identified whereby a duplicated transaction had
a different ‘NUM’, which Mr Jenkins agreed to review. '*” With respect
to this particular issue:
(i)
(ii)
In an email of 23 June 2010, Mr Jenkins wrote to Ms
Thomas and Mr A. Holmes, “/’ve had a look at the Mails
duplicate. I think it is OK. I’ve sent the extract to Richard
O'Neil in Crew to confirm that this is normal.”'% In a
summary of the duplicate transaction issue he wrote “we
have identified scenarios with Postal services transactions
where (details tbs) which result in different transactions
appearing to be duplicates”.'°°
On 24 June 2010, Mr Jenkins emailed both again, “!’ve
now confirmed that the Mails duplicate is OK’ and
amended the summary of the duplicate transaction issue:
“we have identified a scenario with Post Services
transactions where multiple, identical mails items are
accepted (ie the Quantity button is set to greater than 1),
but Postage Labels are printed for each individual item.
This results in separate transactions being generated for
each item, which are identical in the ARQ extracts (there is
another minor different [sic] in the raw data apart from the
Page 56 of 103
WITN06650300
WITN06650300
<Num< attribute, but this different attribute is not currently
included in the ARQ extract)”.?°°
122. The report also noted the need to identify the cases which had progressed to
prosecution since HNG-X had been live, and whether duplicate records had been
included in support evidence. The report references an initial review in which
West Byfleet (R v Misra) and Porters Avenue (R v Hosi) had been identified.2°"
123. In an internal Fujitsu email chain on 23 June 2010, Ms Thomas reported Peak
PC0200468 to Ms Lywood. In her email, Ms Thomas noted that the problem
recorded in Peak PC200468 was a “very significant problem”, and that “In a nut
shell the HNG-X application is not removing duplicate transactions (which may
have been recorded twice on the Audit Server) and they are appearing in the
AR@Q returns”.*°? She also noted that the steps taken following a previous Peak
(PC0194639) were supposed to highlight any duplication of records, but this did
not appear to be happening.?°2
124. Peak PC0194639 had been raised on 16 February 2010 by Andrew Mansfield
(Audit Development, Fujitsu) as there had “been cases recently of HNG-X
messages appearing in the audit files with duplicate JSNs (Journal Sequence
Numbers — message numbers)’ as a result of “an issue with the counter software”
but not being reported clearly to the audit client user. A fix was attributed to
the Peak “to highlight any problem with duplicate message numbers and make it
200 FJ00176310
201 FYJ00097058
202 FUJ00097038
203 FUJ00097038
204 FUJ00176307
Page 57 of 103
WITN06650300
WITN06650300
obvious to the user’.2°5 On 14 June 2010, the same fix was attributed to the Peak
again. Documents relating to the underlying issue in Peak PC01946339 (i.e., the
duplicate JSN issue) have been separately disclosed to the Inquiry.2°° Copied in
this email chain, Graham Allen (Applications Engineering Manager, Fujitsu)
asked Mr A. Holmes, Adam Spurgeon (Data Centre Development Manager,
Fujitsu), and Mr Mansfield whether there were fixes for the Peaks, and if not, how
long it would be before they would be ready.°” With respect to Peak PC0200468,
Mr Mansfield responded that Mr Barnes had started testing a fix to remove the
duplicated messages from the Horizon transaction data.?°° With respect to Peak
PC0194639, he noted that the Peak was less serious as the purpose of the Peak
was to clearly highlight where duplicates had been found.”° A fix would take two
to three days and would require a change to workstation applications only.
125. On 24 June 2010, Ms Thomas updated various members of the Fujitsu team
working on this issue with a more detailed analysis of the ARQs that had been
affected by email. This analysis stated that:
a. 112 ARQs had been affected;
b. 17 ARQs highlighted one or two instances which indicated bona fide
activity;
c. 7 ARQs were works in progress;
205 FUJ00176307
206 FUJ00165083, FUJ00165082; FUJ00157889; FUJ00170220; FUJ00093048; FUJ00142164;
FUJ00142160; FUJ00172032; FUJ00142157; FUJ00091790; FUJ00172090; FUJ00142162;
FUJ00142158; FUJ00097072; and FUJ00172031
207 FUJ00097038
208 FUJ00097038
209 FYJ00097038
Page 58 of 103
WITN06650300
WITN06650300
126.
127.
d. 12 ARQs had been involved in two court cases;
e. 8 ARQs had been involved in three court cases where witness
statements had been requested by POL, but not yet provided; and
f. Court activity was not known in relation to 76 ARQs.?'°
In the same email, Ms Thomas also provided an explanation of the problem and
suggested a workaround that had been drafted by Mr Jenkins for POL.2‘' Mr
Welsh responded that he saw no issue with the content of the report and was
pleased to have a workaround.?'2
Guy Wilkerson (Commercial Director, Fujitsu) responded by asking whether the
duplicated transactions would affect charges against Sub-Postmasters,*"? and
Mr Jenkins advised that “any detailed analysis of the finances of a Branch which
is done with duplicate transactions without realising that there are duplicates (and
so removing them) will give incorrect results” (including analysis of stock units,
cash on hand analysis and possibly the sum of all transactions).2"4 Mr Wilkerson
recommended that Alan D’Alvarez (Programme Director, Fujitsu) or Geoff Butts
(HNG-X Release 1 Programme Manager, Fujitsu) of the HNG-X team look at the
issue, and agreed with Ms Thomas that she should “hold off’ advising her
counterpart at POL of the issue “just until [they] get a chance to speak the HNG-
X team tomorrow”. 216
210 FYJ00097039
2"T FYJ00097039
212 FUJ00172044
213 FUJ00097039
214 FUJ00097039
218 FUJ00097039; FUJ00153121
Page 59 of 103
WITN06650300
WITN06650300
128.
129.
130.
After Ms Thomas confirmed to Mr Butts and Mr Wilkerson that she would not
communicate with POL regarding the issue, she expressed her assumption that
any duplicate records presented to court would need to be replaced.2'* Attached
to her email was a list of ARQs and witness statements, highlighting the then
ongoing cases relating to Porters Avenue (R. v Hosi) and West Byfleet (R. v
Misra).2'7 Additional Post Office branches (High Wycombe, Kirkoswald, New
Cheltenham, Castleton, Rinkfield, Shefford and Wokingham) were highlighted
green with the note: “1 or 2 instances which indicates bone fide activity’ .2"*
Mr Welsh forwarded Ms Thomas’ email to Ms Lywood describing the problem to
a wider Fujitsu team (including Mr D’Alvarez, Mr Allen, Mr Van Achte, Mr Butts,
Mr Thompson, Mark Andrews (Account Manager, Fujitsu), Vince Cochrane
(Head of Infrastructure and Migration, Royal Mail Group Account, Fujitsu), James
Davidson (Operations Executive, Fujitsu), and Debbie Richardson (Programme
Test Manager, Post Office Account, Fujitsu)).2"* Mr Welsh wrote of the problem,
“duplicate records that can not [sic] be differentiated are supplied as evidence.
Thus could allow for legal challenge to the integrity of the system”.?2° Mr Allen
copied Mr Jenkins into the email chain as he had “identified a workaround’ .22'
Mr Jenkins responded that he was not sure whether the workaround would be
acceptable to POL prior to review by the HNG-X team.?22
Mr Wilkerson forwarded the email exchanges on the issue to Peter Beresford
(Sourcing Manager, Fujitsu) (also on 25 June 2010), stating, “/ don’t want to be
26 FUJ00153121
217 FYJ00153122
218 FUJ00153122
28 FYJ00097038
220 FUJ00097038
22" FUJ00097038
222 FUJ00097038
Page 60 of 103
WITN06650300
WITN06650300
131.
132.
telling POL that they can’t rely on Branch numbers or that there’s a risk of more
complaints from Post masters. They could regard this as a serious issue and
delay acceptance on HNG-X. Geoff Butts tells me he’s aware and that they are
working on a permanent fix.”23 Mr Butts forwarded this to David Cooke (Service
Level Manager, Fujitsu) (with Mr Andrews, Mr Allen, Mr Welsh and Mr Jenkins in
copy) warning, “this may become an Acceptance Incident’ and explaining that
Mr Jenkins was assessing the viability of the workaround but there was no
capacity to include a fix within HNG-X Release 1.274 Mr Jenkins confirmed that
the workaround would require Fujitsu to inform POL of the problem and “since
this does relate to evidence used for prosecutions, [...] now we know there is an
issue we do need to tell POL about it asap.”2
On 30 June 2010, Ms Thomas emailed Sue Lowther (Head of Information
Security, POL), Mark Dinsdale (Security Programme Manager, POL) and Jane
Owen (Security Team Advisor, POL) informing them of the duplicate transaction
records problem.22° She explained that with the upgrade from Legacy Horizon
to HNG-X, the HNG-X data retrieval mechanism did not remove duplicated data
in the ARQ extracts and, from a review of the ARQs provided to POL since the
change to HNG-X, it had been indicated that approximately 35% of the ARQs
might contain some duplicate data.22”
Following a call with Mr Dinsdale, Ms Owen and Alan Simpson (Information
Security Incident Senior, POL) on 2 July 2010, Ms Thomas forwarded notes of
223 FYJ00097061
224 FUJ00097061
225 FUJ00097071
226 FUJ00121097
227 FUJ00121097
Page 61 of 103
WITN06650300
WITN06650300
133.
134.
the call to the Fujitsu team (Tom Lillywhite (Principal Security Consultant,
Fujitsu), Ms Munro, Mr Thompson, Mr Welsh and Mr Jenkins).?28 The notes of
the call confirmed that affected ARQs were used in the Porters Avenue and West
Byfleet cases (R. v Hosi and R. v Misra).?29 The notes state that ARQs had been
provided to the defence expert in the cases, but there was uncertainty about
whether the ARQs had been presented to the Court already, which may have
necessitated the provision of replacement data.?°
On 1 July 2010, Mr Allen followed up with Mr Mansfield regarding the fix for Peak
PC0200468.781 The Peak was with the Release Management Forum for
approval and scheduling, as it was reported that Mr Barnes had produced a fix
for HNG-X Release 1 which was ready to go.?°* He had added an impact
statement to the Peak, including a brief statement on testing.?°> Mr Mansfield
further noted that a possible workaround, being discussed by Ms Thomas and
POL, involved “modifying the audit queries so that the message numbers are
included in the output to the spreadsheets (currently they are not). This would
allow the duplicate messages to be identified and removed by running a macro
on the final spreadsheet generated by the application. "25+
Ms Thomas asked Mr Welsh for his help with Peak PC0200468 on 5 July 2010
as “POL have gone to POL legal for guidance and further returns have been
identified this morning as bound for Court.”235 Mr Welsh forwarded this request
228 FUJ00121097
229 FYJ00121097
280 FUJ00121097
231 FUJ00172046
282 FUJ00172046
283 FUJ00172046
234 FUJ00172046
285 FUJ00172046
Page 62 of 103
WITN06650300
WITN06650300
to Mr Butts and Sheila Bamber (Release Manager, Fujitsu) (copying Mr
Lillywhite) asking for confirmation as to when the fix could be fitted, as “there are
more court cases pending and whilst the Briefing to the Investigation has taken
place they are coming back requesting help due to the level of activity and
nervousness regarding the current work-a-round.”*°° Ms Bamber responded that
it would take three days’ work to take the fix with HNG-X Release 2.257. Mr
Mansfield confirmed that Mr Barnes could merge the fix into the HNG-X Release
2 code and deliver the fix by 7 July 2010.298 Chris Hammond (Major Release
Manager, Fujitsu) confirmed this would be deployed on 10 July 2010 with all other
HNG-X Request 2 baselines.”°9 On 7 July 2010, Mr Barnes emailed a wider
Fujitsu team (Ms Bamber, Mr A. Holmes, Mr Mansfield, Mr Spurgeon, Peter
Okely (Application Services, Fujitsu), Vijesh Pandya (Integration Team Leader,
Fujitsu), Matthew Swain (Integration Team, Post Office Account, Fujitsu) and
Nigel Taylor (Integration Team, Fujitsu)) stating that an Audit baseline for Peak
PC0200468 was ready for build.”4°
135. On 7 July 2010, FSL and POL agreed to amend the standard FSL witness
statement (the “Amended Standard Witness Statement’) that accompanied
ARQ data to include the wording: “the duplication of audited records has not, in
any way, affected actual physical transactions recorded on any counter at any
outlet. The duplication of records has occurred during the auditing process when
records were in the process of being recorded purely for audit purposes from the
correspondence servers to the audit servers. It should be noted that this
236 FUJ00172046
237 FYJ00172046
238 FYJ00172046
239 FUJ00172046
240 FUJ00172045
Page 63 of 103
WITN06650300
WITN06650300
136.
137.
duplication of data in the audit stream has always been happening. However the
Horizon retrieval process automatically discarded duplicate records before
creating the ARQ spreadsheets, while the current HNG-X retrieval process for
Horizon data does not do so”.2*1
Ms Thomas and Mr Jenkins reviewed this updated language and shared it with
Mr Lillywhite and Mr Wilkerson.242, POL had also suggested that Mr Jenkins
complete the witness statement as the Fujitsu expert witness, to which Ms
Thomas commented that this would not fit into the current service agreement and
it would not be feasible for him to provide all witness statements going ahead.”4?
Attached to her email was a standard witness statement including the modified
wording regarding duplicate transactions,”44 which had previously been reviewed
by Mr Jenkins.245
In the 7 July 2010 email correspondence, POL also requested that Fujitsu
“provide a witness statement to quantity the above that [they] could attach to
each case (as appropriate), and treat each case where this is not accepted
individually’ .24° On 8 July 2010, Ms Thomas informed Mr Jenkins that she and
Mr Wilkerson had discussed an “additional statement regarding duplicate
records”.247 Mr Wilkerson had initially suggested that Ms Thomas include the
following statement in her witness statement: “The Audit Mechanism cannot alter
the base information and therefore a re-running of the audit process will always
produce the same result’, which she incorporated in an updated pro forma
241 FUJ00122901
242 FUJ00122901
243 FYJ00122901
244 FUJ00122902
245 FUJ00122899
248 FUJ00122901
247 FUJ00122907
Page 64 of 103
WITN06650300
WITN06650300
witness statement.*4® However, Ms Thomas noted to Mr Jenkins that that Mr
Wilkerson was of the view that “if [POL] wanted an expert witness statement we
should provide [sic]”. In addition, Mr Wilkerson had “said that a CR would need
to be raised by POL but when he realised it would only be for a short while and
that we needed one pretty soon for Kirkoswald he thought] we could provide this
FOC."*49 Ms Thomas explained that if Mr Jenkins was happy to provide such a
statement (the "Draft Additional Duplicate Transactions Witness
Statement"), she would remove the extract Mr Wilkerson had proposed from the
standard statement.?5°
138. In addition to various drafting notes, the Amended Standard Witness Statement
included the below wording, which explains that HNG-X did not de-duplicate
records in ARQ extracts as Legacy Horizon had:?*":
"With Horizon counters, the mechanism by which Data is audited has always
worked on the principle that it is acceptable to audit the same data more than
once — in particular if in doubt as to whether or not it has been previously audited
successfully. The Mechanism used on Horizon to retrieve the audit data took
this into account and only presented one instance of such duplicate data in the
ARQ extracts. The Audit Mechanism cannot alter the base information and
therefore a re-running of the audit process will always produce the same result.
In January 2010 a new HNG-X application was introduced to filter transaction
records for presentation to Post Office Limited. It has recently been noticed that
248 FUJ00122908
249 FYJ00122907
250 FUJ00122907
251 FUJ0017631 1; FUJ00122907
Page 65 of 103
WITN06650300
WITN06650300
139.
140.
this HNG-X retrieval mechanism does not remove such duplicates. An
enhancement to the extraction toolset will be developed, tested and deployed
and will remove such duplicate data in the future. However until this
enhancement is deployed, there is a possibility that data is duplicated. The
reliable way to identify a duplicate transaction is to use the <Num> attribute that
is used to generate the unique sequence numbers. This will be included in all
future transaction record returns until the retrieval mechanism is enhanced. A
semi-automated process to copy the returned data, and then to identify and
remove any duplicated records which may be present from this copy by using
the <NUM> attribute, has been agreed with Post Office Limited for use in the
interim period. It is emphasised that the duplication of audited records has not,
in any way, affected actual physical transactions recorded on any counter at
any outlet. The duplication of records has occurred during the auditing process
when records were in the process of being recorded purely for audit purposes
from the correspondence servers to the audit server."252
Later the same day (8 July 2010), Ms Thomas emailed Mr Jenkins that she had
prepared a Draft Additional Duplicate Transactions Witness Statement for him,
253 to which he responded, “that looks fine and I would be happy to sign that if
needed”>4. The statement included the wording set out at paragraph 138 above.
In response to POL’s email on 7 July 2010 agreeing to the amended witness
statement wording and the introduction of the Draft Additional Duplicate
Transactions Witness Statement, Ms Thomas emailed to POL a copy of the Draft
252 FUJ00122908
258 FUJ00122914; FUJ00122915
254 FUJ00176312
Page 66 of 103
WITN06650300
WITN06650300
141.
Additional Duplicate Transactions Witness Statement for POL’s review.2°> Ms
Thomas forwarded this correspondence to John Longman (Security Advisor,
POL), who responded that he was happy with the statement as “it confirms that
[the duplicate transactions issue] has no affect [sic] on Horizon’s accuracy. I
have added an extra paragraph to tie it in with the trial of Seema Misra and
confirm that only ARQ447 has any duplications within the disc you produced as
PT/02."255 Ms Thomas forwarded Mr Longman’s comment to Mr Jenkins with a
revised copy of the Draft Additional Duplicate Transactions Witness
Statement,”°” noting that the additions made related directly to West Byfleet (R v
Misra).2°8 Mr Jenkins responded “This look[s] fine. Do you want me to pop up
and sign it with you as a witness?’ and Ms Thomas confirmed that was okay.7°9
On 15 July 2010, Ms Thomas replied to Mr Longman “we’re happy with your
addition” and arranged for postage of the signed Draft Additional Duplicate
Transactions Witness Statement to POL.7°
The fix to Peak PC0200468 was successfully deployed with HNG-X Release 2
and the Peak was closed on 1 September 2010. Peak PC0194639 was closed
on 16 December 2010.76"
2014 Incident
142. On 19 May 2014, Kathryn Alexander (Network Support BAU Area Manager,
POL) emailed Alistair Kay (Project Manager, Post Office Account, Fujitsu) asking
258 FUJ00122928; FUJO0122929
258 FUJ00122928
257 FYJ00122929
258 FYJ00122928; FUJ00122929
259 FUJ00153145; FUJO0176313
260 FUJ00153133
261 FUJ00176307
Page 67 of 103
WITN06650300
WITN06650300
for help as ARQs for Caereithin FAD 166642 “appear to have duplications on
certain months and from certain time in the day [sic]’ 2°?
143. On 20 May 2014, Jason Muir (Security Operations Analyst, Fujitsu) emailed Mr
Barnes and Mr A. Holmes for advice as six ARQs he had run “returned duplicate
data’.?®3 Mr Barnes responded, “duplicates are OK for Horizon — sometimes
multiple copies of the data were stored” .?°+
144. On 21 May 2014, Mr Muir asked Mr Barnes to prepare an explanation for Fujitsu
to share with POL explaining the reason for duplicate data that had arisen in ARQ
data sent and “alleviate any fears that may have arisen over the data”.26° Mr Muir
reported that the latest occurrence of this was in ARQ 051-056 Caereithin FAD
166642 on 9 May 2014. Mr Barnes described the issue (with Mr Kay, Andy Dunks
(Associate Service Manager, Fujitsu), Kumudu Amaratunga (Security
Operations Manager, Royal Mail Group Account, Fujitsu), Mr Spurgeon and
Brain Lea (Software and Solution Developer, Post Office Account, Fujitsu) in
copy) as being a problem with Riposte where transactions were occasionally
duplicated in different files, adding “it is almost certain that all the other duplicates
are identical too but to be thorough you should really check each and every
one.”26° He emphasised “The key point is duplicate identical transactions are a
known issue which does not matter. However you need to confirm that you have
262 FYJ00176323
263 FUJ00176322
264 FUJ00176322
265 FUJ00172085
266 FUJ00172085
Page 68 of 103
WITN06650300
WITN06650300
not got duplicate transactions which are not identical because that would be a
new issue of concern”.?67
145. Mr Kay asked Mr A. Holmes whether he had anything to add to Mr Barnes’
statement, bearing in mind this would be reported to POL.?°° Mr A. Holmes added
more detail on the reason for duplicates: “Under old Horizon, Riposte Audit data
was extracted from the correspondence servers (central transaction data
repositories) by an agent harvester process. These were then written as flat files
which were picked up and secured by the audit server. The design of this
harvester was such that duplicate records were allowed to be included in the
files. This would only happen in exception conditions and basically operated as
a fail safe i.e. err on the side of including duplicates rather than potentially losing
data. The design of this harvester, and the fact that duplicates may appear is
described in a very old Horizon document: “High Level Design of Common
Agents” AD/DES/042 §2”.2°
146. On 22 May 2014, Mr Kay replied to Ms Alexander's email of 19 May 2014, “we
identified two files containing the duplicate transactions and confirmed that the
first 4 duplicates and the last duplicate were all identical transactions. It is almost
certain that all the other duplicates are identical too.””° He further included the
explanations provided by Mr Barnes at 144, specifically that “the key point is
duplicate identical transactions are a known issue that does not matter’, and by
Mr A. Holmes as outlined in 145 above.2”'
207 FUJ00172085
288 FUJ00172085
269 FUJ00172085
270 FUJ00176323
271 FUJ00176323
Page 69 of 103
WITN06650300
WITN06650300
2016 Incidents
147.
148.
On 7 April 2016, Mr Dunks emailed Mr Barnes and Mr Muir (with Steve Goddard
(Software Development Manager, Fujitsu) copied) noting that a duplicate was
showing on a summary sheet, copied in the email.2”? Mr Barnes responded, “it is
something to be concerned about’, and a Peak should be raised.?”3 He added “/
am copying all the transaction files to the local disk of an audit workstation right
now. I have already got on a USB stick two files which contain the first set of
duplicates so that can be examined [...] I have studied the first message,
5026965, in detail and it looks like the complete and same transaction is in both
files.”274
Mr Dunks raised Peak PC0250729 on 8 April 2016 with the comment “some of
the data spread sheets that are generated via the audit retrieval process are
showing a number of duplicates on the summary sheef’.2”> The Peak was
allocated to Mr Barnes who added “the reason for these duplicates needs to be
identified before the prosecution team submits its spreadsheets”.2”° On 11 April
2016, Mr Barnes investigated the ‘AUD’ files generated on the 24 October 2015,
and found they had gathered twice “due to an error in the process of migrating
the share on which they lay from one platform to another’.2”’ Peak PC0250729
was closed on 26 April 2016 by Mr Muir with the comment “ARQ data sent to
POL"278
272 FUJ00172086
273 FUJ00172086
274 FUJ00172086
275 FYJ00173072
276 FUJ00173072
277 FUJ00173072
278 FUJ00173072
Page 70 of 103
WITN06650300
WITN06650300
WITN06650300
WITN06650300
149. On 14 July 2016 Farzin Denbali (Security Operations Analyst Post Office
Account, Fujitsu) wrote to Mr A. Holmes, “as we discussed a few days ago, there
are a large number of duplicate records in an ARQ output for FAD 173458” and
asked him to investigate the issue.?”? Mr Denbali also asked whether he could
send the ARQ to POL, to which Mr A. Holmes replied “no, don’t send the stuff to
POL until we know what’s going on.”®° Later the same day (14 July 2016) Mr
Denbali shared a file with Mr A. Holmes in an email chain entitled ‘duplicate
records in ARQ output’.261
150. On 14 July 2016, Mr Goddard forwarded Mr Dunks’ email and the subsequent
correspondence regarding duplicate transactions (document FUJ00172086) to
Mr A. Holmes and noted “it turned out that the share on which they sat was
migrated that day and the audit configuration file was modified by hand with two
separate paths to the same migrated share which caused the files to be copied
twice. It was incorrect procedure to modify the audit configuration files by
hand.”?82 Mr A. Holmes forwarded the correspondence again, to Mr Denbali,
adding “Farzin — your problem with duplicates in an ARQ is another occurrence
of the problem detailed below’, and asked Mr Dunks how the issue was “worked
around last time”.?8> Mr Dunks could not remember and deferred to Mr Barnes,
but Mr A. Holmes responded “there a couple of approaches we could take to
work around this and get the ARQ completed, but I think that it is better that we
wait until Gerald comes back on Monday and check what we did last time so we
can adopt a consistent approach. This problem is going to occur again for any
279 FUJ00176324
280 FYJ00176326
281 FUJ00176327; FUJO0176328
282 FUJ00172086
283 FUJ00172086
Page 71 of 103
ARQs that overlap the date range 23rd — 25th Oct 2015. It would be a good idea
to add this, and the eventual workaround, to your list of ‘problem cases’ for
reference the next time it crops up.”284
151. On 15 July 2016, Mr Muir responded to the email chain attaching emails relating
to the 2014 Incident.”8° He wrote that the reason for the duplicate records in 2014
appeared to be different to the reason for the 2016 duplicate records, but that
Fujitsu “never highlighted the duplicates to them back in 2014 and I don’t recall
they ever queried it’.28° While Mr Muir made this statement at the time, it is
apparent from the contents of FUJ00176323 that Fujitsu did provide POL with an
explanation as to the duplications in the 2014 Incident.2°7 Mr Muir suggested
completing a check that had been suggested by Mr Barnes at the time of the
2014 Incident, and confirm that the duplicate transactions were identical, as if
they were not identical, this would be “a new issue of concern”.28° Mr A. Holmes
responded “the reason behind the 2014 duplicates will be different to this
case.”289
152. On 18 July 2016, Mr Barnes replied to Mr Dunks’ email of 14 July 2016 asking
him how the 2014 Incident had been resolved, “if you redo the queries as slow
ARQs then you can produce the spreadsheets but with the duplicates listed. You
then need a separate explanation to be [sent] to the Post Office as to why there
are dupplicates [sic] in this case.”9° Mr Muir replied, “far as we can see the
284 FUJ00172086
285 FUJ00172086; FUJ00172087
286 FUJ00172086
287 FYJ00176323
288 FUJ00172086
289 FUJ00176329
290 FUJ00216439
Page 72 of 103
WITN06650300
WITN06650300
duplicate records are already showing in the ARQ data. We will send the data to
POL with a covering statement to explain why there are duplicate records.”®"
HISTORIC GAPS IN ARQ DATA
153.
154.
In preparing this corporate statement, Fujitsu has identified a small number of
documents in relation to this topic that had not previously been disclosed to the
Inquiry. These documents are exhibited to this statement.?°2
On 1 December 2021, Paul Gauntlett (Customer Solution Architect, Fujitsu)
emailed Steven Browell (Chief Information Security Officer, Fujitsu), with other
Fujitsu personnel in copy (namely Gerald Barnes (Software Engineer, Fujitsu),
Manisha Mistry (Service Delivery Manager, Fujitsu) and Phil Boardman (Service
Architect, Fujitsu)) regarding a “Historical Issue with Audit Data” (the “Historic
ARQ Gaps Issue’). Mr Gauntlett explained that, during a meeting with John Nelis
(Problem Manager, POL) and Dean Bessell (POL) the previous day, an issue
was discussed regarding audit data gathered prior to 2010 existing only on one
audit server and not both. Mr Gauntlett was asked to write up what was
communicated to POL during this meeting, so POL could discuss this with their
legal team. Mr Gauntlett prepared a draft write-up, in relation to which Mr
Boardman, Mr Barnes and Mr Browell provided input.29° According to this write-
up:294
a. Riposte was used in Legacy Horizon “to gather audit transactions from
all the Post Office counters... deployed in the Bootle & Wigan data
29" FUJ00216439
282 See the documents listed at Exhibit numbers 199 to 270 in this regard
298 FUJ00176507
294 FUJ00176508
Page 73 of 103
WITN06650300
WITN06650300
WITN06650300
WITN06650300
centres and on the Post Office counters... Each evening the Riposte
databases in each data centre were copied to the local Audit Archive
— this was done by the local Harvester...there were occasions when
the Harvester in one data centre would exhibit faults which would
interrupt the process of copying the transaction data to the Audit
Archive resulting in some data not being copied to the Audit Archive.
There is no recorded instance of both harvesters failing at the same.
Although data may be missing from one Audit Archive server, it will be
present on the other Audit Archive in the other data centre (and the
ARQ process checks for this when retrieval requests are
processed)”.
b. Riposte was not deployed in the HNG-X solution, and the Wigan and
Bootle Data Centres were migrated to the IRE11 and IRE19 audit
servers. From 2010 onwards, “all transaction and non-transaction files
[were] gathered in IRE11 and harvested to the IRE11 Audit Archive
only. Live data is automatically mirrored to IRE19 — including to the
IRE19 Audit Archive. Under HNG-X, all audit files (transaction and
non-transaction) are consistent across both Audit Archive servers’.
c. In terms of the potential impact of this issue on ARQ Reliability, “ARQ
queries are run against one of the Audit Archives. In the unlikely event
that the results show any gaps, the query is run against the other Audit
Archive to add the missing data and provide a holistic response to the
requestor. There have been no reported instances of gaps in ARQ
295 See for example FUJO0176543
Page 74 of 103
WITN06650300
WITN06650300
retrievals once a query has been executed against both Audit
Archives”.
155. On the same day, Simon Oldnall (Horizon IT Director, POL) emailed Mr Browell
noting that “/t/he PCI team have flagged an issue to me around potential issues
with the data integrity of the audit San. The initial feedback I’m hearing is that
this is a historical issue, however this does raise a number of concerns”. Mr
Oldnall asked Mr Browell to provide Mr Bessell with a “more detailed brief on
what the issues were, the history of these issues and any ongoing concerns that
may exist’.2°6
156. On 3 December 2021, Mr Browell circulated his working notes on the Historic
Gaps Issue to a number of Fujitsu recipients, including Mr Barnes, Mr Gauntlett,
Jason Muir (Information Security Manager, Fujitsu) and Geoff Baker (Information
Security Manager, Enterprise & Cyber Security, EMEIA, Fujitsu). Mr Browell
explained that ARQ retrievals can be run as a “Fast’ retrieval or a “Slow” retrieval.
The Slow ARQ process would look for and handle gaps in the sequencing of
JSNs. The Fast ARQ process, in contrast, “will abort if the sequence of JSNs has
gaps’ (see CP 4867, FUJ00155474%"). If the operator of the ARQ retrieval was
alerted to gaps?%*:
a. In Legacy Horizon, “they should rerun the SLOW query against the
other archive as it is unlikely that BOTH archives will have gaps”.
b. In HNG-X, “they should raise an Incident as this is not expected’.
296 FUJ00176487
297 FUJ00155474
298 FUJ00176516
Page 75 of 103
157.
158.
c. In relation to both (a) and (b) above, Mr Browell’s notes included a
comment that “/this relies on [the operator] deciding to do this and
could be subject to human errory’.
On 6 December 2002, Mr Browell sent an email to Fujitsu personnel including Mr
Barnes, Mr Muir and Mr Gauntlett with the subject “CONFIDENTIAL - Audit
Archive — The action plan?®°. According to this email, Fujitsu’s investigation into
potential gaps in the audit data included (i) confirming whether incomplete ARQ
transaction responses have ever been sent to POL, and whether POL have
always been aware of any such gaps, (ii) confirming whether there were any
gaps in the audit data being stored by Fujitsu at that time, (iii) understanding how
harvester issues were recorded/actioned, and if there were any current harvester
issues that needed to be addressed, and (iv) confirming that Fujitsu is confident
in the design of the HNG-X harvester.
Mr Browell provided Fujitsu’s response to the Historic ARQ Gaps Issue on 11
January 20225°. This included a summary of the issue in similar terms to
paragraph 154 above. In relation to the ARQ process, Mr Browell’s email stated:
a. “There have been a small number of suspected gap issues
investigated. It is understood that all investigations were resolved,
including by re-running the ARQ against the second data store
(Horizon) or by extending the date ranges (HNG-X) resulting in no
gaps in any ARQ responses."
299 FUJ00176528
300 FUJ00176487
Page 76 of 103
WITN06650300
WITN06650300
“The ARQ process includes the adding of warnings for gaps in the
resulting files sent to POL. If there had been any gaps, POL would
have been made aware within the ARQ responses supplied.”
“One known gap (for a specific FAD and date) was identified as part
of ARQ extracts performed against the Bootle archive. That gap is not
in the Wigan audit archive though. Therefore, Fujitsu is not aware of
any gaps in the audit archive data for Horizon when both archives are
used’.
“There have been no gaps identified as part of any ARQ extracts
performed against the HNG-X audit archive’.
159. Mr Oldnall responded to Mr Browell’s email on 14 February 2022 seeking further
information including (i) an explanation of the harvesting process and the
parameters used when running ARQs, and (ii) an explanation of how gaps in the
audit archives manifest in the ARQ data sent to POL and how they are brought
to POL's attention.°°' Mr Browell responded to these questions on 17 March
2022, as follows:
301 FUJ00176487
As a general point, “/n the unlikely event that there are gaps in the
Audit Archive, the ARQ process would spot it and if this could not be
reconciled by using the second Audit Archive store, POL would be
notified of the true gap found. POL would be aware of the gap”.
In relation to the harvesting process, this “archives the data from
BRDB to flat files which are then stored in the Audit Archive. The
Page 77 of 103
WITN06650300
WITN06650300
method of doing this has changed over the years as data storage
changed”.
c. _ Inrelation to the parameters used when running ARQs, these are “the
provided FAD code and date range requested for the type of content
requested (typically branch transaction and event data”.
d. In relation to how gaps are identified and notified to POL, the
application used to run the ARQ requests, AEClient, “checks the
transaction sequence numbers which are the unique and sequential
identifiers of transactions made at a branch. If any gaps in the
sequence numbers are identified, this signals that there is a gap in the
IRE11 audit archive data. The presence of a gap is presented to the
operator...If this happens for pre HNG-X then the audit archive in
IRE19 is checked. If that also shows a gap for the matching search
criteria, then a true gap will have been found. If a true gap had been
found, then the ARQ response spreadsheet would highlight this to
POL".
OTHER POTENTIAL ISSUES RELATING TO ARQ DATA
160. In addition to the issues identified in this statement, I understand that Fujitsu
disclosed to the Inquiry 102 documents from its Peak, PinICL and Known Error
Log (“KEL”) databases on 14 July 2023. These documents were identified by
Fujitsu as records of incidents referring to the ARQ process in the context of court
proceedings.
Page 78 of 103
WITN06650300
WITN06650300
161. Many of the documents disclosed relate to system changes and support issues
rather than issues with the ARQ data that was extracted and provided to POL.
However, there are certain Peaks set out below that could be relevant to the
issue of ARQ Reliability. In the time available, Fujitsu has not been able to
investigate these Peaks in any detail but would be willing to do so if the Inquiry
requests that it does.
a.
302 FUJ00172096
$3 FYJ00173183
304 FUJ00172221
905 FUJ00172215
306 FUJ00173184
PC0088573° (Audit Data Extraction Problems) relates to a risk that
incorrect data may be used in an ARQ report for branches with two
leading zeros in their FAD codes.
PC0272681°" records that only “fast ARQs” can be reliably run in the
evening when the SQL server is shutdown. Running “slow ARQs” at
this time may give rise to unpredictable results.
PC0205806°% relates to duplicate transaction records not being
reported to POL in ARQ reports.
PC0206923°° relates to errors in the filtering process, meaning that
data was returned for one day less than the data range specified.
Fujitsu's prosecution support service stopped processing ARQs until
this was resolved.
PC0280793° relates to issues experienced when running audit
queries for an entire month.
Page 79 of 103
WITN06650300
WITN06650300
WITN06650300
WITN06650300
f. I PC0241862°" relates to issues with audit data accidentally “purging”.
g. PC0225656°° relates to a loophole which could potentially result in
missing transactions on ARQ reports.
h. PC0211833%° relates to a number of ARQ returns which did not
identify transaction reversals.
Statement of Truth
I believe the content of this statement to be true.
Signed: }
Dated: 14 September 2023
307 FUJ00173063
308 FUJ00172286
309 FUJ00172241
Page 80 of 103
INDEX TO THE THIRD CORPORATE STATEMENT OF
FUJITSU SERVICES LIMITED
eae Document Description Control No. URN
CONTRACTUAL BACKGROUND
1. Codified Agreement between Fujitsu and I POINQ0006242F I FUJ00000071
POL dated 28 July 1999
2. Codified Agreement between Fujitsu and I POINQ0003098F I FUJ00000003
POL dated 23 November 2020
THE ARQ SPREADSHEET
3. ARQ Spreadsheet of Marine Drive VIS00011623 LCAS0001383
transactions and events from 2 February I pages 17 and 18 I pages 17 and
2004 only 18 only
4. CCN 0423a dated 1 July 1999 POINQOO06565F I FUJ00000394
5. Service Architecture Design Document POINQ0007602F I FUJ00001431
Change Control Notices Applied v 6.0
dated 23 October 2000
6. Audit Trail Functional Specification v 3.0 I POINQ0007489F I FUJ00001318
dated 1 July 1999
7. ARQ 0506/405 dated 26 October 2005 POINQ0158757F I FUJ00152562
8. ARQ 0506/421 - 423 dated 4 November I POINQ0158759F I FUJ00152564
2005
9. Email from ‘Brian’ to ‘Peter’ with subject POINQ0158769F I FUJ00152574
line “FW: Torquay Road ARQ 0506/368”
dated 23 December 2005
10. Email from ‘Brian’ to ‘Penny’ with subject I POINQ0158764F I FUJ00152569
line “FW: Mr L Castleton - Marine Drive
Post Office, Bridlington” dated 22
November 2005
11. Network Banking Management of POINQ0158403F I FUJ00152209
Prosecution Support Procedure v 2.0
dated 29 February 2005
12. Audit Data Extraction Process v 3.0 POINQ0230499F I FUJ00176265
dated 1 February 2005
AVAILABILITY OF ARQ DATA
13. CCN 1122 dated 5 January 2004 POINQ0007089F I FUJ00000918
14, CT 1542 dated 29 May 2014 POINQ0163141F I FUJ00156946
15. CT 1922 dated 15 October 2015 POINQ0230513F I FUJ00176279
16. CT 2616a dated 10 September 2018 POINQ0230508F I FUJ00176274
17. CWO 0251a dated 2 July 2020 POINQ0230512F I FUJ00176278
18. CWO 0395b dated 26 March 2021 POINQ0230509F I FUJ00176275
19. CWO 0560a dated 28 February 2022 POINQ0230510F I FUJ00176276
20. CWO 0725 dated 23 February 2023 POINQ0230511F I FUJ00176277
21. F extract of Moss: for Marine Drive for the POINQO178138F I; Fusoo171957_001 I
period 12 March 2004 to 2 April 2004 orem
22. Excel spreadsheet containing data from POINQ0178139F I FUJ00171958
Marine Drive for the period 12 March
2004 to 2 April 2004
Page 81 of 103
WITN06650300
WITN06650300
ee Document Description Control No. URN
BROKEN AUDIT TRAIL
23. Letter from J. Holmes to S. Kinghorn with I POINQ0178140F I FUJ00171959
subject “Broken Audit Trail” dated 23 May
2001
24. Document titled “PROBLEM POINQ0158378F I FUJ00152184
REF PC0066318 - Incomplete TMS Audit
Trail” dated 24 May 2001
25. Network Banking Internal Audit POINQ0178148F I FUJ00171967
Requirements v 2.0 dated 13 July 2001
26. Technical Environment Description v 4.8 I POINQ0207221F I FUJ00201501
dated 22 October 2002
27. PC0066318 POINQ0178274F I FUJ00172093
28. Letter from C. Lenton-Smith to K. Baines I POINQ0230531F I FUJ00176297
with subject “Lost Data and Audit
Requests” dated 15 August 2002
29. Customer Service Monthly Report - July POINQ0126708F I FUJ00120516
2001 v 1.0 dated 31 July 2001
30. Document titled “ICL Pathway Monthly POINQ0122204F I FUJ00116033
Progress Report - September 2001”
dated 10 October 2001
31. Contract Administration Meeting Minutes I POINQ0230519F I FUJ00176285
dated 23 May 2001
32. Customer Service Monthly Report - May I POINQ0230516F I FUJ00176282
2001 v 1.0 dated 30 May 2001
33. Pathway/Consignia Audit & Security POINQ0178152F I FUJ00171971
Panel Meeting Minutes dated 18 June
2001
34. Letter from C. Lenton-Smith to K. Baines =I POINQ0230530F I FUJ00176296
with subject “Lost Data and Audit
Requests” dated 15 February 2002
35. Email from A. Clarke to ZDL UKS POINQ0161723F I FUJ00155529
PATCPImpactNotOnline, J. Holmes and
C. Lenton-Smith with subject “FOR
IMPACT - URGENT- CP 3061 - Rebuild
broken Audit Trail due to Missing /
Damaged tapes” dated 27 July 2001
36. CP 3061 POINQ0161724F I FUJ00155530
37. Programme Change Control Board POINQ0230520F I FUJ00176286
Meeting Minutes dated 2 August 2001
38. Programme Change Control Board POINQ0230521F I FUJ00176287
Meeting Minutes dated 16 August 2001
39. Change Control Board Meeting Minutes POINQ0230522F I FUJ00176288
dated 20 August 2001
40. Programme Change Control Board POINQ0230525F I FUJ00176291
Meeting Minutes dated 18 October 2001
41. Programme Change Control Board POINQ0230526F I FUJ00176292
Meeting Minutes dated 25 October 2001
42. Document titled “Audit Trail Break - POINQ0230528F I FUJ00176294
Pathway Position” dated 5 December
2001
Page 82 of 103
WITN06650300
WITN06650300
Exhibit
no.
Document Description
Control No.
URN
43.
Audit of Horizon Data Centres and
Belfast Operations Centre v 2.0 dated 21
November 2001
POINQO086685F
FUJ00080514
44.
Pathway/Consignia Audit & Security
Panel Meeting Minutes dated 12 October
2001
POINQ0178153F
FUJ00171972
45.
Email chain from J. Holmes to R. Dhesi
copying S. Gardiner with subject “RE:
Consignia IA Involvement” dated 26
October 2001
POINQ0178154F
FUJ00171973
46.
Email from J. Holmes to R. Dhesi with
subject “Data Centre Audit Report” dated
27 November 2001
POINQ0178155F
FUJ00171974
47.
Document titled Review of Horizon Data
Centres -November 2001 dated January
2001
POINQ0178160F
FUJ00171979
48.
Email from J. Holmes to R. Dhesi copying
C. Lenton-Smith and G. Hooper with
subject “Data Centre Report - POL
Distribution” dated 6 February 2002
POINQ0178164F
FUJ00171983
49.
Email from R. Dhesi to J. Holmes with
subject “Audit Report - Data Centre
Review” dated 31 January 2022
POINQ0178159F
FUJ00171978
50.
Email chain from J. Holmes to R. Dhesi
with subject “RE: Audit Report — Data
Centre Review” dated 31 January 2002
POINQ0178161F
FUJ00171980
51.
Email from R. Dhesi to J. Holmes copying
H. Stewart with subject “RE: Data Centre
Report — POL Distribution” dated 28
February 2002
POINQ0178162F
FUJ00171981
52.
Email chain from J. Holmes to R. Dhesi
copying G. Hooper and H. Stewart with
subject “Data Centre Report - POL
Distribution” dated 11 February 2002
POINQ0178163F
FUJ00171982
53.
Email chain from J. Holmes to C. Lenton-
Smith with subject “Subject to Legal
Professional Privilege - in contemplation
of Legal proceedings - RE: Audit Report”
dated 13 February 2002
POINQ0178151F
FUJ00171970
Email chain from J. Holmes to C. Lenton-
Smith and M. Blewett copying G. Hooper
with subject “Subject to Legal
Professional Privilege - in contemplation
of Legal proceedings - RE: The missing
date” dated 11 February 2002
POINQ0178149F
FUJ00171968
55.
Document titled “Review of Horizon Data
Centres - November 2001” dated January
2001
POINQ0178165F
FUJ00171984
56.
Email from J. Holmes to C. Lenton-Smith
and G. Hooper with subject “Data Centre
Report” dated 31 January 2002
POINQ0230989F
FUJ00176754
Page 83 of 103
WITN06650300
WITN06650300
Exhibit
no.
Document Description
Control No.
URN
57.
Email from J. Holmes to G. Hooper and
C. Lenton-Smith with subject “DC Audit
Report Issue” dated 5 February 2002
POINQ0230991F
FUJ00176756
58.
Letter from C. Lenton-Smith to K. Baines
with subject “Loss Data & Audit
Requests” dated 19 September 2001
POINQ0230524F
FUJ00176290
59.
Document titled ICL Pathway Monthly
Progress Report - May 2001 dated 6
June 2001
POINQ0122147F
FUJ00115976
60.
Letter from C. Lenton-Smith to K. Baines
with subject “Lost Data” dated 7 August
2001
POINQ0230514F
FUJ00176280
61.
Letter from K. Baines to C. Lenton-Smith
dated 29 August 2001
POINQ0230523F
FUJ00176289
62.
Letter from K. Baines to C. Lenton-Smith
with subject “Lost Data & Audit Requests”
dated 6 February 2002
POINQ0230529F
FUJ00176295
63.
Email chain from C. Lenton-Smith to I.
Monaghan with subject “RE: Settlement
of disputes” dated 16 December 2002
POINQ0230539F
FUJ00176305
64.
Letter from M. Hannon to C. Lenton-
Smith with subject “Lost Data and Audit
Trail” dated 30 August 2002
POINQ0230532F
FUJ00176298
65.
Letter from C. Lenton-Smith to M.
Hannon with subject “Lost Data and Audit
Trail” dated 13 September 2002
POINQ0230533F
FUJ00176299
66.
Letter from C. Lenton-Smith to M.
Hannon with subject “Lost Data and Audit
Trail” dated 13 September 2002
POINQ0230534F
FUJ00176300
67.
Email from C. Lenton-Smith to H. Forrest
copying P. Purewal with subject “Lost
Audit data changes” dated 3 October
2002
POINQ0230535F
FUJ00176301
68.
Email chain from J. Holmes to G. Hooper
and P. Lywood with subject “Single data
centre” dated 24 October 2002
POINQ0230537F
FUJ00176303
69.
Email from A. Clarke to ZDL UKS
PATCCBMinutes/Agenda, zDL UKS
PATPCCBMinutes/Agenda, and zDL
UKS PATCPCommunication copying in
P. Purewal with subject “CCN 1019
(Release:N/A) - APPROVED” dated 21
November 2002
POINQ0230538F
FUJ00176304
OMISSIONS IN ARQ DATA CAUSED BY OPERATOR ERROR
70.
Witness Statement of P. Thomas dated
17 June 2004
POINQ0128201F
FUJ00121987
71.
Witness Statement of N. Lowther dated 1
September 2003 (unsigned)
POINQ0128105F
FUJ00121891
72.
Witness Statement of P. Thomas dated
15 April 2004
POINQ0128209F
FUJ00121995
Page 84 of 103
WITN06650300
WITN06650300
sat Document Description Control No. URN
73. Email chain from G. Ward to POINQ0128189F I FUJ00121975
Fujitsu@royalmail.com and M. William
with subject “Witness Statement request
& Forest” dated 25 May 2004
74. Witness Statement of W. Mitchell dated POINQ0128202F I FUJ00121988
17 June 2004
75. Audit Record Query request dated 10 POINQ0230540F I FUJ00176306
March 2004
76. Email chain from M. William to G. Ward POINQ0178201F I FUJ00172020
copying P. Thomas with subject “Revised
ARQ Issues (Forest gate & Urmston)”
dated 1 June 2004
77. Witness Statement of W. Mitchell dated POINQ0128193F I FUJ00121979
14 June 2004
2008 INCIDENTS RELATED TO RIPOSTE LOCK EVENT
78. PC0152376 POINQ0160879F I FUJ00154684
79. PC0140715 POINQ0161401F I FUJ00155207
80. Email chain from S. Evans to S. Denham I POINQ0161590F I FUJ00155396
copying G. Allen, P. Sewell, A. Holmes,
P. Thomas and G. Jenkins with subject
“Audit and PC01152376” dated 19
December 2008
81. KEL dsed5628Q POINQ0137022F I FUJ00130827
82. Email from G. Barnes to D. Seddon with =I POINQ0161403F I FUJ00155209
subject “dsed5628Q” dated 17 January
2008
83. PC0152421 POINQ0160881F I FUJ00154686
84. PC0153009 POINQ0161418F I FUJ00155224
85. PC0152828 POINQ0161405F I FUJ00155211
86. Email chain from R. Gelder to D. Wilcox POINQ0161404F I FUJ00155210
and K. McKeown with subject “FW:
153009” dated 17 January 2008
87. PC0155120 POINQ0160880F I FUJ00154685
88. KEL kiangl823S POINQ0136671F I FUJ00130476
89. POL/Fujitsu Product & Branch POINQ0161424F I FUJ00155230
Accounting Workshop Action report dated
1 August 2008
90. Email chain from M. Stewart to G. POINQ0161439F I FUJ00155245
Jenkins and A. Chambers copying R.
Birkinshaw with subject “RE: Branch
141832 Craigpark” dated 3 September
2008
91. PC0158102 POINQ0160878F I FUJ00154683,
92. Email chain from P. Thomas to G. POINQ0161427F I FUJ00155233
Jenkins, A. Holmes, S. Meek and P.
Sewell with subject “FW: Branch 141832
Craigpark” dated 12 August 2008
Page 85 of 103
WITN06650300
WITN06650300
Exhibit
no.
Document Description
Control No.
URN
93.
Email chain from P. Thomas to G.
Jenkins, A. Holmes, S. Meek and P.
Sewell copying H. Pritchard and B.
Pinder with subject “RE: Branch 141832
Craigpark” dated 14 August 2008
POINQ0161018F
FUJ00154823
94.
Note of meeting held on 13 August 2008
prepared by P. Thomas dated 14 August
2008
POINQ0161019F
FUJ00154824
95.
Email chain from R. Birkinshaw to G.
Jenkins with subject “RE: Branch 141832
Craigpark” dated 25 August 2008
POINQ0161432F
FUJ00155238
96.
Email from G. Jenkins to R. Birkinshaw
and D. Johns with subject “RE: Potential
Audit Issue.doc” dated 28 August 2008
POINQ0161435F
FUJ00155241
97.
Document titled “Potential Audit Issue -
Horizon” dated 28 August 2008
POINQ0161436F
FUJ00155242
98.
Email chain from J. Burton to G. Jenkins
copying A. Chambers with subject “RE:
Response to Action AP0108003 from
POL/Fujitsu P&BA Workshop 1 August
2008” dated 3 September 2008
POINQ0161440F
FUJ00155246
99.
Email chain from A. Chambers to G.
Jenkins copying M. Peach and J. Burton
with subject “RE: Response to Action
AP0108003 from POL/Fujitsu P&BA
Workshop 1 August 2008” dated 3
September 2008
POINQ0161441F
FUJ00155247
100.
Email chain from M. Stewart to G.
Jenkins copying J. Burton, A. Chambers
and M. Peach with subject “RE:
Response to Action AP0108003 from
POL/Fujitsu P&BA Workshop 1 August
2008” dated 4 September 2008
POINQ0161446F
FUJ00155252
101.
Email chain from A. Chambers to G.
Jenkins with subject “FW: Response to
Action AP0108003 from POL/Fujitsu
P&BA Workshop 1 August 2008” dated 4
September 2008
POINQ0161442F
FUJ00155248
102.
Email chain from G. Barnes to D.
Seddon, S. Evans, G. Jenkins and R.
Birkinshaw copying J. Burton and A.
Chambers with subject “RE: Peak
152376: CAPS Process Locking the
messagestore PC0164429” dated 5
September 2008
POINQ0161452F
FUJ00155258
103.
Email from M. Peach to J. Budworth, S.
Evans, S. Bamber, M. Cumming and G.
Jenkins with subject “RE: Peak 152376:
CAPS Process Locking the
messagestore PC0164429” dated 9
September 2008
POINQ0161453F
FUJ00155259
Page 86 of 103
WITN06650300
WITN06650300
Exhibit
no.
Document Description
Control No.
URN
104.
Email chain from J. Budworth to S. Evans
copying G. Barnes, D. Seddon, M.
Peach, G. Jenkins and others with
subject “RE: Peak 152376: CAPS
Process Locking the messagestore
PC0164429”" dated 10 September 2008
POINQ0161455F
FUJ00155261
105.
PC0164429
POINQ0161560F
FUJ00155366
106.
Post Office Account Release Note
(RNT8601) raised on 29 September 2008
POINQ0161680F
FUJ00155486
107.
Email chain from D. Johns to A.
Chambers copying R. Birkinshaw, A.
Holmes and G. Jenkins with subject “RE:
Audit Review” dated 4 September 2008
POINQ0161447F
FUJ00155253
108.
Email from R. Birkinshaw to P. Sewell, A.
Holmes, S. Meek, P. Ambrose, S. Evans,
A. Chambers, D. Johns, A. Dunks, P.
Thomas and J. Burton with subject
“Recent Exercise to review Audit in
Horizon” dated 5 September 2008
POINQ0161451F
FUJ00155257
109.
Email from A. Chambers to P. Thomas
copying G. Jenkins and P. Sewell with
subject “Riposte Timeout waiting for lock
events” dated 16 September 2008
POINQ0161458F
FUJ00155264
110.
Document titled “Analysis of Audit
timeouts” dated 3 September 2008
POINQ0161450F
FUJ00155256
111.
HNG-X Change Proposal raised on 13
October 2008
POINQ0161466F
FUJ00155272
112.
Email from A. Holmes to P. Sewell, R.
Birkinshaw and G. Jenkins with subject
“Counter Audit CP” dated 13 October
2008
POINQ0161465F
FUJ00155271
113.
Email chain from G. Jenkins to P.
Thomas with subject “RE: Event Errors -
ARQ Service” dated 15 October 2008
POINQ0161468F
FUJ00155274
114.
Meeting invite from P. Thomas to A.
Holmes, R. Birkinshaw, P. Sewell and S.
Meek with subject “Updated: Audit CP -
Words” scheduled on 24 October 2008
POINQ0161472F
FUJ00155278
115.
Email from P. Thomas to P. Sewell
copying H. Pritchard with subject “ARQ
Service” dated 1 December 2008
POINQ0161567F
FUJ00155373
116.
Email chain from P. Sewell to D. Hinde
copying P. Thomas with subject “RE:
Audit Strengthening - potential CP” dated
8 December 2008
POINQ0161574F
FUJ00155380
117.
Email chain from P. Thomas to H.
Pritchard with subject “RE: ARQ Service
problem” dated 4 December 2008
POINQ0161572F
FUJ00155378
118.
Slide deck titled “Prosecution Support
Urgent Issue”
POINQ0161030F
FUJ00154835
Page 87 of 103
WITN06650300
WITN06650300
Exhibit
no.
Document Description
Control No.
URN
119.
Email chain from P. Thomas to G, Allen,
D. Hinde, A. Cousins, S. Evans, P.
Sewell and A. Holmes copying H.
Pritchard and S. Denham with subject
“RE: Proposed Slides for ARQ Service
Issue” dated 15 December 2008
POINQ0161581F
FUJ00155387
120.
Email chain from P. Thomas to G.
Jenkins with subject “RE: Audit Issue”
dated 17 December 2008
POINQ0161029F
FUJ00154834
121.
Meeting invite from S. Denham to G.
Allen, A. Cousins, S. Evans, P. Sewell, A.
Holmes, P. Thomas, W. Warham and H.
Pritchard with subject “Updated: ARQ
Aervice issue” scheduled on 17
December 2008
POINQ0161586F
FUJ00155392
122.
Email chain from P. Sewell to D. Hinde,
G. Allen, P. Thomas, A. Cousins, S.
Evans and Alan Holmes copying H.
Pritchard and S. Denham with subject
“RE: Proposed Slides for ARQ Service
Issue” dated 19 December 2008
POINQ0161588F
FUJ00155394
123.
Email chain from D. Hinde to S. Denham,
G. Allen, P. Thomas, A. Cousins, S.
Evans, P. Sewell and A. Holmes copying
H. Pritchard and J. Jukes with subject
“RE: Proposed Slides for ARQ Service
Issue” dated 19 December 2008
POINQ0161589F
FUJ00155395
124.
Email from P. Thomas to D. Hinde, G.
Allen, A. Cousins S. Evans, P. Sewell
and A. Holmes with subject “Proposed
Slides for ARQ Service Issue” dated 11
December 2008
POINQ0161579F
FUJ00155385
125.
Email from S. Evans to P. Thomas, G.
Jenkins and A. Holmes copying S.
Denham and G. Allen with subject
“Standard_Fujitsu_WS_V8_Jan
2009 _SAE .doc” dated 5 January 2009
POINQ0128806F
FUJ00122592
126.
Draft template witness statement
(undated)
POINQ0128807F
FUJ00122593
127.
Email chain from P. Thomas to S. Evans,
G. Jenkins and A. Holmes copying S.
Denham and G. Allen with subject “RE:
Standard_Fujitsu_WS_V8_Jan
2009_SAE .doc” dated 6 January 2009
POINQ0128810F
FUJ00122596
128.
Draft template witness statement
(undated)
POINQ0128811F
FUJ00122597
129.
Email chain from P. Thomas to D.
Posnett copying H. Pritchard and P.
Sewell with subject “FW: Security
Incident” dated 7 January 2009
POINQ0161593F
FUJ00155399
Page 88 of 103
WITN06650300
WITN06650300
Exhibit
no.
Document Description
Control No.
URN
130.
Email chain from P. Thomas to H.
Pritchard, W. Warham and S. Denham
copying P. Sewell with subject “RE:
Security Incident” dated 8 January 2009
POINQ0161594F
FUJ00155400
131.
Email chain from S. Evans to G. Barnes
with subject “FW: Audit Issue” dated 8
January 2009
POINQ0161596F
FUJ00155402
132.
Email chain from A. Chambers to P.
Thomas copying H. Pritchard, P. Sewell,
A. Holmes, S. Evans, G. Allen, M. Peach
and S. Denham with subject “RE: Outlet
Checking List - Audit Issue” dated 3
February 2009
POINQ0161612F
FUJ00155418
133.
Spreadsheet titled “Outlet Checking List”
(undated)
POINQ0161613F
FUJ00155419
134.
Email chain from P. Thomas to S.
Denham copying H. Pritchard and P.
Sewell with subject “FW: Outlet Checking
List - Audit Issue” dated 4 February 2009
POINQ0161616F
FUJ00155422
135.
Email chain from P. Thomas to A.
Chambers with subject “RE: Outlet
Checking List - Audit Issue” dated 3
February 2009
POINQ0161034F
FUJ00154839
136.
Email chain from P. Thomas to S.
Denham copying H. Pritchard, P. Sewell
and W. Warham with subject “FW: Outlet
Checking List - Audit Issue” dated 27
January 2009
POINQ0161603F
FUJ00155409
137.
Email from P. Thomas to unknown
recipients with unknown subject attaching
a draft template witness statement
(undated)
POINQ0161614F
FUJ00155420
138.
Email from P. Thomas to S. Denham and
P. Sewell with subject “FW: Security
Incident” dated 4 February 2009
POINQ0161615F
FUJ00155421
139.
HNG-X Change Proposal (CP 4867)
raised on 26 February 2009
POINQ0161668F
FUJ00155474
140.
Email chain from K. Westfield to D. Hinde
and Wendy Warham copying G. Allen
and J. Burton with subject “RE: HNG-X
CP0336” Enable analysis of Counter
event messages within the HNG-X Audit
solution” dated 17 July 2009
POINQ0161684F
FUJ00155490
141.
Email from C. Maving to numerous POL
and Fujitsu personnel including S. Jones,
A. Fisher, A. Spencer, T. Baker, N. Taylor
and F. Denbali with subject “Testing
progress 06/07/10” dated 6 July 2010
POINQ0161705F
FUJ00155511
Page 89 of 103
WITN06650300
WITN06650300
Exhibit
no.
Document Description
Control No.
URN
DUPLICATE TRANSACTIONS
142.
Email chain from G. Jenkins to G. Allen,
A. Holmes, A. Mansfield and A.
Spurgeon, with subject “RE: PC0200468
- Duplication of Transaction Records”
dated 24 June 2010
POINQ0103209F
FUJ00097038
143.
Document titled Duplication of
Transaction Records Contained in ARQ
Returns dated 23 June 2010
POINQ0103229F
FUJ00097058
144.
PC0200468
POINQ0178364F
FUJ00172183
145.
Email chain from P. Thomas to G.
Wilkerson with subject “FW: Duplication
of Transaction Records on ARQ Returns”
dated 24 June 2010
POINQ0103228F
FUJ00097057
146.
Email chain from G. Jenkins to P.
Thomas and A. Holmes, with subject
“RE: Duplicate Messages in ARQ” dated
23 June 2010
POINQ0178224F
FUJ00172043
147.
Email from G. Jenkins to P. Thomas and
A. Holmes with subject “Duplicate
Messages in ARQ (updated) dated 24
June 2010
POINQ0230544F
FUJ00176310
148.
PC0194639
POINQ0230541F
FUJ00176307
149.
Email chain from T. Godeseth to J.
Gribben, copying M. Lenton, D. Ibbett
and P. Newsome, with subject “RE:
Questions from Counsel on Comments
on Day 7 and 8” dated 10 April 2019
POINQ0171261F
FUJ00165083
150.
Document titled “Questions on RW and
SSC Comments on Days 7 and 8”
(undated)
POINQ0171260F
FUJ00165082
151.
Email chain from M. Underwood to L.
Keating, T. Godeseth, M. Westbrook, P.
Newsome. E. losifidou, copying in L.
Wolstencroft and J. Gribben, with subject
“RE: Bramble: call to discuss analytics
(Privileged & Confidential)” dated 28
February 2017
POINQ0164065F
FUJ00157889
152.
Email chain from T. Godeseth to L.
Keating, M. Underwood, M. Westbrook,
P. Newsome. E. losifidou, copying in L.
Wolstencroft and J. Gribben, with subject
“RE: Bramble: call to discuss analytics
(Privileged & Confidential)” dated 1
March 2017
POINQ0176401F
FUJ00170220
153.
RMGA HNG-X Counter Application
Review dated 9 February 2010
POINQ0099219F
FUJ00093048
Page 90 of 103
WITN06650300
WITN06650300
Exhibit
no.
Document Description
Control No.
URN
154.
RMGA HNG-X Counter Application
Review dated 9 February 2010 with mark
up by G. Jenkins and comments by D
Johns
POINQ0148361F
FUJ00142164
155.
RMGA HNG-X Counter Application
Review dated 9 February 2010 with mark
up by G. Jenkins
POINQ0148357F
FUJ00142160
156.
Email chain from J. Ballantyne to G.
Jenkins, S. Parker, M. Wright and A.
Holmes, copying in J. Simpkins, I. Turner,
G. Allen, S. Goddard, A. Beardmore and
S. Porter, with subject “RE: Peak
PC0196948" dated 12 April 2018
POINQ0178213F
FUJ00172032
157.
RMGA HNG-X Counter Application
Review, v 1, dated 9 February 2010
POINQ0148354F
FUJ00142157
158.
HNG-X Counter Application High Level
Design v 1 dated 23 August 2010
POINQ0097961F
FUJ00091790
159.
Email from Peak to A. Dunks, with
subject “Complete Call Update
PC0257379: Peak calls assigned to Sec
Opps” dated 23 November 2018
POINQ0178271F
FUJ00172090
160.
RMGA HNG-X Counter Application
Review, v 1, dated 9 February 2010 with
comments by G. Jenkins and D. Johns
POINQ0148359F
FUJ00142162
161.
RMGA HNG-X Counter Application
Review, v 1, dated 9 February 2010 with
comments by G. Jenkins
POINQ0148355F
FUJ00142158
162.
Email chain from G. Butts to D. Cooke,
copying M. Andrews, G. Allen and G.
Jenkins, with subject “FW: ARQs” dated
26 June 2010
POINQ0103243F
FUJ00097072
163.
RMGA HNG-X Counter Application
Review, v 1, dated 9 February 2010
(scanned) with comments
POINQ0178212F
FUJ00172031
164.
Email chain from G. Wilkerson to P.
Thomas and G. Jenkins, copying A.
D’Alvarez and G. Butts, with subject “RE:
Duplication of Transaction Records on
ARQ Returns” dated 24 June 2010
POINQ0103210F
FUJ00097039
165.
Email chain from G. Welsh to P. Thomas,
G.V. Achte, D. Munro, P. Thompson, and
G. Wilkerson, copying G. Jenkins and A.
Holmes with subject “RE: Duplication of
Transaction Records on ARQ Returns”
dated 24 June 2010
POINQ0178225F
FUJ00172044
166.
Email chain from P. Thomas to G.
Wilkerson, G. Butts, G. Jenkins, copying
A. D’Alvarez with subject “RE:
Duplication of Transaction Records on
ARQ Returns” dated 25 June 2010
POINQ0159316F
FUJ00153121
167.
Spreadsheet titled “ARQ Report” dated
24 June 2010
POINQ0159317F
FUJ00153122
Page 91 of 103
WITN06650300
WITN06650300
Exhibit
no.
Document Description
Control No.
URN
168.
Email chain from G. Butts to D. Cooke,
copying M. Andrews, G. Allen, G. Welsh
and G. Jenkins with subject “FW: ARQs”
dated 25 June 2010
POINQ0103232F
FUJ00097061
169.
Email chain from G. Jenkins to G. Butts
and D. Cooke, copying M. Andrews, G.
Allen and G. Welsh, with subject “RE:
ARQs” dated 25 June 2010
POINQ0103242F
FUJ00097071
170.
Email chain from P. Thomas to T.
Lilywhite, P. Thompson, G. Welsh, D.
Munro copying G. Jenkins with subject
“FW: Duplication of Transaction Records
in ARQ Returns - Discussion with POL”
dated 2 July 2010
POINQ0127290F
FUJ00121097
171.
Email chain from G. Barnes to P. Thomas
with subject “FW: PC0200468 -
Duplication of Transaction Records”
dated 21 July 2010
POINQ0178227F
FUJ00172046
172.
Email from G. Barnes to S. Bamber, P.
Okely, V. Pandya, J. Rogers, M. Swain
and N. Taylor, with subject “Audit
baseline
AUDIT_EXTRACT_SVR_0207_V051-
050 now RFB - PC0200468 ‘A’ priority”
dated 7 July 2010
POINQ0178226F
FUJ00172045
173.
Email chain from P. Thomas to T.
Lillywhite, G. Jenkins, G. Wilkerson,
copying G. Welsh with subject “FW:
Duplicatation of Transaction Records in
ARQ Returns” dated 7 July 2010
POINQ0129115F
FUJ00122901
174.
Draft Witness Statement of P. Thomas
(undated and unsigned)
POINQ0129116F
FUJ00122902
175.
Email from P. Thomas to G. Jenkins with
subject “Duplicate Records - Amended
Witness Statement” dated 7 July 2010
POINQ0129113F
FUJ00122899
176.
Email from P. Thomas to G. Jenkins
copying G. Wilkerson with subject
“Duplicate Records - Witness Statement”
dated 8 July 2010
POINQ0129121F
FUJ00122907
177.
Draft Witness Statement of P. Thomas
(undated and unsigned)
POINQ0129122F
FUJ00122908
178.
Email from G. Jenkins to P. Thomas and
A. Holmes, with subject “Duplicate
Messages in ARQ (updated)” dated 24
June 2010
POINQ0230545F
FUJ00176311
179.
Email from P. Thomas to G. Jenkins with
subject “Duplicated Records - WS” dated
8 July 2010
POINQ0129128F
FUJ00122914
180.
Witness Statement of G. Jenkins dated 8
July 2010 (unsigned)
POINQ0129129F
FUJ00122915
Page 92 of 103
WITN06650300
WITN06650300
Exhibit
no.
Document Description
Control No.
URN
181.
Email chain from P. Thomas to G.
Jenkins with subject “RE: Duplicated
Records - WS” dated 8 July 2010
POINQ0230546F
FUJ00176312
182.
Email chain from P. Thomas to G.
Jenkins with subject “FW: Duplicatation
of Transaction Records in ARQ Returns”
dated 15 July 2010
POINQ0129142F
FUJ00122928
183.
Witness Statement of G. Jenkins dated 8
July 2010 (unsigned)
POINQ0129143F
FUJ00122929
184.
Email chain from P. Thomas to G.
Jenkins with subject “RE: Duplicatation of
Transaction Records in ARQ Returns”
dated 15 July 2010
POINQ0230547F
FUJ00176313
185.
Email chain from P. Thomas to J.
Longman copying J. Owen with subject
“RE: Duplicatation of Transaction
Records in ARQ Returns” dated 15 July
2010
POINQ0159328F
FUJ00153133
186.
Email chain from G. Jenkins to P.
Thomas with subject “RE: Duplicatation
of Transaction Records in ARQ Returns”
dated 15 July 2010
POINQ0159340F
FUJ00153145
187.
Email chain from A. Kay to CSPOA
Security with subject “ARQ 051-056
Caereithin FAD 166642, returned on 9th
May 14” dated 22 May 2014
POINQ0230557F
FUJ00176323
188.
Email chain from G. Barnes to J. Muir
and A. Holmes with subject “RE: Issues
with a Slow ARQ” dated 20 May 2014
POINQ0230556F
FUJ00176322
189.
Email chain from A. Holmes to A. Kay
copying G. Barnes with subject “RE: ARQ
- Duplicate Data” dated 22 May 2014
POINQ0178266F
FUJ00172085
190.
Email chain from J. Muir to A. Holmes, A.
Dunks, F. Denbali, copying G. Barnes, S.
Goddard and S. Godfrey with subject
“RE: Duplicates” dated 15 July 2016
POINQ0178267F
FUJ00172086
191.
PC0250729
POINQ0179253F
FUJ00173072
192.
Email from F. Denbali to A. Holmes with
subject “Duplicate records in ARQ output”
dated 14 July 2016
POINQ0230558F
FUJ00176324
193.
Email chain from A. Holmes to CSPOA
Security with subject “RE: Duplicate
records in ARQ output” dated 14 July
2016
POINQ0230561F
FUJ00176326
194.
Email from F. Denbali to A. Holmes with
subject “RE: Duplicate records in ARQ
output” dated 14 July 2016
POINQ0230562F
FUJ00176327
195.
Document attached to email
correspondence from F. Denbali to A.
Holmes dated 14 July 2016
POINQ0230563F
FUJ00176328
Page 93 of 103
WITN06650300
WITN06650300
ae Document Description Control No. URN
196. I Email chain from G. Barnes to CSPOA POINQ0178268F I FUJ00172087
Security with subject “RE: ARQ -
Duplicate Data” dated 21 May 2014
197. I Email from A. Holmes to J. Muir, A. POINQ0230564F I FUJ00176329
Dunks and F. Denbali, copying G.
Barnes, S. Goddard and S. Godfrey with
subject “RE: Duplicates” dated 15 July
2016
198. I Email chain from J. Muir to G. Barnes, A. I POINQ0222161F I FUJ00216439
Dunks, A. Holmes and F. Denbali with
subject “RE: Duplicates” dated 19 July
2016
HISTORIC GAPS IN ARQ DATA
199. I Email chain from S. Browell to G. Barnes, I POINQ0230745F I FUJ00176510
P. Gauntlett and P. Boardman copying M.
Mistry with subject “Historical Issue with
Audit Data” dated 2 December 2021
200. I Audit Extractor Support Guide v 1 dated POINQ0230774F I FUJ00176539
21 August 2000
201. I Audit Data Extraction Process v 1 dated I POINQ0230775F I FUJ00176540
29 May 2002
202. I Audit Data Extraction Process v 2.1 POINQ0230778F I FUJ00176543
dated 26 October 2004
203. I Audit Data Retrieval Low Level Design v I POINQ0230780F I FUJ00176545
1 dated 8 June 2010
204. I Audit Trail Functional Specification v 11.0 I POINQ0008190F I FUJ00002019
dated 4 August 2006
205. I PC0206479 POINQ0230565F I FUJ00176330
206. I PC0268776 POINQ0230566F I FUJ00176331
207. I PC0230183 POINQ0230567F I FUJ00176332
208. I PC0162602 POINQ0230568F I FUJ00176333
209. I PC0186750 POINQ0230569F I FUJ00176334
210. I PC0277613 POINQ0230571F I FUJ00176336
211. I PC0290357 POINQ0230572F I FUJ00176337
212. I PC0093837 POINQ0230573F I FUJ00176338
213. I PC0089043 POINQ0230574F I FUJ00176339
214. I PC0290571 POINQ0230575F I FUJ00176340
215. I PC0133634 POINQ0230576F I FUJ00176341
216. I PC0095192 POINQ0230577F I FUJ00176342
217. I PC0089033 POINQ0230716F I FUJ00176481
218. I PC0286014 POINQ0230717F I FUJ00176482
219. I PC0187097 POINQ0230718F I FUJ00176483
220. I PC0133534 POINQ0230719F I FUJ00176484
221. I PC0289422 POINQ0230720F I FUJ00176485
222. I PC0189343 POINQ0230782F I FUJ00176547
223. I PC0200501 POINQ0230783F_I FUJ00176548
Page 94 of 103
WITN06650300
WITN06650300
Exhibit
no.
Document Description
Control No.
URN
224.
Email chain from S. Browell to S. Oldnall
and W. Warham with subject “Horizon
Audit San” dated 17 March 2022
POINQ0230722F
FUJ00176487
225.
Email chain from P. Gauntlett to G.
Barnes, P. Boardman and S. Browell with
subject “Historical issues with Audit Data”
dated 1 December 2021
POINQ0230725F
FUJ00176490
226.
Email chain from P. Gauntlett to G.
Barnes, P. Boardman and S. Browell
copying M. Mistry with subject “Historical
Issue with Audit Data”, dated 1
December 2021
POINQ0230727F
FUJ00176492
227.
Email chain from G. Barnes to P.
Gauntlett with subject “Summary of
Riposte and HNGx” dated 1 December
2021
POINQ0230728F
FUJ00176493
228.
Email chain from P. Boardman to P.
Gauntlett and S. Browell, copying G.
Barnes and M. Mistry with subject
“Historical Issue with Audit Data”
POINQ0230729F
FUJ00176494
229.
Email chain from G. Barnes to P.
Gauntlett with subject “Summary of
Riposte and HGNx” dated 30 November
2021
POINQ0230730F
FUJ00176495
230.
PC0268776
POINQ0230731F
FUJ00176496
231.
Email chain from G. Barnes to P.
Gauntlett with subject “Summary of
Riposte and HNGx” dated 30 November
2021
POINQ0230732F
FUJ00176497
232.
Email chain from G. Barnes to P.
Boardman, P. Gauntlett and S. Browell
copying M. Mistry with subject “Historical
Issue with Audit Data” dated 1 December
2021
POINQ0230733F
FUJ00176498
233.
Email chain from G. Barnes to F. Denbali
copying P. Gauntlett with subject
“Horizon Queries” dated 29 November
2021
POINQ0230734F
FUJ00176499
234.
Email from F. Denbali to G. Barnes
copying P. Gauntlett with subject
“Horizon Queries” dated 29 November
2021
POINQ0230735F
FUJ00176500
235.
Email chain from G. Barnes to F. Denbali
copying P. Gauntlett with subject
“Horizon Queries” dated 29 November
2021
POINQ0230736F
FUJ00176501
236.
Email chain from S. Browell to G. Barnes,
P. Gauntlett, P. Boardman copying M.
Mistry with subject “Historical Issue with
Audit Data” dated 2 December 2021
POINQ0230737F
FUJ00176502
Page 95 of 103
WITN06650300
WITN06650300
Exhibit
no.
Document Description
Control No.
URN
237.
Email chain from G. Barnes to S. Browell,
P. Gauntlett and P. Boardman copying M.
Mistry with subject “Historical Issue with
Audit Data” dated 2 December 2021
POINQ0230738F
FUJ00176503
238.
Archive Server Configuration v 1.0 dated
21 October 2009
POINQ0230739F
FUJ00176504
239.
Email chain from P. Boardman to S.
Browell, P. Gauntlett, G. Barnes copying
M. Mistry with subject “Historical Issue
with Audit Data” dated 2 December 2021
POINQ0230741F
FUJ00176506
240.
Email from S. Browell to P. Gauntlett, G.
Barnes and P. Boardman copying M.
Mistry with subject “Historical Issue with
Audit Data” dated 2 December 2021
POINQ0230742F
FUJ00176507
241.
Document titled “Inconsistent data in
IRE11 and IRE19 Audit Archives” dated 2
December 2021
POINQ0230743F
FUJ00176508
242.
Email chain from G. Barnes to S. Browell,
P. Gauntlett and P. Boardman copying M.
Mistry with subject “Historical Issue with
Audit Data” dated 2 December 2021
POINQ0230744F
FUJ00176509
243.
Email from G. Barnes to S. Browell, P.
Gauntlett, P. Boardman and E. Ashford
copying M. Mistry with subject “Historical
Issue with Audit Data” dated 2 December
2021
POINQ0230746F
FUJ00176511
244.
Email from G. Barnes to E. Ashford, S.
Browell, P. Gauntlet and P. Boardman
copying M. Mistry, A. R. Gibson and S.
Wilson with subject “Historical Issue with
Audit Data” dated 3 December 2021
POINQ0230747F
FUJ00176512
245.
Email chain from E. Ashford to G.
Barnes, S. Browell, P. Gauntlett and P.
Boardman copying M. Mistry, A. R.
Gibson and S. Wilson with subject “RE:
Historical Issue with Audit Data” dated 3
December 2021
POINQ0230748F
FUJ00176513
246.
Email chain from G. Barnes to E.
Ashford, S. Browell, P. Gauntlett and P.
Boardman copying M. Mistry, A. R.
Gibson and S. Wilson with subject
“Historical Issue with Audit Data” dated 3
December 2021
POINQ0230749F
FUJ00176514
247.
Email chain from S. Browell to P.
Gauntlett, P. Boardman and G. Barnes
with subject “PCI meeting where audit
archive gaps was raised” dated 3
December 2021
POINQ0230750F
FUJ00176515
Page 96 of 103
WITN06650300
WITN06650300
Exhibit
no.
Document Description
Control No.
URN
248.
Email from S. Browell to A. Kemp, A.
Holmes, G. Barnes and P. Gauntlett
copying J. Muir and G. Baker with subject
“CONFIDENTIAL - Notes from our catch
up earlier - ARQ focused” dated 3
December 2021
POINQ0230751F
FUJ00176516
249.
Email chain from G. Barnes to S. Browell,
A. Kemp, A. Holmes and P. Gauntlett
copying J. Muir and G. Baker with subject
“CONFIDENTIAL - Notes from our catch
up earlier - ARQ focused” dated 4
December 2021
POINQ0230752F
FUJ00176517
250.
Email chain from E. Ashford to G.
Barnes, S. Browell, P. Gauntlett, and P.
Boardman copying M. Mistry, A. R.
Gibson and S. Wilson with subject
“Historical Issue with Audit Data” dated 3
December 2021
POINQ0230753F
FUJ00176518
251.
Email chain from E. Ashford to G.
Barnes, S. Browell, P. Gauntlett and P.
Boardman copying M. Mistry, A. R.
Gibson and S. Wilson with subject
“Historical Issue with Audit Data” dated 3
December 2021
POINQ0230754F
FUJ00176519
252.
Email chain from S. Browell to E.
Ashford, G. Barnes, P. Gauntlett and P.
Boardman copying M. Mistry, A. R.
Gibson and S. Wilson with subject
“Historical Issue with Audit Data” dated 6
December 2021
POINQ0230755F
FUJ00176520
253.
Email chain from E. Ashford to S.
Browell, G. Barnes, P. Gauntlett and P.
Boardman copying M. Mistry, A. R.
Gibson and S. Wilson with subject
“Historical Issue with Audit Data” dated 6
December 2021
POINQ0230756F
FUJ00176521
254.
Email chain from S. Browell to E.
Ashford, G. Barnes, P. Gauntlett and P.
Boardman copying M. Mistry, A. R.
Gibson and S. Wilson with subject
“Historical Issue with Audit Data” dated 6
December 2021
POINQ0230757F
FUJ00176522
255.
Email chain from G. Barnes to S. Browell,
E. Ashford, P. Gauntlett and P.
Boardman copying M. Mistry, A. R.
Gibson and S. Wilson with subject
“Historical Issue with Audit Data” dated 6
December 2021
POINQ0230758F
FUJ00176523
256.
Email chain from G. Barnes to E. Ashford
copying P. Gauntlett and A. R. Gibson
with subject “Historical Issue with Audit
Data” dated 6 December 2021
POINQ0230759F
FUJ00176524
Page 97 of 103
WITN06650300
WITN06650300
Exhibit
no.
Document Description
Control No.
URN
257.
Email chain from E. Ashford to G. Barnes
copying P. Gauntlett and A. R. Gibson
with subject “Historical Issue with Audit
Data” dated 6 December 2021
POINQ0230761F
FUJ00176526
258.
Email chain from G. Barnes to S. Browell,
E. Ashford, P. Gauntlett and P.
Boardman copying M. Mistry, A. R.
Gibson and S. Wilson with subject
“Historical Issue with Audit Data” dated 6
December 2021
POINQ0230762F
FUJ00176527
259.
Email from S. Browell to G. Barnes, J.
Muir, E. Ashford, P. Gauntlett, G. Baker,
S. Wilson, M. Hatch and S. Bansal with
subject “CONFIDENTIAL - Audit Archive
- The action plan” dated 6 December
2021
POINQ0230763F
FUJ00176528
260.
Email chain from G. Barnes to S. Browell,
A. Kemp, A. Holmes and P. Gauntlett
copying J. Muir and G. Baker with subject
“CONFIDENTIAL - Notes from our catch
up earlier - ARQ focused” dated 5
December 2021
POINQ0230764F
FUJ00176529
261.
Email chain from E. Ashford to P.
Gauntlett with subject “Mirroring the audit
archives in IRE11 and IRE19” dated 14
January 2022
POINQ0230765F
FUJ00176530
262.
Email chain from H. Kuypers to P.
Gauntlett and M. Mistry with subject
“Problem statement IRE19 DCs” dated
28 January 2022
POINQ0230766F
FUJ00176531
263.
Email chain from H. Kuypers to P.
Gauntlett, M. Mistry and R. Oye-Akinola
with subject “Problem statement IRE19
DCs" dated 31 January 2022
POINQ0230767F
FUJ00176532
264.
Email chain from P. Gauntlett to G.
Barnes with subject “Summary of Riposte
and HNGx” dated 30 November 2021
POINQ0230768F
FUJ00176533
265.
Email from P. Gauntlett to G. Barnes with
subject “Summary of Riposte and HNGx”
dated 1 December 2021”
POINQ0230769F
FUJ00176534
266.
Email chain from P. Gauntlett to S.
Browell copying G. Barnes, M. Mistry and
P. Boardman with subject “Historical
Issue with Audit Data” dated 1 December
2021
POINQ0230770F
FUJ00176535
267.
Email from P. Gauntlett to P. Boardman,
S. Browell and G. Barnes copying M.
Mistry with subject “Historical Issues with
Audit Data” dated 2 December 2021
POINQ0230771F
FUJ00176536
Page 98 of 103
WITN06650300
WITN06650300
oe Document Description Control No. URN
268. I Email chain from P. Gauntlett to S. POINQ0230772F I FUJ00176537
Browell, P. Boardman and G. Barnes with
subject “PCI meeting where audit archive
gaps was raised” dated 3 December
2021
269. I Email chain from P. Gauntlett to S. POINQ0230773F I FUJ00176538
Browell, P. Boardman and G. Barnes with
subject “PCI meeting where audit archive
gaps was raised” dated 3 December
2021
270. I PC0187306 POINQ0230570F I FUJ00176335,
OTHER POTENTIAL ISSUES RELATING TO ARQ DATA
271. I PC0088573 POINQ0178277F I FUJ00172096
272. I PC0272681 POINQ0179364F I FUJ00173183
273. I PC0205806 POINQ0178402F I FUJ00172221
274. I PC0206923 POINQ0178396F I FUJ00172215
275. I PC0280793 POINQ0179365F I FUJ00173184
276. I PC0241862 POINQ0179244F I FUJ00173063
277. I PC0225656 POINQ0178467F I FUJ00172286
278. I PC0211833 POINQ0178422F I FUJ00172241
APPENDIX 1 - AUDIT TRAIL FUNCTIONAL SPECIFICATIONS
279. I Audit Trail Functional Specification v 3.1 POINQ0125542F I FUJ00119343
dated 10 November 1999
280. I Audit Trail Functional Specification v 4.0 I POINQ0124379F I FUJ00118196
dated 10 November 1999
281. I Audit Trail Functional Specification v 4.1 POINQ0161717F I FUJ00155523
dated 10 April 2000
282. I Audit Trail Functional Specification v5.0 I POINQ0007621F I FUJ00001450
dated 15 January 2001
283. I Audit Trail Functional Specification v6.0 I POINQ0007791F I FUJ00001620
dated 25 February 2002
284. I Audit Trail Functional Specification v 7.0 I POINQ0007850F I FUJ00001679
dated 17 September 2002
285. I Audit Trail Functional Specification v 8.0 I POINQO008065F I FUJ00001894
dated 18 October 2004
286. I Audit Trail Functional Specification v 9.0 POINQ0125546F I FUJ00119347
dated 22 November 2004
287. I Audit Trail Functional Specification v 10.0 I POINQ0008116F I FUJ00001945
dated 29 June 2005
288. I Audit Trail Functional Specification v 12.0 I POINQ0008425F I FUJ00002254
dated 8 October 2010
APPENDIX 2 - ARQ DATA EXTRACTION PROCESSES
289. I Conducting Audit Data Extractions at POINQ0158353F I FUJ00152159
CSR Process v 1.0 dated 4 May 2000
290. I Conducting Audit Data Extractions at POINQ0158361F I FUJ00152167
CSR+ Process v 1.0 dated 15 December
2000
291. I Conducting Audit Data Extractions at Live I POINQ0158370F I FUJ00152176
v 2.0 dated 27 November 2001
Page 99 of 103
WITN06650300
WITN06650300
ee Document Description Control No. URN
292. I Audit Data Extractions Process v 0.1 POINQ0230501F I FUJ00176267
dated 31 January 2002
293. I Audit Data Extraction Process v 2.0 POINQ0230504F I FUJ00176270
dated 27 January 2003
294. I Audit Extraction Support Guide v 2.0 POINQ0230505F I FUJ00176271
dated 21 May 2003
295. I Audit Data Extraction Process v 1.0 POINQ0158412F I FUJ00152218
dated 1 March 2011
296. I Audit Data Extraction Process v 2.0 POINQ0158417F I FUJ00152223
dated 23 April 2012.
297. I Audit Data Extraction Process v 3.0 POINQ0158422F I FUJ00152228
dated 4 September 2014
298. I Audit Data Extraction Process v 4.0 POINQ0158426F I FUJ00152232
dated 2 December 2016
APPENDIX 2 - PROSECUTION SUPPORT PROCESSES
299. I Network Banking Management of POINQ0158399F I FUJ00152205
Prosecution Support Procedure v 1.0
dated 26 November 2002
300. I Service Description for the Security POINQ0007914F I FUJ00001743
Management Service v 1.0 dated 6
January 2003
301. I Service Description for the Security POINQ0230507F I FUJ00176273
Management Service v 2.0 dated 2
December 2004
302. I Service Description for the Security POINQ0008171F I FUJ00002000
Management Service v 3.0 dated 6
March 2006
303. I Security Management Service: Service POINQ0094351F I FUJ00088180
Description v 1.0 dated 24 August 2006
304. I Security Management Service: Service POINQ0094508F I FUJ00088337
Description v 2.0 dated 31 December
2008
305. I Management of the Litigation Support POINQ0158406F I FUJ00152212
Service v 1.0 dated 27 October 2009
306. I Security Management Service: Service POINQ0094854F I FUJ00088683
Description v 3.0 dated 15 October 2010
307. I Management of the Litigation Support POINQ0158419F I FUJ00152225
Service v 2 dated 23 April 2012
308. I Security Management Service: Service POINQO095039F I FUJ00088868
Description v 3.5 dated 25 November
2013
309. I Security Management Service: Service POINQO095040F I FUJ00088869
Description v 4.0 dated 4 December 2013
310. I Security Management Service: Service POINQO095068F I FUJ00088897
Description v 5.0 dated 4 April 2014
Page 100 of 103
WITN06650300
WITN06650300
APPENDIX 1
Audit Trail Functional Specifications
Version Date Control Number URN
1. I 3.0 1 July 1999 POINQ0007489F FUJ00001318
2. 13.41 10 November 1999 POINQ0125542F FUJ00119343
3. I 4.0 10 November 1999 POINQ0124379F FUJ00118196
4. 144 10 April 2000 POINQ0161717F FUJ00155523
5. I5.0 15 January 2001 POINQ0007621F FUJ00001450
6. I6.0 25 February 2002 POINQ0007791F FUJ00001620
7. I7.0 17 September 2002 POINQ0007850F FUJ00001679
8. I8.0 18 October 2004 POINQO008065F FUJ00001894
9. I9.0 22 November 2004 POINQ0125546F FUJ00119347
10. I 10.0 29 June 2005 POINQ0008116F FUJ00001945
11.} 11.0 4 August 2006 POINQ0008190F FUJ00002019
12. I 12.0 8 October 2010 POINQ0008425F FUJ00002254
Page 101 of 103
WITN06650300
WITN06650300
WITNO6650300
WITN06650300
APPENDIX 2
Schedules of Approved ARQ Extraction Process Documents and Prosecution Support Process Documents
A. ARQ Data Extraction Processes
Document Date Title Internal Reference I Version Author/ URN
Originator
1. I 4 May 2000 Conducting Audit Data Extractions I IA/PRO/002 1.0 Jan Holmes FUJ00152159
at CSR
2. I 21 August 2000 Audit Extractor Support Guide TD/MAN/018 1.0 Bryan Muir FUJ00176539
3. I 22 December 2000 I Conducting Audit Data Extractions I IA/PRO/003 1.0 Jan Holmes FUJ00152167
at CSR+ Brian Mooney
Anthony Brown.
4. I 27 November 2001 I Conducting Audit Data Extractions I IA/PRO/003 2.0 Jan Holmes FUJ00152176
at Live Jane Bailey
5. 31 January 2002 Audit Data Extractions Process IA/PRO/004 0.1 Jan Holmes FUJ00176267
Jane Bailey
6. I 29 May 2002 Audit Data Extractions Process IA/PRO/004 1.0 Jane Bailey FUJ00176540
7. I 27 January 2003 Audit Data Extractions Process IA/PRO/004 2.0 Jane Bailey FUJ00176270
8. 21 May 2003 Audit Extractor Support Guide TD/MAN/018 2.0 Keith Hibberd FUJ00176271
9. 1 February 2005 Audit Data Extractions Process IA/PRO/004 3.0 Neneh Lowther FUJ00176265
10. I 1 March 2011 Audit Data Extraction Process SVM/SEC/PRO/0018 I 1.0 Penny Thomas FUJ00152218
11. I 14 February 2012 I Audit Data Extraction Process SVM/SEC/PRO/0018 I 2.0 Penny Thomas FUJ00152223
12. I 4 September 2014 I Audit Data Extraction Process SVM/SEC/PRO/0018 I 3.0 Kumudu FUJ00152228
Amaratunga
13._I 2 December 2016 __I Audit Data Extraction Process SVM/SEC/PRO/0018_I 4.0 Farzin Denbali FUJ00152232
Page 102 of 103
WITNO6650300
WITN06650300
B. Prosecution Support Processes
Document Date Title Internal Reference I Version Author/ URN
Originator
1. I 26 November 2002 I Network Banking Management of NB/PRO/003 1.0 Jane Bailey FUJ00152205
Prosecution Support
2. I 6 January 2003 Service Description for the Security I CS/SER/016 1.0 Graham Hooper FUJ00001743
Management Service Peter Sewell
3. I 2 December 2004 I Service Description for the Security I CS/SER/016 2.0 Bill Mitchell FUJ00176273
Management Service Peter Sewell
4. I 29 February 2005 Network Banking Management of NB/PRO/003 2.0 Neneh Lowther FUJ00152209
Prosecution Support
5. I 6 March 2006 Service Description for the Security I CS/SER/016 3.0 Peter Sewell FUJ00002000
Management Service Brian Pinder
6. I 24 August 2006 Security Management Service: SVM/SDM/SD/0017 1.0 Richard Brunskill I FUJ00088180
Service Description
7. I 31 December 2008 I Security Management Service: SVM/SDM/SD/0017 2.0 Peter Sewell FUJ00088337
Service Description
8. I 27 October 2009 Management of the Litigation SVM/SEC/PRO/0017 I 1.0 Penny Thomas FUJ00152212
Support Service
9. 15 October 2010 Security Management Service: SVM/SDM/SD/0017 3.0 Donna Munro FUJ00088683
Service Description
10. I 23 April 2012 Management of the Litigation SVM/SEC/PRO/0017 I 2.0 Penny Thomas FUJ00152225
Support Service
11. I 25 November 2013 I Security Management Service: SVM/SDM/SD/0017 3.5 Kumudu FUJ00088868
Service Description Amaratunga
12. I 4 December 2013 I Security Management Service: SVM/SDM/SD/0017 4.0 Kumudu FUJ00088869
Service Description Amaratunga
13. I 4 April 2014 Security Management Service: SVM/SDM/SD/0017 5.0 Kumudu FUJ00088897
Service Description Amaratunga
Page 103 of 103