WBON0000469
WBON0000469
Deloitte PRIVATE AND CONFIDENTIAL — SUBJECT TO
e LEGAL PRIVILEGE
‘Bramble’ — Draft Report
Draft for discussion
THIS IS A DRAFT DISCUSSION DOCUMENT AND REPRESENTS A WORK IN PROGRESS AND MAY
CONTAIN PRELIMINARY RESULTS OR CONCLUSIONS INCOMPLETE INFORMATION OR INFORMATION
WHICH IS SUBJECT TO CHANGE
7 October 2016
“@
This report and the work connected therewith are subject to the Terms and Conditions of the engagement letter dated 09 April
2014 (amended 11 March 2016) between Post Office Limited (POL) and Deloitte LLP. The report is produced for the General
Counsel of Post Office Limited (POL), solely for the use of Post Office Limited (POL) for the purpose of assessing assurance
sources and the design of certain controls relating to the Horizon system. Its contents should not be quoted or referred to in
whole or in part without our prior written consent, except as required by law. Deloitte LLP will accept no responsibility to any
third party, as the report has not been prepared, and is not intended for any other purpose.
WBD_000339.000001
WBON0000469
WBON0000469
Contents
1 Executive Summary 2
2 Background 8
3. Scope and Approach 12
4 — Work Performed 14
5 Assumptions and Limitations 32
Appendix 33
© 2016 Deloitte LLP Private & Confidential - Subject to Legal Privilege -
DRAFT 1
WBD_000339.000002
WBON0000469
WBON0000469
1 Executive Summary
The reader should acknowledge that all results highlighted within this report are provisional as our QA processes
remain ongoing due to outstanding evidence from Fujitsu and as a result our findings may change as the work
performed is finalised.
1.1. Background
Post Office Limited (POL) continues to respond to allegations that the “Horizon” IT system used to record
transactions in POL branches is defective and the processes associated with it are inadequate (the “Allegations”).
The ‘Allegations’ span a period of over 15 years, some pre date 2000 and others relate to 2016. In response to the
commencement of litigation proceedings, Deloitte has been instructed to plan and execute procedures and respond
to three scope areas supporting POLs ability to understand how Horizon (HNG-X) has been operated to prevent
incorrect system operation that could have resulted in Sub-postmaster detriment.
The scope areas over which Deloitte have been requested to perform procedures are as follows:
(i) Scope Area 1 - To carry out an analysis of the relevant transaction logs for branches within the
Scheme to confirm, insofar as possible, whether any bugs in the Horizon system are revealed by the
dataset which caused discrepancies in the accounting position for any of those branches.
(ii) Scope Area 2 - To carry out a full review of the use of Balancing Transactions throughout the lifetime of
the Horizon system, insofar as possible, to independently confirm from Horizon system records the
number and circumstance of their use.
(iii) Scope Area 3 - To carry out a full review of the controls over the use and capability of authorised
Fujitsu personnel to create, amend or delete baskets within a sealed audit store throughout the lifetime
of the Horizon system, insofar as possible.
Against each of these three scope areas the main body of this report will outline further:
(i) Background and context in relation to this engagement;
(ii) The approach Deloitte have taken to planning the procedures;
(iii) The testing procedures POL has requested Deloitte undertake in response to the planning activities;
and
(iv) Results of these testing procedures.
1.2. Summary of Results
Asummary of key controls tested and results are set out below. A full set of agreed procedures tested and
associated results has been included in Section 3 of this report. These should be reviewed in tandem with the
assumptions and limitations that have been included in Section 5.
6 Deloitte LLP Private & Confidential - Subject to Legal Privilege -
DRAFT 2
WBD_000339.000003
WBON0000469
WBON0000469
1.2.1 Scope Area1
Scope Area 1: To carry out an analysis of the relevant transaction logs for branches within the Scheme to confirm, insofar
as possible, whether any bugs in the Horizon system are revealed by the dataset which caused discrepancies in the
accounting position for any of those branches.
We have performed testing of key inherent system controls, together with review of some of the source code which
supports the correct operation of the system in relation to ‘bugs’ (error, flaw, failure or fault in a system that
causes it to produce an incorrect or unexpected result, or to behave in unintended ways) which may have given
rise to or contributed to the allegations under investigation. These are controls which in our scoping discussion with
POL and Fujitsu have been determined to be fundamental to protecting the integrity of transaction data within the
system.
The key controls identified were:
«All transactions on the Horizon Counter balance to zero — No Relevant Exceptions Noted.
«Transactions are atomically (either in entirety, or not at all) written to the Branch Database — No Relevant
Exceptions Noted.
¢ Digital Signature controls are applied to the Message Journal during initiation of transfer to Branch Database,
ensuring the integrity of data. - No Relevant Exceptions Noted.
e Transaction Acceptance (in relation to interface file receipt for non-Counter originated interface files) is required
by Sub Postmaster’s in order to be accepted into branch accounting records. — No Relevant Exceptions Noted.
« Recovery processes are in place for transactions in the event of connectivity failure. - Relevant Exceptions
Noted.
The exceptions noted were:
- ‘For one of the transaction recovery scenarios tested (whereby a user session is automatically logged
out after a period of inactivity — 59 minutes after the session screen being locked), it was noted that
Post Office business rules are in place for Horizon to automatically commit unprocessed transactions
to the branch database tables. This would have the effect of committing any unprocessed transactions
within a basket to the branch database. However when next authenticating into Horizon, after being
automatically logged out, the user is immediately presented with a till receipt confirming that the
transactions had been committed to the branch database.’
- ‘Where a new product is created, the recovery script could theoretically be coded to do nothing,
meaning no recovery of transactions would occur in the event of connection failure - no rollbacks or
roll-forwards would happen in this case’
The first exception could lead to an increased risk that Sub-postmasters are unaware of transactions being
posted in a power failure, although they are notified by receipt that this has occurred. The second
exception could lead to the risk of inappropriate/inaccurate resolution to a recovery situation.
The above controls were tested at a recent point in time, as they are system controls. Given this limitation the
following procedures were undertaken over change control, as changes to the system are subject to the change
control process in place over the Horizon system:
e A review of sources of assurance around change control was performed and it was noted that three ISAE3402
reports were performed covering the period April-December in 2012, 2013 and 2014 by professional services
firm Ernst & Young LLP. The scope of the report was seen to include ‘Fujitsu's system of IT Infrastructure
Services supporting POL's POLSAP and HNG-X applications’. Within each reports’ scope was a control
objective relating to change management, and in each report reviewed no deviations were noted against this
objective, or any related controls.
iC
16 Deloitte LLP Private & Confidential - Subject to Legal Privilege -
DRAFT 3
WBD_000339.000004
WBON0000469
WBON0000469
Whilst not causing an exception to one of the controls covered by the scope of our work the following exception
relating to General IT Controls over Horizon was noted:
- One Fujitsu user has access to both development and live environments of HNG-X, contravening
typically expected segregation between environments in a change control process.
Fujitsu stated that:
“Whilst we appreciate that there is lack of segregation of duties here for Gerald between Live and
Development, it is felt that there is a strong business need for this access for Gerald. He provides 4th
line/final line support for the audit service and is in regular weekly contact with the Security audit team to
assist them in resolving queries with the audit service. He is the lead designer/developer and system
owner.
Additionally there are compensating controls in place such as CCTV, and the auditing (performed by
Fujitsu) we have in place (and the technical controls around not being able to change audit items for 7
years) acts as a safeguard against anyone with access trying to change anything in an unauthorised way.”
In addition to the system controls noted above, the following analytics procedures were performed to support this
scope area:
e Review of the case data available (relevant to allegations) for transactions indicating items of risk from a
system functionality perspective. The analytical procedures outlined in Appendix 6 were undertaken, and a
number of items of interest were noted, see Appendix 6a for details and summary of findings. One finding of
note is that ‘there were 81 (0.0026%) session ids from a total of 3,074,830 which were out of balance based on
the transactional data received. Those 81 session ids out of balance related to 20 distinct branches from 118 in
total. 12 out of those 18 branches were not balanced by less than £10. The largest balance difference was
£178.09.
POL investigators have been handed this information for further investigation. In short, whilst various
characteristics were noted that could be indicative of risk within the system, further manual investigation will be
required by POL’s investigators to conclude. This has been discussed with POL management during the
course of our work.
1.2.2 Scope Area 2
To carry out a full review of the use of Balancing Transactions throughout the lifetime of the Horizon system, insofar as
possible, to independently confirm from Horizon system records the number and circumstance of their use.
In performing our procedures against this scope area, we have worked with POL and Fujitsu to identify other
methods of posting transactions which impact a branch accounts, without knowledge of the sub-postmaster which
in the context of the allegations is the more generic risk illustrated by Balancing Transactions. This highlighted
other areas of risk, such as:
* ‘Global Users’ — being central users who can access branches remotely for support purposes. Critically such
users are not able to post transactions remotely, but only when physically in the branch.
« Database and Operating System Users with sufficient privileges to post transactions directly to the database
from outside of Horizon, thereby bypassing the system controls to manage activity.
These areas have been brought into scope.
In summary across each of these areas, including Balancing Transactions, controls were noted to be operating
effectively. In particular, based on the procedures we have performed:
16 Deloitte LLP Private & Confidential - Subject to Legal Privilege -
DRAFT 4
WBD_000339.000005
WBON0000469
WBON0000469
« Logical Access rights to these sensitive functions had been appropriately restricted. - No Relevant Exceptions
Noted.
* Any writes by the Shared Service Centre (SSC) to the Branch Database (BRDB) must be audited. The
mechanism for inserting a correction record must ensure that the auditing of that action performed must be
atomic. — No Relevant Exceptions Noted.
e Access to these mechanisms is segregated from key management responsibilities (via system access rights
restrictions), meaning that even should such access rights be abused the digital signature that is included with
=a Counter and Kiosk transaction could not be spoofed. — ac rnrtor ana enorme
e It was also noted via a control walkthrough that any Transaction Corrections created by POL Finance must be
accepted by a Postmaster at branch prior to affecting branch accounts. — No Relevant Exceptions Noted.
e Inherent system controls around Global Users were tested, notably that Global users with a Role of ADMIN
cannot log onto any Branch other than Global (including Remote access controls to branch infrastructure (e.g.
Counter)). — No Relevant Exceptions Noted.
« SSC will have privileges of only inserting balancing / correcting transactions to relevant tables in the database.
SSC will not have any privileges to update or delete records in the database. — Relevant exception noted.
The exception noted was:
« ‘The control wording is not accurate. A small number of users are granted extended privileges which enable
them to update / delete records. However the control is operating in line with management's expectations.
Access to the privileged role is restricted to users explicitly authorised for this access. User actions are audit
logged, and not proactively reviewed.’
The above controls were tested at a recent point in time, as they are system controls. Given the limitations around
this the following procedures were undertaken over change control, as changes to the system are subject to the
change control process in place over the Horizon system:
« A review of sources of assurance around change control was performed and it was noted that three ISAE3402
reports were performed covering the period April-December in 2012, 2013 and 2014 by professional services
firm Ernst & Young. The scope of the report was seen to include ‘Fujitsu's system of IT Infrastructure Services
supporting POL's POLSAP and HNG-X applications’. Within each reports scope was a control objective relating
to change management, and in each report reviewed no deviations were noted against this objective, or any
related controls.
ss
An exception was noted relating to a core General IT Control exception around Segregation of Duties, please see
page 5 above where this issue is described in detail.
In addition to the system controls noted above, the following procedures were performed to support this scope
area:
e All available audit data over the use of Balancing Transactions was inspected (12/03/2010 — 28/05/2016) and it
was noted that only 1 ‘true’ Balancing Transaction was inserted, it did not relate to a branch involved in the
allegations, and the branch was made aware of the transaction prior to insertion. Other uses of the tool used to
insert Balancing Transactions were noted, however they did not affect transactional data and related to the
update of a specific flag (SU) to enable continued processing.
© 2016 Deloitte LLP Private & Confidential - Subject to Legal Privilege -
DRAFT 5
WBD_000339.000006
WBON0000469
WBON0000469
1.2.3. Scope Area 3
Scope Area 3: To carry out a full review of the controls over the user and capability of authorised Fujitsu personnel to
create, amend or delete baskets within a sealed audit store throughout the lifetime of the Horizon system, insofar as possible.
In performing our procedures against this scope area, we have worked with POL and Fujitsu to identify how
baskets of transactions flow from creation at the counter, through the sealed audit store (See Background section
for a high level overview).
Further we have tested controls over the accuracy, completeness and validity of the flow of data into the audit
store, which is used as the master data point for audit purposes. We highlight the following key controls during
scoping as being fundamental to ensuring the accuracy, completeness and validity of this data flow:
«The flow of data from counter to audit store was mapped at a detailed level (See Section 1 for high level
overview). Security controls over data at rest (when held in an intermediate location), and completeness and
accuracy controls over data in transit (transfer of data from one holding location to another) including exception
monitoring were tested. — No Relevant Exceptions Noted.
« Security controls over access to the audit servers, and audit store were tested, specifically that there are
separate roles and a clear segregation between audit server administration staff, who administer the
architecture, and Fujitsu service audit staff, who have access to retrieve data from the audit store via an audit
workstation. — No Relevant Exceptions Noted.
* Access to mechanisms for managing the digital signatures are segregated from database administration
responsibilities (via system access rights restrictions), meaning that even if such access rights be abused the
Be Saree that is included with every Counter and Kiosk transaction could not be spoofed. —
« The ATS (Audit Track Scheduler) collects files for sealing and records a log of its activities to the ATD (Audit
Track Database). In sealing a file the seal is generated using a MD5 hash algorithm. Once a file has had a seal
calculated the file is written to Centera and details are stored in the Audit Track Seal Database. — No Relevant
Exceptions Noted.
¢ Audit tracks and seals are copied to the equivalent import area on the remote audit server as part of Audit
server overnight schedule. On arrival, the sealer on the remote audit server recalculates the seal value of the
imported audit track and compares it with the original value in the imported seal file. Assuming they match, the
file is then written to the remote Audit archive. If the seals do not match, the Audit track and seal file are
moved to a holding area and an event is raised. Manual investigation is necessary to investigate the cause of
the discrepancy (which could be indicative of tampering with the data in between the two Audit servers). — No
Relevant Exceptions Noted.
e Audit tracks that are gathered at one data centre are replicated to the Audit server at the remote data centre. —
No Relevant Exceptions Noted.
e As Audit tracks are retrieved from the archive, their seals are checked (by re-application of the MD5 message
digest function) to ensure that the source data has not been tampered with while it was stored in the archive.
The digital signature check is also applied at this point to ensure data integrity. - No Relevant Exceptions
Noted.
e The remote directories from which the Audit Server gathers Audit Tracks is configured so that only the Audit
Server (or an administrator who has been explicitly given permission) is able to delete files in the directory. —
No Relevant Exceptions Noted.
« Allusers (including administrators) of the Audit Workstation and Audit Server log onto systems using two factor
authentication in conjunction with the HNG-X Active Directory system. Each user is uniquely identifiable. - No
Relevant Exceptions Noted.
© 2016 Deloitte LLP Private & Confidential - Subject to Legal Priv -
DRAFT é
WBD_000339.000007
WBON0000469
WBON0000469
e The following operating system level events on the Audit Server are audited via the System Management event
monitoring facilities:
+ Log on/Log off (including unsuccessful log on attempts)
+ File Creation, Deletion and Modification (on selected files)
+ Modifications to system configuration (incl. software configuration and account details)
+ System start up and shut down
+ Change of user rights
Relevant Exceptions Noted:
- ‘Review of the audit settings for the Audit Server noted that the audit policy change which relates to
change of user rights was set to log success events only, with failure not enabled.’
The above controls were tested at a recent point in time, as they are system controls. Given the limitations around
this the following procedures were undertaken over change control, as changes to the system are subject to the
change control process in place over the Horizon system:
e A review of sources of assurance around change control was performed and it was noted that three ISAE3402
reports were performed covering the period April-December in 2012, 2013 and 2014 by professional services
firm Ernst & Young. The scope of the report was seen to include ‘Fujitsu's system of IT Infrastructure Services
supporting POL's POLSAP and HNG-X applications’. Within each reports scope was a control objective relating
to change management, and in each report reviewed no deviations were noted against this objective, or any
related controls.
ys
An exception was noted relating to a core General IT Control exception around Segregation of Duties, please see
page 5 above where this issue is described in detail.
In addition to the system controls noted above, the following procedures were performed to support this scope
area:
« The process of Journal-Sequence-Numbering (each transaction is given a unique ID of 1 greater than the
previous transaction), whereby completeness checks are performed over these JSNs, is an optional setting
within the system (which assures the completeness of messages from the counter in the audit store). Testing
supported that this control has been enabled since 2010 and not turned off since inception in 2010.
1.3 Fundamental Limitations and Assumptions
Any procedures performed during our work against each scope area are subject to a number of assumptions and
inherent limitations.
Specifically it should be noted that controls tested/to be tested for Phase 1 relating to the system will be tested on
the system (HNG-X) operating at the time of our review, and Finance controls testing will cover controls in place at
the time of our review. It must be noted that at the time of some allegations the Legacy Horizon system was still in
use, and further there is currently a refresh of POL Finance Centre controls underway. In performing our testing we
have commented on the evidence that supports the view that the control was operating in the relevant period
where we were able to do so.
Further all analytical procedures for Phase 1 are subject to the availability of data / evidence, it is noted that while a
full transactional audit log is available for up to 8.5 years, logistical / time constraints limited the volume of data that
is able to be retrieved and interrogated. Also any controls testing is subject to the availability of evidence.
© 2016 Deloitte LLP Private & Confidential - Subject to Legal Privilege -
DRAFT
WBD_000339.000008
WBON0000469
WBON0000469
Finally our work performed for Phase 0 and proposed/tested procedures for Phase 1 are specifically limited to the
three scope areas outlined in the scope section above. Our work is focused on identifying, and performing
procedures to validate, the facts in relation to the Horizon system with regard to the three scope areas as above.
Please see Section 5 for a full list of assumptions and inherent limitations.
6 Deloitte LLP Private & Confidential - Subject to Legal Privilege -
DRAFT 8
WBD_000339.000009
WBON0000469
WBON0000469
2 Background
The Horizon system was developed by Fujitsu and is the core operational and Electronic Point of Sales (EPOS)
platform for the Post Office network. Whilst formal benchmarking data is not available, it is considered by
interviewed stakeholders to be one of the largest computer systems in existence in terms of the number of
transactions it processes on a daily basis, and it sits at the core of a complex systems estate with multiple
interfaces with other Post Office systems as well as third party systems.
The system has been in use for over 15 years and is audited by multiple parties for statutory audit, service auditor
reporting, and accreditation purposes. Given its size and scale, and the considerable intellectual property that
Fujitsu has built within the system, in relation to this piece of work, there is a significant quantity of documentation
articulating how the various modules and features comprising the system operate. Much of this documentation has
formed the focus of our review during Phase 0 of the work.
In understanding Horizon it has been important to distinguish between features which are of relevance today, and
the time period to which that relevance applies. In particular we would highlight the migration between the system
commonly referred to as Legacy Horizon, and the online variant operated today, referred to as Horizon HNG-X.
The key difference between these two iterations of the platform is the way data is stored. In the Legacy version
data was replicated between the data centre and the branches (this system was called Riposte), whilst over the
course of 2010 a migration event occurred whereby the Riposte system was replaced by the Branch Database
model, the Branch Database being a data centre only database storing the transactional and accounting data for
the branches, with a Counter application held locally within the branch which connects to the branch database as
necessary. This change may have influenced the relevance of some of the controls in existence at the present time
and care must be taken to consider this when prioritising procedures.
The Branch Database is also key to understanding the flows of data to the Audit Store given that it acts as a hub
for all branch transactional and accounting records. The diagram below provides clarity on the high level flow of
data from transaction origination through to the Audit Store:
Indicative Data Flow Overview
System Description & Detai
Counter Front end of the system, located behind the ‘counter’
‘Counter [SSK (Kiosk) Branches. Transactions are input here by the Postmaster.
ssk Configured the same way as the Counter, but for Kiosk
(Kiosk) _ outlets.
BAL Transactions are bundled into ‘Baskets’ and sent from the
an I Nraperert
to - Seat Counter / Kiosk to the BAL once they are complete. All
\ baskets must balance to 0 (Debit = Credit). Data is then
transferred from the BAL — BRDB in real time.
BROB The Branch Database is an Oracle database and sits at the
Tame Fae heart of the Horizon system. It receives transactions from
the BAL and also from other sources as illustrated.
Transactions input into BRB from sources other than the
Counter/SSK are fed back to the Counter/SSK and have to
be ‘Transaction Accepted’
‘Audit The Audit Server run a Daemon process which searches
Server for new data in the BRDB. When relevant transactions are
identified they are pulled into the Audit Server from the
BRDB. Data is held in the Audit Server for approximately 5
days.
‘Audit Store After approximately 5 days data is written from the Audit
Server to the Audit Store where it is stored semi-
permanently (currently 8.5 years of data is stored).
[ aust oeseon_] Transactional data is stored in a message journal, whereby
the completeness of the audit data is confirmed by JSN.
I__sequencing,
‘Audit Upon request from POL, Fujitsu audit staff can use the
rE) Desktop Audit Desktop to query the audit store to extract specified
data. Upon extraction from Audit Store ~ Audit Desktop, the
integrity of the data is confirmed via the digital signature
and seal check.
cD ACD is produced with the requested Audit Data.
© 2016 Deloitte LLP Private & Confidential - Subject to Legal Privilege -
DRAFT 9
WBD_000339.000010
WBON0000469
WBON0000469
This diagram shows most but not all of the data feeds associated with the Branch Database, but does show all of
the direct transactional feeds to the Branch Database. It demonstrates the convergence of the data flows at the
Branch database and the chain of subsequent data movements.
In considering these diverse data feeds a key concept is those which use a public key infrastructure (Counter) for
completeness and accuracy of the message journals to the Branch Database, versus those which use a
combination of interface controls (header and footer records) for completeness, combined with manual
interventions from Branch staff around the completeness of the associated data (being the data feeds external to
the Horizon infrastructure e.g. Paystation).
2.1 Potential Risks
Our view of the potential risks which are inherent in the high-level procedures requested by POL are listed below.
In creating this list of potential risks we have considered the high-level procedures themselves, our understanding
of the allegations made by the sub postmasters and our knowledge of the Horizon system through workshops with
POL and Fujitsu personnel.
The table below shows how each potential risk relates to POL’s scope areas:
Requested Scope Areas
1 - To carry out an analysis of the
relevant transaction logs for
branches within the Scheme to
confirm, insofar as possible,
whether any bugs in the Horizon
system are revealed by the dataset
2-To carry out a full review of the
use of Balancing Transactions
throughout the lifetime of the
Horizon system, insofar as
possible, to independently confirm
from Horizon system records the
3 - To carry out a full review of the
controls over the user and
capability of authorised Fujitsu
personnel to create, amend or
delete baskets within a sealed audit
store throughout the lifetime of the
which caused discrepancies inthe number and circumstance of their Horizon system, insofar as
accounting position for any of those _use. possible.
branches.
R1 v
R2 v
R3 v
R4 v v
RS v
Key to potential risks
R1. If Horizon does not process transactions correctly and these are not identified and resolved, these
could lead to sub postmaster financial loss.
R2. If inappropriate transactions can be created centrally by POL or Fujitsu which branch staff and sub
postmasters are unaware of, this would undermine the sub postmasters’ ability to trust the transactions in
Horizon are authentic and could cause sub postmaster financial loss.
R3. If data flow to the audit store is not complete, accurate or valid, the conclusions from the investigations
by case handlers or other parties dependent on these records cannot be relied on.
R4. If once data is in the Audit Store or extracted to support case investigation it is subject to
amendment, modification or deletion, this would also reduce confidence in case handlers’ conclusions.
© 2016 Deloitte LLP Private & Confidential - Subject to Legal Privilege -
DRAFT 0
WBD_000339.000011
WBON0000469
WBON0000469
R5. If suspense accounts mismanaged leading to sub postmaster loss, suspense account transactions are
by nature unusual and require investigation. The risk is that there are suspense account transactions which
relate to a mediated sub postmasters, are capable of being identified as such by POL and if corrected would
be favourable from the perspective of the sub postmaster.
Controls
POL management are responsible for ensuring there is a system of internal control designed to mitigate these
potential risks and that these controls are operating effectively.
No system of internal controls can be expected to guarantee the associated potential risk has not been realised.
For example, in our experience it is not reasonable to expect any enterprise software to be free from bugs
throughout the duration of its use. However, the design of enterprise software should take into account the key
risks to the application’s ongoing security and operation. Where possible inherent system controls should be
developed to prevent these potential risks being realised. Monitoring controls may also be implemented to detect
issues so they can be resolved in a timely manner by the right people. A robust change management process
should be in place to ensure only authorised changes are made and changes are tested thoroughly prior to being
implemented.
© 2016 Deloitte LLP Private & Confidential - Su t to Legal Privilege -
DRAFT 1
WBD_000339.000012
3 Scope and Approach
3.1. Scope of Work
WBON0000469
WBON0000469
We have structured our work around the three scope areas POL have asked us to review, as shown in the table
below:
Scope Area# POL Instruction
1 POL consider instructing a suitably qualified
party to carry out an analysis of the relevant
transaction logs for branches within the
Scheme to confirm, insofar as possible,
whether any bugs in the Horizon system are
revealed by the dataset which caused
discrepancies in the accounting position for any
of those branches.
2 POL instruct a suitably qualified party to carry
out a full review of the use of Balancing
Transactions throughout the lifetime of the
Horizon system, insofar as possible, to
independently confirm from Horizon system
records the number and circumstance of their
use.
3 POL instruct a suitably qualified party to carry
out a full review of the controls over the user
and capability of authorised Fujitsu personnel to
create, amend or delete baskets within a sealed
audit store throughout the lifetime of the
Horizon system, insofar as possible.
3.2 Summary of Approach and Work Performed
Proposal
POL will instruct Deloitte to
determine whether such an
analysis/review is feasible, and
if it is, to provide an indication
of the cost, time and process
that would be incurred.
POL will instruct Deloitte to
determine whether such an
analysis/review is feasible, and
if it is, to provide an indication
of the cost, time and process
that would be incurred.
POL will instruct Deloitte to
undertake this review,
throughout the lifetime of the
Horizon system, insofar as is
possible.
The work was performed in two phases. Phase 0 was ‘Discovery’ and Phase 1 was ‘Testing’.
3.2.1 Phase 0 - Discovery
This phase of work performed constituted ‘the ‘Discovery Phase’, whereby Deloitte performed initial enquiries and
investigations across the three scope areas to identify procedures which POL could undertake for each scope area.
In performing work for Phase 0, Deloitte conducted the following procedures:
« Review of relevant technical documentation as provided for previous ‘Bramble’ work, or requested and
provided by Fujitsu/POL during the course of this engagement. We have set out the documentation reviewed
during the course of this work in Appendix 1.
e Workshops with Finance staff in Chesterfield on 14" and 23 March, and 18" April 2016.
* Workshop with Fujitsu staff in Bracknell on 14" April 2016.
© 2016 Deloitte LLP Private & Confidential - Subject to Legal Privilege -
DRAFT
WBD_000339.000013
WBON0000469
WBON0000469
« Workshop with Case Handlers in Chesterfield on 8 April 2016
The aim of these procedures was:
« To enhance Deloitte’s previous understanding of the key concepts, processes, risks and controls associated
with the Horizon system, relevant to the three scope areas highlighted above (see 2.1).
e To identify the fundamental limitations and assumptions which will need to be made and considered by
management when deciding which procedures they wish to conduct during Phase 1 (see 1.3).
« As aresult of i) and ii) above the identification of possible procedures which could be adopted by management
in order to provide assurance over the risks posed in relation to the three scope areas highlighted above (see
1.3.4). We identified three core procedure types which were then utilised during Phase 1:
- Analytics — Procedures using data tools to analyse large volumes of data for particular characteristics of
interest or the absence thereof. For example verification for a given set of case data that the JSN sequence
is complete.
- Controls review and testing — Verification through walkthrough, enquiry, and subsequent evidence
gathering that controls relating to the Horizon system operate as expected or otherwise, to support in
mitigation of the associated risks. For example testing the population of Fujitsu users who can administer
the Oracle DB estate underpinning Horizon directly is appropriate.
-— Substantive procedures — Direct inspection of selected samples or information for confirmation of its
qualities or characteristics of note (Analytics is an example of ‘full population’ substantive procedures). In
this instance the main substantive procedures expected will be inspection of source code to verify that the
system functions as expected.
3.2.2 Phase 1 - Testing
Deloitte conducted the following procedures:
« Onsite review and visit to Fujitsu to test controls between May 2016 and September 2016.
e Review of case data provided by POL case handlers and tested for characteristics which could illustrate the
Horizon system has not operated as expected.
e Review of relevant technical documentation as provided for previous ‘Bramble’ work, or requested and
provided by Fujitsu/POL during the course of this engagement. We have set out the documentation reviewed
during the course of this work in Appendix 1.
© 2016 Deloitte LLP Private & Confidential - Subject to Legal Privilege -
DRAFT 3
WBD_000339.000014
WBON0000469
WBON0000469
4 Work Performed
4.1. Summary of Work Performed
For each scope area we have laid out our work performed as follows:
i) Setting the Scene — We have described in a narrative format the work we have performed, and our
understanding of the relevant subject matter.
ii) A tabular format of the procedures performed in Phase 0, and the key learnings relevant to our planning.
iii) The procedures which have been performed in Phase 1 as per POL instruction, and the findings obtained
from the performance of those procedures.
4.2 Scope Area 1
Scope Area 1: To carry out an analysis of the relevant transaction logs for branches within the Scheme to confirm, insofar
as possible, whether any bugs in the Horizon system are revealed by the dataset which caused discrepancies in the
accounting position for any of those branches.
4.2.1 Work Performed, and Analysis Results
Our procedures centred on the workshops and documentation reviews highlighted in Section 2.2.1 and 2.2.2. In
addition, specific to this scope area we reviewed the case data which had been provided to us, and assessed the
feasibility of performing analytics over the available case data in order to ascertain whether evidence of the system
not operating in accordance with expectations could be identified.
Our work has highlighted a number of fundamental system controls designed to ensure the integrity of processing,
and correct functionality. Key principles/items identified include:
i) At a holistic level, IT change control processes and procedures operate over the Horizon system, and
the related controls around testing, approval, and the overall software development lifecycle should
provide assurance over the correct operation of the system. The operational effectiveness of this
control framework has, since 2012 been assessed on a regular basis, via Service Auditor Reports
(ISAE3402 produced by EY). Further sources of assurance is provided by regular IS027001
certification and ongoing audit and attestation regime, and ongoing IT focused Internal Audit and
External Audit activity. ‘Bugs’ in the system would be more likely in an environment with inadequate
change control procedures, and the level of comfort that can be gained over such controls provides a
view on the inherent risk of such errors.
ii) There are some fundamental inherent system controls, specifically designed to support correct
processing within the system. These include:
a. Journal Sequence Numbers (JSNs) are applied to each Counter transaction within the Horizon
system. These JSNs are generated using Public Key Encryption and are used by each piece of
Counter Hardware to ‘digitally sign’ a transaction. The digital signature is passed to all latter stages
of the infrastructure including the Audit Store (and beyond). This signing process provides two
critical control points over the data captured:
i. The completeness (‘density’) of the flow of transactions for a particular Branch, meaning that
completeness of the audit trail behind transactions can be ascertained.
)16 Deloitte LLP Private & Confidential - Subject to Legal Privilege -
DRAFT 4
WBD_000339.000015
WBON0000469
WBON0000469
ii. The validity and accuracy of the transactions as any changes to a transaction after the
application of the digital signature would invalidate the signature. The Audit Store extraction
routines check for this at the point of extraction.
b. Transaction Acknowledgements — Whilst JSNs are a powerful inherent system control over the
correct origination and completeness of the Message Journals from the Counter, other feeds to the
Branch Database are not subject to this control. However as an alternative control mechanism the
interface files, which issue data to the Branch Database contain Header and Footer records which
allows Horizon to automatically check the completeness of data. In addition, Branch staff accept
these interface files into their Branch accounts via Transaction Acknowledgements, meaning these
staff are directly responsible for verification that the data being received into the Branch Database
via sources outside the Counter are valid and accurate.
c. Recovery Procedures — In acknowledging that the Horizon system is dependent upon connectivity
between a data centre, a branch, and various third parties, seven recovery processes have been
designed to combat instances when a loss of connection causes an error in the completion of
transaction processes. The recovery process used depend on the nature of the connectivity issue.
Recovery scripts designed by POL are an integral part of this process.
d. The commit of transactions to the Branch Database is all performed as one Oracle DB write action,
i.e. it is atomic in nature.
e. All transactions from the Counter are checked by Horizon to ensure they balance to zero (double
entry principle). If the Counter attempts to write a transaction which does not balance to zero, this
should be rejected via the Counter.
f. External file feeds (i.e. for data feeds not from the Counter or Kiosks) are received by the Branch
Database and into the database by Horizon before being sent to the Audit Store. Alongside this
data flow, the raw interface files are also processed directly to the Audit Store.
iii) Alongside the inherent system controls available for our review, there are two tranches of data
analytics work that we can perform to highlight the inherent risk of system failure or ‘bugs’:
a. Using the case data we have been provided with we can perform specific profiling tests which
support the operation of these inherent controls or rule out the occurrence of particular risky events
from within the relevant data set.
b. The BRSS (Branch Support Database) is a copy of the main Branch Database used by Fujitsu staff
for support purposes. This database contains the most recent six months’ worth of transactional
data (the Branch database itself contains only 5 days’ worth). Using tools already available via
Fujitsu we can profile this data to look for characteristics of risk (such as recovery situations,
Balancing Transactions, transactions posted by staff not related to a Branch etc).
© 2016 Deloitte LLP Private & Confidential - Subject to Legal Privilege -
DRAFT 5
WBD_000339.000016
4.2.2 Summary Table of Phase 0 Procedures and Conclusions
POL Instruction
Carry out an analysis of the relevant
transaction logs for branches within the
Scheme to confirm, insofar as possible,
whether any bugs in the Horizon system are
revealed by the dataset which caused
discrepancies in the accounting position for
any of those branches.
© 2016 Deloitte LLP Private & Confidential
Procedures Performed
Identified relevant business processes and areas of
interest.
Review of existing technical documentation and
identification of key inherent system controls.
Workshops with Case Handlers (POL) in order to
understand how to interpret the case data.
Workshops with Systems Architects (Fujitsu) in order
to understand how to interpret the case data and
technical documentation.
A walkthrough on-screen as to how the system works.
Subject to Legal Privilege - DRAFT
What we have discovered
There are a set of inherent system controls within Horizon
targeting the completeness, accuracy and validity of the flow of
data from Counter and other in-branch data sources, onwards
to Branch Database, and ultimately the Audit Store.
Central to these controls is the digital signature applied to each
message journal of branch transactional data sent from
Counter to Branch Database and beyond.
Connectivity issues are managed via Recovery processes, and
so issues with loss of connectivity have been built into the
design of the system from the outset, in recognition this could
be an area of potential data corruption or loss.
A strategy for our analytic procedures is to profile the available
case data for characteristics of interest in relation to the correct
operation of the system.
WBON0000469
WBON0000469
WBD_000339.000017
4.2.3 Phase 1 Procedures
Performed Procedures
WBON0000469
WBON0000469
*Note where this procedure is subject to some follow up clarification queries as a result of our ‘internal review process’ and delay in provision of evidence from Fujitsu,
and as such the below statements are subject to change. Any change in conclusions will be communicated immediately and also included in the final report.
Procedures
Controls
2. *Review of existing sources of assurance around Change Control and confirmation
of relevant coverage — plus targeted testing to attempt to identify changes relevant
to the key controls on Horizon.
Data
3. Review case data for transactions indicating items of risk from a system functionality
perspective (e.g. recovery transactions are present in the case data). See Appendix
2 and6
4. Review of population of balancing transactions (to validate population of Balancing
Validate inherent system controls around:
a) All transactions on Counter system balancing to zero.
b) Atomic write and commit controls of transactions to the Branch Database.
c) ‘Digital Signature controls applied to Message Journal during initiation of
transfer to Branch Database.
d) Transaction Acceptance in relation to interface file receipt for non-Counter
originated interface files.
e) Recovery of transactions in the event of connectivity failure.
Transactions relative to total transaction volumes)
Substantive
5.
Review source code on screen at Fujitsu headquarters which supports the key
inherent control operation around digitally signing transactions posted from the
Counter to the Branch Database.
Review source code on screen at Fujitsu headquarters which supports the key
inherent control operation around:
3 Deloitte LLP Private & Confidential - Subject to Legal Pr je - DRAFT
Findings
Controls
1a. No issues noted
1b. No issues noted
1c. No issues noted
1d. No issues noted
te. issue noted. ‘For one of the transaction recovery scenarios tested as part of
recovery scenario 6, whereby a user session is automatically logged out after a
period activity, it was confirmed that Post Office business rules are in place for
Horizon to automatically commit unprocessed transactions to the branch
database tables. As part of the walkthrough testing performed, it was observed
that Horizon is configured to automatically lock a user account after 15 minutes
of inactivity, at which point the user is required to re-enter their user credentials.
After a further period of 59 minutes of inactivity, Horizon is configured to
automatically log the user out, ending a user session and committing any
unprocessed transactions within a basket to the branch database. When next
authenticating into Horizon, after being automatically logged out, the user is
immediately presented with a till receipt confirming that the transactions had
‘been committed to the branch database. From review of the printed receipt, an
enhancement point was noted in that there is scope for the till receipt to include
further detail to the user, highlighting that an unattended transaction had
WBD_000339.000018
WBON0000469
WBON0000469
Procedures Findings
a) All transactions on counter balancing to zero. automatically been committed by Horizon to provide greater visibility to Post
b) Atomic write and commit controls of transactions to the Branch Database. Masters that a recovery session had been initiated.’
c) Digital Signature controls applied to Message Journal during initiation of transfer °
to Branch Database.
d) Transaction Acceptance in relation to interface file receipt for non-Counter 2. Issue noted.
originated interface files.
) Recovery of transactions in the event of connectivity failure.
©
It was noted one user has access to both development and live environments of
HNG-X.
Fujitsu stated that
“Whilst we appreciate that there is lack of segregation of duties here for Gerald
between Live and Development, it is felt that there is a strong business need for
this access for Gerald. He provides 4th line/final line support for the audit
service and is in regular weekly contact with the Security audit team to assist
them in resolving queries with the audit service. He is the lead
designer/developer and system owner.
Additionally there are compensating controls in place such as CCTV, and the
auditing we have in place (and the technical controls around not being able to
change audit items for 7 years) acts as a safeguard against anyone with access
trying to change anything in an unauthorised way.”
Data
3. Review of the case data available (relevant to allegations) for transactions
indicating items of risk from a system functionality perspective. The
analytical procedures outlined in Appendix 6 were undertaken, and a
number of items of interest were noted, see Appendix 6a for details and
summary of findings. One finding of note is that 'there were 81 (0.0026%)
session ids from a total of 3,074,830 which were out of balance based on
the transactional data received. Those 81 session ids out of balance related
to 20 distinct branches from 118 in total. 12 out of those 18 branches were
not balanced by less than £10. The largest balance difference was £178.09.
) 2016 Deloitte LLP Priv je - DRAFT
fe & Confidential - Subject to Legal Pr
WBD_000339.000019
WBON0000469
WBON0000469
Procedures Findings
POL investigators have been handed this information for further
investigation. In short, whilst various characteristics were noted that could
be indicative of risk within the system, further manual investigation will be
required by POL's investigators to conclude. This has been discussed with
POL management during the course of our work.
4. No issues noted. 1 Balancing Transaction identified (in the period where
data was available for review 12/03/2010 — 28/05/2016) which did not relate
to a branch involved in the allegations and was appropriately approved and
governed.
Substantive
5. No issues noted
6a. No issues noted
6b. No issues noted
6c. No issues noted
6d. No issues noted
6e Post Office have the ability to create their own EDAPC transactions. So they
can create a product, and a transaction and then also specify the recovery
script which would be initiated when any of the recovery scenarios kick in.
This could, theoretically cause an issue where a new product is created, and
the recovery script is then coded to do nothing. So if the cashier sold that
product for the customer, and then in the event of the connection going down
and the recovery process kicking in - no rollbacks or roll-forwards would happen
in this case.
DRAFT 19
Confidential
16 Deloitte LLP Privat oject to Legal Priv
WBD_000339.000020
WBON0000469
WBON0000469
Procedures Findings
Our testing has shown no evidence which would suggest this has happened,
although we have not specifically performed procedures to verify this.
) 2016 Deloitte LLP Private & Confidential - Subject to Legal Privilege - DRAFT )
WBD_000339.000021
WBON0000469
WBON0000469
4.3 Scope Area 2
Scope Area 2: Carry out a full review of the use of Balancing Transactions throughout the lifetime of the Horizon system,
insofar as possible, to independently confirm from Horizon system records the number and circumstance of their use.
4.3.1. Work Performed, and Analysis Results
Our procedures centred on the workshops and documentation reviews highlighted in Section 2.2.1 and 2.2.2
above.
Balancing Transactions are exceptional processes used by Fujitsu support staff to correct exceptional errors in
system processing/fix issues or bugs in the recording of data. The inherent controls around the integrity of data
recording are designed to ensure that such issues manifest themselves in the data on an exceptionally rare basis,
and therefore volumes of Balancing Transactions should be inherently low (substantive procedures performed
support management representation there has been only 1 true Balancing Transaction since 2010).
Balancing Transactions should not be confused with Transaction Corrections which is a more routine process,
used to centrally correct issues by POL Finance staff, which are then subject to Transaction Acknowledgement by
sub postmasters prior to being accepted into a Branches accounts.
Fujitsu have advised that whilst there have been several hundred instances of Balancing Transactions used
throughout the known lifecycle of the HNG-X system, only one has been a complex usage of the functionality, to
correct a bug around double writing of a transaction, immediately subsequent to the migration to Horizon HNG-X.
The remainder relate to switching a flag on Stock Units (SU are a Counter concept to allocate transactions to a
particular ‘sub-branch’ area to enable users to process transactions on that stock unit (following communications
failure Stock Units occasionally become locked to editing).
Our work has highlighted a number of fundamental controls which are designed within the system to control the
use of Balancing Transactions and to ensure that the use of Balancing Transactions is recorded. Key
principles/items identified include:
i) Balancing Transactions are the only transactions that do not either originate at Branch, or have to be
acknowledged / accepted by branch. As such the use of Balancing Transactions is very rare.
ii) Any writes by Fujitsu Support to BRDB must be audited (record created and stored in audit store). The
mechanism for inserting a correction record must ensure that the auditing of that action is atomic with
the insert of the record.
iii) Fujitsu Support with access to post Balancing Transactions cannot amend the related audit files.
iv) Fujitsu Support will have privileges of only inserting balancing / correcting transactions to relevant
tables in the database. They will not have any privileges to update or delete records in the database.
v) There are various inherent system controls around Balancing Transactions, notably that each
Balancing Transaction must only contain 1 transaction (single SQL statement) and the balancing
transaction module can only be run by limited appropriate personnel.
In assessing the risk posed by Balancing Transactions we have also enquired as to additional ‘privileged account’
transactions which could also be used to post transactions centrally without the knowledge of Branch staff. These
enquiries have highlighted two additional areas of consideration against this risk:
a. Global Users of the Horizon System — These are users that can log on at any HNG-X Branch, and are used
for a number of purposes including global user administration.
'e LLP Private & Confidential - Subject to Legal Privilege -
DRAFT 21
WBD_000339.000022
WBON0000469
WBON0000469
b. Other ‘Superusers’ — At various layers of the Horizon infrastructure there exist accounts with privileged
access rights which could be used to modify or insert data relevant to transactions at branches should they
not be adequately controlled. For example a superuser account on the Oracle DB forming the nucleus of
the Branch Database could insert transactions directly onto the backend (effectively Balancing
Transactions are a specialised ‘legitimised’ way of using such Oracle access).
Anumber of key controls were noted to operate on Horizon to mitigate these broader ‘superuser’ risks:
vi) Global Users are subject to two fundamental controls reducing their risks. The first is that they cannot post
transactions in a branch unless they are physically present at that branch. The second is that the Global
Admins can only create users and there is therefore a Segregation of Duties between users who can
create users, and users who can post transactions.
vii
Superuser activity is monitored via log files which are transferred to the Audit Store following aggregation
by the Event Management System which collects log files from across the Horizon estate. Regardless of
this control, for transactions related to the Counter and Kiosks any attempt to insert transactions into the
database by an individual with the privileged access rights to do so, would be identifiable due to the Digital
Signature process applied to Message Journals from the Counter. To circumvent this a ‘superuser’ would
require the relevant access rights to the key management infrastructure which controls the Digital
Signature processes, and therefore the segregation of duties between such infrastructure and the
remaining Branch infrastructure is a key control.
Alongside the inherent system controls around balancing transactions, and the completeness and accuracy of the
audit log of Balancing Transactions available for our review, there are various data analytics procedures which can
be performed:
vii) As discussed above Fujitsu highlighted that while the Balancing Transaction module has been used
approximately 200 times in the past 7.5 years, only 1 of these uses has been a ‘complex’ Balancing
Transaction. Analytical procedures could be performed to validate the number and nature of Balancing
Transactions which have been performed in:
a. The Case Data available
b. The BRSS most recent 6 months data available
c. The full period of data available — (7.5 years)
Sample (or full population) testing could then be performed to validate that for all Balancing
Transaction records (except the 1 known Balancing Transaction, for which the branch was aware of)
no transactional postings were made using Balancing Transactions.
© 2016 Deloitte LLP Private & Confidential - Subject to Legal Priv -
DRAFT 20
WBD_000339.000023
4.3.2. Summary Table of Phase 0 Procedures and Conclusions
POL Instruction
POL consider instructing a suitably qualified
party to carry out an analysis of the relevant
transaction logs for branches within the
Scheme to confirm, insofar as possible,
whether any bugs in the Horizon system are
revealed by the dataset which caused
discrepancies in the accounting position for
any of those branches.
© 2016 Deloitte LLP Private & Confidential
Procedures Performed
Identified relevant business processes and areas of
interest.
Review of existing technical documentation and
identification of key inherent system controls, and
support in interpreting the transactional data.
Workshops with Systems Architects (Fujitsu) in order
to understand how to interpret the technical
documentation and the availability of Audit Store data.
Awalkthrough on-screen as to how the system works.
Subject to Legal Privilege - DRAFT
What we have discovered
There are a sequence of inherent system controls within
Horizon which ensure Balancing Transactions have certain
standard characteristics, use of them is controlled, and usage
is recorded in the Audit Store.
Other privileged access rights which would lead to similar risks
of central posting of transactions with sub postmaster
knowledge, such as Global Users, and ‘superuser’ accounts on
the Horizon infrastructure, are also subject to key controls,
most notably the segregation of duties between the key
infrastructure for digital signatures and the infrastructure
supporting the processing of Branch transactions. These
controls have been tested at a point in time.
The strategy to be adopted across our analytical procedures
will be to Investigate a sample / full population of all Balancing
Transaction records found to validate the branch was aware of
their usage / no transactional postings were made in the
balancing transaction.
WBON0000469
WBON0000469
WBD_000339.000024
4.3.3 Phase 1 Procedures
Performed Procedures
WBON0000469
WBON0000469
*Note where this procedure is subject to some follow up clarification queries as a result of our ‘internal review process’ and delay in provision of evidence from Fujitsu,
and as such the below statements are subject to change. Any change in conclusions will be communicated immediately and also included in the final report.
Procedures
Controls
1. *Validate inherent system controls around Balancing Transactions (See Appendix 3
for detail of controls A — 1).
2. Validate any writes by Fujitsu support staff to BRDB must be audited. The
mechanism for inserting a correction record must ensure that the auditing of that
action performed is atomic.
3. Validate Fujitsu support staff cannot amend audit files for Balancing Transactions.
4. *Validate Fujitsu support staff only have privileges for only inserting balancing /
correcting transactions to relevant tables in the database. Confirm SSC do not have
any privileges to update or delete records in the database.
5. *Validate broader population of Balancing Transaction controls identified. (See
Appendix 3a for detail of controls A—N)
6. *Validate there is a Segregation of Duties between BRDB Administration and Key
Management Software Administration.
7. Validate inherent system controls around Global Users, notably that Global users
with a Role of ADMIN cannot log onto to any Branch other than Global (Including
Remote access controls to branch infrastructure (e.g. Counter).
Data
8. Review case data for Balancing Transactions to validate population of Balancing
Transactions relative to total transaction volumes (Balancing transactions should be
inherently rare, and only deployed in response to actual loss/bugs in code.)
9. Review full population (already extracted by Fujitsu - 7.5 years) of balancing
transactions (sample vs full population depending on feasibility) to validate the
3 Deloitte LLP Private & Confidential - Subject to Legal Pr je
Findings
Controls
No issues noted
No issues noted
No issues noted
age pp
‘Through discussion with Fujitsu management it was noted that the
control wording is not accurate. A small number of users are granted
extended privileges which enable them to update / delete records.
However in mitigation this access is appropriately restricted to
authorised users. Users do not have the ability to bypass this role
restriction by running SUDO command. User actions are audit logged
and not proactively reviewed, and all instances of users being granted
the APPSUPP role are also captured in audit logs.’
5. Issues noted for control 2A and 2C.
2a finding noted — ‘Through discussion with Fujitsu management it was
noted that the control wording is not accurate. A small number of users are
granted extended privileges which enable them to update / delete records.
However in mitigation this access is appropriately restricted to authorised
users. Users do not have the ability to bypass this role restriction by running
SUDO command. User actions are audit logged and not proactively
reviewed, and all instances of users being granted the APPSUPP role are
also captured in audit logs.’
2c finding noted — ‘The technical document <DESAPPLLD0142> is
inaccurate. The user OPS$SUPPORTTOOLUSER does require update
WBD_000339.000025
WBONO0000469
WBON0000469
Procedures Findings
branch was aware of their usage / no transactional postings were made in the access to the table BRDB_BRANCH_INFO, however the document does not
balancing transaction. reflect this.’ This is a documentation finding only.
Substantive 6. No issues noted.
10. Review source code on screen at Fujitsu headquarters which supports the key 7. Noissues noted
inherent control operation around Balancing Transactions. Léa
at
11. Review of Transaction Correction source code on screen at Fujitsu headquarters to
validate that Transaction Corrections must be accepted by Branches, in order to 8. Review of the case data available (relevant to allegations) for
validate Balancing Transactions are the only transactions Branches would not have transactions indicating items of risk from a system functionality
to accept. perspective. The analytical procedures outlined in Appendix 6 were
12. Review the 9 Balancing Transaction Templates to validate balancing transactions undertaken, and a number of items of interest were noted, see Appendix
would, if the template was followed, logically perform as expected. 6a for details and summary of findings. One finding of note is that ‘there
were 81 (0.0026%) session ids from a total of 3,074,830 which were out
of balance based on the transactional data received. Those 81 session
ids out of balance related to 20 distinct branches from 118 in total. 12
out of those 18 branches were not balanced by less than £10. The
largest balance difference was £178.09.
13. Walkthrough a Transaction Correction being raised by SCC, and the notification /
acceptance of it by a branch.
POL investigators have been handed this information for further
investigation. In short, whilst various characteristics were noted that
could be indicative of risk within the system, further manual investigation
will be required by POL’s investigators to conclude. This has been
discussed with POL management during the course of our work.
9. No issues noted. 1 Balancing Transaction identified (in the period where
data was available for review 12/03/2010 — 28/05/2016) which did not
relate to a branch involved in the allegations and was appropriately
approved & governed.
Substantive
10. No issues noted
11. No issues noted
3 Deloitte LLP Private & Confidential - Subject to Legal Pr je - DRAFT 5
WBD_000339.000026
WBONO0000469
WBON0000469
12. No issues noted
13. No issues noted
© 2016 Deloitte LLP Private & Confidential - Subject to Legal Privilege - DRAFT 26
WBD_000339.000027
WBON0000469
WBON0000469
4.4 Scope Area 3
Scope Area 3: Carry out a full review of the controls over the user and capability of authorised Fujitsu personnel to create,
amend or delete baskets within a sealed audit store throughout the lifetime of the Horizon system, insofar as possible.
4.4.1 Work Performed, and Analysis Results
Our procedures centred on the workshops and documentation reviews highlighted in Section 2.2.1 and 2.2.2
above.
For this specific scope area our procedures centred on understanding the specific controls and processes around
protecting the integrity of data from inception to Branch Database, and subsequently to the Audit Store. Our work
highlighted a number of core concepts relevant to understanding the related risks and controls during this data
flow:
In essence the data journey can be divided into a number of distinct phases:
a. Transaction initiation within either the Counter, Kiosk, or ‘third party interface source’, and
subsequent interface to the Branch Database.
b. Archival from the Branch Database to the Audit Server.
c. Sealing of Audit Tracks via MD5 Message Digest and Archive to the Audit Store itself (Now based
on Eternis technology).
d. Subsequent Retrieval of Tracks, validation via the ARQ (Audit Track Retrieval) process, and
Investigator validation on the received data.
e. Non-Branch Transaction Data Records of Relevance
A. Transaction Initiation within either the Counter, Kiosk or ‘third party interface source’
i) For Counter and SSK (Kiosk) initiated transaction data, the JSN remains a core element of control for
the Audit Store process as it validates the origination and completeness of data for a particular Counter
and is independent of the MD5 message digest elements.
ii) Given the wealth of ‘data at rest’ (stored in a directory/database awaiting onward processing) and ‘data
in transit’, security controls over access to ‘data at rest’ and interface controls over monitoring
completeness and accuracy of ‘data in transit’ are both pertinent. However the JSN concept provides
assurance regardless given interruptions in the sequence, or mis-match between signature value and
message content, would highlight downstream risks of data corruption.
iii) The other interfaces pertinent to our understanding have been represented by Fujitsu systems
architects to be:
a. Logistic Feeder Service
b. Post and Go (discontinued in 2015, but relevant prior to that date)
c. Near Real Time (NRT) feeds
d. Paystation
e. Camelot
iv) For non-Counter and Kiosk interfaces to the Branch Database completeness is provided by the
interface file header and footer record, with accuracy and validity provided by manual inspection by
Branch staff themselves via the Transaction Acknowledgements process.
)16 Deloitte LLP Private & Confidential - Subject to Legal Privilege -
DRAFT
WBD_000339.000028
WBON0000469
WBON0000469
v) For many of these interfaces the Post Office Data Gateway (PODG) provides the point of entry to POL
infrastructure.
B. Archival from the Branch Database to the Audit Server
i) Archival from the Branch Database of data take place to the Audit Server (which is the gateway to the
Audit Store infrastructure) in accordance to an automated routine which is central to the operation of
the Horizon system. If archival did not take place then very quickly the system would run out of
available capacity. Two intermediate directories are used to hold records prior to transfer to the Audit
Server.
ii) As referenced above both ‘data at rest’ and ‘data in transit’ controls are therefore relevant to this stage
of the process.
C. Sealing of Audit Tracks via MD5 Message Digest and Archive to the Audit Store itself
i) The Audit Track Gatherer (ATG) is a routine which is permanently scanning for new Audit files on the
upstream infrastructure (including the Branch Database) which are then copied to the Audit Server,
sealed by the Audit Track Sealer (ATS), using the MD5 message digest algorithm, copied to the Audit
Store Eternis architecture itself, and then purged from the Audit Server when copied across.
ii) The Audit Server maintains a database of sealed files and their seal values, for later interrogation when
locating files, and validating their integrity has not been violated.
iii) Therefore once again both ‘data at rest’ and ‘data in transit’ controls are relevant to this stage of the
process.
iv) Once on the Eternis hardware which has now replaced the EMC Centera hardware solution, the data is
subject to a number of controls around access, deletion and amendment, all of which are designed to
maintain the integrity of the audit trail during storage. Both EMC Centera (historical solution) and
Eternis (current solution) are specialised hardware solutions for the storage of audit trail data intended
to be used forensically.
v) Previously there was a seven year limit to the retention of data in the Audit Store, after which it was
purged by the system in line with Retention requirements. Given recent history this policy has recently
been changed to indefinite retention of all Audit Store data. As a result all transactions should be
available for as long as the Audit Store continues to exist from 04/10/2007, and therefore a complete
audit trail of all transactions ever posted on Horizon HNG-X should exist (given the migration date).
D. Subsequent Retrieval of Tracks, validation via the ARQ (Audit Track Retrieval) process, and Investigator
validation on the received data itself
i) Extraction of the data from the Audit Store is via a defined process known as the ARQ process. A
specialised Audit Desktop estate is utilised to interrogate the Audit Server database, retrieve relevant
sealed files, process the data, and burn to CD (or email as a data file), whereby it is made available to
POL investigative staff.
ii) There are a number of logical access controls operating over this process, including role based access
mechanisms, a strict ‘segregation of duties’ from POL staff and audit logs over the process.
iii) Upon receipt of the data files POL investigators carry out a number of additional checks themselves in
order to validate the data integrity.
E. Non-Branch Transaction Data Records of Relevance
i) Alongside the Branch Database data flowing into the Audit Store there are a number of other relevant
data sources:
© 2016 Deloitte LLP Private & Confidential - Subject to Legal Privilege -
DRAFT
WBD_000339.000029
WBON0000469
WBON0000469
ii) Interface files received from third party systems which are then processed into the Branch database,
are also sent directly to the Audit Store as raw files, allowing potential future reconciliation between the
two data sources.
iii) The Event Management System captures System Audit Logs from across the Horizon estate, and
processes these to the Audit Store.
Given the above understanding of the process gained from our work to date, our approach to assurance against
this scope area is largely based upon controls assurance, in combination with some limited analytics procedures to
support completeness, security and integrity of the data throughout the relevant data flows.
6 Deloitte LLP Private & Confidential - Subject to Legal Privilege -
DRAFT 29
WBD_000339.000030
4.4.2 Summary Table of Phase 0 Procedures and Conclusions
POL Instruction
Carry out a full review of the
controls over the user and
capability of authorised Fujitsu
personnel to create, amend or
delete baskets within a sealed
audit store throughout the lifetime
of the Horizon system, insofar as
possible.
____ Procedures Performed
Identified relevant business processes
and areas of interest.
Review of existing technical
documentation and identification of key
inherent system controls, and support in
interpreting the transactional data.
Workshops with Systems Architects
(Fujitsu) in order to understand
technical documentation.
Awalkthrough on-screen as to how the
system works.
Walkthrough of Audit Store specific
controls in order to determine relevance
and accuracy for inclusion within the
scope of our work.
‘e LLP Private & Confidential - Subject to Legal Privilege -
WBON0000469
WBON0000469
What we have discovered
The Branch Database is a key point in the
data journey at which all Branch relevant
data whether generated by the Counter or by
a third party data source external to Horizon
will interface to.
There are a number of intermediate points at
which data is at rest during the flow of data to
the Audit Store, and understanding the
Security controls over such data will support
the integrity of data flowing into the Audit
Store.
Regardless of the opportunity or otherwise
for interception and tampering of data pre its
arrival in the Audit Store, for key data
originating from the Counter and the Kiosks,
the digital signatures should highlight any
tampering with data prior to its usage within
the Cases.
The Case data provided can be reviewed
with a view to re-performing the key integrity
checks performed by investigators, over the
completeness and accuracy of the data.
The Audit Store controls should have
remained relatively constant over the period
of allegations when considering those
relating to infrastructure downstream of the
Branch Database. This is due to the HNG-X
project which has influenced a number of
other key control areas, leaving the Audit
Store architecture relatively untouched.
WBD_000339.000031
WBON0000469
WBON0000469
4.4.3. Phase 1 Procedures
Performed Procedures
*Note where this procedure is subject to some follow up clarification queries as a result of our ‘internal review
process’ and delay in provision of evidence from Fujitsu, and as such the below statements are subject to change.
Any change in conclusions will be communicated immediately and also included in the final report.
Procedures
Controls Controls
1. *Validate Audit Store controls identified (See Appendix 4 for detail of controls 1A— 1. No issues noted
10).
2. No issues noted
2. *Digital Signature controls applied to Message Journal during initiation of transfer to
Branch Database. 3. No Issues Noted except fc
3. *Additional Audit Store Controls identified (See Appendix 4a for detail of controls 3A 3A finding - ‘Review of the
-3F). the audit policy change wt
log success events only, v
4. “Identification of Audit Store Data Flows at a Detailed Level, including security 4. No issues noted
controls over data at rest, and completeness, accuracy and validity controls over
data in transit.
Data
Data
NA N/A
Substantive Substantive
5. Review source code on screen at Fujitsu headquarters which supports the key 5. No issues noted
inherent control operation around digitally signing transactions posted from the
Counter to the Branch Database. re
6. Identification of changes relevant to the Audit Store from review of historical
documentation, and validation that the Audit Store has remained broadly consistent
over time from a controls perspective for the period relevant to the allegations.
© 2016 Deloitte LLP Private & Confidential - Su t to Legal Privilege -
DRAFT 1
WBD_000339.000032
WBON0000469
WBON0000469
5.1 Assumptions and Limitations
Our work has been subject to the following exclusions:
1. We have not verified or tested any information or assertions provided directly by you, or directly or
indirectly by third parties;
2 For scope areas 1, 2 and 3, only matters relating to Horizon Features and Audit Store within the
Horizon processing environment have been considered during our workshops and discussions;
3. We have not provided a legal or any other opinion as to the completeness and accuracy of processing
of Horizon at any point throughout the work;
4. We have not had direct contact with any third parties other than named contacts that you have
provided to us (Appendix 1);
ce We have not reviewed any contractual provisions in place between you and third parties;
6. Our work was limited by gaps existing in the information available, relating to both the granularity of
information and the existence of the Horizon Features’ over the entire timeline of operation of Horizon
and suspense account process documentation. The effect of which is that there are in gaps within what
we are able to comment upon over this timeline;
7. We have not validated or commented on the quality of the Assurance Work? supplied to us.
Our work was also based on the assumption that the documents provided and assertions made are a complete and
accurate representation of the Horizon design, audit store process and suspense account process. We therefore
cannot comment as to whether other processes would need consideration in the context of the Matters.
We have performed work on control in place and operating at the time of the review, and not those operating at the
time of the allegations. Other evidence has been obtained, where available, to provide evidence as to whether the
control was likely to have operated at the time of the allegations.
1 “Horizon Features” is a term we have introduced to represent those features of the Horizon processing environment, including IT management
and business use controls, which provide that:
Movements in Branch ledgers have the full ownership and visibility of sub-postmasters; and
* Audit trails kept by the system are complete and accurate.
2 Since its implementation in branches, POL has commissioned or has received a number of pieces of work relating to the Horizon processing
environment, to provide comfort over its integrity. This work, referred to in our report as the “Assurance Work”, provides documented assertions
relating to aspects of the design and operation of the Horizon processing environment. The Assurance Work includes IT project documents;
operational policies and procedures; internal and external investigations and reviews; independent audits; and emails confirming otherwise
verbal assertions.
© 2016 Deloitte LLP Private & Confidential - Subject to Legal Privilege -
DRAFT 32
WBD_000339.000033
Appendix 1
Documents Reviewed
Document Ref
DES/APP/HLD/0047
DES/APP/HLD/0020
DES/APP/HLD/0030
DES/APP/HLD/0029
ARC/SOL/ARC/0006
DEV/APP/LLD/O065
DEV/APP/LLD/0014
DEV/APP/LLD/0142
DES/APP/SPG/0001
DEV/APP/LLD/0199
DES/APP/HLD/0035
DES/APP/IFS/0002
DES/APP/IFS/0012
DES/APP/HLD/0083
DES/APP/HLD/0021
DES/APP/IFS/0007
DES/APP/IFS/0001
DES/APP/HLD/0049
DES/APP/HLD/0057
ARC/SOL/ARC/0001
DEV/APP/LLD/0071
POLSAP/DES/APP/STG/0001
DEV/INF/ION/0001
DES/SEC/HLD/0003
DES/APP/HLD0041
DES/APP/IFS/0018
DES/APP/HLD/0012
ARC/SEC/ARC/0003
DEV/APP/LLD/0204
DES/APP/HLD/0070
DEV/APP/LLD/0151
© 2016 Delo
DRAFT
LLP Prive
Donfidential - Subj
Document Title
HNG-X Counter Application High Level Design
Branch Database High Level Design
Audit Data Collection & Storage High Level Design
Audit Data Retrieval High Level Design
HNG-X Architecture - Global Users
BRDBCO002 — BRDB Message Journal Auditing LLD
Host Branch Database Audit Archive Purge Low Level Design
Host BRDB Transaction Correction Tool Low Level Design
Host branch database support guide
Schema definition for branch database, standby branch database
and branch support system
Exceptions and logging frameworks high level design.
HNG-X:RDDS to Branch Database - Counters & HBS Reference
Data and Memo Submission Interface Specification
BAL Service Interface Specification
HNG-X Counter Subsystem : Recovery Management
_ Branch Database Scheduling High Level Design
Branch Database to Legacy Host Interface Specification
HNG-X: RDMC / RDDS to Branch Database Application Interface
Specification
HNG-X Generic Reports Data Extract HLD
HNG-X Counter Infrastructure: Service and Process Control High
Level Design
I HNG-X Solution Architecture Outline
_ Audit Data Retrieval Low Level Design
POLSAP Archiving Strategy
Archive Server Configuration
HNG-X KEY MANAGEMENT HIGH LEVEL DESIGN
HNG-X Counter Applications: Business Logic Subsystem High
I Level Design
XML Message Audit between Counter or HBS and BAL/OSR
DVLA Internal Web Service High Level Design
HNG-X Technical Security Architecture
Host BRDB Update Outstanding Recovery Transaction Tool Low
Level Design
Host Applications Monitoring High Level Design
HNGX BRDB HOST: BRANCH SUPPORT DATABASE LOW
LEVEL DESIGN
st to Legal Privile
WBON0000469
WBON0000469
WBD_000339.000034
Individuals Interviewed
Name
Patrick Bourke
Mark Underwood
Job Title
POL — ‘Bramble’ Project Manager
POL — ‘Bramble’ Project Manager
WBON0000469
WBON0000469
Rodric Williams
Rod Ismay
Lorraine Garvey
Sarah Haywood
Tracy Middleton
Michael Harvey
Pete Newsome
Torstein Godeseth
Steve Bansal
Alan Holmes
Gerald Barnes
Gareth Seemungal
© 2016 Deloitte LLP Private & Conf
DRAFT
POL — POL Legal
POL - Head of Finance Service Centre
POL - Enquiries Manager
POL - Finance Team Leader
POL - Finance Team Leader
Fujitsu -Head of Commercial
Fujitsu - Business Change Manager
Fujitsu - Chief Architect
Fujitsu - Senior Service Delivery Manager
Fujitsu - Customer Solution Architect
Fujitsu -Senior Software & Solutions Designer
Fujitsu - Senior Software & Solutions Designer
idential - Subject to Legal Privilege -
WBD_000339.000035
WBON0000469
WBON0000469
Appendix 2
Scope area I — Potential Analytics Procedures
Ref Analytics Procedure
A Completeness Test - Identify gaps in audit log sequencing
B Completeness Test - Identify gaps in transaction times during working hours
c Completeness Test - Identify two user logon events in sequence without the expected logoff event in
between, an indicator of a connectivity issue
D Completeness Test - Identify recovery transactions
E Accuracy Test - Identify zero valued transactions
F Accuracy Test - Identify branches which are out of balance based on transactional data available (should
not be possible based on inherent system controls).
G Integrity Test - Identify transactions posted by non-branch users without subsequent branch
acknowledgement.
H Integrity Test - Identify balancing transactions.
© 2016 Deloitte LLP Private & Confidential - Subject to Legal Privilege -
DRAFT 35
WBD_000339.000036
WBON0000469
WBON0000469
Appendix 3
Scope area 2 — Balancing Transactions Controls
Ref
A
Control Description
SSC will have privileges of only inserting balancing / correcting transactions to relevant tables in the
database. SSC will not have any privileges to update or delete records in the database.
If the process fails (e.g. transaction file is found to be invalid), then the transaction file will not be moved
and an error message will be written to standard output.
Any writes by the SSC to BRDB must be audited. The mechanism for inserting a correction record must
ensure that the auditing of that action performed must be atomic. There also needs a level of obfuscation
to ensure that the audit mechanism is robust.
Appendix 3a
Scope area 2 — Balancing Transactions Controls (Broader population)
Ref
A
B
c
DRAF
Control Description
All inserts will be audited in the table BRDB_TXN_CORR_TOOL_JOURNAL.
The PL/SQL package PKG_BRDB_TXN_CORRECTION will be owned by Oracle user
“OPS$SUPPORTTOOLUSER’.
The PL/SQL package PKG_BRDB_TXN_CORRECTION will execute with the permissions of the
OPS$SUPPORTTOOLUSER account and can only insert rows into the transaction tables as controlled by
an entry in BRDB_SYSTEM_PARAMETERS. The account will not have update or delete privileges.
Each of the transaction tables that are allowed to have balancing transactions inserted on them has an
associated template file. Each file contains a template of an INSERT statement for that table, in the
required format, and listing all of the columns on the table. Users should create their own transaction file
based upon the relevant template file, substituting the values they require into the SQL. Note that some of
the column values specified in the template should not be changed — these are annotated with comments.
as appropriate.
When execution is complete the file is then moved to directory ‘/app/brdb/trans/support/brdbx015/output’
and the log file is created in directory ‘/app/brdb/trans/support/brdbx015/log’. Log file will be named using
the following convention:
<transaction_file_name>_<CCYYMMDDHHMISS>.log
Access to these 2 directories is appropriately restricted.
It is expected that only a small number of skilled staff will run this tool and that they will have detailed
guidance as to when and how to use the tool (For example by restriction of staff to
“OPS$SUPPORTTOOLUSER’).
From the Unix command prompt, execute the following
./BRDBX015.sh MyTransactionFile.sql 2001
where the first parameter is the transaction file name and the second parameter is the branch code where
the balancing transaction is going to be applied. Note that the branch code must exist in the database, and
must not be for a closed branch. If this is not the case, then an error message will be shown and the run
aborted.
The correction tool places a number of constraints on the contents of the transaction file. These are
necessary in order to provide a defined baseline upon which it can base its operation. If any of the
constraints are violated then validation will detect it and abort the run with a meaningful error message.
The constraints are as follows:
6 Deloitte LLP Private & Confidential - Subject to Legal Privilege -
WBD_000339.000037
WBON0000469
WBON0000469
Ref Control Description
¢ The transaction file must be less than 32K in size
e The transaction file must only contain Unix-style end of line markers (EOL), not DOS format end of line
markers (CR/EOL)
¢ The transaction file can only contain a single SQL statement. If more than one balancing transaction is
required then more than one transaction file must be created, each of which is executed with a separate run
of the tool
e If the transaction file contains an introductory comment, then it must be a ‘/* ...... *P style comment, not
A eee * style comment
¢ The closing ‘*/’ of the introductory comment must have a trailing space (i.e. ‘..... */‘)
¢ The run symbol at the end of the SQL must be a ‘;’ , not ‘/’, and must have a trailing space (i.e. *.....; ‘)
e The SQL must be a valid SQL statement according to the normal Oracle SQL parsing rules (e.g. valid
syntax, objects accessible etc)
¢ The SQL must begin with ‘INSERT INTO OPS$BRDB.’ and be of the form ‘INSERT INTO ..... SELECT
tame FROM dual, (SELECT ..... FROM .... WHERE .....)’.
° The table name must be one of the tables named in the
BRDB_TXN_CORRECTION_ALLOWED_TABLES1 or
BRDB_TXN_CORRECTION_ALLOWED_TABLES2 configuration parameters
¢ All of the columns that exist on the table in question must be explicitly named. It is not necessary for
every listed column to be on a separate line, but this is advisable for readability.
e The values to be inserted must be provided by the ‘SELECT ... FROM dual ...’. Each value must be on
a separate line. Trailing comments are allowed, but must be a ‘-- .....’ style comment. Any such comment
must not include any commas. All columns must have values provided for them (even if that value is NULL).
* Certain columns are common between a subset of the transaction tables. In some cases, these columns
should be set to the same value no matter what table is in use. With the exception of the bind variables
listed earlier, the value that the SQL will try to insert is under the control of the user (i.e. it is determined by
the value specified in the SQL). However, the tool can be configured to validate that the value specified in
the SQL matches that expected. In order to do this, set the
BRDB_TXN_CORRECTION_ENFORCED_VALUES configuration parameter to include the field and the
required value.
The parameter is populated as a comma-delimited list of name/value pairs, where the name is the name of
the column name, and the value is the value to be enforced. As released, this configuration parameter is
set to:
NODE_IID=99,APP_SERVER_NODE_NAME=999,BRANCH_USER=:bind_SSC_user,BRDB_INSTANCE
_NAMI ind_instance_name
which, for example. ensures that if a ‘node_id’ column exists on the transaction table, it's value is specified
as 99. If there is no ‘node_id’ on the transaction table, then no value is enforced for that field. Note that if
the parameter does not exist, then no values are enforced in the SQL.
I The SQL statement being executed will be logged in the table BRDB_TXN_CORR_JOURNAL. The format
of the data to be written to the column JOURNAL_XML is:
“<?xml version="1.0" encoding="UTF-8"?>
<Support_Insert>
<Unix_User>Unix User Name</Unix_User>
<Oracle_User>Oracle User Name</Oracle_User>
<Sql>SQL Statement</Sql>
</Support_Insert>”
where :
e Unix User Name is the Unix user name under which the user logged in
* Oracle User Name is Oracle user that is carrying out the actual insert i.e. SUPPORTTOOLUSER
¢ SQL Statement is the final (i.e. after substituting actual values for bind variables) SQL that is executed
to insert the balancing transaction
© 2016 Del
DRAFT
fe LLP Private & Confidential - Sul
Privilege -
WBD_000339.000038
WBON0000469
WBON0000469
Ref Control Description
J As records are being written to the audit files, the process must optionally be able to monitor if the set of
Journal-Sequence-Numbers for a node in a Branch is dense. The check should only be performed when
the value of mandatory System-Parameter ‘JOURNAL_SEQ_DENSE_SET_CHECK_ENABLED’ is
“TRUE”. When a missing journal entry is encountered, a message should be written on standard output
along the lines of “...records between sequence numbers M and N are missing...”. Once the list of
auditable messages for a node is completed, an Operational exception should be raised to indicate the
count of missing sequence numbers. Duplicate records are not possible due to the primary key on this
table.
K Unix shell script BRDBX015.sh which is in the /app/brdb/trans/support/brdbx015 directory. It is deliberately
kept separate from the standard $BRDB_SH directory so that access to the script and the associated
components can be restricted to authorised users. The shell script calls the PL/SQL package
PKG_BRDB_TXN_CORRECTION.
L PL/SQL package PKG_BRDB_TXN_CORRECTION, which resides within the Branch Database and is
owned by Oracle user OPS$SUPPORTTOOLUSER. The PL/SQL package is the component that validates,
creates and audits the balancing transaction.
M If an Oracle node/instance failure occurs, the utility will fail with an error code of 99. For all other failures, it
will fail with an error code of 1 and log an operational exception in BRDB_OPERATIONAL_EXCEPTIONS.
N The SQL in the transaction file is validated as follows. Any validation failures are displayed to standard
output and logged to the log file.
e Check that the file does not contain any carriage returns, indicating DOS format EOL markers
* Check that the SQL in the transaction file parses according to the standard Oracle rules (e.g. syntax,
privileges etc). This is done using the standard Oracle DBMS_SQL.PARSE procedure.
e Check that there is only a single SQL statement in the transaction file. Note that in most cases, this will
be detected by the previous parsing step. However, the fact that the parsing does this is not described in
the Oracle documentation, so it may be changed in future releases of Oracle. Therefore, this validation
provides security if the behaviour of the Oracle procedure is changed at a later date.
¢ Check that the SQL begins with ‘INSERT INTO OPS$BRDB.”
e Check that the table named in the SQL is one of the tables listed in the two
BRDB_TXN_CORRECTION_ALLOWED_TABLES<n> configuration parameters. Note that as long as the
privileges are set up correctly (i.e. OPS$SUPPORTTOOLUSER only has insert privileges on the allowed
tables), any attempt to insert a balancing transaction on a non-allowed table will cause the previous parsing
step to fail (because the user would not have the necessary privileges). Therefore, this validation provides
security in case the privileges are not correctly set up.
¢ Check that all the columns named in the SQL exist on the table, and that all the columns on the table
are named in the SQL
« Check that the values to be inserted are provided by a SELECT ... FROM dual, (SELECT ... FROM ...
WHERE) i.e. not a VALUES
e Check that if any of the name/value pairs that are listed in the
BRDB_TXN_CORRECTION_ENFORCED_VALUES configuration parameter are present on the table, they
are set to the listed value.
fe} Balancing transaction audit files (BRDBC033), unlike the files produced by BRDBC002, are not
compressed, but are still encrypted.
Appendix 4
© 2016 Deloi
DRAFT
fe LLP Private & Confidential - Sul
t to Legal Privilege -
WBD_000339.000039
Scop
Ref
A
© 2016
DRAFT
WBON0000469
WBON0000469
e area 3 — Audit Store Controls Listing
Control Description
Audit tracks that are gathered at one data centre are replicated to the Audit server at the remote data
centre. This replication process is managed by the Audit Track Sealer. As Audit tracks are secured to the
Audit archive, they are moved to an export area awaiting transfer to the remote campus. A second file,
containing the calculated seal value for the audit track is also stored in the export area.
Audit tracks and seals are copied, using robocopy, to the equivalent import area on the remote audit server
as part of Audit server overnight schedule. On arrival, the sealer on the remote audit server recalculates the
seal value of the imported audit track and compares it with the original value in the imported seal file.
Assuming they match, the file is then written to the remote Audit archive. If the seals do not match, the
Audit track and seal file are moved to a holding area and an event is raised. Manual investigation is
necessary to investigate the cause of the discrepancy.
There will be a single instance of the ATS that concurrently accepts files for sealing/seal checking from ATG
and ATR and notifies sealed files to the ATD and into the Sealer Database for subsequent use by the Audit
Track Extractor.
The ATS shall collect files for sealing via I-ATS-4 and shall write a log of its activities to the ATD via I-ATS-
2. In sealing a file the seal shall be generated using a secure hash algorithm, the MD5 algorithm has been
selected.
Once a file has had a seal calculated the file will be written to Centera and details will be stored in the Audit
Track Seal Database via I-ATS-5.
Access to the Audit Track files for gathering shall be via Samba (for Unix systems) or NTFS (for Windows
systems). Access to the sub directory shall be limited to the application generating the Audit Track and the
Audit Track Gatherer. Audit track files should be written in write-append mode.
All users (including administrators) of the Audit Workstation and Audit Server shall log onto systems using
two factor authentication in conjunction with the HNG-X Active Directory system. Each user shall be
uniquely identifiable.
The remote directories from which the Audit Server gathers Audit Tracks will be configured so that only the
Audit Server (or an administrator who has been explicitly given permission) is able to delete files in the
directory.
All Audit Server and Audit Workstation and Centera hardware shall be held in physically secure areas where
physical access to the systems is controlled.
There shall be separate roles for:
. Audit Server (inc. Audit Workstation) Administration
. Fujitsu Services Audit Staff
The roles shall be mutually exclusive, i.e. no one individual shall be given access rights of more than one
role.
The Fujitsu Services Audit Staff role shall not have any write, modify or delete access to the Audit Archive.
The following integrity checks will be applied to the data
. Completeness of data — contiguous message sequence numbers
. Integrity of individual messages
° For Riposte data the message CRC should be checked
° For HNG-X data the message signature will be verified
Separate Riposte and HNG-X summaries of the results of the integrity checks are generated. They should
detail:
. Summary of the message sequence runs broken down by counter Id. This should include start and
end date/times and start and end message sequence numbers. Any gaps in the message sequence runs
must be highlighted.
. Summary of messages that have failed individual message integrity checks
Any failure of the data integrity checks will not prevent subsequent execution of the query. The audit
workstation user will be warned of the failure via the server process status notification mechanism.
Deloitte LLP Private & Confidential - Sul
t to Legal Privilege -
WBD_000339.000040
Ref
A
WBON0000469
WBON0000469
Control Description
As Audit tracks are retrieved from the archive, they are seal checked (by re-application of the MD5 message
digest function) to ensure that the source data has not been tampered with while it was stored in the
archive.
Only authorised users may access the Audit workstation applications. Authorised users are required to log
on to the workstation using two factor authentication and the HNG-X Identity Management system. An
Active Directory group named AUDIT_USER will be created with the rights required to utilise the workstation
applications. Authorised users will be added to this group.
All retrievals of audit data are performed using the Audit Extractor Client, and all such user actions are
themselves audited. It is not possible for users to access the archive by any other means.
Audit workstations and Atalla NSPs are located in secure areas. Only authorised users are given physical
access to these areas.
All auditable messages logged during a calendar day will be made available to the audit system in
uncompressed form as a part of Branch Database batch overnight processing.
The message journal is implemented in the form of a single Oracle table named
BRDB_RX_MESSAGE_JOURNAL. Uniqueness is controlled at the level of a Branch counter using a dense
sequence known as the Journal-Sequence-Number
ppendix 4a
Scope area 3 — Audit Store Controls Listing (broader population)
Ref I Control Description
A__I The following operating system level events on the Audit Server will be audited via the System Management
event monitoring facilities:
« Log on/Log off (including unsuccessful log on attempts)
e File Creation, Deletion and Modification (on selected files)
* Modifications to system configuration (inc software configuration and account details)
« System start up and shut down
e Recovery actions
« Exception conditions
« Change of user rights
B The Audit Server Administrator role shall have full access to manage all of the Audit Server and Audit
Workstation file stores and shall be granted the necessary Windows privileges.
Cc POL staff will not be given direct access to the Audit Workstation to safeguard other parts of the HNG-X
system. Instead nominated Fujitsu Services personnel will supply audit information as requested by Post
Office.
D User Log/On events are included in the Windows event log. Users are allocated to a specific role which
enables them to access the Audit databases.
E Baskets are stored for a defined period of time. The configuration of this parameter and the audit trail
around changes to it need to inspected in order to provide assurance over the maintenance time period for
audit purposes.
F POL controls around processing of received data from Fujitsu following a successful ARQ including
validation checks.
DRAFT 40
WBD_000339.000041
Appendix 5
Change Control — list of controls and their change dates.
Appendix 6
Case Data Analytics Overview
WBON0000469
WBON0000469
The below analytical procedures were performed on ‘Case Data’. 'Case data’ refers to transactional data provided
by POL, which had been extracted by Fujitsu from the audit store, and relates specifically to the branches involved
in the ‘allegations’. The data extracted is in 1 month periods relating specifically to the period of the allegations for
each specific branch.
Scope Relevant Analytics
I Area QC's Advice _ Proposal _ _ Procedures Analytic
1 POL consider instructing a suitably POL will instruct Deloitte I Reviewcase data 1, 2, 3, 4, 4a,
qualified party to carry out an analysis of I to determine whether for transactions 5, 6, 6a, 7
the such an analysis/review indicating items of
relevant transaction logs for branches is risk from a system
within the Scheme to confirm, feasible, and if itis, to functionality
insofar as possible, whether any bugs in I provide an indication of perspective
the Horizon system are revealed the cost, time and (e.g. recovery
by the dataset which caused process that would be transactions are
discrepancies in the accounting position I incurred. present in the
for any of those branches. case data).
Tab Index I Description
Analytic 1 I ently gaps in audit log sequencing
Analytic 2 Identify gaps in transaction times during working hours
Identify two user logon events in sequence without the expected logoff event in between; an
Analytic 3 indicator of a connectivity issue
Analytic 4 Identify recovery transactions
Analytic 4a Identify recovery transactions that indicate a connectivity issue
Analytic 5 Count of zero valued transactions summarised by product
Identify branches which are out of balance based on transactional data available (should not be
Analytic 6 possible based on inherent system controls).
Analytic 6 Group Identify branches which are out of balance based on transactional data available (should not be
and Session id possible based on inherent system controls).
© 2016 Deloitte LLP Private & Confidential - Subject to Legal Privilege -
DRAFT Bal
WBD_000339.000042
I Tab index
Description
WBON0000469
WBON0000469
I Analytic 7
Appendix 6a
Case Data Summary Findings
Identify transactions posted by non-branch users without subsequent branch acknowledgement.
POL investigators have been handed this information for further investigation. In short, whilst various
characteristics were noted that could be indicative of risk within the system, further manual investigation will be
required by POL’s investigators to conclude. This has been discussed with POL management during the course of
our work.
Procedure
Comments
Summary
Analytic 1: Identify gaps in
audit log sequencing
Analytic 2: Identify gaps in
transaction times during
working hours
Analytic 3 : Identify two user
logon events in sequence
without the expected logoff
event in between, an indicator
of a connectivity issue
In order to identify gaps in audit log
sequencing, the transactions data was
sorted into ascending order on session
id and txn id, and any gaps in the
sequence at both the session and txn
level were identified.
In order to identify gaps in transaction
times during working hours, the
transaction data was ordered by
branch, date and time. Gaps that were
significantly higher than the average
gaps in transaction times were
identified, only transactions with the
same date were compared.
Transactions with a stock unit of ATM,
LOT, OOH or BUR were excluded.
In order to identify two user logon
events in sequence without the
expected logoff event in between, an
indicator of a connectivity issue the
events data was ordered by date and
time and logon events (event code 12
or “EPOSSTransaction.Ti of Logon
Completed”) not followed directly by a
log off event (event code 13, 27 and
102 or “EPOSSTransaction.Ti of Logoff
Completed”) were identified.
There were 212,372 (1.60%) gaps in audit
log sequencing from a total of 13,307,999
transactions
There were 46,528 (0.35%) gaps in
transaction times that were more than 20
times higher than the average transaction
gap of all stores with the same number of
positions from a total of 13,307,999
transactions
There were a total of 1,064 (0.93%) logon
events in sequence without the expected
logoff between; from a total of 114,491 log
on/off events.
Analytic 4: Identify recovery
transactions
In order to identify recovery
transactions the eventDetailMsg column
of the Events data was searched for
words like ‘successfully recovered’ but
not like 'No recovery required."
There were 30 (0.00057%) recovery
transactions identified from a total of
5,289,369 transactions in the events data
© 2016 Deloitte LLP Private & Confidential - Subject to Legal Privilege -
DRAFT
WBD_000339.000043
Procedure
Comments
WBON0000469
WBON0000469
Summary
Analytic 4a: Identify recovery
transactions that indicate a
connectivity issue
In order to identify connectivity issues of
none recovery transactions the
eventDetailMsg column of the Events
data was searched for words like ‘could
not recover’ and 'No recovery required.’
There were 258 ‘no recovery’ transactions
that indicate a connectivity issue from a
total of 5,289,369 transactions in the
events data
Analytic 5: Identify zero valued
transactions
In order to Identify zero valued
transactions, all transactions with a sale
value of 0, a quantity not equal to zero
and a mode of either 1 or SC for ‘Serve
Customer’ were identified and a
summary per item is produced.
There was a total 1,314,761 (9.88%) zero
valued transactions with a quantity not
equal to zero from a total of 13,307,999.
These transactions were against a total of
814 products
Analytic 6: Identify branches
which are out of balance based
on transactional data available
(should not be possible based
on inherent system controls).
In order to identify branches which were
out of balance based on transactional
data available (which should not be
possible based on inherent system
controls), the transactions data was
summarised by branch (Group) and
session id and those branches that do
not sum to zero, 1 or -1 (due to
rounding) were identified, and are
ordered by balance descending.
There were 81 (0.0026%) session id's
from a total of 3,074,830 in 20 (16.94%)
distinct branches from 118 in total which
were out of balance based on
transactional data
Analytic 7: Identify transactions
posted by non-branch users
without subsequent branch
acknowledgement.
DRAFT
In order to identify transactions posted
by non-branch users without
subsequent branch acknowledgement,
any users whose id did not take the
usual format (6 digits - 15* letter of
forename followed by 1*' and 2” letters
of surname and numeric 001) were
identified. A user id of *PS98 are
paystation transactions and were
ignored here, a user id beginning with a
* are identified as global users
There were 17 (3.03%) users (*DSI02,
*JBAO3, *TAKO1, *PJO07, *BMA01,
*JCA01, *RCRO1, *DCU02, *JHOOS,
*RLY01, *DWAO1, *MWEO1, *STU03,
*GDRO1, *NST01, *PJO02 and *“GMU01)
from a total of 561 users classified as non-
branch users who posted transactions
WBD_000339.000044
WBON0000469
WBON0000469
© 2016 Deloitte LLP Private & Confidential - Subject to Legal Privilege -
DRAFT “4
WBD_000339.000045
WBON0000469
WBON0000469
Statement of Responsibility
We take responsibility for this report which is prepared on the basis of the limitations set out below.
The matters raised in this report are only those which came to our attention during the course of our work and are
not necessarily a comprehensive statement of all the weaknesses that exist or all improvements that might be made.
Recommendations for improvements should be assessed by you for their full impact before they are implemented.
Deloitte LLP
London
October 2016
Other than as stated below, this document is confidential and prepared solely for your information and that of other
beneficiaries of our advice listed in our engagement letter. Therefore you should not, refer to or use our name or this
document for any other purpose, disclose them or refer to them in any prospectus or other document, or make them
available or communicate them to any other party. If this document contains details of an arrangement that could
result in a tax or National Insurance saving, no such conditions of confidentiality apply to the details of that
arrangement (for example, for the purpose of discussion with tax authorities). In any event, no other party is entitled
to rely on our document for any purpose whatsoever and thus we accept no liability to any other party who is shown
or gains access to this document.
© 2016 Deloitte LLP. All rights reserved.
Deloitte LLP is a limited liability partnership registered in England and Wales with registered number OC303675
and its registered office at 2 New Street Square, London EC4A 3BZ, United Kingdom.
Deloitte LLP is the United Kingdom member firm of Deloitte Touche Tohmatsu Limited (“DTTL”), a UK private
company limited by guarantee, whose member firms are legally separate and independent entities. Please see
www.deloitte.co.uk/about for a detailed description of the legal structure of DTTL and its member firms.
Member of Deloitte Touche Tohmatsu Limited
6 Deloitte LLP Private & Confidential - Subject to Legal Privilege -
DRAFT 45
WBD_000339.000046