POL00029824
POL00029824
Horizon Spot Review 5 — Response
Bracknell & Centrally Input Transactions
Summary
An assertion has been made by Mr Michael Rudkin that during a visit to the Fujitsu Bracknell site on
Tuesday 19" August 2008, he observed an individual based in the basement of the building who
demonstrated the ability to access ‘live’ branch data and directly adjust transactions on the Horizon
system.
Given the amount of time that has passed, neither POL nor Fujitsu have any record of Mr Rudkin
attending the Bracknell site.
It has however been determined that the basement of Fujistu's building contained a Horizon test
environment. This environment was not physically connected to the live Horizon environment. It
was therefore impossible for anyone in this room to have adjusted any live transaction records
though they may have shown Mr Rudkin some form of adjustment to the test environment.
Spot review scope
This Spot Review covers 4 key areas:
1. What Horizon functions and access were available from the basement room of Fujistu's
Bracknell Site visited by Mr Rudkin?
2. What happened during Mr Rudkin's visit to the Bracknell site?
3. What access and authorisations were available from the test environment at the Bracknell
site? This section addresses the 7 specific questions put forward by Second Sight in the
email from lan Henderson to Steve Allchorn of 7 June 2013.
4. What access was historically available to live Horizon data?
Issue 1: What Horizon functions and access were available from the basement room of Fujistu's
Bracknell Site witnessed by Mr Rudkin?
The Bracknell Site Set-Up In 2008
In August 2008 there were 4 separate test environments set up in the basement area of the Fujitsu
building:
e BTC1 & BTC3 : These two test environments were used for functional testing of changes
being made to Horizon at that time, e.g. the introduction of MoneyGram (the Functional
Test Environment)
POL00029824
POL00029824
e V&l: The Horizon Volume test environment (the Volume Test Environment)
e@ REL: The Horizon release test environment (where deployment of software packages were
tested) (the Release Test Environment)
It is these test environment that is believed to have been witnessed by Mr Rudkin.
Along with these environments, preparation activities were underway in the basement to build a
volume and release environment for other Post Office IT systems, however, this environment would
not have been in a working state at the time of Mr Rudkin’s visit.
POL had access to theFunctional Test Environment.
Fujitsu controlled the Volume and Release Test environments. Their focus was not counter
functionality; they were designed to provide performance capability and software deployment to
the data centre platforms rather than counter.
Test environment only
The key point here is the phrase ‘test environment’. In August 2008, the live Horizon Data Centre
was dual-located in Wigan and Bootle. Access to the live site was strictly controlled and one could
not interfere with the live transaction databases from the test environments at Bracknell.
To create the test environment at Bracknell, POL/Fujitsu physically built a completely separate set of
servers that reflected the live configuration in Wigan/Bootle. These servers were hosted in the
basement in Bracknell, along with test counters to connect to them. Access to the test environments
then (and which remains the case now) was controlled via secure rooms and user logon
authentication.
Critically, there was no physical connection between the live and test environments. The test
environments at Bracknell could not access nor manipulate any data in the live environment.
However, as a test environment, there would have been terminals where interrogation of the test
copies of the live databases would have been possible. To a lay person, this may look like activity in
the live environment. But, to be clear, this would have been interrogation of the test databases only,
as there was complete physical and technological separation between the test and live systems.
Issue 2: What happened during Mr Rudkin's visit to the Bracknell site?
Post Office has made a concerted attempt to corroborate Mr Rudkin’s stated visit and purpose
through existing records held for that date and POL projects being managed during that time in
2008. The intention here has been to try to verify Mr Rudkin’s statement through identification of
the individual(s) who came into contact with him during his visit.
It is worth noting that Mr Rudkin states that he was unaccompanied when visiting Bracknell and
cannot remember the name of the individual who greeted him on site and escorted him through to
the basement (test environment) area during his visit.
POL00029824
POL00029824
We have additionally attempted to establish the Bracknell visitor logs for the date stated in the Spot
Review to verify Mr Rudkin’s attendance and his contact on the day, however Fujitsu have confirmed
that these records are not retained for as far back as 2008.
Fujitsu have additionally made the effort go through all email, documents and archived information
to hand but do not have any information for Tuesday 19" August 2008 that would suggest they had
visitors to the site.
Mr Rudkin states that the intention for his visit to the Bracknell site on that day was for ‘bureau de
change automation’. We have looked at the Horizon test plan for 2008 and the only subject related
project that was scheduled for transaction testing at that time was the Bureau Pre-Order
Automation project. Minutes from meetings for June and July 2008 for that project mention an
action to set up a Bracknell site visit for ‘FED’ to view the test transactions on the 19 August 2008.
Logic would suggest that the intention to hold this test session for the project on that day correlates
with Mr Rudkin’s stated visit. However, a number of individuals listed as being involved in the
project — including the Bureau product manager and assigned business analyst - have no recollection
of either the formal visit having taken place or recognition of Mr Rudkin as an invited individual to
the Bracknell site. One of the individuals spoken to was, at the material time, in post as the test
manager assigned to the project. Having searched his Lotus Notes time records, he has confirmed
that he was working in Bracknell on Tuesday 19 August 2008, however he has no recollection of Mr
Rudkin or a related visit to the test environment on that day.
Further investigation into the Lotus Notes records of the test team in Bracknell indicates that there
were just three POL test managers present on site in Bracknell on the 19 August 2008. None of them
have any calendar records relating to a visit by Mr Rudkin. The account provided by Mr Rudkin
suggests that he came across at least five individuals on his journey from the first floor in Bracknell
into the basement area.
Mr Rudkin’s account of the journey he took with his building contact, from reception to upstairs and
then down through security doors and down a stairwell to gain access to the basement, aligns with
the structural layout in the building.
Based on the above, neither POL nor Fujitsu appear to have any record of Mr Rudkin attending the
Bracknell site however, that is not to say that his visit did not happen.
Alleged Comments
Mr Rudkin mentions that he heard the basement test environment being referred to as the ‘covert
operations’ room. There is no evidence to suggest that this was ever a term used to describe the
basement area and none of the test team questioned have heard of the phrase even in a joking
context.
Mr Rudkin additionally mentions that whilst in the basement area he saw access to the live Horizon
system. This would not have been the case as the test environment was an independently built
environment from the live production system. It is possible that Mr Rudkin did not fully understand
the functions of the test environment that he visited and made certain assumptions regarding what
he saw and heard.
POL00029824
POL00029824
Issue 3: What access and authorisations were available from the test environment at the
Bracknell site?
This section addresses the 7 specific questions put forward by Second Sight in the email from lan
Henderson to Steve Allchorn of 7 June 2013.
Question 1:
What capabilities did the POL Bracknell team have? (As far as TC or Rem Out type transactions or
Journal adjustments are concerned).
Response:
The POL Bracknell Team had no access to the live system therefore could not conduct any of these
transactions.
Question 2:
What were the PHYSICAL or LOGICAL controls over their use of the systems available to them?
Response:
There was no Physical or Logical connection to the live system from the areas in Bracknell being
discussed/ investigated. Detailed documentation has been supplied of the testing processes and
procedures recently audited and the design documents to support this position.
Question 3:
What audit trail is available to show the extent that they posted TC or Rem Out type transactions, or
Journal adjustments?
Response:
When any transactions are posted to the database they are contained in the audit trail. As the
Horizon test systems were available to the test teams in that period, the test area and the test data
is often refreshed and changed it would not be possible to identify any transactions from this period
in the test system. Specifically we do not keep audits of test systems, only the Live system. As stated
in response to question 1, the teams in the area of Bracknell concerned would have no access to the
live system.
Question 4:
Can Post Office reply on the COMPLETENESS of the audit trail? ie. does it record all transactions or
just transactions meeting certain criteria? Is it protected from user manipulation?
POL00029824
POL00029824
Response:
The detailed answer to this is included in two papers Horizon Data Integrity and Horizon Online Data
Integrity for Post Office Ltd which have been presented as evidence in a number of previous court
cases.
Question 5:
What USER ID was used if TC type transactions or journal adjustments were posted?
Response:
On the old Horizon System (which was Live in 2008) Data introduced to the system in the Data
Centre would not be marked with any user ID. To be clear, however, it was impossible to introduce
data into the live system from the test environment.
Question 6:
Could the POL Bracknell team log on with either super user or SMPR credentials?
Response:
Not into the live system for reasons already outlined.
Question 7:
How would TC, Rem Out or Journal Adjustment type transactions executed by the POL Bracknell
team be seen by SPMR of Branches affected by those actions?
Response:
The PO Bracknell team did not have access to the live system.
Issue 4: What access was historically available to live Horizon data?
As referenced in the Spot Review, the Horizon Operating Manual from 2006 notes that the
introduction of a new system meant that POL could "no longer adjust client accounts on site". POL
has been asked to clarify whether this meant that POL could access and change live Horizon data.
In parallel to Horizon, POL operates a finance IT system. This finance system manages the
relationships between POL and its product suppliers. These relationships are the "client accounts"
referred to in the Operating Manual.
In 2006, POL upgraded its finance system to a new SAP finance system. Before this upgrade,
transaction records were sent from Horizon to the old finance system. When certain types of error
were made in recording transactions in branch, POL's Product and Branch Accounting (P&BA) team
based in Chesterfield could make manual adjustments to the finance system records so that the
client accounts would be corrected. This is what the Operations Manual meant by an adjustment
being "on site" — the site being the Chestefield site of the P&BA team.
POL00029824
POL00029824
For clarity, the manual adjustments to the finance system did not change the Horizon records and
therefore did not change the branch's local accounting position.
Post 2006 and the introduction of the SAP system, POL changed this process. The errors that would
historically have been corrected by manual adjustment are now corrected by way of a transaction
correction being issued to the branch. On the SPMR accepting the transaction correction, the
Horizon data is updated and this flows through to the SAP finance system.
It was this change of process that led to the above entry in the Operations Manual. This change, and
the ability to access the old financial system, is also entirely unrelated to the test environment at
Bracknell.