Letter to the City of London Police dated 23 August 2024
APPENDIX 1
POTENTIALLY RELEVANT RECORDS GENERATED BY THE HORIZON SYSTEM AND SUPPORTING SERVICES
FUJ00243304
FUJ00243304
Page 1 of 22
Data Category
Description of Data
Records
Dimensions: The Post
Office Account Team
document library
Overview
The Post Office Account Team uses a document repository to maintain a controlled library of documents relating to the
Horizon system and its operation. This library, known as Dimensions, includes contractual, architectural and process
documents, many of which have been agreed between FSL and Post Office. The library uses document status, review
and version control processes, some of which is facilitated by a Dimensions mailbox.
Records
There are various documents stored in Dimensions which FSL legal considers an IT expert may need to consider when
investigating the Horizon system’s operation, including but not limited to:
Architecture and design documents
Process and procedure documents for managing problems and incidents
Service Descriptions
Service Management documents
Test documents
Documents relating to Releases
Interface specifications
Reports and audits
eee ee eee
Some of the documents retained in the Dimensions system are referred to in this table in footnotes. Such documents
normally contain a formal reference code, together with a version number and a date of issue or approval and this
information is reflected in the footnotes where available. Documents normally contain version histories, reviewer and
contributor names, glossaries and list associated documents.
To give a sense of the scale of this repository, in relation to Horizon Online, there are currently approximately 3,500
live documents. There are also over 2,000 “withdrawn” documents. Superseded and prior versions of Dimensions
Documents stored in Dimensions
are controlled and are generally
available for the lifetime of
Horizon
Fujitsu
Fujitsu Services Limited, Registered in England no 96056, Registered Office Fujitsu, Lovelace Road, Bracknell, England, RG12 8SN
Letter to the City of London Police dated 23 August 2024
FUJ00243304
FUJ00243304
Page 2 of 22
Data Category
Description of Data
Records
documents are also retained in Dimensions and there are a total of over 37,000 documents stored in the Horizon Online
repository. A separate Dimensions repository contains documentation relating to Legacy Horizon, created mostly
between 1996 and 2010.
The Peak system
Overview
In summary, Peak is an incident and release management system and work allocation tool. It is used by many teams in
FSL to support the Horizon system, including 3” Line support, 4" Line support, release management, integration and
test teams. It is a live database used operationally to manage, amongst other things, identified incidents, including
reconciliation errors and software defects. The Peak system has been in place since 2003 (prior to which the PinICL
system was used).
Records
There are currently approximately 260,000 Peaks in the live Peak database. Peaks have creation dates and should note
when they were updated. Peaks may contain various searchable attributes such as the unique Post Office branch code
known as a “FAD Code”.
The Peak Database is generally
available for the lifetime of
Horizon, although it is understood
that certain attachments to Peaks
may no longer be available.
Knowledge Base
Articles (“KBs”)
(formerly Known
Error Logs (“KELs”))
Overview
AKBis raised when a member of FSL’s architecture, development, test or support team identifies information that may
be beneficial to share with the FSL support team.
Governance
Aweekly KB review forum is held and is managed by the Service Management team (Incident Management).
Records
FSL can produce an extract from
the KB database.
Please note, for various reasons,
not all KBs which have been
created during the lifetime of
Horizon are retained in the KB
database’.
1 The availability of KBs is discussed in Mr Justice Fraser’s judgment in the Bates v Post Office group litigation (at which time they were referred to as KELs) and it has also been the subject
of evidence in the Inquiry.
Fujitsu
Fujitsu Services Limited, Registered in England no 96056, Registered Office Fujitsu, Lovelace Road, Bracknell, England, RG12 8SN
FUJ00243304
FUJ00243304
Page 3 of 22
Letter to the City of London Police dated 23 August 2024
Data Category Description of Data Records
KBs are stored in a database, so that when an issue is identified, support staff can crosscheck the database to see if a
similar issue has been seen before and how such issues have previously been handled. The KB database can be accessed
via the SSC Website (see row 18 below). KBs record information on many types of issues related to the Horizon system’s
operation. This should include information regarding an event, failure, fault, user error or a potential bug.
Minutes and actions should be produced and sent out after the weekly KB review forum. An agenda should also be sent
out prior to the meeting by the Incident Management team.
Remote Access I Overview RA Report of February 2021
Documents
‘As noted in the cover letter to this Appendix, remote access powers enabling FSL to access and to amend data affecting I PAM RAM Assurance Report
branch accounts have existed and continue to exist within the Horizon system. The use of such powers would need to
be examined to ascertain if they could potentially have caused or affected the accounting discrepancies or transactions I Horizon Data Changes Process
under investigation. FSL has produced relatively recent reports to POL to explain FSL’s Remote Access capabilities and I Work Instruction
which contain more details of these remote access capabilities. In particular:
TfSNow (e.g. TfSNow incidents,
the RA Report? which explains how FSL at that time implemented and managed Remote Access to Horizon. CHGs and CTasks) (see row 11
* the PAM RAM Assurance Report? which was a response to various question posed to FSL by Post Office regarding I below)
remote access and privileged access.
Peak (see row 3 above)
Horizon Data Changes
The SYS.AUDS table
FSL support users may need to make corrections to various databases, including the BRDB (see row 12 below).
Corrections to the BRDB may include deleting, inserting and updating data in the BRDB. The process currently to be I Microsoft Teams and Outlook
followed by FSL support staff when making certain data changes to the live Horizon system is contained in the Horizon I messages
Data Changes Process Work Instruction.*
Counter Access Reports (circa mid
In order to perform certain remote access activities, FSL staff may need to be granted a type of privileged access role I 2022 onwards only)
known as APPSUP (or Application Support). The Horizon Data Changes Process Work Instruction provides information
2 COM/MGT/REP/4165, version 1.0, 12 February 2021
3 COM/MGT/REP/4818, 20 April 2023
4 SVM/SDM/PRO/4293, version 3.0, 3 May 2024
Fujitsu
Fujitsu Services Limited, Registered in England no 96056, Registered Office Fujitsu, Lovelace Road, Bracknell, England, RG12 8SN
Letter to the City of London Police dated 23 August 2024
FUJ00243304
FUJ00243304
Page 4 of 22
Data Category
Description of Data
Records
regarding the processes, procedures and audit of the granting and revoking of the APPSUP role.° Certain actions
performed using the APPSUP role are logged to a table in the BRDB (specifically the “SYS.AUD$” table).°
Records
Records capturing the granting and revoking of the APPSUP role and corrections undertaken by FSL staff members using
it may include:
* TFS Now (e.g. TFS Now incidents, CHGs and CTasks’);
e = Peak;
* The “SYS.AUD$” table in the BRDB. Note the actions written to the “SYS.AUDS” table are subsequently written to
the Audit Archive.
Microsoft Teams and Outlook messages may also be used in the course of following the procedures set out in the
Horizon Data Changes Process Work Instruction to arrange for the APPSUP role to be granted or revoked.
Counter Access
FSL also provides a regular weekly Counter Access Report to Post Office. These are relatively recent reports (circa mid
2022 onwards) which provide details of commands executed against a counter by FSL support staff. These reports relate
to read-only access to live counters. Whilst these reports will not demonstrate any corrections made by FSL support
staff, they may help identify Peaks where FSL staff are requested to investigate a branch.
Older records
As noted below, some change records (such as historical OCRs/OCPs*) contain information and authorisation regarding
FSL staff members using remote access powers which could affect the financial position of a Post Office branch.
OCRs/OCPs/MSCs (see row 11
below)
5 See for example section 5.3.
6 See for example page 15 (ref 2.8) of the PAM RAM Assurance Report.
7 See Row 11 regarding TfSNow.
® It is understood that the OCRs and OCPs ceased to be used in approximately 2010
Fujitsu
Fujitsu Services Limited, Registered in England no 96056, Registered Office Fujitsu, Lovelace Road, Bracknell, England, RG12 8SN
Letter to the City of London Police dated 23 August 2024
Page 5 of 22
FUJ00243304
FUJ00243304
Data Category
Description of Data
Records
Management of Bugs,
Errors and Defects
(“BEDs”)
Overview
‘As noted in the covering letter to this Appendix, Horizon has had, and continues to have BEDs. Such BEDs either have
affected or have the potential to affect financial accounts. There are a number of processes and associated records
relating to identifying, tracking and resolving BEDs which are retained by the Post Office Account Team. Below is a non-
exhaustive list of such records and processes. Further, in February 2021 FSL prepared a report to Post Office (titled BED
Current Process Report) explaining the process for the management of bugs, errors and defects (as it existed at that
time).° The POA Live Defect Management Procedures document” seeks to provide a consolidated view of the Post
Office Account Team’s end-to-end process for managing live defects.
Governance and records
The Horizon Defects Review (“HDR”) is a joint forum held weekly (from around June 2021) between FSL and Post Office
with the purpose of governing the defect management process." The weekly meeting is intended to track the status
of open defects from their identification to resolution’.
Typically, in advance of the meeting FSL sends to Post Office a report known as the “HDR Defects Update Report”
(discussed further below in this row). This report is then considered at the meeting. Following the meeting, Post Office
typically sends FSL two documents: i) an Excel “Defect Tracker” document which includes Post Office’s view of defects
contained in the HDR Defects Update Report and ii) an Excel document (called “Horizon Defects Review Forum —
Meetings Notes”) which contains, amongst other things, meeting attendees and agreed actions arising from the
meeting. Emails between FSL and POL relating to the HDR meeting also exist, some of which are centrally stored in a
SharePoint site.
There have been various predecessor meetings to the HDR, such as the Horizon Known Error Review Forum (the
“HKERF”) which was established in approximately September 2019. It is understood that this forum has over time been
referred to by different names, such as the KEL Review Forum, Horizon Governance Working Group and the Horizon
Post Office’s Defect Tracker
Post Office’s Horizon Defects
Review Forum — Meetings Notes
HKERF related documents are
available from September 2019 to
March 2021
HDR meeting related documents,
available from June 2021 to
present
HDR Defects Update Report
(weekly). available from June
2021 to present.
POA HNG-X All Live Defects
Report (fortnightly) is available
from August 2022 to present.
Internal FSL Live Defects Report
(monthly), available from January
2022 to present.
9 COM/MGT/REP/4184, version 1.0, dated 26 February 2021
2 SVM/SDM/PRO/4313, version 1.0, 17 October 2023
14 Note the HDR covers defects in the live Horizon system; defects in test systems are understood to be covered by a separate process.
22 Horizon Defects Review Terms of Reference, version 3.2 dated 3 January 2023. Note this is a Post Office controlled document and so does not have an FSL/Dimensions reference
number.
Fujitsu
Fujitsu Services Limited, Registered in England no 96056, Registered Office Fujitsu, Lovelace Road, Bracknell, England, RG12 8SN
Letter to the City of London Police dated 23 August 2024
FUJ00243304
FUJ00243304
Page 6 of 22
Data Category
Description of Data
Records
Known Errors Joint Review Working Group. Similar records to those used by the HDR are assumed to exist for these
predecessor meetings but we have not verified this at the time of writing save to confirm that, for the HKERF, it is
understood that Post Office produced a weekly email summarising which Peaks and KBs were being focussed on by
that forum. The transition from HKERF to the new format HDR meeting took place between around March to June
2021.
Reports
FSL produces various reports recording issues identified in Peaks"? which have been tagged as HDR Defects, Live Defects
or both. Definitions of these are set out in section 1.3 of POA Live Defect Management Procedures’ and are
summarised below:
© A “Live Defect”*® is any issue which (i) is present on the live system, (ii) falls within FSL’s scope of obligations
and (iii) is, or appears to be, inconsistent with the agreed design or service specification. Such defects are
therefore likely to need fixing.
¢ A“HDR Defect” is a sub-set of Live Defects and is an issue which has the potential to affect (i) branch financial
outcomes, (ii) the way an end user of Horizon in branch (such as a sub-postmaster) is required use the system
or (iii) the experience of a Post Office customer or client. HDR Defects are classed as either “Financial”
impacting (which Post Office refers to as “Impact” and which FSL refers to as “Financial”) or “Experience”
impacting.
When FSL raises a HDR Defect, Post Office should assign its own corresponding problem reference number in its Service
Now tooling for tracking and management purposes. FSL uses a process to extract the latest data from the Peak
database in order to compile the three reports summarised below:
© HDR Defects Update Report: A weekly report is sent to Post Office in Excel format (with a date/time stamp)
containing all Peaks tagged as HDR Defects. As noted above, this report is considered at the weekly HDR meeting.
It contains information including a description of each HDR Defect, root cause analysis, proposed steps to resolve
the issue and any known branch impact as identified by FSL at that time. The Excel also includes the corresponding
3 See row 2 for more information regarding Peaks.
+8 SVM/SDM/PRO/4313, version 1.0, 17 October 2023
* Live Defects have also been referred to as “Live Affecting Defects” operationally by certain teams in the Post Office Account.
Fujitsu
Fujitsu Services Limited, Registered in England no 96056, Registered Office Fujitsu, Lovelace Road, Bracknell, England, RG12 8SN
Letter to the City of London Police dated 23 August 2024
FUJ00243304
FUJ00243304
Page 7 of 22
Data Category
Description of Data
Records
Post Office problem reference number in Service Now (once provided to FSL) and an embedded extract of certain
fields in Peak.
* POAHNG-x All Live Defects Report: FSL generally sends to Post Office on a fortnightly basis a report on alll Live
Defects (which will include all HDR Defects given they are a subset of All Live Defects). The report is in PDF format
with a date/time stamp. It contains a summary front page followed by a table of all Live Defects.
Live Defect Management Update (internal FSL): FSL generally produces a monthly report internally in PowerPoint
format which is sent to certain FSL Post Office Account Team managers. It contains totals of All Live Defects
(including HDR Defects), the releases that fixes of the defects are targeted at, annual trends, information regarding
the numbers of open defects for previous years and summaries of particular defects from the Peak system.
Release Management
Overview
The Horizon system is updated by the implementation of new releases. In broad terms, releases are a documented
collection of software and/or data. They can also include system infrastructure items and reference data. Software
releases result from a number of different requirements such as a change to services (e.g. adding new functionality),
supporting new procedures, improving performance and, importantly, fixing BEDs.
The following Dimensions documents contain further information about the release management process and its
operation:
e — Fujitsu POA Release Policy?
© — Post Office Account Release Management Strategy?”
_HNGxDBM Release Management Process™®
There are different types of releases, including:
Fujitsu POA Release Policy
Post Office Account
Management Strategy
Release
HNGxDBM Release Management
Process
POA Live Defect Management
Procedures
Release Notes
Release Planning Meeting/
Release Management Forum
16 PA/STR/003, version 8.0, 24 July 2024 (latest draft currently being agreed with Post Office).
17 SVM/SDM/PRO/1520, version 1.1, 27 July 2020. Note this document is likely to be withdrawn shortly subject to certain of its content being incorporated into “HNGxDBM Release
Management Process.
18 PGM/PAS/PRO/0010, version 1.0, 20 May 2024
Fujitsu
Fujitsu Services Limited, Registered in England no 96056, Registered Office Fujitsu, Lovelace Road, Bracknell, England, RG12 8SN
FUJ00243304
FUJ00243304
Page 8 of 22
Letter to the City of London Police dated 23 August 2024
Data Category Description of Data Records
i) Programme Releases, which can include changes of business functionality requested by Post Office, infrastructure I Demand Planning Forum
changes, fault fixes and service level improvements. These were previously known as Major Releases. Generally, these
should be released a minimum of 3 months apart. Forward Schedule of Change (or
FSC) meeting
ii) Counter Releases, which are deployed to counters in branches. These can contain fixes, service improvements and
feature changes. Since counter hardware and some counter software is currently managed for Post Office by a third- I Release Schedule
party supplier (currently DXC), Counter Releases generally involve a collaboration between FSL, Post Office and DXC.
The Quality Filtering Process
iii) Maintenance Releases, which may contain one or a number of smaller changes or fixes which may be implemented I meeting
in between Programme Releases (for example fault fixes and security patches).
Release related Peaks
iv) Emergency Changes, which provide a more urgent route to delivering changes and fixes, for example, to recover a
service or to ensure that a service remains operational. RAM/RAB
In addition, there are certain minor operational changes implemented to Horizon which fall outside of the four release I BIF/PTF
types described above. These are logged in TfSNow as TFSNow Changes (see row 11 below) and which are approved
under that toolset. HDR (which now covers CBIF-type
matters)
Governance and records
Agree Release Backlog
Various governance measures are used to manage the delivery of the above-mentioned releases and FSL holds I Maintenance Forum
documentation in relation to these measures. A key document is the Release Schedule document (the “Release
Schedule”), in Excel format, which the team uses to plan, manage and track releases. A new Release Schedule is I CAB/ECAB
understood to be created each calendar year.
For Programme Releases, governance meetings and documents include the Release Planning Meeting (sometimes
referred to as the Release Management Forum), the Demand Planning Forum, the Forward Schedule of Change (or FSC)
meeting, and the Quality Filtering Process meeting.
‘As noted above, since Counter Releases require collaboration between FSL, Post Office and DXC, joint meetings are
held between FSL and Post Office to manage them. Two key meetings are the Release Acceptance Meeting (“RAM”)
and the Release Authorisation Board (“RAB”). The RAM focusses on reviewing release requirements, test outputs and
Fujitsu
Fujitsu Services Limited, Registered in England no 96056, Registered Office Fujitsu, Lovelace Road, Bracknell, England, RG12 8SN
Letter to the City of London Police dated 23 August 2024
FUJ00243304
FUJ00243304
Page 9 of 22
Data Category
Description of Data
Records
related documentation. Taking into consideration outputs from the RAM, the RAB focusses on reviewing deployment
readiness for a release and service acceptance. The RAB will decide whether to authorise the deployment of the release.
In practice, the RAM and the RAB are held as a single meeting. FSL holds documentation associated with the RAM and
RAB meetings, including minutes of meetings.
For Maintenance Releases, the Business Impact Forum (“BIF”) and the Peak Targeting Forum (“PTF”) are both internal
FSL governance forums (chaired by the Release Management team) and are intended to identify and monitor Peaks
requiring software fixes to be delivered in a Maintenance Release. The Terms of Reference for the internal BIF/PTF
meeting are contained within POA Live Defect Management Procedures?® (see section 2.7 “Key Meetings”). FSL
understands minutes for these meetings were created in the past, but ceased to be taken approximately in 2021.
Currently, outputs from the BIF/PTF meeting are captured and recorded in the BIF/PTF tab of the Release Schedule.
In addition to the above, FSL and Post Office should discuss items escalated from the BIF which are determined to
require Post Office consideration at the HDR (referred to in row 5 above). These discussions previously took place in a
separate meeting known as the Customer Business Impact Forum (“CBIF”). FSL understands CBIF minutes remain
available from July 2019 to July 2021.
There is a further monthly internal FSL forum known as the Agree Release Backlog Maintenance Forum which is used
to review Peaks which the BIF has decided not to prioritise for fixing in a maintenance release. It is understood that
these Peaks may be resolved by a future release.
For Emergency Changes, it is understood by FSL legal that these can be approved by the Change Advisory Board (“CAB”)
or via the Emergency Change Advisory Board (“ECAB”). FSL holds minutes in relation to the CAB meeting. We
understand that ECAB approvals are handled via messages between CAB members rather than at a specific ECAB
meeting.
FSL holds Release Notes, raised in Peak, which detail changes, enhancements or fault fixes that are to be deployed in
the live Horizon estate.
+8 SVM/SDM/PRO/4313, version 1.0, 17 October 2023
Fujitsu
Fujitsu Services Limited, Registered in England no 96056, Registered Office Fujitsu, Lovelace Road, Bracknell, England, RG12 8SN
FUJ00243304
FUJ00243304
Page 10 of 22
Letter to the City of London Police dated 23 August 2024
Data Category Description of Data Records
The Reconciliation I Overview End to End _ Reconciliation
Service
In broad terms, the Reconciliation Service compares transaction and other types of data i) as recorded by Horizon with
ii) the corresponding data held by third parties, such as financial institutions. Its purpose is to check transactions
processed through Horizon.
The Reconciliation Service checks if the data matches, produces reports regarding the same and identifies where the
data does not match. Where the data does not match, this is known as a reconciliation error. Reconciliation errors could
be caused by Horizon system failures or faults and could affect transactions in Post Office branches.
FSL used Data Reconciliation Service version 1 (“DRSv1”) until approximately 2020, at which time it was largely replaced
by the current system in use, known as Data Reconciliation Service version 2 (“DRSv2”).2°
More detailed information regarding the Reconciliation Service is contained in the Reconciliation Service Description”.
Reports
Various reports are generated by the Reconciliation Service. A list of these reports is set out in the ‘End to End
Reconciliation Reporting’ document.”
By way of example, FSL regularly produces and sends to Post Office ‘NB102 Reports’. This is a group of reports
highlighting mismatches in banking withdrawals and deposits, payments and E-TopUp transactions.
Business Incident Management Service (“BIMS”) and associated reports
Reconciliation reporting activity is currently the responsibility of the FSL Major Account Controller (or MAC) team.
Where a reconciliation error is identified, a BIMS Peak is raised containing details of the error. Details of the
reconciliation error from the BIMS Peak are entered into an Access database (the “BIMS Access database”). BIMS
Reporting document
Reconciliation Service Description
NB102 Reports
Data Reconciliation Database
BIMS Access database
BIMS Reports stored in the BIMS.
Network Drive (from circa 2011
onwards)
Outlook reconciliation mailbox
2° Note some transactions are still routed via DRSv1.
21 SVM/SDM/SD/0015, version 8.0 dated 12 February 2024
22 SVM/SDM/SD/0020 version 4.0, 4 September 2017
Fujitsu
Fujitsu Services Limited, Registered in England no 96056, Registered Office Fujitsu, Lovelace Road, Bracknell, England, RG12 8SN
Letter to the City of London Police dated 23 August 2024
FUJ00243304
FUJ00243304
Page 11 of 22
Data Category
Description of Data
Records
Peaks are often investigated by the SSC team. Where appropriate, details of BIMS issues are entered into a template
Word document to create a “BIMS Report”. BIMS Reports are then sent to Post Office for its investigation from an
Outlook reconciliation mailbox used by FSL.”3
FSL is able to perform an extract of the BIMS Access database.
Some BIMS Reports are stored in a fileshare known as the BIMS Network Drive. The FSL MAC team has access to data
in the BIMS Network Drive dating back to approximately 2011.?*
Some BIMS Reports should also be available from the sent items of the Outlook reconciliation mailbox (referred to
above).
Audit Archive
Certain of the Reconciliation Reports are also harvested into the Audit Archive. At the time of writing, FSL’s legal team
has not yet determined what reports and other data are retained in the Audit Archive.
Service Management
Overview
FSL provides a Service Management Service to Post Office which monitors, manages and maintains the delivery of
Horizon operational services. This includes the management of problems and complaints and the measurement and
management of customer satisfaction. More detailed information regarding the Service Management Service is
contained in the Service Management Service: Service Description”.
Governance and associated records
A key meeting governing the Service Management Service is the joint FSL and Post Office monthly Service Management
Review Forum, also known as the “SMR”. The Service Management Review Forum has operated in one form or another
for anumber of years. We understand its structure, name, composition and frequency will have changed over the years.
SMR Forum meeting minutes,
agendas, Service Level reports
and other performance reports
available generally for recent
years
Similar records are assumed to
exist for Operational Review
Forums
23 PostOfficeAccount.reconciliation@fujitsu.com
24 Further, older data dating back to 2005 might also be retrievable.
25 SVM/SDM/SD/0007, version 8.0 dated 13 December 2023
Fujitsu
Fujitsu Services Limited, Registered in England no 96056, Registered Office Fujitsu, Lovelace Road, Bracknell, England, RG12 8SN
Letter to the City of London Police dated 23 August 2024
FUJ00243304
FUJ00243304
Page 12 of 22
Data Category
Description of Data
Records
Papers produced to this forum should generally include meeting minutes, agendas, Service Level reporting statistics
and other performance reports.
The Service Management Review Forum is supported by various subsidiary Operational Review Forums that are each
responsible for reviewing the performance of a specific service(s) and any related operational issues”. It is assumed
but has not been verified at the time of writing that documents available in relation to these Operational Review Forums
should include meeting minutes, actions and reports.
Beyond these governance forums, it is understood that other communications occur between FSL and Post Office
regarding the operational services. Recently, these have included weekly priority calls and a hot topics forum run by
the head of the Post Office Account Team.”” At the time of writing, we have been unable to ascertain what records
have been generated by these meetings.
Systems
Management Service,
Systems
Management Centre
and Tivoli
Systems Management Service
The Systems Management Service comprises an event management service, a systems monitoring service and a
software distribution service. Events are the indications of conditions that have operational significance, including
software, hardware or security conditions which may require investigation”®. A description of the Systems Management
Service is contained in the Systems Management Service: Service Description.” The document SYSMAN4 HNG-x System
and Estate Management: Monitoring*®® contains further information regarding service monitoring and event
management.
The Systems Management Centre
The Systems Management Centre team (“SMC”) provides support in relation to the Systems Management Service. The
SMC uses various tools and systems to monitor the Horizon system, such as the Realtime Active Dashboard, Active
Event List, Spectrum, HORIce, PAN Manager and IBM Netcool. The SMC produces various reports based on its
Systems Management Service:
Service Description
SMC Reports
Tivoli Reporter Database
26 although note such meetings/forums may not formally fall within the Service Management Service.
27 Note that the two meetings referred to here are executive level meetings and, in terms of governance, fall outside of the Service Management Service. Note further that the hot topics
forum meeting has ceased to be convened.
28 The Systems Management Service: Service Description gives an example of a low battery on a PIN Pad as an event.
23 SVM/SDM/SD/0006, version 8.0, 14 March 2024
3° ARC/SYM/ARC/0003, version 4.0, 4 January 2016
Fujitsu
Fujitsu Services Limited, Registered in England no 96056, Registered Office Fujitsu, Lovelace Road, Bracknell, England, RG12 8SN
Letter to the City of London Police dated 23 August 2024
FUJ00243304
FUJ00243304
Page 13 of 22
Data Category
Description of Data
Records
monitoring. Such reports are in Word and Excel formats and should be contained in the Post Office Account Team’s
SharePoints and emails. At the time of writing, we have been unable to ascertain what records are retained from
monitoring systems and associated tools.
Itis understood that events come through a collection layer and that certain events are filtered for the SMC’s purposes.
Events can be extracted into Excel spreadsheets by the SMC team up to 30 days.
Tivoli
Tivoli is a set of third-party system management toolsets. It has been used by the Post Office Account Team to provide
event management, system monitoring, and software distribution activities. In particular, the toolset was used to
manage the distribution of software to Post Office counters, each of which is individually identified by a unique node
that includes the FAD code and a counter number. FSL can extract various reports from the Tivoli reporter database.
10.
Reference Data
Overview
Reference data, in broad terms, is data about products and other operational elements. For example, the price of a
stamp is controlled by reference data. As the price of a stamp will change over time, we understand that a design
decision was taken not to hard code the price into the system since changes in price would require new code. Instead,
the price is contained (and updated) in a set of Reference Data that the Post Office branch counters can refer to. Issues
with reference data have caused and could cause discrepancies in Post Office branches. The Reference Data
Management Service: Service Description? and the Interface Agreement for Operational Business Change — Reference
Data®® contain further information regarding reference data.
Records
Reference data delivered from Post Office to FSL is loaded into a Reference Data Management Centre database (the
“RDMC database”). The RDMC database contains multiple versions of reference data over time (i.e. reference data for
the past, present and future).
RDMC database
Reference Data Delivery Service
database
HNG-X: RDMC Host High Level
Design
31 SVM/SDM/SD/0013, version 6.0, 23 November 2021.
32 CS/PRD/058, version 18.0, 10 October 2016
Fujitsu
Fujitsu Services Limited, Registered in England no 96056, Registered Office Fujitsu, Lovelace Road, Bracknell, England, RG12 8SN
Letter to the City of London Police dated 23 August 2024
FUJ00243304
FUJ00243304
Page 14 of 22
Data Category
Description of Data
Records
‘As noted in row 14, the counter will periodically refresh the reference data set it holds locally. There is a separate
database, derived from the RDMC database, which holds only the reference data set for the single point in time that is
today (the “Reference Data Delivery Service database”). This is the reference data set which is used by the Post Office
Branch counters when they periodically refresh their reference data.
FSL’s legal team understands from the Post Office account that not all reference data originates from Post Office. It is
understood that certain reference data originates from FSL and that such reference data is also released to the RDMC.
iL
TfS Now
Overview
TfSNow is the current service management toolset used by the Post Office Account Team to record and manage Incidents,
Problems and Changes. It has an integration to Peak for FSL’s third and fourth line support teams and also to the Post.
Office ServiceNow toolset to allow shared tickets to replicate specified updates. It has been in use since approximately
2018.3,
Within TFS Now there are records of (i) Incidents (actual or perceived faults), (ii) Problems (recurring or underlying
issues) and (iii) Changes (requirements to amend the configuration) records as summarised below.
TES Now Incidents (Inc): TfS Now is used as an incident ticketing system to raise incidents and also to bond together
incidents from different systems (such as Peak tickets and Post Office raised tickets notified to the FSL MAC team).
Note that not all information is transferred between the TfsNow and Peak systems and therefore tickets passed
between these systems should be looked at together to understand more about the incidents they relate to.
TfS Now Problems (Prb): A TfS Now Problem is usually raised where a number of Incidents appear to have a
common theme.
© TFS Now Change (CHG): Examples of Changes include deploying a Programme Release or Maintenance Release. As
noted above in row 6 certain, some operational changes are implemented using TfSNow Change. We understand
that Changes can be subdivided in the system into sub-tasks known as CTasks.
Records
These Change, Incident and Problem records can be exported into a PDF or Excel document.
2018 to present: TfS Now
(Earlier systems in use included
Hera, Apollo, OCP, OCR and the
MSC toolset)
33 Previous systems in use included Hera, Apollo, OCP, OCR and the MSC toolset.
Fujitsu
Fujitsu Services Limited, Registered in England no 96056, Registered Office Fujitsu, Lovelace Road, Bracknell, England, RG12 8SN
Letter to the City of London Police dated 23 August 2024
FUJ00243304
FUJ00243304
Page 15 of 22
Data Category
Description of Data
Records
12. The Branch Database I Overview BRDB/BRSS tables are retained for
(“BRDB”) varying periods of time. Generally
The BRDB is a key operational database in Horizon into which counter transactions and other data are recorded. The I these tables are retained for a
design of the BRDB is described in the Branch Database High Level Design document.>* short period of time (days, weeks
or months).
Records
Some will be available from the
To give a sense of scale, as noted above, the BRDB is comprised of over 2,000 tables. The BRSS* retains a copy of BRDB I Audit Archive for a much longer
table information and, typically in resolving issues, FSL will retrieve data from the BRSS rather than the BRDB as data is I period of time.
retained in the BRSS for longer periods of time.
Audit Archive
Certain information in the BRDB is sent to the Audit Archive. FSL is not currently in a position to confirm precisely which
information is sent to the Audit Archive although we note that the Branch Database High Level Design does contains
further information in this regard. The FSL Archive Server Configuration document* also indicates there are various
audit points and subpoints in the BRDB which are harvested into the audit archive.
13. IThe Branch Access I Overview Message.log (available for eight
Layer (“BAL”)
In broad terms, the BAL exchanges messages with Post Office branch counters and other parts of the Horizon system.
FSL legal understands that it acts as a distribution and load balancing service allocating activity. The BAL sits between
the counters and services that the Counter Business Application (“CBA”)°” at the Post Office branch utilises (such as the
BRDB, the CDP*8 and the GWS?%). Messages and transactions which happen at the counter are first passed to the BAL
and then potentially onwards to the BRDB.
days)
Osr.log (available for eight days)
Syslog (available for eight days)
34 DES/APP/HLD/0020, version 26, 3 May 2024
35 A database used for support purposes.
3© DEV/INF/ION/0001, version 21, 21 September 2023, (see section 5).
37 See row 14 below regarding the CBA.
38 See row 14 below regarding the CDP.
39 See row 15 below regarding the GWS.
Fujitsu
Fujitsu Services Limited, Registered in England no 96056, Registered Office Fujitsu, Lovelace Road, Bracknell, England, RG12 8SN
Letter to the City of London Police dated 23 August 2024
FUJ00243304
FUJ00243304
Page 16 of 22
Data Category
Description of Data
Records
Records
The BAL platform consists of 10 individual BALs. Two applications, known as Online Service Routers (OSRs), run on each
of the 10 BALs (i.e. there are 20 OSRs in total). The OSRs hold data including:
© The Message.log - this contains request messages, including transactions, sent to and received from the BAL. This
log may be used to investigate transactions. It could be compared with the message log generated at a counter.
e The Osr.log — the OSR application log records data regarding the functioning of it as an application. If, for example,
the OSR has an error sending a transaction to an endpoint, it will be recorded in the Osr.log whereas the counter
may only see a timeout.
e The Syslog — this is the BAL event log. It is analogous to the Windows event log retained on PCs in Post Office
branches.
We have not yet ascertained the extent to which such logs are sent to the Audit Archive and which may therefore be
available for extraction.
14.
Post Office Counters
in branches
Overview
In addition to Horizon data stored in datacentres operated by FSL, certain Horizon data is stored locally within counters
in Post Office branches. The counter generates various logs and also holds a copy of the reference data currently in
use (in XML format).
The CBA is a Java application supplied by FSL which runs on equipment procured by the Post Office from other third-
party suppliers. It provides an interface for the counter operator to transact a range of products, including sending or
receiving information to or from third parties (such as financial institutions and utilities suppliers). The CBA sends
messages to the BAL which in turn routes the messages to an appropriate endpoint, including the BRDB or other
service such as the Common Digital Platform (or CDP)*?.
PostOfficeCounter.log (retained
for six months)
Message.log (retained for six
months)
PBSIL.log (retained for six months)
C3.log and C3Crypt.log
CounterMemoryProfile.log
Stopcba.log
* See row 13 for information regarding BAL/OSR
“1 Note the CDP is operated by a third-party, Accenture, rather than FSL, but records for it may be available from Post Office or Accenture
Fujitsu
Fujitsu Services Limited, Registered in England no 96056, Registered Office Fujitsu, Lovelace Road, Bracknell, England, RG12 8SN
Letter to the City of London Police dated 23 August 2024
FUJ00243304
FUJ00243304
Page 17 of 22
Data Category
Description of Data
Records
Records
Records held on the Post Office counters in branches should include:
© The PostOfficeCounter.log containing records of interface interactions undertaken by the counter operator (such
as messages that were displayed at the counter, communications between PINPads and the counter) and reports
sent to the tally roll or back office printer (depending on the configuration and hardware of the relevant Post
Office branch).
e The Message.log containing messages, including transactions, sent to and received from the BAL.
e The PBSIL.log containing data of interactions between the CBA and the counter PinPads. PBSIL stands for Payment
and Banking Service Isolation Layer.
© The C3.log and C3Crypt.log containing data related to counter PINPad interactions. Both are created by third-
party WorldLine applications and are not readable by FSL without support from Worldline. In some circumstances
Worldline will request these logs from FSL for their own investigations. Note the C3Crypt.log ceased to be created
recently.
© CounterMemoryProfile.log which records of memory metrics from the CBA. This could potentially be useful, for
example, to investigate the cause of a slow running counter.
— Stopcba.log which records start-up information from the CBA before the PostOfficeCounter.log is created. The
CBA does not contain any user actions; it records the status of the system before a user logs on and may be looked
at if there have been problems in the operation of the CBA.
e Reference Data working day set — the reference data that is currently in use by the counter. Post Office maintains
acentral reference data server and the counter periodically refreshes/updates its Reference Data working day set
to ensure up to date reference data is used.
The Post Office counters are Windows devices (e.g. desktop computers). They are currently supplied and
maintained by third party suppliers rather than FSL. The Windows event log relating to the counter should contain
Reference Data working day set
Fujitsu
Fujitsu Services Limited, Registered in England no 96056, Registered Office Fujitsu, Lovelace Road, Bracknell, England, RG12 8SN
Letter to the City of London Police dated 23 August 2024
FUJ00243304
FUJ00243304
Page 18 of 22
Data Category
Description of Data
Records
application, system and security information, but this is not accessible by FSL as it does not supply the Post Office
counters used in branches
Note that for some of the logs above, for data security and protection purposes (e.g. to protect customer financial
information), these logs contain some obfuscation of data.
15. I Generic Web Service I Overview Application log for the GWS
(“ews”)
GWS is a mechanism for the branch counter to communicate with third parties. GWS has been used by multiple services I Messages received from Post
in the past, but is now understood to be only used regarding travel money. Specifically, for: (i) FRES*? Money Click and I Office Branch counters and sent
Collect (“FMCC”) and (ii) FRES*? Money Card Verify (“FMCV”). The GWS could be used to, for example, check that a top- I to third parties
up to a card does not exceed the allowed amount of foreign currency.
Metrics for each service processed
Records by the GWS.
Data held on the GWS“* includes:
an application log for the GWS;
© Messages received from Post Office Branch counters and sent to the third parties; and
© Metrics for each service processed by the GWS.
Operationally, FSL staff may examine the information sent to FRES and the response back if, for example, someone in
branch queries a FMCC order.
16. Horizon Business I Overview The BRDB indicates there are a
Server (“HBS”) large number of kiosks active, and
HBS is a platform and translation engine that FSL uses to interface information from third parties to the BRDB. It was
originally built to interact with kiosks supplied by third parties but the HBS now handles multiple services. Transactions
each kiosk may contain various
logs
“ First Rate Exchange Service
* [bid.
“ At the time of writing this letter, we have not been able to ascertain for what periods of time such data has been retained for.
Fujitsu
Fujitsu Services Limited, Registered in England no 96056, Registered Office Fujitsu, Lovelace Road, Bracknell, England, RG12 8SN
FUJ00243304
FUJ00243304
Page 19 of 22
Letter to the City of London Police dated 23 August 2024
Data Category Description of Data Records
can be conducted at kiosks (e.g. a self-service kiosk where a customer can print off and pay for a postage label). For
example one of the services handled by HBS is a web page based help and information service for Post Office branch
staff, called the Counter Help Service.
Records
The HBS produces a number of logs. FSL understands these are retained operationally for a maximum of between 3 to
5 days. If a kiosk transaction is questioned, the HBS logs may be used to understand what records there are of
transactions performed on the kiosk nodes.
We have not yet ascertained the extent to which such logs are sent to the Audit Archive and which may therefore be
available for extraction.
17. Horizon Estate I Overview The EMDB has existed from
Management around the end of 2009
FSL maintains an Estate Management Database (“EMDB”), a suite of databases housing Estate Management
configuration and change data required by or supplied to various clients and endpoints. The data includes items such
as IP addresses, counter numbers per branch and other information used by various interfaces (e.g. by a merchant
acquirer interface used to source terminal identifiers that are required to support certain in branch transactions)
Design documents exist for Estate Management in Dimensions, such as the Estate Management Database (EMDB) High
Level Design‘® and the System and Estate Management — Overall Architecture.*®
Records
The EMDB contains history tables which record changes made to the EMDB. Log files of access to EMDB data are also
created, Back-ups of the EMDB are taken nightly but these are only retained for two days. It is presently unclear to the
FSL legal team whether and to what extent other back up processes are undertaken of the EMBD which could have
longer retention periods.
EMDB history tables
Log files of clients/ access to
EMDB data
Back-ups of the EMDB
45 DES/SYM/HLD/0031, version 10.0, 15 November 2023
46 ARC/SYM/ARC/0001, version 4.1, 2 October 2015
Fujitsu
Fujitsu Services Limited, Registered in England no 96056, Registered Office Fujitsu, Lovelace Road, Bracknell, England, RG12 8SN
Letter to the City of London Police dated 23 August 2024
FUJ00243304
FUJ00243304
Page 20 of 22
Data Category
Description of Data
Records
18.
SSC Intranet/Website
Overview
The SSC maintains an intranet site to provide staff access to various records, including the Peak and KB databases. The
website provides access to Horizon support information, including records which are only stored on the SSC website.
Records
Records only stored for operational purposes on the SSC website include:
¢ OCP - Operational Change Proposal (referred to in row 4 above)
OCR - Operational Correction Request (referred to in row 4 above)
© KB - Knowledge Base (referred to above)
¢ WI- Work Instructions (e.g. documents providing guidance for support staff when performing certain tasks,
including activities performed using the APPSUP role)
* — St- Support Information (e.g. documents containing certain potentially sensitive information to allow the SSC to
carry out their role, such as passwords and IP addresses)
¢ MI- Management Information (provides an overview of various HNG-X services in a non-technical way, often
includes diagrams etc)
— Rotas - Support staff daily rotas
e Events - Notable dates of activities such as planned outages
Website maintained for a number
of years to present. We envisage
that an extract can be prepared of
the current contents of the
website (noting that it is used
operationally and will be updated
after that extract is taken).
Fujitsu
Fujitsu Services Limited, Registered in England no 96056, Registered Office Fujitsu, Lovelace Road, Bracknell, England, RG12 8SN
FUJ00243304
FUJ00243304
Page 21 of 22
Letter to the City of London Police dated 23 August 2024
APPENDIX 2
Sample BRDB Tables
Data Type Description (including table name)
Session Data All transaction lines that make up a basket (BRDB_RX_REP_SESSION_DATA).
Journal Journalizable messages such as logon, Basket Settle, Label print (BRDB_RX_IMESSAGE_JOURNA).
Transfers Incomplete and complete transfers between stock units (BRDB_SU_PENDING_TRANSFER).
TC Transaction Corrections received from Post Office (TPS_TXN_CORRECTION_DETAILS).
TA Transaction Acknowledgements from Post Office for 3rd party such as Paystation (TPS_TXN_ACK_DETAILS).
Recovery Transaction recovery details, created for each recoverable transaction (BRDB_RX_RECOVERY_TRANSACTIONS).
EPOSS Additional data relating to EPOSS transactions (BRDB_RX_EPOSS_TRANSACTIONS).
APS Automated Payment - including complex scripted transactions (BRDB_RX_APS_TRANSACTIONS / BRDB_F_RX_APS_Transactions).
ETU Electronic Top Up transactions (BRDB_RX_NWB_TRANSACTIONS).
Dcs Kiosk Payment transactions (BRDB_RX_DCS_TRANSACTIONS).
PBS Payment and Banking transactions including PBS reference and Response Code (BRDB_RX_PBS_TRANSACTIONS).
Bureau Bureau transactions including margin rates (BRDB_RX_BUREAU_TRANSACTIONS).
Pouch Pouch dispatch status (BRDB_REM_OUT_POUCH_HEADER).
BAP Barcode All Parcels (BRDB_RX_NRT_TRANSACTIONS).
Users Users in a branch (BRDB_BRANCH_USERS / BRDB_BRANCH_USER_ROLES / BRDB_BRANCH_USER_SESSIONS).
Node Info Nodes and kiosks in a branch (BRDB_BRANCH_NODE_INFO).
Stock Units tockunits, associations, TP, BP etc (BRDB_BRANCH_STOCK_UNITS).
Rep Events Report Events, including Declarations, Rollovers, Logons etc (BRDB_RX_REP_EVENT_DATA).
LFS PLO Planned Orders (LFS_PLO_HEADER / LFS_PLO_DETAILS).
LFS RDC Cash Replenishment (LFS_RDC_HEADER / LFS_RDC_DETAILS).
Poll Last time a node polled the datacentre, what version of CBA software it is running and ref data packages
(BRDB_BRANCH_NODE_MONITOR).
Fujitsu
Fujitsu Services Limited, Registered in England no 96056, Registered Office Fujitsu, Lovelace Road, Bracknell, England, RG12 8SN
Letter to the City of London Police dated 23 August 2024
FUJ00243304
FUJ00243304
Page 22 of 22
Data Type
Declarations
Description (including table name)
Report Reprints
Opening Figures
Branch
Fujitsu
Last Cash / Stamp declarations for each stock unit (BRDB_BRANCH_DECL / BRDB_BRANCH_DECL_ITEM).
Report reprints including Stock Unit Rollover, Branch Trading Statement (BRDB_RX_REPORT_REPRINTS).
Opening position for stock and cash per stock unit following a roll over (BP or TP) (BRDB_SU_OPENING_BALANCE).
All Horizon branches including Payzone, HiH and training Offices (BRDB_BRANCH_INFO).
Fujitsu Services Limited, Registered in England no 96056, Registered Office Fujitsu, Lovelace Road, Bracknell, England, RG12 8SN