FUJ00176502
FUJ00176502
From: Browell, Steven[/O=EXCHANGELABS/OU=EXCHANGE ADMINISTRATIVE GROUP
(FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=3D3D7C6D3423416C862CDAOE068EF742-
BROWELL, ST]
Sent: Thur 02/12/2021 3:29:11 PM (UTC)
To: Barnes, Gerald
; Gauntlett, Paulf”
Boardman, Phill.
Ce: Mistry, Manishaf””
Subject: RE: Historical Issue with Audit Data
Thanks Gerald,
I will remove the bullet point re FADs using specific DCs.
As for the p.s. Iam afraid I am not sure what that means and how it affects what we are describing. If you are simply
enlightening me as to how it was configured — noting there were some differences to how other things were done —
then thank you. However, if you are saying that the way it was configured was wrong then that warrants more
checking.
Also, how can we pro-actively confirm no gaps in the combined audit archives for pre-HNG-X - instead of just waiting
for ARQ anomalies to pick up on something? Can we write a script/program to hunt for any possible issues?
Steve Browell
From: Barnes, Gerald <_
Gauntlett, Paul
“>; Boardman, Phil
c: Mistry, Manisha <
Subject: RE: Historical Issue with Audit Data
Hi Steve,
Apologies for the delay in replying. Too much going on including one of our cars in the garage!
. By design, some FADs used Bootle as their primary data centre, and others used Wigan. Some used both
This is not right,
My understanding is that Wigan and Bootle are equal.
There were separate harvesters on each.
Any transaction is saved both in a Wigan file and a Bootle file. Both audit servers were active. The Wigan audit
server saved the Wigan file and the Bootle audit server saved the Bootle file.
Regards,
Gerald Barnes
p.s. Both audit servers by a configuration change could in principal have copied each file one to other but my guess is
FUJ00176502
FUJ00176502
that the designers thought that you would end up with 4 copies of each transaction which was overkill. From the
notes in DEV/INF/ION/0001 it looks like most other files other than these transaction files were copied one to other. I
attach an earlier copy of DEV/INF/ION/0001. You will see most of the time in the table at the end “Transfer Data” is
“YES” apart from the below
55 Cluster1B 8 TMS 8 TMSA TMS transactions gather MBOCORO1\DS\Audit\Done HOARD+4320 84 0 1 tmp NO NONE
from Correspondence
Server Cluster 1
56 Cluster2B 8 TMS 7 TMSB TMS transactions gather \\MBOCORO2\D$\Audit\Done HOARD+4320 84 0 1 tmp NO NONE
from Correspondence
Server Cluster 2
57 Cluster3B 8 TMS 8 TMSA TMS transactions gather —_\\MBOCORO3\DS\Audit\Done HOARD+4320 84 0 1 tmp NO NONE
from Correspondence
Server Cluster 3
58 Cluster4B 8 TMS 7 TMSB TMS transactions gather \\MBOCORO4\DS\Audit\Done HOARD+4320 84 0 1 tmp NO NONE
from Correspondence
Server Cluster 4
59 ClusterlW 8 TMS 8 TMSA TMS transactions gather \\MWICORO1\DS\Audit\Done HOARD+4320 84 0 1 tmp NO NONE
from Correspondence
Server Cluster 1
60 Cluster2W 8 TMS 7 TMSB TMS transactions gather MWICORO2\D$\Audit\Done HOARD+4320 84 0 1 tmp NO NONE
from Correspondence
Server Cluster 2
61 Cluster3W 8 TMS 8 TMSA TMStransactions gather —_\\IWICORO3\DS\Audit\Done HOARD+4320 84 0 1 tmp NO NONE
from Correspondence
Server Cluster 3
62 Cluster4W 8 TMS 7 TMSB TMStransactions gather —_\\MWICORO4\DS\Audit\Done HOARD+4320 84 0 1 tmpI/NO NONE
from Correspondence
Server Cluster 4
371 Cluster5B 8 TMS 14 DUMMY Unused MBOCOROS\DS\Audit\Done HOARD+1440 84 0 1 tmp NO NONE
372 Cluster6B 8 TMS 14 DUMMY Unused \\MBOCORO6\DS\Audit\Done HOARD+1440 84 0 1 tmpINO NONE
373 Cluster7B 8 TMS 14 DUMMY Unused \\MBOCOROZ\DS\Audit\Done HOARD+1440 84 0 1 tmp NO NONE
374 Cluster8B 8 TMS 14 DUMMY Unused \\MBOCOROS\DS\Audit\Done HOARD+1440 84 0 1 tmpINO NONE
From: Browell, Steven <_ >
Sent: Thursday, December 2, 2021 1:34 PM -
To: Gauntlett, Paul < GRO >; Barnes, Gerald <q GRO ““f>; Boardman, Phil
Subject: RE: Historical Issue with Audit Data
I attach my amended notes for your critique please before I pass to various parties for review and permission to
release.
FUJ00176502
FUJ00176502
Steve Browell
Mob§_
From: Gauntlett, Paul <.
>; Boardman, Phil ¢ >; Browell, Steven
istry, Manisha <
Subject: RE: Historical Issue wit!
Hi All,
Please see updated statement below.
* Accepted the changes that Phil added directly
© Orange updates based on comments below from Phil/Gerald.
Steve/Gerald/Phil — please confirm if happy with the below. I’m guessing there may be some further tweaks needed
so will setup a short call for tomorrow am so we can finalise - I can cancel if not needed!
Problem Summary:
Post Office counter audit transaction files gathered prior to the HNGx software rewrite in 2010 are not
consistent across both IRE11 & IRE19 servers. Holistically no data is missing but in rare instances transaction
data exists only on one server or the other. The problem was caused by the occasional malfunction of the
Harvester process of the previous Horizon system - Riposte.
Background:
Audit Gathering Method in Horizon up to approx 2010
* Originally a system called Riposte was used, as part of Horizon, to gather audit transactions from all the Post
Office counters.
© Riposte was a big distributed database.
* At that time Riposte was deployed into Bootle & Wigan data centres.
* Each evening all Post Office counter transactions were harvested and stored in files.
© By design transactions were either harvested to Wigan or Bootle or both data centres.
* Different files were produced in each data centre. The files were different because although the same
transactions were harvested they were processed in a different order and for different FADs. Filenames were
also different.
* The “CopyData” option (robocopy) was not set for transaction files which were gathered to both data centres.
© It is now known that due to the occasional malfunction of the Harvester process, for those transactions
harvested to both datacentres, data may be missing from the files gathered in one or other data centre.
e There is no recorded instance of both harvesters failing at the same time so although data may be
missing from one server it will be present in the other.
e NOTE: The data gathered using this method, in accordance with the contract, should have been deleted
by now. It is only being retained currently, beyond Fujitsu Services’ contracted obligations, at Post Office’s
request (below text from CWO0395b)
Post Office have requested under RTQSRO003106 (previously RTQSRO002349 and RTQSRO002456) that
Fujitsu Services continue to preserve data (including Post Office Personal Data) generated on the HNG-X
System as per the previous work orders (CT2616a & CW0251a) until 30 April 2022.
Audit Gathering Method for HNG-X from 2010 onwards
FUJ00176502
FUJ00176502
e Around 2010 the contract came up for renewal.
e — Fujitsu’s proposal was a rewrite called HNGx which eliminated Riposte (for which there was a big annual
licence fee) and to migrate the datacentres from Wigan and Bootle to IRE11 and IRE19.
e =Asapart of this rewrite each Post Office counter transaction was only processed into a one file ona
single server.
e This drove the decision to change the Audit gathering approach.
¢ From this point forward all transaction and non-transaction files were gathered by the IRE11 Audit Server
only and robocopied to the IRE19 Audit Server.
¢ Therefore from 2010 all audit files (transaction and non-transaction) are consistent across both audit
servers.
Impact on ARQ requests
¢ When ARQ requests for a FAD code are made all relevant files are retrieved from the target audit server .
«Then the files are processed by the Query Manager service on the audit server.
e¢ — Each Horizon or HNGx transaction has a unique number associated with it. The Query Manager checks
that the transactions
= have not been tampered with
= that there are no duplicates or gaps in the sequence of transactions.
¢ Due to the Riposte harvester issue, for ARQ queries from 2010 or earlier, there may be gaps in the results
from one or other of the servers.
e — If this occurs then a Peak is raised and the ARQ request is rerun on the other server. This resolves any
issues with data gaps.
*® There have been no reported instances of unsuccessful ARQ request once a query has been executed on
the second server.
From: Barnes, Gerald < GRO b
Sent: Wednesday, December 1, 2021 1:29 PM
To: Boardman, Phil
; Gauntlett, Paul ; Browell, Steven
istry, Manisha <.
Subject: RE: Historical Issue with Audit Data
Hi Phil,
Answered inline.
Regards,
Gerald Barnes
From: Boardman, Phil ¢
Sent: Wednesday, December 1, 2021 1
To: Gauntlett, Paul
Cc: Barnes, Gerald <.
Subject: RE: Historical Issue with Audit Data
Browell, Steven
; Mistry, Manisha
Hi Paul
I think these two butllet-points (in the pre-2010 section) are liable to cause confusion/consternation
FUJ00176502
FUJ00176502
The Bootle audit server gathered Bootle transaction files and the Wigan audit server gathered Wigan
transactions files.
e Although there was an option to robocopy files from one server it was switched off because there were
already two copies of each transaction (one in each data centre).
... aS I understand it (from Gerald's explanation (and I may have mis-understood)), by design transactions were either
harvested to Wigan or Bootle or BOTH ... and the robocopy was ONLY turned off for those transactions configured to
be copied to both datacentres ... I think it’s important that we show that this was not designed to leave single copies of
data anywhere. Please correct me if I’m wrong there Gerald.
My understanding (from reading historic copies of DEV/INF/ION/0001 and from practical experience) is that the main
case of the “CopyData” option (the robocopy) not being set was these transaction files. The logic behind that I guess (I
was not the designer) is that two copies of each transaction were made anyway — one on Bootle and one Wigan. If we
need more detail on this I will need to get out from Dimensions historic copies of the configuration file to examine.
if !’m not wrong, then this statement “due to the occasional malfunction of the Harvester process in one or other of the
data centres transactions may be missing from the files gathered in that data centre” needs to be clarified that that’s
only true for those transactions configured to be harvested to BOTH datacentres.
That is right. Pretty certain that was all Horizon (pre 2010) transaction data.
I’ve also proposed some changes to the text inline below (in this colour), trying to make it more clear that this was a
change instigated by the Horizon to HNG-X change.
I think we also need to consider how we should include the details that the data in question is only being retained
currently, beyond Fujitsu Services’ contracted obligations, at Post Office’s request (below text from CWO0395b) ...
Post Office have requested under RTQSRO003106 (previously RTQSRO002349 and RTQSRO002456) that Fujitsu
Services continue to preserve data (including Post Office Personal Data) generated on the HNG-X System as
per the previous work orders (CT2616a & CW0251a) until 30 April 2022.
and (in accordance to our contract) should have been deleted, by now.
Yes that is right — normally the files would have been deleted long ago!
Regards, PhilB
From: Gauntlett, Paul <.
Sent: Wednesday, December 1, 2021 11:51 AM
To: Browell, Steven 4 :
Cc: Barnes, Gerald <!” GRO i>; Mistry, Manisha <____¢ GRO >; Boardman, Phil
Subject: Historical Issue with Audit Data
Hi Steve
lam Migration Lead for the Audit Migration to AWS.
Myself and Gerald Barnes (Audit SME) are currently working with POL to define requirements for the migration of
historical Audit data
An historical issue has been identified and is detailed below.
This issue was discussed yesterday with John Nelis the POL PM & also Dean Bessell who I understand is your POL
opposite.
POL requested that we write up what was communicated in the meeting so they can take it to their legal team ahead
of making a decision regarding the scope of the data to be migrated.
On that basis I don’t wish to send anything over without it being reviewed and agreed by relevant parties. Happy to
have a call to discuss further.
FUJ00176502
FUJ00176502
Problem Summary:
Post Office counter audit transaction files gathered prior to the HNGx software rewrite in 2010 are not
consistent across both IRE11 & IRE19 servers. Holistically no data is missing but in rare instances transaction
data exists only on one server or the other. The problem was caused by the occasional malfunction of the
Harvester process of the previous Horizon system - Riposte.
Background:
Audit Gathering Method in Horizon up to approx 2010
Originally a system called Riposte was used, as part of Horizon, to gather audit transactions from all the
Post Office counters.
e — Riposte was a big distributed database.
e Atthat time Riposte was deployed into Bootle & Wigan data centres.
e — Each evening all Post Office counter transactions were harvested and stored in files.
* The Bootle audit server gathered Bootle transaction files and the Wigan audit server gathered Wigan
transactions files.
« Different files were produced in each data centre. The files were different because although the same
transactions were harvested they were processed in a different order and for different FADs. Filenames
were also different.
e Although there was an option to robocopy files from one server it was switched off because there were
already two copies of each transaction (one in each data centre).
e — Itis now known that due to the occasional malfunction of the Harvester process in one or other of the
data centres transactions may be missing from the files gathered in that data centre.
«There is no recorded instance of both harvesters failing at the same time so although data may be
missing from one server it will be present in the other.
e TBC-Ina DR situation data could have been lost was this captured as an operational risk and
communicated to POL?
Audit Gathering Method for HNG-X from 2010 onwards
e Around 2010 the contract came up for renewal.
e — Fujitsu’s proposal was a rewrite called HNGx which eliminated Riposte (for which there was a big
annual licence fee) and to migrate the datacentres from Wigan and Bootle to IRE11 and IRE19.
e Asapart of this rewrite each Post Office counter transaction was only processed into a one file ona
single server.
e This drove the decision to change the Audit gathering approach.
e From this point forward all transaction and non-transaction files were gathered by the IRE11 Audit
Server only and robocopied to the IRE19 Audit Server.
e Therefore from 2010 all audit files (transaction and non-transaction) are consistent across both audit
servers.
Impact on ARQ requests
e When ARQ requests for a FAD code are made all relevant files are retrieved from the target audit
server .
«Then the files are processed by the Query Manager service on the audit server.
* — Each Horizon or HNGx transaction has a unique number associated with it. The Query Manager checks
that the transactions
© have not been tampered with
© that there are no duplicates or gaps in the sequence of transactions.
* Due to the Riposte harvester issue, for ARQ queries from 2010 or earlier, there may be gaps in the
results from one or other of the servers.
« — If this occurs then a Peak is raised and the ARQ request is rerun on the other server. This resolves any
FUJ00176502
FUJ00176502
issues with data gaps.
* There have been no reported instances of unsuccessful ARQ request once a query has been executed
‘on the second server.
Regards,
Paul Gauntlett
Customer Solution Architect
Cloud Transformation & Development - AMCS.
Fujitsu
Central Park, Northampton Road, Manchester, M40 SBP
United Kingdom
@: www fujitsu.com/uk