POL00448622
POL00448622
Tab 7.1 Belfast Data Centre Fortification
@
POST OFFICE LIMITED
BOARD REPORT
Title: Belfast Data Centre Fortification Meeting Date: I 24' January 2023
. Jeff Smyth, Enterprise Cloud & . Jeff Smyth, Enterprise Cloud &
Author: Data Transformation Director Sponsor: Data Transformation Director
Input Sought: Decision
Board is requested to: -
« Review _and endorse the recommendation for delegated funding approval to GE for
Capex to allow the Data Centre Fortification Project to approve work orders
I Fujitsu to deliver the scope the Board agreed to on the 27th September 2022.
to compile contingent options should a data centre support contract extension
m March 2024-March 2025) with Fujitsu not be reached.
Previous Governance Oversight
+ A decision was made to cease Belfast Cloud Migration on 27th September 2022, a
revised focus on Application Modernisation - to enable NBIT and further progression
on the Belfast Data Centre fortification planning/cost estimation was agreed.
Executive Summary
« At the September 2022 Board, the Board agreed to halt Belfast Cloud Migration,
and this stimulated the need to recontract, upgrade and fortify the Belfast Data
Centres to allow Fujitsu to continue operating Horizon until March 2025, at which
point all branch core trading will be executed on NBIT.
e In parallel, Post Office has commenced negotiations for an additional
expires in March 2023 until Ti.
extension with Fujitsu is already secured through to
Data Centre support contract has proved to be very challenging, as the Fujitsu
position since 2020 has consistently been that they wish to exit the Horizon
contract and that they harbour multiple concerns about supporting out of date
hardware and software in the Data Centres. The shared expectation of Fujitsu and
Post Office was that much of the end-of-life hardware/software would have been
eliminated had the Data Centre transition to AWS cloud activity been completed as
originally expected.
e The Horizon infrastructure, although stable, is old and most of the hardware and
Confidential
92 of 165 POL Board Meeting-24/01/23
POL-BSFF-WITN-021-0000004
Tab 7.1 Be
POL00448622
POL00448622
fast Data Centre Fortification
@
As of the compilation date of this paper, Fujitsu are currently only willing to agree
to a single year Data Centre support contract extension from March 2023 until
March 2024 subject to the current Data Centre Fortification Project scope being
agreed, funded and delivery initiated. Fujitsu have also indicated that there may be
some additional technical investments in additional to the existing scope required
as an enabling condition for a March 2024 unti xtension which relate
to the upgrade of some very old operating systems. This is an additional, newly
emerging condition which is under review (and robust scrutinisation by Post Office).
In overall terms, Fujitsu have limited the period of the extension for two reasons: -
iu are concerned that the NBIT roll out will not have completed by {
w(i.e., Horizon will still be in operational use) and that Fujitsu will be required
to continue support for an indeterminate further period on a hardware/software
environment during which support capabilities are diminishing.
e Fujitsu have already signalled that any further extension of the Data Centre
support contract from March 2024 unti ill be conditional on
additional hardware/software upgrades (including the old operating system
upgrades referenced earlier) and an Oracle Database refresh which is un-costed
activity in this current proposal. Historically, Oracle Database upgrades,
executed by Fujitsu, have cost in the range of however Post Office
are exploring the expiring support position and upgrade options directly with
Oracle to validate the execution cost estimation range.
The Data Centre Fortification Project mitigates hardware failure for the components
in scope that are being fortified, either through direct replacement or by creating an
extensive bank of spares. However, the project is not a full refresh of the entire
infrastructure in Belfast; replacement scope has been aligned to those components
with the greatest security vulnerabilities, unsupported technologies and end of
serviceable life software which is not receiving vulnerability patching.
At the previous Board submission, it was estimated that “would be
required to complete Belfast datacentre fortification scope. This has been revised to
(through reconciliation of Project and BAU run costs assumptions). This
st estimate based on current knowledge and could increase due to risk
materialisation, or if we are forced to accept the additional operating system request
investment from Fujitsu during the contract negotiations for the March 2024 Data
Centre support extension. Delegated Approval for: Capex is requested to GE
related to this activity.
Informal discussions between Post Office and Fujitsu indicate that there is zero
appetite at Fujitsu t er application or infrastructure support contracts
for Horizon beyond Fujitsu have repeatedly reinforced their desire to
withdraw from all aspects of the Horizon contract and their stance and posture
seem to be a unified position for Fujitsu UK and the Fujitsu Japanese parent.
If Horizon is required post March 2025 the Post Office will require a backup plan to
provide Data Centre and Application support services for Horizon until the NBIT roll
out is complete, should a further extension not be achievable with Fujitsu. This is a
major operational risk and Post Office have limited time to compile a realistic
Confidential
POL Board Meeting-24/01/23
f 165
POL-BSFF-WITN-021-0000004_0001
Tab 7.1 Be
POL00448622
POL00448622
fast Data Centre Fortification
@
94 of 165
alternative strategy that fulfils procurement, commercial, technical and
transitioning activities. GE have requested urgent analysis of this scenario to
determine the full technical, commercial, and organisational implications of ongoing
Operational support should a stalemate with Fujitsu occur. GE have allocated up to
“jof investment to execute this work and Board are requested to delegate
approval of up to to GE to facilitate urgent completion of the background
analysis and options planning work.
Overview
1, What is the Horizon Support Position to! IRRELEVANT I:
« Fujitsu have been consistently communicating, since 2020, that they want to exit the
Horizon contract and even offered Post Office incentivisation for the early transfer of
infrastructure and application support ahead of March 2023 contractual expiries.
* We negotiated (in December 2020) a revised !!
that provided Post Office application support unti
with no further provisions to extend.
pplication support contract
e Our current Data Centre infrastructure support contract expires in March 2023.
end of life software impacts emerge and total delivery costs are potentially onerous for
Post Office. The absolute present limit of what Fujitsu are prepared to offer is an
extension until March 2024. The Fujitsu delivery position appears to have hardened in
the last 3 months which may be partially influenced by the degree of external
scrutinisation by the Horizon Inquiry and Fujitsu’s escalating concerns about the
aging/supportability of the Horizon platform and the inherent reputational impact of a
major security/operational incident.
¢ One of the main end of life components that Fujitsu are concerned about is the Oracle
database which is supporting all of Horizon’s branch transaction:
u will almost certainly seek an upgr
extension which could cost up to} This is not budgeted in
our current delivery estimates and would represent a significant level of unplanned
activity to be understatement in parallel with present technical commitments occurring
in the Horizon live environment.
« Fujitsu have stated that to even consider a support extension beyond March 2024 we
will need to agree a further package of end-of-life hardware/software upgrades beyond
the present commitments for Data Centre Fortification by June 2023. If we cannot
reach agreement Fujitsu will provide no further Data Centre support after March 2024
which will leave the Post Office in a position of having no Data Centre support available
from April 2024 onwards. Clearly this is an untenable operational situation, and GE
3
Confidential
POL Board Meeting-24/01/23
S a condition of March
POL-BSFF-WITN-021-0000004_0002
Tab 7.1 Be
POL00448622
POL00448622
fast Data Centre Fortification
@
2. What are the options for a Horizon extension after i
have initiated compilation of the full range of options available should Fujitsu support
be withdrawn after March 2024. This will include analysis of our market procurement
requirements, identification of appropriate technical support partners, a high-level
transitioning strategy and a costed programme delivery plan. GE have allocated up to
‘0 be overseen and approved by IADG.
Informal discussions indicate that there is zero appetite from Fujitsu to extend either
application or j frastructure support contracts for Horizon beyond
support beyond: is unavailable from Fujitsu, then the Post Office will either
need to recontré ternative partner or undertake direct Horizon support.
Solution options and the organisational/technical/commercial impact of these scenarios
are neither modelled nor costed within the present scope of the Data Centre
Fortification project. The options analysis activity referred to earlier, will provide high
level options and a contingency pl: oth the March 2024+ ‘}period and
for support arrangements post; IRRI
3. What is the Data Centre Fortification Scope?
« The Data Centre Fortification Project delivery. acti ity has already commenced based
‘on the scope approved previously. An initial ! ‘unding allocation was assigned
to begin discovery and design activities with Fujitsu. This initial funding has been
used for the following items: -
- Discovery, Scoping and Design of all elements by Post Office resources and
Fujitsu technical design analysis of the proposed scope to produce work orders.
- The project has started and has commissioned Fujitsu to deliver of the first
elements in the Cybersecurity Infrastructure area: -
= Replacement of the Citrix Load Balancer Replacements
= Upgrade the Intrusion Detection Sensors in the Network
. Deployment of new w Vulnerabllity Management appliances
3. Business ‘Continuity Plant - Refresh of ‘Cooling systems in IRE11
4. Business Continuity Infrastructure - Procurement of Hardware spares
4, Summary of Funding Position and Drawdown Request?
* Originally in the previous board submission, the project was estimated to cost
between with an indicative allocation set at/iri
Confidential
POL Board Meeting-24/01/23
95 of 165
POL-BSFF-WITN-021-0000004_0003
POL00448622
POL00448622
Tab 7.1 Belfast Data Centre Fortification
96 of 165
5. What are the benefits for the Post O
EVANTI
The project is tracking t
REL
, this could be exceeded due to the following risks:-
« The costs are based on Fujitsu initial estimates which are non-committed forecast
of costs. The Project team are reviewing (and challenging) each scoping decision
against a risk of hardware failure to control costs. However, Fujitsu have already
indicated that several work packages may well exceed their original estimates
which would increase the total toi!"**“"";, Given nascent security events in tandem
with Fujitsu’s diminishing risk tolerance of end-of-life hardware/software
vulnerabilities it may be extremely difficult to abate these additional investment
requirements if we are to maintain our support arrangements for Horizon.
« Further initial fortification activity may be discovered during the design phase
which either Post Office decides it must resolve due to the inherent risk to Horizon,
or which Fujitsu insists must be resolved as a prerequisite to any further Data
Centre contract extensions. This could take the total cost beyond by an
amount that cannot currently be quantified.
Even with judicious challenge Post Office may not be able to resist any further scope
changes (and increased costs) due to the weakness of our negotiating position with
Fujitsu. If this risk crystallises the project will revert with a further paper.
The requested drawdown amount delegated to GE is:-
apex to allow us to commit to work orders with Fujitsu. Together with
which was previously approved to allow to project to begin this would
future date.
iCapex to compile contin
extension (from March 202:
s should a data centre support contract
with Fujitsu not be reached.
e and NBIT of Fortification
The present hardware support situation in the Data Centres represents a low likelihood
but high impact risk for the Post Office. If a component experienced a critical issue
which was unresolvable by reboots or localised workarounds, the Post Office could be
without a branch trading platform for an indeterminate downtime period. The Data
Centre Fortification Project mitigates this risk for the components in scope that are
being fortified, either through replacement or by creating an extensive bank of spares.
However, the project is not a 100% refresh of the entire hardware/software
infrastructure or plant in the Belfast Data Centres.
The programme of work is a necessary enabler of retaining our Data. Centre support
contract with Fujitsu minimally until March 2024 and potentially until {IRRELEVANT with
conjunction with Application Modernisation deliverables, for the Post Office to dual run
core branch transaction processing on Horizon and migrate to NBIT. Should Horizon be
required beyond Ti, alternatives to Fujitsu provided support will need to
planned, mobilised and funded.
Confidential
POL Board Meeting-24/01/23
POL-BSFF-WITN-021-0000004_0004