Dave;
In response to your note of Friday, I do understand your concern regarding outages being
contributed to by operational changes in the live estate and had already enforced the
message of rigorous change control across our technical and service management teams.
As a result of continuous improvement activity and in support of the deployment, we do have
a volume of operational change activity. However, in this instance the problems experienced
on the 26th of March were not due to operational changes in the live estate other than the
migration of 208 branches as part of the agreed deployment plan.
We have established that the Incident was as a result of the Oracle database holding open
connections and this in turn created timeouts which subsequently degraded the performance
of the Branch Access Layer. This prevented the successful processing of transactions. We
have identified a gap in the monitoring in relationship to these connections between branch
access layer and the database and a defect has been raised to correct this.
In terms of the overall monitoring of the infrastructure we are actively addressing the profile
of the alerts and events raised. This has been an on-going activity since Data Centre
migration. We have also taken proactive measures to fine-tune the escalation and
communication of issues relating to HNG between the respective Service Delivery
organisations. To complement this activity we are holding daily 7:30 calls with the internal
delivery capabilities to review and confirm the health of our infrastructure and processing of
the overnight batch routines.
It has been already been agreed and acknowledged that our prime objective is to minimise
disruption and the ability of the branch network to trade and I am acutely aware of the
support and tolerance of the Postmasters and Postmistresses in the HNG pilot and recognise
their patience. To that end for the last 4 weeks we have been proactively calling the PMs in
the pilot to gauge their experiences and to provide feedback on any issues they had logged
to date. We have also attended the recent NFSP meeting in conjunction with POL
Programme.
We do recognise from previous Horizon experiences especially the introduction of network
banking in 2004 that a combination of joint working is key to overcoming the challenges
presented with the remote application rollout of this scale.
The conflicting needs for deployment progression and stabilisation is always a challenge;
which we recognise. However we are striving to demonstrate stability and continuous
improvement to the existing Horizon Online branches. This ranges from the aforementioned
proactive calls to the PMs, uplifting resource and re-profiling shifts at the service desk, uplift
in the engineering resource, increased out of hours cover for the technical architecture, focus
on trend analysis in both events and calls raised from the estate. This is being achieved
whilst maintaining the high levels of service for Horizon and the on-going deployment of the
branch router.
Regards
Gavin
FUJ00094959
FUJ00094959