Attendees
Antonio Jarnasb (AJ)
Emma Langfield (EL)
Alan Simpson (AS)
Julia Marwood (JM)
lan Trundell (IT)
Andrew Winn (AW)
Mike Stewart (MS)
John Simpkins (JS)
Gareth Jenkins (GJ)
Mark Wright (MW)
What is the issue?
Receipts/Payments Mismatch issue notes
I ;
Service Delivery
Service Delivery
Security
Network I
IT I
POL Finance
Fujitsu SOM
Fujitsu Security
Fujitsu Technical Specialist
Fujitsu Technical Specialist
I
I
I
{
I
I
4
I
FUJ00083406
FUJ00083406
Appendix 1 to Cs' Responsive Note
Fe)
FUJITSU
' I
Discrepancies showing at the Horizon counter disapp ear when the branch follows certain process steps, but will still show within the back end branch
account. This is currently impacting circa 40 Branc hes since migration onto Horizon Online, with an ov erall cash value of circa £20k loss. This issue wil I
only occur if a branch cancels the completion of the trading period, but within the same session continues to roll into a new balance period.
\
At this time we have not communicated with branches affected and we do not believe they are exploiting this bug intentionally
The problem occurs as part of the process when moving discrepancies on the Horizon System into'Local Suspense.
When Discrepancies are found during Stock Unit roll over into a new Transaction Period, then the User is asked if the discrepancy should be moved to
Local Suspense. If the branch presses cancel at this point the Discrepancy is zeroed on the Horizon System.
Page 1 of 5 I
COMMEDCIAL IN-CONEINENCE
I
i
I
FUJ00083406
FUJ00083406
Receipts/Payments Mismatch issue notes
I [oe]
I FUJITSU
Note at this point nothing into feeds POLSAP and Credence, so in affect the POLSAP and Credence shows the discrepancy whereas the Horizon
system in the branch doesn't. So the branch will then believe they have balanced.
If at the next screen the rollover is completely cancelled, then no harm is done. However if the Rollover is re-attempted at this point, the rollover will
continue without any discrepancy meaning Horizon doesn’t Match POLSAP or Credence
This has the following consequences:
= There will be a Receipts and Payment mismatch corresponding to the value of Discrepancies that were “lost”
i
Note the Branch will not get a prompt from the system to say there is Receipts and Payment mismatch, therefore the branch will believe they have
balanced correctly..
I
= When the Branch begins the new Branch Trading period the discrepancies will show at Zero, however the Receipts and Payment mismatch will carry
over into the next period. I
Impact
e The branch has appeared to have balanced, whereas in fact they could have a loss or a gain.
e Our accounting systems will be out of sync with what is recorded at the branch
e If widely known could cause a loss of confidence in the Horizon System by branches
¢ Potential impact upon ongoing legal cases where branches are disputing the integrity of Horizon Data.
e [t could provide branches ammunition to blame Horizon for future discrepancies
Page 2 of 5 I
I COMMEDCIAL IN_CONEINENCE
FUJ00083406
FUJ00083406
“ Receipts/Payments Mismatch issue notes
[oe]
I FUJITSU
I
The Receipts and Payment mismatch will result j in an error code being generated which will allow Fujitsu to isolate branches affected this by this
problem, although this is not seen by the branchés. We have asked Fujitsu why it has taken so long to react to and escalate an issue which began in
May. They will provide feedback in due course. I
Fujitsu are writing a code fix which stop the discrepancy disappearing from Horizon in the future. They are aiming to deliver this into test week
commencing 4" October. With live proving at the model office week commencing 11" October. With full roll out to the network completed by the 21° of
October. We have explored moving this forward and this is the earliest it can be released into live.
The cade fix will on stop the issue occurring in the future, but it will not fix any current mismatch)at branch.
I
Proposal for affected Branches I
@ orancnes I
There are three potential solutions to apply to the impacted branches, the groups recommendation is that solution two should be progressed.
\ ‘
SOLUTION ONE - Alter the Horizon Branch figure at the counter to show the discrepancy. Fujitsu would have to manually write an entry value to the
local branch account.
IMPACT - When the branch comes to complete next Trading Period they would have a discrepancy, which they would have to bring to account.
RISK- This has significant data integrity concerns and could lead to questions of “tampering” with the branch system and could generate questions
around how the discrepancy was caused. This solution could have moral implications of Post Office, changing branch data without informing the branch.
\ :
SOLUTION TWO - P&BA will journal values from the discrepancy account into the Customer Account and recover/refund via normal processes. This will
need to be supported by an approved POL communication. Unlike the branch “POLSAP” remains in balance albeit with an account (discrepancies) that
should be cleared. I
IMPACT - Post Office will be required to explain the reason for a debt recovery/ refund even though there is no discrepancy at the branch.
RISK - Could potentially highlight to branches that Horizon can lose data.
SOLUTION THREE - It is decided not to correct the data in the branches (ie Post Office would prefer to write off the “lost”
IMPACT - Post office must absorb circa £20K loss I
RISK - Huge moral implications to the integrity of the business, as there are agents that were potentially due a cash gain on their system
Page 3 of 5 I
COMMEDCIAL IN-CONEINENCE
FUJ00083406
FUJ00083406
Receipts/Payments Mismatch issue notes
FUJITSU
Action Point Summary
I
I
I
I
I
I
I
Audit accounts Audit of Credence and ROLSAP to check both systems match. 8" October
‘Tisolate branch issues I Provide list of Horizon Desk calls made by affected branches to see if any other issue 8" October MS
impacting branches I
Forward Fix plan Provide timeline for permanent for fix introduction to branches 6" October MS
i
Branch Performance I Confirm with Shaun Turner any future audits for Branches and any performance i issues 8" October JM
review flagged I i
Branch Investigation I Confirm any existing of future investigations for impacted branches I 8" October AS
review I i
NBSC Script Confirm NBSC scripts for receipts and payments 6" October TS
HSD Script Confirm Horizon System Desk script for receipts and payments 6" October MS
Branch weekly report I Fujitsu to provide a weekly report updating the affected Branches until rgraion of fix is 8" October GJ
completely rolled out I
Further Branch Provide Fujitsu a list of branches to carry out further investigations I 6" October EL
investigation I I
\
Page 4 of 5
COMMEDCIAL IN-CANCINENCE I
FUJ00083406
FUJ00083406
fl
~ Receipts/Payments Mismatch issue notes
I
I
\
I
Fujitsu Fujitsu to supp ly output of further investigations : " ctober
investigations I
Communications I Engage Post Office Communication teams to understand what should be shared with 6" October TJ
Branches I ]
Stakeholder Produce report for Post! Office Stakeholders I 6" October TJ
Report I :
I I
I I
I
I I
I
I I
I
I
Page 5 of 5 f
i
COMMEDCIAL IN_-CONECINENCE
Ref:
FUJ00083406
FUJ00083406
Appendix 2 to Cs' Responsive Note
Correcting Accounts for “lost” Discrepancies
g:\gij documents\notes\lost discrepancies.doc
Author: Gareth I Jenkins
Date:
1.
2.
Change Control
29/09/2010 10:50:00
Introduction
This note relates to Peaks PC0204765 and PC0204263 (and also PC0203864 which is
a duplicate of PC0204263).
Are these really duplicates? I’m a bit confused as to which one to refer to. Can one be closed
as a duplicate of the other?
PC0204263 describes a problem with SU Balancing that will result in a Receipts
payments mismatch. A fix is available for this peak which needs to be scheduled via
“RMF. However any brarich encountering the problem will have corrupted accounts
and Peak PC0204765 is a Master Peak to record al affected branches and also to
define the process for correcting the accounts.
The purpose of this note is to:
= Summarise the problem in terms that are meaningful to Post Office Ltd
= Define a process for identifying all affected branches
= Explain what analysis is needed on each affected branch
= Define what ongoing monitoring is required to pick up further occurrences of the
issue until the root cause of the problem is fixed
= Provide a basis for agreeing the necessary data fixes with Post Office Ltd and how
they are to be applied
«Explain how each problem branch can be fixed
Initial version 28/09/2010 12:49:00: This version of the note is an initial draft for
discussion within development.
Updated version 29/09/2010 10:50:00: Updated following feedback from Dev to be
distributed to SSC
Overview
The problem occurs as part of the process of moving discrepancies into Local
Suspense.
When Discrepancies are found when rolling a SU over into a new TP, then the User is
asked if they should be moved to Local Suspense (MSG31316). Should they Cancel
at this point the Discrepancy is zeroised in the Local Cache (but nothing is written to
the BRDB). Note that there is no corresponding Balancing Transaction generated in
the Local Cache and so the Local Cache is in an Unbalanced state.
c:\documents and settings\jarnail.a.singh\local settings\temporary internet files\olk18\lost discrepancies
290910.
doc Printed at 16:38:24 on 8/10/2010 Page 1 of 4
FUJ00083406
FUJ00083406
If at the next screen (where the options are to: print or preview the trial balance again;
to re-attempt the rollover; or to cancel the rollover) the rollover is Cancelled, then no
harm is done. However if the Rollover is re-attempted at this point, the rollover will
continue with the corrupted Local Cache. This has the following consequences:
= There will be a Receipts and Payment mismatch corresponding to the value of
Discrepancies that were “lost”
Note that if the User doesn’t check their Final Balance Report carefully they may
be unaware of the issue since there is no explicit message when a Receipts and
Payment mismatch is found on the Final balance (the User is only prompted when
one is detected during a Trial balance)
= The Local Suspense will have no knowledge of this specific Discrepancy
m The Opening Figures for Discrepancies in the new Period will be zero rather than
the actual value of the Discrepancy
= The data used for the BTS will also have a zero value for Discrepancies at the end
of the period:” When the BTS is produced this will result in asimilar Receipts and ~~~
Payment mismatch
Note that if the bug was not present, then the Discrepancy would have been
transferred to Local Suspense and that would have been cleared, so there are a
number of things wrong with the BTS. However the impact of the bug is that the
discrepancy is lost and so the simplest way to correct it is to re-introduce the lost
discrepancy in a subsequent period and allow the normal rollover process to
correct it.
Note that if more than one SU has the issue then the value will be the total value of all errors.
m The level of Discrepancies when viewed at the Branch will no longer match the
level as seen in POL SAP or POL MIS
3. Identifying Affected Branches
~The Receipts and Payment mismatch will result in an NT event being generated.
These use Event id of 902 when detected during SU balancing and 903 when detected
during BTS production.
Processes should be in place such that SMC pick up these events and raise a peak for
each occurrence of these events.
1 don’t believe that this has happened and this needs to be investigated further.
Therefore a check of the Event archives is required to produce all occurrences of these
events from HNG-X.
Mark Wright has produced a list of 16 occurrences of event 903 in the last 30 days. This
needs to be extended
Also application event 116 or 117 should be written to the
BRDB_RX_REP_EVENT_DATA.
c:\documents and settings\jarnail.a.singh\local settings\temporary internet files\olk18\lost discrepancies
290910.doc Printed at 16:38:24 on 8/10/2010 Page 2 of 4
FUJ00083406
FUJ00083406
Looking at the BRDB when Dev reproduced the problem only 117 events were found. I need
to check what the difference is between 116 and 117.
Therefore an extract from BRSS of all instance of Events 116 and 117 will provide a
further check.
Please can SSC arrange to get extracts of the relevant NT and Application events asap
(before things get archived) so that we can get the scope of the problem.
4. Analysis Required for each Affected Branch
For each Branch need to ascertain the following:
m= When the Receipts Payments mismatch occurred
m What is the value of the Lost discrepancy
= Isita gain ora loss?
m Is there a corresponding Application Event? as
m= Affected SU, TP and BP
= Has a call been raised by the Branch?
= Has a call been raised by SMC?
= Has the Branch rolled over to a new TP?
5. Ongoing Monitoring
We need to ensure that SMC processes are changed such that Peaks are generated for
each occurrence of events 902 or 903.
As a backstop we should also ensure that a monthly check as described in Section 3 is
carried out to ensure that nothing has been forgotten. Note that this check shouldn’t
come up with any new branches if the processes have been put in place correctly.
6. Communication with Post Office Ltd
Once we have the information from Section 4 which will enable us to identify the full
scope of the issue we need to communicate this to Post Office Ltd through the
problem management mechanisms. We will then need to get Post Office Ltd to agree
if / how we should be correcting the data.
Post Office Ltd should also be able to check up on POL SAP to confirm that these
discrepancies are still visible even though they have been lost in the Branch.
It should be noted that as Discrepancies are normally Losses, then a Lost Discrepancy
would normally work in the Branches favour and so there is no incentive for the
Branch to report the problem. Also if we do amend the data to re-introduce the
Discrepancy, this will need to be carefully communicated to the Branches to avoid
questions about the system integrity.
c:\documents and settings\jarnail.a.singh\local settings\temporary internet files\olk1 8\lost discrepancies
290910.doc Printed at 16:38:24 on 8/10/2010 Page 3 of 4
7.
Of the cases so far identified there is one for £30,611.16, one for £4,826.00 and the
rest are all less than £350.
I've been unable to work out yet if these are losses or gains!
Fixing the Data for each Affected Branch
The data can be corrected by adjusting the appropriate Opening Figures and BTS Data
that relates to the current TP. This will result in the Discrepancy needing to be
processed when rolling over into the next TP.
I propose that if we are to do this then we take a copy of the data for one branch and
check out the proposed changes on a test system and then rollover the branch on the
test system to ensure that the discrepancy is handled correctly before we attempt to
correct Live data. Having done one example in this way, we then need to agree a
~timetable-with Post" Office Ltd-to-correct the other “branches~and- ensure: that ‘this “is
communicated with the Branches to ensure that everyone involved is happy.
Note that if it is decided not to correct the data in the branches (ie POL would prefer
to write off the “lost” discrepancy), then adjustments will be required to the
Discrepancy account in POL SAP to align this with the actual level of discrepancy
seen at the Branches.
c:\documents and settings\jarnail.a.singh\local settings\temporary internet files\olk18\lost discrepancies
290910.doc Printed at 16:38:24 on 8/10/2010 Page 4 of 4
FUJ00083406
FUJ00083406