POL00117863
POL00117863
I i Appendix 1 to Cs' Responsive Note
Receipts/Payments Mismatch issue notes} 2
: FUJITSU
Attendees
Antonio Jamasb (AJ) Service Delivery
Emma Langfield (EL) Service Delivery
- Alan Simpson (AS) Security I
Julia Marwood UM) Network
lan Trundell (IT) IT
Andrew Winn (AW) POL Finance I
Mike Stewart (MS) Fujitsu SOM I
John Simpkins (JS) Fujitsu Security
Gareth Jenkins (GJ) Fujitsu Technical Specialist
Mark Wright (MW) Fujitsu Téchnical Specialist
What is the issue? I
I I
Discrepancies showing at the Horizon counter disappear when the branch follows certain process steps, but will still show within the back end branch
account. This is currently impacting circa 40 Branches since migration onto Horizon Online, with an overall cash value of circa £20k loss. This issue will
only occur if a branch cancels the completion of the trading period, but within the same session continues to roll into a new balance period.
I
I
At this time we have not communicated with branches affected and we do not believe they are exploiting this bug intentionally
\
I I
The.problem occurs as part of the process when moving discrepancies on the Horizon System into Local Suspense.
I
When Discrepancies are found during Stock Unit rollover into a new Transaction Period, then the User is asked if the discrepancy should be moved to
Local Suspense. If the branch presses cancel at this point the Discrepancy is zeroed on the Horizon; System.
I
i
i
i
I
Page 1 of 5 he _
I COMMEDSIAI IN-CONEINENCE I
POL00117863
POL00117863
Receipts/Payments Mismatch issue notes, I
I FUsTSU
I
Note at this point nothing into feeds POLSAP and Credence, so in affect the POLSAP and Credence shows the discrepancy whereas the Horizon
system in the branch doesn’t. So the branch will then believe they have balanced. I
If at the next screen the rollover is completely cancelled, then no harm is done. However if the Rollover is re-attempted at this point, the rollover will
continue without any discrepancy meaning Horizon doesn’t Match POLSAP or Credence
This has the following consequences:
I I
‘= There will be a Receipts and Payment mismatch corresponding to the value of Discrepancies that were “lost”
Note the Branch will not get a prompt from the system to say there is Receipts and Payment mismatch, therefore the branch will believe they have
balanced correctly.. I
a When the Branch begins the new Branch Trading period the discrepancies will show at Zero,
I
I “ F
however the Receipts and Payment mismatch will carry
over into the next period. I I
I
I
I
j
I
Impact I
The branch has appeared to have balanced, whereas in fact they could have a loss or a gain.
Our accounting systems will be out of sync with what is recorded at the branch
If widely known could cause a loss of confidence in the Horizon System by branches I
Potential impact upon ongoing legal cases where branches are disputing the integrity of Horizon Data.
lt could provide branches ammunition to Blame Horizon for future discrepancies I
Identifying the issue and forward resolution I
i
bo
Page 2 of 5 “ “I
COMMEDSCIAI INCONTINENCE
POL00117863
POL00117863
I I
~ ~~ Receipts/Payments Mismatch issue notesI
I
I FUJITSU
I
The Receipts and Payment mismatch will result in an error code being generated which will altow Fujitsu to isolate branches affected this by this
problem, although this is not seen by the branches. We have asked Fujitsu why it has taken so long to react to and escalate an issue which began in
May. They will provide feedback in due course.
Fujitsu are writing a code fix which stop the discrepancy disappearing from Horizon in the future. iThey are aiming to deliver this into test week
commencing 4" October. With live proving at thel model office week commencing 11 October. With full roll out to the network completed by the 21* of
October. We have explored moving this forward and this is the earliest it can be released into live!
The code fix will on stop the issue occurring in the future, but it will not fix any current mismatch;at branch.
Proposal for affected Branches I
There are three potential solutions to apply to the Impacted branches, the groups recommendation is that solution two should be progressed.
I :
SOLUTION ONE - Alter the Horizon Branch figure at the counter to show the discrepancy. Fujitsu ould have to manually write an entry value to the
local branch account. poo ]
I
IMPACT - When the branch comes to complete next Trading Period they would have a discrepancy, which they would have to bring to account.
RISK- This has significant data integrity concerns and could lead to questions of “tampering” with the branch system and could generate questions
around how the discrepancy was caused. This solution could have moral implications of Post Office, changing branch data without informing the branch.
SOLUTION TWO - P&BA will journal values from he discrepancy account into the Customer Account and recover/refund via normal processes. This will
need to be supported by an approved POL cormut ication. Unlike the branch “POLSAP” remains in balance albeit with an account (discrepancies) that
should be cleared.
IMPACT - Post Office will be required to explain the reason for a debt recovery/ refund even though there is no discrepancy at the branch.
RISK - Could potentially highlight to branches that Horizon can lose data. I
\
SOLUTION THREE - It is decided not to correct the data in the branches (ie Post Office would prefer to write off the “lost”
IMPACT - Post office must absorb circa £20K loss I
RISK ~ Huge moral implications to the integrity of the business, as there are agents that were potentially due a cash gain on their system
I i
L- al
Page 3 of § [ve Ne
I COMMERCIAL IN-CANEINENCE i
POL00117863
POL00117863
I
Receipts/Payments Mismatch issue notes’
i}
FUJITSU
Action Point Summary
ee
Audit accounts
F Audit of Credence and BOLSAP. to check both systems match.
"I 8° October
“Tsolate branch issues
I
Provide list of Horizon Desk calls made by affected branches to see if any other issue
investigation
8" October ‘MS
impacting branches I
Forward Fix plan Provide timeline for permanent for fix introduction to branches 6® October MS’
Branch Performance I Confirm with Shaun Turner any future audits for Branches and any perforrnance issues 8" October JM
review flagged I
Branch Investigation I Confirm any existing of future investigations for impacted branches I 8" October AS
review I
NBSC Script Confirm NBSC scripts ii receipts and payments 6" October TJ
HSD Script Confirm Horizon System Desk script for receipts and payments I 6" October MS
; I
Branch weekly report I Fujitsu to provide a weekly report updating the affected Branches until migration of fix is 8" October GJ
completely rolled out I
Further Branch 6" October EL
!
Provide Fujitsu a list of frente to carry out further investigations
Page 4 of 5 r-
I
COMMEDCIAL IN-COMEINENCE
POL00117863
POL00117863
Receipts/Payments Mismatch issue notes!
investigations
to supply output! of further investigation:
i
8" October
' Engage Post Office Communication teams to understand what should b
Communications @ shared with 6" October TJ
: Branches i
Stakeholder Produce report for Post! Office Stakeholders 6" October TJ
Report .
Page 5 of 5
COMMEDCIAI -IN-CONEINENCE
POL00117863
POL00117863
Appendix 2 to Cs' Responsive Note
Correcting Accounts for “lost” Discrepancies
Ref: _g:\gij documents\notes\lost discrepancies.doc
Author: Gareth I Jenkins
Date: 29/09/2010 10:50:00
1. Introduction
This note relates to Peaks PC0204765 and PC0204263 (and also PC0203864 which is
a duplicate of PC0204263).
Are these really duplicates? I’m a bit confused as to which one to refer to. Can one be closed
as a duplicate of the other?
PC0204263 describes a problem with SU Balancing that will result in a Receipts
. payments mismatch. A fix is available for this peak which needs to be scheduled via
oN. RMF, However any biaiich éiiconiitering the probleni will have corrupted accounts
and Peak PC0204765 is a Master Peak to record al affected branches and also to
define the process for correcting the accounts. .
The purpose of this note is to:
= Summarise the problem in terms that are meaningful to Post Office Ltd
= Define a process for identifying all affected branches
= Explain what analysis is needed on each affected branch
«Define what ongoing monitoring is required to pick up further occurrences of the
issue until the root cause of the problem is fixed
m Provide a basis for agreeing the necessary data fixes with Post Office Ltd and how
they are to be applied
«Explain how each problem branch can be fixed
11 Change Control
Initial version 28/09/2010 12:49:00: This version of the note is an initial draft for
discussion within development.
Updated version 29/09/2010 10:50:00: Updated following feedback from Dev to be
distributed to SSC
2. . Overview
The problem occurs as part of the process of moving discrepancies into Local
Suspense.
When Discrepancies are found when rolling a SU over into a new TP, then the User is
asked if they should be moved to Local Suspense (MSG31316). Should they Cancel
at this point the Discrepancy is zeroised in the Local Cache (but nothing is written to
the BRDB). Note that there is no corresponding Balancing Transaction generated in
the Local Cache and so the Local Cache is in an Unbalanced state.
cA\documents and settings\jarnail.a.singh\local settings\temporary internet files\olk18\lost discrepancies
290910.doc Printed at 16:38:24 on 8/10/2010 Page 1 of 4
POL00117863
POL00117863
If at the next screen (where the options are to: print or preview the trial balance again;
to re-attempt the rollover; or to cancel the rollover) the rollover is Cancelled, then no
harm is done. However if the Rollover is re-attempted at this point, the rollover will
continue with the corrupted Local Cache. This has the following consequences:
«There will be a Receipts and Payment mismatch corresponding to the value of
Discrepancies that were “lost”
Note that if the User doesn’t check their Final Balance Report carefully they may
be unaware of the issue since there is no explicit message when a Receipts and
Payment mismatch is found on the Final balance (the User is only prompted when
one is detected during a Trial balance)
= The Local Suspense will have no knowledge of this specific Discrepancy
The Opening Figures for Discrepancies in the new Period will be zero rather than.
the actual value of the Discrepancy
«The data used for the BTS will also have a zero value for Discrepancies at the end
—_———-— of the period: When the BTS is produced this will result ina similar Receipts‘and———— ~~
Payment mismatch
Note that if the bug was not present, then the Discrepancy would have been
transferred to Local Suspense and that would have been cleared, so there are a I
number of things wrong with the BTS. However the impact of the bug is that the I
discrepancy is lost and so the simplest way to correct it is to re-introduce the lost
discrepancy in a subsequent period and allow the normal rollover process to
correct it.
Note that if more than one SU has the issue then the value will be the total value of all errors. ]
m The level of Discrepancies when viewed at the Branch will no longer match the I
level as seen in POL SAP or POL MIS I
3. Identifying Affected Branches
These use Event id of 902 when detected during SU balancing and 903 when-detected
during BTS production. I
Processes should be in place such that SMC pick up these events and raise a peak for
each occurrence of these events. .
I don’t believe that this has happened and this needs to be investigated further.
Therefore a check of the Event archives is required to produce all occurrences of these
events from HNG-X.
Mark Wright has produced a list of 16 occurrences of event 903 in the last 30 days. This
needs to be extended
Also application event 116 or .117 should be written to the
BRDB_RX_REP_EVENT_DATA.
c:\documents and settings\jarnail.a.singh\local settings\temporary internet files\olk18\lost discrepancies
290910.doe Printed at 16:38:24 on 8/10/2010 Page 2 of 4
POL00117863
POL00117863
Looking at the BRDB when Dev reproduced the problem only 117 events were found. I need /
to check what the difference is between 116 and 117.
Therefore an extract from BRSS of all instance of Events 116 and 117 will provide a
further check.
Please can SSC arrange to get-extracts of the relevant NT and Application events asap I
(before things get archived) so that we can get the scope of the problem.
4. Analysis Required for each Affected Branch
For each Branch need to ascertain the following: I
m When the Receipts Payments mismatch occurred
» What is the value of the Lost discrepancy
m Isita gain ora loss?
“w Is therea correspondiiig Application Event? “~~
u Affected SU, TP and BP
= Has a call been raised by the Branch?
m Has a call been raised by SMC? I
m Has the Branch rolled over to anew TP?
5. Ongoing Monitoring
We need to ensure that SMC processes are changed such that Peaks are generated for
each occurrence of-events 902 or 903.
As a-backstop we should also ensure that a monthly check as described in Section 3 is
carried out to ensure that nothing has been forgotten. Note that this check shouldn’t
come up with any new branches if the processes have been put in place correctly.
6. Communication with Post Office Ltd I
Once we have the information from Section 4 which will enable us to identify the full I
scope of the issue we need to communicate this to Post Office Ltd through the I
problem management mechanisms.. We will then need to get Post Office Ltd to agree
if / how we should be correcting the data.
Post Office Ltd should also be able to check up on POL SAP to confirm that these
discrepancies are still visible even though they have been lost in the Branch.
It should be noted that as Discrepancies are normally Losses, then a Lost Discrepancy
would normally work in the Branches favour and so there is no incentive for the
Branch to report the problem. Also if we do amend the data to re-introduce the
Discrepancy, this will need to be carefully communicated to the Branches to avoid
questions about the system integrity.
c:\documents and settings\jarnail.a.singh\local seitings\temporary internet files\olk18\lost discrepancies
290910.doc Printed at 16:38:24 on 8/10/2010 Page 3 of 4
POLO00117863
POL00117863
Of the cases so far identified there is one for £30,611.16, one for £4,826.00 and the
rest are all less than £350.
I’ve been unable to work out yet if these are losses or gains!
7. Fixing the Data for each Affected Branch
The data can be corrected by adjusting the appropriate Opening Figures and BTS Data I
that relates to the current TP. This will result in the Discrepancy needing to be
processed when rolling over into the next TP.
I propose that if we are to do this then we take a copy of the data for one branch and
check out the proposed changes on a test system and then rollover the branch on the I
test system to ensure that the discrepancy is handled correctly before we attempt to
: correct Live data. Having done one example in this way, we then need to agree a
I .-—-—timetable-with-P ost Office Ltd: to-correct-the- other-branches~and-ensure- that -this-is---- ~~
communicated with the Branches to ensure that everyone involved is happy.
Note that if it is decided not to correct the data in the branches (ie POL would prefer
to write off the “lost” discrepancy), then adjustments will be required to the
Discrepancy account in POL SAP to align this with the actual level of discrepancy
seen at the Branches.
c:\documents and settings\jamail.a.singh\local settings\temporary intemet files\olk18\lost discrepancies
290910,doc Printed at 16:38:24 on 8/10/2010 Page 4 of 4