WBONO000202
WBON0000202
pete.newsome(
"Dave. Ibbett,
Importance: Normal
Inline-Images: image001.png; image002.png; image003.png; imaged2829d.PNG; image6ef5c4.PNG;
image0542d6.PNG
Matthew,
An urgent question has arisen in relation to paragraph 30 of Steve's second statement. In that paragraph Steve states
that transactions were only injected into the counter "in the following circumstances while Mr Roll was employed by
Fujitsu (emphasis added):~
29.1 fixing a Riposte Index at the counter;
29.2 removing a historic message that was influencing the balancing process on a replaced counter;
29.3 correcting configuration data after a PinPad change;
29.4 removing redundant configuration items;
29.5 the example given above involving five corrupted bureau transactions; and
29.6 removing historic recovery information."
This is based on the content of row 6 in the table below. Steve's statement goes on to say that this only happened on
14 occasions and only one of those involved transaction data. The 14 occasions were: PC0060114 {POL-0234909},
PC0112293 {POL-0283845}, PC0112293 {POL-0283845}, PC0112397 {POL-0283948}, PC0112650 {POL-0284204},
PC0112659 {POL-0284213}, PC0118037 {POL-0289559},PC0122806 {POL-0293307}, PC0170799 {POL-0341013},
PC0175821 {POL-0345994}, PC0182141 {POL-0352240}, PC0198266 {POL-0368128}, PCO0201613 {POL-0371420},
PC0203896 {POL-0373686}. You can ignore the POL numbers.
It appears that the 14 occasions actually span the life of Legacy Horizon, rather than the period during which Roll was
employed. Is that right?
Please would you get back to me ASAP?
Kind regards
Jonny
Jonathan Gribben
Managing Associate
Womble Bond Dickinson (UK) LLP
WBD_000072.000001
WBONO000202
WBON0000202
‘Stay informed: sign up to our e-alerts
WOMBLE womblebonddickinson.com
BOND
DICKINSON A)
From: Matthew.Lenton: i[mailto:!.
Sent: 25 January 2019
To: Jonathan Gribben
Cc: SHenderson@~ } Lucy Bremner; ParkerSP¢ yl
Gareth Jenkins <7" GRO. I Andrew
Subject: RE: Roll 2 [WBDUK-AC.FID27032497]
Jonny,
Please see below an update which we believe completes the response to action 3 as the remaining 16
incidents referred to yesterday have now been analysed.
Additions in red are additional events not present in the data sent to you on 24-Jan-2019. Changes in text are
shown with strikethrough.
Matthew Lenton
Post Office Account Document Manager
P&PS, Digital Technology Services
Fujitsu
Lovelace Road, Bracknell, Berkshire, RG12 8SN
From: Lenton, Matthew
Sent: 24 January 2019 17:58
To:
Cc ‘Lucy Bremner’
i Parker, Steve { GRO. >; Ibbett, Dave
Newsome, Pet Gareth Jenkins
GRO. ; ‘Andrew Parsons’
‘AC.FID27032497]
Jonny,
Please see below, a response is now added for action 3, which we think is mostly complete but will update
further. No other changes to the table.
WBD_000072.000002
WBONO000202
WBON0000202
Matthew Lenton
Post Office Account Document Manager
P&PS, Digital Technology Services
Fujitsu
Lovelace Road, Bracknell, Berkshire, RG12 8SN
Web: hos wn fjsu.comyolobal
From: Lenton, Matthew
a 24 January 2019 13:31
GRO. +; Parker, Steve
; Newsome, Pete
Subject: RE: Roll 2 [WBDUK-AC, erect ie
Jonny,
Please see below revised table with responses added for actions 6 and 8.
Actions 3 and 9 are still being worked on, but an update is included in those rows.
ction] Paragraph I Action Assigned IFujitsu
of Roll 2 to
Provide a list of events that give rise to a receipts and payments mismatch ae /
[Matthew
Because of the volume of data here (735 incidents) and the need to [Lenton]
eyeball each one we’re restricted the initial analysis to the 390 calls [Response
opened between 1999 and Jan 2002 (inc.). After this the beat rate provided
significantly decreased (only 345 in the subsequent 8 years). This is lat left.
believed to be due to the version M1 rollout (summer 2001) which
appears to have significantly increased the reliability in this area. I\Updated
25-Jan-
2019
Analysis
WBD_000072.000003
Category Calls Residue Comment
Orange Prepay 99 291
Tssue
Newly migrated 61 230
offices (paper to
PC)
Erroneous 39 191
settlement of
Transfer Out and
Transfer In
transactions to
Cash
(KEL DRoweS0K)
‘Balancing Error: 14 177
Receipts and
payments do not
match, please
investigate. The
error may be
corrected using
Reversal Function.
WARNING:
Continuing may
lead to an
unbalanced Cash
Account"
(KEL
DRowe1625K)
Event
Jan 2002. Software
Reference data I error
/ software
issue.
Urgent
software fix
applied within
a week.
Oct 1999 —
Nov 2001.
Migration
Hot spots July
— Sep 2000,
March 2001.
Migration
figures
accepted
inevitably lead
to R&P issue.
No software
fault.
April 2001 - Software
June 2001. error
Corrected cash
accounts
provided to
Post Office
Networks
(PON).
Counter
software fix @
release M1,
which rolled
out from May
2001.
March 2001 — I Software
July 2001. error
Reconciliation
data has been
provided to
PON (suspect
this was
corrected cash
accounts).
Counter
software fix @
release M1,
which rolled
out from May
2001.
WBONO000202
WBON0000202
WBD_000072.000004
Stock unit being 8 169
rolled over twice
before the Cash
Account is rolled.
(KEL
LKiang1222L,
GMaxwelll59r)
Single Counter 17 152
Outlet (SCO) was
replaced, without
synchronising the
messagestore.
(KEL
JBallantyne5328R)
Software fixes. 2F Bs
May be related to I 29 123
above KELs, or
other issues.
March 2001 — I Software
May 2001. error
Corrected cash
accounts
provided to
PON.
Counter
software fix @
release M1,
which rolled
out from May
2001.
November Software
2000 — error
November
2001.
Reconciliation
Data provided
to PON.
Mismatch
between
receipts and
payments is
due to a self
originated
message which
overwrote a
transaction on
the counter
messagestore.
MSU noted in
Nov 2001:
This type of R
& P incident is
the only one
we still get
regularly. Is
there anything
that can be/is
being done to
fix it?
Software fix @
release BI2.
April 2000 — Software
December error
2001.
12 @CI4.
10@MI.
S@ other:
7 @ other.
WBONO000202
WBON0000202
WBD_000072.000005
Reference data. 13
Either rollout
timetable not
followed, resulting
in unavailable
local products such
as OBCS, or
products ending
and stock
remaining.
Duplicate incidents I 34
within the set
being analysed
e.g. branch reports
the same issue
flagged on
Fujitsu’s host cash
account report, or
vice versa
Reconciliation 41
resolved.
May be related to
above KELs, or
other issues.
No fault, not R&P I 11
Peaks, ete
Temp Closed 5
offices
Hardware swaps 5
110
76
35
24
19
Reference
Data Error
July 2000 —
December
2001.
OBCS
products will
have become
available, later
than expected.
Admin
Ignore
September
2000 —
December
2001.
August 2000 — I Unknown
December
2001.
Identified by
data centre
reporting.
Root cause
cannot be
determined
from Peak
Information
provided to
POL to give
correct view of
accounts
September Admin
2000 — January I Ingore
2002
May 2001 — POL
January 2002 _ Process
Error
Correct outlet
close process
not followed.
Information
archived (e.g.
Balance
brought
forward) by
system.
Faby 2000—
October 200+
Engineering
process
error
WBONO000202
WBON0000202
WBD_000072.000006
May 2000 —
November
2001
User 4 6
August 2000 —
July 2001
A&G for PM
or Trainers,
which
sometimes
wasn’t
followed
(PC0065358).
PM ignoring
on screen
messages
(PC0053164).
One call where
PM accepted
shortage, then
acall was
raised
(PC0067250),
possibly
indicating lack
of
understanding.
Another call
(PC0068191)
reads like lack
of PM
understanding
of the Cash
Account.
Training
Unclear 6 0
July 2000 —
June 2001
Insufficient
evidence to
comment.
Unclear
For 2002, 101 of the 124 calls raised that year were opened in
January. 99 of those were for the Orange Prepay issue. Only I call
was opened in February.
Provide a list of reasons for which transaction data would need to be
injected at the counter.
Issue with Riposte index at counter
Potential financial
WBONO000202
WBON0000202
WBD_000072.000007
Last historic message stored at counter
was incorrectly being considered as part
of a balancing process
Config data relating to PinPad needs to
be deleted if PinPad is removed from.
counter. AKA PinPad LPO delete.
Old configuration objects local to
counter needed to be removed.
LPO Delete.
Five corrupted bureau transactions on
counter
PM left AP recovery for too long.
Usually same / next day not months. Ref
data for product referenced in AP
recovery removed. Impossible for PM to
complete recovery. Objects deleted. LPO
delete
* LPO=Local Persi:
impact because the
wrong value or quantity
was being used for a
product
No financial impact. PM
recognised that data
presented was too old.
No financial impact
No Financial impact
Financial impact
(PC0175821)
Changes approved by
POL
Documented on BIMS
Possible but unlikely
financial impact due to
age of recovery
information.
nt object. Configuration object used by the
Riposte system. By its nature, requires intervention at counter.
Note: Last case (RiposteObject command) still being worked on.
This relates to configuration information (similar to LPO above) and
will not have any financial impact so is for completeness only.
Method
We searched the following databases to try and identify the incidents
for which transaction data has been inserted at the counter:
KEL: Known Error Log
OCP: Operational Change Processes OCR / OCP
Peak: Incident management system
System — Search Keywords
KEL RiposteMessageFile
KEL LPO Delete
WBONO000202
WBON0000202
rom the
sampling
referred to
below at 9?
(Matthew
\Lenton]
IResponse
provided
jat left, 24-
lan-2019.
WBD_000072.000008
KEL Marooned
OcP RiposteMessageFile
ocP LPO Delete
ocP Marooned
OcP RiposteObject put
Peak RiposteMessageFileRiposteMessage
Peak LPO Delete
Peak JBallant498)
Peak MYoung5043M
Peak Marooned
RiposteObject put
Did: (1) Belfast team; and (2) privileged users have the ability to inject
transaction data between 2001 and 2004? Do they have that ability now?
Gareth Jenkins: With Horizon Online, there is the Transaction
Correction Tool which can inject transactions and this is controlled by
SSC. It is audited when it runs and we have only used it once in
March 2010. The DBAs in Belfast can in theory do anything to the
BRDB. In practice they will run scripts tested by dev as part of a
systems upgrade if DB changes are required. Any such access is
audited and since 2015 the actual commands run are also audited.
With old Horizon, control was weaker. SSC could inject into
Correspondence Servers and also at the counter.
Belfast team: Belfast had administrative access to the
correspondence servers and had a theoretical ability to inject data
into the messagestores, but dont believe that they had the technical
understanding to do so.
Belfast had no access to counters, UNIX/NT team having no users and
no knowledge of administrative user accounts/passwords.
They would not have injected any data unless it happened to be done
by scripts that they were asked to run and which were provided under
change control. Direct manipulation of the messagestore wasn t
something that they knew how to do and would not have attempted to
do lest it break the running applications which harvested/inserted
data. Their understanding of the actual messages was very low/non-
existent so would have had no confidence in making any insertion.
WBONO000202
WBON0000202
[Matthew
[Lenton]
[Response
provided
lat left, 24-
Van-2019.
21
Review a sample of OCPs to give an indication as to how frequently
transaction data was injected.
Fujitsu _‘ISteve/
ISSC —
relates to 6
jabove?
WBD_000072.000009
[Matthew Lenton] This is proving difficult to provide. The original
plan was to examine sample months of change control data and
produce rough figures. As Pete Newsome already discussed with you,
this lead to it becoming apparent that support did not use formal
change control in the earlier years for BAU support actions. We relied
on the audit trail within the incidents (Peaks) to document support
actions. We had auditability of the work done but no change control
entries. We assume that the reasoning behind this was to allow
implementation of support actions ASAP, and the audit trail being
good enough where there was no financial impact.
Therefore we are still looking at how / if we can provide an accurate
answer to this question for the earlier years.
Matthew Lenton
Post Office Account Document Manager
P&PS, Digital Technology Services
Fujitsu
Lovelace Road, Bracknell, Berkshire, RG12 8SN
Web: https://www.fujitsu.com/global/
WBONO000202
WBON0000202
[Matthew
[Lenton]
\Update at
eft
From: Jonathan Gribben [mailto?
Sent: 24 January 2019 09:56
To: Lenton, Matthew } GRO i
Ce: SHenderson( GRO : Lucy Bremner + GRO >; Parker, Steve
{GRO}: Ibbett, Dave ¢ GRO >: Newsome, Pete
t GRO p; Gareth Jenkins + GRO
>; Andrew Parsons {
‘Subject: RE: Roll 2 [WBDUK-AC.FID27032497
Importance: High
Matthew,
Please would you provide an update in relation to the below this morning?
Kind regards
WBD_000072.000010
WBONO000202
WBON0000202
Jonny
Jonathan Gribben
Managing Associate
Womble Bond Dickinson (UK) LLP
‘Stay informed: sign up to our e-alerts
WOMBLE womblebonddickinson.com
BOND
DICKINSON + Ain)
From: Matthew.Lenton;
Sent: 22 January 2019 15:46
To: Jonathan Gribben
; Lucy Bremner; ParkerSP¢
Gareth Jenkins <{"~
Subject: RE: Roll 2 [WBDUK-AC.FID27032497]
Jonny,
I’ve numbered the actions 1 — 11 below, and added the responses so far to actions I and 11 in the Actions
column, and some notes on progress etc. to the Fujitsu column.
Matthew Lenton
Post Office Account Document Manager
P&PS, Digital Technology Services
Fujitsu
Lovelace Road, Bracknell, Berkshire, RG12 8SN
Web: https:/www.fujitsu.comi/global/
From: Jonathan Gribben [mailto}, GRO }
Sent: 21 January 20. ~ -
To: Parker, Steve < GRO }; Ibbett, Dave I i>; Newsome, Pete
>; Lenton, Matthew <. GRO. +; Gareth Jenkins
GRO. b
Ce: Simon Henderson } GRO >; Lucy Bremner
WBD_000072.000011
WBONO000202
WBON0000202
I GRO,
Subjec RE: Roll 2 [WBDUK-AC.FID27032497]
Dear all,
Privileged & Confidential
Thank you for your time earlier. Here's a list of the actions that I captured from today's calls. Please let me know if
there's anything you'd like to add or change:-
Action] Paragraph I Action Assigned [Fujitsu
of Roll 2 to
1 8 Keyword search for incidents containing the words I Fujitsu [Steve / SSC
"laptop" and/or "luggable" and/or "outreach" etc.
[Matthew Lenton]
The Peaks referenced below Details of Peaks
provided at left.
PC0100174 March Ist 2004 to Sth March
2004
FAD317309 reporting: Horizon Kit rebooting
itself for no apparent reason.
Helpdesk user: "Over the past 2-3 weeks
engineers have been to site and have replaced
2xBU's and 2xPSU's but the problem
persists."
BU = base units = PC itself. PSU = Power
supply units within the base units
RR "Evidence (from event logs) shows that
the power is being switched off every morning
shortly (ie 5 or 6 minutes) before the PM logs
ont
RR: "After carrying out tests on our rigs, I
have been able to duplicate the problem here
on ONE of our rigs but not on others. It
seems that the Screen Power Button is
incorrectly connected to the motherboard."
RR: "We have now identified two instances of
this, one in live. This is a hardware build
quality issue."
This was followed by:
PC0100899 18th March 2004 to 24th March
2004.
WBD_000072.000012
Hardware returned from site to Bracknell for
examination.
RR: "Tests carried out on screen power
switch - working correctly, no further action
required."
Your questions 2deig
d) Is his example true, or could it have been
true: Yes. Can find no data on the origin for the
statement: "This is a hardware build quality
issue". Could be a discussion with engineering
which was not recorded on the incident progress.
Information we have only describes the hardware
issue being seen internally to FJ on one instance
of test rig hardware. No hardware error proved
on the site.
If so, how often did that sort of problem occur:
Very rare. Only one other found using keywords
"standby", "laptop", "luggable". PCO0S5550 which
was a problem on prototype hardware going into
standby mode.
What would have caused it: Inconclusive. No
information on root cause of issue reported by
the Post Master onsite. Could be a hardware
problem, could be user miss-operation of
hardware.
Could it have affected/did it affect branch
accounts: No. Once powered on the unit would
function as normal.
If so, might its effect on branch accounts never
have been detected with the result that some
SPMs might have been wrongly held liable for
false deficits: No
e) Would Rolls have disassembled laptops and
done the other things he describes in para 8:
Have to assume he did as per the incident
updates. I expect he had some assistance
(especially with kit on test rigs - different team
totally) but unable to substantiate.
Would he have had/did he have the
conversation with his manager he describes in
para 8: Just can’t answer this. My analysis of the
issue would suggest that it turned out to be
unimportant because there was no proof that this
ever happened in the live estate and that his
comment of "This is a hardware build quality
issue" is simply conjecture. However, he may
have discussed with engineering and truly
discovered a batch of faulty hardware. I would
have expected an update in the incident reading
"Discussed with xxxxxxxx in engineering and we
determined that........ Bad batch...... etc" No such
updates are present.
f) Was the problem referred to in para kept
secret, as claimed at the end of para 8: No
evidence either way. I would not expect that to
WBONO000202
WBON0000202
WBD_000072.000013
be the case. It is not in Fujitsu's interest to have
faulty equipment that is not corrected damaging
reputation.
g) Would Fujitsu management have known/did it
know about this problem? Would/did Post
Office? If not, why not: No way of knowing.
Information no longer exists
WBONO000202
WBON0000202
= 8 Check what the experts and witnesses say about WBD
KEL psteed2847n.
3 9 Provide a list of events that give rise to a receipts Fujitsu Steve / SSC
and payments mismatch.
Examples only, or all
scenarios that caused
hem in reality?
[Matthew Lenton] May
take rest of this week orI
lnore. Requires eyeball
searching.
4 12 Did Post Office review TC volumes in order to WBD to pick
identify potential software issues. up with POL
5 16 Review the contract between POL and Fujitsuand I WBD
summarise SLAs/penalties.
6 20 Provide a list of reasons for which transaction data I Fujitsu ISteve / SSC
would need to be injected at the counter.
[Can this be ascertained
rom the sampling referredI
fo below at 21?
(Matthew Lenton] SSC
forming a query to find
this from OCP data,
lalso determining when
transaction would be
injected at the counter.
7 20 Review Peak reference 107043 (example of WBD
transaction being injected into counter).
8 21 Did: (1) Belfast team; and (2) privileged users have I Fujitsu IGareth: answer 1 and 2
the ability to inject transaction data between 2001 jand perhaps explain again
and 2004? Do they have that ability now? difference between old
land new?
9 21 Review a sample of OCPs to give an indication as Fujitsu ISteve / SSC — relates to
to how frequently transaction data was injected. 120 above?
(Matthew Lenton] See
action 6 above
10 I 22 Search for documents relating to the controls WBD
around transaction data being injected
(DE/HLD/002 is an example).
11 ‘I General Provide details of Fujitsu's document storage Fujitsu IMatthew
practices and retention policies. Are emails, word
documents etc. from 2001 — 2004 available? (Matthew Lenton] See
answer at left
[Matthew Lenton] Emails cannot be retrieved
from the accounts of former Fujitsu
employees from that period, and back ups are
WBD_000072.000014
not held for that period of time. The only
records of such a person’s emails would be if
they are part of a current employee’s email
account or pst archive, in which case it would
be only the subset of their emails that were to
or from the other user. Similarly, for other
documentation that was held locally be
individual employees on their laptops, that
would have been deleted when the user left.
As we have already seen, some limited
information from this period does exist, stored
in Dimensions and other networked
repositories, some of which we have already
provided in connection with this case.
WBONO000202
WBON0000202
We are aiming to get a draft response to Roll 2 into circulation by early tomorrow afternoon.
Kind regards
Jonny
Jonathan Gribben
Managing Associate
Womble Bond Dickinson (UK) LLP
Stay informed: sign up to our e-alerts
WOMBLE
BOND
DICKINSON
From: Jonathan Gribben
Sent: 21 January 2019 11:05
To: 'ParkerSP@ GRO } Dave. Ibbett¢
Matthew.Lenton@.... j
Cc: Simon Henderson:_
Subject: Roll 2 [WBDU
Privileged & Confidenti
To discuss
Jonny
Lucy Bremner
WBD_000072.000015
WBONO000202
WBON0000202
Please consider the environment! Do you need to print this email?
tify jonathan, gribbent”
achments is prohibit nlawful. Information about how we use
Unautho
personal data is in our Privacy Policy on our website,
Any fil
Womble Bond Dickinson (UK) LLP accepts no liability for
fore opening any attachment.
Content of this email which does not relate to the official business of Womble Bond Dickinson (UK) LLP, is neither given nor endorsed by it,
on (UK) LLP whi Our registered
of the LLP, or an
sd in England and Wales under numb
rection, We use the term partner to refer
limited liability partnership re
of
1w firms providing
e acts or omissions of,
member of Womble Bond Dickinson (International) Limited, which consists of independent and autonomous
found the world, Each Womble Bond Dickinson entity is a separa
Womble Bond Dickinson (International) Limited does not practice law. Ple:
Womble Bond Dickinson (UK) LL
services in the US, K
ry and is not responsi
Womble Bond Dickinson (UK) LLP is authorised and regulated by the Solicitors Regulation Authority
Unless otherwise stated, this email has been sent from Fujitsu Services Limited (registered in England No
96056); Fujitsu EMEA PLC (registered in England No 2216100) both with registered offices at: 22 Baker
Street, London W1U 3BW; PFU (EMEA) Limited, (registered in England No 1578652) and Fujitsu
Laboratories of Europe Limited (registered in England No. 4153469) both with registered offices at: Hayes
Park Central, Hayes End Road, Hayes, Middlesex, UB4 8FE.
This email is only for the use of its intended recipient. Its contents are subject to a duty of confidence and
may be privileged. Fujitsu does not guarantee that this email has not been intercepted and amended or that it
is virus-free.
Unless otherwise stated, this email has been sent from Fujitsu Services Limited (registered in England No
96056); Fujitsu EMEA PLC (registered in England No 2216100) both with registered offices at: 22 Baker
Street, London W1U 3BW; PFU (EMEA) Limited, (registered in England No 1578652) and Fujitsu
Laboratories of Europe Limited (registered in England No. 4153469) both with registered offices at: Hayes
Park Central, Hayes End Road, Hayes, Middlesex, UB4 8FE.
This email is only for the use of its intended recipient. Its contents are subject to a duty of confidence and
may be privileged. Fujitsu does not guarantee that this email has not been intercepted and amended or that it
is virus-free.
WBD_000072.000016