FUJ00163098
FUJ00163098
From:
Sent:
To:
Ce:
Subject: FW: Roll 2 [WBDUK-AC.FID123822914]
Jonny,
I’ve added in the open and close dates of each of the Peaks in the email below; R Roll was employed 05-Mar-2001 to
17-Sep-2004, therefore only the first Peak (highlighted) has any overlap with his period of employment. From what I
can understand of that Peak however, it looks like the actual change being referred to was done on 11-Jan-2001, the
rest of the call is about trying to obtain an events report, but in the end it appears that they give up as the issue has
been fixed.
Matthew Lenton
Post Office Account Document Manager
Business & Application Services
Fujitsu
From: Jonathan Gribben ¢
Sent: Thursday, March 7, 2019 11:08 AM
To: Lenton, Matthew 7. ~
Cc: Newsome, Pete
bbett, Dave >; Andrew Parsons
; Lucy Bremner <!
Subject: FW: Roll 2 [WBDUK-AC.FID123822914]
Matthew,
An urgent question has arisen in relation to paragraph 30 of Steve's second statement. In that paragraph Steve states
that transactions were only injected into the counter "in the following circumstances while Mr Roll was employed by
Fujitsu (emphasis added):-
29.1 fixing a Riposte Index at the counter;
29.2 removing a historic message that was influencing the balancing process on a replaced counter;
29.3 correcting configuration data after a PinPad change;
29.4 removing redundant configuration items;
29.5 the example given above involving five corrupted bureau transactions; and
29.6 removing historic recovery information."
This is based on the content of row 6 in the table below. Steve's statement goes on to say that this only happened on
14 occasions and only one of those involved transaction data. The 14 occasions were:
PC0112293 {POL-0283845}, [Lenton, Matthew] 09-Dec-2004 - 14-Dec-2004
PC0112293 {POL-0283845}, [Lenton, Matthew] Duplicate of above
PC0112397 {POL-0283948}, [Lenton, Matthew] 13-Dec-2004 - 13-Dec-2004
PC0112650 {POL-0284204}, [Lenton, Matthew] 17-Dec-2004 - 20-Dec-2004
PC0112659 {POL-0284213}, [Lenton, Matthew] 17-Dec-2004 - 20-Dec-2004
PC0118037 {POL-0289559},/Lenton, Matthew] 24-Mar-2005 - 24-Mar-2005
PC0122806 {POL-0293307}, [Lenton, Matthew] 05-Jul-2005 - 05-Jul-2005
PC0170799 {POL-0341013}, [Lenton, Matthew] 03-Dec-2008 - 05-Dec-2008
FUJ00163098
FUJ00163098
PC0175821 {POL-0345994}, [Lenton, Matthew] 19-Feb-2009 - 20-Mar-2009
PC0182141 {POL-0352240}, [Lenton, Matthew] 02-Jun-2009 - 04-Jun-2009
PC0198266 {POL-0368128}, [Lenton, Matthew] 28-Apr-2010 - 16-Jul-2010
PC0201613 {POL-0371420}, [Lenton, Matthew] 15-Jul-2010 - 29-Jul-2010
PC0203896 {POL-0373686}. [Lenton, Matthew] 03-Sep-2010 - 07-Sep-2010
It appears that the 14 occasions actually span the life of Legacy Horizon, rather than the period during which Roll was
employed. Is that right?
Please would you get back to me ASAP?
Kind regards
Jonny
Jonathan Gribben
Managing Associate
Womble Bond Dickinson (UK) LLP
womblebonddickinson.com
WOMBLE
BOND
DICKINSON y ©
From: Matthew.Lenton{_
Sent: 25 January 2019 13:02
To: Jonathan Gribben
Cc: SHendersoné
ete.newsomel
Parsons
Subject: RE: Roll 2 [WBDUK-AC.FID27032497]
I
3 Andrew
Gareth Jenkins <.
Jonny,
Please see below an update which we believe completes the response to action 3 as the remaining 16 incidents
referred to yesterday have now been analysed.
Additions in red are additional events not present in the data sent to you on 24-Jan-2019, Changes in text are shown
with strikethrough.
Matthew Lenton
Post Office Account Document Manager
P&PS, Digital Technology Services
Fujitsu
RG12 SN
From: Lenton, Matthew
FUJ00163098
FUJ00163098
Sent: 24 January 2019 17:58
To: ‘Jonathan Gribben' < GRO i
"SHendersOnt. ened GRO i ‘Lucy Bremner’
">; Parker, Steve GRO. >; Ibbett, Dave 4 GRO $; Newsome, Pete
‘Gareth Jenkins < GRO. iy
GRO rew Parsons’ <_
Subject RE: Roll 2 [WBDUK-AC.FID27032497]
Jonny,
Please see below, a response is now added for action 3, which we think is mostly complete but will update further.
No other changes to the table.
Matthew Lenton
Post Office Account Document Manager
P&PS, Digital Technology Services
Fujitsu
From: Lenton, Matthew
Sent: 24 January 2019 1
To: ‘Jonathan Gribben'
Cc: SHendersoné.. GRO Lucy ; Parker, Steve
; Ibbett, Dave ; Newsome, Pete
GRO. }) Gareth Jenkins < GRO }
‘>; Andrew Parsons
Subject: RE: Roll 2 [WBDUK-AC.FID27032497]
Jonny,
Please see below revised table with responses added for actions 6 and 8.
Actions 3 and 9 are still being worked on, but an update is included in those rows.
ction} Paragraph I Action Assigned [Fujitsu
of Roll 2 to
Provide a list of events that give rise to a receipts and payments
mismatch Matthew
Lenton]
Because of the volume of data here (735 incidents) and the need to esponse
eyeball each one we’re restricted the initial analysis to the 390 calls rovided at
opened between 1999 and Jan 2002 (inc.). After this the beat rate left.
significantly decreased (only 345 in the subsequent 8 years). This is lUpdated 25-
believed to be due to the version M1 rollout (summer 2001) which lan-2019
appears to have significantly increased the reliability in this area.
FUJ00163098
FUJ00163098
Analysis
Category
Orange Prepay
Issue
Newly migrated
offices (paper to
Pc)
Erroneous
settlement of
Transfer Out and
Transfer In
transactions to
Cash
(KEL DRowe50K)
‘Balancing Error:
Receipts and
payments do not
match, please
investigate. The
error may be
corrected using
Reversal Function.
WARNING:
Continuing may
lead to an
unbalanced Cash
Account"
99
Calls
61
39
ve
Residue Comment Event
291 Jan 2002. Software
Reference data — error
/ software
issue.
Urgent software
fix applied
within a week.
230 Oct 1999-Nov Migration
2001.
Hot spots July —
Sep 2000,
March 2001.
Migration
figures accepted
inevitably lead
to R&P issue.
No software
fault.
191 April 2001 - Software
June 2001. error
Corrected cash
accounts
provided to Post
Office Networks
(PON).
Counter
software fix @
release M1,
which rolled out
from May 2001.
v7 March 2001- Software
July 2001. error
Reconciliation
data has been
provided to
PON (suspect
this was
corrected cash
accounts).
Counter
software fix @
release M1,
which rolled out
FUJ00163098
FUJ00163098
(KEL DRowe1625K) I from May 2001.
Stock unit being 8 169 March 2001 — Software
rolled over twice May 2001. error
before the Cash Corrected cash
Account is rolled. accounts
(KEL LKiang1222L, provided to
GMaxwell159r) PON.
Counter
software fix @
release M1,
which rolled out
from May 2001.
Single Counter 17 152 November 2000 Software
Outlet (SCO) was — November error
replaced, without 2001.
synchronising the Reconciliation
messagestore. Data provided
(KEL to PON.
JBallantyne5328R) Mismatch
between
receipts and
payments is due
toaself
originated
message which
overwrote a
transaction on
the counter
messagestore.
MSU noted in
Nov 2001: This
type of R&P
incident is the
only one we still
get regularly. Is
there anything
that can be/is
being done to
fix it?
Software fix @
release BI2.
Software fixes. 27 2s April 2000 — Software
May be relatedto 29 123 December 2001. error
above KELs, or 12 @Cl4.
other issues. 10 @ M1.
5-@-other
7 @ other.
Reference data. 13
Either rollout
timetable not
followed, resulting
in unavailable
local products such
as OBCS, or
products ending
and stock
remaining.
Reconciliation 41
resolved.
May be related to
above KELs, or
other issues.
No fault, not R&P 11
Peaks, etc
Temp Closed 5
offices
110
76
35
24
19
May 2001 —
September 2000 Admin
— December Ignore
2001.
August 2000 — Unknown
December 2001.
Identified by
data centre
reporting.
Root cause
cannot be
determined
from Peak
Information
provided to POL
to give correct
view of
accounts
September 2000 Admin
—January 2002 Ingore
POL Process
January 2002 Error
Correct outlet
close process
not followed.
Information
archived (e.g.
Balance brought
forward) by
system.
FUJ00163098
FUJ00163098
FUJ00163098
FUJ00163098
Hardware swaps = 5
10
duly 2000—
October-2002
May 2000 —
November 2001
August 2000 —
July 2001
A&G for PM or
Trainers, which
sometimes
wasn’t followed
(PCO065358).
PM ignoring on
screen
messages
(PC0053164).
One call where
PM accepted
shortage, then a
call was raised
(PC0067250),
possibly
indicating lack
of
understanding.
Another call
(PC0068191)
reads like lack
of PM
understanding
of the Cash
Account.
Training
Unclear 6
July 2000 — June
2001
Insufficient
evidence to
comment.
Unclear
For 2002, 101 of the 124 calls raised that year were opened in
January. 99 of those were for the Orange Prepay issue. Only 1 call
was opened in February.
FUJ00163098
FUJ00163098
20 Provide a list of reasons for which transaction data would need to be Steve / SSC
injected at the counter. 7 (Can this be
Issue with Riposte index at counter Potential financial pecertained
impact because the sampling
wrong value or quantity referred to
was being used for a below at 9?
product ‘Matthew
Lenton]
IResponse
Last historic message stored at counter No financial impact. PM provided at
was incorrectly being considered as part I recognised that data left, 24-Jan-
of a balancing process presented was too old. 12019.
Config data relating to PinPad needs to No financial impact
be deleted if PinPad is removed from
counter. AKA PinPad LPO delete.
Old configuration objects local to counter I No Financial impact
needed to be removed.
LPO Delete.
Five corrupted bureau transactions on Financial impact
counter (PC0175821)
Changes approved by
POL
Documented on BIMS
PM left AP recovery for too long. Usually I Possible but unlikely
same / next day not months. Ref data for financial impact due to
product referenced in AP recovery age of recovery
removed. Impossible for PM to complete I information.
recovery. Objects deleted. LPO delete
* LPO=Local Persistent object. Configuration object used by the
Riposte system. By its nature, requires intervention at counter.
Note: Last case (RiposteObject command) still being worked on. This
relates to configuration information (similar to LPO above) and will
not have any financial impact so is for completeness only.
Method
We searched the following databases to try and identify the incidents
for which transaction data has been inserted at the counter:
KEL: Known Error Log
OCP: Operational Change Processes OCR / OCP
Peak: Incident management system
I Search Keywords
FUJ00163098
FUJ00163098
21
KEL RiposteMessageFile
KEL LPO Delete
KEL Marooned
ocP RiposteMessageFile
ocp LPO Delete
ocP Marooned
ocP RiposteObject put
Peak RiposteMessageFileRiposteMessage
I
Peak LPO Delete
Peak JBallant498)
Peak MYoung5043M
Peak Marooned
Peak RiposteObject put
Did: (1) Belfast team; and (2) privileged users have the ability to inject
transaction data between 2001 and 2004? Do they have that ability
now?
Gareth Jenkins: With Horizon Online, there is the Transaction
Correction Tool which can inject transactions and this is controlled by
SSC. It is audited when it runs and we have only used it once in March
2010. The DBAs in Belfast can in theory do anything to the BRDB. In
practice they will run scripts tested by dev as part of a systems
upgrade if DB changes are required. Any such access is audited and
since 2015 the actual commands run are also audited.
With old Horizon, control was weaker. SSC could inject into
Correspondence Servers and also at the counter.
(Matthew
Lenton]
[Response
provided at
left, 24-Jan-
2019.
FUJ00163098
FUJ00163098
Belfast team: Belfast had administrative access to the
correspondence servers and had a theoretical ability to inject data
into the messagestores, but don’t believe that they had the technical
understanding to do so.
Belfast had no access to counters, UNIX/NT team having no users and
no knowledge of administrative user accounts/passwords.
They would not have injected any data unless it happened to be done
by scripts that they were asked to run and which were provided under
change control. Direct manipulation of the messagestore wasn’t
something that they knew how to do and would not have attempted
to do lest it break the running applications which harvested/inserted
data. Their understanding of the actual messages was very low/non-
existent so would have had no confidence in making any insertion.
Review a sample of OCPs to give an indication as to how frequently
transaction data was injected.
[Matthew Lenton] This is proving difficult to provide. The original
plan was to examine sample months of change control data and
produce rough figures. As Pete Newsome already discussed with you,
this lead to it becoming apparent that support did not use formal
change control in the earlier years for BAU support actions. We relied
on the audit trail within the incidents (Peaks) to document support
actions. We had auditability of the work done but no change control
entries. We assume that the reasoning behind this was to allow
implementation of support actions ASAP, and the audit trail being
good enough where there was no financial impact.
Therefore we are still looking at how / if we can provide an accurate
answer to this question for the earlier years.
Steve / SSC
+ relates to
6 above?
[Matthew
Lenton]
\Update at
left.
Matthew Lenton
Post Office Account Document Manager
P&PS, Digital Technology Services
Fujitsu
Wet
https://www.fujitsu.com/global/
From: Jonathan Gribben} GRO. j
Sent: 24 January 2019 09:56
To: Lenton, Matthew ¢. GRO
Ce: SHenderso:
__» Lucy Bremner
Parker, Steve
FUJ00163098
FUJ00163098
bbett, Dave Newsome, Pete
ndrew Parsons <
Subject: RE: Roll 2 [WBDUK-AC.FID27032497]
Importance: High
Matthew,
Please would you provide an update in relation to the below this morning?
Kind regards
Jonny
Jonathan Gribben
Managing Associate
Womble Bond Dickinson (UK) LLP
Stay informed: sign up to our e-alerts
womblebonddickinson.com
WOMBLE
/ BOND
DICKINSON
From: Matthew.Lenton! i
Sent: 22 January 2019 15:46
To: Jonathan Gribben
Cc: SHenderson: Lucy Bremner; ParkerSP@ GRO
pete.newsom “i, Gareth Jenkin:
Subject: RE: Roll 2 [WBDUK-AC.FID27032497]
Jonny,
I’ve numbered the actions 1 — 11 below, and added the responses so far to actions 1 and 11 in the Actions column, and
some notes on progress etc. to the Fujitsu column.
Matthew Lenton
Post Office Account Document Manager
P&PS, Digital Technology Services
Fujitsu
Lovelace
RG12 8SN
From: Jonathan Gribben if
Sent: 21 January 2019 1
To: Parker, Steve
; Ibbett, Dave Newsome, Pete
FUJ00163098
FUJ00163098
; Lenton, Matthew <_ __>; Gareth Jenkins
Cc: Simon Henderson;
>; Lucy Bremner
Subject: RE: Roll 2 [WBDUK-AC.FID27032497]
Dear all,
Privileged & Confidential
Thank you for your time earlier. Here's a list of the actions that I captured from today's calls. Please let me know if
there's anything you'd like to add or change:-
IActionI Paragraph I Action Assigned IFujitsu
of Roll 2 to
4 8 Keyword search for incidents containing the words Fujitsu Steve / SSC
"laptop" and/or "luggable" and/or "outreach" etc. ‘Matthew Lenton] Details
The Peaks referenced below lof Peaks provided at left.
PC0100174 March 1st 2004 to 5th March 2004
FAD317309 reporting: Horizon Kit rebooting itself for
no apparent reason.
Helpdesk user: "Over the past 2-3 weeks engineers
have been to site and have replaced 2xBU's and
2xPSU's but the problem persists."
BU = base units = PC itself. PSU = Power supply units
within the base units
RR "Evidence (from event logs) shows that the power
is being switched off every morning shortly (ie 5 or 6
minutes) before the PM logs on"
RR: "After carrying out tests on our rigs, I have been
able to duplicate the problem here on ONE of our
rigs but not on others. It seems that the Screen
Power Button is incorrectly connected to the
motherboard."
RR: "We have now identified two instances of this,
one in live. This is a hardware build quality issue."
This was followed by:
PC0100899 18th March 2004 to 24th March 2004.
Hardware returned from site to Bracknell for
examination.
RR: "Tests carried out on screen power switch -
working correctly, no further action required."
Your questions 2d,e,f,g
d) Is his example true, or could it have been true:
Yes. Can find no data on the origin for the statement:
"This is a hardware build quality issue". Could be a
discussion with engineering which was not recorded
on the incident progress. Information we have only
describes the hardware issue being seen internally to
FJ on one instance of test rig hardware. No hardware
FUJ00163098
FUJ00163098
error proved on the site.
If so, how often did that sort of problem occur: Very
rare. Only one other found using keywords
"standby", "laptop", "luggable". PCO055550 which
was a problem on prototype hardware going into
standby mode.
What would have caused it: Inconclusive. No
information on root cause of issue reported by the
Post Master onsite. Could be a hardware problem,
could be user miss-operation of hardware.
Could it have affected/did it affect branch accounts:
No. Once powered on the unit would function as
normal.
If so, might its effect on branch accounts never have
been detected with the result that some SPMs might
have been wrongly held liable for false deficits: No
e) Would Rolls have disassembled laptops and done
the other things he describes in para 8: Have to
assume he did as per the incident updates. I expect
he had some assistance (especially with kit on test
rigs - different team totally) but unable to
substantiate.
Would he have had/did he have the conversation
with his manager he describes in para 8: Just can’t
answer this. My analysis of the issue would suggest
that it turned out to be unimportant because there
was no proof that this ever happened in the live
estate and that his comment of "This is a hardware
build quality issue" is simply conjecture. However, he
may have discussed with engineering and truly
discovered a batch of faulty hardware. I would have
expected an update in the incident reading
"Discussed with xxxxxxxx in engineering and we
determined that........ Bad batch...... etc" No such
updates are present.
f) Was the problem referred to in para kept secret,
as claimed at the end of para 8: No evidence either
way. I would not expect that to be the case. It is not
in Fujitsu's interest to have faulty equipment that is
not corrected damaging reputation.
g) Would Fujitsu management have known/did it
know about this problem? Would/did Post Office?
If not, why not: No way of knowing. Information no
longer exists
Check what the experts and witnesses say about KEL I WBD
psteed2847n.
Provide a list of events that give rise to a receipts and I Fujitsu Steve / SSC
payments mismatch.
Examples only, or all
scenarios that caused them]
in reality?
FUJ00163098
FUJ00163098
(Matthew Lenton] May
take rest of this week or
more. Requires eyeball
searching.
4 12 Did Post Office review TC volumes in order to identify I WBD to
potential software issues. pick up with
POL
5 16 Review the contract between POL and Fujitsu and WBD
summarise SLAs/penalties.
6 20 Provide a list of reasons for which transaction data Fujitsu Steve / SSC
would need to be injected at the counter. (Can this be ascertained
{from the sampling referred
fo below at 217
(Matthew Lenton] SSC
forming a query to find
this from OCP data, also
\determining when
transaction would be
injected at the counter.
7 20 Review Peak reference 107043 (example of WBD
transaction being injected into counter).
8 21 Did: (1) Belfast team; and (2) privileged users have Fujitsu (Gareth: answer 1 and 2
the ability to inject transaction data between 2001 and land perhaps explain again
2004? Do they have that ability now? difference between old and
Inew?
9 I 21 Review a sample of OCPs to give an indication as to I Fujitsu Steve / SSC - relates to 20
how frequently transaction data was injected. labove?
(Matthew Lenton] See
laction 6 above
10 I 22 Search for documents relating to the controls around =I WBD
transaction data being injected (DE/HLD/002 is an
example).
11. ‘I General Provide details of Fujitsu's document storage Fujitsu Matthew
practices and retention policies. Are emails, word ‘Matthew Lenton] See
documents etc. from 2001 — 2004 available? lanswer at left
[Matthew Lenton] Emails cannot be retrieved from
the accounts of former Fujitsu employees from that
period, and back ups are not held for that period of
time. The only records of such a person’s emails
would be if they are part of a current employee’s
email account or pst archive, in which case it would
be only the subset of their emails that were to or
from the other user. Similarly, for other
documentation that was held locally be individual
employees on their laptops, that would have been
FUJ00163098
FUJ00163098
deleted when the user left.
As we have already seen, some limited information
from this period does exist, stored in Dimensions and
other networked repositories, some of which we
have already provided in connection with this case.
We are aiming to get a draft response to Roll 2 into circulation by early tomorrow afternoon.
Kind regards
Jonny
Jonathan Gribben
Managing Associate
Womble Bond Dickinson (UK) LLP
womblebonddickinson.com
yO
From: Jonathan Gribben
Sent: 21 January 2019 11:05
To: 'ParkerSP¢
Matthew.Lenton{
Cc: Simon Hendersot
Subject: Roll 2 [WBDUK-AC.FID27032497]
5 pete.newsomet
Gareth Jenkins ¢
Privileged & Confidential
To discuss
Jonny
Please consider the environment! Do you need to print this em:
entong’
rivileged and protected by Ia
an.gribber
ation or attachments is prohibited and ma
only is authorised to
nd delete any copies.
and any attachments is confidential and
a hments. If you are not matt iont
Unauthorised use, dissemination, distribution, publication or copying of this comm!
personal data is in our Privacy Policy on our website.
The information in this e x
access this e-mail and ‘as soon as possibl
“Ge unlawful. Information about how we use
please notify
Any files attached to this e-mail will have been checked by us with virus dé
loss or damage which may be caused by software viruses and you should carry out your own virus checks before opening any attachment
ection software before transmission. Womble Bond Dickinson (UK) LLP accepts no liability for any
Content of this email which does not relate to the official business of Womble Bond Dickinson (UK) LLP, is neither given nor endorsed by it.
This email is sent by Womble Bond Dickinson (UK) LLP which is a limited liability partnership registered in England and Wales under number 0C317661. Our registered office
is 4 More London Riverside, London, SE1 2AU, where a list of members’ names is open to inspection. We use the term partner to refer to a member of the LLP, or an employee
or consultant who is of equivalent standing. Our VAT registration number is GB123393627.
Womble Bond Dickinson (UK) LLP is a member of Womble Bond Dickinson (International) Limited, which consists of independent and autonomous law firms providing
FUJ00163098
FUJ00163098
services in the US, the UK, and elsewhere around the world. Each Womble Bond Dickinson en
can bind or
eparate legal entity and is not responsible for the acts or omissions of, nor
1
ate, another W 1) Limited does not practice law. Please see
son entity. Womble Bond Dickinson (Inter:
www. womb ickinson.com/} further details,
Womble Bond Dickinson (UK) LLP is authorised and regulated by the Solicitors Regulation Authority
Unless otherwise stated, this email has been sent from Fujitsu Services Limited (registered in England No
96056); Fujitsu EMEA PLC (registered in England No 2216100) both with registered offices at: 22 Baker
Street, London W1U 3BW; PFU (EMEA) Limited, (registered in England No 1578652) and Fujitsu
Laboratories of Europe Limited (registered in England No. 4153469) both with registered offices at: Hayes
Park Central, Hayes End Road, Hayes, Middlesex, UB4 8FE.
This email is only for the use of its intended recipient. Its contents are subject to a duty of confidence and may
be privileged. Fujitsu does not guarantee that this email has not been intercepted and amended or that it is
virus-free.
Unless otherwise stated, this email has been sent from Fujitsu Services Limited (registered in England No
96056); Fujitsu EMEA PLC (registered in England No 2216100) both with registered offices at: 22 Baker
Street, London W1U 3BW; PFU (EMEA) Limited, (registered in England No 1578652) and Fujitsu
Laboratories of Europe Limited (registered in England No. 4153469) both with registered offices at: Hayes
Park Central, Hayes End Road, Hayes, Middlesex, UB4 8FE.
This email is only for the use of its intended recipient. Its contents are subject to a duty of confidence and may
be privileged. Fujitsu does not guarantee that this email has not been intercepted and amended or that it is
virus-free.