POL00153248
POL00153248
Strictly Confidential
Review of the first ten Final Case Review Reports (v1.0)
1. Introduction
Aim and method
1.1 The aim of this document is to provide a short desktop review of the first ten Final Case Review
Reports (CRRs) prepared by Second Sight." Areas of enquiry include:
— What contents do the CRRs contain?
— What methodology are the CRRs based on?
— How have the CRRs been prepared?
1.2 The document is based on a review of the Draft and Final CRRs and the Post Office response
letters to each Draft CRR (Annex B) along with other sources listed in Annex C. The estimation of
the cost and timeliness of the production of the CRRs has involved a manual reconciliation of
data which may contain errors and has not been externally validated (Annex A).”
Background
1.3 On 27 August 2013 Post Office launched The Initial Complaint Review and Mediation Scheme
and a related Working Group. The Scheme is intended to help resolve the concerns of
Subpostmasters regarding the Horizon system and other associated issues.* Second Sight were
commissioned to work with each Scheme Applicant and Post Office to gather information,
investigate cases, and produce a Case Review summarising the findings and making a
recommendation on whether the case is suitable for mediation.
1.4 According to the Overview of the Initial Complaint Review and Mediation Scheme the Case
Review is intended to ‘bring clarity to many cases’ (p.2) and ‘Second Sight will seek to determine
whether there was a problem with Horizon (or any associated issue) that had an impact on [the
Applicant]. If so, Second Sight will also try to determine the scale and scope of that impact on
[the Applicant’s] case’ (p.7).
1.5 On 1 July 2014 Second Sight signed a letter of engagement.’ The scope of services includes:
— Investigating the specific complaints raised by each Subpostmaster who has been accepted
into the Scheme...
— Acting independently in providing the Services and any assessment or opinion given by
Second Sight shall be without bias and based on the facts and evidence available
— Acting with the skill and care expected of qualified and experienced accountants
M006, M009, M022, M028, M048, M054, M057, M062, M076, and M127.
Second Sight invoices (#86-#93) and Huddle upload data presented in the Excel Workbook entitled Second
Sight Activity, December 2013 - July 14 (v0.4).
Post Office (2014). Scheme timeline. p.9. This followed publication on 8 July 2013 ofthe Second Sight
Interim Report into alleged problems with the Horizon system, with Second Sight work on Horizon related
business having originally began in June 2012
Post Office (2014). Second Sight Engagement Letter in relation to the Initial Complaint Review & Mediation
Scheme. 1 July. pp.6-7
3
Page 1 of 10
POL00153248
POL00153248
Strictly Confidential
— Conducting the Services in an efficient manner and with a view to ensuring that the costs of
the Scheme are reasonable
— Using its reasonable endeavours to comply with any deadlines or timeframes set by the
Working Group
2. What contents do the CRRs contain?
Structure
2.
Bb
The ten Final CRRs all adopt a standard structure, using the following section headings:
1) Introduction
2) Points of common ground between the Post Office and the Subpostmaster
3) Points of disagreement between Post Office and the Subpostmaster
4) Where there is disagreement, a logical and fully evidenced opinion on the merits of that
Subpostmaster’s complaint where it is possible to do so
5) Asummary of any points on which it is not possible to offer a fully evidenced opinion due to
a lack of evidence/information
6) Is this case suitable for mediation?
These headings fall short of the requirements set out in the scope of services. Specifically, in
Sections 2 and 3 to provide ‘An assessment of...’ and in Section 6 to provide ‘A view on...’.
Emphasis added.
Length
2.2 The ten Final CRRs are an average four and a half pages long, ranging from two and a half pages
for M048 to ten pages for M022. Some of the content is standard across all the CRRs, reducing
the bespoke content of each CRR by about a page. For example, each Introduction contains a
standard description of Second Sight Support Services Limited and a Terms of Reference
attributed to the Working Group (although the source for this is not mentioned).
2.3 Section 1 and Section 6 of each CRR are of consistent length, at about four paragraphs and one
paragraph respectively. Section 2 to Section 5 vary greatly in length. For example, in MOO9
Section 4 is three paragraphs long whereas in M006 it is fourteen paragraphs (see Annex A i).
3. What methodology are the CRRs based on?
Method
3.
BS
The introductions to the ten Final CRRs do not provide a description of the methodology by
which investigations were conducted, only a reference to sources consulted and the Terms of
Reference for the work. It is therefore hard to know how the CRRs have been prepared.
Sources
3.
ie
According to the Overview of the Initial Complaint Review and Mediation Scheme, Second Sight’s
investigation ‘will be principally based on information provided by [the Applicant] and Post
Office...’ (p.8). Each CRR provides a list of sources that the report should be read in conjunction
Page 2 of 10
3.
3.
3:
3
4
iu
POL00153248
POL00153248
Strictly Confidential
with (Paragraph 1.2) and a list of documents that Second Sight have been provided with
(Paragraph 1.4). Detailed references for these documents are not given in either Paragraph. All
of the CRRs mention the Applicant Initial Application, Case Questionnaire Response and Post
Office's Investigation Report. Three of the CRRs (M022, M054 and 127) also mention the
Applicant and/or Professional Advisor's response to further Second Sight questions.
Use of evidence
The CRRs contain reported speech from the Applicant although the source is not directly
indicated. For example, CR127 states ‘The Applicant remarks on “numerous problems with the
machine” (p.4). The CRRs also attribute comments from the Applicants that are not presented
as reported speech and the source is again not indicated. For example, CRO54 states ‘The
Applicant told us that...’ (p.4). This can make it difficult to distinguish between what the
applicant said and what is an interpretation of what the applicant said.
The CRRs make reference to Post Office evidence. For example, M048 mentions that ‘Post Office
records also show that...’ (p.4), but further detail of these records is not provided. M127 states
that ‘Post Office has informed us...’ and that ‘Post Office has subsequently advised that...’ (p.5),
but no further details are given. M022 includes comments that are unattributed, stating ‘we are
advised that...’ (p.5) and ‘we were also told...’ (p.8), but the source for these comments is not
stated.
In Section 2 the CRRs present points of common ground between the Applicant and Post Office.
For example, M006 states ‘it is common ground that...’ (p.3 ). In Section 3 the CRRs present
Applicant and Post Office perspectives. For example M028 and M048 both use the phrases ‘The
Applicant states...’ and ‘Post Office believe...’ (p.4). In Section 5 the CRRs allege that missing
documents contribute to areas where an opinion is not given. For example, M054 states that
“..key documents, available at the time were not preserved’ (p.6) and M009 mentions ‘the
expiry of document retention’ (p.5).
Assessment
3.6 Section 4 of the CRRs offer some opinion on the cases. For example, M022 states ‘Taking all of
3.
N
these facts and circumstances into account, we believe that Post Office should bear a significant
proportion of the responsibility for the losses that did occur’ (p.10, emphasis added) and MO09
states ‘we do not find the argument by Post Office... particularly compelling.’ (p.4, emphasis
added). M057 states ‘we believe on the balance of probabilities, that the applicant was
responsible for the loss’ (p.5, emphasis added) and M048 states ‘On the balance of probabilities
we believe that the losses were caused either by user error or theft...’ (p.5, emphasis added). No
explanation is given as to how the probabilities have been balanced.
M054 and M127 give the clearest presentation of the points of disagreement presented in
Section 4. Both use bold subheadings to highlight the main points and introduce them with the
phrase ‘The following issues represent points of disagreement between Post Office and the
Applicant’ (p.4 and p.6 respectively). More than once, M054 states ‘There is therefore a conflict
of evidence on this point...’ (p.4). No indication is given of how this ‘conflict’ should be resolved.
Page 3 of 10
POL00153248
POL00153248
Strictly Confidential
3.8 For Section 6 all the CRRs recommend mediation. Limited (or no) reasoning is given and limited
(or no) reference is made to the preceding content in the CRR. The fullest explanations are given
in:
— M054, ‘In our opinion, and recognising the conflicts of evidence reported in Section 3 above
that remain unresolved, we believe that this case is suitable for Mediation...’ (p.6). No
indication is given as to how the ‘conflicts’ could be resolved.
— MO009, ‘In our opinion this case is a weak candidate for mediation... there are however, some
matters such as the failure by Post Office to fully resolve the issues reported to it in 2005
and 2006 to the Applicant’s satisfaction, that may benefit from mediation’ (p.5). No
explanation is given as to how the term ‘weak’ should be interpreted between cases.
— M062, ‘In our opinion this case is a weak candidate for Mediation, primarily due to the fact
that the Applicant accepted a Caution for false accounting. There are however, other
matters such as the allegations of mis-advice and the difficulty in resolving the reported
problems that may benefit from mediation.’ (p.6)
— M057, ‘In our opinion this case is a weak candidate for Mediation, primarily due to the fact
that the key events occurred so long ago. However, some of the matters raised by the
Applicant's Professional Advisor in commenting on our previously issued Draft Report, may
benefit from mediation’. (p.6) The ‘matters’ are not specified.
The shortest explanations are given in M006, M022, M028, M048, M076, M127, which simply
state ‘We consider that...’ or ‘In our opinion this case is suitable for Mediation and the following
issue should be considered’.
3.9 The Final CRRs have been used to: help inform the Working Group decision on whether to
proceed to mediation; provide the Applicant with a third party analysis of the application; and
support the mediation process or help the Applicant understand why their case has not
progressed to mediation. The provision of a clearer and more detailed explanation as to why
cases are (or are not) felt to be suitable for mediation could help reduce the need for plenary
discussion of cases at Working Group.
4. How have the CRRs been prepared?
Timeliness and cost
4.
A
The ten Final CRRs took an average of 11 weeks to produce, from the time the Post Office
Investigation Reports were shared: seven weeks to reach Draft and an average four further
weeks to reach Final. See Annex A ii). Work on the CRRs sometimes preceded receipt of the Post
Office Investigation Report. For example, Invoice #86 attributes work to M009 in December
2013, although the Post Office Investigation Report wasn’t shared until 30 April 2014.
4.
ND
The ten CRRs were subject to an average three weeks of slippage to reach Draft and two working
days slippage to reach Final. Only one of the Draft CRRs was produced ahead of forecast (M022),
whereas four of the Final CRRs were produced ahead of forecast (M022, M048, M054 and
M076). This suggests the production of Draft CRRs was subject to more slippage than Final CRRs.
See Annex A iii). A consequence of slippage has been to extend the time it takes to resolve cases
Page 4 of 10
POL00153248
POL00153248
Strictly Confidential
and to hamper onward planning. Also, if multiple cases are delivered at once this also generates
administrative challenges and delay.
4.3 The average cost attributed in the Second Sight invoices (#86-93) for the production of the ten
Final CRRs is £4.5K per Report. This equates to an average of just under £1K per page and is likely
to be an underestimate due to gaps in the invoice data. For example, M006 has only one hour of
time attributed against it. The three cases with the highest attributed cost are M022, M127 and
M009 (£12K, £11K and £8K respectively).
Revisions
The style of the CRRs has been subject to discussion at Working Group. In March 2013, the Chair
prepared some tentative suggestions for Second Sight to consider as a way of approaching their
work on the claims.° On 1 May the Working Group discussed M022 - with a review to follow a
week later to check how the comments had been taken on board.’ A range of concerns about
quality were raised at this point by Post Office, including that:
= The Applicant would need the Part One report’
— The depth of analysis was not sufficient
— Aclearer articulation was needed of the factual basis upon which conclusions were made
— Neutral language needed to be used
— The evidence used needed to be clearly balanced with any counterpoint brought forward
— Un-evidenced statements needed to be avoided
— Raising real or implied questions needed to be avoided
— It was going beyond Second Sight’s areas of expertise
4.4 Upon receipt, each Draft CRR received a response letter from Post Office (Annex B). Each CRR
was then subject to edits prior to delivery as a Final version.® There is evidence from some of the
CRRs (e.g. M028, M048 and M127) to suggest that the edits made tend to reflect the more
minor points of detail raised by Post Office and do not always (fully) reflect or acknowledge the
more major and substantive points raised by Post Office in their response letters.
4.
in
For example, the Final CRR for M127 has edits which pick up comments made by Post Office on
length of tenure (p.3) and the figure for total losses (p.5). It also has edits which provide a
qualification that ‘Although not accepted by Post Office...’ (p.4) and a change from ‘Both parties
agree’ to ‘the Applicant has advised’ (p.5). The frequency of power cuts is also reduced from ‘a
relatively large number’ to ‘a number’ (p.6).
4.6 However, no edits were made to M127 to reflect or acknowledge that ‘Post Office does not
agree that the Applicant’s approach to managing the ATM might indicate a failure on Post
Office’s part to offer access to appropriate training.’ (Post Office Response Letter, p.8, and
summarised on p.1). The Final CRR still reads ‘The subsequent problems and failure to follow
SAH. (2014). Note from SAH. March
Working Group. (2014). initial Complaint Review and Mediation Scheme, Minute. 1 May. pp.2-3
Second Sight (2014). Mediation Briefing Report: Part One. Draft. 16 May
Final CRRs with Track Changes displayed were reviewed for this exercise.
Page 5 of 10
4.
4.
4.
7
bo
)
POL00153248
POL00153248
Strictly Confidential
procedures by the Applicant provides further confirmation that the training provided was
inadequate in the circumstances.’ (M127, p.8).
Another instance is where Post Office comment that ‘...no evidence of theft or any other
criminal wrong-doing has been found.’ (Post Office Response Letter, p.10). The text of the Final
CRR remains unchanged stating ‘It is entirely possible that these two branches were subjected to
such attacks, and Post Office’s failure to investigate either of the two substantial shortfalls...
means that these types of external theft may have contributed to the losses.’ (M127, p.9).
Although reference is made to ‘elegant techniques’, how such theft might work and why it could
reasonably be expected to have occurred is not explained.
For case M028, the Final CRR has edits which change ‘the losses are... exceptionally large’ to ‘the
losses are... significant’ (p.4). This follows a comment that ‘Post Office considers it would be
helpful if supporting evidence were provided to show why the losses are considered to be
exceptionally large in relation to the turnover of the branch’ (Post Office Response Letter,
p.1).The requested supporting evidence is not provided in the Final CRR.
The text of the Final CRR for M028 states ‘It is unfortunate that further steps were not taken at
the time to conclusively establish the cause of the losses. This appears to demonstrate a
weakness in the support framework available to the Applicant’ (M028, p.5). This does not
acknowledge or reflect the point made by Post Office that ‘The CRR does not explain what
additional steps could have been taken to establish the cause of the losses conclusively. In fact
Post Office went to significant lengths to investigate the cause of the shortfalls...’ (Post Office
Response Letter, p.10).
4.10 With M048 Post Office state that it ‘has not seen any evidence of theft and, on balance,
5.
A
considers that user error was the more likely cause of the losses’ (Post Office Response Letter,
p.1). This is not acknowledged in the Final CRR which states ’On the balance of probabilities we
believe that the reported losses were caused either by user error or theft’ (M048, p.5).
Findings and recommendations
Findings
This short review has found that the first ten Final CRRs:
- Use a standard structure of sub-headings (which elide ‘assessment of’ and ‘view on’)
- Vary (sometimes greatly) in overall length and in the length of individual Sections
- Provide no methodology, only mentioning major source documents and a Terms of
Reference
- Do not provide detailed referencing for the sources cited
- Present Applicant and Post Office perspectives and in places offer opinion on the cases,
particularly in Section 4
= Donot always provide supporting evidence or reasoning to substantiate the points made
- (for some CRRs) tend to reflect the more minor points of detail raised by Post Office and do
not always (fully) reflect or acknowledge the more major and substantive points raised by
Post Office in their response letters.
Page 6 of 10
5.
ie)
POL00153248
POL00153248
Strictly Confidential
Provide limited (or no) explanation (or evidence) for why the case is suitable for mediation
Took an average seven week to Draft plus a further average four weeks to Final
Were subject to an average three weeks slippage to Draft and two days slippage to Final
Have an attributed cost of £4.5K per Report or just under £1K per page of content, which is
likely to be an underestimate due to gaps in the invoice data
Have been subject to minuted Working Group concerns over their style and quality
Recommendations
Based on this short review of the first ten Final Reports, and in order for the CRRs to deliver
against what is required from the Terms of Reference and Scheme documentation , it is
recommended that:
The production of remaining Final CRRs are subject to cost, quality and timeliness control
The level of assessment offered in the CRRs is increased, with the reasoning made clear and
links provided to the relevant evidence
More detailed explanation is given as to why cases are (or are not) suitable for mediation -
with reference to the findings in the report
Explanation of alternative views is provided and an opinion given as to which is to be
preferred and why
The precision of the referencing and citations is improved
Delivery is regularly measured against the Scope of Services and Scheme objectives
The level of case specific detail in the supporting invoices is increased
Page 7 of 10
POL00153248
POL00153248
Strictly Confidential
Annexes
A. Data
i) Cost, timeliness and content data for the 10 Final CRRs (Source: Second Sight Invoices).
Attributed cost likely an underestimate for some cases (e.g. M006).
Paragraph Count (Total and by Section}
‘Case [Work ntarted] Final shared [Working Days] Weeks [Atwibuted hra[Atributed cont[Page CountIcost/PageIrowi] 1 I 2 I 3 I 4 I s I 6
‘woos I 20-1un I O8-Aug 26 5 10 [é wot 6 Ie a{2{f7]if*s{w{ifa
‘woos I —22-Dee I —20-Aug 173 35 se fe eae 3 le a7vofefe{i1]3]3]2]a
‘wo22 I —02-Ape I —13-Jun 53 rn m2 Ie uae 10 le ima[ Ie [uous I
‘wae I —12-Jun OL 14 3 m2 fe _1as[ 3 fe os a furtif3]s]a]a
‘wos I —05-1un I _20um rr 2 ws _[e 1s[ 2s fe mfaftstilta}te}2 a
1wos4 I —14-Apr I —i-Jun 3 2 v3 fe a30[ 3 le ao7[ foots ]6l7]2]a
wos? [26-11 I 08-Aug 40 z wos fe 1ssol 3s fe asfafulife2fsIifa
1w052I —20-Jun aul 30 6 153 fe 220 [3 le veal fuof 12 [s]2]a
‘wo76 I —o7-4un I 20,un 10 2 site az9f 3 fe asf fs fi fete [sa
miz7_[08-Apr_I_13Jun 48 10 wa fe m2ze{ se le iasI s3 [ae] as[aofs I 7 [a
a2 rn moi le asm[ ac fe 983 Ja0sfu0 fs [7 [7 [3 [a
ii) Timeliness of production for the 10 CRRs to Draft and Final (Source: Huddle data).
POIR to Draft Draft to Final POIR to Final
Case__[POIRshared[Draft shared] Work Days I Weeks _IFinal shared] Work Days I Weeks I Work Days I Weeks
moo6 I 06-Jun O1-Jul 18 4 04-Aug 25 5 43 9
moog I 30-Apr. 28-Jul 64 B 20-Aug 18 4 82 16
mo22 I 24-Apr_I_ 16-May 7 3 13-Jun 2 4 38 8
mo28 I 02-May I _13Jun 31 6 O1-Jul B 3 44 9
moag I 15-Apr_I_06-Jun 39 A 20-Jun Fr 2 50 10
mos4 I 10-Apr_I_ 29-May 36 7 iiJun 10 2 46 9
mos7_I _08-May 04-Jul 42 A 08-Aug 26 5 68 14
mo62_I 30-Apr 20-Jun 38 A 31-Jul 30 6 68 14
move I 17-Apr_I _06-Jun 37 7 20-Jun i 2 48 10
miz7_I 24-Apr_I_ 23-May 22 4 13-Jun 16 3 38 8
34 7 18 4 53 Ft
Average Average
iii) Change in Draft and Final due date for the 10 Final CRRs (Source: Huddle data).
DraftCRR ] DrafeCAR ] DraftCRR I Draft CRR POIRto Draft Final CRR] Final CRA] Final CRR Draft to Final
case IPoIRshared] duedate I duedate I duedate I duedate Jorattshared] slippage due date [due date Idue date [Final shared I slippage
() (2) @ (4) (workdays) I (ay I 2) I a) (work days)
moos I o6jun I 204un I 27-Jun : = oLul 8 38sul_I O-Aug O4Aug 2
moos I 30-Apr I 28May I 27jun I tisul I 25s I 28dul a 13-Aug 1-Aug 2
woz I 26-Apr_ I 22-May - : : 16-May 5 aun I 13un 13Jun 4
1wo2s I o2-May I 30-May I 13-Jun = = 13Jun rT or I > OLul 1
twos I 15-Apr_[ 13May I 05-Jun z z 06 un 19 2asun I 5 2oJun 3
1mos4_I 10-Ape I 08 May I _28-Ma = : 29-M 16 a34un I 7 jun 3
mos? I 08-May I _05Jun I 20-1un I _27-Jun = ful 2 2Jul_I OrAug I O&Aug I 08 Aug 7
mos2 I 30-Apr I 28May I 13Jun I 20-Jun 201un 18 29Jul_I OFAug sul 3
wo7s_I17-Apr I 15-May I 06-Jun. - (O6s1un 7 2éclun_[ 20-4un 20un 3
waz7 I 26-Apr I 22-May : 5 23-May 2 aijun I 13Jun 3
152 24
‘Average Average
Page 8 of 10
POL00153248
POL00153248
Strictly Confidential
B. Draft and Final Case Review Reports and Post Office Responses
Post Office (2014). Post Office’s Response to Second Sight’s CRR on case M006. 9 July
Post Office (2014). Post Office’s Response to Second Sight’s Draft CRR on case MOO9. 6 August
Post Office (2014). Post Office’s Response to Second Sight’s CRR on case M022. 5 June
Post Office (2014). Post Office’s Response to Second Sight’s Draft CRR on case M028. 24 June
Post Office (2014). Post Office’s Response to Second Sight’s Draft CRR on case M048. 17 June
Post Office (2014). Post Office’s Response to Second Sight’s CRR on case M054. 6 June
Post Office (2014). Post Office’s Response to Second Sight’s CRR on case M057. 15 July
Post Office (2014). Post Office’s Response to Second Sight’s CRR on case M062. 1 July
Post Office (2014). Post Office’s Response to Second Sight’s Draft CRR on case M076. 17 June
Post Office (2014). Post Office’s Response to Second Sight’s CRR on case M127. 4 June
Second Sight. (2014). Case Review Report. Case Reference: M006. 1 August
Second Sight. (2014). Case Review Report. Case Reference: M006. DRAFT. 30 June
Second Sight. (2014). Case Review Report. Case Reference: M009. 18 August
Second Sight. (2014). Case Review Report. Case Reference: M009. DRAFT. 25 July
Second Sight. (2014). Case Review Report. Case Reference: M022. 13 June
Second Sight. (2014). Case Review Report. Case Reference: M022. DRAFT. 13 May
Second Sight. (2014). Case Review Report. Case Reference: M028. 30 June
Second Sight. (2014). Case Review Report. Case Reference: M028. DRAFT. 13 June
Second Sight. (2014). Case Review Report. Case Reference: M048. 20 June
Second Sight. (2014). Case Review Report. Case Reference: M048. DRAFT. 6 June
Second Sight. (2014). Case Review Report. Case Reference: M054. 11 June
Second Sight. (2014). Case Review Report. Case Reference: M054. DRAFT. 28 May
Second Sight. (2014). Case Review Report. Case Reference: M057. 8 August
Second Sight. (2014). Case Review Report. Case Reference: M057. DRAFT. 4 July
Second Sight. (2014). Case Review Report. Case Reference: M062. 31 July
Second Sight. (2014). Case Review Report. Case Reference: M062. DRAFT. 20 June
Page 9 of 10
POL00153248
POL00153248
Strictly Confidential
Second Sight. (2014). Case Review Report. Case Reference: M076. 20 June
Second Sight. (2014). Case Review Report. Case Reference: M076. DRAFT. 7 June
Second Sight. (2014). Case Review Report. Case Reference: M127. 13 June
Second Sight. (2014). Case Review Report. Case Reference: M127. DRAFT. 23 May
C. Other sources
Post Office. (2014). Scheme Timeline
Post Office. (2014). Second Sight Activity, December 2013 - July 14 (v0.4).
Post Office. (2014). Second Sight Engagement Letter in relation to the Initial Complaint Review &
Mediation Scheme. 1 July
Post Office. (2013). Overview of the Initial Complaint Review and Mediation Scheme.
SAH. (2014). Note from SAH. March
Second Sight (2014). Interim Report into alleged problems with the Horizon system. 8 July
Second Sight (2014). Mediation Briefing Report: Part One. Draft. 16 May
Working Group. (2014). Initial Complaint Review and Mediation Scheme, Minute, 1 May, pp.2-3
Page 10 of 10