POL00234921 - Blog of Tim McCormack - The Error in Horizon

Evidence on official site

POL00234921
POL00234921

The Error in Horizon

November 10, 2015 Tim McCormack

The Error

Several of my previous blog posts have alluded to an error in the
Horizon system that was discovered recently. I now need to
document in detail what transpired. Last week Computer Weekly
ran a story highlighting the fact that the CWU Subpostmasters
Section were so concerned about this that they wrote to all of their
members warning them to be vigilant as this error could occur at
any time and result in unexplainable losses at their branches.

I have written to Paula Vennells on two occasions about this error
yet she has failed to respond. This is a very sad reflection on her
competence to manage a large organization such as POL.

Seemingly unknown to Paula, internal documents I have seen from
POL not only acknowledge the error but incredibly they admit they
will ensure it does not happen again by releasing a software patch
to Horizon in March of next year! There is more to this admission
than meets the eye which I will attempt to explain later — but for
the moment I would like to point out that in my opinion they don’t
know what caused the error so how they are going to fix it in March
would be extremely interesting to know!

Timeline of Events

8/10 Sub postmistress sees that Horizon has replicated the same
transaction she has just entered three additional times. The value
of the transaction was £8k. The result of the 3 replicated
transactions meant her branch accounts as calculated by HOL had
£24k more in cash than actually existed. At that moment in time,
because of the SPMR contract she had signed, she was liable to POL
for £24k. £24k that never existed.
POL00234921

POL00234921

8/10 Sub postmistress requests help on a Facebook site from fellow
subpostmasters. Advice given is to call the help line and report the
problem. She does this — note that this is an error of £24k — and
according to the sub postmistress the help desk advise her that it
must be a “system” problem and to call the technical support

team. It is shocking to think that an SPMR calling in an error of
such a significant amount is handled by front line help desk support
and not immediately passed up the line to senior management to
check.

8/10 Sub postmistress calls the technical help desk and reports to
the Facebook group that they did not understand the problem.
They state that it will ‘probably’ be rectified remotely.

8/10 The CWU Subpostmasters branch emails the Service Manager
at ATOS with details of the problem and asks for assistance on
behalf of the member.

9/10 Sub postmistress reports to the Facebook group that no one
has called back from POL help desks. Note that is still an error of
£24k — is this such a common occurrence of such a large amount
that everyone in POL back office ignores it?

9/10 the ATOS service manager replies to the CWU saying that he
asked someone to look into it.

12/10 The sub postmistress receives a brief call from ATOS stating
that they are looking into it and she should receive a call from
NBSC. (I’m not sure if she ever did get that call)

13/10 FIVE days after the error occurred and after close of business
the sub postmistress advises the Facebook group that no one has
called back. This results in another member of the FB group
suggesting that she contact directly a person in the Financial
POL00234921
POL00234921

Services Centre (FSC) that has helped her with a SIMILAR problem
in the past.

Without access to this Facebook group who knows how long it
would have taken POL to get back to her if at all. It’s £24k for
goodness sake. Cash that does not physically exist but the sub
postmistress remains liable to POL for £24k in Physical cash!

13/10 CWU reminds ATOS that nothing has been done about the
error

14/10 an early morning discussion on FB discloses worrying
similarities between previous Horizon errors that had to be
corrected manually by the FSC for other FB subpostmasters.

14/10 later the sub postmistress phones the contact at the FSC and
explains the problem (6 DAYS AFTER £24k WENT MISSING). The
person from the Cash Currency Stock Control Team finds the
transactions that have caused the error and attempts to ‘fix’ the
problem in Chesterfield. Unbelievably he explains that he was not
able to do this because the branch where the error occurred does
not have a ‘unique’ number! WOW! He advises the sub
postmistress to ‘move’ the missing money from the branch where
the error occurred to another of the sub postmistress’s branches.
He would then be able to ‘fix’ the problem centrally.

The sub postmistress quite rightly refuses to do this as it would be a
fraudulent transaction.

What is extremely worrying at this stage is the nonchalance of the
FSC. They seem unconcerned about how the error occurred and
even less concerned about the amount involved.

The FSC however allegedly admits in the phone call to the SPMR
that he had dealt with a similar case only the week previously.
POL00234921
POL00234921

14/10 I send email to Paula Vennells telling her of the problem.

15/10 CWU send a detailed synopsis of what has transpired so far to
ATOS and the complete lack of action on the part of ATOS to
resolve this matter.

19/10 Sub postmistress reports on FB that the matter still has not
been resolved. From a financial viewpoint I understand this to be
down to the need to get a unique code allocated to the branch.
Quite utterly bizarre that it can take this long to do that.

23/10 CWU receives email from ATOS “explaining” what has
happened and how they are going to make sure it doesn’t happen in
the future. This email is the most damning piece of evidence that I
have ever seen that confirms these people have no idea what they
are talking about and by ‘these people’ I refer to POL, ATOS and
Fujitsu.

Subsequently to receiving this email we tested exactly the same
scenario in another branch which was identical in all respects to the
one in which the original error occurred. We tried everything we
could to replicate the error and could not.

The email suggests the ‘root cause’ of the error occurred because of
a process that simply could NOT HAVE HAPPENED in this
branch. It goes on to suggest that they have had the same error in
other branches so what in effect they are saying is that they have a
similar error that they have not fixed but they have failed to realize
that this is a completely new error. It could not be the same as the
one they describe because it is not physically possible for it to have
happened in the way they describe.

They say they can prevent this happening again by a code fix that
they will introduce into Horizon next MARCH! This is frankly
unbelievable. We tested the same transaction in an identical
POL00234921
POL00234921

branch without the error occurring. We tested the same
transaction in the same branch without the error re-occurring.
What on earth are they attempting to fix here? How can they fix
something that they don’t know what causes it to happen?

WHY oh WHY oh WHY does a £24k discrepancy not ring alarm
bells and if the code change (whatever they are going to fix) is not
going to be released until next March WHY oh WHY of WHY do
they not inform the network (in their own words — across the whole
estate) that this error can occur and watch out for it.

The Sub postmistress eventually received a transaction correction
for the missing £24k and then successfully completed a similar
transaction to the one that she had noticed the error on. As the
Sub postmistress commented on Facebook — prior to the error she
had performed the same transaction hundreds of times previously
with no error.

There could be no clearer indication that the error that caused this
to happen was an intermittent error.

Intermittent errors can be caused by many things, including an
unlikely sequence of events or even faulty hardware. It should be
noted that no engineer has visited the branch in question to
perform checks on the hardware

3/11 CWU Subpostmasters branch after carefully checking and
double checking the full facts of this case issue a warning email to
its members to be on the lookout for such an error.

3/11 I send another email to Paula Vennells pointing out to her that
she needs to get personally involved in this chain of events.

5/11 Computer Weekly publish a front page article highlighting the
memo that the CWU felt necessary to send out to their members.
Computer Weekly asks POL for comment but POL re-iterate there
POL00234921
POL00234921

are no errors in Horizon (the POL PR team blissfully unaware that
an error is to be fixed next March)

Conclusion

There can be no other conclusion here than Fujitsu, ATOS and POL
do not have a clue as to what is going on in their respective
organizations. No doubt POL will respond and say that their
internal systems would have picked up this error in due course but
the fact is that they did not automatically do this. It required
manual intervention — in this case by the CWU — in order to get to
the bottom of this matter.

Very importantly, even if POL can show that their internal systems
would pick up such an error (they can’t but that is another story)
this particular event shows a serious delay in that occurring. In this
case the delay was only a few weeks before the missing amount was
credited back to the SPMR but during that time the SPMR was
liable to POL for the amount in error. Past court cases brought by
POL against SPMRs for account discrepancies have relied totally on
the contract the SPMR has signed with POL to show that the SPMR
is liable for these unexplained errors.

While the amount in this case is significant and should have rang
larger internal alarm bells than normal, nevertheless the actual
amount could just as easily have been more than £24k or as little as
£1. The erroneous transactions were entered by the computer
system and not the operator. In this case there were three
duplications but there could have been far more as well as being
only one.

The error in this case was easily spotted by the operator but could
just as easily have been overlooked. The discrepancy in the branch
accounts may only have come to the attention of the SPMR several
weeks later and would have been incredibly difficult to find. The
POL00234921
POL00234921

amount of the discrepancy would have been a multiple of the
original transaction so searching through transaction history for
the amount of the discrepancy would not have shown anything.

The FSC employee reported that he had dealt with a similar case
only the previous week. Just how many of these errors are out
there? The system did not automatically flag these erroneous
transactions as causing discrepancies that required manual
intervention. How are they located?

The explanation from ATOS is utterly incredible. We noticed
immediately that the reason given for the error could not have
happened in this particular branch. How on earth did Fujitsu and
Atos not realize this? The explanation also gives cause for concern
in that they readily admit it occurs across the network and they
have seen previous instances of it — nothing at all to do with the
error we are talking about — so another error exists that SPMRs are
unaware of?

Fujitsu believe that they will fix this error in March — yet they have
not informed the network that the error exists. This is almost
criminal given that these errors can give rise to unexplained losses
to SPMRs that the SPMRs themselves are liable for.

There is no doubt that this error is intermittent. It didn’t happen
before when the SPMR entered the same transaction on to the
system and it did not happen after the event when she repeated the
transaction. It did not occur in another similar branch when we
tested the same set of events.

By their own admission, both from Fujitsu (although they may have
been talking about another similar error) and the FSC, this error
has happened before. They do not know, they cannot know, when
the error first started as they have not identified the cause
POL00234921
POL00234921

properly. It could go back to the very start of Horizon in 2000 for
all we know.

I wrote to Paula Vennells twice about this error and she failed to
respond. What sort of chief executive would not investigate such a
serious problem — one that totally contradicts the oral evidence she
provided to parliament on the reliability of the Horizon system.

There are no excuses. This is undeniable evidence of both systemic
failure within the software system itself but also failure in the
internal procedures within the POL organization. Failures that
many people, including politicians and the media have highlighted
for many years.

Are POL truly unaccountable?

Who can do something about this?

Share this: