FUJ00086456 - Peak Incident Summary: call ref PC0132133; re: large discrepancy.

Evidence on official site

FUJ00086456
FUJ00086456

Peak Incident Management System

Call Reference PC0132133 Call Logger Customer Call_ -- EDSC
Release Targeted At -- BI 3890R —_ Top Ref FSTK_2_0_WP23712
Call Type Live Incidents Priority : B -- Business restricted
Contact EDSC Call Status Closed -- S/W Fix Available to Call Logger
Target Date 13/02/2006 Effort (Man Days) 0
Summary FAD378420: large discrepancy
All References Type Value

Powerhelp E-0602100449

Fast Track Fix FSTK 2 0 WP23712

Release PinICL PCO133131

Release PinICL PC0133486

SSCKEL KEL MScardifield2219S

: : Work Package PWY_WP_23712

Progress Narrative

Date:10-Feb-2006 14:03:04 User: Customer Call_

CALL PCO132133 opened

Details entered are:-

summary:PM states that she had desprepency that seemed to
call Type:1

call Priority:B

target Release
routed to:EDSC

T_3S82R
_Unassigned_

Priority: B
contact Name: Irene
contact Phone:

riginator:
riginator's reference: E-0602100449
product Type: riposte

product Serial No:

Product Site: 378420

the descrepency vanished and normal figures remained normal

10/02/06 13:17 uk957320

Information: PM noticed this on daily cash report previews
10/02/06 13:17 uk957320

information: PM referred by NBSC

0/02/06 13:20 uk957320

information: Descrepency was at its highest £1000 shortage.
jsut PM insists within minutes the cash balance preview
stated that the balance was fine and descrepency was gone.
10/02/06 13:30 uk957320_,
information:

time / date
lode ID 2
10/02/06 13:43 uk957320

information: PM staes that on the 4th of Feb pm was doing her cash
ldescrepency and the figure was showing a shortage of 470.00 pm
then came out of this pm then served customer s pm then amened

the cash the figure was around 5.00-10.00 pm then states

that the descrepancy then went up to around 1,000 pm rechecked

the system and the cash went up to 1,453 pm states that the

figure pm then states that the system corrected it self and

lis now back with the correct descrepancy

10/02/06 13:49 uk957320

information: pm states she did a trail balance and was showing the 74
jpence shortage which pm was happy with

10/02/06 13:51 uk957320

information: pm stat
H2.10 username 18

this was on sat 4th feb pm declared cash around
ode 2 stock unit EE

10/02/06 13:53 uk957320
KEL Ref No.: no kel found
Ho/02/06 13:54 uk957320

10/02/06 13:12 PM states that she had desprepency that seemed to become greater over the course of 20mins.

Then a few minutes later

FUJ00086456
FUJ00086456

down loading PS log
8 uk957320
file id 710717 node 2
58 uk957320

0/02/06 13:
Recommend: please check why when pm was doing her cash desprepency
la figure of 1,000 showed on her report when pm did the trail
balance it showed correct 74 pence shortage pm wants to

know why this figure of 1,000 showed in her desprepe:
see log for more details

0/02/06 14:01 SYSADM

lopen OTI: Automatic Open OTT

**Updated by Ricky Law at 10/02/2006 14:01:48
10/02/06 14:01 uk957320

IREASSIGN: Call # E-0602100449 was Reassigned from Ricky Law, Group
lis to Group EDSC1

cy please

jDate:10-Feb-2006 14:40:52 User:Jim Anscomb
Product EPOSS & DeskTop -- EPOSS added.

lDate:10-Feb-2006 14:41:16 User:Jim Anscomb
the Call record has been assigned to the Team Member: Cheryl Card
Progress was delivered to Powerhelp

lDate:10-Feb-2006 14:45:00 User:Cheryl Card
[the call summary has been changed from:—

lem states that she had desprepency that seemed to
[he call summary is now:

FAD378420: large discrepancy

JDate:15-Feb-2006 10:09:47 User:Gheryl Card
[the Call record has been assigned to the Team Member: David Seddon
progress was delivered to Powerhelp

jDate:15-Feb-2006 16:12:19 User:David Seddon
lsvidence Added =

Date:15-Feb-2006
evidence Added ~ Al

[Date:15-Feb-2006
levidence Added -

David Seddon
t/Audit logs plus messagestore extracts

jbate:15-Feb-2006 17:01:10 Uscr:David Seddon
[Start of Response]

been done between these two declarations.

so at 12:32 the system when it calculated a -ve £1453.

lstockunit EE on counter 2.

[End of Response}

response was delivered to Powerhelp
Hours spent since call received: 0 hours

can see that on 4th Feb the PM made numerous cash declarations for stockunit EE.
jetting different variances. What does stick out though is the cash declaration done on counter 2 at 12:32. PM declared £1218.82
land the system calculated a -ve £1453.28 discrepancy. Moments later the PM logs onto counter 1 and declared £1218.82 cash once

again. This time the system calculated a -ve £93.12 discrepancy, which is roughly what the PM expected, even though no txns had

8 discrepancy it thought that there was £2672
cash txns it would actually have had this amount at 08:47 4th Feb. This is about the time that a Trial Balance was carried out for

The PM was declaring all sorts of figures and

-10 of cash. Totting up the

Ht would appear that when working out the cash discrepancies on counter 2 the system has used an old ‘data tree’ (the one it used
lat the earlier trial balance) rather than creating a new one so the discrepancies were wrongly calculated. It wasn't until the PM
hater moved to counter 1 that a new ‘data tree’ was produced and the discrepancies were calculated correctly.

lkesponse code to call type L as Category 40 -- Pending -- Incident Under Investigation

bate:15-Feb-2006 17:03:22 User:David Seddon
[Start of Response]

Passing call over to development for comment/investigation.
[End of Response]

lkesponse code to call type L as Category 40
response was delivered to Powerhelp

Hours spent since call received: 0 hours

Pending ~- Incident Under Investigation

Jbate:15-Feb-2006 17:03:27 User:David Seddon
[the Call record has been transferred to the team: QFP
Progress was delivered to Powerhelp

FUJ00086456
FUJ00086456

[pate:iS-Feb-2006 17:16:56 UseriLionel Higman
the Call record has been assigned to the Team Member: Mark Scardifield
progress was delivered to Powerhelp

Joate:15-Feb-2006 17:36:23 User:David Seddon
[Start of Response]

ready spoken with Irene the PM at the office. She was most concerned about what had happened and had spent some time worrying
Jbout where the large discrepancy had come from. Apologised and explained what looks to have happened. Reassured her that she
hasn't lost any txns or got any suspect txns on her system.

[End of Response]

Ikesponse code to call type L as Category 40 -- Pending -- Incident Under Investigation

Response was delivered to Powerhelp

tours spent since call received: 0 hours

bate:16-Feb-2006 12:56:16 Uscr:Mark Scardifield
the Call record has been transferred to the team: EPOSS-Dev

IThe Call record has been assigned to the Team Member: Gerald Barnes
Progress was delivered to Powerhelp

ldate:16-Feb-2006 14:36:22 Uscr:David Seddon
Didn't spot it éarlier but I have since found a KEL that looks like it might fit the problem - KEL MScardifield2219S. Does it fit
lor is it slightly different?

[Date:16-Feb-2006 1.
Reference Adde

bate:23-Feb-2006 14:37:24 User:Gerald Barnes
[Start of Response]

ly totting up all cash transactions done up to the point of the spurious -ve £1453.28 discrepancy I can see that what has happened
nas there has been a failure of the Notify mechanism from the point of the Trial Balance done on counter 2 on 4th. February
8:41:49. I can also see from the audit log that no Notifys were received after this point. I imported the message store to this
point and did the trial balance again; but afterwards when I sold a stamp the notify did work.

iginally this failure in the Notify was thought to be an Escher bug – see PC0121925. However the fact that in this case the
lotify mechanism was working perfectly well before the trial balance and then suddenly stopped working gives me cause to suspect
that there may be some other cause.

If therefore recommend a small amount of extra diagnostics in DataServer. First of all I would like an extra audit line just before
the Notify mechanism is actually started – this is not done at the moment. It is controlled by a flag and if, for some
jreason, this flag was not set you would get all the symptoms seen. In addition I would like an extra conditional audit line in the
jroutine in DataServer called on Notify – at the moment it does audit in certain conditions «#8211; I would like to leave this
jas it is but in addition audit the entire command passed in if nothing is audited at all (in other words the command is slightly
different from the expected format which would, if it occurred, also give rise to exactly the symptoms described) .

in addition I would like this PEAK cloned to address a problem I discovered whilst trying to duplicate this problem which in fact
lives rise to exactly the symptoms described as far as the clerk would see (but is not what happened because it would give a
ifferent signature in the audit log).

tf you start to rollover the Stock Unit and get the message
lacning
jarning The end data for TP 12 is not today #8211; please double check that you wish to roll over into the next TP.

jand you, at that stage, click Cancel then after that point all Notifys are ignored and subsequent cash declarations will not take
into account any additional transactions done. The fix is required in StockUnit where it says

case "RollEarlyResponse”

If (ObjattributeValue (sCommand, "ButtonClicked")) = "1" Then
obj Desktop. DesktopButtonDisabled = True

lsstock = CurrentStockUnit ()

e.Callinterface ObjMake("CmdStr", ObjMake("Cmd", “ProcessRolloverCommit") & ObjMake("ButtonClicked", "1"))
RolloverQuestion = False

lk1se
~PCO117205 MMc 13/04/05 Clear previous display messages and unlock desktop..
)~ WW PCO121111 2/6/2005 [BEGIN] ~ Remove screen flash during SU rollover
lUserAbortFromFinalBalance

"~ objDesktop. HideMessage
'~ objDesktop.DesktopButtonDisabled = False

!~ LockDesktopRequest vbNullString, False, sUser, objDesktop
t= HideWait objDesktop

~ Ww PCO121111 2/6/2005 [END]

lend rf

land what is required is an additional call

JsResult = objCurrentFigures.UnFreeze

)

[just after the UserAbortFromFinalBalance

FUJ00086456
FUJ00086456

LIVE FIX IMPACT
lEPoSSDataServer diagnostics

cost: (man days in development

h man day

IMPACT ON DEVELOPMENT:

very little

IMPACT ON USER:

ery little

IMPACT ON OPERATIONS:

ery little

RISK: (any associated to releasing the fix, or risks if fix is not released
very little

[TARGET RELEASE:

s9or

FORECAST DATE OF DELIVERY WITHIN TARGE'
27/02/06

RELEASE:

JIVE FIX IMPACT
IePossstockUnit fix

cost: (man days in development)

h man day

HMPACT ON DEVELOPMENT:

very little

HMPACT ON USER:

11 stop cash declarations giving spurious results after a particular error
ath has been taken on StockUnit rollover

IMPACT ON OPERATIONS:

ill reduce the incidents of incorrect cash declaration results
RISK: (any associated to releasing the fix, or risks if fix is not released)
negligible

[TARGET RELEASE:

soor

FORECAST DATE OF DELIVERY WITHIN TARGET RELEASE:

28/02/06

[End of Response]
Response code to call type L as Category 42
liours spent since call received: 21 hours

Pending -- Product Error Diagnosed

JDatc:23-Feb-2006 14:37:46 Uscr:Gerald Barnes
Ihe Call record has been transferred fo the team: RelMngmntForum
Progress was delivered to Powerhelp

lDate:24-Feb-2006 11:57:55 User:Mark Scardifield
Irry and drop at the same time as 130089

JDate:24-Feb-2006 16:15:42 User:Tyrone Cozens
[fhe call Target Release has been moved to Targeted At -~ BI_3890R

Date:24-Feb-2006 16:16:33 User:Tyrone Cozens
[Start of Response]

RMF authorise a fix for S90R. Routing to Eposs Dev for progression.
{End of Response]

esponse code to call type b as Category 56
fours spent since call received: 0 hours

Pending -- Live Fix Authorised

Feb-2006 16:16:48 User:Tyrone Cozens
the Call record has been transferred to the team: EPOSS-Dev
Progress was delivered to Powerhelp

jbate:24-Feb-2006 16:18:15 User: Tyrone Cozens
fo be delivered alongside PC0129639.

bate: 24-Feb-2006 17:27:53 User:Mark Scardifield
ignore my comment: about dropping at the same time as 130089

JDate:24-Feb-2006 17:28:21 User:Mark Scardifield
[the Call record has been assigned to the Team Member: Gerald Barnes
Progress was delivered to Powerhelp

joate:27-Feb-2006 13:32:34 User:Gerald Barnes

[Start of Response]

A new release of EPOSSStockUnit has been produced to fix the problem that if you click Cancel on the Message "Warning The end data
for TP 12 is not today – please double check that you wish to roll over into the next TP." produced during Stock unit

FUJ00086456
FUJ00086456

[Follover and then do more transactions they are not added to the tree and Declare Cash shows the wrong discrepancy+

new release of DataServer has been prepared with extra diagnostics around the area of the notification mechanism. One immediate
result is that when the notify mechanism is enabled you will get a message of the form "DS
lclsInternalSession.CompletePopulateTree: Just about to create a message port number 4 and filter criteria
kFilter:<TxnData.Container:IND><EPOSSTransaction.PM.L5:3017>>". In addition if the Notify mechanism receives an unrecognised
command you will get a message "DS C/I Unprocessed command received

kcmast r :<Cmd: Dat aServerNewRecord005><Data:<Filter:<?xnData Container: IND><EPOSSTransact ion. PM. L
:1><Num:3013><Date:27-Feb-2006><Time:12:56:12><User :MIGRO1><Expir anStartNum:3012><TxnData:<Session!
><TxnId: 44-901777-1-3011-3><Container: IND><Start : <Date:27-Feb-2006><Time: 12: 56:12><TF:4>><End:<Date:27-Feb-
2006><Time:12:56:12><TF : 6>><Mode: SC>><App] icat ion: EPOSSAppMain><EPOSSTransact ion: <CofA:551000><ProductNo: 1><PVe
h><SaleValue:~

lo. 3><BlackBoxData:<M:SC><V:15><UnitPrice:0.3><S:1>><TranType:S><PM: <I :1001><L2:1000><L3: 3003><L4: 3008><15:3017>><SM:><TM: >><Debit:
B0><CRC:717FSDE7>>>>". It is possible to integration test the former message but not the latter which will only occur in exception
conditions.

(017>><Message:<GroupId:901777><Id
4-901777-1-3011-

:43><oty

[End of Response]
lkesponse code to call type L as Category 46 -- Pending ~- Product Error Fixed
Hours spent since call received: 7 hours

bate:27-Feb-2006 13:33:00 User:Gerald Barnes
the Call record has been transferred to the team: EPOSS-Rel
Progress was delivered to Powerhelp

lbate:28-Feb-2006 08:40:47 User:Mike Coon
[Start of Response]

Fix released (with diagnostic) in WP23712 for BI3S90R
[End of Response]

esponse code to call type L as Category 46
fours spent since call received: 0 hours

Pending -- Product Error Fixed

Date: 28-Feb-2006 08:41:02 Uscr:Mike Coon
Reference Added: Work Package PWY_WP_23712

Date:28-Feb-2006 08:41:05 Uscr:Mike Coon
fOr Reference set to: Work Package PWY WP 23712

Jbate:28-Feb-2006 08:44:08 User:Mike Coon
[the Call record has been transferred to the team: Dev-Int-Rel
Progress was delivered to Powerhelp

bate:28-Feb-2006 11:15:11 User:PIT Automated User
Reference Added: Fast Track Fix FSTK 2 0 WP23712 (TOP Reference)

lOate:28-Feb-2006 12:33:47 User:Arun Singh
the Call record has been transferred to the team: Live Supp.Test
progress was delivered to Powerhelp

[Date:07-Mar-2006 1!
Reference Added: Release PinIc

(8:52 User:Edward Willis
PC0133131

JDate:15-Mar-2006 11:57:32 User:Edward Willis
Reference Added:  PinICL PCO133486

bate: 22-Mar-2006
[Start of Response]

'ested in LST (See release Peak for details). Please close

[End of Response]

Response code to call type L as Category 60 -- Final -- S/W Fix Released to Call Logger
Routing to Call Logger following Final Progress update.

fours spent since call received: 0 hours

6:09 User:Sheila Bamber

jbate:22-Mar-2006 14:36:43 Uscr:Lorraine Guiblin
the Call record has been assigned to the Team Member: David Seddon
Progress was delivered to Powerhelp

jate:23-Mar-2006 1
[Start of Response]
Response from development:

7:10 User:David Seddon

new release of EPOSSStockUnit has been produced to fix the problem that if you click Cancel on the Message "Warning The end data
for TP 12 is not today ? please double check that you wish to roll over into the next TP." produced during Stock unit rollover and

FUJ00086456
FUJ00086456

fren do more transactions they are not added to the tree and Declare Cash shows the wrong discrepancy:

new release of DataServer has been prepared with extra diagnostics around the area of the notification mechanism. One immediate
jresult is that when the notify mechanism is enabled you will get a message of the form "DS
lclsInternalSession.CompletePopulateTree: Just about to create a message port number 4 and filter criteria
kFilter:<?xnData.Container: IND><EPOSSTransaction-PM.15:3017>>". In addition if the Notify mechanism receives an unrecognised
command you will get a message "DS C/I Unprocessed command received

‘cmd : DataServerNewRecord005><Data:<Filter:<TxnData, Container: IND><EPOSSTransact ion. PM.L5:3017>><Message:<GroupId:901777><Id
:12><User: <Expiry: 42><TranStartNum: 3012><TxnData:<SessionId:44-901777-1-3011-
h><txnId:44-901777-1-3011-3><Container: IND><Start:<Daté:21~Feb-2006><Time: 12:56: 12><TF: 4>><End:<Date:27-Feb-
2006><Time:12:56:12><TF: 6>><Mode : SC>><App] icat ion: EPOSSAppMa in><EFOSSTransact ion:<CofA:551000><ProductNo: 1><PVer: 43><Qty
jl><SaleValue:-
jo. 3><BlackBoxData:<M:SC><V:15><UnitPrice:0.3><S:1>><TranType:S><PM: <I: 1001><L2:1000><L3: 3003><L4: 3008><L5:3017>><SM:><TM:>><Debit:

30><CRC:717F5DE7>>>>". It is possible to integration test the former message but not the latter which will only occur in exception
conditions.

[End of Response]

Response code to call type L as Category 40 -- Pending -- Incident Under Investigation

Response was delivered to Powerhelp
fours spent since call received: 0 hours

bate: 23-Mar-2006 1
[Start of Response]

lew versions of software have been released to the live estate both to fix a specific variant of the problem and also to provide
ladditional diagnostics to help identify the root cause of other variants.

0:32 User:David Seddon

KEL MScardifield2219S updated.

Returning call for closure.
[End of Response]

Response code to call type L as Category 60 -- Final -- S/W Fix Released to Call Logger
touting to Call Logger following Final Progress update.

service Response was delivered to Powerhelp

JDate:23-Mar-2006 16:20:32 User:David Seddon
CALL PC0132133 closed: Category 60 Type L

Jbate:23-Mar-2006 16:20:32 Uscr:David Seddon
Hours spent since call received: 0 hours
lbefect cause updated to 14 -- Development ~ Code

Date:23-Mar-2006 16:23:16 User: Customer Call_
consumer Phelp has received the call closure

Root Cause Development - Code

Logger : _Customer Call_ -- EDSC

Subject Product EPOSS & DeskTop -- EPOSS (version unspecified)
Assignee _Customer Call_ -- EDSC

Last Progress 23-Mar-2006 16:23 -- Customer Call_