FUJ00154684
FUJ00154684
Peak Incident Management System
Call Reference PC0152376 Call Logger _Customer Call -- EDSC
Release Proposed For -- T80 Top Ref : 82747
Call Type Live Incidents Priority B -- Business restricted
Contact EDSC Call Status Closed -- Avoidance Action Supplied
Target Date 10/01/2008 Effort (Man Days) 2.00
Summary FAD005948 BM stock unit was rolled over it was forced to clear the local suspense account
AllReferences Type Value
TRIOLE for Service 82747
SSCKEL KEL dsed5628Q
Clone Call PCO152421
Clone Call PC0164429
Progress Narrative
pate:20-Dec-2007 12:35:19 User:_Customer Call_
CALL PC0152376 opened
Details entered are:-
summary: Ibrahim from the NBSC has asked that an issue be i
call Type:
call Priority:B
jfarget Release: T70
outed to:EDSC - _Unas
jbate/Time Raised: Dec 20 2007 1
priority: B
ontact Name: 1
ontact Phone
riginator: XXXXXX@TFSO1
riginator's reference: 82747
Product Serial No:
product Site: 005948
dag - NBSC
Ibrahim from the NBSC has asked that an issue be investigated by our software team regarding discrepancies still showing when the
WS stock unit is rolled to clear the local supsense account.
incident History:
2007-12-20 11:53:19 [ Brooks, Katrina]
INIT : create a new request/incident/problem/change/issue
2007-12-20 12:01:32 [ Brooks, Katrina]
LOG : The following information has been sent to me via Email from Ibrahim @ NBSC
n Wednesday 12/12 the BM stock unit had a gain of £465.73. As this stock unit rolled over it was forced to clear local suspense
c1083.76- . The gain of £465.73 did not go to local suspense and is not included in the £1083.76-.
[this was not the last stock unit to roll over. The last stock unit to roll over was MIS at 10:20 on 13/12. This stock unit had no
Iaiscrepancies. MIS is a correction stock unit and was not inactive as it is rolled every BP.
the suspense account and final balances corroborate the above as the office has sent us copies.
IThe Trading statement agrees with the suspense account and that BM stock cleared suspense but did not send its gain to suspense.
fhe Trading position line should always show zero. Under the BM stock column it shows £465.73~.
It have had a trial done on BM stock to see if this is showing the £465.73 but it is not.
2007-12-20 12:02:28 [ Brooks, Katrina]
0G : I contacted the PO to gain more details but the pm was on the other phone. I was asked to call back in 15 mins.
2007-12-20 12:04:59 [ Brooks, Katrina]
0G : Ibrahim stated that this might be the same issue for branch code 003020 (incident number 76918) .
2007-12-20 12:25:58 [ Brooks, Katrina]
L0G : I have spoken to the PM for more details:
lsu - BM that has the problem
TSC SU is one they use to roll over the office.
User name JBALZG
Have rolled into TP9
jode 1
2007-12-20 12:33:53 [ Brooks, Katrina]
0G : Ibrahim from the NBSC states that this might be related to Branch Code 003020 (76918) that I have sent back across for
investigation.
FUJ00154684
FUJ00154684
P007=12-20 12:57:31 [I Brooks, Katrina]
toc : Can you please investigate as to why when the BM stock unit was rolled over it was forced to clear the local suspense
Jaccount. This was not the last stock unit to be rolled over.
thanks
2007-12-20 12:38:11 [ Brooks, Katrina]
[TR : Transfer ‘group’ from 'HSH7' to 'PEAK'
Dat e:20-Dec-2007 12:41:36 Usor:Lorraine Guiblin
[fhe call summary has been changed from:
Ibrahim from the NBSC has asked that an issue be i
[the call summary is now:
IFAD005948 BM stock unit was rolled over it was forced to clear the local suspense account
jDate:20-Dec-2007 12:41:51 User:Lorraine Guiblin
Product EPOSS & DeskTop -- Counter Common (version unspecified) added.
JDate:20-Dec-2007 12:42:07 Uscr:Lorraine Guiblin
[the Call record has been assigned to the Team Member: David Seddon
lprogress was delivered to Provider
[Date:21-Dec-2007 13:46:12 User:David Seddon
cloning call so original can be passed to development for further investigation and clone can be passed to MSU for reconcilaition
jpurposes.
jDate:21-Dec-2007 13:46:19 User:David Seddon
call has been cloned to Call:PC0152421 by User:David Seddon
jbate:24-Dee-2007
kvidence Added -
0:29 User:David Seddon
5948 - Complete Me
Sagestore
jDate:21-Dec-2007
jbate:21-Dec-2007
svidence Added -
Date:21-Dec-2007 1
[Start of Response]
Istockunit BM was being rolled over on counter 1 at the same time that the various EOD of day processes were being run in the
background around 7pm. It was during the CABSProcess that the following message was written to the audit log...
su: £Post'T'xnsToLocalSuspense (:-1056374781) Timeout occurred waiting for lock. (0xC1090003) CreateMessageEx:
IkiposteCreateMessageEx call failed.
the messages that should have posted the £465.73 gain in stockunit BM to local suspense failed to be written. Consequently, when
liocal suspense was cleared (written off to P&L in this case) the £465.73 wasn't taken into account and this resulted in the -
:465.73 trading position seen on the Branch Trading Statement.
outing call to development to investigate further and improve the error handling so that following the failure to write messages
the system doesn't carry on regardless.
[End of Response]
lkesponse code to call type L as Category 40 -- Pending -- Incident Under Investigation
Response was delivered to Consumer
IDate:21-Dee-2007 1:
[Start of Respon
Itt is not believed that there will be any ongoing impact of this problem at the branch and the branch is not personally out of
jpocket given that losses were written off to P&L. However, there is an impact on POLFS which will need to be corrected. The
detail for this is contained in call PC0152421 which has been passed to the MSU for onward progression to POL.
[End of Response]
Response code to call type L as Category 40 -- Pending -- Incident Under Investigation
lkesponse was delivered to Consumer
1:22 User:David Seddon
Date:21-Dec-2007 15:01:29 User:David Seddon
[the Call record has been transferred to the team: QFP
Progress was delivered to Provider
[Date:02-Jan-2008 08:29:51 Uscr:Lionel Higman
[fhe Call record has been assigned to the Team Member: Mark Scardifield
progress was delivered to Provider
FUJ00154684
FUJ00154684
[Date:02-Jan-2008 09:51:02 UscriMark Scardifield
the Call record has been transferred to the team: EPOSS-Dev
[the Call record has been assigned to the Team Member: Gerald Barnes
Progress was delivered to Provider
joate:02-dan-2008 13:17:58 User:Gerald Barnes
target Date/Time updated: new value is 10/01/2008 12:35
Development Cost updated: new cost is 2 (Man Days)
[Start of Response]
[fhe fact that EPOSS code is not resilient to errors is endemic. There seems little point fixing it in this one particular case
because there will be many others to catch you out. For example when I tried to balance with CABSProcess running I found that
\teclaring cash failed with the same sort of error message!
It may be worth passing on the general message to the HNGx team that in many cases code should always try and exit gracefully
jafter an error and not just blunder on regardless.
[This is a perfect example of why. Had the balancing code exited gracefully then if the user had tried again after CABSProcess had
finished working then all would have been well.
for the time being I propose a much cheaper solution then re-writing a lot of EPOSS error handling.
[fhe problem is that because of a previous PEAK (PC0140715) CABSProcess writes out messages atomically. It does a StartTransaction
fmite early on (which creates the lock), then initiates writing lots of transactions with CreateMessage and persistent objects
ith PutObject and finally really writes them with a call to Endfransaction (which ends the lock). If something else tries to
rite a transaction whilst CABSProcess has things locked then it will time out after 10 seconds. Hence if CABSProcess takes more
than 10 seconds to run you could get this sort of problem. In this case CABSProcess took 33 séconds to run which gives a
significant window of opportunity for this sort of problem to occur. I suggest addressing this matter directly by having
caBsProcess store all that it wants to write out to a collection and then only really write it out at the very end. In this way
the system will be locked for less than 10 seconds and there will be no possibility of this sort of problem.
lerx IMPACT
omplete Forecast Date and Development (man days) fields below this text box.
include a brief statement for each of the headings below these instructions.
n return to Details window Set Target Release Type to "Proposed for" and Target Release to that proposed.
fo the Developer:
(1) Put yourself in the shoes of people downstream and provide information that they are likely to need to process this fix. eg
[the testing and rollout costs may add significantly to the COST of the fix
(2) Check that the statements are still accurate post-implementation
IMPACT ON DEVELOPMENT:
effort in mandays.
2 man days
There will be an expectation at RMF that this approximates to the timescale for delivery so if there are reasons why this might
Inct be the case please note them here.
IMPACT ON TEST:
hat independent test coverage does development recommend?
[this will often be about the level of regression testing required.
lust some independent tests that CABSProcess is still producing the same results as before.
IMPACT ON USER:
Benefit of making the fix.
Tt will no longer matter if CABSProcess is running when the user tries to do many sorts of different things, balancing included.
at_ does the user have to do to get this problem?
jbo anything which involves writing a transaction whilst CABSProcess is running (after 19:00) when CABSProcess has sufficient work
to do so that it takes more than 10 seconds to run (so probably on the larger offices).
How does it affect them when it occurs?
whatever is being tried on the counter at this time can potentially fail.
IMPACT ON OPERATIONS:
Benefit of fix that may not visible to end user.
tess support calls.
IRISKS (of releasing and of not releasing proposed fix):
hat live problems will there be if we do not issue this fix?
Problems will continue to occur if the counter is being used whilst CABSProcess is running in those cases when it takes more than
1.o seconds to run.
at are the risks of this fix having unexpected interactions with other areas?
lone.
is this a high-risk area in which changes have caused problems in the past?
les. However the fix proposed is self contained and is considered unlikely to cause any problems.
should we consider a pilot rollout and of what sort:
FUJ00154684
FUJ00154684
[think a pilot is well worth while in all cases: Wowover as stated before 1 do not consider this a dangerous fix:
LIST OF LIKELY DELIVERABLES:
capsProcess
LIST OF THE ABOVE ALREADY DELIVERED FOR THE PROPOSED RELEASE:
None
IIST OF THE ABOVE ALREADY DELIVERED TO A RELEASE LATER THAN THAT PROPOSED:
LIST OF THE ABOVE LIKELY TO BE REDELIVERED INTO THE PROPOSED OR A LATER RELBAS
jone
IYTHING ELSE THAT SHOULD BE KNOWN ABOUT THIS CHANGE:
jothing
[End of Response]
Response code to call type L as Category 55 -- Pending —- Live Fix Impact Supplied
joatc:02-Jan-2008 13:20:43 Uscr:Gerald Barnes
the call Target Release has been moved to Proposed For ~~ T70
Date:02-gan-2008 1.
{Start of Response]
Ir have put proposed for 170. However I think it really wants to be 180. There is no T80 option at the moment.
[End of Response]
Response code to call type L as Category 55 -- Pending -- Live Fix Impact Supplied
1:40 User:Gerald Barnes
jat¢:02-Jan-2008 13:23:17 User:Gerald Barnes
the Call record has been transferred to the team: RelMngmntForum
Progress was delivered to Provider
jat¢:08-Jan-2008 15:19:29 User:John Boston
{the call Target Release has been moved to Proposed For ~~ 180
date:40-dan-2008 14:31:17 User:Tyrone Cozens
[Start of Response]
Routing to EDSC for KEL and close.
nse}
lkesponse code to call type L as Category 68 -~ Final -- Administrative Response
kouting to Call Logger following Final Progress update.
jDate:10-Jan-2008 14:39:30 Uscr:Lorraine Guiblin
{the Call record has been assigned to the Team Member: David Seddon
Progress was delivered to Provider
Jbate:10-Jan-2008 1
Ineference Added: 5.
8:49 Uscr:David Seddon
Date:10-Jan-2008 16:06:12 User:David Seddon
[Start of Response]
it has been decided that no fix will be carried out for the time being given the rarity of the problem. Should the problem become
nore prevalent then the need for a fix will be reviewed once again. In the meantime KEL dsed5628Q
has been created to cover the problem.
ith regard to this instance of the problem we have already passed details and corrective actions necessary on to Post Office
limited by means of a BIM issued by the MSU (PC0152421). Therefore no further action is necessary and this call can simply be
closed.
[End of Response]
Response code to call type L as Category 70 -- Final -
Routing to Call Logger following Final Progress update.
service Response was delivered to Consumer
Avoidance Action Supplied
JDate:10-Jan-2008 16:06:13 User:David Seddon
CALL PC0152376 closed: Category 70 Type L
jbate:10-Jan-2008 16:06:12 User:David Seddon
jbefect. cause updated to 14 -- Development — Code
FUJ00154684
FUJ00154684
fdate:10-Jan-2008 16:14:50 Uscr: customer Call_
Consumer XXXXXX@TFS01 has acknowledged the call closure
Dat e:05-Sep-2008 12:56:52 User:David Seddon
call has been cloned to Call:PC0164429 by User:David Seddon
Root Cause Development - Code
Logger _Customer Call_ -- EDSC
Subject Product EPOSS & DeskTop -- Counter Common (version unspecified)
Assignee _Customer Call_ -- EDSC
Last Progress 05-Sep-2008 12:56 -- David Seddon