FUJ00081870
FUJ00081870
Peak Incident Management System
Call Reference PC0203085 Call Logger _ Customer Call_ -- EDSC
Release Targeted At -- HNG-X 03.21 Top Ref BAL_SRV_OSR_0321 D240
Call Type HNG-X Data Centre Raised Incidents/Defects Priority C -- Progress Restricted
Contact EDSC Call Status Closed -- Fix Released to Call Logger
Target Date 22/08/2010 Effort (Man Days) 0
Summary FAD126109 Pouch remmed in on two counters at same time
All References Type Value
Product Baseline BAL _SRV_OSR_ ROUTING _0321 D240
Other DB PATCH BAL_03 20 00.15 TO BAL_03 21 00 3
Product Baseline BAL_SRV_OSR_0321_D240
Release PEAK PC0207466
DevintRel-Director Live Supp.Test
Product Baseline BAL_SRV_OSR_0321_V240
SSCKEL KEL acha42210
Product Baseline BAL_SRV_OSR_ROUTING_0321_V240
TRIOLE for Service 2736482
Impact
5 iat & User Date
Unknown 21-Oct-2010 17:59:23
The same pouch can be remmed in to the system more than once, resulting in a shortage at the branch
which POL have to rectify by issuing a Transaction Correction.
1. call has to be processed, and corrective action taken, by SSC, MSU and POL
2. visible to POL and the branch when it happens
3. very rare
Progress Narrative
te:17-Aug-2010 13:03:42 User: Customer Call_
ALL PC0203085 opened
Details entered are:~
Summary:rem has came in pm has scanned it and printed the recipt whi...
lrarget. Rele: x 01.00
Routed to:EDSC - _Unassigned_
jats:17-Aug-2010 13:03:42 User: Customer Call_
[INCIDENT MANAGEMENT
jpate/Time Raised: Aug 17 2010 12:45PM
Priority: ¢
contact Nam
contact Phone:
riginator: XX)
riginator's refer
roduct Serial No:
Product Site: 126109
rem has
came in pm has scanned it and printed the recipt which was blank so the pm has gone to another counter and has gone though
Incident: History:
0010-08-17 12:45:32 [ Steptoe, Jack]
HNIT : create a new request/incident /problem/change/issue
2010-08-17 12:48:02 [ Steptoe, Jack]
pneut_en_rmg : Transfer Notification
2010-08-17 12:48:02 [ Steptoe, Jack]
pen Notification
neun_en rmg ¢
=17 12:48:06 [ Steptoe, Jack]
M has done a weekly report and it is showing as double
on _1-350379 16/09/2010 remmed in £410
FUJ00081870
FUJ00081870
session 2-195226 16/09/2010 vemmed in £410
f010-08-17 12:49:41 [ Steptoe, Jack
FLD : FIEL flag’ OLD="NO' NI
0010-08-17 12:49:45 I Steptoe, Jack]
0G : pm has tried revering the 2nd session and gets message "the session was transacted in a mode that is not allowed to be
reversed"
ze
010-08-17 12:50:47 [ Steptoe, Jack]
.0G : pm states that she is now showing a loss
pm has now checked the weekly remm report which is showing it twice
2010-08-17 12:51:26 [ Steptoe, Jack’
IoG : pm has spoken to nbsc who walked the pm thought doing another weekly remm report and it showing the same thing
b010-08-17 12:52:45 [ Steptoe, Jack.
LOG : Node: 2
Jsername
f'ime: session 1 - 10:05 session 2 - 10:07
pate: 16208/2010
Ie: 5
Bp: 1
stock Unit: AA
full error message when trying to do the reversal: The session was trasacted in a mode that cannot be revered
ey stokes: back office, rem and transfers , pouch delivery, then the pm scaned the barcode
010-08-17 12:56:59 [ Steptoe, Jack
L0G : was unable to look for any kels
2010-08-17 12:57:14 [ Steptoe, Jack
L0G : PEAK: please investigate why the pm is unable to reverse the rem which has been scanned in twice
(010-08-17 13:01:30 [ Steptoe, Jack
LoG : call checked by Rachel
lef given
2010-08-17 13:02:40 [ Steptoe, Jack:
lneut_en_rmg : Transfer Notification
010-08-17 13:02:41 [ Steptoe, Jack
FR : Transfer assignee from ‘Steptoe, Jack’ to
transfer group from 'HSH6' to ‘PEAK
17-Aug-2010 13:04:27 User:Lorraine Elliott
fhe call stimmary has been changed from:
em has came in pm has scanned it and printed the recipt wh.
the call summary is now:~
FAD126109 pm is unable to reverse the rem
Jate:47-Aug-2010 13:04:37 User:Lorraine Elliott
Product EPOSS & DeskTop -- Counter Common (version unspecified) added.
joate:17-Aug-2010 13:05:44 User: Lorraine Elliott
[the Call record has been assigned to the Team Member: Anne Chambers
progress was delivered to Consumer
Tori 7-Aug-2010 15:48:59 User:Anne Chambers
[Start of Response]
fh cash pouch was remmed in twice at branch 126109:
Pouch barcode 399347067204
bp coin £60
0p coin £250
bp coin £100
ion 1-350379 16/09/2010 10:08
Session 2-195226 16/09/2010 10
[fhe PM cannot reverse the transaction since rem reversal isn't allowed
[This is NOT another example of the duplicate rem problem that we have ssn in the past, where use of the Prev key accepted the same
wich twice. In this case the pouch was processed on both counters...
2 get pouch status, retrieve pouch details
el get pouch status, retrieve pouch details
@2 settle pouch delivery
1 settle pouch delivery
FUJ00081870
FUJ00081870
here were some printer problems on counter 2 which probably explain why this was done
please send this info to POL via BIMS, because the branch now has a shortage of £410 as a result of this double rem
need a correction. Then return the call to me and I'll get development to check whether it is working as intended.
[End of Response]
esponse code to call type J as Category 40 -- Pending -- Incident Under Investigation
Response was delivered to Consumer
in, and will
[Date:i7-Aug-2010 15:49:13 Uscr:Anne Chambers
[fhe Call record has been transferred to the team: MSU-Indt Mgt
rogress was delivered to Consumer
ate:18-Aug-2010 1!
[thanks Anne,
6:30 User:Andrew Nash
Final BIMS issued to POL. Passing back to SSC, for attention of Anne.
jDate:48-Aug-2010 15:46:36 Uscr:Andrew Nash
Ihe Call record has been transferred to the team: EDSC
Progress was delivered to Consumer
Date:18-Aug-2010 15:48:09 Uscr:Anne Chambers
[the Call record has been assigned to the Team Member: Anne Chambers
rogress was delivered to Consumer
jate:18-Aug-2010 18:31:40 Uscr:Anne Chambers
Evidence Added - log extracts.txt - File will be Obfuscated and made available to download soon.
jabe:i8-Aug-2010 18:33:19 Uscr:Anne Chambers
I:vidence Obfuscated File (CTRMSG) Added - Ext:
e:18-Aug-2010 18:35:06 User:Anne Chambers
[Start of Response]
POL have been notified that the problem has caused a shortage at the branch, Sending
11 to development for further
[End of Response]
lesponse code to call type J as Category 40 -- Pending -~ Incident Under Investigation
lesponse was delivered to Consumer
investigation.
Date :18-Aug-2010 16:35:20 Uscr:Anne Chambers
It checked whether there were any exceptions in the BAL OSR logs for any of the messages, there was nothing.
Gareth Jenkins thinks that it should not be possible to complete the rem in on both counters. Please investigate.
18-Aug-2010 18:37:26 User:Anne Chambers
all summary has been changed from:-
FAD126109 pm is unable to reverse the rem
Ihe call summary is now:
ffAD126109 Pouch remmed in on two counters at same time
Date:18-Aug-2010 18:37:37 Uscr:Anne Chambers
[the Call record has been transferred to the team: xCtr GDC
ser:Anne Chambers Confirmed that this Incident may be passed to the external company with the attached evidence.
ogress was delivered to Consumer
Date: 19-Aug-2010 07:41:24 User:Suresh Chitikela
the Call record has been transferred to the team: _CSM_
he Call record has been assigned to the Team Member: Chaitanya Pothapragada
rogress was delivered to Consumer
Date:19-Aug-2010 09:10:07 User:Steven Porter
It think I have seen a similar peak relating to race condition with rem’ing of poucl
JDate:49-Aug-2010 10:54:22 Uscr:Chaitanya Pothapragada
tam looking into this and update will the root cause by EOD today.
jbate:19-Aug-2010 15:
roduct. HING-X Counter
0:14 User:Chaitanya Pothapragada
BUC-Cash&Stock (version unspecified) added.
FUJ00081870
FUJ00081870
e:19-Aug-2010 15:09:09 User:Chaitanya Pothapragada
Problem
Ihe issue is with race condition for auto remin pouch delivery. For auto remin, the pouch details should be present in
ILFS_RDC_HEADER
TR 1
scan the pouchid, preview the Delivery Receipt and come back to
print/preview/LateDate. Now, if enter is hit here, auto remin completes
uaccessfully. Donot hit enter and leave the system here,
TR 2
scan the same pouch id used in CTR 1 above and complete the remin successfully.
CTR 1
low at. print/preview/LateDate, when enter is hit to complete remin, the remin
completes successfully resulting in duplicate lines in RSD.
expected Result is MSG90955 - Settlement Error with Error code
POUCH DELIVERY ALREADY PROCESSED ERROR".
oot Cause & Solution
hen an auto remin pouch id is settled successfully, the system updates the
COUNTER READ TIMESTAMP in LFS RDC HEADER to a not null value for that pouch
5
The race condition for auto remin pouch delivery is handled at
sett lePouchDel iveryServiceSett lement Processor.processPouch () «
[this method checks during settlement whether the COUNTER READ TIMESTAMP in
FS_RDC_HEADER is null or not null value.
Ite null, the pouch id is good and settlement completes successfully.
lif not null, the pouch id is already processed and error is thrown.
fhe query that gets the COUNTER READ TIMESTAMP from LFS RDC HEADER is
t Set lePouchDel iveryPreCheck'.
in this query, the input parameter for pouch id is defined incorrectly.
Itt is given "pouchBarcode[String]", but in dyno the pouch id is “pouchia".
his is the root cause why the query always returns null although the
COUNTER READ TIMESTAMP is not null.
Ihe solution is to define the pouch id input param to “pouchId[String]" in
t Set t lePouchDeliveryPreCheck'.
VELOPMENT IMPACT OF FIX:
PECIFY THE HNG-X PLATFORMS IMPACTED:
oduct Group : HNG-X Counter
Product : BUC-CashaStock
IECHNICAL SUMMARY:
3 above
LIST OF KNOWN DIMENSIONS DESIGN PARTS AFFECTED BY THE CHANGE
jone
ARE ANY OF THESE DESIGN PARTS AFFECTED BY APPROVED CPs/PEAKS in HNGX Release 2:
ione
RELEASE 2 IMPACT:
DEPENDENCIES:
jone
DOES THE FIX REQUIRE ANY MANUAL DEPLOYMENT BASELINES:
ione
DEV EFFORT IN MANDAYS:
Ikess than 1 one day for dev and testing.
IMPACT ON USER:
Ithe race condition will be operative for auto remin pouch delivery. A pouch id cannot be settled by two users at a time.
JiMPACT ON OPERATIONS:
jone
HAVE RELEVANT KELS BEEN CREATED OR UPDATED?
lone
IMPACT ON HORIZON TO HNGX BRANCH MIGRATIONS
lone
FUJ00081870
FUJ00081870
FMPACT ON TEST?
[the test scenario is described above.
SKS (of releasing and of not releasing proposed fix):
this is low risk and low complex issue.
tS? OF LIKELY DELIVERABLI
jpdated sql of SettlePouchDeliveryPreCheck.
[Date:49-Aug-2010 15:09:41 Uscr:Chaitanya Pothapragada
ction placed on Team:xCtr_C , User:Chaitanya Pothapragada
‘e:19-Aug-2010 15:09:49 User:Chaitanya Pothapragada
fhe Call record has been assigned to the Team Member: Suresh Chitikela
Progress was delivered to Consumer
jbate:20-Aug-2010 13:25:53 User:Suresh Chitikela
i agree with analysis of the root cause.
[Datc:20-Aug-2010 13:25:59 Uscr:Suresh Chitikela
Ihe Call record has been assigned to the Team Member: Chaitanya Pothapragada
Progress was delivered to Consumer
ate: 25-Aug-2010 1!
[Start of Response]
9:21 User:Chaitanya Pothapragada
[End of Response]
Response code to call type J as
fiesponse was delivered to Consumer
ategory 41 -- Pending -- Product Error Diagnosed
[Datc:i3-Sep-2010 13:26:36 Uscr:Chaitanya Pothapragada
Evidence Added - Solutio
Sion Heke
jbate:13-Sep-2010 13:29:33 User:Chaitanya Pothapragada
[Start of Response]
INGX CODE FIX
FIX DESCRIPTION
the issue is with race condition for auto remin pouch delivery.
Ithe query that gets the COUNTER READ TIMESTAMP from LFS_RDC_HEADE!
! sett. lePouchDeliveryPreCheck'.
in this query, the input parameter for pouch id is defined incorrectly.
it is given “pouchBarcode[String]", but in dyno the pouch id is “pouchIa".
[this is the root cause why the query always returns null although the
COUNTER READ TIMESTAMP is not null.
[fhe solution is to define the pouch id input param to “pouchId[String]" in
h SettlePouchDeliveryPreCheck'.
is
PROPOSED BRANCH
Bp.
SOUNTER JAVA FILES CHANGED
SOUNTER PDL FILES CHANGED
[COUNTER REFDATA FILE:
jone.
CHANGED
SHARED CODE FILES CHANGED
BAL, JAVA CODE FILES
jone.
CHANGED
SOL FILES CHANGED
Set t lePouchDel iveryPreCheck
[OTHER FILES CHANCED
one.
PROPRIATE CODE COMMENTS
es.
DEPENDENCIES
jone.
RELATED PROBLEMS
FUJ00081870
FUJ00081870
jones
UNIT TESTING EVIDENCE
EGRESSION TES
PCO203085 RemInCashTest.
IBACKWARDS COMPATIBILITY
A.
DEVELOPMENT DOCUMENTATION
jone.
QUIREMENTS
lone.
HELP
one.
[End of Response]
Response code to call type J as Category 46 -~ Pending —- Product Error Fixed
esponse was delivered to Consumer
jate:13-Sep-2010 13:30:00 User:Chaitanya Pothapragada
fhe Call record has been assigned to the Team Member: Suresh Chitikela
Progress was delivered to Consumer
Suresh Chitikela
Date: 7-Sep-2010 14:37:15 User
and the code patch is resolving the issue.
Ir tested the patch in my workspa:
leql patch is updating the Column SQL STATEMENT addition to IN Params as mentioned in PC019151.
[Date:17-Sep-2010 1
evidence Added -
[Date:47-Sep-2010 14:39:41 Uscr:Suresh Chitikela
The Call record has been assigned to the Team Member: Chaitanya Pothapragada
Progress was delivered to Consumer
ate:20-Oct-2010 15:27:39 User:Suresh Chitikela
new Business Impact has been added:
impact of this peak is the same pouch can be rem in to the
emin amount
system for more than once which can make the branch shortage of the
[this issue is in particular when two counters try to remin the pouch at the same time .in all these cases we can see an occurance
pf this incident.
this affects all the branches where pouch remin happens at the same time at two counters
6 User:Anne Chambers
ha42219
jate:a@i-oct-2010 1
eference Addec
[Date:21-Oct-2010 17:59:23 Uscr:Anne Chambers
The Busini Impact has been updated:
[the same pouch can be remmed in to the system more than once, resulting in a shortage at the branch which POL have to rectify by
issuing a Transaction Correction.
Hh. call has to be processed, and corrective action taken, by SSC, MSU and POL
visible to POL and the branch when it happens
very rare
e:22-Oct-2010 11:48:20 User:Prashanth Pamidimukkala
Product HNG-X Platforms -- HNG-X Counter (CNT) (version unspecified) added.
2 User:Prashanth Pamidimukkala
BUC-CashsStock deleted.
[Date:22-Oct-2010 11:
Product HNG-X Counter
ate:22-Oct-2010 11:49:34 User:Prashanth Pamidimukkala
oduct HING-X Platforms -- HNG-X Counter (CNT) updated to Subject.
Date:22-Oct-2010 11:49:36 User:Prashanth Pamidimukkala
roduct EPOSS & DeskTop -- Counter Common updated to Subject.
FUJ00081870
FUJ00081870
[oate:22-Oct-2010 11:50:32 UscriPrashanth Pamidimukkala
product HNG-X Platforms -- BAL Server (BAL) (version unspecified) added.
‘Te:22-Oct-2010 1
roduct. TING
0:45 User:Prashanth Pamidimukkala
X Platforms -- HNG-X Counter (CNT) deleted.
Date:22-Oct-2010 12:09:22 User:Prashanth Pamidimukkala
[fhe call Target Release has been moved to Proposed For ~- HNG-X 03.20
jate:25-Oct-2010 12:03:08 Uscr:Prashanth Pamidimukkala
ction has been removed from the call
Date:25-Oct-2010 12:13:52 User:Prashanth Pamidimukkala
tion placed on Team:RelMngmntForum
jbate:27-Oct-2010 12:03:12 User:dohn Boston
RMF 26/10/10 Agreed ~ to target at 03.17.
jate:27-Oct-2010 12:03:31 User:John Boston
fhe call Target Release has been moved to Targeted At -- HNG-X 03.17
Te:27-Oct-2010 12:03:45 Uscr:John Boston
ction has been removed from the call
[Date:28-Oct-2010 12:08:14 User:John Boston
[the call Target Release has been moved to Targeted At -- HNG-X 03.21
Date:28-Oct-2010 12:08:22 User:dohn Boston
s agreed by Release Planning group all 03.17 Peaks retargeted at 03.21.
Date:08-Dec-2010 10:04:55 Uscr:Chaitanya Pothapragada
ction placed on Team:xCtr_CSM GDC, User:Chaitanya Pothapragada
[Date:08-Dec-2010 10:05:41 lscr:Chaitanya Pothapragada
the Call record has been transferred to the team: xCtr_REL GDC
Progress was delivered to Consumer
Jpate:43-Dec-2010 14:50:13 Uscr:Kishor GaneshRao
Rig: 4LS CIT
elease: BALO3.21.00.00
saseline: DB PATCH BAL 03 20 00 15
tO_BAL_03_21_00_3.sql
[his peak has been successfully verified and observed that now we are getting the expected message i.e. MSG90955 when performed the
low scenario.
eference: Test evidence is attached for further referenc
enario ty
tec
pre-requisite for test dat
[there should be an auto Rem in pouch id available in the database, which can be loaded from the attached SQL script.
oTR 1
an the pouch id and preview the Delivery Receipt and come back to
rint/preview/Late Date. Now, if enter is hit here, auto Rem in completes
uccessfully. Do not hit enter and leave the system here.
TR 2
Scan the same pouch id used in CTR 1 above and complete the Rem in s
asfully.
jow at _print/preview/Late Date, when enter is hit to complete Rem in, the Rem in
FUJ00081870
FUJ00081870
ompletes s
ecesstully resulting in duplicate lines in RSD:
expected Result:
SG90955 should be displayed - Settlement Error with Error code
"POUCH DELIVERY ALREADY PROCESSED ERROR".
Hissue/actual result:
[fhe issue was with race condition for auto Rem in pouch delivery
IFS RDC HEADER
For auto Rem in,
the pouch details should be present in
[Date:13-Dec-2010 1
evidence Added -
1:28 User:Kishor GaneshRao
£
[Datc:43-Dec-2010 14:56:12 Uscr:Kishor GaneshRao
ividence Added - SQL script for the auto remin
bate:13-Dec-2010 14:57:38 User
eference Added: Other DB_PAT
shor GaneshRao
H_BAL_03_20 00 15 TO BAL 03 21 00 3
Date:13-Dee-2010 14:
TOP Reference set to
7:41 User:Kishor GaneshRao
Other DB_PATCH BAL 03 20 00.15 TO BAL 03 _21 00 3
jate:22-Dec-2010 09:30:03 User :PiT Automated User
eference Added: Product Baseline BAL SRV_OSR 0321 _v240
Jato: 22-Dec-2010 09:35:04 Uscr: Pil Automated User
eference Added: Product Baseline BAL SRV_OSR ROUTING 0321 v240
te:22-Dec-2010 1
3:59 User:Pavan Vejendla
“IX RELEASE CHECKLIS
"IX IMPACT TEMPLATE
ULLY COMPLETED?
RMF APPROVAL GRANTED?
es
CODE REVIEW TEMPLAt
Yes
FULLY COMPLETED?
CODE REVIEW PASSED?
es
SODE CHECK-IN BRANCH SPECIFIED?
ED-IN FILES HAVE PEAK REFERENCE IN
COMMEN
FIX PASSED CIT AND REGRESSION TESTING?
EL UPDATED TO REFLECT PEAK FIX?
CIT TEST SCRIPT UPDATED TO REFLE
es
PEAK?
IT TEST RESULTS PUBLISHED?
[Date:22-Dec-2010 16:35:04 Uscr:PIn Automated User
lkeference Added: Product Baseline BAL SRV _OSR ROUTING 0321 D240
‘Te:22-Dec-2010 16:45:04 Uscr:PiT Automated User
eference Added: Product Baseline BAL SRV. 0321_D240
Jate:23-Dec-2010 09:24:39 User:Pavan Vejendla
the Call record has been transferred to the team: Dev-Int-Rel
Jser:Pawan Vejendla Confirmed that this Incident may be passed to the external company with the attached evidence.
rogress was delivered to Consumer
FUJ00081870
FUJ00081870
Date:05-Jan-2011 08:25:03 User:Lionel Higman
[Start of Response]
[End of Response]
esponse code to call type J as Category 49
esponse was delivered to Consumer
[the Call record has been transferred to the team: Live Support Team
he Call record has been assigned to the Team Member: Unassigned_
Date:21-Jan-2011 13:42:43 User:Sheila Bamber
[the Call record has been assigned to the Team Member: Release to Live
@-Mar-2011 15:21:52 Uscr:Mark Ascott
Ihe Call record has been transferred to the team: RM-x
Progress was delivered to Consumer
Jbate:03-May-2011 14:40:58 User:gohn Budworth
[Start of Response]
[fhe 03.21 BAL upgrade was applied to live as part of PSPID Release HRU7284B PR for 03.21.00.50 23/1/2011. See release PEAK 207466.
[End of Response]
Response code to call type J as Category 71 -- Final -- Fix Released to Call Logger
outing to Call Logger following Final Progress update.
esponse was delivered to Consumer
jefect cause updated to 14 -- Development - Code
Date:03-May-2011 14:41:16 User:John Budworth
eference Added: Bel BAK _PCO207466
[Date:03-May-2011 14:48:32 Usecr:Lorraine Elliott
[fhe Call record has been assigned to the Team Member: Anne Chambers
progress was delivered to Consumer
joate:03-May-2011 15:20:43 User:Anne Chambers
[Start of Response]
Code fix applied to live 23/1/2011. Please close call.
[End of Response]
esponse code to call type J as Category 71 -- Final
outing to Call Logger following Final Progress updat
Service Response was delivered to Consumer
Fix Released to Call Logger
Jate:03-May-2011 15:20:43 Uscr:Anne Chambers
FALL PC0203085 closed: Category 71 Type J
Root Cause Development - Code
Logger _Customer Call_ -- EDSC
Subject Product EPOSS & DeskTop -- Counter Common (version unspecified)
Assignee : _Customer Call_ -- EDSC
Last Progress 03-May-2011 15:20 -- Anne Chambers