FUJ00086586
FUJ00086586
Peak Incident Management System
Call Reference PC0195561 Call Logger _Customer Call_ -- EDSC
Release Targeted At -- HNG-X 01.22.01 Top Ref 2091569
Call Type Live Incidents/Defects Priority __ B -- Business restricted :
Contact EDSC Call Status Closed -- S/W Fix Available to Call Logger
Target Date 07/03/2010 Effort (Man Days) 0
Summary FAD226542 pm was issued with 2 x 4000pds receipts
All References Type Value
DevIntRel-Director ITU SV&I
OCP OCP. 25882
TRIOLE for Service 2091569
SSCKEL KEL cardce262S
SSCKEL KEL carde262S
Collections ‘Name User Date
PrescanCounter Lorraine Elliott 15-Apr-2010 08:05:13
Progress Narrative
fe:04-Mar-2010 12:50:40 User: customer Call_
CALL PC0195561 opened
Details entered are:-
summary:see call 2083169. pm was trying to tansfer out 4000 pds. the...
call Type:L
call Priority:aA
ffarget Release: 786
Routed to:EDSC - _Unassigned_
JDate:04-Mar-2010 12:50:40 User: Customer Call_
INCIDENT MANAGEMENT
Date/Time Raised: Mar 4 2010 12:33PM
riority: A
contact Name:
Kontact Phon
wiginator: XXXXKX@TFSO1
riginator's reference: 2091569
Product Serial No:
product Site: 226542
sce call 2083169. pm was trying to tansfer out 4000 pds. the system crashed. pm was issued with 2 x 4000pds receipts.
lent History:
b010-03-04 12:33:39 [ Vasse, Anthony]
INIT : create a new request /incident /problem/change/issue
010-03-04 12:36:08 [ Vasse, Anthony]
neun_en_rmg : Open Notification
b010-03-04 1)
eneut en rmg
6:08 [ Vasse, Anthony]
Transfer Notification
10-03-04 12:36:20 [ Vasse, Anthony]
0G : nbsc said that the error should have resolved itself overnight.
b010-03-04 1:
FLD
o:41 [ Vasse, Anthony]
FIELD='zcbflag' OLD="NO' NEW='YES"
2010-03-04 12:36:50 [ Vasse, Anthony]
toc : pm is horizon online.
username CDI001
tock Unit BB individual
transferring BB into MS shared main stock unit
he transfer out receipt came up at 1506 on the 2/3
n got two receipts for 4000pds.
hey have the same =
session 2-192266 F/590/1
sion number on them.
FUJ00086586
FUJ00086586
fiessage at bottom of receipt reads “this session way oF may hot have worked
Itt will not be recovered. You will need to check its effect when you next log in.
lt 1615 pm logged into ms on the transfer screen.
bm had two transfers at 4000pds each to complete.
jsoth had the same session id.
h-537246
2010-03-04 12:41:59 [ Vasse, Anthony]
ILD : FIELD='category’ OLD='RMGA.S Software.SD21 Reported software error’ NEW='RMGA.S Software.SD22 HNG Migration Issue.CB3 HNG
ligration issue’
010-03-04 12:42:04 [ Vasse, Anthony]
LOG : pm transferred one amount back to stock unit bb.
session id 1-537252 at 16:16
transfer in slip for stock unit bb 2-192288
lm did a cash declaration in MS that was fine.
bm did a cash declartion in BB. that was out by 4000 pounds. ( a loss of 4000 pds.)
2010-03-04 12:45:34 [ Vasse, Anthony]
LOG : pm is not sure what has happend with a 4000 pds transie
om is now showing a 4000pds loss in stock unit BB.
Please see details.
had a system crash when she was completing the transfer.
2010-03-04 12:47:08 [ Vasse, Anthony]
LOG : pm had a number of crashes yesterday "
unable to contact the data center errors
m is not back in office until 1400 today.
010-03-04 12:49:32 [ Vasse, Anthony]
[IR : Transfer ‘group’ from 'HSH6' to ‘PEAK’
2010-03-04 12:49:32 [ Vasse, Anthony]
pneut_en_rmg : Transfer Notification
jDate:04-Mar-2010 12:59:33 User:Lorraine Elliott
oduct EPOSS & DeskTop -- Counter Common (version unspecified) added.
‘e:04-Mar-2010 12:59:54 Uscr:Lorraine Elliott
he call summary has been changed from
ee call 2083169. pm was trying to tansfer out 4000 pds. the...
fthe call summary is now:—
FAD226542 pm was issued with 2 x 4000pds receipts
€e:04-Mar-2010 13:13:36 User:Lorraine Elliott
the Call record has been assigned to the Team Member: Cheryl Card
progress was delivered to Consumer
Pate:05-Mar-2010 1
[Start of Response]
yn. 02/03/10 on counter 2 at 15:04, the clerk attempted a Transfer Out of 4000.00 from stock unit BB to MS. Due to a system problem,
he Transfer Out doubled up, so when the Transfer In was done on counter 1 at 16:15, it was for 8000.00. The branch now has a loss of
1000.00.
8:10 User:Gheryl Card
JI phoned the PM and explained that the problem was under investigation. The PM would like to have it sorted out before she rolls into
the next TP, which is due on Wed 17th March.
[End of Response]
esponse code to call type 1 as Category 40 ~~ Pending -~ Incident Under Investigation
esponse was delivered to Consumer
[Date:09-Max-2010 1
KEL_cardc262S author
Date:i0-Mar-2010 08:43:45 Uscr:Gheryl card
[Start of Response]
Ihe PM attempted a Transfer Out of 74000. The settlement request timed out and the PM was logged out. Two Transfer Out slips were
printed, each with the following text at the bottom:
[this session may or may not have worked. It will not be recovered. You will need to check its effect when you next login.
hen the PM did the corresponding Transfer In, it was for ?8000 instead of the expected 24000. The branch now has a loss of ?4000.
[fhe Transfer Reconciliation Report, printed by the clerk, contains:
192266 BB MS 2 02-Mar 15:04 TO 8000.00-
N-537246 BB MS 2 02-Mar 16:15 TI 8000.00
elevant counter logs and database extracts are attached.
[End of Response]
tesponse code to call type L as Category 40 —- Pending -~ Incident Under Investigation F/590/2
FUJ00086586
FUJ00086586
Response was delivered to Consumer
Pe:10-Mar-2010 08:44:53 User:Cheryl Card
lsvidence Added - 9. and_logs_for bra
1226542 on 02/03/10
Date:10-Mar-2010 08:51:33 User:Cheryl Card
[Start of Response]
fter discussion with Gareth Jenkins, the suggested correction is to negate the duplicate transfer out by writing 2 lines to the
RDB_RX_REP_SESSION and BRDB RX EPOSS TRANSACTIONS tables, with:
jl) Product 1, Quantity 1, Amount 4000.00, Counter mode id 7 (TI)
) Product 6276, Quantity -1, Amount -4000.00, Counter mode id 7 (tI)
Ihis should be done using the Transaction Correction tool. An OCP approved by POL will be needed.
[End of Response]
Response code to call type L as Category 40 -- Pending
esponse was delivered to Consumer
Incident Under Investigation
te:ii-Mar-2010 1.
[Start of Response]
Ii spoke to the PM yesterday about the proposed repair.
5:17 User: Cheryl Cara
fhe source stock unit BB is down by 4000 and has not been used since the problem occurred. The destination stock unit MS has balanced
orrectly and has rolled into a new BP. Therefore I propose to make the correction by writing 2 additional records to both
IsRDB_RX_REP SESSION and BRDB_RX_EPOSS TRANSACTIONS to transfer out an additional 4000 from BB as follows:
H) Product 1, Mode 13 (TO), Amount 4000, Quantity 1
k) Product 6277, Mode 13, Amount -4000, Quantity -1
[this reflects how they have handled it at the branch, and also matches the 2 records in the BRDB SU PENDING TRANSFER DET table.
Ihe PM is waiting for me to do the repair, then they will roll the BB stock unit into a new BP this week (TP rollover is next week).
[End of Response]
kesponse code to call type L as Category 40 -- Pending -- Incident Under Investigation
kkesponse was delivered to Consumer
jDate:ii-Mar-2010 15:22:14 User:Cheryl cara
[Start of Response]
\dvised PM to print a balance snapshot. Ran the Transaction Correction tool (BRDBX015). Advised PM to print a balance snapshot again
lnd she confirmed that it now looks correct. She will now balance and do 2 BP rollovers to catch up with the other stock units. Will
eep call open until she has rolled into a new TP (17th March).
[End of Response]
esponse code to call type L as Category 40 -- Pending -- Incident Under Investigation
lkesponse was delivered to Consumer
jDate:ii-Max-2010 17:26:59 User
Cheryl Card
Isvidence Added - SQi updates and :
te _from the TC tool
jbate:12-Mar-2010 16:54:08 User:Cheryl Card
[Start of Response]
checked the BRDB DAILY SUMMARY table, which would have been populated overnight, and this shows the Transfer Out of 4000.00 against
products 1 and 6277 for stock unit BB.
[End of Response]
esponse code to call type L as Category 40 -- Pending -- Incident Under Investigation
esponse was delivered to Consumer
Dabe:i2-Mar-2010 17:25:00 User:Gheryl Card
Ihe call Priority has been changed from A
the call Priority is now B
[Date:12-Mar-2010 17:25:18 User:Gheryl card
[Start of Response]
educing priority to B now that the repair has been successfully done.
[End of Response]
esponse code to call type L as Category 40 -- Pending -- Incident Under Investigation
esponse was delivered to Consumer
ToriS-Mar-2010 12:29:40 User:Gheryl Card
[Start of Response]
on 02/03/10 between 15:03 and 15:06, on counter 2, the PM attempted to do a Transfer Out of 4000.00 cash from stock unit BB to MS.
[fhe settlement request timed out and the PM was logged out. Two Transfer Out slips were printed, each with the following text at the
ttom:
[this session may or may not have worked. It will not be recovered. You will need to check its effect when you next login.
BAL logs show 2 instances of:
equestEvent... for BasketSettlementService/SettleTransferOuthas timed out but is executing. Ignoring timeout
F/590/3
FUJ00086586
FUJ00086586
fiva entries for 4000.00 were written to table BROB SU PENDING TRANSFER DET, but only one entry should have bean written. This caused
[the branch a loss of 4000.00 (which has now been corrected using the Transaction Correction tool).
outing to 4th line for investigation into why the Transfer Out doubled up. Evidence (logs and database extract) attached.
[End of Response]
esponse code to call type L as Category 40 -- Pending -- Incident Under Investigation
eaponse was delivered to Consumer
[Date:45-Max-2010 12:29:50 Uscr:Cheryl Card
[fhe Call record has been transferred to the team: xCtr GDC
User:Cheryl Card Confirmed that this Incident may be passed to the external company with the attached evidence.
rogress was delivered to Consumer
te:iS-Mar-2010 12:34:08 User:Suresh Chitikela
he Call record has been transferred to the team: xCtr_CSM GDC
[fhe Call record has been assigned to the Team Member: Chaitanya Pothapragada
rrogress was delivered to Consumer
Jbate:i5-Mar-2010 1
[Start of Response]
[his looks like it would need to go into HNGX R1 at some point, and could not wait for R2.
1:16 User
teven Porter
[End of Response]
esponse code to call type L as Category 40 -- Pending -~ Incident Under Investigation
esponse was delivered to Consumer
[Date:47-Max-2010 07:14:46 Uscr:Chaitanya Pothapragada
[Start of Response]
hen the BAL service is timed out, the data used to commit to the database before.
ith the release of PC0194893 in CTRO25 10 HOTFIX, which resolved the issue of a transaction commiting data once a request has
ut. The fix will rollback the data.
le are trying to reproduce the issue in CTR024 04 HOTFIX to get the receipt twice and check the db if the transaction data is
committed and check the same in CTRO25 10 HOTFIX to find the transaction data not getting committed.
timed
[End of Response]
jesponse code to call type L as Category 40 -- Pending -- Incident Under Investigation
esponse was delivered to Consumer
[Date:47-Max-2010 08:34:18 User:Chaitanya Pothapragada
[Start of Response]
ranch used to reproduce the issue : CTRO24 04 HOTFIX
Steps to reproduce
tun the OSR in debug mode setting breakpoints.
the transfer out in counter and hit settle to hit the break point in OSR.
net the request timeout.
ISG10010 will be displayed on CTR.
iit Retry and let the request timeout for the second time.
low hit Cancel on MSG10010.
[his will print two transfer out receipt with same session id.
n successfully completing the printing of second receipt, ctr logs off.
jt am able to produce two receipts for the same transfer out transaction and found the transaction details in
rdb_su_pending transfer det and brdb_su pending transfer.
fter contacting OSR team, found that this issue is a duplicate of PCcO194893.
Pc0194893 went to release in CTRO25 10 HOTFIX
lot able to reproduce the issue in CTRO25 10 HOTFIX.
hen BAL request is timedout, data is not committed to the database.
[End of Response]
esponse code to call type b as Category 40 -- Pending -- Incident Under Investigation
esponse was delivered to Consumer
Date:i?-Maz-2010 08:34:30 Uscr:Chaitanya Pothapragada
Ihe Call record has been assigned to the Team Member: Suresh Chitikela
rogress was delivered to Consumer
Date:17-Mar-2010 01
[Start of Response]
hen the request timedout BAL used to commit the data to databse.This has been addressed as part of peak PC0194893.
8:32 User:Suresh Chitikela
nalysis looks fine
[End of Response]
Response code to call type L as Category 40 -~ Pending -- Incident Under Investigation
sponse was delivered to Consumer
F/590/4
FUJ00086586
FUJ00086586
[later 7-Mar-2010 08:49:55 UserSuresh Chitikela
ction placed on Team:xCtr_OSR SME, User:Steven Porter
e:17-Mar-2010 14:05:17 User:Martin Tonge
[Start of Response]
jot totally convinced you have re-created what occured in live. You are running breakpoints on the settlement request. This is
ifferent to what occured in live.
h: Settlement requests have journal entries and in your test scenario the second request would have failed because the fisrt request
jould have commited a journal entry and you would have a duplicate entry.
: In live one of the settlements would have failed because of the journal entry - if this was a genuine retry.
can you check your testing and get the OSR message logs to show that you have recreated what occured in live. My conscern is that
somthing else has occured here. One thing that is bothering me is cash declaraion on with respect to 2 different stock units.
[End of Response]
Response code to call type L as Category 40 -- Pending
esponse was delivered to Consumer
Incident Under Investigation
Eo:d7-Mar-2010 14:19:54 User:Martin Tonge
[Start of Response]
jot totally convinced you have re-created this with what occured in live.
settlement requests have journal entries and in your test scenario the second request would have failed because the first request
ould have commited a journal entry and you would errored with a duplicate entry with the second request.
tf as in live the retry was committed before the first request completed then the first request would have failed with a duplicate
journal entry.
can you check your testing and get the OSR message logs to demonstrate what happended. Although the DB was the trigger for this and
he hew OSR behaviour may avoid this issue - I'm not convinced we understand how the duplicate occured.
leeds more analysis. Please ring if it will to discuss this one.
(End of Response]
lesponse code to call type L as Category 40 -- Pending —- Incident Under Investigation
esponse was delivered to Consumer
te: 7-Mar-2010 14:20:15 User:Martin Tonge
ction has been removed from the call
te:18-Mar-2010 11:45:59 Use!
[Start of Response]
It am working on this and will put my analysis here in a shortwhile.
Chaitanya Pothapragada
ls the fix is already released, I would like to request the priority of the issue be downgraded to € as we are trying to investigate
the root cause only. Please let me know your thoughts on this.
[End of Response]
esponse code to call type lL as Category 40 -- Pending ~~ Incident Under Investigation
esponse was delivered to Consumer
Jbate:18-Mar-2010 11:46:35 User:Chaitanya Pothapragada
the Call record has been transferred to the team: xCtr_OSR_SME
[fhe Call record has been assigned to the Team Member: Martin Tonge
User:Chaitanya Pothapragada Confirmed that this Incident may be passéd to the external company with the attached evidence
rogress was delivered to Consumer
Jbate:18-Mar-2010 13:17:54 User:Martin Tonge
[Start of Response]
hat is missing from this Peek is an explanation of the events in terms of the requests, how they were ordered and when any was
lcomitted. Only then can we qualify the priority. The assumption is that we have a fix. The facts are -
h
A settement request to timed.
: A retry of request timeout occured.
: According to the DB entries both later succeeded.
low unlike other reconciallation Peeks this stands out because only one of the requests are specific to settlement. They she
prkad Kecaliae there Gould have becn @ unique cometeaine Giglacion on the (even! entey of one of thew and 12 ve are not ge F/S90/5i¢
FUJ00086586
FUJ00086586
hen this 1s Still an issn
o what next ~
le can't reduce the priority unless we understand what is going on.
Hh: Need to look at the logs and work out the sequence of events.
b: If you have genuinely recreated this issue diring test. We also need to work out the sequence of events and to determine why no
exception was thrown.
[End of Response]
esponse code to call type L as Category 40 -- Pending -- Incident Under Investigation
was delivered to Consumer
JDate:18-Mar-2010 13:18:05 User:Martin Tonge
the Call record has been assigned to the Team Member: Martin Tonge
rogress was delivered to Consumer
719:17 User:Martin Tonge
Whe Call record has been assigned to the Team Member: Martin Tonge
Progress was delivered to Consumer
fe:18-Mar-2010 13:19:32 User:Martin Tonge
Ihe Call record has been assigned to the ‘Team Member: Martin Tonge
ogress was delivered to Consumer
jate:18-Mar-2010 13:20:18 User:Martin Tonge
fhe Call record has been transferred to the team: xCtr_CSM_GDC
he Call record has been assigned to the Team Member: Chaitanya Pothapragada
ser:Martin Tonge Confirmed that this Incident may be passed to the external company with the attached evidence.
rogress was delivered to Consumer
jate:d@-Mar-2010 14:
[Start of Response]
es, the second request is failing and the first request is committing the data in db.
6:04 User:Chaitanya Pothapragada
[the below message is seen in the logs :
010-03-18 10:01:01,296 UTC [settlement queue pool-13-thread-9] com. fujitsu.poa.bals.filters.MessageJournalFilter INFO [3407-2-vU-
1809-19] - [3407]- Retry attempted for a request that already succeeded - BasketSettlementService/SettleTransferout from counter [2]
branch code{3407]. No further processing required
lease find attached the message logs.
nen a transfer out is done to another SU, the data is committed in two tables, BRDB SU PENDING TRANSFER and
IBRDB SU PENDING TRANSFER DET , but here the duplicate row is added only in BRDB_SU PENDING TRANSFER DET which contains the details of
the transaction.
tam not able to get the duplicate row in BRDB SU PENDING TRANSFER DET in my testing.
i would like to request for more logs to investigate furthur as to how the duplicate row in BRDB_SU PENDING TRANSFER DET is added in
hive.
i)Need OSR message log for the following request ids :
26542-2-KH-0215=2
b26542-2-xH-0215-3
b26542-2-xH-0215-4
'2.6542-2-KH-0215-5.
ii)Also the “osr.log" file for the following OSR instances
hiprpbal005 : 20560
Aprpba1008 : 20560!
iprpbal006:20560" for the day 02-MAR-2010 from 15:00 to 15:20 for the branch id "226542" and counter id "2".
[End of Response]
esponse code to call type L as Category 40 -- Pending -- Incident Under Investigation
Response was delivered to Consumer
Joate:18-Mar-2010 14:32:21 User:chaitanya Pothapragada
[Start of Response]
lease provide the below information to proceed furthur on the issue :
li)Need OSR message log for the following request ids :
b26542-2-xH-0215-2
b26542-2-xH-0215-3
b26542-2-xH-0215~4
b2654
H.i)Also the “osr.log" file for the following OSR instances :
‘prpba1005:20560"
‘Iprpbal008:20560"
Iprpbal006:20560" for the day 02-MAR-2010 from 15:00 to 15:20 for the branch id "226542" and counter id "2".
F/590/6
FUJ00086586
FUJ00086586
ffTiy DB extract of RDB SU PENDING TRANSFER DET Including the full data of all the columns for the rows given below, which you have
rovided earlier.
efore it was BRDB_SU PENDING TRANSFER DET:
INSERT TIMESTAMP PROD ID AMOUNT QUANTITY IS BUREAU
2-MAR-2010 15:08:22 1 4000 1N
2-MAR-2010 15:08:31 1 4000 1N
[End of Response]
Response code to call type 1 as Category 40 -- Pending
lesponse was delivered to Consumer
Incident Under Investigation
joate:18-Mar-2010 14:33:37 User:Chaitanya Pothapragada
Ihe Call record has been transferred to the team: EDSC
jser:Chaitanya Pothapragada Confirmed that this Incident may be passed to the external company with the attached evidence.
lprogress was delivered to Consumer
e:d8-Mar-2010 15:47:48 Uscr:Mark Wright
he Call record has been assigned to the Team Member: Cheryl Card
rogress was delivered to Consumer
19-Mar-2010 14:16:06 Uscr:Cheryl Card
rested
fo:i9-Mar-2010 14:18:02 User:Gheryl card
fhe Call record has been transferred to the team: xCtr_CSM GDC
fhe Call record has been assigned to the Team Member: Chaitanya Pothapragada
Jser:Cheryl Card Confirmed that this Incident may be passed to the external company with the atta:
Progress was delivered to Consumer
ed evidence.
[bate:24-Max-2010 1
[Start of Response]
Iimeouts were the underying cause of the issue and that there were long delays waiting on the DB to process the 4 request:
eeded.
5:49 User:Chaitanya Pothapragada
in this case two of the requests were commited and two correctly detected that the transaction had already
[there is an issue with the 2 commits because this shoudn't have happened.
fowever the behaviour of the OSR from CTR25.07 onwards is to roll the
transaction back on a timeout. In this scenario all the requests would have
cailed and no reconciliation is required.
le would like to find the root cause of the issue as to how the duplicate entry was committed in the db.
in consultation with FJ SME, request to kindly downgrade the priority to C.
[End of Response]
esponse code to call type Las Category 40 -- Pending -- Incident Under Investigation
esponse was delivered to Consumer
Te:24-Mar-2010 15:30:48 Uscr:Chaitanya Pothapragada
[Start of Response]
3 per above, please downgrade the priority to C.
ror furthur investigating the root cause of the issue, request to provide the below
H. For branch accounting code = 226542, counter id = 2, JSN = 3192266
Please provide the message journal record(JOURNAL XML) in
IBRDB RX MESSAGE JOURNAL.
incase, if there are two records existing for the same JSN, pls provide both.
b. Also, provide the BAL request logs/message journal records for
previous (3192265) and next (3192267) JSN requests
thanks.
[End of Response]
esponse code to call type L as Category 40 -- Pending -- Incident Under Investigation
esponse was delivered to Consumer
Jbate:24-Mar-2010 15:35:54 User:Chaitanya Pothapragada
the Call record has been transferred to the team: EDSC
Pser:Chaitanya Pothapragada Confirmed that this Incident may be passed to the external company with the attached evidence.
Progress was delivered to Consumer
‘24-Mar-2010 15:37:38 User: Gheryl Gard
he call Priority has been changed from B
Ihe call Priority is now C
Date:24-Mar-2010 15:50:00 User
[fhe Call record has been assigned
jarran Avenell
Fo the Team Member: Cheryl Card F/590/7
FUJ00086586
FUJ00086586
Progress was delivered to Consumer
Po:25-Mar-2010 14:31:31 Uscr:_customer Call_
foiced Cheryl Card (SSC) and advised on the latest update in the call she states she was going to check if it was the same thing that
append before.
Ol are want to know why this has happend? = why does it keep happening? - can you advise on this.
thanks
[Date:25-Mar-2010 16:26:58 Uscr:Cheryl card
[Start of Response]
jave checked the logs and it does not appear to be a system error. Checked the buttonpress details in the PostOffieCounter log and
this shows that the clerk:
ressed Enter to select the transfer
pressed Print to print the Transfer In slip
Pressed Enter to accept the transfer - which also prints the Transfer In slip.
I checked the relevant database tables and they look consistent. Phoned the PM to explain why the Transfer In slip was printed twice.
She maintains that the stock unit is now down by 182.00. I agreed to do further checks tomorrow.
[End of Response]
sponse code to call type L as Category 40 -- Pending -- Incident Under Investigation
esponse was delivered to Consumer
jate:26-Mar-2010 11:56:43 User:Gheryl card
[Start of Response]
A transaction to the value of 182.00 was done in stock unit OOH on counter 1, on 20/03/10 at 09:23. A Transfer Out of 182.00 from OOH
to AA was done immediately afterwards.
[fhe corresponding Transfer In of 182.00 was done on counter 2 at 09:24.
Balance
Snapshot for OOH was done on counter 1 at 09:27 showing zero cash. The PM has confirmed that this is as expected.
A Balance Snapshot for AA was done on counter 2 at 12:17 showing 3285.01 cash. The previous balance snapshot was done on a previous
Hay. I phoned the PM and suggested she contact the NBSC to see if the transactions carried about between the 2 balance snapshots can
be checked.
[End of Response]
Response code to call type L as Category 40 -- Pending
Response was delivered to Consumer
Incident Under Investigation
Date: 27-Mar-2010 1
evidence Added - =:
User:Gheryl Card
om Journal and BAL logs
jbate:27-Mar-2010 10:40:24 User:Gheryl card
[Start of Response]
Extracts from Journal and BAL logs attached as requested by Chaitanya Pothapragada
[End of Response]
esponse code to call type L as Category 40 -- Pending -- Incident Under Investigation
lkesponse was delivered to Consumer
[Date:27-Max-2010 10:40:58 User:Gheryl Card
he Call record has been transferred to the team: xCtr_CSM GDC
[fhe Call record has been assigned to the Team Member: Chaitanya Pothapragada
User:Cheryl Card Confirmed that this Incident may be passed to the external company with the atta’
ogress was delivered to Consumer
ed evidence.
ate:44-Ape-2010 16:21:19 UscriChaitanya Pothapragada
[Start of Response]
equest to provide a few more logs to investigate the root cause of the issue.
lbc0195561 is now routed to you.
je need osr.log for Iprpbal005:20560 & lprpbal010:20559 from 15:04 to 15:10 for the day, 2010-03-02.
so, please run the query below and provide us the result :
SELECT insert timestamp
FROM brdb_rx_message journal
HERE branch accounting code =~ 226542
AND journal seq number = 3192266
[End of Response]
esponse code to call type L as Category 40 -- Pending -- Incident Under Investigation
esponse was delivered to Consumer
Date:14-Apr-2010 16:21:51 User:Chaitanya Pothapragada
Ihe Call record has been transferred to the team: EDSC
User:Chaitanya Pothapragada Confirmed that this Incident may be passed to the external company with the attached
Progress was delivered to Consumer
F/590/8_
FUJ00086586
FUJ00086586
[ateriS-Ape-2010 12°35:50 UscriaAnne Chambers
evidence Added - OSRlog extracts
jate:15-ApE-2010 12:39:16 Uscr:Anne Chambers
[Start of Response]
SELECT insert. timestamp
FROM brdb_rx_message_journal
WERE branch accounting code = 226542
AND journal seq number = 3192266
found nothing because the journal entries are not kept for long on the database. I've found the journal entry in theaudit file but T
an't see that it includes the brdb insert timestamp.
It ran a similar check on brdb rx_rep session data; the insert timestamp of the 2 messages was 02-MAR-10 15:08:22.4782
[End of Response]
esponse code to call type L as Category 40 -- Pending -- Incident Under Investigation
esponse was delivered to Consumer
Date:15-Apr-2010 12:51:53 User:Lorraine Elliott
Ihe Call record has been transferred to the team: xCtr_CSM GDC
he Call record has been assigned to the Team Member: Chaitanya Pothapragada
User:Lorraine Elliott Confirmed that this Incident may be passed to the external company with the attached evidence.
Progress was delivered to Consumer
jbate:22-Apr-2010 09:58:59 Uscr:Cheryl Card
the call Priority has been changed from C
Ihe call Priority is now B
[22-Apr-2010 10:00:46 User: Gheryl card
[Start of Response]
(hanging priority to B - since this involves a loss of 4000.00 cash and POL are now concerned that this problem may reoccur.
[End of Response]
esponse code to call type L as Category 40 -- Pending -- Incident Under Investigation
lkesponse was delivered to Consumer
te:22-Apr-2010 15:35:54 User:Chaitanya Pothapragada
[Start of Response]
It have gone through the counter logs, OSR logs and the DB dumps provided in the peak. Lets analyze this from the scratch.
Peak has been raised when a clerk attempted a Transfer Out of 4000.00 from stock unit BB to MS. Due to a system problem, the Transfer
ut doubled up, so when the Transfer In was done on counter 1 at 16:15, it was for 8000.00. The branch now has a loss of 4000.00.
I checked the counter logs and analyzed that the request ?BasketSettlementService/SettleTransferOut? with request id ?226542-2-xH-
215-22 (with JSN 3192266) has timed out. But note that in the BAL/OSR side this request was ignored by the time out monitor and
ontinued to execute and hence updated the table
Since in the counter side the request was timed out, PM has retried the same request 3 more times with request ids (226542-2-XH-0219-
+ 226542-2-XH-0215-4, 226542-2-XH-0215-5) .
ince the first request 226542-2-XxH-0215-2, updated the DB tables, the following retries should fail in the Message Journal Filter
itself. In this filter, system will check whether the JSN entry is already exists in the Journal table. Retried requests (226542-2-XH
215-4 and 226542-2-XH-0215-5) were failed with the same reason, since there already exists journal record due to 226542-2-xH-0215-:
ut, I have noticed that the retried request with id ?226542-2-xH-0215-3? was ignored by time out monitor in the BAL side and
continued to execute. But from the osr.log file and OSR message log, I couldn?t find this request was failed due to the duplicate JSN
lrecord in the journal table (which was expected and the normal behavior of OSR), didn?t happen in this case.
Ii have requested for the journal table dump, to check whether duplicate JSN entries exists in the table. But from the DB dump, I
couldn?t find any duplicates.
ltven I have requested OSR logs for to investigate further. I didn?t get any clue on the request ?226542-2-XH-0215-3?. I could only
le to see the following in the osr.log file.
2010-03-02 15:05:18,536 UTC lprpbal010:20559 [time out thread] com. fujitsu.poa.bal.osr.event.RequestEvent INFO [226542-2-xH-0215-3] —
[]- Event:com, fujitsu.poa.bal.osr.event .RequestEvent@5d0efe for BasketSett lementService/Settle?ransferOuthas timed out but is
executing. Ignoring timeout-@@-
Ihe OSR message log shows the following logs.
10-03-02 15:04:48,280 UTC 1prpbal010:20559 [httpWorkerThread-20559-9] message logger INFO [226542-2-KH-0215-3] - []- Request
Received: Uri [BasketSettlementService/SettleTransferOut]: Request (user-agent: Jakarta Commons-HttpClient /3.0.1
ontent=length: 1170
etry: true
imeout: 30000
ontent~encoding: gzip
ost: vbal001:9000
sageid: 226542-2-XH-0215-3
ecept=encoding: gzip
ontent-type: text/xml
buthorization-signature:
-a1 17 £BegQ£0/WGkTGPz 6PkwRN/xbtg7kWxh zn ZMoU£X luTYWCI pUMa 9CrxZQhLZ0CkxOsTl tN4c9ZRIUCRVTY51i peGEbyR1MU8S /42aFCeZ9CUcTPo6t+Vzmg7\ 2a
i Yoda TUSRGg/8JVWdg? £ZwgZbjuh gE} jdNz40— F/590/9
FUJ00086586
FUJ00086586
[fe TdSN? 3192206]: Request length[5750]—ee—
010-03-02 15:08:48,369 UTC Iprpbal010:20559 [settlement queue pool-13-thread-37] message logger INFO [226542-2-XH-0215~3] ~ [226542]
equest Complete: Status (Response Sent]: uri [BasketSettlementService/SettleTransferOut]: Response length{143] : Duration [240090]-@@-
It suspect. there must be something gone wrong with this request i.e. 226542-2-XH-0215-3. But unfortunately no clue on this. I am not
sure why this might have happened. Normally, since this is a retried one it should have failed at the Journal filter stage.
laut from the DB dump, in the BRDB_SU_PENDING TRANSFER DET table we had 2 records as follows.
INSERT TIMESTAMP PROD ID AMOUNT QUANTITY IS BUREAU
2-MAR-2010 15:08:22 1 4000 1 N
-MAR-2010 15:08:31 1 4000 1.N
0, we can see that the insert time stamp is different for these 2 records and hence it might have entered from 2 different requests.
ft have no doubt that one of the records was inserted by the request id ?226542-2-xH-021
jot sure how the second record was inserted. But I have doubt on the retried request 222
journal filter stage.
2, which is the original request. But I am
-KH-0215-3?, which didn?t fail at the
bo, I would request you to suggest on this, since there wasn?t any evidence which shows that 2 txn has happened and updated the
tables.
[End of Response]
esponse code to call type L as Category 40 ~~ Pending -~ Incident Under Investigation
esponse was delivered to Consumer
[Date:22-Apr-2010 15:50:33 Uscr:Tyrone Cozens
Ihe call Target Release has been moyed to Targeted At -- HNG-X 01.22.01
Joate:22-Apr-2010 15:50:42 User: Tyrone Cozens
[Start of Response]
wthorised for 01.22.01.00 as agreed in RMF
[End of Response]
lesponse code to call type L as Category 56 -- Pending -- Live Fix Authorised
esponse was delivered to Consumer
Jpate:23-Apr-2010 08:17:31 Uscr:Chaitanya Pothapragada
ction placed on Team:xCtr OSR SME, User:Martin Tonge
Date: 23-Apr-2010 0
[Start of Response]
Ht think we have done as much as we can on this one. In conclusion although we haven't been able to totally explain the behaviour, the
risk of this type Peek occuring again has been minimised in live due to a change of behaviour in the BAL with respect to
transactions. PC0194893 in CTRO25 10 HOTFIX will not allow any timed out request to commit. The timeouts are essence of this Peek and
the underlying cause.
[this may now be marked as a duplicate of PC0194893.
[End of Response]
esponse code to call type L as ©
esponse was delivered to Consumer
tegory 56 -~ Pending -- Live Fix Authorised
te:23-Apr-2010 08:32:30 User:Martin Tonge
ction has been removed from the call
Jbate:23-Apr-2010 12:03:24 User:Chaitanya Pothapragada
[Start of Response]
Huplicate of PC0194893
[End of Response]
esponse code to call type L as Category 72 -- Final -- Duplicate Call
outing to Call Logger following Final Progress update.
Response was delivered to Consumer
Defect cause updated to 40 -- General - User
lat
23-Apr-2010 15:09:05 User: Cheryl card
[Start of Response]
I am sending this call back with Response Rejected.
losing a call as "Duplicate Call' results in a black mark against me. It basically means that I should not have sent the call over
since the same problem has already been sent over in a previous call.
lrco195561 (duplicate transfer of 4000.00 cash) may have been caused by the same underlying fault as PC0194893 (banking
}econciliation), however I could not have been reasonably expected to link the 2 calls and take the decision that it was not
ecessary to send PC0195561 over for further investigation.
lease close this call with category ‘Advice After Investigation’
[End of Response]
lesponse code to call type L as Category 52 -- Pending -- Response Rejected
Ikesponse was delivered to Consumer
F/590/10
FUJ00086586
FUJ00086586
Jate?23-Ape-2010 15:41:09 Uscr:Gheryl Card
[the Call record has been transferred to the team: xCtr CSM GDC
[fhe Call record has been assigned to the Team Member: Chaitanya Pothapragada
heryl Card Confirmed that this Incident may be passed to the external company with the attached evidence.
Progress was delivered to Consumer
Date: 26-Apr-2010 09:06:26 Uscr:Steven Porter
[Start of Response]
[hus we need to ensure there is a KEL for this (either an existing one,
er a new one), then return to ssc.
[End of Response]
esponse code to call type L as Category 52 -- Pending ~~ Response Rejected
esponse was delivered to Consumer
6-Apr-2010 09:54:25 Uscr:Chaitanya Pothapragada
placed on Team:xCtr_GDC, User:Suresh Chitikela
JDate:26-Apr-2010 09:56:34 User:Chaitanya Pothapragada
[Start of Response]
ctioned to xctr_gde for visibility of the peak to GDC in future.
[End of Response]
Response code to call type L as Category 52 -- Pending -- Response Rejected
esponse was delivered to Consumer
‘$e:26-ApE-2010 10:02:38 Uscr:Chaitanya Pothapragada
[Start of Response]
PC0195561 covers the fix for this issue.
[End of Response]
esponse code to call type L as Category 95 -- Final -- Advice after Investigation
outing to Call Logger following Final Progress update.
esponse was delivered to Consumer
Jate:26-ApE-2010 10:03:57 Uscr:Chaitanya Pothapragada
ly above comment is incorrect.
c0194893 covers the fix for this issue.
[Date:26-Apr-2010 11:42:41 Uscr:Cheryl card
[the Call record has been assigned to the Team Member: Cheryl Card
progress was delivered to Consumer
j0ate:04-May-2010 13:41:57 Uscr:Cheryl card
[Start of Response]
[fhe problem was fixed in release BAL SRV_OSR ROUTING 0108 D048-p047 /
IAL_SRV_OSR 0108 D056-D055, which went onto Live on 04/04/10. I have updated KEL carde2628 with this information. Call can now be
closed.
[End of Response]
esponse code to call type L as Category 60 -- Final -- S/W Fix Released to Call Logger
Routing to Call Logger following Final Progress update.
service Response was delivered to Consumer
Date:04-May-2010 13:41:58 User:Gheryl Card
CALL PC0195561 closed: Category 60 Type L
jbate:04-May-2010 15:23:21 User:_Customer Call_
Consumer XXXXXX@TFS01 has acknowledged the call
Root Cause General - User
Logger _Customer Call_ -- EDSC
Subject Product EPOSS & DeskTop -- Counter Common (version unspecified)
Assignee _Customer Call_ -- EDSC
Last Progress 04-May-2010 15:23 -- Customer Call_
F/590/11