FUJ00176341
FUJ00176341
Peak Incident Management System
Call Reference PC0133634 Call Logger _Customer Call_ -- EDSC
Release _ Targeted At -- T10 Top Ref E-0603200374
Call Type Live Incidents Priority B -- Business restricted
Contact EDSC Call Status Closed -- S/W Fix Available to Call Logger
Target Date 31/03/2006 Effort (Man Days) 2.50
Summary missing data on Wigan and Bootle for Oct / Nov 05
All References Type Value
Release PinICL PCO131438
Fast Track Fix FSTK 2 0 WP23847
Work Package PWY_WP_ 23847
Powerhelp E-0603200374
Clone Call PC0134190
Progress Narrative
bate: 20-Mar-2006 10: at 23 User: Customer Call_
CALL PC0133634 open
Details entered are:-
Summary:missing data on Wigan and Bootle for Oct / Nov 05
call Type:L
call Priority:a
ffarget Release:BI_3582R
bate/Time Raise
priority: A
contact Name
ontact Phon
riginator:
riginator's reference:
Product Type: ADMIN
product Serial No:
Product Site: PATHO51
0/06/06 10129 wiasi iq dara on Wigan cad Goctla @on Ock / Noy OB on cle audie carvers
20/03/06 10:35 062523
information: security issue.
20/03/06 10:35 SYSADM
lopen OTI: Automatic Open OTT
**Updated by Jovan Vranjkovic at 20/03/06 10:35:25
20/03/06 10:35 062523
IREASSIGN: Call # E-0603200374 was Reassigned from Jovan Vranjkovic,
croup SMC7-Distrib to Group EDSC1
Date: 20-Mar-2006 10:41:59 User: Customer Call_
EMPTY 20/03/06 10:38 062523 SMC7-Distrib Information: as requested by caller -call to be assigned to SSC.
Date: 20-Mar-2006 10:44:47 Uscr:Lorraine Guiblin
Product EPOSS & DeskTop -- EPO:
jbate:20-Mar-2006 10:45:07 User:Lorraine Guiblin
\The Call record has been assigned to the Team Member: Clive Turrell
Progress was delivered to Powerhelp
[Date:20-Mar-2006 10:50:03 Uscr:Glive Turrell
Please route to Audit-Dev
Date: 20-Mar-2006 10:50:17 User:Clive Turrell
[the Call record has been transferred to the team: Audit-Dev
progress was delivered to Powerhelp
[Date:20-Max-2006 10:53:34 Usci:_Customer Call_
EMPTY 20/03/06 10:51 534768 CFM1 Information: Please Note: This call should have been logged under path
FUJ00176341
FUJ00176341
[lode PATHOSS - POA Security/Audit. SG
Jbate: 20-Mar-2006 11:32:48 Uscr:Oddette Moronfolu
evidence is attached in the form of messagestore-dat files for both servers where the missing sequence numbers were found.
butput from RQuery has also been found in the form of Excel spreadsheets.
[fhe missing message numbers seem to have occurred on or about 8am on the 14th Oct 2005, for both servers but the gaps between
these message numbers seem to vary even though they occurred at the same time.
je need an investigation into the possible cause of the gaps happening at both servers at the same time on the same day. We will
Jalso need to know if there is any other possible way of getting the missing data from other audited sources.
[Date:20-Mar-2006 11:33:52 Uscr:0ddette Moronfolu
Evidence Added - Wigan outpe
from ROuery
bate: 20-Mar-2006 11:38:52 User:Oddette Moronfolu
evidence Added - 2
Dats: 20-Mar-2006 9:29 Usor:Oddette Moronfolu
evidence Added -
JDatc:20-Mar-2006 11:40:29 Uscr:Oddette Moronfolu
evidence Added - 5a
(again RQuery out
[Date:20-Mar-2006 11:41:53 Usci:Oddette Moronfolu
[fhe Messagestores failed to be added, i will bring the disk up to the SSC, it will be quicker.
JDate:20-Mar-2006 11:46:01 Uscr:Oddette Moronfolu
rhe Call record has been transferred to the team: SSC-4th
Progress was delivered to Powerhelp
[Dat e:20-Mar-2006 11:51:17 User:Lorraine Guiblin
the Call record has been assigned to the Team Member: John Ballantyne
lPragress was delivered to Powerhelp
Joate:20-Mar-2006 14:46:12 Uscr:John Ballantyne
[fhe Call record has been transferred to the team: EDSC
Progress was delivered to Powerhelp
jDate:20-Mar-2006 14:46:33 Uscr:John Ballantyne
{the Call record has been assigned to the Team Member: John Ballantyne
Progress was delivered to Powerhelp
Date: 20-Mar-2006 1
[Start of Response]
extracted data to the Users work area on the Archive/Audit server contains the required data. Recommend that the application
lextracting the data from these files is examined.
[End of Response]
lkesponse code to call type L as Category 38 -- Pending -- Potential Problem Identified
Response was delivered to Powerhelp
Hours spent since call received: 0 hours
0:08 User:dohn Ballantyne
[Date:20-Mar-2006 14:50:26 User:John Ballantyne
[fhe Call record has been transferred to the team: Audit-Dev
the Call record has been assigned to the Team Member: Oddette Moronfolu
Progress was delivered to Powerhelp
jDate:20-Mar-2006 15:33:07 Uscr:Oddette Moronfolu
Jiohn has found that the data does in fact exist in the message store.
[Date:20-Mar-2006 17:28:33 Uscr:0ddette Moronfolu
[fhe last comment should have read:
"John has found that the data does NOT in fact exist in the message store.”
However the Me:
sage ID's missing from the message store exist in the retrieved TMS files. Then they appear to be missing once the
lgenerate messagestore process has completed.
[fhe Generate message store is created from "Agent_Load Audit _Data.exe" created by the Agents team. It takes the TMS files and
turns them into a messagestore.dat file for use by RQueryUK (an audit application).
FUJ00176341
FUJ00176341
fhe TMS file containing the message ID's has been identified at “NOL IMS Clusteraw W_5 20051013 18151400 v00l-are this has been
Inade available to the Agent team for diagnostics.
jbate:20-Mar-2006 17:31:28 Uscr:Oddette Moronfolu
the Call record has been transferred to the team: TSC-Dev
[fhe Call record has been assigned to the Team Member: Anne Mohan
Progress was delivered to Powerhelp
bate:21-Mar-2006 10:03:32 User:Anne Mohan
[the TMS file mentioned above has been found to contain a record with some non-ascii characters in it. These characters are
treated as End-Of-File by Agent Load Audit Data causing the agent to stop processing that file and move on to the next - hence
the missing records in the reconstructed message store.
[the suspect record is for Group=5868, Id=3, Num=2242256 and the hex character '1A' occurs at the end of attributes
land RawData in the record. We think these characters may have come from a bar code reader.
aR, TD, Rref
ie have reproduced the problem on our development test rig. It is not clear at the moment what the remedial action should be.
Jbate:21-Mar-2006 1
evidence Added - Re
ning non-ascii chars
JDate:21-Mar-2006 11:45:54 User:gohn Rayner
Ite investigation ie continuing to discover:
h) where the offending hex '1A' character came from
2) whether the audit utilities can be amended to handle it
lHowever it should be possible to circumvent the problem by removing the offending record, which is not part of the audit data
that is needed from the audit file : FNO1 TMS Cluster2W W_5 20051013 18151400 V001.arc.
lit should then be possible to run the audit utilities and obtain the required data.
tt does not look as if the audit file contains any other records with a hex '1A' character.
bate: 21-Mar-2006 16:15:18 User:Rex Dixon
he Call record has been assigned to the Team Member: John Rayner
Progress was delivered to Powerhelp
jbate:22-Max-2006 08:30:09 Uscr:Lionel Higman
fhe call Priority has been changed from A
fhe call Priority is now B
Date: 22-Mar-2006 08:32:05 User:Lionel Higman
priority reduced based on the following mail extractfrom John Rayner/Oddette Moronfolu:
the priority of this peak can now be lowered as agreed with Brian Pinder.
bate: 22-Mar-2006 12:34:51 User:John Rayner
ffarget Date/Time updated: new value is 31/03/2006 10:37
Development Cost updated: new cost is 2.5 (Man Days)
[Start of Response]
[fhe immediate problem has been solved by providing an copy of the audit file with the offending record removed on a CD.
it is proposed that a complete solution should be provided by producing a version of Agent Load Audit Data which reads audit
files in binary mode.
such a version of the program will see the CR + LF pairs in the audit file and will have to remove the CR.
lan experiment has been run to prove this approach and load a message containing the hex'1A' character into Riposte.
Fix IMPACT
HMPACT ON DEVELOPMENT: low
IMPACT ON TEST: low
HMPACT ON USER: low
IMPACT ON OPERATIONS: low
IkISKS (of releasing and of not releasing proposed fix):
the proposed fix is low risk.
he risk of not releasing the fix is that similar incidents will occur and require individual investigation.
[End of Response]
Response code to call type L as Category 42
tours spent since call received: 0 hours
Pending -- Product Error Diagnosed
JDatc:22-Mar-2006 1:
6:49 User:dohn Rayner
FUJ00176341
FUJ00176341
fhe Call record has been transferred to the team: RelMngmitrorum
Progress was delivered to Powerhelp
Date: 22-Mar-2006 18:04:45 User:fyrone Cozens
fhe call Target Release has been moved to Targeted At -- T10
IDate:22-Mar-2006 1
[Start of Response]
IMF authorise a fix for T10, routing to
[End of Response]
Response code to call type L, as Category 56 -- Pending -- Live Fix Authorised
Hours spent since call received: 0 hours
5:06 User: Tyrone Cozens
Date:22-Max-2006 18:05:15 User:Tyrone Cozens
lthe Call record has been transferred to the team:
Progress was delivered to Powerhelp
Jbate:22-Mar-2006 18:05:22 User:Tyrone Cozens
product, DevIntRel-Director Live Supp.Test added.
bate: 22-Mar-2006 18:17:31 User:Peter Ambrose
[fhe Call record has been assigned to the Team Member: John Rayner
progress was delivered to Powerhelp
bate: 28-Mar-2006 17:04:25 User:Oddette Moronfolu
Please note that these messages have now happened twice since this peak was logged. It seems as though the message data in the
PSTOKEN1829, a TV Licensing Client message holds incorrect/unexpected data in values:
kcur:
kr:
kRref:
[fhe TMS files containing incorrect data have been found in files dated around the middle of October 2005. We must stress that
luntil a fix is released into the live environment the delay in filtering through and editing these files may impact the ARQ
service and development time editing these files.
jDate:30-Mar-2006 15:34:04 User:dohn Rayner
Fixed in new version of Agent Load Audit Data.exe.
jDate:30-Mar-2006 15:34:14 User:John Rayner
[the Call record has been transferred to the team: TSC-Rel
lthe Call record has been assigned to the Team Member: John Rayner
Progress was delivered to Powerhelp
JDate:30-Mar-2006 15:34:52 User:John Rayner
Fixed for T10 in version 9.6.005.
JDate:30-Mar-2006 15:35:11 User:dohn Rayner
Reference Added: Work Package PWY WP 23847
jDate:30-Mar-2006 15:35:17 User:Jdohn Rayner
IfoP Reference set to: Work Package PWY WP 23847
Date:30-Mar-2006 15:35:30 User:John Rayner
the Call record has been transferred to the team: Dev-Int-Rel
progress was delivered to Powerhelp
IDate:30-Mar-2006 17:33:04 User:PIT Automated User
ference Added: Fast Track Fix FSTK 20 WP23847 (TOP Reference)
JDate:30-Mar-2006 1
[Start of Response]
I"Pasttrack fix released, now ready for test."
[End of Response]
Response code to call type L as Category 46 (Product Error Fixed)
[fhe incident has been transferred to the team: Live Supp.Test
Progress was delivered to Powerhelp
3:11 User:PIT Automated User
FUJ00176341
FUJ00176341
[Date:03-Apr-2006 16:08:13 Uscr:dohn Boston
Reference Added: FinIChL PCO1314
JDate:04-Apr-2006 13:22:53 User:Lionel Higman
call has been cloned to Call:PC0134190 by User:Lionel Higman
date: 20-Apr-2006 09:23:28 User:John Boston
[Start of Response]
P23847 successfully tested and applied to live. Routing to call logger for closure.
[End of Response]
lkesponse code to call type L as Category 60 -- Final -- S/W Fix Released to Call Logger
touting to Call Logger following Final Progress update.
Hours spent since call received: 0 hours
O-Apr-2006 09:33:11 User:Lorraine Guiblin
(the Call record has been assigned to the Team Member: Clive Turrell
Progress was delivered to Powerhelp
[Dat ¢:20-Apr-2006 10:02:06 User:Clive Turrell
lthe Call record has been transferred to the team: DataExtraction
Progress was delivered to Powerhelp
[Date:12-Jul-2006 09:41:02 User:Lionel Higman
for Reference set to: Powerhelp £-0603200374
JDate:12-gul-2006 09:41:35 User:Lionel Higman
[the Call record has been transferred to the team:
Progress was delivered to Powerhelp
jbate:12-dul-2006 09:42:02 User:Lionel Higman
[this call was raised via Powerhelp, and must be closed in the same way.
Date:42-dul-2006 09:48:58 Uscr:Jagdeep Bhambra
the Call record has been assigned to the Team Member: Clive Turrell
Progress was delivered to Powerhelp
Date:12-Jul-2006 14:30:07 User:Clive Turrell
[Start of Response]
Please route call to logger for closure - please see comments above.
{End of Response}
lkesponse code to call type L as Category 60 -- Final -~ S/W Fix Released to Call Logger
routing to Call Logger following Final Progress update.
service Response was delivered to Powerhelp
jDate:12-Jul-2006 14:30:07 User:Clive Turrell
CALL PCO133634 closed: Category 60 Type L
Date:42-Jul-2006 14:30:07 User:Clive Turrell
Hours spent since call received: 0 hours
Defect cause updated to 41 -- General - in Procedure
JDate:42-3ul-2006 14:33:47 User: Customer Call_
consumer Phelp has received the call closure
Root Cause General - in Procedure
Logger _Customer Call_ -- EDSC
Subject Product EPOSS & DeskTop -- EPOSS (version unspecified)
Assignee _Customer Call_ -- EDSC
Last Progress 12-Jul-2006 14:33 -- Customer Call_