FUJ00176333
FUJ00176333
Peak Incident Management System
Call Reference PC0162602 Call Logger _Customer Call_ -- EDSC
Release Proposed For --T86 Top Ref 398871
Call Type Live Incidents Priority C -- Non-critical
Contact EDSC Call Status Closed -- Programme approved - No fix required
Target Date 03/11/2011 Effort (Man Days) 0
Summary Gaps in Bootle cluster 3.
All References Type Value
SSCKEL KEL gbar402R
TRIOLE for Service 398871
_ SSCKEL KEL gbar402R
Progress Narrative
foate:30-dul-2008 11:50:58 User: Customer Call_
CALL Pc0162602 opened
jbetails entered are:~
summary:Gaps in Bootle cluster 3.
call Type:L
call Priority:B
lfarget Release: T82
Routed to:EDSC - _Unassigned_
Date/Time Raised: Jul 30 2008 11:26AM
Priority: B
contact Name: Nick Audley
contact Phone:!”"""GRO
wiginator: XxkXXX@rfsur
iginator's reference: 398871
Product Serial No:
RMGA NT
Product Site
JAudit Penny Thomass has reported that Bootle cluster 3 has gaps in audit data. Please investigate
incident History:
2008-07-30 11:26:09 [ Audley, Nicholas]
INIT : create a new request/incident/problem/change/issue
2008-07-30 11:28:51 [ Audley, Nicholas]
zneun_en_rm : Open Notification
2008-07-30 11:31:34 [ Audley, Nicholas]
I[R : Passing call over for investigation / comment.
2008-07-30 11:31:53 [ Audley, Nicholas]
Jzneut_en_rm : Transfer Notification
2008-07-30 11:32:22 [ Steve, Gardiner]
L0G : Audit User Penny Thomas (7302-5968) has reported that Bootle cluster 3 has gaps in audit data. Please investigate. Thank
you
oo8-07-30 11:33:46 [ Steve, Gardiner]
toc : We understand the period of time in question is the 14th June ~ 2nd July. Problems were experienced with MBOCORO3 during
the aforementioned time period. MBOCORO3 had a hardware fault and subsequently required a server rebuild.
oo08-07-30 11:37:42 [ Steve, Gardiner]
L0G : Please pass this call into PEAK FAO the EDSC in the first instance. The call should then be transferred to the Audit~Dev
team FAO Steven Meck. Thank you
2008-07-30 11:39:38 [ Steve, Gardiner]
Rt Please pass this call into PEAK. Thank you
2008-07-30 11:41:03 { Steve, Gardiner]
neut_en rm : Transfer Notification
2008-07-30 11:50:09 [ Vi
Jzneut_en_rm : Transfer Notification
o008-07-30 11:50:09 { Vincent, Niall]
I(R : Transfer ‘group! from 'SMC1' to ‘PEAK’
Jbate:30-Jul-2008 12:10:11 User:Clive Turrell
product General/Other/Misc -- Unknown General/Other/Misc (version unspecified) added.
FUJ00176333
FUJ00176333
fbate:30-dul-2008 12:
{Start of Response]
can this call please be routed to Audit-Dev team FAO Steven Meek
[End of Response]
Response code to call type L as Category 40 -- Pending -- Incident Under Investigation
Response was delivered to Consumer
Grid Useri¢live Turrell
jbate:30-Jul-2008 12:12:27 User:Clive Turrell
Ithe Call record has been transferred to the team: QFP
Progress was delivered to Provider
jbate:30-dul-2008 13:01:24 User:Nick Lawman
the Call record has been transferred to the team: Audit~Dev
[fhe Call record has been assigned to the Team Member: Steven Meek
Progress was delivered to Provider
Jbate:31-dul-2008 10:26:46 User:Steven Meek
IMBOCORO3 failed during a re-boot on the 14th June 2008. The machine was out of action until 2nd July when it was brought back
into service.
[throughout the 2nd July there appears to have been repeated problems with available disk space, presumably due to the large
number of messages being replicated from Wigan.
[the normal flow of Audit files was re-established on 5th July 2008.
initial checks suggest that messages originating at the counters for this period have been replicated from Wigan to Bootle, and
subsequently appear in the MBOCORO3 audit files, and that the gaps identified relate to messages originating at the call centres.
[this is probably due to the differences in the Correspondence servers retention periods for messages (42 days for counter
messages, 4 days for data centre messages) -
thorough check is being conducted to confirm this, and to identify all gaps. Once completed the same checks will be applied to
the corresponding Wigan files for this period to confirm that no data has been lost
[Date:14-Oct-2008 15:49:32 User:Steven Meek
[fhe call Target Release has been moved to Proposed For ~~ T86é
Jbate:15-Oct-2008 12:30:18 User:Steven Meek
ffarget Date/Time updated: new value is 28/11/2008 11:50
[Start of Response]
[the checks to identify and fill gaps is on-going.
[End of Response]
Ikesponse code to call type L as Category 40 -- Pending -- Incident Under Investigation
jbate:1-Nov-2008 13:35:31 User:Steven Meek
fhe Call record has been assigned to the Team Member: Gerald Barnes
Progress was delivered to Consumer
lbate:28-Nov-2008 11:43:25 User:Steve Evans
target. Date/Time updated: new value is 31/12/9999 00:00
[Start of Response]
ngoing - don't know quite when it will be complete.
[End of Response}
Response code to call type L as Category 40 -- Pending -- Incident Under Investigation
lbate:05-Feb-2009 1
target Date/Time updated:
{Start of Response]
It hope to complete this within a fortnight.
[End of Response]
Response code to call type L as Category 40 -- Pending -- Incident Under Investigation
2:07 User:Gerald Barnes
new value is 19/02/2009 00:00
Jbate:19-Feb-2009 1.
target Date/Time update
[Start of Response]
It have some urgent work to do concerned with making sure that all audit source is in VSS. I will address this PEAK in a week's
time.
[End of Response]
Response code to call type L as Category 40 -~ Pending -- Incident Under Investigation
6:56 User:Gerald Barnes
new value is 26/02/2009 00:00
Date: 24-Feb-2009 17:01:00 User:Gerald Barnes
Target Date/Time updated: new value is 05/03/2009 00:00
[Start of Response]
further works has cropped up and so I am putting this back another week.
FUJ00176333
FUJ00176333
Tend of Response]
Response code to call type L as Category 40 -- Pending -- Incident Under Investigation
Date: 04-Mar-2009 10:02:46 User:Gerald Barnes
ffarget Date/Time updated: new value is 31/12/9999 00:00
[Start of Response]
lat a recent meeting it was considered whether this excercise should be broadened to check the consistency of both audit servers
thoughout there life. I am setting this to no forecast whilst it is decided whether this particular problem mentioned here is to
Ibe addressed or this more broad issue.
[End of Response]
lkesponse code to call type L as Category 40 -~ Pending -- Incident Under Investigation
Date: 09-Mar-2009 1
[Start of Response]
laudit User Penny Thomas (7302-5968) has requested for this call to be updated to an ?A? priority, I understand that this issue
Inceds to be resolved prior to the migration of the Centera?s.
[End of Response]
lkesponse code to call type L as Category 40 -~ Pending -~ Incident Under Investigation
9:45 User:Stephen Gardiner
Date:09-Max-2009 1.
[Start of Response]
Judit User Penny Thomas (7302-5968) has requested for this call to be updated to an A priority, I understand this issue needs to
joe resolved prior to the migration of the Centera?s.
[End of Response]
Response code to call type L as Category 40 -- Pending -- Incident Under Investigation
0:33 User:Stephen Gardiner
lDate:09-Mar-2009 15:05:39 User:John Simpkins
he call Priority has been changed from B
fhe call Priority is now A
lDate:09-Mar-2009 16:
ffarget Date/Time update
[Start of Response]
Somehow the forecast got messed up.
[End of Response]
Response code to call type L as Category 40 -- Pending -- Incident Under Investigation
1:43 User:Steve Evans
new value is 31/12/9999 00:00
lbate:12-Mar-2009 1
ffarget Date/Time update
[Start of Response]
ur first priority here is to ensure a backup copy of all the data which is missing on one side or the other.
[this PEAk should remain as an A priority until this is done.
5:09 User:Steve Evans
new value is 30/04/2009 12:00
fter that we will need to devise a method to inject the data to the hardware where it is missing. This can be done under a lower
priority.
It am projecting a date for the first part of ths solution only, i.e. I am providing a traget date at which this PEAK may be
jreduced in priority.
[End of Response]
Response code to call type L as Category 40 -- Pending -- Incident Under Investigation
Date:12-Max-2009 15:46:39 Uscr:Gerald Barnes
(Start of Response]
lan initial set of files has been identified which needs restoring. I attach them as evidence labelled “Files Needing Restoring".
they will first be copied to a NAS.
[End of Response]
Response code to call type L as Category 40 -- Pending -- Incident Under Investigation
Date:12-Mar-2009 15:47:07 User:Gerald Barnes
evidence Added - Files Needing Restoring
JDate:18-Mar-2009 11:11:35 User:Gerald Barnes
[Start of Response]
11 the “Priority A" area yellow areas on the "Files Needing Restoring" spreadsheet have now been copied to the NAS as follows -
lewo1 TMS ClusteriW W ~ 20021213 13312800 VOOl.arc 1 OTH3069B
Jewo1_TMS_Cluster1W_W_1 20041103 08025000 _V001.arc 2_OTH3070B
IFNO1_TMS_Cluster4W_W_3 20030128 01323300 _VO01.arc 3 OTH3071B
Files have been copied from 2 days before to 3 days after each incident.
Ir am now starting to deal with the original files with queries as shown in the attached “Original Files to Restore".
So far I have started query OTH3050W to retrieve files from 12th June to 17th June 2008.
{End of Response]
Response code to call type L as Category 40 -- Pending -- Incident Under Investigation
FUJ00176333
FUJ00176333
flours Spent Since call received: 3 hours
jDate:18-Mar-2009 11:12:57 Uscr:Gerald Barne:
evidence Added ~ Original Files to
Date: 20-Mar-2009 17:33:36 User:Gerald Barnes
[Start of Response]
ith regards to the evidence labelled "Original Files to Restore".
[the query OTH3050W has now been run and I am in the process of copying all the files to the NAS.
la query OTH3053W for the period 18th June to 24th June has been run as well. I will start copying the
hen the previous copy has finished.
[End of Response]
Iresponse code to call type L as Category 40 -- Pending -- Incident Under Investigation
files from this to the NAS
Date: 30-Mar-2009 12:02:29 User:Gerald Barnes
[Start of Response]
[the files from OTH3053W have now been copied to the NAS and the query closed.
lhe query OTH3055W has been created to deal with the files from 25th June 2008 to Ist. July 2008.
{End of Response]
Response code to call type L as Category 40
Pending ~~ Incident Under Investigation
[Date:01-Apr-2009 15:40:05 User:Steve Evans
[Start of Response]
lciven that priority ‘A’ data has been successfully copied, we can now reduce trhis to a 'B'.
[End of Response]
Response code to call type L as Category 40
Pending -- Incident Under Investigation
Date: 09-Apr-2009 17:26:33 User:Gerald Barnes
[Start of Response]
YTH3O56W was created to deal with 2nd July 2008 to @ July 2008. This was run and all files copied to t
ls_ova3z05ew.
Hence now all high priority files are on the NAS.
[End of Response]
lkesponse code to call type L as Category 40 -- Pending -- Incident Under Investigation
he NAS in directory
Date: 28-Apr-2009 12:21:55 User:Gerald Barnes
target Date/Time updated: new value is 11/05/2009 12:00
[Start of Response]
fll high priority files are saved to the NAS. I have other work to do which is more pressing.
[End of Response]
Response code to call type Las Category 40 -- Pending -- Incident Under Investigation
IDate:08-May-2009 17:04:38 Uscr:Steve Evans
target Date/Time updated: new value is 31/12/9999 00:00
[Start of Response]
As Gerlad has said, all high priority files are saved to the NAS.
iven that we believe weekend D to be delayed, we are able then to work on the lesser priority data in
11 probably be resolved in June.
[End of Response]
Response code to call type L as Category 40
Pending ~- Incident Under Investigation
slower time.
This issue
bate:02-dul-2009 10:09:00 User:Chet Patel
[the call Priority has been changed from A
the call Priority is now c
Date:02-ul-2009 10:09:26 User:Chet Patel
[Start of Response]
IAS discussed, with Stephen Gilbert and Steve Evans. This is required in T86.
[End of Response]
lkesponse code to call type L as Category 40 -- Pending -- Incident Under Investigation
Response was delivered to Consumer
Date:03-dul-2009 17:32:48 User:Steve Evans
target Date/Time updated: new value is 31/12/9999 00:00
[Start of Response]
fhe willbe managed long-term.
[End of Response]
Response code to call type L as Category 40
Pending -- Incident Under Investigation
FUJ00176333
FUJ00176333
[Date:08-dul-2009 16:33:06 User:Lionel Higman
lkeference Added: HNG-X_Vet Status To be reviewed
Jbat¢:16-Jul-2009 16:26:23 User:Lionel Higman
keference Added: QFP Review IN PROGRESS
Oate:30-dul-2009 12:29:16 User:Steve Evans
fhe call Target Release has been moved to:Proposed For -- T86
[Start of Response]
le are planning to update this PEAK with a final code be the close of August
[End of Response]
Response code to call type L as Category 42
ffarget Response changed to 2009-08-28 18:00:00.0
Jbate:03-Aug-2009 14:04:35 User:Lionel Higman
Reference Added: QFP Review IN PROGRESS SheilaB
[Date:03-Aug-2009 14:04:42 Uscr:Lionel Higman
fo Reference set to: QFP Review IN PROGRESS SheilaB
[Date:03-Aug-2009 14:04:43 Uscr:Lionel Higman
Reference Deleted: QFP Review IN PROGRESS
jDate:04-Aug-2009 10:47:35 User:Lionel Higman
Reference Added: QFP Review COMPLETE
jbate:04-Aug-2009 10:47:37 User:Lionel Higman
IfOP Reference set to: HNG-X Vet Status To be reviewed
jbate:04-Aug-2009 10:47:39 User:Lionel Higman
Reference Deleted: QFP Review IN PROGRESS SheilaB
JDate:04-Aug-2009 19:45:45 Uscr:Sheila Bamber
[Start of Response]
this is a live call so decision should be via RMF _ removing HNG-x statuses
[End of Response]
Response code to call type L as Category 42 -- Pending -- Product Error Diagnosed
Response was delivered to Consumer
[Date:04-Aug-2009 19:46:00 Uscr:Sheila Bamber
TOP Reference set t
lbate:04-Aug-2009 19:46:09 User:Sheila Bamber
lkeference Deleted: HNG-X Vet Status To be reviewed
jDate:04-Aug-2009 19:46:12 Uscr:Sheila Bamber
Reference Deleted: QFP Review COMPLE’
02-Sep-2009 18:15:45 Uscr:Steve Evans
target Date/Time updated: new value is 31/12/9999 00:00
[Start of Response]
11 reassess this PEAk v soon, and give update.
[Bnd of Response]
lkesponse code to call type L as Category 42 -- Pending -- Product Error Diagnos
Response was delivered to Consumer
Jbate:07-Sep-2009 16:38:09 User:Lionel Higman
[Start of Response]
lke-organising Response Categories
[End of Response}
response code to call type L as Category 41
response was delivered to Consumer
Jbate:12-Aug-2010 12:32:52 User:Andrew Mansfield
[the data that has been backed up to the NAS (143 GB) now needs to be uploaded to the Centera from which it is missing.
lo will need to generate MD5 checksums
x these files and upload them in small batches so that it doesn't interfere with normal
FUJ00176333
FUJ00176333
fperations:
lzstimate 2 -3 days preparation time (prepare tooling for generating MD5s etc)
then 1 hour/day for 5 — 6 weeks to initiate each day's upload.
[there are no deliverables for this Peak.
Date:12-Aug-2010 16:51:06 User:Gerald Barnes
ftarget Date/Time updated: new value is 03/11/2011 00:00
{Start of Response]
the priority A tracks that have been recovered will all have expired on 3rd November 2011. When this date is reached the NAS can
be wiped of the recovered files.
[End of Response]
Response code to call type L as Category 40 -- Pending -- Incident Under Investigation
Response was delivered to Consumer
Hours spent since call received: 1 hours
Date:17-Sep-2010 10:23:24 User:John Simpkins
[Start of Response]
Requested to be closed by Saheed Salawu following review with CS operations.
[End of Response]
Ikesponse code to call type L as Category 63
service Response was delivered to Consumer
joaté:17-Sep-2010 10:23:29 User:John Simpkins
CALL PC0162602 closed: Category 63 Type L
jDate:17-Sep-2010 12:10:34 User: Customer Call_
consumer XXXXXX@TFS01 has acknowledged the call closure
41-Sep-2020 17:11:55 Uscr:Mark Wright
authorised
Root Cause General - Unknown
Logger _Customer Call_ -- EDSC
Subject Product General/Other/Misc -- Unknown General/Other/Misc (version unspecified)
Assignee _Customer Call_ -- EDSC
Last Progress 21-Sep-2020 17:11 -- Mark Wright