FUJ00173063 - Peak Incident Management System Log PC0241862 - Issue with audit purging after it was re-enabled on Wednesday

Evidence on official site

FUJ00173063
FUJ00173063

Peak Incident Management System

Call Reference PC0241862

Release Targeted At -- HNG-X 11.73.01.40
Call Type Live Incidents
Contact EDSC
Target Date 14/03/2015
Summary
All References Type
C Release PEAK
Call reference
Product Baseline
Product Baseline
TRIOLE for Service
Jira
DevIntRel-Director
Product Baseline
Collections Name
BIFApproved
Impact
ee ee

Gerald Barnes

Call Logger _Customer Call_ -- EDSC

Top Ref AUDIT EXTRACT SVR_1173_D078-D076
Priority A -- Business stopped

Call Status Closed -- S/W Fix Available to Call Logger

Effort (Man Days) 0

Issue with audit purging after it was re-enabled on Wednesday

Value

PC0242265

PC0241860
AUDIT_EXTRACT_SVR_1173_V078-V076
AUDIT EXTRACT SVR_1173_V078
A7901817

POA-1886

Live Supp.Test

AUDIT_EXTRACT SVR_1173_D078-D076
User Date

Lorraine Guiblin 23-Mar-2015 10:15:04

Date
16-Mar-2015 12:58:48

There are two tables DailyDeleteCenteralnitialTrack and DailyDeleteCenteraDupInitialTrack loaded with
tracks to delete prior to audit data being kept for an additional year. If the purger is accidentally started
from the TWS schedules they will be purged! It has been started accidentally before - that is what created
these tables in the first place. These tables must be cleared and then purging can be started again but will not
purge anything until we get to about June of this year when the additional year given to the data is
exceeded. In addition, for PARC001 only, we wish to reload the metadata pointing at the purged tracks on
the Eternus which fortunately is still there. Then PARCOO1 can be safely used for retrieving data in year 8

as well as PARC002.
Progress Narrative

pate:43-Mar-2015 12:03:38 User: Customer Call_
CALL PC0241862 opened

Details entered are:~

lsummary: 1s
call Type:L

call Priority:c

target Release:HNG-x 10.00
outed to:EDSC - Unassigned_

ue with audit purging after it was re-enabled on Wednesday

jate:13-Mar-2015 12:03:38 User: Customer Call_
INCIDENT MANAGEMENT

bate/Time Raised: Mar 13 2015 11:44AM
Priority: c
contact, Name:
contact Phone
riginator: XXXX
riginator's referen
Product Serial N
Product Site: POA

transfer Note: As per last update.

FAO audit-dev:
Received the below e-mail from Andrew G

ibson Andrew R
‘riday, March 13,
IN. POA_SMC

ston Neil J; Hemingway Andrew R
: Calls to log

2015 11:43 AM

FUJ00173063
FUJ00173063

ii, could I have 2 calls logged please

IB priority
Pass to UNIX team

2) Issue with audit purging after it was re-enabled on Wednesday

landy, we would be looking a very quick turnaround on the Ist one if possible because it is clouding our investigation into an
longoing problem with hich is hitting us for 3-4 hrs callout per night

Regards, Andrew

drew Gibson
Principal Technical Services Specialist
Hosting, Networks & Security
rujitsa
trident House,

dent History:
2015-03-13 11:44:09 [ Subramani, Santhosh Kumar]

INIT : Create a new request /incident /problem/change/issue

2015-03-13 11:45:21 [ Subramani, Santhosh Kumar]

neut_en_poa : Transfer Notification

2015-03-13 11:45:21 [ Subramani, Santhosh Kumar]

neun_en_poa : Open Notification

2015-03-13 11:47:14 [ Subramani, Santhosh Kumar]

neut_en_poa : Transfer Notification

2015-03-13 11:54:00 [ Gibson, Andrew Richard], =

oc : When audit purging was re-enabled on {IRRELEVANTI earlier this week, we saw a large number of files being unexpectedly
jpurged. This needs to be investigated by audit-dev. All further purging has been suspended

2015-03-13 11:55:20 [ Gibson, Andrew Richard]

zneut_en_poa : Transfer Notification

2015-03-13 12:03:11 [ Variyam, Parthasarathy Kumaranellur]

Jneut_en_poa : Transfer Notification

s per last update. FAO audit-dev

jbate:13-Mar-2015 12:17:33 User:Sarah English
Product HNG-x Platforms -- Audit Server (ARC) (version unspecified) added.

Jbate:43-Mar-2015 12:17:45 User:Sarah English
fhe Call record has been transferred to the team: Audit-Dev
Progress was delivered to Consumer

[Date:13-Mar-2015 13:48:34 User:Gerald Barnes
fhe Call record has been assigned to the Team Member: Gerald Barnes
Progress was delivered to Consumer

JDate:16-Mar-2015 1:
Reference Adde

741:35 User:Gerald Barnes

Date:16-Mar-2015 11:49:09 Uscr:Gerald Barnes
[Start of Response]
It have got to the bottom of this. See PPC0241860 for the details. We have two PEAKS on the same matter — PPC0241860 and this one.

in this PEAK I will hand over utility scripts to purge the DailyDeleteCenteraInitialTrack and DailyDeleteCenteraDupinitialTrack
tables and to import new tracks into the sealer database. They can then in general be tested in LST. I will use the other PEAK to
drive an MSC to call the utility scripts on PARCOO1 and (purge only) on PARCOO2

[End of Response]
lkesponse code to call type L as Category 41 -- Pending -- Product Error Diagnosed
lkesponse was delivered to Consumer

fours spent since call received: 1 hours

Date:16-Mar-2015 12:08:03 User:Andy Dunks
can this be upgraded to an “A" as there is a danger of more data being accidently purged until this is fixed

[Date:16-Mar-2015 12:21:22 User:John Simpkins
fhe call Priority has been changed from C
Ihe call Priority is now A

FUJ00173063
FUJ00173063

Date: 16-Mar-2015 12:22:18 User:Gerald Barnes
Product HNG-X Platforms -- Audit Server (ARC) (version unspecified) added.

Oate:16-Mar-2015 12:24:34 User:Gerald Barnes
fhe call Target Release has been moved to Proposed For -- HNG-X 10.63

JDate:16-Mar-2015 12:52:51 Uscr:Gerald Barnes
[the call Target Release has been moved to Proposed For -- HNG-X 11.73

Jbate:16-Mar-2015 12:58:48 User:Gerald Barnes
lA new Business Impact has been adde
[there are two tables DailyDeleteCenteraInitialfrack and DailyDeleteCenteraDupInitialfrack loaded with tracks to delete prior to
audit data being kept for an additional year. If the purger is accidentally started from the TWS schedules they will be purged!
it has been started accidentally before - that is what created these tables in the first place. These tables must be cleared and
then purging can be started again but will not purge anything until we get to about June of this year when the additional year
given to the data is exceeded. In addition, for PARCOO1 only, we wish to reload the metadata pointing at the purged tracks on the
ternus which fortunately is still there. Then PARCO01 can be safely used for retrieving data in year 8 as well as PARCO02.

JDate:16-Mar-2015 1:
[Start of Response]
DEVELOPMENT IMPACT OF FIX:

6:09 User:Gerald Barnes

SPECIFY THE HNG-X PLATFORMS IMPACTE!

[the platform is specified and it is the audit server.
[TECHNICAL SUMMARY:

[then purpose of this handover is to supply two scripts. One will purge tables DailyDeleteCenteralnitialfrack and
Dai lyDeleteCenteraDupInitialfrack from the RFISQ1 database. The other will import tracks into the SEALERSQL “Initial Track
table". These scripts can be used in general but will be called by MSC right now to get purging working safely and to recover
host metadata on PARCOO1.

LIST OF KNOWN DIMENSIONS DESIGN PARTS AFFECTED BY THE CHANG

lAUDI'T_EXTRACT_SVR
DEPENDENCIES:

[there are no dependencies. PEAK PC0241862 will be used to create an MSC to call the scripts once to recover the audit servers.
DEPLOYMENT DETAIL:

lew files that can be applied at any time.

DEV ERFORT IN MANDAYS:

2 man days

HMPACT ON USER:

the prosecution team will be able to retrieve data from year 8 on PARCOO1 again.

HMPACT ON OPERATIONS:

operations will be able to switch the purger back on again.

]HAVE RELEVANT KELS BEEN CREATED OR UPDATED?

lo KELs are needed in this case.

IMPACT ON TEST:

rest can check that one script purges tables DailyDeleteCenteralnitialTrack and DailyDeleteCenteraDupInitialTrack from RFISQL and
also that the other script imports data inot the "Initial Track table" of SEALERSQL.

IRISKS (of releasing and of not releasing proposed fix):

lat the moment purging is switched off manually from the TWS schedules. If it is accidentally switched on (and this has happenned
before) data in year 8 will be purged from the audit servers when it should not be.

tf the prosecution team forgets that I told them not to get data back in year 8 from PARCOO1 then incomplete spreadsheets may be
sent to the Post Office.

LIST OF LIKELY DELIVERABLES:

[they are all to be in D:\Archiveserver\Databases\Mssql\Utilities and are ClearPurgerTables.bat, ClearPurgerTable:
import InitialfrackData.bat and ImportInitialTrackData.sql.

[End of Response]

Response code to call type L as Category 55 -- Pending -- Live Fix Impact Supplied

lkesponse was delivered to Consumer

Hours spent since call received: 2 hours

sql,

FUJ00173063
FUJ00173063

Date:16-Mar-2015 13:17:47 User:Gerald Barnes
ction placed on Team:BIF.

bate:23-Mar-2015 10:14:58 User:Lorraine Guib:
[this will need to go as a hotfix as 11.73 has all ready been deployed, so it will need to be 11.73.01.40 and you will need to
raise an MSC for this

jDate:23-Mar-2015 10:15:07 User:Lorraine Guiblin
ction has been removed from the Gall

date: 24-Mar-2015 1
fhe call Target Release has been moved to

3:38 Usor:Lorraine Guiblin
‘argeted At -- HNG-X 11.73.01.40

loate:24-Mar-2015 17:24:01 User:Gerald Barnes
[Start of Response]

[fhe utilities have been prepared and are documented in section 5 of DEV/APP/SPG/0020.
[End of Response]

IResponse code to call type L as Category 40 -- Pending -- Incident Under Investigation
lkesponse was delivered to Consumer

Hours spent since call received: 1 hours

loate:24-Mar-2015 1
[Start of Response]

It attach a document describing the testing done with the utilities as evidence with description "fest Plan".
[End of Response]

response code to call type L as Category 40 -- Pending -- Incident Under Investigation

lkesponse was delivered to Consumer

5:03 Uscr:Gerald Barnes

Jbate:24-Mar-2015 1
evidence Added = ‘Tc

7:12 Uscr:Gerald Barnes
Plan

Date:24-Mar-2015 18:08:15 Uscr:Gerald Barnes
[Start of Response]

clearPurgerTables.bat and ImportInitialTrackData.bat can be run as any user with the right privileges - for example
\dministrator.

[they are to be run in live to cure a specific problem.

I attach notes of how LST can check them in general as evidence with description “Testing Notes for LST".
[End of Response]

lkesponse code to call type L as Category 40 -- Pending -~ Incident Under Investigation

lkesponse was delivered to Consumer

ours spent since call received: 1 hours

Date:24-Mar-2015 1)

51 User:Gerald Barnes

a:
ng Notes Fe

JDate:24-Mar-2015 1
Reference Added: Product Baseline AUDIT EXTRACT

5:02 User:Dimensions Automated User
173 _V078

Dat ¢:24-Mar-2015 19:00:01 User:Dimensions Automated User
Reference Added: Product Baseline AUDIT EXTRACT SVR_1173_VO78-VOT6

bate: 25-Mar-2015 10:40:38 User:Gerald Barnes
[Start of Response]
Fixed by baseline AUDIT

XTRACT SVR _1173_V078-V076.

[End of Response]
Response code to call type L as Category 46 -- Pending -- Product Error Fixed
Response was delivered to Consumer

Hours spent since call received: 1 hours

Date:25-Mar-2015 10:41:43 Uscr:Gerald Barnes
Defect cause updated to 21: Requirements

Date:25-Max-2015 10:41:57 User:Gerald Barnes
[the Call record has been transferred to the team: Dev-Int-Rel
Progress was delivered to Consumer

FUJ00173063
FUJ00173063

Date:25-Mar-2015 11:35:02 User:Dimensions Automated User
Reference Added: Product Baseline AUDIT_EXTRACT SVR_1173_D078-D076

Joate:25-Mar-2015 12:23:58 Uscr:Gerald Barnes
Reference Added: Jira POA-1886

jbate:25-Mar-2015 15:48:40 Uscr:Gerald Barnes
eference Adde: 0241860

Jate:25-Mar-2015 15:48:49 Uscr:Gerald Barnes
Reference Deleted: Call reference PPC0241860

User:Lerraine Guiblin

lDate:26-Mar-2015 10:06:42 Uscr:PIT Automated User
[Start of Response]
peak 0241862 handled by integration auto handler

[the following baselines attached to this peak have the targeting flags set:
lAUDIT EXTRACT SVR_1173_D078-D076 FOR (LIVE:YES TEST:YES RDT:YES) Integrator

Geoff Inglis

[these baselines have completed integration testing, moving to holding stack awaiting peak ejection.
[End of Response]

Response code to call type L as Category 47 (Fix Processed by PIT)

fhe incident has been transferred to the Team: Int-Rel

Progress was delivered to Consumer

Date: 26-Mar-2015 10:08:51 User:PIT Automated User
[Start of Response]
## AUTOMATED UPDATE - INTEGRATION PEAK BOT #f

lrix processed by integration, routing to dev-int-rel director...

IPLEASE NOTE: If this fix has failed, to send this peak back to integration it MUST have the response code Fix Failed or Response
ejected on it, otherwise the peak will bounce.

[End of Response]

lesponse code to call type L as Category 49 (Fix Available for IndependentTest)

fhe incident has been transferred to the Team: Live Supp.Test

progress was delivered to Consumer

Jbate:31-Mar-2015 14:26:59 User:John Rogers
IDPVB applied to LST rig under RN HRU11738.

the delivered scripts were tested suc
jon SharePoint.

sfully on the LS

rig and the testing is documented in document Hydra 0584 which is held

the scripts are to be run on Live under the control of MSC 04330432565.

jbate: 31-Mar-2015 14:27:14 User:dohn Rogers
{the Call record has been assigned to the Team Member: Release to Live
progress was delivered to Consumer

jDate:15-Apr-2015 0
[Start of Response]

Applied to Live

[End of Response]

Response code to call type L as Category 60 -- Final -~ S/W Fix Available to Call Logger
routing to Call Logger following Final Progress update.

Response was delivered to Consumer

8:25 Uscr:Lorraine Guiblin

joate:15-Apr-2015 08:51:00 Uscr:Hanif Adat
[Start of Response]

IPRESCAN

comments: Please see last comment , SW fix available
KEL Reference:

lAssigned User

[End of Response]
lkesponse code to call type L as Category 60 -- Final -- S/W Fix Available to Call Logger

FUJ00173063
FUJ00173063

Routing to Call Logger following Final Progress update.
Service Response was delivered to Consumer

“Apr-2015 08:51:00 User:Hanif Adat
0241862 closed: Category 60 Type L

Root Cause Requirements

Logger _Customer Call_ -- EDSC

Subject Product HNG-X Platforms -- Audit Server (ARC) (version unspecified)
Assignee _Customer Call_ -- EDSC

Last Progress 15-Apr-2015 08:51 -- Hanif Adat