FUJ00171944
FUJ00171944
Peak Incident Management System
Call Reference PC0280437 Call Logger _Mail Manager_ -- EDSC
Release Reported In -- HNG-X Rel. Ind. Top Ref.
Call Type Operational (SSC) Priority C -- Progress restricted
Contact — EDSC Call Status Closed -- Advice after Investigation.
Target Date 28/02/2020 Effort (Man Days) 0
Summary FW: Counter and BAL clocks not synchronised, causing event date problems (PC0280046)
Collections Name User Date :
-SSCCBA Anne Best 03-Oct-2019 11:58:45
Progress Narrative
fbate:03-Oct-2019 11:40:02 User: Mail Manager_
CALL PCO280437 opened
Details entered are:~
lsummary:FW: Counter and BAL clocks not synchronised, causing event date problems (PC0280046)
call typ
Call Priority:¢
target Release: HNG-X Rel. Ind.
routed to:EDSC - _Unassigned_
jDate:03-Oct-2019 11:40:02 User: Mail Manager
IL GENERATED INCIDENT
Jbate/Time Raised: 2019-10-03 11:40:02
priority: ¢
Requested Team: EDSC
Requested Product: SSC Adhoc Request
Requested Release: HNG-X Rel. Ind.
lrmail Attachments: 4
riginator Email: "john. simpkin:
JDate:03-Oct-2019 11:40:01 User: Mail Manager_
lAddea evidence item 'OriginalEmail.eml' from Email attachment
external Progress Update Re
iginator
sent Date
subject : FW: Counter and BAL clocks not synchronised, causing event date prob.
ems (PC0280046)
lrrom: Hulme, Jon?
sent: Thursday,
to: Simpkins, John
cc: Brindley, Susan } Arain, Tariq
subject: Counter and BAL Gidcks hot synchronised, Causing event date problems (PCO2B00d6)
i John,
s discussed, I have routed PC0280046 to QFP, who can assess whether it is safe to route to GDC (copied Susan Brindley to be
laware of this).
separately to this, please can SSC investigate why the BAL clock was consistently 34s adrift from the counter clocks (seen all
kay long on 30/08/2019 for branch 102868 nodes 2 and 5, but may be a wider issue than that).
Are the clocks still adrift?
Is this an estate wide problem?
What is the counter or BAL that was out from real time?
thanks,
[Ton
Won Hulme
Domain Architect
Isusiness & Application Services
Planned leave/training:
H3th-27rd Sep
it7th Oct
23-25th Oct
Fujitsu
Jtovelace Road, Bracknell, Berkshire, RG12 8SN
FUJ00171944
FUJ00171944
jeb: NEEPT//GETPOITESW: CoNANEED! 77 ak PUERTO S
[cid: image002 .png@01CDCD78 .A2445810] <http: //www. facebook. com/fujitsuuk>
[cid: image003-png@01CDCD78 .A2445810}<https://twitter.com/#!/fujitsu_uk>
[cid: imaqe004.png@01CDCD78 .A2445810]<http: //www. youtube . com/user/FujitsuUK>
[eid: imaqe005.png@01CDcD78 .A2445810) <http: //www. linkedin. com/company/fujitsu-uk-and-ireland>
lp Please consider the environment - do you really need to print this email?
lAdded evidence item ’image001.png’ from Email attachment
\dded evidence item 'image002.png' from Email attachment
\dded evidence item 'image003.png' from Email ment
lAdded evidence item 'image004.png' from Email attachment
[Date:03-Oct-2019 11:59:07 User:Anne Best
{fhe Call record has been assigned to the Team Member: John Simpkins
bate:03-Oct-2019 15:54:33 User:RCAClient Live
PEAK { PCO280437 ] Branch ID { 102868 ] Node ID [ 2 ] SSN
lac Application /rd:true /f:text
Attempting command execution: wevtutil
Date:03-Oct-2019 15:58:30 User:RCAClient Live
PEAK [ PC0280437 ] Branch ID [ 102868 ] Node ID [ 2 ] SSN
Jsuccessfully:wevtutil ge Application /rd:true /f:text
Command execution completed
IDate:03-Oct-2019 16:06:51 User:RCAClient Live
PEAK [ PC0280437 ] Branch TD [ 102868 ] Node ID [ 5 I SSN
lac Application /rd:true /f:text
User [
Attempting command execution: wevtutil
Jdate:03-Oct-2019 16:10:43 User:RCAClient Live
PEAK [ PCO280437 ] Branch ID [ 102868 ] Node ID [ 5 ]
ssfully:wevtutil ge Application /rd:true /f:text
User [ ] Command execution completed
Date: 03-Oct-2019 16:33:59 User:RCAClient Live
PEAK { PC0280437 ] Branch ID [ 102868 ] Node ID [ 2 ] SSN
lac System /rd:true /f:text
jsint
Attempting command execution: wevtutil
Dat ¢:03-Oct-2019 16:38:51 User:RCAClient Live
PEAK [ PC0280437 ] Branch ID [ 102868 ] Node ID [ 2 ] SSN User [ ] Command execution completed
jsuccessfully:wevtutil ge System /rd:true /f:text
bate: 10-Oct-2019 16:58:55 User:RCAClient Live
PEAK [ PCO280437 ] Branch ID [ 203311 ] Node ID [ 1 ] SSN ] User [ } Attempting command execution: 1s -1 &&
late
lbate:40-Oct-2019 1
PEAK [ PCO280437 ] Branch ID [ 203311 ] Node ID [1 ]
successfully:1s -1 &&@ date
8:57 User:RCAClient Live F
[ IRRELEVANT! } User [
1 Command execution completed
jDate:10-Oct-2019 17:00:37 User:RCAClient Live = =
PEAK [{ PC0280437 ] Branch ID [ 73131 ] Node ID [ 1 ] SSN [ }IRRELEVANT!) User [ jsif-
date
<i] Attempting command execution: 1s -1 &&
Jbate:10-Oct-=2019 17:00:40 User:RCAClient Live Sk 2
PEAK { PC0280437 ] Branch ID { 73131 ] Node ID [ 1 ] SSN [ }IRRELEVANT}] User [ jsinf<J ] Command execution completed
lsuccessfully:1s -1 && date = =
jate:10-Oct-2019 17:17:42 User:RCAClient Live = = :
PEAK [ PCO280437 ] Branch ID [ 102868 ] Node ID [ 2 ] SSN [{IRRELEVANT! ] User [ jsin:
fiat
] Attempting command execution: 1s -1 &&
Jbate:10-Oct-2019 17:17:44 User:RCAClient Live
IPEAK [{ PC0280437 ] Branch ID [ 102868 ] Node ID [ 2
sfully:ls -1 && date
SSN. [TIRRELEVANT!
User [ jsinfemd ] Command execution completed
FUJ00171944
FUJ00171944
[Dateri0-Oct-2019 17:18:06 User:RCACIient Live
PEAK [ PC0280437 ] Branch ID [ 102868 ] Node ID [ 3 ] SSN [ JIRRELEVANT!] User [ ] Attempting command execution: ls -1 &&
date
Date:10-Oct-2019 17:18:08 User:RCAClient Live
PEAK [ PC0280437 ] Branch ID [ 102868 ] Node ID [ 3 ] SSN [ User [ } Command execution completed
lcuccessfully:1s -1 && date
Date:40-Oct-2019 17:18:27 User:RCAClient Live
PEAK [ PCO280437 ] Branch ID { 102868 ] Node ID [ 5 ] SSN [ {IRRELEVANT [ ] Attempting command execution: Is -1 &&
uate z
Date:10-Oct-2019 17:18:29 User:RCAClient Live S
PEAK [ PC0280437 ] Branch ID [ 102868 ] Node ID [ 5 ] SSN [ {IRRELEVANT ] User [ jsinf- ] Command execution completed
lsuccessfully:1s -1 && date
Date:40-Oct-2019 17:18:50 User:RCAClient Live E =
PEAK [ PC0280437 ] Branch ID [ 102868 ] Node ID [ 1 ] SSN [{IRRELEVANT#] User [ J Attempting command execution: 1s -1 &&
uate : Some
Jbate:10-Oct-2019 17:18:52 User:RCAClient Live =
PEAK [ PCO280437 ] Branch ID [ 102868 ] Node ID [ 1 ] SSN [}IRRELEVANT}] User [ } Command execution completed
lsuccessfully:1s -1 && date =
Date:10-Oct-2019 17:19:25 User:RCAClient Live : 7
PEAK [ PCO280437 ] Branch ID [ 102868 ] Node ID [ 4 ] SSN [} IRRELEVANT! ] Attempting command execution: 1s -1 &&
fas L i
JDate:10-Oct-2019 17:19:27 User:RCAClient Live
PEAK [ PC0280437 ] Branch ID [ 102868 ] Node ID [ 4] (AIRRELEVANT! I User [ } Command execution completed
lsuccessfully:1s -1 && date S =
lbate:10-Oct-2019 17:20:05 User:RCAClient Live
PEAK [ PCO280437 ] Branch ID { 73131 ] Node ID [ 1 ] SSN [TIRRELEVANT! User [ jsinlmud ] Attempting command execution: 1s -1 &&
date IT
Date:40-Oct-2019 17:20:07 Uscr:RCAClient Live S =
PEAK [ PC0280437 ] Branch ID [ 73131 ] Node ID [ 1 ] SSN { {IRRELEVANT} I User [ jsimj--] ] Command execution completed
lsuccessfully:1s -1 &@ date
Date:29-dan-2020 11:22:53 User:RCAClient Live =
Peak [ PCO280437 ] Branch ID [ 102868 ] Node ID [ 1 ] SSN [{IRRELEVANT] ] User [ Attempting command execution: Is -1 &&
date ‘
Date:29-dan-2020 11:22:35 User:RCAClient Live SSS
PEAK [ PC0280437 ] Branch ID [ 102868 ] Node ID [ 1 ] SSN [}IRRELEVANT} ] User [ ] Command execution completed
jsuccessfully:1s -1 && date
Oate:29-gan-2020 11:23:00 User:RCAClient Live 5
PEAK { PCO280437 ] Branch ID [ 102868 ] Node ID [ 2 ] SSN ] User [ jin} ] Attempting command execution: 1s -1 &&
uate
JDate:29-dan-2020 11:23:02 User:RCAClient Live F
PEAK [ PCO280437 ] Branch ID [ 102868 ] Node ID { 2 ] SSN [JIRRELEVANT!} User [ } Command execution completed
Isuc ully:1s -1 && date ‘
bate: 29-Jan-2020 11:23:33 User:RCAClient Live ncn
IPEAK { PCO280437 } Branch ID [ 102868 I Node ID [ 3 I SSN [j IRRELEVANT! User [ I Attempting command execution: 1s -1 &&
date =
Date:29-dan-2020 11:23:34 User:RCAClient Live = =
PEAK [ PCO280437 ] Branch ID [ 102868 ] Node ID [ 3] SSN [{IRRELEVANT! ] User [ Command execution completed
successfully:1s -1 && date
FUJ00171944
FUJ00171944
[Date:29-Jan-2020 11:23:59 Uscr:RCACLient Live
PEAK [ PCO280437 ] Branch ID { 102868 I
late
Node ID [ 4 ] SSN ] Attempting command execution: ls -1 &&
kiate
Jbate:29-Jan-2020 11:24:01 Uscr:RCAClient Live =
PEAK [ PCO280437 ] Branch ID [ 102868 ] Node ID { 4] SSN [T IRRELEVANT! User [ I Command execution completed
lsuccessfully:ls -1 && date Ss a
[Date :29-Jan-2020 11:24:20 User:RCAClient Live
IPEAK { PCO280437 ] Branch ID [ 102868 ] Node ID [ 3 ] (PIRRELEVANT! I User [ ] Attempting command execution: 1s -1 &&
date
4:21 User:RCAClient Live
PEAK [ PC0280437 ] Branch ID [ 102868 ] Node ID [ 3 ] User [ ] Command execution completed
lsuccessfully:ls =1 && date
IDate:29-Jan-2020 11:24:44 User:RCAClient Live
PEAK { PCO280437 ] Branch ID [ 102868 ] Node ID [ 5 ] User [ ] Attempting command execution: 1s -1 &&
date
bate:29-San-2020 11:24:46 User:RCAClient Live
PEAK { PCO280437 ] Branch ID [ 102868 ] Node ID [ 5 I ] User [ ] Command execution completed
lsuccessfully:1s -1 && date
IDate:29-Jan-2020 11:27:05 User:RCAClient Live r *
PEAK [ PCO280437 ] Branch ID [ 102868 ] Node ID [ 1 ] SSN [{IRRELEVANT}) User [ jsinf==3 ] Attempting command execution: ls -1 &&
jdate
Date:29-dan-2020 11:27:06 User:RCAClient Live
PEAK [ PC0280437 ] Branch ID [ 102868 ] Node ID [1 ] { ] Command execution completed
lsuccessfully:ls -1 && date
IDate:29-Jan-2020 11:29:07 User:John Simpkins
counter 1 date response is:
jed Jan 29 11:27:05 GMTST 2020
this matches well with the Peak timestamp for completion of:
bate:2020-01-29 11:27:06 User:RCAClient Live
Date:29-dan-2020 11:29:35 User:RCAClient Live =
PEAK [ PCO280437 ] Branch ID [ 102868 ] Node ID [ 2 ] SSN ] Attempting command execution: -1 66
lbate:29-dan-2020 11:29:36 Uscr:RCAClient Live
PEAK [ PCO280437 I Branch ID [ 102868 I Node
lcuccessfully:1s -1 && date
ID [
User [ join,
] Command execution completed
pate:29-dan-2020 11:30:12 User:John Simpkins
counter 2 date response is:
jed Jan 29 11:29:35 GMTST 2020
[his matches well with the Peak timestamp for completion of:
Jpate:2020-01-29 11:29:36 User:RCAClient Live
jDate:29-Jan-2020 11:30:33 User:RCAClient Live
IPEAK [ PC0280437 ] Branch 1D [ 102868 ] Node ID [
laate
3] SSN [{IRRELEVANT: } User [ jsint
J 1 Attempting command execution: 1s -1 &&
29-Jan-2020 11:30:34 User:RCAClient Live
IPEAK [ PC0280437 ] Branch ID [ 102868 ] Node ID [
jsuccessfully:ls -1 && date
3] SSN (IRRELEVANT! I User { jeint
} Command execution completed
JDate:29-Jan-2020 11:33:39 User:John Simpkins
counter 3 date response is:
led Jan 29 11:30:19 GMTST 2020
this differs with the Peak timestamp for completion of:
FUJ00171944
FUJ00171944
[ate:2020-01-25 11:30:34 User: RCACIione Live
[the RCAClient to PEAK delta is 15 seconds the normal delta appears to just be 1 second so this counter appears to have drifted by
Jabout 14 seconds.
jate:29-Jan-2020 11:33:59 User:RCAClient Live
PEAK [ PC0280437 ] Branch ID [ 102868 ] Node ID [ 4 ] SSN
date
J Attempting command execution: 1s -1 &&
oate:29-dan-2020 11:34:00 User:RCAClient Live = on
PEAK [ PC0280437 ] Branch ID [ 102868 ] Node ID [ 4] SSN [ JIRRELEVANT}] User
successfully:1s -1 &&§ date
} Command execution completed
Date:29-dan-2020 11:34:41 User:John Simpkins
counter 4 date response is:
led Jan 29 11:33:59 GMTST 2020
[this matches well with the Peak timestamp for completion of:
pate: 2020-01-29 11:34:00 User:RCACLient Live
JDate:29-Jan-2020 11:35:00 User:RCAClient Live
PEAK { PC0280437 ] Branch ID [ 102868 ] Node ID [ 5 ] SSN [ {IRRELEVANT!] User
ate =
] Attempting command execution: 1s -1 &&
jbate:29-Jan-2020 11:35:02 User:RCAClient Live
PEAK [ PCO280437 ] Branch ID [ 102868 ] Node ID [ 5 ]
jsuccessfully:1s -1 && date
N [JIRRELEVANT!] User [ jsinfuz} ] Command execution completed
bat ¢:29-Jan-2020 11:36:51 User:John Simpkins
counter 5 date response is:
jed Jan 29 11:34:47 GMTST 2020
this differs with the Peak timestamp for completion of:
pate: 2020-01-29 11:35:02 User:RCAClient Live
[fhe RCAClient to PEAK delta is 15 seconds the normal delta appears to just be 1 second so this counter also appears to have
drifted by about 14 seconds.
Joate:29-Jan-2020 11:40:21 User:John Simpkins
target Date/Time updated: new value is 28/02/2020 11:40
[Start of Response]
tt would therefore seems that nodes 3 and 5 are still adrift from the rest of the other counters in the branch by about 14
seconds.
It will try and investigate if there are any events created by these counters that will identify why they are not time synching
correctly. If I can find a key event we may then be able to look for that across the estate.
[End of Response]
Response code to call type 0 as Category 40 -- Pending -- Incident Under Investigation
jbate:29-an-2020 11:42:13 User:RCAClient Live
PEAK [ PCO280437 ] Branch ID [ 102868 ] Node ID [ 3 ] SSN [ [IRRELEVANT!)] User [ ] Attempting command execution: wevtutil
lac Application /rd:true /fitext
loate:29-gan-2020 11:44:13 User:RCAClient Live S
PEAK [ PC0280437 ] Branch ID [ 102868 ] Node ID [ 3 ] SSN [ {IRRELEVANT} ] User [ } Command execution completed
lsuccessfully:wevtutil ge Application /rd:true /f:text
lbaté:29-gan-2020 11:45:45 User:RCAClient Live
PEAK [ PC0280437 ] Branch ID { 102868 ] Node ID [ 5 ] SSN User [ ] Attempting command execution: wevtutil
lac Application /rd:true /f:text
Date:29-Jan-2020 11:49:29 User: RCAClient Live
PEAK { PCO280437 } Branch ID [ 102868 ] Node ID [ 5 ]
lsuccessfully:wevtutil ge Application /rd:true /f:text
Command execution completed
Date:29-dan-2020 11:56:48 User:RCAClient Live
PEAK [ PC0280437 ] Branch ID [ 102868 ] Node ID [ 3 ]
lac System /rd:true /f:text
] Attempting command execution: wevtutil
FUJ00171944
FUJ00171944
[bate:29-Jan-2020 12:01:44 User:RCAClient Live ;
PEAK [ 2C0280437 ] Branch ID [ 102868 ] Node ID [ 3] SSN [{
jsuccessfully:wevtutil ge System /rd:true /f:text
] Command execution completed
bate:03-Feb-2020 16:09:31 User:RCAClient Live =
PEAK { PCO280437 ] Branch ID { 178844 ] Node ID { 03 ] SSN [ User [ jsinf—-{ ] Attempting command execution: get
/cygdr ive/c/ProgramData/Fujitsu/CounterBusinessApplicat ion/1dy7PusturrréeCounter.log.2020-02-02.zip evidence/178844_01 POC _20-02~
jo2.zip
lbate:03-Feb-2020 16:09:42 User:RCAClient Live
PEAK [ PCO280437 } Branch ID [ 178844 ] Node ID [ 03 ] §
lsuccessfully:get /cygdrive/c/ProgramData/Fujitsu/CounterBusi
levidence/178844_01 POC _20-02-02.zip
N [fIRRELEVANT! I User [ jsim{=
‘Cation/log/Postort:
Command execution completed
‘ceCounter. log. 2020-02-02.2ip
joate: 23-Jun-2020 08:57:08 Uscr:John Simpkins
[Start of Response]
lAll nodes are currently in time synch, no time related events were visible for the time where the counters were out of time.
Ii believe that to go any further wee would need to talk with CC as they own the time synchronization features used on the
jcounters.
closing this call as I cannot make any further progress with the information available.
[End of Response]
esponse code to call type 0 as Category 95 -- Final -- Advice after Investigation
outing to Call Logger following Final Progress update.
lbefect cause updated to 42 -- Gen - Outside Program Control
Joate:23-gun-2020 08:57:16 User:John Simpkins
CALL PCO280437 closed: Category 95 Type O
Root Cause Gen - Outside Program Control
Logger _Mail Manager_ -- EDSC
Subject Product General/Other/Misc -- SSC Adhoc Request (version unspecified)
Assignee _Mail Manager_ -- EDSC
Last Progress 23-Jun-2020 08:57 -- John Simpkins