FUJ00095094
FUJ00095094
To: Brimicombe Johi
From: Allen Graham (BRA\
Sent: Tue 4/6/2010 12:32:54 PM (UTC)
Subject: RE: Red Alert - Independent Technical Review
I'll do the facilitating but I'll have to leave the oracle code to whoever has employed him. I'll prod though.
Graham Allen
Application Services ~ Post Office Account
FUJITSU
Lovelace Road, Bracknell, Berkshire. RG12 8SN.
Tel: +44
Mob: +
E-mail
Web
"GRO
bprrorcrunestceorn™
ss Please consider the environment - do you really need to print this email?
Fujitsu Se ered Office:
Baker Street, London, W1U 3
ices Limited, Registered in England no 96056, Reg!
This e-mail is only for the use of its intended recipient. Its contents are subject to a duty of confidence and may be privileged. Fujitsu Services does not guarantee
that this e-mail has not been intercepted and amended or that it is virus-free.
From: Brimicombe John
Sent: 06 April 2010 13:31
To: Allen Graham (BRAO1)
Subject: RE: Red Alert - Independent Technical Review
For two days .. hopefully with his knowledge and experience he can at least ask the right questions and put in place some
recommendations. Please can you make available an SST code for James and also facilitate any access he needs to resources
(including people) as necessary.
Kind Regards,
John.
John R. Brimicombe
FUJITSU
Lovelace Road, Bracknell, Berkshire, RG12 8SN, UK
Telephone: +44 (0)
Mobile: +44 (Oy
please wait as diverts to a mobile)
Web: _http://uk.fujitsu.com
Fujitsu Services Limited, Registered in England no 96056, Registered Office 22 Baker Street, London, W1U 3BW
This e-mail is only for the use of its intended recipient. Its contents are subject to a duty of confidence and may be privileged. Fujitsu does not guarantee
that this e-mail has not been intercepted and amended or that it is virus-free.
From: Allen Graham (BRAO1)
Sent: 06 April 2010 11:25
To: Brimicombe John
Subject: FW: Red Alert - Independent Technical Review
Freed him then??
Graham Allen
Application Services ~ Post Office Account
FUJITSU
Lovelace
FUJ00095094
FUJ00095094
ss Please consider the environment - do you really need to print this email?
Fuj
Services Limited, Registered in England no 96056, Registered Office: 22 Baker Street, London, W1U 3BW
This e-mail is only for the use of its intended recipient. Its contents are subject to a duty of confidence and may be privileged. Fujitsu Services does not guarantee
that this e-mail has not been intercepted and amended or that it is virus-free.
From: D'Alvarez Alan
Sent: 06 April 2010 11:09
To: Lywood Pat; Stinchcombe James
Cc: Butts Geoff; Van Achte Gaetan; Wood Mike; Bounds Gavin; Allen Graham (BRAO1); Richardson Debbie DB
Subject: Red Alert - Independent Technical Review
James,
As discussed, the account has gone on Red Alert due to service impacting issues. The initial Red Alert call had two aspects:
1. Stability of the solution
2. Service readiness
It's the stability of the technical solution that we have requested you to come in and independently review. The brief history is:
* Evening of 25" March we started high volume pilot — successfully migrating an additional 208 branches (from a target
deployment of 215 branches that night). Taking the total number of branches on HNGX to 612
* Up until 12.30pm everything was looking OK. From that point we had a catastrophic failure of the BRDB nodes causing
total loss of HNGX for over an hour, followed by a 30 minute loss of part of the estate around 3pm. Features of the loss of
service were:
© Unusually high number of time outs
© The way the timeouts/retries were ‘hitting’ the BRDB
© Oracle 3136 error message triggering the BRDB to ‘spin’ and hang
* On Monday further HNGX service disruption (not complete outage) caused by loss of single instances of the BRDBs.
© Adecision was taken to switch off of Oracle Streams at this time which co-incided with the system settling down
* On Wednesday — some additional loss of HNGX service for short periods of time also caused by loss of single instances of
the BRDBs
* Current technical progress
© Oracle has identified a known issue that results in the same symptoms we have experienced through the Oracle
3136 error message and stated this was fixed in Oracle Patch Set 3. As a consequence we have implemented this into
Live this weekend
© A programme to upgrade Redhat has been put in place which is implemented on the BRDB and a number of other
critical databases, but there are still a number to be upgraded
© Root cause of issue still uncertain (why are we getting the timeouts in the first place) — the following lines of
investigation are in hand:
= The Network Team identified packet loss and have noted that one single 1gig port is used for all four BRDB
nodes on the Bladeframe and was currently running at >600mb traffic with just 5.5% of the Branches live on
HNGX. A plan is in place to spread the load across 4 ports — one for each BRDB instance
= We had also identified a number of performance related fixes in VOL testing associated with the BRDB which
we also applied to Live over the weekend
= Some of the counter SQL queries may be inefficient - number of these have/are being rewritten
* On Thursday, Horizon PCI banking suffered severe service disruption with 80% of all transactions failing at the PEAK time
(9.30am through to 11.15am) — HNGX was not impacted
* Another service impacting issue hit Horizon PCI banking this morning
© No conclusive evidence as yet on whether these are connected — although there was ‘talk’ that the BRDB might be
using Horizon PCI addressing, but this has not been confirmed
Pat,
At Thursday's Red Alert cal it was agreed that an independent review of how we are progressing the technical issues be undertaken.
I put forward James’ name as being a suitable candidate due to his technical expertise and knowledge of the architecture (to avoid
the exercise becoming a distracting ‘learning’ exercise for someone who was totally new to the solution). James will be able to start
this afternoon to get an overview of the issues and any changes to the solution since he last worked on it. He will also be able to
assist from tomorrow afternoon and the rest of the week. Can you/Graham Allen facilitate this.
FUJ00095094
FUJ00095094
Thanks
Alan D' Alvarez
Programme Director, RMG BU
Associate P&PM Academy
FUJITSU
Mob : +44 (0) E-mai
Fujitsu Services Limited, Registered in England no. ., London W1U 3BW
This e-mail is only for the use of its intended recipient. Its contents are subject to a duty of confidence and may be privileged. Fujitsu Services does not guarantee that this e-mail
has not been intercepted and amended or that it is virus-free.