FUJ00095350
FUJ00095350
; Kostuch
llen Graham (BRAO1
IANGE/OU=ADMINGROUP1/CN=RECIPIENTS/CN=COCHRANEV]
Sent: Wed 4/21/2010 4:07:47 PM (UTC)
Subject: FW: HNG Red Alert fixes -- update as of Wed the 21st
All,
This is an update on the changes that have been made over the past 3 weeks, together with the top priority fix
we want to apply to the Live environment this Sunday or earlier if agreed. Please copy to others as necessary.
The following are the changes made to date (apart from the LCK fix — see second row below):
Sun 28/03/2010 Updated BAL and Oracle configs
29/03/2010 09:01:00 Alert_BRDB2.log shows LCK I LCK process was stalled, this will effectivel
process, 25353 was RAC until it's freed Oracle will need to back
temporarily stalled this into the release we are running -- reque
off Oracle confirmed this is being done
15/4 - Fix going through Integration for pac
16/4 - Take on to Vol Monday - will take a ¢
test but need rig in right state - test Tuesda
take Wednesday before overnight
20/4 - Should be approved by Vol today pre
LST tomorrow Wednesday.
Sun 04/04/2010 08:00 Numerous fixes for streams
Mon 05/04/2010 15:00 15:15 Firewall limits hit
Wed 07/04/2010 19:00 22:00 Remove from all bladeframe trunk ports the
which are not specifically required by hosts
Thu 08/04/2010 11:45 12:00 POLSAP box multicasting
Wed 14/04/2010 18:00 18:05 bdb1-4 Shared pool issue
Thu 15/04/2010 18:00 18:05 BRSS Parallelism & hash table
Fri 16/04/2010 10:00 BRSS De frag of tables
Sun 18/04/2010 08:00 babs Numerous fixes for streams
rs
Mon 19/04/2010 15:00 15:15 bdb1-4 RMAN backup failing
Tue 20/04/2010 18:00 IDGEN parallel max server & library cache
changed
Tue 20/04/2010 18:00 Add extra storage to flash area
The target date for Live for the Oracle fix for the LCK problem is 25" April. We have considered an option to
bring this LCK fix into Live on Thursday evening and believe this is possible:
FUJ00095350
FUJ00095350
1. Just before the ref data refresh at 00:00 i.e. between 22:30 and 00:00
e We made a similar change @ this time on a previous occasion with the important task being to
restart the BAL's to ensure they are all connected after the work.
e If the system is available at 22:30, the team is comfortable with the timescales for application of the
fix to the BDB and possible regression before 23:45.
2. Just after the ref data refresh at 00:30
+ This is slightly different and hasn't been done before but there should be no major differences between
this and 1) apart from slightly different TWS schedule considerations
+ If, for some unknown reason, we had to wait later than 22:30, we can easily adapt the second strategy
of running it in after 00:30 as the schedule changes are very similar
My view would be to proceed with number 1 above but if the early evening schedule runs late, we have the
fallback to number 2.
If it goes on to LST with no issues, we'd be comfortable running it in to Live during either of these periods on Thursday
night, we'll also verify the 30 minutes for the BDB fix on LST.
Following successful application of the LCK fix on Thursday evening, we’d like to agree a Friday afternoon
slot to switch Streams on to test the LCK problem in Live.
In addition to the above changes and fixes, we are also working on other options to address current
behaviours on the system, some of these fixes may come through test on time for a Sunday morning upgrade
or later next week as OCP’s.
I would like to discuss this by conf call, probably tomorrow morning,
Regards Vince