FUJ00088797
FUJ00088797
o HOST BRANCH DATABASE SUPPORT GUIDE @
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
Document Title: HOST BRANCH DATABASE SUPPORT GUIDE
Document Reference:
DES/APP/SPG/0001
Document Type: SUPPORT GUIDE
Release: HNG-X Release 5
Abstract: This Support Guide details information in support and maintenance
of the Branch, the Branch Support and the Standby databases
Document Status: APPROVED
Andrew Aylward, HNG-X Host Development [20/09/2011]
Chris Walker, HNG-X Host Development [09/09/2009]
Wing Pang, HNG-X Host Development [09/09/2009]
Author & Dept: Tony Dolton, HNG-X Host Development [23/10/2009]
Rajdeep Dhaliwal, HNG-X Host Development [12/01/2010]
Gareth Seemungal, HNG-X Host Development [01/08/2011]
Pete Jobson, Technical Architecture & Consulting [22/10/2010]
Vishnu Ramachandran, HNG-X Host Development [19/05/2011]
External Distribution:
None
Approval Authorities:
Name Role Signature Date
Steve Parker ssc
Note: See Royal Mail Group Account HNG-X Reviewers/Approvers Role Matrix (PGM/DCM/ION/0001) for
guidance.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 1 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
0 Document Control
0.1 Table of Contents
0 DOCUMENT CONTROL.
0.1 Table of Contents.
0.2 Document History.
0.3 Review Details.
0.4 Associated Documents (Internal & External).
0.5 Abbreviations.
0.6 Glossary...
Changes Expected..
2 BRDB HOST PROCESSES.
2.1 Approach used for Support Gu
2.2 Table of BRDB Host Processes.
2.2.1. BRDB Environment Variables
2.3 BRDB Host Processes - Overview. 20
2.3.1 Individual Programs.. 20
2.3.2 _ Interface Feeds. 20
2.3.3 Data Aggregations. 20
2.3.4 Support Differences...
2.4 _BRDB Host Processes — Support Detail.
2.4.1 Host Interface Feeds — additional support details.
2.4.2 Agent Interfaces — additional support details.
2.5 Error Logging/Notificatior
2.5.1 Program Return Code.
2.5.2 Screen Output...
2.5.3 Operational Exception:
2.5.4 Process Control.
2.5.5 Feed Data Except
2.6 Troubleshooting.....
3 BRDB SCHEDULING.
3.1 Multi-Instance Batch Job:
3.1.1 Rerunning Failed Multi-instance Batch Jobs..
3.2. Any Active Node Batch Jobs......
3.3. Branch Database Jobs in other Schedule:
3.4. Monitoring Jobs.....
3.5 Repeating/Daemon Processe:
3.5.1. Node Failures...
3.5.2 Manually Stopping Daemon Processe:
3.5.3 Manually Starting Daemon Processes.
3.5.4 Track and Trace Feed
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 2 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.5.5 Guaranteed Reversals Feed.
3.5.6 Transaction Confirmation Feed to APO!
3.6 File Import Daemons (BRDBC038).....
3.6.1 BRDB_EXT_INTERFACE_FEEDS Table.
3.6.2 Single Node Job.
3.6.3 Post Office Esser
3.6.4 BRDB Postcode Address File Complete [BRDBC040)
3.6.5 BRDB Postcode Address File Additional [BRDBC040]
3.6.6 I BRDB Postcode Address File — End-to-End Process.
3.7 BRDB Schedules and Failove:
3.8 Schedule BRDB_PAUSE_FEE!
3.8.1 Dependencies...
3.8.2 Job BRDBX011_PAUSE_NPS_TT_COPY.
3.8.3. Job BRDBX011_PAUSE_NPS_GREV_COPY.
3.9 Schedule BRDB_STARTUP.
3.9.1 Dependencies.
3.9.2 I Job BRDBCO01..
3.10 Schedule BRDB_START_FEED:
3.10.1 Dependencies...
3.10.2 Job BRDBX011_START_NPS_TT_COPY....
3.10.3 Job BRDBX011_START_NPS_GREV_COPY.
3.11 Schedule BRDB_TT_TO_NPS3.
3.11.1. Dependencies...
3.11.2 Job BRDBX003_TT_TO_NPS_1
3.12 Schedule BRDB_GREV_NPS3.
3.12.1 Dependencies...
3.12.2 Job BRDBX003_GREV_TO_NPS_1
3.13 Schedule BRDB_PAUSE_FEED1..
3.13.1. Dependencies...
3.13.2 Job BRDBX011_PAUSE_NPS_TT_COP’
3.13.3 Job BRDBX011_PAUSE_NPS_GREV_COPY.
3.14 Schedule BRDB_COMPLETE.
3.14.1. Dependencies...
3.14.2 Job CREATE_BRDB_COMPLETE_FLAG.
3.15 Schedule BRDB_SOD.
3.15.1 Dependencies...
3.15.2 Job DELETE_BRDB_COMPLETE_FLAG.
3.15.3 Job DELETE_BRDB_COMPLETE_FLAG.
3.16 Schedule BRDB_START_FEED1
3.16.1 Dependencies... 50
3.16.2 Job BRDBX011_START_NPS_TT_COPY. 50
3.16.3 Job BRDBX011_START_NPS_GREV_COPY. 51
3.17 Schedule BRDB_START_LFS. 51
3.17.1 Dependencies... 51
3.17.2 Job BRDBX011_START_LFS_PCOL_COP
3.17.3 Job BRDBX011_START_LFS_PDEL_COPY.
3.18 Schedule BRDB_START_APOP....
3.18.1 Dependencies... wee
3.18.2 Job BRDBX011_START_APOP_TC_COPY.
3.19...
3.20 Schedule BRDB_TT_TO_NPS1.
3.20.1 Dependencies.
3.20.2 Job BRDBX003_TT_TO_!
3.21 Schedule BRDB_GREV_NPS1
3.21.1 Dependencies...
3.21.2 Job BRDBX003_GREV_TO_NPS_1...4_ NOPAGE
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN Ref: DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 3 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.22 Schedule BRDB_PCL_TO_LFS..
3.22.1 Dependencies...
3.22.2 Job BRDBX003_PCOL_TO_LFS_1
3.23 Schedule BRDB_PDL_TO_LFS..
3.23.1 Dependencies.
3.23.2 Job BRDBX003_I _TO_l
3.24 Schedule BRDB_TC_TO_APOP.
3.24.1 Dependencies...
3.24.2 Job BRDBX003_TC_TO_APOP.
3.25.
3.26 chedule bs
3.26.1 Dependencies.
3.26.2 Job COMPLETE.
3.27 Schedule BRDB_REF_DATA_SLA.
3.27.1 Dependencies... .
3.27.2 Job BRDBX032_BRDB_REF_DATA_SLA. 54
3.28 Schedule BRDB_ONCH_AGG.
3.28.1 Dependencies..............
3.28.2 Job BRDBX007_ONCH. _AGG_ 1.
3.28.3 Job BRDBC008_CHECK_ONCH_AGG.
3.29 Schedule BRDB_CSH_TO_LFS.
3.29.1 Dependencies...
3.29.2 Job BRDBX003_CASH_TO_LFS_1...4.
3.29.3 Job BRDBC008_CHECK_CASH_TO_LFS.
3.30 Schedule BRDB_FROM_EMDB..
3.30.1 Dependencies.......
3.30.2 Job BRDBX003_I BRDATA FROM_EMDB.
3.31 Schedule BRDB_CLR_BRANCH...
3.31.1. Dependencies...
3.31.2 Job BRDBX037_CLEAR_I BRDATA.
3.32 Schedule BRDB_PAUSE_LFS.
3.32.1 Dependencies...
3.32.2 Job BRDBX011_PAUSE_LFS_PCOL_COP
3.32.3 Job BRDBX011_PAUSE_LFS_PDEL_COPY.
3.33 Schedule BRDB_PAUSE_APOP.
3.33.1 Dependencies...
3.33.2 Job BRDBX011_PAUSE_APOP_TC_COPY.
3.34 Schedule BRDB_EPOS_TO_TP:
3.34.1 Dependencies...
3.34.2 Job BRDBX003_EPOSS_TO_TPS_.
3.34.3 Job BRDBC008_CHECK_EPOSS_TO_TPS
3.35 Schedule BRDB_APS_TO_TPS.
3.35.1 Dependencies.
3.35.2 Job BRDBX00:
3.35.3 Job BRDBCO008 _( CHECK ‘APS. TO! “TP!
3.36 Schedule BRDB_NWB_TO_TPS.
3.36.1 Dependencies...
3.36.2 Job BRDBX003_NWB_TO_TPS_1
3.36.3 Job BRDBC008_CHECK_NWB_TO_TPS.
3.37 Schedule BRDB_DCS_TO_TPS.
3.37.1 Dependencies...
3.37.2 Job BRDBX003_DCS_TO_TPS_1...4.
3.37.3 Job BRDBC008_CHECK_DCS. TO, _TI
3.38 Schedule BRDB_BDC_TO_TPS.
3.38.1 Dependencies...
3.38.2 Job BRDBX003_BUREAU_TO_TPS ~
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 4 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.38.3 Job BRDBC008_CHECK_BUREAU_TO_TPS.
3.39 Schedule BRDB_EVT_TO_TPS..
3.39.1 Dependencies...
3.39.2 Job BRDBX003_I EVENTS - TO_TPS_’
3.39.3. Job BRDBC008_CHECK_EVENTS_TO_TPS.
3.40 Schedule BRDB_COFS_TO_TPS..
3.40.1 Dependencies...
3.40.2 Job BRDBX003_COFF_SUMM_TO_TPS_1
3.40.3 Job BRDBC008_CHECK_COFF_SUMM_TO_TPS.
3.41 Schedule BRDB_TA_FROM_TPS.
3.41.1. Dependencies...
3.41.2 Job BRDBX003_TA_FROM_TPS
3.42 Schedule BRDB_TC_FROM_TPS..
3.42.1 Dependencies...
3.42.2 Job BRDBX003_TC_I FROM. TPS
3.43 Schedule BRDB_TPS_COMPL.
3.43.1. Dependencies...
3.43.2 Job COMPLETE.
3.44 Schedule BRDB_TPS_TOTALG....
3.44.1 Dependencies...
3.44.2 Job BRDBX007_TPS_TXN. I TOTALS 4
3.44.3. Job BRDBC008_CHECK_TPS_TXN_TOTALS.
3.45 Schedule BRDB_TOTL_TO_TP: 66
3.45.1 Dependencies. 66
3.45.2 Job BRDBX00: 66
3.45.3 Job BRDBCO008 _( CHECK _’ TXN. I TOTALS - To TPS.
3.46 Schedule BRDB_APS_TOTALS.
3.46.1 Dependencies...
3.46.2 Job BRDBX007_APS_TXN_TOTALS_1...4..
3.46.3 Job BRDBC008_CHECK_APS_TXN_TOTAL:
3.47 Schedule BRDB_TOTL_TO_APS.
3.47.1. Dependencies...
3.47.2 Job BRDBX003_TXN_TOTALS_TO_APS_1...4.
3.47.3 Job BRDBC008_CHECK_TXN_TOTALS_TO_APS.
3.48 Schedule BRDB_TXNS_TO_APS.
3.48.1 Dependencies...
3.48.2 Job BRDBX003_TXNS_TO_APS_1...4.
3.48.3. Job BRDBC008_CHECK_TXNS_TO_APS.
3.49 Schedule BRDB_APS_COMPL.
3.49.1 Dependencies.
3.49.2 Job COMPLETE.
3.50 Schedule BRDB_NWB_TO_DRS.
3.50.1 Dependencies.
3.50.2 Job BRDBX003_NWB_TO_DRS_-
3.50.3 Job BRDBC008_CHECK_NWB_TO_DRS
3.51 Schedule BRDB_DCS_TO_DRS.
3.51.1. Dependencies...
3.51.2 Job BRDBX003_I DCs, TO! ‘DRS 1.
3.51.3 Job BRDBC008_CHECK_DCS_TO_DRS.
3.52 Schedule BRDB_DRS_COMPI
3.52.1 Dependencies.
3.52.2 Job COMPLETE.
3.53 Schedule BRDB_)
3.53.1 Dependencies.
3.53.2 Job COMPLETE.
3.54 Schedule BRDB_FEED_ERRORS.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 5 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.54.1 Dependencies...
3.54.2 Job BRDBX007_RAISE_FEED_DATA_EXCEPTIONS.
3.55 Schedule BRDB_NCU_TXN_AGG
3.55.1 Dependencies...
3.55.2 Job BRDBX007_NON_‘ CUMU_- “TXN I_TOTALS_1
3.55.3 Job BRDBC008_CHECK_NON_CUMU_TXN_AG
3.56 Schedule BRDB_CU_TXN_AGG.
3.56.1 Dependencies...
3.56.2 Job BRDBX007_CUMU_TXN_AGGR_1...4.
3.56.3 Job BRDBC008_CHECK_CUMU_TXN_AGGR.
3.57 Schedule BRDB_BBNI_MAINT.
3.57.1 Dependencies...
3.57.2 Job BRDBX031_JSN_USN_SSN.
3.58 Schedule BRDB_SUMMARY_DTE.
3.58.1 Dependencies...
3.58.2 Job BRDBX011_SET_I DAILY. “SUMMARY _I DATI
3.59 Schedule BRDB_GEN_REP.
3.59.1 Dependencies..............
3.59.2 Job GENERIC. CREATE _I "REPORT _VIEWS..
3.59.3 Job GENERIC_CREATE_RECON_REPORTS
3.60 Schedule BRDB_TO_DWH.
3.60.1 Dependencies...
3.60.2 Job BRDBX020_BRDB_XFER_TO_DWH.
3.61 Schedule BRDB_AGG_COMPL..
3.61.1 Dependencies.
3.61.2 Job COMPLETE.
3.62 Schedule BRDB_FROM_RDDS.
3.62.1 Dependencies...
3.62.2 Job BRDBX003_REFDATA_FROM_RDDS
3.63 Schedule BRDB_FROM_TPS.
3.63.1 Dependencies...
3.63.2 Job BRDBX003_REFDATA_FROM_TPS.
3.64 Schedule BRDB_AUD_FEED.
3.64.1 Dependencies...
3.64.2 Job BRDBC002_AUDIT_:
3.64.3 Job BRDBC008_CHECK_AUDIT_FEED.
3.64.4 Job BRDBC033_AUDIT.....
3.65 Schedule BRDB_ORA_STATS.
3.65.1 Dependencies...
3.65.2 Job BRDBX005_SCHEMA.
3.66 Schedule BRDB_ADMIN.
3.66.1 Dependencies.
3.66.2 Job BRDBC004..
3.66.3 Job BRDBX006.
3.66.4 Job BRDB_HKP_ORAFILES:
3.66.5 Job BRDB_HKP_ORAFILES:
3.67 Schedule BRDB_PAUSE_FEED;
3.67.1 Dependencies...
3.67.2 Job BRDBX011_PAUSE_NPS_TT_COP
3.67.3 Job BRDBX011_PAUSE_NPS_GREV_COPY.
3.68 Schedule BRDB_EOD.
3.68.1 Dependencies.
3.68.2 Job BRDBCOO9..
3.69 Schedule BRDB_START_FEED:
3.69.1 Dependencies...
3.69.2 Job BRDBX011_START_NPS_TT_COPY
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 6 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.69.3 Job BRDBX011_START_NPS_GREV_COPY.
3.70 Schedule BRDB_TT_TO_NPS2.
3.70.1 Dependencies...
3.70.2 Job BRDBX003_TT_TO_NPS_:
3.71 Schedule BRDB_GREV_NPS2.
3.71.1. Dependencies...
3.71.2 Job BRDBX003_GREV_TO_NPS__
3.72 Schedule BRDB_START_BKP.
3.72.1 Dependencies.
3.72.2 Job COMPLETE.
3.73 Schedule BRDB_I
3.73.1 Dependencies...
3.73.2 Job BRDB_LVLO_BACKUP.
3.74 Schedule BRDB_BACKUP_’
3.74.1. Dependencies... we
3.74.2 Job BRDB_LVL1_BACKU:
3.75 Schedule BRDB_BKP_COMPI
3.75.1 Dependencies...........cccceccecee
3.75.2 Job CREATE_BRDB_COMPLETE_FLAG.
3.76 Schedule BRDB_MONITOR.
3.76.1 Dependencies...
3.76.2 Job BRDB_MON_STARTUP.
3.76.3 Job BRDB_MON_PAUSE_FEED
3.76.4 Job BRDB_MON_AUD_FEE!
3.76.5 Job BRDB_MON_EOD...
3.77 Schedule BRDB_POE_LOAD.
3.77.1 Job BRDBC038_POE_FROM_POLSAP.
3.78 Schedule BRDB_PAFCD_LOAD.......
3.78.1 Job BRDBC038_PAF_FROM_CD.
3.79 Schedule BRDB_PAFADD_LOAD.
3.79.1 Job BRDBC038_PAF_ADD_LOAD..
4 BACKUP AND RECOVERY.
4.1 BRDB & BRSS Backups.
4.44 Backup Duration...
4.2 Restoring files with RMAI
4.3 Failure and Recovery...
4.3.1 Escalation and Notification.
4.3.2 Media Failure and Recovery.....
4.3.3. Instance/Node Failure and Recover
5 GENERAL AND TROUBLESHOOTING NOTE:
5.1 Database...
5.1.1 Oracle Database Listeners.
5.1.2 I General Recommendations.
5.1.3 Password Management
5.2. Backups...
5.2.1 Database Backups.
5.2.2 Disk Backups....
5.3 Partition Managemen’
5.3.1 Introduction.
5.3.2 I Assumption:
5.3.3 Overview..
5.3.4 Troubleshooting
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 7 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
5.4 Standby Database.
5.4.1 Introduction.
5.4.2 Assumptions.
5.4.3. Troubleshooting
5.5 Oracle Streams.
5.5.1 Introduction.
5.5.2 Assumption:
5.5.3 Overview.
5.5.4 Troubleshooting
5.6 SCC Transaction Correction Tools.
5.6.1. BRDBX015 — Transaction Correction Too!
5.6.2 BRDB Clear Stock Unit Lock (clear_su_lock.sh).
5.6.3 BRDB Clear Rollover Lock (clear_ro_lock.sh)..
5.6.4 BRDB Update Outstanding Recovery Transaction Tool (upd_rvy_txn.sh
5.7 BRDBC004 Archival/Purge Logi
5.8 BRDB Software Updates/Installatio:
5.9 Querying/Updating BRDB/BRSS during the online da’
6 APPENDIX A - STANDBY DATABASE.
6.1 Oracle Data Guard Broker (DGMGRL) Failover
6.2 SQL*Plus Failover...
6.3 Standby Database Re-instantiation..
6.3.1 Tripwire Configuration...
6.4 Opening Standby Database “READ ONLY”.
6.5 Standby Cluster - Software Installation...
7 APPENDIX B — BRANCH SUPPORT.
7.1. Cleanup and Re-instantiation of Oracle Stream:
7.4.1 Assumptions.. ceseeeeeeeeeee
7.1.2 Cleanup and Re-instantiation Procedure.
7.2. Managing Streams Lag.....
7.21 Context and Assumption:
7.2.2 Lag Evaluation and Escalation.
7.2.3 Lag Assessment and Action Procedur
7.2.4 Post Lag Action Procedure...
7.3 Streams DML Behaviour on OPS$BRDB Tables.
7.4 Data Aggregations....
7.5 I Table of BRSS Host Processes.
7.6 BRSS Scheduling.
7.6.1 Schedule BRSS_TRACE_STOP
7.6.2 Schedule BRSS_SOD.
7.6.3 Schedule BRSS_CLR_BRANCH..
7.6.4 Schedule BRSS_TRACE_STRT1
7.6.5 Schedule BRSS_JRNL_TRACE1.
7.6.6 Schedule BRSS_DXC.
7.6.7 Schedule BRSS_GEN_RE!
7.6.8 Schedule BRSS_ORA_STATS.
7.6.9 Schedule BRSS_ADMIN
7.6.10 Schedule BRSS_START_E .
7.6.11 Schedule BRSS_BACKUP_0O.
7.6.12 Schedule BRSS_BACKUP_1
7.6.13 Schedule BRSS_STARTU
7.6.14 Schedule BRSS_COMPLETE..
7.6.15 Schedule BRSS_MONITOR..
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 8 of 183
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE ®
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN &
CONFIDENCE)
8 APPENDIX C - TRANSACTION CORRECTION TEMPLATES.
8.1 Templates. 183
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 9 of 183
Fe)
FUJITSU
HOST BRANCH DATABASE SUPPORT GUIDE
FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
FUJ00088797
FUJ00088797
0.2 Document History
Associated Change
Version No. Date Summary of Changes and Reason for Issue SPEARS
4 22" June 2009 Initial Version N/A
02 18" September 2009 _I First major update to all sections NA
03 23" October 2009 Updated with schedule details and other information. NIA
Updated with general review comments and additions to
a
04 29 October 2008 Streams and Standby procedures. NA
05 29" October 2009 Updated with Streams related information. NIA
14 5" November 2009 I Added new Hydra functionality cP404
12 12" January 2010 Added Transaction Acknowledgement copy cP 4914S
‘n ‘Added stock unit unlock, update outstanding recovery txn I PC0191404,
13 18° January 2010 and branch rollover unlock functionality. PC0191168, PC0189018
14 17" February Added process BRDBX035 PC0194351
15 47" March 2010 Couple of corrections plus adding bookmarks for schedule I 1,
document hyperlinks
16 47” May 2010 Couple of corrections plus adding bookmarks for schedule I 11,
document hyperlinks.
17 28" June 2010 Added BRSS schedule, TT/GREV changes 0200577, PC0200019
18 9" July 2010 ‘Added manual start/stop feed commands NIA
Corrections due to review process (comments from SSC,
19 20" October 2010 ISD), section added for service outages, changes to I PCO203999
recovery, changes to BRDB schedules (remove HYDRA)
Added AEI Near-Real Time Interface.
New Sections — cP4ot
2.3.2.2, 2.4.2 through to 2.4.2.4
1.10 27" October 2010 Updated Sections —
2.2,2.3.2,23.4,25.3
+
Updated Transaction Correction templates (all templates in I PC0195962
Section 7 - Appendix C)
Changes due to ISD review
1 17% December 2010 NIA
Changed BRDBX005 details to match new implementation
2.00 3rd February 2011 Document status set to ‘APPROVED' NIA
Release 4 branch closure process BRDBXO037.sh, new
associated schedule + description
24 1otn February 2011 I gpg -> BRDB description update CPS5B5, CP510
TPoS - new table added
Pc0208496
Release 4 changes to BRDB purge process [BRDBC004]
Release 4 Capacity Management Reporting solution in
23 19" May 2014 BRSS (new modules) cP639
Release 5 BRDB Transaction Confirmation feed to APOP.
(new Host interface feed) cpez9
24 26" May 2011 Release 5 Post Office Essentials cpse2
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN _ Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED PageNo: 10 of 183
Fe)
FUJITSU
HOST BRANCH DATABASE SUPPORT GUIDE
FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
FUJ00088797
FUJ00088797
25 ‘August 2014 Post Office Address File Processing and other I CP633
9 amendments including Approver/Reviewer matrix updates.
3.0 21* September 2011 I Document status set to ‘APPROVED’ NA
0.3 Review Details
Review Comments by
17" August 2011
Review Comments to
Mandatory Review
Role
Andrew Aylward; RMGADocumentManagement
Name
ssc
‘Steve Parker*
Solution Design / Host Branch Database
Andy Beardmore
Solution Design / Host Batch Systems
Pete Jobson
Core Division
Optional Review
Role
Gibson Andrew”
Name
Head of Service Management
Tony Atkinson
Problem & Incident Lead SDM
Saheed Salawu
Service Introduction Manager
Adam Bowe
Service Manager - Retail and RMGA
Pete Thompson (as interim)
HNG-X Host Development
Steve Goddard
HNG-X Host Development
Wing Pang
HNG-X Host Development
Vishnuvardhan Ramachandran
HNG-X Host Development
Gareth Seemungal
Core Division — NI Oracle Support
Wayne Calvert
Core Division — NI Oracle Support
Paul Simpson
Core Division — NI Unix Support
Paul Stewart”
Business Continuity Adam Parker
Issued for Information — Please restrict this
distribution list to a minimum
Position/Role Name
(*) = Reviewers that returned comments
© Copyright Fujitsu Ltd 2011
UNCONTROLLED IF PRINTED
FUJITSU RESTRICTED (COMMERCIALIN Ref.
CONFIDENCE) Version:
Date:
Page No:
DES/APP/SPG/0001
3.0
21-Sep-11
11 of 183
Fe)
FUJITSU
HOST BRANCH DATABASE SUPPORT GUIDE
FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
FUJ00088797
FUJ00088797
@
0.4 Associated Documents (Internal & External)
Reference Version Date Title Sot
PGM/DCM/TEM/000 Fujitsu Post Office A t HNG-X D. q
4 Aor ujitsu Post Office Accoun' -X Document
2.0 16-Apr-07 Template - PORTRAIT Dimensions
(DO NOT REMOVE)
DES/APP/HLD/0020 Branch Database High Level Design Dimensions
DES/APP/LLD/0152 Branch Database Low Level Design Dimensions
DES/APP/HLD/0021 Branch Database Scheduling High Level Design Dimensions
DES/APP/HLD/0023 Branch Support Database High Level Design Dimensions
DES/APP/LLD/0151 Branch Support Database Low Level Design Dimensions
DES/APP/HLD/0025 Branch Support Database Scheduling High Level I Dimensions
Schema Definition for the Branch Database,
DEV/APP/LLD/0199 Standyby Branch Database and Branch Support I Dimensions
‘System
DEV/APP/LLD/0011 HOST BRANCH DATABASE SUPPORT GUIDE Dimensions
DEV/APP/LLD/0802 Host BRDB Near-Real Time Service Interface — Dimensions
Low Level Design
DES/APP/HLD/0732 NRT Interface Agent High Level Design Dimensions
DES/APP/DPR/0671 AE! Near-Real Time Design Proposal Dimensions
DEV/APP/LLD/1230 BRDBIBRSS Branch Closure and Archive Dimensions
rocess
DEV/APP/SPG/0025, LFS Support Guide Dimensions
DEV/APP/LLD/0050 BRDB Host System Interfaces Low Level Design Dimensions
DEV/APP/LLD/1394 BRSS Host: Data Aggregation and De-I pimensions
normalisation Low Level Design
Unless a specific version is referred to above, reference should be made to the current approved
versions of the documents.
0.5 Abbreviations
Abbreviation Definition
ACE Cisco Application Control Engine
ASM Automatic Storage Management
BAL Branch Access Layer
BDB Acronym for Branch Database
BDS Acronym for Branch Standby Database
BRDB Branch Database Oracle SID
BRS Acronym for Branch Support Database
CRS Oracle Cluster Ready Services
FAN Oracle Fast Application Notification
© Copyright Fujitsu Ltd 2011
UNCONTROLLED IF PRINTED
FUJITSU RESTRICTED (COMMERCIAL IN Ref:
CONFIDENCE) Version:
Date:
Page No:
DES/APP/SPG/0001
3.0
21-Sep-11
12 of 183
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE @
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
GREV Guaranteed Reversals
HLD High Level Design
ITM IBM Tivoli Manager
JSN Journal Sequence Number
LCR Logical change record (generated by the Streams capture process)
LPAN Logical Processing Area Network
LFS Logistics Feeder Service
NPS Network Persistant Store
OCFS Oracle Cluster File System
OcR Oracle Cluster Registry
PAN Processing Area Network
RFS Oracle Remote File Server (a process)
RHEL Red Hat Enterprise Linux
RMAN Oracle Recovery Manager
SAN Storage Area Network
SCN Oracle System Change Number
SHLD Schedule High Level Design
sal Structured Query Language
SSN Session Sequence Number
T Track & Trace
USN User Sequence Number (in the context of the counter user)
NRT Near-Real Time
AEI Application & Enrolment Identity
APOP Automated Payment Out Pay
0.6 Glossary
Term Definitior
BladeFrame A BladeFrame is a chassis which contains processing blades (pBlade) and control
blades, as well as integrated interconnect and power connections. The BladeFrame
is connected to networks and storage with fully redundant cables.
Branch Access Layer The middle-tier that carries out the data storage, retrieval and transfer on behalf of
the Counter.
Cluster A cluster is a group of loosely coupled computers that work together closely so that
in many respects they can be viewed as though they are a single computer.
Clusters are usually deployed to improve performance and/or availability over that
provided by a single computer,
Database A collection of records stored in a systematic way. The software used to manage
and query records is known as the Database Management System. This document
uses the term ‘Database’ to cover both meanings.
Host System The collection of host systems including TPS, APS, DRS, LFS, NPS, RDDS and
RDMC
Hydra Phase covering the dual-running of Horizon and HNG-X
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN Ref. DES/APP/SPG/0001
CONFIDENCE) Version 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 13 of 183
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE @
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
Instance A database instance — this is composed of memory structures and the Oracle
background processes that run on a server.
Node Any device connected to a network such as a server. In the document, the term
‘Node’ includes the Oracle Instance.
pBlade A processing blade which contains processors and memory, but not network or disk
devices.
pServer A logical representation of a pBlade.
Real Application Clusters I An Oracle Real Application Cluster is a group of loosely coupled computers that
work together closely so that in many respects they can be viewed as though they
are a single computer. Clusters are usually deployed to improve performance
and/or availability over that provided by a single computer.
0.7 Changes Expected
Changes
Changes from time-to-time in subsequent versions of the all HLDs and LLDs may require changes to this
document.
0.8 Accuracy
Fujitsu endeavours to ensure that the information contained in this document is correct but, whilst every effort is
made to ensure the accuracy of such information, it accepts no liability for any loss (however caused) sustained as
a result of any error or omission in the same.
0.9 Copyright
© Copyright Fujitsu Limited 2011. All rights reserved. No part of this document may be reproduced, stored or
transmitted in any form without the prior written permission of Fujitsu
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 14 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
1 Introduction
1.1 Document Overview
This Support Guide details information in support of the Branch Database solution by documenting the
operational processes that run for the application and in support of the infrastructure surrounding the
application. Procedures for supporting and troubleshooting the Branch Database solution are also
included.
The Branch Database has been designed to be able to fail over to a standby server in the event of a
disaster but requires operator intervention because of the inherent complexity of the solution. Relevant
procedures are provided for this purpose.
The Branch Support Database has been designed as a data store for support personnel. Keeping this
database in step with BRDB is very important, the BRDB HLD indicates that the Branch Support
Database should not lag BRDB by more than 15 minutes.
The BRDB schedule must run once for each and every calendar day. BRDB keeps a track of the current
working day, in order to guarantee that data is correctly stored, processed and replicated.
Text which is highlighted if yellow like'this indicates important information that should be noted.
1.2 Scope
This document is to serve as guide in support of the Branch and the Branch Support Databases. It is not
a build manual, nor does it explain all the inner workings of Oracle or the operating system. Guidance
for important tasks and troubleshooting scenarios are also included.
It is also to be noted that much of the detailed information for the support guide has already been
documented in the associated specifications and designs. The main sources for this information are the
BRDB High Level Design [DES/APP/HLD/0020], the BRSS High Level Design [DES/APP/HLD/0023] and
the BRDB Low Level Design [DES/APP/LLD/0151].
1.3 Assumptions
This Support Guide assumes the Branch Database has been successfully built and is in operation.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 15 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
Fe)
FUJITSU
2 BRDB Host Processes
2.1 Approach used for Support Guide
Much of the relevant information for this section of the support guide has already been documented in
the associated specifications and designs. The main source of information is:
The Branch Database High Level Design (DES/APP/HLD/0020)
The relevant information in this reference is already presented under repeating headings for the
processes (i.e. the same headings for each process in turn), making it ideally suited for use as a support
reference. This section of the document mainly serves to identify the relevant information, and indicate
where it can be found. Pertinent information that is not covered by the existing documents has been
added as appropriate.
The relevant process section of the Branch Database High Level Design is Section 7.2 - Host Processes.
For further information on the Host Processes and their integration in the overnight schedule, see
Section 0 - BRDB Scheduling.
2.2 Table of BRDB Host Processes
The following table lists the current Branch Database Host processes, a brief description of each and the
names of the executables used to run them. The process name corresponds to the name that is
registered in table BRDB_PROCESSES and, where applicable, the name that is used to control
processing via table BRDB_PROCESS_CONTROL.
No. Executable BRDB Pro Name ription
BRDBCO0O01 BRDBCO001 Start of Day
BRDBC002 BRDBC002 Message Journal Auditing
BRDBX003.sh
BRDB_APS_TXN_FROM_TPS
BRDB APS transactions from TPS feed
BRDBX003.sh
BRDB_APS_TXN_TO_APS
BRDB APS transactions to APS feed
BRDB_APS_TXN_TO_TPS
BRDB APS transactions to TPS feed
BRDBX003.sh
BRDB_BDC_TXN_FROM_TPS
BRDB BDC transactions from TPS feed
BRDBX003.sh
BRDB_BDC_TXN_TO_TPS
BRDB BDC transactions to TPS feed
BRDBX003.sh
BRDB_CASH_TO_LFS
BRDB Cash Declarations to LFS feed
1
2
3
4
5 I BRDBX003.sh
6
7
8
9
BRDBX003.sh
BRDB_CNTR_REF_FROM_RDDS
BRDB_ Counter
RDDS feed
Reference Data from
10 BRDBX003.sh
BRDB_CUTOFF_SUMM_TO_TPS
BRDB Cut Off Summaries to TPS feed
1 BRDBX003.sh
BRDB_DCS_TXN_FROM_TPS
BRDB DCS transactions from TPS feed
12 BRDBX003.sh
BRDB_DCS_TXN_TO_DRS
BRDB DCS transactions to DRS feed
13 BRDBX003.sh
BRDB_DCS_TXN_TO_TPS
BRDB DCS transactions to TPS feed
14 BRDBX003.sh
BRDB_EMDB_INTERFACE
BRDB Estate Management Interface feed
15 BRDBX003.sh
BRDB_EPOSS_EVNT_TO_TPS
BRDB EPOSS events to TPS feed
16 BRDBX003.sh
BRDB_EPOSS_TXN_FROM_TPS
BRDB EPOSS transactions from TPS feed
17 I BRDBX003.sh
BRDB_EPOSS_TXN_TO_TPS
BRDB EPOSS transactions to TPS feed
© Copyright Fujitsu Ltd 2011
UNCONTROLLED IF PRINTED
FUJITSU RESTRICTED (COMMERCIAL IN Ref:
CONFIDENCE)
DES/APP/SPG/0001
Version: 3.0
Date: 21-Sep-11
Page No: 16 of 183
FU
Re)
ITSU
HOST BRANCH DATABASE SUPPORT GUIDE
FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
FUJ00088797
FUJ00088797
18 BRDBX003.sh BRDB_HOST_REF_FROM_RDDS. BROS Host Reference Data from RDDS
jeer
19 I BRDBX003.sh BRDB_INDAY_XML_FROM_TPS Redundant since R2 decommissioning
BRDB In-Day Migration Blob from TPS
20 I BRDBX003.sh BRDB_MEMOS_FROM_RDDS BRDB Desktop Memos from RDDS feed
21 BRDBX003.sh BRDB_NWB_TXN_FROM_TPS. BRDB NWB transactions from TPS feed
22 I BRDBX003.sh BRDB_NWB_TXN_TO_DRS BRDB NWB transactions to DRS feed
23 I BRDBX003.sh BRDB_NWB_TXN_TO_TPS. BRDB NWB transactions to TPS feed
24 I BRDBX003.sh BRDB_PCOL_TO_LFS BRDB Pouch Collections to LFS feed
25 I BRDBX003.sh BRDB_PDEL_TO_LFS BRDB Pouch Deliveries to LFS feed
26 I BRDBX003.sh BRDB_PLO_FROM_LFS pepe Planned Order details from LFS
jeer
27 I BRDBX003.sh BRDB_RDC_FROM_LFS BRDB_ Replenishment Delivery details
from LFS feed
28 BRDBX003.sh BRDB_RECON_XML_FROM_TPS. BRDB Reconciliation Blob from TPS feed
29 BRDBX003.sh BRDB_REF_COPY_FROM_TPS BRDB Outlets/Transaction Modes from
TPS feed
30 I BRDBX003.sh BRDB_REV_TXN_TO_NPS BRDB Reversal Records to NPS feed
31 BRDBX003.sh BRDB_TT_TXN_TO_NPS PROB Track and Trace Records to NPS
feed
32 I BRDBX003.sh BRDB_TXN_CORR_FROM_TPS ir Transaction Corrections from TPS
jeer
33 I BRDBX003.sh BRDB_TXN_TOT_TO_APS BRDB Transaction Totals to APS feed
34 I BRDBX003.sh BRDB_TXN_TOT_TO_TPS BRDB Transaction Totals to TPS feed
35 I BRDBX003.sh BRDB_TXN_CONF_TO_APOP. BROS Transaction Confirmation to APOP:
jeer
36 BRDBCO004 BRDBCO04 Audit, Archive, Purge
37 I BRDBX005.sh BRDBX005.sh Gather Optimiser Statistics
38 I BRDBX006.sh BRDBX006 File Housekeeping
39 I BRDBX007.sh BRDB_APS_TXN_TOTALS Data aggregation to calculate APS
transaction totals
40 I BRDBX007.sh BRDB_CUMU_TXN_AGGR Data aggregation for daily cumulative
summary
41 BRDBX007.sh BRDB_NON_CUMU_TXN_AGGR Data aggregation for daily summary
42 BRDBX007.sh BRDB_TPS_TXN_TOTALS Data aggregation to calculate outlet
transaction totals
43 BRDBX007.sh OVERNIGHT_CASH_ON_HAND Data aggregation to calculate ONCH
figures.
44 BRDBX007.sh RAISE_FEED_DATA_EXCEPTIONS Inserts into operational exceptions if Feed
data exceptions
45 I BRDBCO08 BRDBCO008 Check Job Completion
46 I BRDBCOO9 BRDBCO09 End Of Day
47 BRDBX011.sh BRDBX011 Updates system parameters
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref: DES/APP/SPG/0001
CONFIDENCE) Version 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED PageNo: 17 of 183
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE @
FUIITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
48 I BRDBX015.sh None Transaction correction tool
49 I BRDBX020.sh* None Redundant since R2 decommissioning
File transfer for BRDB Branch Migration
Status data feed
50 I BRDBX021.sh None Pause or restart Oracle Streams
propagation
51 I BRDBX030.sh BRDBX030_INDAY Redundant since R2 decommissioning
Hydra XML processing (INDAY)
52 I BRDBX030.sh BRDBX030_RECON_CATCHUP Redundant since R2 decommissioning
BRDBX030_RECON_NORMAL Hydra XML processing (RECON)
53 I BRDBX031.sh* BRDBX031 Reset JSN, USN and SSN
54 I BRDBX032.sh* BRDB_REF_DATA_SLAS. Reference Data SLAs
55 I BRDBCO33* BRDBC033 Transaction Correction Journal Auditing
56 I BRDBX033.sh BRDBX033_PREP_RECON_CATCHUP I Redundant since R2 decommissioning
BRDBX033_PREP_RECON_NORMAL I Hydra XML processing (RECON)
57 I BRDBX034.sh BRDBX034 Redundant since R2 decommissioning
Hydra - Maintain filter table of branches
due to migrate and undergo ‘normal
processing in BRDBX030/BRDBX033.
58 I BRDBX035.sh BRDBX035 Redundant since R2 decommissioning
Hydra - Extracts checking version of the
Branch Trading Statement report for
migrating branches.
59 I GREPX001.sh* GREPX001 Create generic views for reporting
60 I GREPX002.sh* GREPX002 Create generic reports
61 BRDBX003.sh BRDB_TXN_ACK_FROM_TPS. BRDB Transaction Acknowledgement from
TPS feed
62 I PKG_BRDB_NRT_ I BRDB_NRT_TXN_TO_AGENT BRDB Near-Real Time Service Interface to
TXN_TO_AGENT* Agents
63 BRDBX036.sh BRDBX036 Athene - performance/graphing tool
64 I BRDBX037.sh BRDBX037 BRDB Branch Closure Process
BRDB_CLR_BRANCH_DATA
Table 1: Branch Processes
Note
At the time of writing, the processes/executables marked with an asterisk (*) in the table above have not
yet been added to the High Level Design document, and therefore do not have the support information
available for reference. They have been included here for completeness and early notification (rather
than waiting until the details have been added to the design document).
Unlike other Host processes, PKG_BRDB_NRT_TXN_TO_AGENT does not get executed by any script
in the Batch Database schedule. Instead, NRT Agents directly access the package as detailed in
subsequent sections. Branch Database HLD is yet to be updated with AEI NRT related changes but the
low level design document [DEV/APP/LLD/0802] provides detailed information on the various
procedures that constitute package PKG_BRDB_NRT_TXN_TO_AGENT and how NRT Agents connect
to the Branch Database to process AEI NRT messages.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 18 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
Fe)
FUJITSU
2.2.1 BRDB Environment Variables
The following set of environment variables are relevant for the BRDB batch users which are used by
TWS when calling batch jobs. The table below is a representation of brdbblv1. and includes only BRDB
application related variables.
Environment Variable
BRDB_EXCP_USER
BRDB_TCT_FILE_TEMP
BRDB_AUDIT_FILE_TEMP
NCHOME
NLS_DATE_FORMAT
EXPORT_DIR
BRDB_TCT_AUDIT_OUTPUT
BRDB_MSU_OUTPUT
BRDB_ARCHIVE_OUTPUT
BRDB_HOST_AUDIT_OUTPUT
BRDB_COUNTER_AUDIT_OUTPUT
ORACLE_HOME
OMNIHOME
INPUTRC
G_BROKEN_FILENAMES.
ORACLE_SID
LANG
NETCOOL_LICENSE_FILE
BRDB_CONNECT_STR
LOGNAME
BRDB_SH
HISTSIZE
REPOSITORY
LESSOPEN
BRDB_MSU_WORKING
FAN_EVENT_LOG_DIR
Variable Value
ORAEXCPLV/EXCPLV123
/app/brdb/trans/support/working
/app/brdb/trans/support/working
Jopt/netcool
DD-MON-YYYY
/var/tmp
/app/brdb/trans/audit/tctaudit
/app/brdb/trans/support/reportoutput
Japp/brdb/trans/support/archive
/app/brdb/trans/audit/hostaudit
/app/brdb/trans/audit/counteraudit
/u01/app/oracle/product/10.2.0/db_1
Jopt/netcool/omnibus
/etc/inputre
1
BRDB1
Cc
27000@lltpbdb001
BRDB
brdbblv1
Japp_sw/brdb/sh
1000
/pw/stagonl/repository
Jusr/bin/lesspipe.sh %s
/app/brdb/trans/support/working
Japp_swibrdb/log
BRDB_PROC Japp_swibrdb/c
SSH_ASKPASS Jusr/libexec/openssh/gnome-ssh-askpass
BRDB_SQL /app_sw/brdb/sqI
EXCP_USER ORAEXCPLV/EXCP123
Table 2: Branch Environment Variables
2.3 BRDB Host Processes - Overview
The BRDB Host processes and how they are implemented fall into 3 main categories:
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref: DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED PageNo: 19 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
2.3.1 Individual Programs
These are individual shell scripts or Pro*C programs that perform a specified task. Typically, they have
been migrated (with minimal change) from existing Horizon processes. e.g. “Start of Day” (BRDBC001),
“Audit, Archive Purge” (BRDBC004) and “File Housekeeping” (BRDBX006). They are invoked by a
direct call (from a Linux shell) to an executable.
2.3.2 Interface Feeds
2.3.2.1 Host Interface Feeds
These are new for the Branch Database, and load data between the BRDB and the legacy Host systems
(in both directions). There are currently over 30 different Feeds, with each being performed by a
separate, specific database package. All of the Feeds have a common interface/parameter list and are
invoked via a single shell script (BRDBX003.sh). The first parameter passed to this script controls which
Feed process (packaged procedure) is executed.
For example, line 17 of the Table of BRDB Host Processes shows that the Feed of EPOSS transactions
from BRDB to TPS, is performed by a call to BRDBX003.sh with a first parameter of
“BRDB_EPOSS_TXN_TO_TPS’.
The corresponding database packages are named according to the following convention:
PKG_<Feed name> e.g. PKG_BRDB_EPOSS_TXN_TO_TPS
See 2.4.1 for feed information and troubleshooting guides.
2.3.2.2 Agent Interfaces
These interfaces are new for the Branch Database introduced at HNG-X Release 3 to cater for various
Near-Real Time (NRT) Service messages. In Release 3, Application and Enrolment Identity (AEI) NRT
Service has been implemented as a NRT interface within the Branch Database. Unlike other Host
Interface feeds these Interfaces do not get invoked from BRDB batch schedule via shell script
BRDBX003.sh; instead they get invoked directly by NRT Agents connecting to the Branch Database.
Wherever applicable these interfaces re-use feed procedures and exception handling mechanisms that
are common to Host Interface feeds.
2.3.3. Data Aggregations
These are also new for the Branch Database, and are similar to the Interface Feeds in that different
processes (currently numbering six) are invoked via a single shell script (BRDBX007.sh) with a
controlling first parameter. However, they differ from the Feeds in that the program code is stored in the
database as raw SQL or PL/SQL, with no corresponding database packages.
2.3.4 Support Differences
The differences between the categories outlined above will translate into variations from a support
perspective. For example, issues with database links, synonyms, grants etc. may manifest as package
compilation errors for the Feeds, but run-time errors for the Aggregations.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 20 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
An invalid Feed package can be re-compiled for verification (before running) after certain problems have
been resolved (e.g. when a missing database link has been restored). A recompilation can be performed
using the “ALTER PACKAGE” command from SQL*Plus:
e.g. ALTER PACKAGE PKG_BRDB_EPOSS_TXN_TO_TPS COMPILE;
In contrast, an Aggregation or Pro*C executable cannot be re-validated against the database in advance,
it can only be re-run.
Another difference between the categories outlined above concerns the amount of information that is
output when the processes are run. The Interface Feeds and main executables (see sections 2.3.1 and
2.3.2 above) provide the option to specify a debug level in order control the amount of output from within
each process/Feed. Typically, the default debug settings provide milestone information only. However,
should the need arise, for example whilst investigating a possible problem, the amount of output can be
easily increased via meta-data (i.e. without changing the program concerned) - the debug levels are held
as numeric system parameters with a higher number (e.g. 1) producing more detailed output than a
lower number (e.g. 0) - see HLD for further details.
From Support perspective, Agent Interfaces vary from Host Interface Feeds. The extent of 3” line
support required is limited within the Branch Database as operational control lies with the NRT Agents.
Within the Branch Database, support will be confined to any exceptions encountered and archiving of
processed messages.
The Aggregations are more limited in this respect. The mechanism that calls each Aggregation issues
output and has the debug capability, but the Aggregations themselves do not.
Differences relating to the support of the program return codes when a Node/Instance failure is
encountered are detailed in section 2.5.1 Program Return Code.
2.4 BRDB Host Processes — Support Details
Much of the detailed information required for support purposes is contained in the following sections of
the BRDB High Level Design:
HLD: Section 7.2 Host Processes
This section of the HLD contains details of each of the Host Processes, and has been written with
support requirements in mind. The information is presented under the following headings for each
process:
e Application Type — indicates the programming language in which the module has been
developed e.g. PL/SQL packages, Pro*C etc
e Inputs — lists the input parameters and whether they are mandatory or optional.
e Outputs — indicates the program return codes.
e Location - states the Linux directory in which the executable code resides.
e Scheduling — gives an overview of the scheduling
e Processing details — gives high level details of the processing performed, along with information
on the more important and specific functionality.
e Handling Failures and Rerun ability - gives information on the likely failure conditions, plus
instructions on how to proceed.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 21 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
A significant part of the BRDB daily processing concerns the loading of data between the Branch
database and numerous Host applications (in both directions) by the Host Interface Feeds. Because of
the variety of processing involved, further details are contained in a separate section of the HLD:
HLD: Section 5.3.4 Host Interfaces
This section of the HLD contains detailed information on the data and requirements for BRDB Host
Interfaces. It includes details of the data being processed, the Host applications, and how the data is
selected for processing.
Although much of this information will be too detailed for initial support purposes, it is referenced here in
case more detailed analysis and understanding of a process(es) is required.
2.4.1 Host Interface Feeds — additional support details
This section gives further details and support information on the Interface Feeds:
The Host Interface Feeds have been designed and written to be robust and should therefore require very
little support. For example, all of the Feed processes can simply be re-run (when the underlying problem
has been resolved) if they fail to complete successfully. They all write the details of any ‘show-stopper’
errors to the standard output, as well as logging the necessary information to the operational exceptions
table (BRDB_OPERATIONAL_EXCEPTIONS). Output is also generated under normal circumstances,
providing useful information on the actions performed, time taken etc.
In addition, certain foreseeable issues/events such as a Node or database instance failure have been
catered for within the logic of the Feed programs and the daily schedule.
2.4.1.1 Process Control
Where relevant, the Feeds utilise the existing ‘process control’ functionality — to store information on
when the processes were run and whether they completed successfully etc. Table
BRDB_PROCESS_CONTROL can be queried for this information. This table is also used to enforce
requirements such as ensuring that certain processes can only be run once for a given trading date.
2.4.1.2 FAD Hashes
As part of the high level design, the processing of the largest volumes of data has been sub-divided -
into FAD hashes (currently numbering 128). Under normal circumstances, the processing of the FAD
Hashes is evenly distributed across the Nodes (currently numbering 4) within the Real Application
Cluster (RAC).
2.4.1.3 Node/Instance Failure
If one of the Nodes or database instances goes down, the loss is automatically detected and flagged
using Oracle’s Fast Application Notification (FAN). FAN then allows the processing that would have
normally taken place on the failed Node to be automatically re-allocated across the remaining Nodes
(when the processes are re-run — see below).
Further details of the FAN event processing are contained in the HLD.
Details of how the failed Node should (when fixed) should be reintroduced to the Cluster (i.e. made
available to the Host processes) are contained within the database support section of this document.
2.4.1.4 Scheduled Re-Run of Multi-Node Feeds
The daily BRDB schedule does not automatically re-run multi-node Feed processes in the event of a
single or multi-node failure. If these processes/jobs were in the state of executing when a node failure is
experienced they will still appear to be executing until such time as the TWS agent re-establishes
communications. Operational support will be notified in the event of such a failure.
Therefore, in order to process any FAD Hashes that have been re-allocated from a failed Node,
Operational support will need to be involved in any steps of intervention.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN _ Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 22 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
2.4.1.5 Data Exceptions
One of the high level design assumptions was that because the Feeds load data between internal
systems (to/from the Branch Access Layer and to/from the Host applications) the data being processed
should be error-free. To this end, the Feeds have (where possible) been designed to perform optimally
when this is the case. However, because the unexpected can (and does) happen, many of the Feeds
(where appropriate) incorporate a mechanism to handle any data errors. This means loading the valid
records, whilst writing any exception records to a separate exceptions table for investigation.
In order to prevent such BRDB data errors from going un-noticed, there is a job
(RAISE_FEED_DATA_EXCEPTIONS) within the normal, daily schedule that highlights any such
exceptions by inserting a summary record into the operational exceptions table. This record provides an
alert to the SMC, and includes the following information:
e Number of interface Feeds that encountered a data exception(s)
« Total number of data exceptions
e Processing date on which the exceptions were encountered
e The names of the affected Feeds and how many exceptions each one encountered
e The name of the database table where the exceptions have been stored
e A statement/instruction to indicate that investigation is required.
It should be noted that such exceptions are DATA errors - caused by issues with the data or underlying
specification of the data format - and NOT Feed errors. The presence of a data error(s) will not cause the
Feed process to fail unless the quantity of such errors is significant — the allowable limit is configurable
for each Feed, and is currently set to 1000.
The nature of this type of exception means that they are unexpected, and therefore cannot be easily
fixed by a support procedure etc. The correct action from a support perspective is to notify the
development team of the situation - so that they can investigate the actual data and data specifications
etc. in order to identify where the problem/discrepancy lies. They will also need to determine whether to
re-process the data that could not be loaded, and if so, how it will be done.
2.4.1.6 Data Exception Thresholds
Every feed has a data exception (numeric) threshold stored in BRDB_SYSTEM_PARAMETERS
identified by a parameter name of the form '<FEED NAME>_MAX_DATA_ERRORS'.
BRDBX011.sh can be used to change a threshold value e.g. the following changes the exception
threshold value for the Track and Trace feed to 10,000:
$BRDB_SH/BRDBX011.sh -n "BRDB_TT_TXN_TO NPS_MAX DATA ERRORS" -t "N" -v 10000
2.4.2 Agent Interfaces — additional support details
This section gives further details and support information on Agent Interfaces:
The Agent Interfaces have been designed and written to be robust and should therefore require very little
support. For example, if there are NRT Agent connection failures or node instance failures then NRT
Agents will have to call the initialise method and continue to process NRT service messages. All
procedures within the NRT Interaface return the details of any ‘show-stopper’ errors to the calling NRT
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 23 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
Agent, as well as logging the necessary information to the operational exceptions table
(BRDB_OPERATIONAL_EXCEPTIONS). Since Agent Interfaces are not batch jobs execution output
(stdlist) is not applicable.
On Windows platforms Agent events are written to the Windows Application Event Log whilst on Linux
systems Agent events are written to syslog (See DES/APP/SPG/0002 section 3.1).
2.4.2.1 Process Control
As all the procedures implemented within the package PKG_BRDB_NRT_TXN_TO_AGENT are
independent, atomic and directly accessible by the NRT Agents there is no need for process control
within the Branch Database for Agent Interfaces.
2.4.2.2 FAD Hashes
Similar to Host Interfaces, the processing of the largest volumes of data has been sub-divided - into FAD
hashes (currently numbering 128). Under normal circumstances, the processing of the FAD Hashes is
evenly distributed across the Nodes (currently numbering 4) within the Real Application Cluster (RAC).
2.4.2.3. Node/Instance Failure
If one of the Nodes or database instances goes down, the loss is automatically detected and flagged
using Oracle’s Fast Application Notification (FAN). The mechanism then allows the processing that
would have normally taken place on the failed Node to be automatically re-allocated across the
remaining Nodes.
Further details of the FAN event processing are contained in the Branch Database HLD.
Details of how the failed Node (when fixed) should be reintroduced to the Cluster (i.e. made available to
the Host processes) are contained within the database support section of this document.
Details of how the NRT Agents will recover and re-connect to the Branch Database in the event of Node
/ Database Instance failure are contained in NRT Interface Agent High Level Design
[DES/APP/HLD/0732].
2.4.2.4 AEI NRT Interface
HNG-X Counters will write AEl NRT Service messages (triggered by new AP-ADC data type
AssociateNRT) to a table called BRDB_RX_NRT_TRANSACTIONS in OPS$BRDB schema of the
Branch Database. These NRT messages will be set initially with a processed_yn value of ‘N’. All such
unprocessed messages will be picked up and processed, one by one, by NRT Agents.
NRT Agents — There will be four NRT Agents connecting to the Branch Database through Nodes 1I2I3/4
respectively. A NRT Agent connecting through a specific BRDB Node will connect to the Branch
Database and access the AEI NRT Interface package using respective database user
LVAGENTUSER{1I2I3/4}. Similarly, while processing NRT messages a NRT Agent will only process
those messages allocated through FAD hash load-balancing for a particular node — this includes Node /
Database Instance failure scenario also.
Processed NRT messages will be set with processed_yn to ‘Y’ and an appropriate processed_timestamp
in BRDB_RX_NRT_TRANSACTIONS table. Such processed messages will be archived based on meta-
data defined in BRDB_ARCHIVED_TABLES.
For an end-to-end overview of the AEI NRT solution in HNG-X refer to AE] Near-Real Time Design
Proposal document [DES/APP/DPR/0671].
2.5 Error Logging/Notification
When an error is detected within one of the BRDB Host processes it is highlighted and logged using the
following standard procedures:
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 24 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
2.5.1 Program Return Code
Processes that fail return a non-zero number to the calling environment. Typically, 0 represents
successful completion, 1 represents a failure and 99 indicates that a Node or Instance failure has been
encountered.
Note
Within the Host processes, two different mechanisms have been used to identify whether an error code
encountered within a program corresponds to a Node/Instance failure:
e Dynamic - The Interface Feeds use a dynamic, meta-data driven mechanism, using
BRDB_ORACLE_ERROR_CODES as a look-up table.
e ‘Hard-coded’ - The 2 other categories of Host process (Individual Programs and Data
Aggregations) have fixed (‘hard-coded’) error codes within the programs.
Therefore, if another, ‘new’ Oracle error code is found to correspond to a Node/Instance failure (and
therefore the Host processes need to return a code of 99), the support activity required will differ
accordingly:
For the Interface Feeds, a new record for the error code will need to be added to the look-up table, with
column INSTANCE_CONN_ERROR_YN set to ‘Y’. None of the Feed programs will need to be changed.
For the other processes, the hard-coded list in each affected shell script/Pro*C program will need to be
updated, and each program re-released.
2.5.2 Screen Output
Most of the BRDB Host processes will output the details of an error (what the problem is, where it was
encountered etc.) to the standard output.
2.5.3 Operational Exceptions
When an error is encountered, the details are logged in table BRDB_OPERATIONAL_EXCEPTIONS,
including what the error is and where and when it was encountered. Agent Interfaces also pass the
exception message and Oracle database error code, if applicable, back to the calling NRT Agent.
In addition, table BRDB_HYDRA_EXCEPTIONS was used to log errors associated with Hydra
processing (prior to release 2 decommissioning).
2.5.4 Process Control
As with many existing Host applications, most of the BRDB processes use _ table
BRDB_PROCESS_CONTROL to manage re-starting, and to control whether an invoked process should
be allowed to run. This table can be queried (using SQL*Plus or TOAD) to determine when a process
started and iffwhen it completed successfully etc. The column
OPS$BRDB.BRDB_PROCESS_CONTROL.PROCESS_NAME will map to those processes listed in 2.2.
This is not applicable for Agent Interfaces.
2.5.5 Feed Data Exceptions
See section 2.4.1.5 (Data Exceptions) for details.
2.6 Troubleshooting
With error logging and notification being detailed in the sections above, the other useful bit of
information is that of troubleshooting failures when the reason for their failure is unclear.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN __ Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 25 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
In most cases the logging information displayed in stdout and the exception information available in
BRDB_OPERATIONAL_EXCEPTIONS will suffice in determining the cause of a particular feed (or other
scheduled job for that matter). A very useful way of determining a higher level of detail in the logging
information (and possibly the exception information — however an exception is not likely to change from
the original when executed a second time) is by increasing the DEBUG level of the job/feed in question.
The table BRDB_LSYSTEM_PARAMETERS holds a parameter for each of these which will generally be
the naming convention, according to the type of job as follows: -
Feeds: <Feed_Name>_DEBUG_LEVEL e.g. BRDB_PDEL_TO_LFS_DEBUG_LEVEL
Jobs: DEBUG_LEVEL_FOR_<Job_Name> e.g. DEBUG_LEVEL_FOR_BRDBCO001
The valid values of the debug level are from 0 to 3, 0 being default logging through to 3 for verbose.
An update to the debug level of a job or feed can be performed as follows: -
U
ATE brdb system parameters
parameter number = '<
HERE parameter_name
w
Be sure to set the parameter back to the default once the more verbose option is no longer required.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 26 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3 BRDB Scheduling
The Branch Database schedule is run each day, and controls how and when most of the processes are
executed. Sections 3.1 to 0.1 describe features of the schedule as a whole, and sections 3.8 onwards
describe the individual schedules that it is composed of.
3.1 Multi-Instance Batch Jobs
Scheduling HLD: Section 5.2 Common Approach for multi-instance batch jobs
The main BRDB processes are scheduled across the nodes of the Real Application Cluster (RAC). Some
of these processes are simply restarted when a failure occurs, but, most are implemented with built-in
delays and reruns in the case of an initial failure. This approach means that a support call is only raised
when a failure condition persists i.e. after an automatic retry has been attempted.
Please note: Currently, all scheduled processes/jobs will raise an alert upon failure. Therefore in all
cases Operational support will be aware of each failure and respond accordingly.
In the schedule listings from sections 3.8 onwards, only the main jobs which perform the relevant task
are listed. However, they are implemented using a common schedule template consisting of the main
job running on each of the four nodes, and additional jobs to perform the waiting, checking and
rerunning, as per the following table.
Job Name Job Dependency Rerun Action
15_min_wait
Job-Instance-1 On failure continue
Job-Instance-2 On failure continue
Job-Instance-3 On failure continue
Job-Instance-4 On failure continue
Check-Job-Instance-1 Follows 15_min_wait
Check-Job-Instance-2 Follows 15_min_wait
Check-Job-Instance-3 Follows 15_min_wait
Check-Job-Instance-4 Follows 15_min_wait
CHECK_FOR_INTRO Follows 15_min_wait RERUN ABENDPROMPT "One or
more jobs are stuck at INTRO.
Investigate before re-run."
Check-DB-Job Follows Job-Instance-1...4 I On success or failure continue
vob to be run on an active node
15_min_wait_rerun Follows Check-DB-Job
Job-Instance-1-rerun Follows Check-DB-Job On failure continue
Follows Check-DB-Job On failure continue
Follows Check-DB-Job On failure continue
Job-Instance-4-rerun Follows Check-DB-Job On failure continue
Check-Job-Instance-1-rerun Follows 15_min_wait_rerun
Check-Job-Instance-2-rerun, Follows 15_min_wait_rerun
Check-Job-Instance-3-rerun, Follows 15_min_wait_rerun
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref: DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 27 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
Check-Job-Instance-4-rerun, Follows 15_min_wait_rerun
CHECK_FOR_INTRO_RERUN Follows 15_min_wait_rerun I RERUN ABENDPROMPT "One or
more jobs are stuck at INTRO.
Investigate before re-run."
Check-DB-Job-rerun Follows Job-Instance-1
rerun
On failure Alert Operations
Job to be run on an active node
Schedule-complete Follows Check-DB-Job,
Check-DB-Job-rerun
3.1.1. Rerunning Failed Multi-Instance Batch Jobs
If the built-in rerun of any particular multi-instance job fails then
«the cause of the failure should be resolved
e the job should be rerun on all nodes
e the associated check job should then be rerun on all nodes
3.2 Any Active Node Batch Jobs
Certain BRDB processes can be run on any node of the Real Application Cluster (RAC).
In the schedule listings from sections 3.8 onwards, only the main jobs which perform the relevant task
are listed. However, they are implemented using a common schedule template consisting of the main
job running on each of the four nodes, and an additional parent job to co-ordinate them, as follows:
Job Name Job Dependency Rerun Action
RERUN ABENDPROMPT "Unable to determine an active
JobName BRDB node. Re-run?"
JobName1 Follows JobName STOP ABENDPROMPT "Appropriate Message"
JobName2 Follows JobName STOP ABENDPROMPT "Appropriate Message”
JobName3 Follows JobName STOP ABENDPROMPT "Appropriate Message"
JobName4 Follows JobName STOP ABENDPROMPT "Appropriate Message"
In this approach, once an available node has been selected the jobs defined for the other nodes are
cancelled.
3.3 Branch Database Jobs in other Schedules
(Scheduling HLD: Section 5.5 Branch Database Jobs in other schedules)
Although most of the BRDB processes are called from within the BRDB schedule, there are a number of
BRDB processes called from other application TWS schedules such as LFS and RDDS. This section
lists the schedules concerned.
RDDS: Scheduling HLD is DES/APP/HLD/0097
LFS: Scheduling HLD is DES/APP/HLD/0088
3.4 Monitoring Jobs
The BRDB schedule includes several monitoring jobs. These are jobs which raise an alert if a specified
process has not been completed by a required point in time. These jobs have been collected within a
single schedule, BRDB_MONITOR - see section 3.76 for details.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 28 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.5 Repeating/Daemon Processes
Currently there are 5 BRDB Host Interface Feeds that are run as ‘daemon’ processes within the daily
schedule:
e Guaranteed Reversals (Feed to NPS)
e Track and Trace (Feed to NPS)
* Pouch Collections (Feed to LFS)
e Pouch Deliveries (Feed to LFS)
e Transaction Confirmation (Feed to APOP)
After starting, these processes enter a cycle of ‘sleep and repeat’ - where they perform any necessary
processing, then sleep for a pre-defined time before ‘waking’ and running again. Each daemon process is
controlled by a separate system parameter, named after the Feed with a ‘_STOP_YN’ suffix, as follows:
* BRDB_REV_TXN_TO_NPS_STOP_YN
* BRDB_TT_TXN_TO_NPS_STOP_YN
* BRDB_PDEL_TO_LFS_STOP_YN
* BRDB_PCOL_TO_LFS_STOP_YN
* BRDB_TXN_CONF_TO_APOP_YN
When this parameter is set to ‘Y’ (from within the schedule using BRDBX011.sh) the daemon Feed
process will stop, although it should be noted that there will be a time delay between setting the
stop flag to ‘Y’ and the process actually terminating. This is because the daemon processes only
check the stop flag after ‘waking’ from a sleep or completing processing.
Additional metadata concerning the feeds (e.g. sleep time) can be queried in table
BRDB_HOST_INTERFACE_FEEDS as per the following:
°T interface_d
brdb_host_ _
HERE interface feed_name
yn, sleep_repeat_s
3.5.1 Node Failures
The daemon feed processes have been designed and developed to cope with node/instance failures
automatically. If a FAN event occurs for a node then:
« Database Column OPS$BRBD.BRDB_OPERATIONAL_INSTANCES.IS_AVAILABLE will be set
to 'N' for the failed instance
e View BRDB_FAD_HASH_CURRENT_INSTANCE will automatically redistribute the
FAD_HASHes of the failed node amongst the other operational nodes.
« Each of the daemon jobs reference the view BRDB_FAD_HASH_CURRENT_INSTANCE when
waking from sleep therefore the remaining operational nodes will work on any unprocessed data
from the FAD_HASHées associated with the failed node.
The failed TWS job can be set to SUCC. Refer to 4.3.3 for instance recovery.
3.5.2 Manually Stopping Daemon Processes
N.B. Stopping daemon feeds could result in the breaching of one or more service level
agreements.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 29 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
If there is a need to stop one of the above daemons manually then running the required job from the
following table will accomplish this:
Feed TWS Job
NPS Track & Trace BRDBX011_PAUSE_NPS_TT_COPY
NPS Guraranteed Reversals BRDBX011_PAUSE_NPS_GREV_COPY
LFS Pouch Collections BRDBX011_PAUSE_LFS_PCOL_COPY
LFS Pouch Deliveries BRDBX011_PAUSE_LFS_PDEL_COPY
APOP Transaction Confirmation BRDBX011_PAUSE_APOP_TC_COPY
3.5.3 Manually Starting Daemon Processes
N.B. Be aware that there should only be one feed job per instance running for each daemon,
ensure the jobs are NOT started more than once. Duplicate running feeds may result in a number of
unexpected and unpredictable failures (TT and GREV might be subject to deadlocking for example).
If there is a need to restart a stopped daemon manually then running the required jobs (i.e. changing the
start/stop flag and then restarting the daemon process on each node) from the following table will
accomplish this:
Feed Job - Daemon Process
NPS Track & Trace BRDBXO011_START_NPS_TT_COPY BRDBX003_TT_TO_NPS_Allld NOPAGE'
NPS Guraranteed Reversals BRDBX011_START_NPS_GREV_COPY I BRDBX003_GREV_TO_NPS_filll4 NOPAGE
LFS PCOL BRDBX011_START_LFS_PCOL_COPY I BRDBX003_PCOL_TO_LFS_flll4_NOPAGE
LFS PDEL BRDBX011_START_LFS_PDEL_COPY I BRDBX003_PDEL_TO_LFS_filll4 NOPAGE
APOP Transaction Confirmation I BRDBX011_START_APOP_TC_COPY I BRDBX003_TC_TO_APOP_filli4_NOPAGE
3.5.4 Track and Trace Feed
TT transactions will be flagged with 'Y' in column PROCESSED_YN once those transactions have been
inserted into the remote NPS database. Any transactions failing to be inserted due to some exception
will:
* have the PROCESSED_YN flag set to 'Y' if the exception was due to some data error’,
NPS_DELIVERED_TIMESTAMP will be left as NULL to allow support groups (SMC, SSC,
HOST) time to examine the exceptions before the archive/purge job removes the source rows.
* be left unprocessed if the exception is due to a network or instance failure; this allows the row to
be resent once the problem has been resolved (e.g. network is back up, NPS is back up etc)
3.5.5 Guaranteed Reversals Feed
GREV transactions will be flagged with 'Y' in column PROCESSED_YN once those transactions have
been inserted into the remote NPS database. Any transactions failing to be inserted due to some
exception will:
‘ 404 indicates that the job should be run concurrently on each BDB instance/node
? As defined in table OPS$BRDB.BRDB_ORACLE_ERROR_CODES where data_error_yn = 'Y'
’ As defined in table OPS$BRDB.BRDB_ORACLE_ERROR_CODES where data_error_yn = 'Y'
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN _ Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 30 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
* have the PROCESSED_YN flag set to 'Y' if the exception was due to some data error’,
NPS_DELIVERED_TIMESTAMP will be left as NULL to allow support groups (SMC, SSC,
HOST) time to examine the exceptions before the archive/purge job removes the source rows
«be left unprocessed if the exception is due to a network or instance failure; this allows the row to
be resent once the problem has been resolved (e.g. network is back up, NPS is back up etc)
3.5.6 Transaction Confirmation Feed to APOP
Transaction Confirmation feed to APOP differs from other Host Interface feeds in terms of transferring
transactions across to the remote APOP Database. Instead of inserting transactions into a target table in
the remote database the feed will invoke a PL/SQL package in the remote APOP Database and pass the
required transaction details as input parameters. The call to the remote PL/SQL package is made for
every unprocessed transaction on a record-by-record basis.
If a successful response is received from the remotely called package then the APOPConfirm
transaction in BRDB_RX_NRT_TRANSACTIONS table will be stamped as processed:
- processed_yn flag will be set to ‘Y’
- processed_timestamp will be set to systimestamp
- update_timestamp will be set to systimestamp
If an unsuccessful response is received then
- ‘tetry_attempts’ value will be incremented by 1
- update_timestamp will be set to systimestamp
However, the transaction belonging to the unsuccessful transfer will remain unprocessed and the feed
will pick the record up for transfer in its next processing cycle. If the number of re-try attempts exceeds a
set threshold value, as defined by a parameter called
‘BRDB_TXN_CONF_TO_APOP_RETRY_ATTEMPTS’ in System Parameters, then the feed will log an
exception in BRDB_HOST_INTERFACE_FEED_EXCP table. Still, the feed will continue to re-process
the transaction in its every processing cycle until the remote PL/SQL package returns a successful
response.
Before an APOPConfirm transaction can be transferred to the remote APOP Database the feed will
perform a set of validations to ensure that the NRT Payload is valid and to ensure that all required
transactional details to be passed as input parameter to the PL/SQL package are available. If any of the
validation check fails then the following attributes will be updated against the transaction and an
exception will be logged in BRDB_HOST_INTERFACE_FEED_EXCP table:
- processed_yn flag will be set to ‘Y’
- update_timestamp will be set to systimestamp
‘processed_timestamp’ column will be left as NULL to indicate that the transaction was not transferred to
the remote APOP Database. Note that transactions that fail during validation checks won't be re-
processed in the feed’s next processing cycle i.e., retry attempt is not applicable to such transactions as
no matter how many times the invalid transactions are re-processed they will fail the validation checks
every time due to invalid NRT Payload content.
Detailed information on this feed is available in the low level design document DEV/APP/LLD/0050.
5 As defined in table OPS$BRDB.BRDB_ORACLE_ERROR_CODES where data_error_yn
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 31 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.6 File Import Daemons (BRDBC038)
File imports are controlled by process BRDBC038 which in turn spawns child processes [BRDBC039,
BRDBC040]. The following are BRDB file imports that occur within the daily schedule:
e Post Office Essentials (POe) POLSAP PDF Load process (BRDB_POE_FROM_POLSAP)
[BRDBC039}
e Postcode Address File (PAF) Complete Load Process (BRDB_PAF_FROM_CD) [BRDBC040]
* Postcode Address File (PAF) Incremental/Additional Load Process (BRDB_PAF_ADD_LOAD)
[BRDBC040]
BRDBC038 uses the metadata stored in BRDB table BRDB_EXT_INTERFACE_FEEDS (see 0.1.1) to
control its behaviour - it can act as a daemon process (with a sleep repeat loop) or as a one off import.
Each instance of BRDBC038 will
e look in the INPUTSHARE_DIR_NAME directory for any files that fit the format mask as defined
in EXT_FILENAME_SEARCH_PATTERN.
e Each relevant file is registered in BRDB_FILE_AUDIT_TRAIL
© file is copied to AUDIT_DIR_NAME (if IS_AUDITABLE='Y')
o file is copied to BRDB_INPUT_DIR_NAME
o file is deleted from INPUTSHARE_DIR_NAME
e The command COMMAND_TO_RUN is invoked to process the registered files.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 32 of 183
HOST BRANCH DATABASE SUPPORT GUIDE ®
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.6.1 BRDB_EXT_INTERFACE_FEEDS Table‘
Column Name Data Type Desc
EXT_INTERFACE_FEED_NAME VARCHAR2(30) I Unique name of feed - Primary key
EXT_INTERFACE_DESC VARCHAR2(250) I Description of interface feed
INPUTSHARE_DIR_NAME. VARCHAR2(128) I Share (source files) path
AUDIT_DIR_NAME
VARCHAR2(128)
Optional - audit directory to copy files in Share to
BRDB_INPUT_DIR_NAME
VARCHAR2(128)
Input directory to move files from Share into
BRDB_LOAD_DIR_NAME VARCHAR2(128) I Local working directory accessible by Oracle [dir BRDB_LOAD_DIR]
EXT_FILENAME_SEARCH_PATTERN I VARCHAR2(128) I String to search for files in sinputShareDir
COMMAND_OR_SCHEDULE VARCHAR2(8) Issue command or generate schedule
Command I Invoke COMMAND_TO_RUN
Schedule _} Do not invoke any command, leave to TWS
COMMAND_TO_RUN VARCHAR2(200) I Invoke Path + executable
NOt if sExecutePerFile = Y then invoke
Path + executable + path_of file/filename
REMOTE_APPLICATION VARCHAR2(8) Description of remote application (e.g. POLSAP)
PROCESSED_SUFFIX VARCHAR2(3) File extension to rename existing extension once processing is completed on a file
SLEEP_REPEAT_YN VARCHAR2(1) Daemon (sleep and loop) or execute once flag
Value Description
Y Daemon feed
N Process is invoked once by TWS
EXECUTE_PER_FILE VARCHAR2(1) Execute command for each file found or at end
4 Extracted from DEV/APP/LLD/1354
© Copyright Fujitsu Ltd 2011
UNCONTROLLED IF PRINTED
FUJITSU RESTRICTED (COMMERCIAL IN _ Ref. DES/APPISPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
Page No: 33 of 183,
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE ®
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
WAIT_FOR_SCHEDULE_COMPLETE
VARCHAR2(1)
Wait for schedule job to finish before creating next job
IS_AUDITABLE
VARCHAR2(1)
Copy file to audit directory Y or N
Value Description
Y
Copy appropriate files in SHARE to audit dir
N
Skip copying to audit dir
SLEEP_REPEAT_SECS
NUMBER(S)
Time to sleep between iterations for a daemon feed. Time to sleep when looking for at least 1 file to
process in a non-daemon feed
ALERT_AFTER_SECS
[)
Number of iterations without finding a file to process before recording exception
Value Description
No exception logged if zero files found
>0
Log exception if non-daemon process and zero
files found within timeframe
A
Log exception if daemon process and zero files
found on exit of loop
© Copyright Fujitsu Ltd 2011
UNCONTROLLED IF PRINTED
FUJITSU RESTRICTED (COMMERCIAL IN. Ref: DES/APP/SPG/0001
CONFIDENCE)
Version: 3.0
Date: 21-Sep-11
Page No: 34 of 183
FUJ00088797
FUJ00088797
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.6.2 Single Node Job
File import daemons are designed to run on only one node at any one time (See 3.2)
3.6.3. Post Office Essentials [BRDBC039]
POLSAP PDF files are made available to BRDB via a share. BRDBC038 registers those PDF files and
invokes BRDBC039 which
e Loops through files in BRDB_FILE_AUDIT_TRAIL (where process_name =
‘BRDB_POE_FROM_POLSAP' and file_status = 'N')
* converts each PDF to one or more PNG files (1 PNG for each PDF page)
« uploads each PNG file into BRDB table OPS$BRDB.BRDB_EXT_FEED_REPORTS
e sets the column FILE_STATUS in BRDB_FILE_AUDIT_TRAIL to 'C' (complete)
* exceptions are logged in OPS$BRDB.BRDB_HOST_INTERFACE_FEED_EXCP
* each processed file (whether PDF or PNG) has its extension uppercased in order to allow BRDB
housekeeping to remove after an appropriate period of time has elapsed.
3.6.4 BRDB Postcode Address File Complete [BRDBC040]
3.6.4.1 Process Overview
The means by which the Post Office queries British postcodes via the Counter, was through the solution
known as QAS. QAS was hosted on an Apache Web Server (Windows Server) in the datacentre and
the data provided through a service.
BRDB PAF is known as PAF Replacement because it replaced the previous solution (provided by an
external provider) with an in-house solution accessed by the counter directly within the Branch Database.
The Load Process at a very high level does in essence: -
- Find and validate files
- Prepare the table and load the data
- Ready the table for access by the estate and complete.
It is important to note that there are two PAF tables. The main table, referred to as
PAF_ADDRESS_POINT and then a secondary table, PAF_ADDRESS_POINT_SAV which holds the
data from the previous load iteration of the load process. When the load process is therefore envoked,
the older table is prepared and loaded such that, should there be a failure of any kind during the load
process, the risk to the estate of not being able to access PAF data is non existent.
3.6.4.2 Process Execution and Flow
BRDBC040 gets executed by the BRDBC038 parent process (see section 3.6). BRDB_PAF_FROM_CD
is the “external feed” identifier for BRDB PAF Complete and is specifically executed as a process when
the following call is made: -
${BRDB_PROC}/BRDBC038 BRDB_PAF_FROM_CD “BRDBBDAY“
Section 3.6 details the activities of BRDBC038, but for completeness it is mentioned here too.
BRDBC038, in the context of BRDB PAF Complete (please see the table below — section 3.6.4.4 - for all
metadata values, including file formats, directories, et cetera) has the following logic flow: -
i. It looks for the files in the INPUTSHARE_DIR_NAME directory, of the form defined for
EXT_FILENAME_SEARCH_PATTERN, which in this case is: *compstc*.*.paf
ii. For every file found: -
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 35 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
a. It registers the file in the table BRDB_FILE_AUDIT_TRAIL with a file_status of ‘N’
(for New)
b. Copies the file from the source directory (see i. above) to the BRDB_INPUT_DIR_NAME
directory
c. Only once ail files are successfully complete, will the transaction commit, i.e. ail files will
either show a file_status of ‘N’ or there will be no record at all
iii. It then executes BRDBC040 using the command-line call in COMMAND_TO_RUN , which in
this case is (see also section 3.78.1): -
${BRDB_PROC}/BRDBC040 BRDB PAF FROM CD
iv. BRDBC040 then using the file-metadata found in BRDB_FILE_AUDIT_TRATL will verify that
all file headers and all file trailers are valid and expected
v. It then prepares the database table PAF_ADDRESS_POINT_SAV for loading by: -
a. Truncating the table and ...
b. Removing the primary key and all remaining indexes
vi. It then calls the PAF Importer (pafimporter.jar Java program) which loads the data (~30
million rows) one file at a time.
The importer can be configured using loader properties found in
/app_sw/brdb/java/paf/config/pafimport.properties such as commit size,
amoungst others. The importer also uses a_ posttown-to-county mapping file
(/app_sw/brdb/java/paf/config/post_town_counties_mapping.csv) when
importing the PAF data in order to populate county data correctly, which is not likely to
create any problems but is merely mentioned here for completeness.
BRDBC040 passes three parameters to the PAF Importer: -
- The type of load, in this case a “full” load (‘M’)
- The absolute path of the file to load (executed in order at a global level)
- The table in which to load the data
vii. It then, having loaded all files successfully, will insert all records from PAF_ADDRESS_POINT
into PAF_ADDRESS_POINT_SAV, previously added by an execution of the PAF Additional
process (see Section 3.6.5) prior to the execution of this process. This insert will include a
SQL query based on the following predicate:
WHERE additional_data = 'T'
viii. It then performs some post-load processing to finalise the PAF table for access by the
estate, this includes: -
a. Creating the primary key and all other indexes (of which there are 14; with the PK, 15)
b. Analyzing the table, providing Oracle with accurate statistics.
c. Updating BRDB metadata in BRDB_LSYSTEM_PARAMETERS with the value of the
current LIVE synonym. The parameter is called PAF_TABLE_SET and will have a value
of ‘A’ or ‘B’, depending on whichever table is the live table.
d. Finally, the synonyms that dictate which table is primary and which the secondary, are
then switched. In this case the secondary table is loaded and then becomes the primary
at the end of the process, i.e. the synonym switch is the very last step.
e.g. Assume that the PAF_ADDRESS_POINT_A table is the table being loaded (this is
the case if the PAF_ADDRESS_POINT_SAV synonym references this table). When the
switch occurs, the PAF_ADDRESS POINT A table is assigned the
PAF_ADDRESS_POINT synonym and the “B” table (former primary) the
PAF_ADDRESS_POINT_SAV synonym.
ix. BRDBC040 then finishes by setting the £ile_status for all files to ‘c’ and completes,
handing control back to BRDBC038.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref: DES/APP/SPG/0001
CONFIDENCE) Version 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 36 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.6.4.3. PAF Load Process - Failure and Recovery
BRDBC040 is not re-runnable. There are a number of reasons for this, the most important of which is
the fact that this process deals with files which are delivered by an external party. Therefore the cause
of the failure must be determined in order to find the best possible set of recovery actions to perform,
including the possibility that the files are corrupt or that they contain erroneous data.
Should there be a failure during this load process, the TWS stdout job log will be required in order to
determine what the next step should be in order to get the PAF data loaded with the least amount of
hassle as possible.
dated ses ant bea pe enon 388) heb fr he BROS snd
3.6.4.3.1 Failure Scenario 1 — Pre-load Failure
Scenario 1 assumes that the PAF Importer (Java program) has not yet been called by BRDBC040 and a
failure occurs:
i. The TWS job log will be required to determine the point of failure.
ii. The likely causes are: -
a. Available space in the BRDB_INPUT_DIR_NAME (see 3.6.4.4) directory has been
exhausted. Solution: Free up disk space for the required files.
b. The file being read has been removed mid-execution. Solution: Find the cause of the
removal.
c. The files being loaded have already been loaded, i.e. as every file name should be
unique, if files are named the same as files which have been loaded before, the process
will skip and inform of a duplicate. Solution: Determine the reason for the file being
duplicated. This should never be the case unless files are manually created/renamed.
3.6.4.3.2 Failure Scenario 2 — PAF Loader (Java code) Failure
Scenario 2 assumes that the load process has been running for a length of time and having loaded 1 or
more files, fails:
i. The TWS job log will be required to determine the error
ii. The likely causes are: -
a. Available space in either the PAF_DATA or PAF_INDEX tablespaces has been
exhausted. Solution: Increase the size of the tablespaces
b. An erroneous record has been read by the PAF Importer. Solution: An exercise to
determine the erroneous record will be required. Activities in this regard could include
comparing the failed file to that of a previous (successful) month.
c. The file being read has been removed mid-execution. Solution: Find the cause of the
removal.
3.6.4.3.3 Failure Scenario 3 — Post-load Failure
Scenario 3 assumes that the load process has completed successfully with all data having been loaded.
As the post-load process is an Oracle PL/SQL procedure:
i. The TWS job log will be required to determine the Oracle error.
ii. The likely causes are: -
a. Available space in either the PAF_INDEX or BRDB_TEMP4 tablespaces has been
exhausted. Solution: Increase the size of either of the tablespaces.
b. An unexpected Oracle error occurred. Solution: Once the error is known, and the
appropriate advice from the DBA Support or Host Development teams has been sought,
the appropriate task to correct the error can be undertaken.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 37 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
c. The associated BRDB instance either crashed or was mistakenly shutdown during the
process. Solution: Startup the instance.
3.6.4.3.4 Recovery Tasks
Following a failure of BRDBC040, a number of tasks will be required, the first of which are described in
the sections prior to this. It is important to know: -
i. In the first instance why the PAF Load process failed (see all point i.’s above) and ...
ii. Thereafter determining the extent to which the job had completed, e.g. which of the above
failure scenarios is applicable.
Once the failure is known, the likely recovery task(s) would include analysis and investigation (initially by
Development) and then actions on the LIVE server to follow; the solutions to most of which, are detailed
in the above scenarios.
Ultimately though, the re-running of the Load process will need to occur and the following is a set of
guidelines and tasks to complete in order to successfully re-run BRDBCO040. Invariably all failure
scenarios and subsequent recovery will include a combination of the following sections.
3.6.4.3.4.1 I Scenarios Regarding File Processing
In all failure scenarios, the table BRDB_FILE_AUDIT_ TRAIL will show a file status of ‘E’
(‘Errorred’) for all files processed in that run of the PAF Loader. The following SQL will help show file
status’ (change accordingly - in the SQL below - to the date the job ran): -
' file name, file status
ops$brdb.brdb file audit trail
process name = 'BRDB PAF OM CD'
file name LIKE '%<TODAY MDD>%"
ORDER BY status_change_timestamp;
In every scenario then, the files will either be in the source directory (INPUTSHARE_DIR_NAME) or the
input directory (BRDB_INPUT_DIR_NAME) and an entry for each file will exist in the database.
Therefore in order to re-run the process: -
1. Either the conditions at which the original process ran, need to be re-created
a. All files need to be located and moved back to the source directory ensuring that the file
extentions are all *.paf and not *. PAF
b. The file entries (in BRDB_FILE_AUDIT_TRAIL) for this particular instance of
BRDBC040 must be removed
E
ROM ops$brdb.brdb file audit trail
WHERE process name = 'BRDB PAI
AND file name LIKE '%<T
2. Or artificial conditions need to be created and BRDBC040 manually re-run
a. All files in the input directory need to be located and then ensure that file extentions are
all *.paf and not *. PAF
b. The file entries (in BRDB_FILE_AUDIT_TRAIL) for this particular instance of
BRDBC040, setting file_status to ‘N’ (‘New’)
UPDATE ops$brdb.brdb file_audit_trail
file status = 'N'
2E process name = 'BRDB P. FROM CD"
AND file_name LIKE '%<TODAY YYYY DD>%';
c. Manually execute BRDBC040 as specified against point (iii.) of section 3.6.4.2
3.6.4.3.4.2 __ Scenarios Regarding Data Loading
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 38 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
As above, in all failure scenarios, the table BRDB_FILE_AUDIT_TRAIL will show a file_status of
‘E’ (‘Errorred’) for all files processed in that run of the PAF Loader. The PAF_ADDRESS_POINT_SAV
table will either be partially, or fully populated or not at all.
This section is relevant to Failure Scenarios 2 or 3 above. Therefore in order to re-run the process: -
1. Either the table is partially populated, in which case a re-run of the process (referring to section
3.6.4.3.4.1) is required.
2. Or the table is completely and correctly populated. In order to not have the initial load process
repeated (and waste time and resource repeating it), manual actions to complete the process are
recommended: -
a. Determine which table the synonym PAF_ADDRESS_POINT_SAV currently references:
ECT table name
FROM all synonyms
WHERE synonym_name = "PAF ADDRESS POINT SA
b. Check to see whether the table has had any indexes created on it.
ECT COUNT (1)
ROM all indexes
; table name =
AND owner =
c. If (b) is NO and in order to not have the entire load process repeated, execute the
following to complete the process: -
i. Create a dummy index:
CREATE INDEX paf owner.pap x c ind
ON paf owner.<TABLE FROM ABOVE SQL> (county)
paf index I
d. If (b) is YES then execute the post-load process (as brdbblv4 on BRDB4): -
EXEC paf_owner.pkg_brdb paf_common.post_paf_dataload;
e. Update all file entries (in BRDB_FILE_AUDIT_TRAIL) for this particular instance of
BRBC040, setting £ile_status to ‘c’ (‘Complete’): -
UPDATE ops$brdb.brdb file_audit_trail
file status =
E process name
AND file_name LIKI
WH "BRDB P:
'S<TODAY YY
3.6.4.3.4.3. Switching Synonyms
This section details the switching of the PAF table synonyms in the event this task is required. It is
highly unlikely that this section will ever be used. However in a scenario where it is found that the full
data having just been loaded is in some way causing a problem or is corrupt, then the following
commands would help in enabling a synonym switch, effectively allowing the former LIVE (now
secondary) table to be made LIVE (primary) again:
1. First determine which PAF table is being referenced as the primary table: -
LECT synonym name, table_name
FROM all synonyms
table owner = 'PAF
D synonym_name
OWNER'
ADDRE
2. Then update the BRDB metadata to reflect the change to new primary: -
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 39 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
E ops$brdb.brdb system parameters
parameter text = '<LIVE TABLE
parameter_name = 'PAF_TA
parameter_text = 'A'
3. Then make the switch by first changing the secondary to the primary and then visa-versa: -
TE OR REPLA IC SYNONYM paf address point
FOR paf_owner.<SECONDARY_TABLE_FROM_ABOVE>;
OR REPLACE JONYM paf address _point_sav
paf_owner.<PRIMARY_TABLE_FROM_ABOVE>
3.6.4.4 External Feed Metadata
ci MN NAME DESCRIPTION VALUE
Note: This metadata is stored in BRDB_EXT_INTERFACE_FEEDS, identified by the row "WHERE ext_interface_feed_name
'BRDB_PAF_FROM_CD".
INPUTSHARE_DIR_NAME PAF file source directory (DAT) Japp/brdb/trans/supportiworking
BRDB_INPUT_DIR_NAME BRDB input directory Japp/brdb/trans/externalinterface/input
AUDIT_DIR_NAME BRDB audit directory NIA
BRDB_LOAD_DIR_NAME BRDB PAF load directory Japp/brdbitrans/externalinterfacelloaddir
EXT_FILENAME_SEARCH_PATTERN I PAF file wildcard *compstc*.*.paf
COMMAND_TO_RUN ‘Command that BRDBCO38 runs ${8RDB_PROCYBRDBCO40 BROB_PAF_FROM_CD
EXECUTE_PER_FILE BRDBC038 number of executions I N
REMOTE_APPLICATION Data description POLPAFM
PROCESSED_SUFFIX File post-process suffix indicator PAF
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 40 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.6.5 I BRDB Postcode Address File Ad nal [BRDBC040]
The majority of the information in Section 3.6.4 BRDB Postode Address File Complete is applicable
here, however this section is concerned more with the information pertaining to the Additional Load
Process.
3.6.5.1 Process Overview
This process differs from the PAF Complete process in that the main table that this particular process
accesses is PAF_ADDRESS_POINT and is the LIVE table used by the Counter. This process does not
reference or work with the PAF_ADDRESS_POINT_SAV table in any way.
3.6.5.2 Process Execution and Flow
The PAF Additional process adds data to PAF_ADDRESS_POINT table when required. This process is
triggered when an additional file is found by the process
As in Section 3.6.4, BRDBC040 gets executed by the BRDBC038 parent process. However,
BRDB_PAF_ADD_LOAD is the “external feed” identifier for BRDB PAF Additional. As in the case of
PAF Complete, it is executed as a process when the following call is made: -
${BRDB_PROC}/BRDBC038 BRDB_PAF_ADD_LOAD “BRDBBDAY“
BRDBCO038, in the context of BRDB PAF Additional (see table below in Section 3.6.5.4 for related
metadata) has the following logic flow: -
i. It looks for the files in the INPUTSHARE_DIR_NAME directory, of the form defined for
EXT_FILENAME_SEARCH_PATTERN, which in this case is: *compstd* .*.paf
ii. There is expected to ever only be a single file for every execution of this job. When the file
is found: -
a. It registers the file in the table BRDB_FILE_AUDIT_TRAIL with a file_status of ‘N’
b. Copies the file from the source directory to the BRDB_INPUT_DIR_NAME directory
c. Creates an additional copy of the file in the AUD/T_DIR_NAME directory
d. Only once the fileis successfully complete, will the transaction commit, i.e. an entry will
either show a file_status of ‘N’ or not at all
iii. It then executes BRDBC040 using the command-line call in COMMAND_TO_RUN , which in
this case is (see also section 3.79.1): -
${BRDB_PROC}/BRDBC040 BRDB_PAF_ADD_LOAD
iv BRDBC040 then using the file-metadata found in BRDB_FILE_AUDIT_TRAIL will verify that
the file header and it's trailer is valid and expected
v. It then calls the PAF Importer (pafimporter.jar Java program) which: -
a. Will delete all records in PAF_ADDRESS_ POINT added by a previous execution of a PAF
Additional process (between the last execution of PAF Complete and now). This delete
is based on the following predicate:
w
b. Will then Load only the new records found in the PAF Additional file.
BRDBC040 passes three parameters to the PAF Importer for PAF Additional: -
- The type of load, in this case a “additional” load (‘D’)
- The absolute path of the file to load (executed at a global level)
- The table in which to load the data
— additional_data 'v!
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 41 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
NOTE:
e All subsequent additional files should be cumulative, i.e. should include all data
delivered by POL in previous additional files.
e« Any subsequent PAF Complete loads should always include all data previously
delivered by POL in additional files.
vi. BRDBC040 then finishes by setting the £ile_status for all files to sc’ and completes,
handing control back to BRDBC038.
3.6.5.3 PAF Load Process - Failure and Recovery
BRDBC040 is not re-runnable. Should there be a failure during this load process, the TWS stdout job
log will be required in order to determine what the next step should be in order to get the PAF data
loaded.
sedan roto be tg aon 360) oo fre BROS sad
3.6.5.3.1 Failure Scenario 1 — Pre-load Failure
Scenario 1 assumes that the PAF Importer (Java program) has not yet been called by BRDBC040 and a
failure occurs:
i. The TWS job log will be required to determine the point of failure.
ii. The likely causes are: -
a. Available space in the BRDB_INPUT_DIR_NAME (see 3.6.4.4) directory has been
exhausted. Solution: Free up disk space for the required files.
b. The file being read has been removed mid-execution. Solution: Find the cause of the
removal.
c. The file being loaded has already been loaded. Solution: Determine the reason for the
file being duplicated. This should never be the case unless the file was manually
created/renamed.
3.6.5.3.2 Failure Scenario 2 — PAF Loader (Java code) Failure
Scenario 2 assumes that the load process has been executed and fails:
i. The TWS job log will be required to determine the error
ii. The likely causes are: -
a. Available space in either the PAF_DATA or PAF_INDEX tablespaces has been
exhausted. Solution: Increase the size of the tablespaces
b. The DELETE of records in the table (marked additional_data = ‘T’) has failed. Solution:
See following section.
c. The INSERT of records in the table has failed (similar to (a.) above). Solution: See
following section.
d. An erroneous record has been read by the PAF Importer. Solution: An exercise to
determine the erroneous record will be required. Activities in this regard could include
comparing the failed file to that of a previous (successful) month.
e. The file being read has been removed mid-execution. Solution: Find the cause of the
removal.
3.6.5.3.3 Recovery Tasks
Following a failure of BRDBC040, a number of tasks will be required, the first of which are described in
the sections prior to this. It is important to know: -
i In the first instance why the PAF Load process failed (see all point i.’s above) and ...
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 42 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
ii. Thereafter determining the extent to which the job had completed, e.g. which of the above
failure scenarios is applicable.
Once the failure is known, the likely recovery task(s) would include analysis and investigation (initially by
Development) and then actions on the LIVE server to follow; the solutions to most of which, are detailed
in the above scenarios.
Ultimately though, the re-running of the Load process will need to occur and the following is a set of
guidelines and tasks to complete in order to successfully re-run BRDBC040.
3.6.5.3.3.1 I Scenarios Regarding File Processing
In all failure scenarios, the table BRDB_FILE AUDIT TRAIL will show a file status of ‘E’
(‘Errorred’) for any files processed in that run of the PAF Loader. The following SQL will help show file
status’ (change accordingly - in the SQL below - to the date the job ran): -
T file name, file status
ops$brdb.brdb file audit trail
process name = 'BRDB PAF ADD L
file name LIKE '%<TODAY ¥
BY status_change_timestamp
In every scenario then, the files will either be in the source directory (INPUTSHARE_DIR_NAME) or the
input directory (BRDB_INPUT_DIR_NAME) and an entry for each file will exist in the database.
Therefore in order to re-run the process: -
1. Either the conditions at which the original process ran, need to be re-created
a. The file needs to be located and moved back to the source directory ensuring that it's
file extention is *.paf and not *. PAF
b. The file entry (in BRDB_FILE_AUDIT_TRAIL) for this particular instance of BRDBC040
must be removed
E
ops$brdb.brdb file audit trail
WHERE process name = 'BRDB PA D LO
AND file_name LIKE '%<TODAY Y
2. Or artificial conditions need to be created and BRDBC040 manually re-run. As PAF Additional
processes a single file, the benefits of leaving just that single file in the target directory are
outweighed by the benefits of a clean run (as in 1. above).
a. The file in the input directory needs to be located and ensure that it's extention is *.paf
and not *. PAF
b. The file entry (in BRDB_FILE_AUDIT_TRAIL) for this particular instance of BRDBC040,
setting file_status to ‘N’ (‘New)
UPDAT
ops$brdb.brdb file_audit_trail
file status = 'N'
E process name = 'B)
AND file _name LIKE '%<
ADD LOAD'
AY YYYYMMDD>%'";
c. Manually execute BRDBC040 as specified against point (iii.) of section 3.6.5.2
3.6.5.3.3.2 I Scenarios Regarding Data Loading
As above, in all failure scenarios, the table BRDB_FILE_AUDIT_TRAIL will show a file status of
‘E’ (‘Errorred’) for the file processed in that run. The PAF_ADDRESS_POINT table will have additional
data, either partially deleted or inserted or neither (old data still exists).
This section is relevant to Failure Scenarios 2 above. Therefore in order to re-run the process: -
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 43 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
1. Either the table still has all the additional data from the previous run populated, in which case a
re-run of the process is required.
2. Or the table is partially populated, in which case a re-run of the process is required. Counting
the number of records which need to be deleted is not the best of ideas, but will give an
indication of the extent to which the process failed; whether that was a failure of the delete or the
insert it is difficult to tell without the TWS stdout log/evidence: -
INT (1)
paf address point
additional_data = 'T';
3.6.5.4 External Feed Metadata
COLUMN NAME DESCRIPTION VALUE
Note: This metadata is stored in BRDB_EXT_INTERFACE_FEEDS, identified by the row "WHERE ext_interface_feed_name =
'BRDB_PAF_ADD_LOAD”
INPUTSHARE_DIR_NAME PAF file source directory (DAT) Japp/brdb/trans/supportworking
BRDB_INPUT_DIR_NAME BRDB input directory Japp/brdb/trans/externalinterface/input
AUDIT_DIR_NAME BRDB audit directory /app/brdb/trans/audit/externalinterfaceaudit/paf
BRDB_LOAD_DIR_NAME BRDB PAF load directory Japp/brdb/trans/externalinterface/loaddir
EXT_FILENAME_SEARCH_PATTERN I PAF file wildcard *compstd*.*.paf
COMMAND_TO_RUN ‘Command that BRDBC038 runs ‘${BRDB_PROC}/BRDBC040 BRDB_PAF_ADD_LOAD
EXECUTE_PER_FILE BRDBC038 number of executions N
REMOTE_APPLICATION Data description POLPAFD
PROCESSED_SUFFIX File post-process suffix indicator PAF
3.6.6 BRDB Postcode Address File — End-to-End Process
This section exists to give background information on the current end-to-end process; from receiving the
files from the Post Office to the final data load.
The process is as follows:
1. POL to Refdata: Fujitsu receives the files from the Post Office on a CD in compressed format
2. Refdata: The Reference Data team “unpack” the files into a format recognised by the DAT Host
process (*.gz) that will copy the files.
3. Refdata to DAT: The files are then manually copied to a local SAMBA share which is mounted
to the DAT server. The target directory on the DAT server is specified as
/bvnw01/rdmc/Z_PAF. This becomes the source for the next step.
4. DAT to BRDB: A script (paf_copy.ksh) is then executed, which will unzip the files, rename
them to a filename format expected by the BRDB TWS Schedule and then copies them to a
separate, but locally mounted NAS share specified as /nas/brdb_sup/working. This share
is a NAS share and as such is mounted locally mounted on all nodes of the BRDB cluster as
/app/brdb/trans/support/working. As mentioned in previous sections, this directory is
seen by BRDBCO40 as the INPUTSHARE_DIR_NAME and is the directory from which the
process finds the files to process.
5. BRDB: When the PAF-related TWS Scheduled jobs are executed the files are “picked up” and
processed as described above.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 44 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.7 BRDB Schedules and Failover
The Scheduling tool used for running BRDB (and other HNG-X schedules) is TWS. TWS needs to
undergo a number of steps in a failover scenario. These are detailed in the relevant TWS and
scheduling documentation. However, it is still the case that TWS (as with other applications) requires
the DNS reconfigured before post-failover testing can begin. To clarify, failover refers only to the
database failover from the primary database cluster (Iprpbdb001 - Iprpbdb004) to the standby database
cluster (Iprpbds001 - Iprpbds004) and not a full campus failover, e.g. IRE11 to IRE19.
See Steps [7.] and [8.] of Section 6.1 for more on allowing applications seamless access to BRDB on
database primary-to-standby cluster post-failover.
3.8 Schedule BRDB_PAUSE_FEED3
This schedule is run daily. It stops the two NPS copy processes prior to the starting of the daily BRDB
schedule. It consists of two tasks which can be run on any active node; see section 0 above for details.
Only the two parent jobs are included here, which are:
BRDBX011_PAUSE_NPS_TT_COPY
BRDBX011_PAUSE_NPS_GREV_COPY
3.8.1. Dependencies
Schedule BRDB_PAUSE_FEED3 depends on the completion of schedule BRDB_BKP_COMPL.
3.8.2 Job BRDBX011_PAUSE_NPS_TT_COPY
This job stops the copying of Track and Trace transactions to NPS, by setting a system parameter (see
section 3.5).
3.8.2.1. Implementation
This job is implemented by a call to the shell script BRDBX011.sh specifying the relevant system
parameter name BRDB_TT_TXN_TO_NPS_STOP_YN and value “Y” (i.e. System parameter in
BRDB_SYSTEM_PARAMETER. parameter_text named 'BRDB_TT_TXN_TO_NPS_STOP_YN' is set to
Y%).
3.8.2.2 Rerun Action
Rerun the job once the underlying problem has been resolved, unless the the node on which it was
running is now down; rerun one of the cancelled jobs from one of the other instances instead.
3.8.3 Job BRDBX011_PAUSE_NPS_GREV_COPY
This job stops the copying of Reversals transactions to NPS, by setting a system parameter (see section
3.5).
3.8.1.1. Implementation
This job is implemented by a call to the shell script BRDBX011.sh specifying the relevant system
parameter name BRDB_REV_TXN_TO_NPS_STOP_YN and value “Y” (i.e. System parameter in
BRDB_SYSTEM_PARAMETER parameter_text named 'BRDB_REV_TXN_TO_NPS_STOP_YN' is set
to'Y’).
3.8.1.2 Rerun Action
Rerun the job once the underlying problem has been resolved, unless the the node on which it was
running is now down; rerun one of the cancelled jobs from one of the other instances instead.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 45 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.9 Schedule BRDB_STARTUP
This schedule is run daily. It runs the BRDB start of day utility. It consists of a single task which can be
run on any active node; see section 0 above for details. Only the parent job BRDBCO001 is included here.
Additional monitoring is required so that an alert is raised if this job has not completed by 06:00. This is
implemented within the BRDB_MONITOR schedule — see section 3.76.
3.9.1 Dependencies
Schedule BRDB_STARTUP depends on the completion of schedule BRDB_PAUSE_FEED3.
3.9.2 Job BRDBCO001
This job runs the BRDB start of day utility in order to create "tomorrow's" partitions. BRDB's system date
is incremented by one.
3.9.2.1. Implementation
This job is implemented by a call to the executable BRDBC001
3.9.2.2 Rerun Action
Check the partition metadata is as expected (refer to 5.3.3.1), if the metadata appears OK then fix the
underlying problem (that caused the abend), raise a high priority call with 4th line support and then rerun
the job.
If the rerun fails then do not attempt to rerun a 3rd time, liase with 4th line support - the resolution should
be reached before 6 p.m. that day.
3.10 Schedule BRDB_START_FEED3
This schedule is run daily. It prepares for the running of the two NPS copy processes by reversing the
changes that stopped them earlier in the schedule. It consists of two tasks which can be run on any
active node; see section 0 above for details. Only the two parent jobs are included here, which are:
BRDBX011_START_NPS_TT_COPY
BRDBX011_START_NPS_GREV_COPY
3.10.1 Dependencies
Schedule BRDB_START_FEED3 depends on the completion of schedule BRDB_STARTUP.
3.10.2 Job BRDBX011_START_NPS_TT_COPY
This job prepares for the starting of the copying of Track and Trace transactions to NPS, by setting a
system parameter (see section 3.5).
3.10.2.1 Implementation
This job is implemented by a call to the shell script BRDBX011.sh specifying the relevant system
parameter name BRDB_TT_TXN_TO_NPS_STOP_YN and value “N”.
3.10.2.2 Rerun Action
Alert Operations on failure!
3.10.3. Job BRDBX011_START_NPS_GREV_COPY
This job prepares for the starting of the copying of Reversals transactions to NPS, by setting a system
parameter (see section 3.5).
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 46 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.10.3.1 Implementation
This job is implemented by a call to the shell script BRDBX011.sh specifying the relevant system
parameter name BRDB_REV_TXN_TO_NPS_STOP_YN and value “N’”.
3.10.3.2 Rerun Action
Alert Operations on failure,
3.11 Schedule BRDB_TT_TO_NPS3
This schedule is run daily to start the Track and Trace NPS data feed. It consists of a single task which is
run on each active node by jobs named BRDBX003_TT_TO_NPS_1...4_NOPAGE.
3.11.1 Dependencies
Schedule BRDB_TT_TO_NPS3 depends on the completion of schedule BRDB_START_FEED3.
3.11.2 Job BRDBX003_TT_TO_NPS_1...4 NOPAGE
These jobs (one per node) start the feed that copies the Track and Trace transactions to NPS.
3.11.2.1 Implementation
These jobs are implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_TT_TXN_TO_NPS.
3.11.2.2. Database Link Information
NBX_TT_HARVESTER_AGENT_1@NPS1
3.11.2.3. Rerun Action
RSRUHISMMMUE] See 3.5.1
3.12 Schedule BRDB_GREV_NPS3
This schedule is run daily to start the Reversals NPS data feed. It consists of a single task which is run
on each active node by jobs named BRDBX003_GREV_TO_NPS_1...4_NOPAGE.
3.12.1 Dependencies
Schedule BRDB_GREV_NPS3 depends on the completion of schedule BRDB_START_FEED3.
3.12.2 Job BRDBX003_GREV_TO_NPS_1...4_NOPAGE
These jobs (one per node) start the feed that copies the Reversals transactions to NPS.
3.12.1.1 Implementation
These jobs are implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_REV_TXN_TO_NPS.
3.12.1.2 Database Link Information
NBX_GREV_AGENT_1@NPS2
3.12.1.3 Rerun Action
FSR See 3.5.1
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 47 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.13 Schedule BRDB_PAUSE_FEED1
This schedule is run daily at 07:50. It stops the two NPS copy processes prior to the start of day
processing. It consists of two tasks which can be run on any active node; see section 0 above for details.
Only the two parent jobs are included here, which are:
BRDBX011_PAUSE_NPS_TT_COPY
BRDBX011_PAUSE_NPS_GREV_COPY
Additional monitoring is required so that an alert is raised if this job has not completed by 08:00. This is
implemented within the BRDB_MONITOR schedule — see section 3.76.
3.13.1. Dependencies
Schedule BRDB_PAUSE_FEED1 depends on the completion of schedules BRDB_STARTUP and
BRDB_START_FEED3.
3.13.2 Job BRDBX011_PAUSE_NPS_TT_COPY
This job stops the copying of Track and Trace transactions to NPS, by setting a system parameter (see
section 3.5).
3.13.2.1 Implementation
This job is implemented by a call to the shell script BRDBX011.sh specifying the relevant system
parameter name BRDB_TT_TXN_TO_NPS_STOP_YN and value “Y”.
3.13.2.2. Rerun Action
AlertOperations on failure!
3.13.3. Job BRDBX011_PAUSE_NPS_GREV_COPY
This job stops the copying of Reversals transactions to NPS, by setting a system parameter (see section
3.5).
3.13.3.1 Implementation
This job is implemented by a call to the shell script BRDBX011.sh specifying the relevant system
parameter name BRDB_REV_TXN_TO_NPS_STOP_YN and value “Y”.
3.13.3.2 Rerun Action
Alert Operations on failure.
3.14 Schedule BRDB_COMPLETE
This schedule is run daily. It checks that the BRDB schedule has completed and creates a flag file via
the job CREATE_BRDB_COMPLETE_FLAG.
3.14.1. Dependencies
Schedule BRDB_COMPLETE depends on the completion of schedules BRDB_BKP_COMPL,
BRDB_STARTUP and BRDB_PAUSE_FEED1.
3.14.2 Job CREATE_BRDB_COMPLETE_FLAG
This job creates the flag file /opt/tws/FLAGS/BRDB_COMPLETE_FLAG.
3.14.2.1 Implementation
This job is implemented by a call to the “touch” command with the relevant file name.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 48 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.14.2.2. Rerun Action
+ Prompts for rerun = action?
3.15 Schedule BRDB_SOD
This schedule is run daily at 08:00. It checks that the BRDB has completed start of day processing.
3.15.1 Dependencies
Schedule BRDB_COMPLETE depends on the existence of the flag files
/opt/tws/FLAGS/BRDB_COMPLETE . flag and /opt/tws/FLAGS/BRDB_BKUP_COMPLETE. flag.
3.15.2 Job DELETE_BRDB_COMPLETE_FLAG
This job deletes the flag file /opt/tws/FLAGS/BRDB_complete.FLAG.
3.15.2.1 Implementation
This job is implemented by a call to the “rm” command with the relevant file name.
3.15.2.2 Rerun Action
Alert Operations on failure?
3.15.3 Job DELETE_BRDB_COMPLETE_FLAG
This job deletes the flag file /opt/tws/FLAGS/BRDB_BKUP_complete.FLAG.
3.15.3.1 Implementation
This job is implemented by a call to the “rm” command with the relevant file name.
3.15.3.2 Rerun Action
Alert Operations on failure?
3.16 Schedule BRDB_START_FEED1
This schedule is run daily at 08:02. It prepares for the running of the two NPS copy processes by
reversing the changes that stopped them earlier in the schedule. It consists of two tasks which can be
run on any active node; see section 0 above for details. Only the two parent jobs are included here,
which are:
BRDBX011_START_NPS_TT_COPY
BRDBX011_START_NPS_GREV_COPY
3.16.1 Dependencies
Schedule BRDB_START_FEED1 depends on the completion of schedule BRDB_SOD.
3.16.2 Job BRDBX011_START_NPS_TT_COPY
This job prepares for the starting of the copying of Track and Trace transactions to NPS, by setting a
system parameter (see section 3.5).
3.16.2.1 Implementation
This job is implemented by a call to the shell script BRDBX011.sh specifying the relevant system
parameter name BRDB_TT_TXN_TO_NPS_STOP_YN and value “N”.
3.16.2.2 Rerun Action
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 49 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.16.3 Job BRDBX011_START_NPS_GREV_COPY
This job prepares for the starting of the copying of Reversals transactions to NPS, by setting a system
parameter (see section 3.5).
3.16.3.1 Implementation
This job is implemented by a call to the shell script BRDBX011.sh specifying the relevant system
parameter name BRDB_REV_TXN_TO_NPS_STOP_YN and value “N”.
3.16.3.2 Rerun Action
Alert Operations on failure:
3.17 Schedule BRDB_START_LFS
This schedule is run daily at 08:02. It prepares for the running of the two LFS copy processes by
reversing the changes that stop them from running. It consists of two tasks which can be run on any
active node; see section 0 above for details. Only the two parent jobs are included here, which are:
BRDBX011_START_LFS_PCOL_COPY
BRDBX011_START_LFS_PDEL_COPY
3.17.1. Dependencies
Schedule BRDB_START_LFS depends on the completion of schedule BRDB_SOD.
3.17.2 Job BRDBX011_START_LFS_PCOL_COPY
This job prepares for the starting of the copying of Pouch Collections to LFS, by setting a system
parameter (see section 3.5).
3.17.2.1 Implementation
This job is implemented by a call to the shell script BRDBX011.sh specifying the relevant system
parameter name BRDB_PCOL_TO_LFS_STOP_YN and value “N’”.
3.17.2.2. Rerun Action
Alert Operations on failure.
3.17.3 Job BRDBX011_START_LFS_PDEL_COPY
This job prepares for the starting of the copying of Pouch Deliveries to LFS, by setting a system
parameter (see section 3.5).
3.17.3.1 Implementation
This job is implemented by a call to the shell script BRDBX011.sh specifying the relevant system
parameter name BRDB_PDEL_TO_LFS_STOP_YN and value “N”.
3.17.3.2 Rerun Action
Alert Operations on failure,
3.18 Schedule BRDB_START_APOP
This schedule is run daily at 08:02. It prepares for the running of the APOP copy process by reversing
the changes that stop them from running. It consists of a single two task which can be run on any active
node. Only the parent job is included here, which is:
BRDBX011_START_APOP_TC_COPY
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 50 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.18.1 Dependencies
Schedule BRDB_START_APOP depends on the completion of schedule BRDB_SOD.
3.18.2 Job BRDBX011_START_APOP_TC_COPY
This job prepares for the starting of copying Transaction Confirmations to APOP, by setting a system
parameter (see section 3.5).
3.18.2.1 Implementation
This job is implemented by a call to the shell script BRDBX011.sh specifying the relevant system
parameter name BRDB_TXN_CONF_TO_APOP_STOP_YN and value “N’”.
3.18.2.2. Rerun Action
AlertOperations on failure!
3.19
3.20 Schedule BRDB_TT_TO_NPS1
This schedule is run daily at 08:05 to restart the Track and Trace NPS data feed. It consists of a single
task which is run on each active node by jobs named BRDBX003_TT_TO_NPS_1...4_NOPAGE.
3.20.1 Dependencies
Schedule BRDB_TT_TO_NPS1 depends on the completion of schedule BRDB_START_FEED1.
3.20.2 Job BRDBX003_TT_TO_NPS_1...4.NOPAGE
These jobs (one per node) start the feed that copies the Track and Trace transactions to NPS.
3.20.2.1 Implementation
These jobs are implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_TT_TXN_TO_NPS.
3.20.2.2 Database Link Information
NBX_TT_HARVESTER_AGENT_1@NPS1
3.20.2.3. Rerun Action
FSR See 3.5.1
3.21 Schedule BRDB_GREV_NPS1
This schedule is run daily at 08:05 to restart the Reversals NPS data feed. It consists of a single task
which is run on each active node by jobs named BRDBX003_GREV_TO_NPS_1...4_NOPAGE.
3.21.1 Dependencies
Schedule BRDB_GREV_NPS1 depends on the completion of schedule BRDB_START_FEED1.
3.21.2 Job BRDBX003_GREV_TO_NPS_1...4_NOPAGE
These jobs (one per node) start the feed that copies the Reversals transactions to NPS.
3.21.2.1 Implementation
These jobs are implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_REV_TXN_TO_NPS.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 51 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.21.2.2 Database Link Information
NBX_GREV_AGENT_1@NPS2
3.21.2.3. Rerun Action
FSR) See 3.5.1
3.22 Schedule BRDB_PCL_TO_LFS
This schedule is run daily at 08:05 to start the Pouch Collection to LFS data feed. It consists of a single
task which is run on each active node by jobs named BRDBX003_PCOL_TO_LFS_1...4_NOPAGE.
3.22.1 Dependencies
Schedule BRDB_PCL_TO_LFS depends on the completion of schedule BRDB_START_LFS.
3.22.2 Job BRDBX003_PCOL_TO_LFS_1...4_NOPAGE
These jobs (one per node) start the feed that copies the Pouch Collections to LFS.
3.22.2.1 Implementation
These jobs are implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_PCOL_TO_LFS.
3.22.2.2 Database Link Information
LFSBRDB@LFS
3.22.2.3. Rerun Action
RERUN! See 3.5.1
3.23 Schedule BRDB_PDL_TO_LFS
This schedule is run daily at 08:05 to start the Pouch Deliveries to LFS data feed. It consists of a single
task which is run on each active node by jobs named BRDBX003_PDEL_TO_LFS_1...4 NOPAGE.
3.23.1 Dependencies
Schedule BRDB_PDL_TO_LFS depends on the completion of schedule BRDB_START_LFS.
3.23.2 Job BRDBX003_PDEL_TO_LFS_1...4 NOPAGE
These jobs (one per node) start the feed that copies the Pouch Deliveries to LFS.
3.23.2.1 Implementation
These jobs are implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_PDEL_TO_LFS.
3.23.2.2 Database Link Information
LFSBRDB@LFS
3.23.2.3. Rerun Action
ERs See 3.5.1
3.24 Schedule BRDB_TC_TO_APOP
This schedule is run daily at 08:05 to start the Transaction Confirmation to APOP data feed. It consists of
a single task which is run on each active node by jobs named
BRDBX003_TC_TO_APOP_1...4_NOPAGE.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 52 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.24.1 Dependencies
Schedule BRDB_TC_TO_APOP depends on the completion of schedule BRDB_START_APOP.
3.24.2 Job BRDBX003_TC_TO_APOP_1...4._NOPAGE
These jobs (one per node) start the feed that copies the Transaction Confirmations to APOP.
3.24.2.1 Implementation
These jobs are implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_TXN_CONF_TO_APOP.
3.24.2.2 Database Link Information
APOPBRDB@APOP
3.24.2.3. Rerun Action
Rerunon failure] See 3.5.1
3.25
3.26 Schedule BRDB_SOB
This schedule is run daily at 19:00. It marks the start of the evening BRDB schedule.
3.26.1 Dependencies
None.
3.26.2 Job COMPLETE
This job simply echoes a message before exiting.
3.26.2.1 Implementation
This job is implemented by a call to the echo command.
3.26.2.2. Rerun Action
None.
3.27 Schedule BRDB_REF_DATA_SLA
This schedule is run daily. It runs the BRDB utility to generate Reference Data SLAs. It consists of a
single task which can be run on any active node; see section 0 above for details. Only the parent job
BRDBX032_BRDB_REF_DATA_SLA is included here.
3.27.1 Dependencies
Schedule BRDB_REF_DATA_SLA depends on the completion of schedule BRDB_SOB.
3.27.2 Job BRDBX032_BRDB_REF_DATA_SLA
This job runs the BRDB utility that generates Reference Data SLAs.
3.27.1.1 Implementation
This job is implemented by a call to the shell script BRDBX032.sh.
3.27.1.2 Rerun Action
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 53 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.28 Schedule BRDB_ONCH_AGG
This schedule is run daily. It aggregates the overnight cash on hand (ONCH) figures as well as setting
the last good ONCH date for relevant rows, in column
OPS$BRDB.BRDB_BRANCH_STOCK_UNITS.LAST_GOOD_ONCH_DATE. It performs two tasks,
firstly running the aggregation itself on all active nodes, with automatic waiting and rerunning; see
section 3.1 above for details. Only the main jobs BRDBX007_ONCH_AGG_1...4 are included here. The
second task checks for completion of the previous task, and can be run on any active node; see section
0 above for details. Only the parent job BRDBCO008_CHECK_ONCH_AGG is included here.
3.28.1 Dependencies
Schedule BRDB_ONCH_AGG depends on the completion of schedule BRDB_SOB.
Job BRDBC008_CHECK_ONCH_AGG depends on jobs BRDBX007_ONCH_AGG_1...4.
3.28.2 Job BRDBX007_ONCH_AGG_1...4
These jobs (one per node) perform the aggregation of the overnight cash on hand (ONCH) figures.
3.28.2.1 Implementation
These jobs are implemented by a call to the shell script BRDBX007.sh specifying the relevant
aggregation name OVERNIGHT_CASH_ON_HAND.
3.28.2.2 Rerun Action
As specified in section 03.1, alert Operations if rerun fails.
3.28.3 Job BRDBC008_CHECK_ONCH_AGG
This job checks for the successful completion of the previous job for all FAD-Hashes.
3.28.3.1 Implementation
This job is implemented by a call to the executable BRDBC008 specifying the relevant aggregation
name OVERNIGHT_CASH_ON_HAND.
3.28.3.2 Rerun Action
As specified in section 0, alert Operations if rerun fails.
3.29 Schedule BRDB_CSH_TO_LFS
This schedule is run daily. It runs the Cash Declarations to LFS feed. It performs two tasks, firstly
running the feed itself on all active nodes, with automatic waiting and rerunning; see section 3.1 above
for details. Only the main jobs BRDBX003_CASH_TO_LFS_1...4 are included here. The second task
checks for completion of the previous task, and can be run on any active node; see section 0 above for
details. Only the parent job BRDBC008_CHECK_CASH_TO_LFS is included here.
3.29.1 Dependencies
Schedule BRDB_CSH_TO_LFS depends on the completion of schedule BRDB_ONCH_AGG.
Job BRDBC008_CHECK_CASH_TO_LFS depends on jobs BRDBX003_CASH_TO_LFS_1...4.
3.29.2 Job BRDBX003_CASH_TO_LFS_1...4
These jobs (one per node) run the feed that copies the Cash Declarations to LFS.
3.29.2.1 Implementation
These jobs are implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_CASH TO LFS.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 54 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.29.2.2 Database Link Information
LFSBRDB@LFS
3.29.2.3 Rerun Action
As specified in section 0, alert Operations if rerun fails.
3.29.3 Job BRDBC008_CHECK_CASH_TO_LFS
This job checks for the successful completion of the previous job for all FAD-Hashes.
3.29.3.1 Implementation
This job is implemented by a call to the executable BRDBC008 specifying the relevant feed name
BRDB_CASH_TO_LFS.
3.29.3.2 Rerun Action
As specified in section 0, alert Operations if rerun fails.
3.30 Schedule BRDB_FROM_EMDB
This schedule is run daily at 19:30. It runs the Estate Management interface feed. It consists of a single
task which can be run on any active node; see section 0 above for details. Only the parent job
BRDBX003_BRDATA_FROM_EMDB is included here.
3.30.1 Dependencies
Schedule BRDB_FROM_EMDB depends on the completion of schedules BRDB_SOB and
EST_BRDB_UPD.
3.30.2 Job BRDBX003_BRDATA_FROM_EMDB
This job runs the Estate Management interface feed.
3.30.2.1 Implementation
This job is implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_EMDB_INTERFACE.
The SUSPEND_DISTRIBUTION flag is maintained by this BRDBX003 job.
This process references the following EMDB maintained tables:
Table Name Description
OPS$BRBD.EMDB_POST_OFFICE Maintained by EMDB, contains information relevant to each individual PO branch (e.g.
total number of counters/inodes, CTO_FLAG).
OPS$BRDB.RDDS_BRANCH_OPENING_PERIODS is used to update the address
information back into OPSBRDB.EMDB_POST_OFFICE
OPS$BRDB.EMDB_MANAGED_NODE Maintained by EMDB, contains information relevant to each individual counter per branch -
most relevantly the IP address associated with the counter.
The process updates the following tables which are referenced by the BAL
Table Name Description
OPS$BRBD.BRDB_BRANCH_INFO Uses EMDB_POST_OFFICE to set information such as the cto_flag,
suspend distribution flag)
OPS$BRDB.BRANCH_BRANCH_NODE_INFO Uses EMDB_MANAGED_NODE to set information such as the counter IP
address, suspend distribution flag)
OPS$BRDB.BRDB_FAD_HASH_OUTLET_MAPPING I New branches are inserted into this table, uses MOD(branch_code, 128) to
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 55 of 183
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE ®
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
resolve the FAD_HASH value.
OPS$BRDB.BRDB_TXN_CORR_TOOL_CTL New branches are inserted into this table in order to allow SSC correction
tools to maintain a running CURRENT_JSN value.
OPS$BRDB.BRDB_BRANCH_STOCK_UNITS, A default (DEF) stock unit is inserted for each new branch created
CP 585 — Branch Closures and Re-openings:
BRDB_EMDB_INTERFACE package has been fixed to mark a branch as ‘Closed’ in addition to clearing
out IP_SUBNET and IP_ADDRESS_1 details in BBI / BBNI
The feed marks a branch as ‘New’ if the branch is already ‘Cleared’ and gets re-activated by EMDB feed.
If the EMDB feed tries to re-activate a suspended branch that is marked as ‘Closed’ in Branch Database
but has not yet been ‘Cleared’ of transactional data then an alert will be raised through
BRDB_OPERATIONAL_EXCEPTIONS to notify such branches.
3.30.2.2 Rerun Action
Alert Operations on failure??
3.31 Schedule BRDB_CLR_BRANCH
This schedule runs after BRDB_FROM_EMDB completes and is stopped at 01:05. The called job
archives and then deletes transactions for all closed branches. This schedule is run on 1 instance at any
one time.
3.31.1 Dependencies
Schedule BRDB_CLR_BRANCH depends on the completion of schedule BRDB_FROM_EMDB. This
job is stopped at 01:05 irrespective of whether it has completed already (outstanding transactions will be
rolled back and picked up the following night).
3.31.2 Job BRDBX037_CLEAR_BRDATA
This job runs the BRDB automated closure process (BRDBX037.sh). Transactions are committed by
FAD_HASH (not individually by branch).
3.31.2.1 Implementation
This job is implemented by a call to the shell script BRDBX037.sh, along with the TWS business date
and instance number.
The process identifies all branches to be cleared by the following query
SELECT fad_hash, branch_accounting_code
FROM brdb_branch_info
WH = 'Closed'
RE branch_statu
AND suspend_distribution
All transactions for those closed branches in a number of tables (identified in column
BRDB_CLEARED_CONTROL_DATA.source_table) are loaded into archive tables (identified in column
BRDB_CLEARED_CONTROL_DATA.target_table) and then deleted from the original tables.
Note that these transactions are not replicated to BRSS, BRSS has an equivalent process
(BRSSX037.sh) that carries the closures independently of BRDB.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 56 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
Closed, cleared and archived branches are recorded in table BRDB_CLEARED_CLOSURE_DATA, with
column brdb_closure_date identifying when the branch was cleared on BRDB.
3.31.2.1.1LFS High Watermarks
As part of the branch clear down, associated LFS high watermarks are deleted from LFS. The following
is taken from the LFS support guide:
As part of removing a Temporarily Closed Branch on Horizon a process Remove Node from Cluster was
run which would trigger LFS to clear down Agent Marker tables and these High Water Marks.
Now that Riposte is no longer part of the solution, then Remove Node from Cluster is no longer run as
part of the Temporarily Closure process and so these High Water Marks are not cleared.
Therefore when we do a clear down of branch transactional data as part of closure procedure, the LFS
High Watermark for corresponding branch will be cleared down as well.
This is done by a package named “PKG_BRDB_CLR_BRANCH_DATA’ in Branch Database. As part of
branch closure process this package will perform the following deletes in LFS database through database
link -
DELETE FROM CTL_TMS RX_CASH_HDR@LFS WHERE GROUP_ID = <brn code>;
3.31.2.2. Exceptions
BRDBX037.sh checks each branch (to be cleared) has not traded within the last 5 days by querying
BRDB_BRANCH_NODE_INFO.last_logout_timestamp.
If a branch does show activity then an exception is logged in BRDB_LOPERATIONAL_EXCEPTIONS
with an exception code of BRDB35110. The branch will continue to log exceptions until the last logout
timestamp is older than TWS business date - 5 days.
3.31.2.3. Rerun Action
This job can be rerun if ISD's opinion is that there is enough of a window to process at least one FAD
HASH before 01.05am. If there is not enough time to complete then the following night's schedule will
pick up from where BRDBX037 stopped previously.
3.32 Schedule BRDB_PAUSE_LFS
This schedule is run daily at 20:00. It stops the two LFS feed processes, to allow the LFS batch jobs to
run overnight without activity occurring in the relevant tables. It consists of two tasks which can be run on
any active node; see section 0 above for details. Only the two parent jobs are included here, which are:
BRDBX011_PAUSE_LFS_PCOL_COPY
BRDBX011_PAUSE_LFS_PDEL_COPY
3.32.1 Dependencies
Schedule BRDB_PAUSE_LFS depends on the completion of schedule BRDB_SOB.
3.32.2 Job BRDBX011_PAUSE_LFS_PCOL_COPY
This job stops the copying of Pouch Collections to NPS, by setting a system parameter (see section 3.5).
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 57 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.32.2.1 Implementation
This job is implemented by a call to the shell script BRDBX011.sh specifying the relevant system
parameter name BRDB_PCOL_TO_LFS_STOP_YN and value “Y”.
3.32.2.2 Rerun Action
Alert Operations on failure,
3.32.3 Job BRDBX011_PAUSE_LFS_PDEL_COPY
This job stops the copying of Pouch Deliveries to NPS, by setting a system parameter (see section 3.5).
3.32.3.1 Implementation
This job is implemented by a call to the shell script BRDBX011.sh specifying the relevant system
parameter name BRDB_PDEL_TO_LFS_STOP_YN and value “Y”.
3.32.3.2 Rerun Action
Alert Operations on failure!
3.33 Schedule BRDB_PAUSE_APOP
This schedule is run daily at 20:00. It stops the APOP feed process, to allow the APOP batch jobs to run
overnight without activity occurring in the relevant tables. It consists of a single task which can be run on
any active node; see section 0 above for details. Only the parent job is included here, which is:
BRDBX011_PAUSE_APOP_TC_COPY
3.33.1. Dependencies
Schedule BRDB_PAUSE_APOP depends on the completion of schedule BRDB_SOB,
BRDB_START_APOP.
3.33.2 Job BRDBX011_PAUSE_APOP_TC_COPY
This job stops the copying of Transaction Confirmations to APOP, by setting a system parameter (see
section 3.5).
3.33.2.1 Implementation
This job is implemented by a call to the shell script BRDBX011.sh specifying the relevant system
parameter name BRDB_TXN_CONF_TO_APOP_STOP_YN and value “Y”.
3.33.2.2_ Rerun Action
Alert Operations on failure!
3.34 Schedule BRDB_EPOS_TO_TPS
This schedule is run daily. It runs the EPOSS transactions to TPS feed. It performs two tasks, firstly
running the feed itself on all active nodes, with automatic waiting and rerunning; see section 3.1 above
for details. Only the main jobs BRDBX003_EPOSS_TO_TPS_1...4 are included here. The second task
checks for completion of the previous task, and can be run on any active node; see section 0 above for
details. Only the parent job BRDBC008_CHECK_EPOSS_TO_TPS is included here.
3.34.1 Dependencies
Schedule BRDB_EPOS_TO_TPS depends on the completion of schedule BRDB_SOB.
Job BRDBC008_CHECK_EPOSS_TO_TPS depends on jobs BRDBX003_EPOSS_TO_TPS_1...4.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 58 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.34.2 Job BRDBX003_EPOSS_TO_TPS_1...4
These jobs (one per node) run the feed that copies the EPOSS transactions to TPS.
3.34.2.1 Implementation
These jobs are implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_EPOSS_TXN_TO_TPS.
3.34.2.2. Database Link Information
TPSBRDB@TPS
3.34.2.3 Rerun Action
As specified in section 03.1, alert Operations if rerun fails.
3.34.3 Job BRDBC008_CHECK_EPOSS_TO_TPS
This job checks for the successful completion of the previous job for all FAD-Hashes.
3.34.3.1 Implementation
This job is implemented by a call to the executable BRDBC008 specifying the relevant feed name
BRDB_EPOSS_TXN_TO_TPS.
3.34.3.2 Rerun Action
As specified in section 0, alert Operations if rerun fails.
3.35 Schedule BRDB_APS_TO_TPS
This schedule is run daily. It runs the APS transactions to TPS feed. It performs two tasks, firstly running
the feed itself on all active nodes, with automatic waiting and rerunning; see section 3.1 above for
details. Only the main jobs BRDBX003_APS_TO_TPS_1...4 are included here. The second task checks
for completion of the previous task, and can be run on any active node; see section 0 above for details.
Only the parent job BRDBC008_CHECK_APS_TO_TPS is included here.
3.35.1 Dependencies
Schedule BRDB_APS_TO_TPS depends on the completion of schedule BRDB_SOB.
Job BRDBC008_CHECK_APS_TO_TPS depends on jobs BRDBX003_APS_TO_TPS_1...4.
3.35.2 Job BRDBX003_APS_TO_TPS_1...4
These jobs (one per node) run the feed that copies the APS transactions to TPS.
3.35.2.1 Implementation
These jobs are implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_APS_TXN_TO_TPS.
3.35.2.2 Database Link Information
APSBRDB@APS
3.35.2.3 Rerun Action
As specified in section 3.1, alert Operations if rerun fails.
3.35.3 Job BRDBC008_CHECK_APS_TO_TPS
This job checks for the successful completion of the previous job for all FAD-Hashes.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 59 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.35.3.1 Implementation
This job is implemented by a call to the executable BRDBC008 specifying the relevant feed name
BRDB_APS_TXN_TO_TPS.
3.35.3.2 Rerun Action
As specified in section 0, alert Operations if rerun fails.
3.36 Schedule BRDB_NWB_TO_TPS
This schedule is run daily. It runs the NWB transactions to TPS feed. It performs two tasks, firstly
running the feed itself on all active nodes, with automatic waiting and rerunning; see section 3.1 above
for details. Only the main jobs BRDBX003_NWB_TO_TPS_1...4 are included here. The second task
checks for completion of the previous task, and can be run on any active node; see section 0 above for
details. Only the parent job BRDBC008_CHECK_NWB_TO_TPS is included here.
3.36.1 Dependencies
Schedule BRDB_NWB_TO_TPS depends on the completion of schedule BRDB_SOB.
Job BRDBC008_CHECK_NWB_TO_TPS depends on jobs BRDBX003_NWB_TO_TPS_1...4.
3.36.2 Job BRDBX003_NWB_TO_TPS_1...4
These jobs (one per node) run the feed that copies the NWB transactions to TPS.
3.36.2.1 Implementation
These jobs are implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_NWB_TXN_TO_TPS.
3.36.2.2 Database Link Information
TPSBRDB@TPS
3.36.2.3. Rerun Action
As specified in section 3.1, alert Operations if rerun fails.
3.36.3 Job BRDBC008_CHECK_NWB_TO_TPS
This job checks for the successful completion of the previous job for all FAD-Hashes.
3.36.3.1 Implementation
This job is implemented by a call to the executable BRDBC008 specifying the relevant feed name
BRDB_NWB_TXN_TO_TPS.
3.36.3.2 Rerun Action
As specified in section 0, alert Operations if rerun fails.
3.37 Schedule BRDB_DCS_TO_TPS
This schedule is run daily. It runs the DCS transactions to TPS feed. It performs two tasks, firstly running
the feed itself on all active nodes, with automatic waiting and rerunning; see section 3.1 above for
details. Only the main jobs BRDBX003_DCS_TO_TPS_1...4 are included here. The second task checks
for completion of the previous task, and can be run on any active node; see section 0 above for details.
Only the parent job BRDBC008_CHECK_DCS_TO_TPS is included here.
3.37.1. Dependencies
Schedule BRDB_DCS_TO_TPS depends on the completion of schedule BRDB_SOB.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 60 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
Job BRDBC008_CHECK_DCS_TO_TPS depends on jobs BRDBX003_DCS_TO_TPS_1...4.
3.37.2 Job BRDBX003_DCS_TO_TPS_1...4
These jobs (one per node) run the feed that copies the DCS transactions to TPS.
3.37.2.1 Implementation
These jobs are implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_DCS_TXN_TO_TPS.
3.37.2.2 Rerun Action
As specified in section 3.1, alert Operations if rerun fails.
3.37.3 Job BRDBC008_CHECK_DCS_TO_TPS
This job checks for the successful completion of the previous job for all FAD-Hashes.
3.37.3.1 Implementation
This job is implemented by a call to the executable BRDBC008 specifying the relevant feed name
BRDB_DCS_TXN_TO_TPS.
3.37.3.2 Database Link Information
TPSBRDB@TPS
3.37.3.3. Rerun Action
As specified in section 0, alert Operations if rerun fails.
3.38 Schedule BRDB_BDC_TO_TPS
This schedule is run daily. It runs the BDC transactions to TPS feed. It performs two tasks, firstly running
the feed itself on all active nodes, with automatic waiting and rerunning; see section 3.1 above for
details. Only the main jobs BRDBX003_BUREAU_TO_TPS_1...4 are included here. The second task
checks for completion of the previous task, and can be run on any active node; see section 0 above for
details. Only the parent job BRDBC008_CHECK_BUREAU_TO_TPS is included here.
3.38.1 Dependencies
Schedule BRDB_BDC_TO_TPS depends on the completion of schedule BRDB_SOB.
Job BRDBC008_CHECK_BUREAU_TO_TPS depends on jobs BRDBX003_BUREAU_TO_TPS_1...4.
3.38.2 Job BRDBX003_BUREAU_TO_TPS_1...4
These jobs (one per node) run the feed that copies the BDC transactions to TPS.
3.38.2.1 Implementation
These jobs are implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_BDC_TXN_TO_TPS.
3.38.2.2 Database Link Information
TPSBRDB@TPS
3.38.2.3 Rerun Action
As specified in section 3.1, alert Operations if rerun fails.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 61 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.38.3 Job BRDBC008_CHECK_BUREAU_TO_TPS
This job checks for the successful completion of the previous job for all FAD-Hashes.
3.38.3.1 Implementation
This job is implemented by a call to the executable BRDBC008 specifying the relevant feed name
BRDB_BDC_TXN_TO_TPS.
3.38.3.2 Rerun Action
As specified in section 0, alert Operations if rerun fails.
3.39 Schedule BRDB_EVT_TO_TPS
This schedule is run daily. It runs the EPOSS events to TPS feed. It performs two tasks, firstly running
the feed itself on all active nodes, with automatic waiting and rerunning; see section 3.1 above for
details. Only the main jobs BRDBX003_EVENTS_TO_TPS_1...4 are included here. The second task
checks for completion of the previous task, and can be run on any active node; see section 0 above for
details. Only the parent job BRDBC008_CHECK_EVENTS_TO_TPS is included here.
3.39.1 Dependencies
Schedule BRDB_EVT_TO_TPS depends on the completion of schedule BRDB_SOB.
Job BRDBC008_CHECK_EVENTS_TO_TPS depends on jobs BRDBX003_EVENTS_TO_TPS_1...4.
3.39.2 Job BRDBX003_EVENTS_TO_TPS_1...4
These jobs (one per node) run the feed that copies the EPOSS events to TPS.
3.39.2.1 Implementation
These jobs are implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_EPOSS_EVNT_TO_TPS.
3.39.2.2_ Database Link Information
TPSBRDB@TPS
3.39.2.3 Rerun Action
As specified in section 3.1, alert Operations if rerun fails.
3.39.3 Job BRDBC008_CHECK_EVENTS_TO_TPS
This job checks for the successful completion of the previous job for all FAD-Hashes.
3.39.1.1 Implementation
This job is implemented by a call to the executable BRDBC008 specifying the relevant feed name
BRDB_EPOSS_EVNT_TO_TPS.
3.39.1.2 Database Link Information
TPSBRDB@TPS
3.39.1.3 Rerun Action
As specified in section 0, alert Operations if rerun fails.
3.40 Schedule BRDB_COFS_TO_TPS
This schedule is run daily. It runs the Cut Off Summaries to TPS feed. It performs two tasks, firstly
running the feed itself on all active nodes, with automatic waiting and rerunning; see section 3.1 above
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 62 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
for details. Only the main jobs BRDBX003_COFF_SUMM_TO_TPS_1...4 are included here. The second
task checks for completion of the previous task, and can be run on any active node; see section 0 above
for details. Only the parent job BRDBC008_CHECK_COFF_SUMM_TO_TPS is included here.
3.40.1 Dependencies
Schedule BRDB_COFS_TO_TPS depends on the completion of schedule BRDB_SOB
Job BRDBC008_CHECK_COFF_SUMM_TO_TPS depends on jobs
BRDBX003_COFF_SUMM_TO_TPS_1...4.
3.40.2 Job BRDBX003_COFF_SUMM_TO_TPS_1...4
These jobs (one per node) run the feed that copies the Cut Off Summaries to TPS.
3.40.2.1 Implementation
These jobs are implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_CUTOFF_SUMM_TO_TPS.
3.40.2.2 Database Link Information
TPSBRDB@TPS
3.40.2.3. Rerun Action
As specified in section 3.1, alert Operations if rerun fails.
3.40.3 Job BRDBC008_CHECK_COFF_SUMM_TO_TPS
This job checks for the successful completion of the previous job for all FAD-Hashes.
3.40.3.1 Implementation
This job is implemented by a call to the executable BRDBC008 specifying the relevant feed name
BRDB_CUTOFF_SUMM_TO_TPS.
3.40.3.2 Rerun Action
As specified in section 0, alert Operations if rerun fails.
3.41 Schedule BRDB_TA_FROM_TPS
This schedule is run daily. It runs the Transaction Acknowledgement from TPS interface feed. It consists
of a single task which can be run on any active node; see section 0 above for details. Only the parent job
BRDBX003_TA_FROM_TPS is included here.
3.41.1 Dependencies
Schedule BRDB_TA_FROM_TPS depends on the completion of schedules BRDB_SOB and TPS_TA.
3.41.2 Job BRDBX003_TA_FROM_TPS
This job runs the Transaction Acknowledgement from TPS interface feed.
3.41.2.1 Database Link Information
TPSBRDB@TPS
3.41.2.2 Implementation
This job is implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_TXN_ACKS_FROM_TPS.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 63 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.41.2.3. Rerun Action
Alert Operations on failure!
3.42 Schedule BRDB_TC_FROM_TPS
This schedule is run daily. It runs the Transaction Corrections from TPS interface feed. It consists of a
single task which can be run on any active node; see section 0 above for details. Only the parent job
BRDBX003_TC_FROML_TPS is included here.
3.42.1 Dependencies
Schedule BRDB_TC_FROM_TPS depends on the completion of schedules BRDB_SOB and TPS_TC.
3.42.2 Job BRDBX003_TC_FROM_TPS
This job runs the Transaction Corrections from TPS interface feed.
3.42.2.1 Implementation
This job is implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_TXN_CORR_FROM_TPS.
3.42.2.2. Rerun Action
Alert Operations on failure,
3.43 Schedule BRDB_TPS_COMPL
This schedule is run daily. It marks the end of the TPS schedule.
3.43.1 Dependencies
Schedule BRDB_TPS_COMPL depends on the completion of schedules BRDB_EPOS_TO_TPS,
BRDB_APS_TO_TPS, BRDB_NWB_TO_TPS, BRDB_DCS_TO_TPS, BRDB_BDC_TO_TPS,
BRDB_EVT_TO_TPS and BRDB_COFS_TO_TPS.
3.43.2 Job COMPLETE
This job simply echoes a message before exiting.
3.43.2.1 Implementation
This job is implemented by a call to the echo command.
3.43.2.2 Rerun Action
None.
3.44 Schedule BRDB_TPS_TOTALS
This schedule is run daily. It aggregates the outlet transaction totals. It performs two tasks, firstly running
the aggregation itself on all active nodes, with automatic waiting and rerunning; see section 3.1 above
for details. Only the main jobs BRDBX007_TPS_TXN_TOTALS_1...4 are included here. The second
task checks for completion of the previous task, and can be run on any active node; see section 0 above
for details. Only the parent job BRDBCO08_CHECK_TPS_TXN_TOTALS is included here.
3.44.1 Dependencies
Schedule BRDB_TPS_TOTALS depends on the completion of schedule BRDB_SOB.
Job BRDBC008_CHECK_TPS_TXN_TOTALS depends on jobs BRDBX007_TPS_TXN_TOTALS_1...4.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 64 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.44.2 Job BRDBX007_TPS_TXN_TOTALS.1...4
These jobs (one per node) perform the aggregation of the outlet transaction totals.
3.44.2.1 Implementation
These jobs are implemented by a call to the shell script BRDBX007.sh specifying the relevant
aggregation name BRDB_TPS_TXN_TOTALS.
3.44.2.2. Database Link Information
TPSBRDB@TPS
3.44.2.3 Rerun Action
As specified in section 3.1, alert Operations if rerun fails.
3.44.3 Job BRDBC008_CHECK_TPS_TXN_TOTALS
This job checks for the successful completion of the previous job for all FAD-Hashes.
3.44.3.1 Implementation
This job is implemented by a call to the executable BRDBC008 specifying the relevant aggregation
name BRDB_TPS_TXN_TOTALS.
3.44.3.2 Rerun Action
As specified in section 0, alert Operations if rerun fails.
3.45 Schedule BRDB_TOTL_TO_TPS
This schedule is run daily. It runs the Transactions Totals to TPS feed. It performs two tasks, firstly
running the feed itself on all active nodes, with automatic waiting and rerunning; see section 3.1 above
for details. Only the main jobs BRDBX003_TXN_TOTALS_TO_TPS_1...4 are included here. The
second task checks for completion of the previous task, and can be run on any active node; see section
0 above for details. Only the parent job BRDBCO008_CHECK_TXN_TOTALS_TO_TPS is included here.
3.45.1 Dependencies
Schedule BRDB_TOTL_TO_TPS depends on the completion of schedule BRDB_TPS_TOTALS.
Job BRDBC008_CHECK_TXN_TOTALS_TO_TPS depends on jobs
BRDBX003_TXN_TOTALS_TO_TPS_1...4.
3.45.2 Job BRDBX003_TXN_TOTALS_TO_TPS_1...4
These jobs (one per node) run the feed that copies the Transactions Totals to TPS.
3.45.2.1 Database Link Information
TPSBRDB@TPS
3.45.2.2. Implementation
These jobs are implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_TXN_TOT_TO_TPS.
3.45.2.3. Rerun Action
As specified in section 3.1, alert Operations if rerun fails.
3.45.3 Job BRDBC008_CHECK_TXN_TOTALS_TO_TPS
This job checks for the successful completion of the previous job for all FAD-Hashes.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 65 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.45.3.1 Implementation
This job is implemented by a call to the executable BRDBC008 specifying the relevant feed name
BRDB_TXN_TOT_TO_TPS.
3.45.3.2 Rerun Action
As specified in section 0, alert Operations if rerun fails.
3.46 Schedule BRDB_APS_TOTALS
This schedule is run daily. It aggregates the APS transaction totals. It performs two tasks, firstly running
the aggregation itself on all active nodes, with automatic waiting and rerunning; see section 3.1 above
for details. Only the main jobs BRDBX007_APS_TXN_TOTALS_1...4 are included here. The second
task checks for completion of the previous task, and can be run on any active node; see section 0 above
for details. Only the parent job BRDBC008_CHECK_APS_TXN_TOTALS is included here.
3.46.1 Dependencies
Schedule BRDB_APS_TOTALS depends on the completion of schedule BRDB_SOB.
Job BRDBC008_CHECK_APS_TXN_TOTALS depends on jobs BRDBX007_APS_TXN_TOTALS_1...4.
3.46.2 Job BRDBX007_APS_TXN_TOTALS_1...4
These jobs (one per node) perform the aggregation of the APS transaction totals.
3.46.2.1 Implementation
These jobs are implemented by a call to the shell script BRDBX007.sh specifying the relevant
aggregation name BRDB_APS_TXN_TOTALS.
3.46.2.2 Database Link Information
APSBRDB@APS,
3.46.2.3 Rerun Action
As specified in section 3.1, alert Operations if rerun fails.
3.46.3 Job BRDBC008_CHECK_APS_TXN_TOTALS
This job checks for the successful completion of the previous job for all FAD-Hashes.
3.46.3.1 Implementation
This job is implemented by a call to the executable BRDBCO008 specifying the relevant aggregation
name BRDB_APS_TXN_TOTALS.
3.46.3.2 Rerun Action
As specified in section 0, alert Operations if rerun fails.
3.47 Schedule BRDB_TOTL_TO_APS
This schedule is run daily. It runs the Transactions Totals to APS feed. It performs two tasks, firstly
running the feed itself on all active nodes, with automatic waiting and rerunning; see section 3.1 above
for details. Only the main jobs BRDBX003_TXN_TOTALS_TO_APS_1...4 are included here. The
second task checks for completion of the previous task, and can be run on any active node; see section
0 above for details. Only the parent job BRDBC008_CHECK_TXN_TOTALS_TO_APS is included here.
3.47.1 Dependencies
Schedule BRDB_TOTL_TO_APS depends on the completion of schedule BRDB_APS_TOTALS.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 66 of 183
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE ®
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
Job BRDBC008_CHECK_TXN_TOTALS_TO_APS depends on jobs
BRDBX003_TXN_TOTALS_TO_APS_1...4.
3.47.2 Job BRDBX003_TXN_TOTALS_TO_APS_1...4
These jobs (one per node) run the feed that copies the Transactions Totals to APS.
3.47.2.1 Implementation
These jobs are implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_TXN_TOT_TO_APS.
3.47.2.2 Rerun Action
As specified in section 3.1, alert Operations if rerun fails.
3.47.3 Job BRDBC008_CHECK_TXN_TOTALS_TO_APS
This job checks for the successful completion of the previous job for all FAD-Hashes.
3.47.3.1 Implementation
This job is implemented by a call to the executable BRDBC008 specifying the relevant feed name
BRDB_TXN_TOT_TO_APS.
3.47.3.2 Rerun Action
As specified in section 0, alert Operations if rerun fails.
3.48 Schedule BRDB_TXNS_TO_APS
This schedule is run daily. It runs the APS transactions to APS feed. It performs two tasks, firstly running
the feed itself on all active nodes, with automatic waiting and rerunning; see section 3.1 above for
details. Only the main jobs BRDBX003_TXNS_TO_APS_1...4 are included here. The second task
checks for completion of the previous task, and can be run on any active node; see section 0 above for
details. Only the parent job BRDBC008_CHECK_TXNS_TO_APS is included here.
3.48.1 Dependencies
Schedule BRDB_TXNS_TO_APS depends on the completion of schedules BRDB_SOB and
APS_BULK_HARV.
Job BRDBC008_CHECK_TXNS_TO_APS depends on jobs BRDBX003_TXNS_TO_APS_1...4.
3.48.2 Job BRDBX003_TXNS_TO_APS_1...4
These jobs (one per node) run the feed that copies the APS transactions to TPS.
3.48.2.1 Implementation
These jobs are implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_APS_TXN_TO_APS.
3.48.2.2 Database Link Information
APSBRDB@APS
3.48.2.3 Rerun Action
As specified in section 3.1, alert Operations if rerun fails.
3.48.3 Job BRDBC008_CHECK_TXNS_TO_APS
This job checks for the successful completion of the previous job for all FAD-Hashes.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 67 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.48.3.1 Implementation
This job is implemented by a call to the executable BRDBC008 specifying the relevant feed name
BRDB_APS_TXN_TO_APS.
3.48.3.2 Rerun Action
As specified in section 0, alert Operations if rerun fails.
3.49 Schedule BRDB_APS_COMPL
This schedule is run daily. It marks the end of the APS schedule.
3.49.1 Dependencies
Schedule BRDB_APS_COMPL depends on the completion of schedules BRDB_TXNS_TO_APS and
BRDB_TOTL_TO_APS.
3.49.2 Job COMPLETE
This job simply echoes a message before exiting.
3.49.2.1 Implementation
This job is implemented by a call to the echo command.
3.49.2.2 Rerun Action
None.
3.50 Schedule BRDB_NWB_TO_DRS
This schedule is run daily. It runs the NWB transactions to DRS feed. It performs two tasks, firstly
running the feed itself on all active nodes, with automatic waiting and rerunning; see section 3.1 above
for details. Only the main jobs BRDBX003_NWB_TO_DRS_1...4 are included here. The second task
checks for completion of the previous task, and can be run on any active node; see section 0 above for
details. Only the parent job BRDBC008_CHECK_NWB_TO_DRS is included here.
3.50.1 Dependencies
Schedule BRDB_NWB_TO_DRS depends on the completion of schedule BRDB_SOB.
Job BRDBC008_CHECK_NWB_TO_DRS depends on jobs BRDBX003_NWB_TO_DRS_1...4.
3.50.2 Job BRDBX003_NWB_TO_DRS_1...4
These jobs (one per node) run the feed that copies the NWB transactions to DRS.
3.50.2.1 Implementation
These jobs are implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_NWB_TXN_TO_DRS.
3.50.2.2 Database Link Information
DRSBRDB@DRS
3.50.2.3. Rerun Action
As specified in section 3.1, alert Operations if rerun fails.
3.50.3 Job BRDBC008_CHECK_NWB_TO_DRS
This job checks for the successful completion of the previous job for all FAD-Hashes.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 68 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.50.3.1 Implementation
This job is implemented by a call to the executable BRDBC008 specifying the relevant feed name
BRDB_NWB_TXN_TO_DRS.
3.50.3.2 Rerun Action
As specified in section 0, alert Operations if rerun fails.
3.51 Schedule BRDB_DCS_TO_DRS
This schedule is run daily. It runs the DCS transactions to DRS feed. It performs two tasks, firstly running
the feed itself on all active nodes, with automatic waiting and rerunning; see section 3.1 above for
details. Only the main jobs BRDBX003_DCS_TO_DRS_1...4 are included here. The second task checks
for completion of the previous task, and can be run on any active node; see section 0 above for details.
Only the parent job BRDBC008_CHECK_DCS_TO_DRS is included here.
3.51.1 Dependencies
Schedule BRDB_DCS_TO_DRS depends on the completion of schedule BRDB_SOB.
Job BRDBC008_CHECK_DCS_TO_DRS depends on jobs BRDBX003_DCS_TO_DRS_1...4.
3.51.2 Job BRDBX003_DCS_TO_DRS_1...4
These jobs (one per node) run the feed that copies the DCS transactions to DRS.
3.51.2.1 Implementation
These jobs are implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_DCS_TXN_TO_DRS.
3.51.2.2 Database Link Information
DRSBRDB@DRS
3.51.2.3 Rerun Action
As specified in section 3.1, alert Operations if rerun fails.
3.51.3 Job BRDBC008_CHECK_DCS_TO_DRS
This job checks for the successful completion of the previous job for all FAD-Hashes.
3.51.3.1 Implementation
This job is implemented by a call to the executable BRDBC008 specifying the relevant feed name
BRDB_DCS_TXN_TO_DRS.
3.51.3.2 Rerun Action
As specified in section 0, alert Operations if rerun fails.
3.52 Schedule BRDB_DRS_COMPL
This schedule is run daily. It marks the end of the DRS schedule.
3.52.1 Dependencies
Schedule BRDB_DRS_COMPL depends on the completion of schedules BRDB_NWB_TO_DRS and
BRDB_DCS_TO_DRS.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 69 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.52.2 Job COMPLETE
This job simply echoes a message before exiting.
3.52.2.1 Implementation
This job is implemented by a call to the echo command.
3.52.2.2 Rerun Action
None.
3.53 Schedule BRDB_XFR_COMPL
This schedule is run daily. It marks the end of the transfer schedule.
3.53.1. Dependencies
Schedule BRDB_XFR_COMPL depends on the completion of schedules BRDB_TOTL_TO_TPS,
BRDB_TXNS_TO_APS and BRDB_DRS_COMPL.
3.53.2 Job COMPLETE
This job simply echoes a message before exiting.
3.53.2.1 Implementation
This job is implemented by a call to the echo command.
3.53.2.2. Rerun Action
None.
3.54 Schedule BRDB_FEED_ERRORS
This schedule is run daily. It runs the process to raise operation exceptions for data feed errors. It
consists of a single task which can be run on any active node; see section 0 above for details. Only the
parent job BRDBX007_RAISE_FEED_DATA_EXCEPTIONS is included here.
3.54.1 Dependencies
Schedule BRDB_FEED_ERRORS depends on the completion of schedule BRDB_XFR_COMPL.
3.54.2 Job BRDBX007_RAISE_FEED_DATA_EXCEPTIONS
This job runs the process to raise operation exceptions for data feed errors.
3.54.1.1 Implementation
This job is implemented by a call to the shell script BRDBX007.sh specifying the relevant process name
RAISE_FEED_DATA_EXCEPTIONS.
3.54.1.2 Rerun Action
Alert Operations on failure!
3.55 Schedule BRDB_NCU_TXN_AGG
This schedule is run daily at 1:15. It performs data aggregation for the daily summary. It performs two
tasks, firstly running the aggregation itself on all active nodes, with automatic waiting and rerunning; see
section 3.1 above for details. Only the main jobs BRDBX007_NON_CUMU_TXN_TOTALS_1...4 are
included here. The second task checks for completion of the previous task, and can be run on any active
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 70 of 183
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE ®
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
node; see section ie} above for details. Only the parent job
BRDBC008_CHECK_NON_CUMU_TXN_AGGR is included here.
3.55.1 Dependencies
Job BRDBC008_CHECK_NON_CUMU_TXN_AGGR depends on jobs
BRDBX007_NON_CUMU_TXN_TOTALS_1...4.
3.55.2 Job BRDBX007_NON_CUMU_TXN_TOTALS_1...4
These jobs (one per node) perform data aggregation for the daily summary.
3.55.2.1 Implementation
These jobs are implemented by a call to the shell script BRDBX007.sh specifying the relevant
aggregation name BRDB_NON_CUMU_TXN_AGGR.
3.55.2.2 Rerun Action
As specified in section 3.1, alert Operations if rerun fails.
3.55.3 Job BRDBC008_CHECK_NON_CUMU_TXN_AGGR
This job checks for the successful completion of the previous job for all FAD-Hashes.
3.55.3.1 Implementation
This job is implemented by a call to the executable BRDBC008 specifying the relevant aggregation
name BRDB_NON_CUMU_TXN_AGGR.
3.55.3.2 Rerun Action
As specified in section 0, alert Operations if rerun fails.
3.56 Schedule BRDB_CU_TXN_AGG
This schedule is run daily. It performs data aggregation for the daily cumulative summary. It performs
two tasks, firstly running the aggregation itself on all active nodes, with automatic waiting and rerunning;
see section 3.1 above for details. Only the main jobs BRDBX007_CUMU_TXN_AGGR_1...4 are
included here. The second task checks for completion of the previous task, and can be run on any active
node; see section 0 above for details. Only the parent job BRDBCO008_CHECK_CUMU_TXN_AGGR is
included here.
3.56.1 Dependencies
Schedule BRDB_CU_TXN_AGG depends on the completion of schedule BRDB_NCU_TXN_AGG.
Job BRDBC008_CHECK_CUMU_TXN_AGGR depends on jobs BRDBX007_CUMU_TXN_AGGR_1...4.
3.56.2 Job BRDBX007_CUMU_TXN_AGGR_1...4
These jobs (one per node) perform data aggregation for the cumulative daily summary.
3.56.2.1 Implementation
These jobs are implemented by a call to the shell script BRDBX007.sh specifying the relevant
aggregation name BRDB_CUMU_TXN_AGGR.
3.56.2.2 Rerun Action
As specified in section 3.1, alert Operations if rerun fails.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 71 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.56.3 Job BRDBC008_CHECK_CUMU_TXN_AGGR
This job checks for the successful completion of the previous job for all FAD-Hashes.
3.56.3.1 Implementation
This job is implemented by a call to the executable BRDBC008 specifying the relevant aggregation
name BRDB_CUMU_TXN_AGGR.
3.56.3.2 Rerun Action
As specified in section 0, alert Operations if rerun fails.
3.57 Schedule BRDB_BBNI_MAINT
This schedule is run daily. It runs the BRDB utility to reset sequence numbers. It consists of a single task
which can be run on any active node; see section 0 above for details. Only the parent job
BRDBX031_JSN_USN_SSN is included here.
3.57.1 Dependencies
Schedule BRDB_BBNI_MAINT depends on the completion of schedule BRDB_CU_TXN_AGG.
3.57.2 Job BRDBX031_JSN_USN_SSN
This job runs the BRDB utility that resets the sequence numbers.
3.57.2.1 Implementation
This job is implemented by a call to the shell script BRDBX031.sh.
3.57.2.2 Rerun Action
#4 Prompts for rerun = action?
3.58 Schedule BRDB_SUMMARY_DTE
This schedule is run daily. It sets the last daily summary date. It consists of a single task which can be
run on any active node; see section O above for details. Only the parent job
BRDBX011_SET_DAILY_SUMMARY_DATE is included here.
3.58.1 Dependencies
Schedule BRDB_LSUMMARY_DTE depends on the completion of schedule BRDB_BBNI_MAINT.
3.58.2 Job BRDBX011_SET_DAILY_SUMMARY_DATE
This job sets the last daily summary date, a system parameter.
3.58.1.1 Implementation
This job is implemented by a call to the shell script BRDBX011.sh specifying the relevant system
parameter name BRDB_LAST_DAILY_SUMMARY_DATE and relevant date value.
3.58.1.2 Rerun Action
Alert Operations on failure:
3.59 Schedule BRDB_GEN_REP
This schedule is run daily. It generates the reconciliation reports. It consists of two tasks which can be
run on any active node; see section 0 above for details. Only the two parent jobs are included here,
which are:
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 72 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
GENERIC_CREATE_REPORT_VIEWS
GENERIC_CREATE_RECON_REPORTS
3.59.1 Dependencies
Schedule BRDB_GEN_REP depends on the completion of schedule BRDB_REF_DATA_SLA.
Job GENERIC_CREATE_RECON_REPORTS depends on job GENERIC_CREATE_REPORT_VIEWS.
3.59.2 Job GENERIC_CREATE_REPORT_VIEWS
This job creates the generic views for reconciliation reporting.
3.59.2.1 Implementation
This job is implemented by a call to the shell script GREPX001.sh.
3.59.2.2 Rerun Action
** Prompts for refun = action? *
3.59.3 Job GENERIC_CREATE_RECON_REPORTS
This job creates the generic reconciliation reports.
3.59.3.1 Implementation
This job is implemented by a call to the shell script GREPX002.sh.
Outputs files to the following directories below.
Usage BRDBBLV1 Environment Variable
Working directory BRDB_MSU_WORKING
BRDB reports directory BRDB_MSU_OUTPUT
Files in the working directory are immediately cleaned up on successful completion while files within the
reports directory are removed after 9 days.
3.59.3.2 Rerun Action
Prompts for rerun = action?
3.60 Schedule BRDB_TO_DWH
This schedule is run daily. It performs the file transfer for the BRDB Branch Migration Status data feed. It
consists of a single task which can be run on any active node; see section 0 above for details. Only the
parent job BRDBX020_BRDB_XFER_TO_DWUH is included here.
3.60.1 Dependencies
Schedule BRDB_TO_DWH depends on the completion of schedule BRDB_GEN_REP.
3.60.2 Job BRDBX020_BRDB_XFER_TO_DWH
This job performs the file transfers for the BRDB Branch Migration Status and Reference data feeds.
3.60.2.1 Implementation
This job is implemented by a call to the shell script BRDBX020.sh.
Outputs files to the following directories below.
Usage BRDBBLV1 Environment Variable
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN Ref: DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 73 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
[ BRDB reports directory l REPOSITORY I
3.60.2.2 Rerun Action
Alert Operations on failure. This job may be re-runable, depending on the error (see failures below for
deciding if re-runable or not).
3.60.2.2.1 Failures
“Source file <n> <filename> does not exist”
Ensure Job GENERIC_CREATE_RECON_REPORTS completed successfully and if all expected
reports are present in ${BRDB_MSU_OUTPUT}
Expected reports are:
* DW Branch _Migration_Extract.csv
* DW Reference Data_SLA.csv
Once the cause of the ‘missing’ reports is resolved, ensure the following files are removed (if present)
e ${REPOSITORY}/brdb/YYMMDD/YYMMDD00.bac
¢ ${REPOSITORY} /brdb/YYMMDD/YYMMDD00.bms
BRDBX020_BRDB_XFER_TO_DWH may then be rerun.
“Destination file <n> <filename> already exists”
The above error suggests that the script has already been run successfully. Alert Operations as this will
require more investigation into why the script has failed.
3.61 Schedule BRDB_AGG_COMPL
This schedule is run daily. It marks the end of the aggregation schedule.
3.61.1 Dependencies
Schedule BRDB_LAGG_COMPL depends on the completion of schedules BRDB_SUMMARY_DTE and
BRDB_TO_DWH.
3.61.2 Job COMPLETE
This job simply echoes a message before exiting.
3.61.2.1 Implementation
This job is implemented by a call to the echo command.
3.61.2.2 Rerun Action
None.
3.62 Schedule BRDB_FROM_RDDS
This schedule is run daily at 00:10. It runs the Host Reference Data from RDDS data feed. It consists of
a single task which can be run on any active node; see section 0 above for details. Only the parent job
BRDBX003_REFDATA_FROM_RDDS is included here.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 74 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.62.1 Dependencies
Schedule BRDB_FROM_RDDS depends on the completion of schedules BRDB_SOB and
RDDS_COPY_SCHED.
3.62.2 Job BRDBX003_REFDATA_FROM_RDDS
This job runs the Host Reference Data from RDDS data feed.
3.62.2.1 Implementation
This job is implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDeHOST_REF_FROM_RDDS. The job populates the following BRDB tables via the LFS database
RDDS_PACKAGE_TYPE
RDDS_PACKAGE
RDDS_PACKAGE_CONTENT
RDDS_DELIVERY_TYPE
RDDS_DELIVERY
. RDDS_BRANCH_BUREAU_REGION
See DEV/APP/LLD/0050 for detailed information.
3.62.2.2 Database Link Information
RDDSBRDB@RDDS
3.62.2.3 Rerun Action
SF Prompts for rerun = action?
3.63 Schedule BRDB_FROM_TPS
This schedule is run daily at 00:10. It runs the Outlets/Transaction Modes data from TPS data feed. It
consists of a single task which can be run on any active node; see section 0 above for details. Only the
parent job BRDBX003_REFDATA_FROML_TPS is included here.
3.63.1 Dependencies
Schedule BRDB_FROM_TPS depends on the completion of schedules BRDB_SOB and
TPSEOD.TPSC207.
3.63.2 Job BRDBX003_REFDATA_FROM_TPS
This job runs the Outlets/Transaction Modes data from TPS data feed.
3.63.1.1 Implementation
This job is implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_REF_COPY_FROM_TPS.
3.63.1.2 Database Link Information
oar WN =
TPSBRDB@TPS
3.63.1.3. Rerun Action
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN Ref: DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 75 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.64 Schedule BRDB_AUD_FEED
This schedule is run daily at 01:05. It performs journal auditing. It performs three tasks, firstly running the
message journal auditing on all active nodes, with automatic waiting and rerunning; see section 3.1
above for details. Only the main jobs BRDBC002_AUDIT_1...4 are included here. The second task
checks for completion of the previous task, and can be run on any active node; see section 0 above for
details. Only the parent job BRDBC008_CHECK_AUDIT_FEED is included here. The third task performs
Transaction Correction journal auditing, and can be run on any active node; again see section 0 above
for details. Only the parent job BRDBC033_AUDIT is included here.
Additional monitoring is required so that an alert is raised if this job has not completed by 04:00. This is
implemented within the BRDB_MONITOR schedule — see section 3.76.
3.64.1 Dependencies
Schedule BRDB_AUD_FEED depends on the completion of schedule BRDB_SOB.
Job BRDBC008_CHECK_AUDIT_FEED depends on jobs BRDBC002_AUDIT_1...4.
Job BRDBC033_AUDIT depends on job BRDBC008_CHECK_AUDIT_FEED.
3.64.2 Job BRDBC002_AUDIT_1...4
These jobs (one per node) generate text files for the input day's auditable messages.
3.64.2.1 Implementation
These jobs are implemented by a call to the executable BRDBCO002.
Outputs files to the following directories below.
Usage BRDBBLV1 Environment Variable
Working directory BRDB_AUDIT_FILE_TEMP
BRDB reports directory BRDB_COUNTER_AUDIT_OUTPUT
3.64.2.2 Rerun Action
As specified in section 3.1, alert Operations if rerun fails.
3.64.3 Job BRDBC008_CHECK_AUDIT_FEED
This job checks for the successful completion of the previous job for all FAD-Hashes.
3.64.3.1 Implementation
This job is implemented by a call to the executable BRDBC008 specifying the relevant process name
BRDBC002.
3.64.3.2 Rerun Action
As specified in section 0, alert Operations if rerun fails.
3.64.4 Job BRDBC033_AUDIT
This job generates text files for the input day's auditable transaction correction messages.
3.64.4.1 Implementation
This job is implemented by a call to the executable BRDBC033
Outputs files to the following directories below.
Usage BRDBBLV1 Environment Variable
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN _ Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 76 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
Working directory BRDB_TCT_FILE_TEMP
BRDB reports directory BRDB_TCT_AUDIT_OUTPUT
3.64.4.2 Rerun Action
As specified in section 3.1.1, alert Operations if rerun fails.
3.65 Schedule BRDB_ORA_STATS
This schedule, which runs daily, gathers statistics on date range partitioned tables every Monday
(excluding English bank holidays) and daily for all other tables. It consists of a single task which can be
run on any active node; see section 3.1.1 above for details. Only the parent job BRDBX005_SCHEMA is
included here.
3.65.1 Dependencies
Schedule BRDB_ORA_STATS depends on the completion of schedules BRDB_AUD_FEED,
BRDB_AGG_COMPL and BRDB_XFR_COMPL.
3.65.2 Job BRDBX005_SCHEMA
This job gathers the Oracle optimiser statistics.
3.65.2.1 Implementation
This job is implemented by a call to the shell script BRDBX005.sh. The input parameters [-i & -s] are
present for backward compatibility only.
Statistics for tables as per those in table BRDB_ANALYZED_OBJECTS are normally gathered on a
Monday (controlled by system parameter BRDBX005_GATHER_WEEK_DAY), those statistics are then
copied into future partitions every night (until the following Monday).
Statistics for tables not present in BRDB_ANALYZED_OBJECTS are gathered every night.
3.65.2.1.1 Associated BRDB System Parameters
Parameter Name Parameter Value Description
DEBUG_LEVEL_FOR_BRDBX005 3 from parameter_number] I Controls detail of stdist output
BRDBX005_ADJUST_HIGH_LOW FLAG _ I Y [from parameter_text] Controls method of copy table stats
BRDBX005_GATHER_WEEK_DAY MON [from parameter text] I Day to gather stats on partitioned tables,
BRDBX005_EXPORT_STATS N {from parameter_text] Controls whether stats are copied to BRDB_OBJECT_STATS_ARC
3.65.2.2 Rerun Action
The statistics gathering job is able to resume from where it last failed so it is feasible to rerun the job (if
the failure was, for example, due to a full tablespace then that would need resolving first).
3.66 Schedule BRDB_ADMIN
This schedule is run daily. It performs administration of the BRDB database. It includes two tasks which
can be run on any active node; see section 0 above for details. Only the parent jobs BRDBC004 and
BRDBXO006 are included here. It also includes two tasks which are run on each active node by jobs
named BRDB_HKP_ORAFILES1 and BRDB_HKP_ORAFILES2.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 77 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.66.1 Dependencies
Schedule BRDB_ADMIN depends on the completion of schedules BRDB_AUD_FEED,
BRDB_AGG_COMPL and BRDB_XFR_COMPL.
3.66.2 Job BRDBC004
This job runs the Audit, Archive and Purge process. See Section 5.7 for the latest in BRDBC004
archival and purging logic.
3.66.2.1 Implementation
This job is implemented by a call to the executable BRDBC004.
3.66.2.2 Rerun Action
Pee Prompis for rerun = action? *4
3.66.3 Job BRDBX006
This job runs the BRDB File Housekeeping process.
3.66.3.1 Implementation
This job is implemented by a call to the shell script BRDBX006.sh.
3.66.3.2 Rerun Action
SP Prompts for renin > action?’
3.66.4 Job BRDB_HKP_ORAFILES1
This job (run on each node) runs the Oracle File Housekeeping process for the BRDB.
3.66.4.1 Implementation
This job is implemented by a call to the shell script HouseKeepOrafiles.sh with the database name
BRDB.
3.66.4.2 Rerun Action
Prompts for fenin = action? ™
3.66.5 Job BRDB_HKP_ORAFILES2
This job (run on each node) runs the Oracle File Housekeeping process for ASM.
3.66.5.1 Implementation
This job is implemented by a call to the shell script HouseKeepOrafiles.sh with the database name
“+ASM’.
3.66.5.2 Rerun Action
Prompts for rerun = action? **
3.67 Schedule BRDB_PAUSE_FEED2
This schedule is run daily. It stops the two NPS copy processes prior to end of day processing. It consists
of two tasks which can be run on any active node; see section 0 above for details. Only the two parent
jobs are included here, which are:
BRDBX011_PAUSE_NPS_TT_COPY
BRDBX011_PAUSE_NPS_GREV_COPY
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 78 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.67.1 Dependencies
Schedule BRDB_PAUSE_FEED2 depends on the completion of schedules BRDB_ADMIN and
BRDB_CSH_TO_LFS.
3.67.2 Job BRDBX011_PAUSE_NPS_TT_COPY
This job stops the copying of Track and Trace transactions to NPS, by setting a system parameter (see
section 3.5).
3.67.2.1 Implementation
This job is implemented by a call to the shell script BRDBX011.sh specifying the relevant system
parameter name BRDB_TT_TXN_TO_NPS_STOP_YN and value “Y”.
3.67.2.2. Rerun Action
Alert Operations on failure.
3.67.3. Job BRDBX011_PAUSE_NPS_GREV_COPY
This job stops the copying of Reversals transactions to NPS, by setting a system parameter (see section
3.5).
3.67.3.1 Implementation
This job is implemented by a call to the shell script BRDBX011.sh specifying the relevant system
parameter name BRDB_REV_TXN_TO_NPS_STOP_YN and value “Y”.
3.67.3.2 Rerun Action
Alert Operations on failure!
3.68 Schedule BRDB_EOD
This schedule is run daily. It runs the BRDB end of day utility. It consists of a single task which can be
run on any active node; see section 0 above for details. Only the parent job BRDBC009 is included here.
Additional monitoring is required so that an alert is raised if this job has not completed by 04:00. This is
implemented within the BRDB_MONITOR schedule — see section 3.76.
3.68.1 Dependencies
Schedule BRDB_EOD depends on the completion of schedule BRDB_PAUSE_FEED2.
3.68.2 Job BRDBC009
This job runs the BRDB end of day utility; resets BRDB_OPERATIONAL_INSTANCES.IS_AVAILABLE
to 'Y' if the instance was previously down but is now available.
3.68.1.1 Implementation
This job is implemented by a call to the executable BRDBCO009.
3.68.1.2 Rerun Action
Prompts for tenun = action? 4
3.69 Schedule BRDB_START_FEED2
This schedule is run daily. It prepares for the running of the two NPS copy processes by reversing the
changes that stopped them earlier in the schedule. It consists of two tasks which can be run on any
active node; see section 0 above for details. Only the two parent jobs are included here, which are:
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 79 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
BRDBX011_START_NPS_TT_COPY
BRDBX011_START_NPS_GREV_COPY
3.69.1 Dependencies
Schedule BRDB_START_FEED2 depends on the completion of schedule BRDB_EOD.
3.69.2 Job BRDBX011_START_NPS_TT_COPY
This job prepares for the starting of the copying of Track and Trace transactions to NPS, by setting a
system parameter (see section 3.5).
3.69.2.1 Implementation
This job is implemented by a call to the shell script BRDBX011.sh specifying the relevant system
parameter name BRDB_TT_TXN_TO_NPS_STOP_YN and value “N”.
3.69.2.2. Rerun Action
Alert Operations on failure:
3.69.3 Job BRDBX011_START_NPS_GREV_COPY
This job prepares for the starting of the copying of Reversals transactions to NPS, by setting a system
parameter (see section 3.5).
3.69.3.1 Implementation
This job is implemented by a call to the shell script BRDBX011.sh specifying the relevant system
parameter name BRDB_REV_TXN_TO_NPS_STOP_YN and value “N”.
3.69.3.2 Rerun Action
AlertOperations on failure!
3.70 Schedule BRDB_TT_TO_NPS2
This schedule is run daily to restart the Track and Trace NPS data feed after end of day processing. It
consists of a single task which is run on each active node by jobs named
BRDBX003_TT_TO_NPS_1...4_NOPAGE.
3.70.1 Dependencies
Schedule BRDB_TT_TO_NPS2 depends on the completion of schedule BRDB_START_FEED2.
3.70.2 Job BRDBX003_TT_TO_NPS_1...4_NOPAGE
These jobs (one per node) start the feed that copies the Track and Trace transactions to NPS.
3.70.1.1 Implementation
These jobs are implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_TT_TXN_TO_NPS.
3.70.1.2 Database Link Information
NBX_TT_HARVESTER_1@NPS2
3.70.1.3. Rerun Action
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 80 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.71 Schedule BRDB_GREV_NPS2
This schedule is run daily to restart the Reversals NPS data feed after end of day processing. It consists
of a_ single task which is run on each active node by jobs named
BRDBX003_GREV_TO_NPS_1...4_NOPAGE.
3.71.1 Dependencies
Schedule BRDB_GREV_NPS2 depends on the completion of schedule BRDB_START_FEED2.
3.71.2 Job BRDBX003_GREV_TO_NPS_1...4.NOPAGE
These jobs (one per node) start the feed that copies the Reversals transactions to NPS.
3.71.2.1 Implementation
These jobs are implemented by a call to the shell script BRDBX003.sh specifying the relevant feed name
BRDB_REV_TXN_TO_NPS.
3.71.2.2_ Database Link Information
NBX_GREV_AGENT_1@NPS1
3.71.2.3. Rerun Action
Refui on failure!
3.72 Schedule BRDB_START_BKP
This schedule is run daily. It marks the start of the backup schedule.
3.72.1 Dependencies
Schedule BRDB_START_BKP depends on the completion of schedule BRDB_EOD.
3.72.2 Job COMPLETE
This job simply echoes a message before exiting.
3.72.2.1 Implementation
This job is implemented by a call to the echo command.
3.72.2.2_ Rerun Action
None.
3.73 Schedule BRDB_BACKUP_0
This schedule is run on Sundays and Wednesdays. It performs the level 0 backup. It consists of a single
task which can be run on any active node; see section 0 above for details. Only the parent job
BRDB_LVLO_BACKUP is included here.
3.73.1. Dependencies
Schedule BRDB_BACKUP_O depends on the completion of schedule BRDB_START_BKP.
3.73.2 Job BRDB_LVLO_BACKUP
This job performs the file transfer for the BRDB Branch Migration Status data feed.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 81 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.73.2.1 Implementation
This job is implemented by a call to the shell script RMANBackup.sh with database name BRDB and
level value 0.
3.73.2.2. Rerun Action
28 Prompts for rerun ~ action?
3.74 Schedule BRDB_BACKUP_1
This schedule is run on every day except Sundays and Wednesdays. It performs the level 1 backup. It
consists of a single task which can be run on any active node; see section 0 above for details. Only the
parent job BRDB_LVL1_BACKUP is included here.
3.74.1 Dependencies
Schedule BRDB_BACKUP_1 depends on the completion of schedule BRDB_START_BKP.
3.74.2 Job BRDB_LVL1_BACKUP
Kicks off an RMAN level 1 backup.
3.74.2.1 Implementation
This job is implemented by a call to the shell script RMANBackup.sh with database name BRDB and
level value 1.
3.74.2.2. Rerun Action
#5 Prompts for rerun = action?
3.75 Schedule BRDB_BKP_COMPL
This schedule is run daily. It checks that the backup schedule has completed and creates a flag file via
the job CREATE_BRDB_BKUP_COMPLETE_FLAG.
3.75.1 Dependencies
Schedule BRDB_BKP_COMPLETE depends on the completion of whichever of schedule
BRDB_BACKUP_0 or BRDB_BACKUP_1 that applies on the appropriate day.
3.75.2 Job CREATE_BRDB_COMPLETE_FLAG
This job creates the flag file /opt/tws/FLAGS/BRDB_BKUP_complete.FLAG.
3.75.2.1 Implementation
This job is implemented by a call to the “touch” command with the relevant file name.
3.75.2.2 Rerun Action
Prompts for rerun = action?
3.76 Schedule BRDB_MONITOR
This schedule is run daily. It checks that other jobs have completed by a specified time. (See section
3.4.)
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 82 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.76.1 Dependencies
None
3.76.2 Job BRDB_MON_STARTUP
This checks that the BRDB_STARTUP job has completed by the required time of 06:00.
3.76.2.1 Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name
and time.
3.76.2.2 Rerun Action
None.
3.76.3. Job BRDB_MON_PAUSE_FEED1
This checks that the BRDB_PAUSE_FEED1 job has completed by the required time of 07:59.
3.76.3.1 Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name
and time.
3.76.3.2 Rerun Action
None.
3.76.4 Job BRDB_MON_AUD_FEED
This checks that the BRDB_AUD_FEED job has completed by the required time of 04:00 EGE
3.76.4.1 Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name
and time.
3.76.4.2 Rerun Action
None.
3.76.5 Job BRDB_MON_EOD
This checks that the BRDB_EOD job has completed by the required time of 04:00.
3.76.5.1 Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name
and time.
3.76.5.2 Rerun Action
None.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 83 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.77 Schedule BRDB_POE_LOAD
This schedule is run daily and converts any available POLSAP PDF documents into PNG format and
loads into table OPS$BRDB.BRDB_EXT_FEED_REPORTS.
3.77.1 Job BRDBC038_POE_FROM_POLSAP
3.77.1.1 Implementation
This job calls executable BRDBC038 which will look for any PDFs in the POLSAP share directory (see
table below for details). If no files are found then sleep for 600 seconds, look again - do this for 3
iterations and log an exception if no files found but exit 0.
The following is a list of directories used by this job: -
Note: The list is stored as values in the following table columns for the row "WHERE
ext_interface_feed_name = ‘BRDB_POE_FROM_POLSAP’.
Description Column Name Value
POLSAP share directory INPUTSHARE_DIR_NAME Japplbrdb/trans/polsap
BRDB input directory BRDB_INPUT_DIR_NAME Japp/brdb/trans/externalinterface/input
BRDB audit directory AUDIT_DIR_NAME Japp/brdb/trans/audit/externalinterfaceaudit/poe
BRDB PNG load directory BRDB_LOAD_DIR_NAME Japp/brdb/trans/externalinterface/loaddir
3.77.1.2 File Retention Periods
Processed PDF & PNG files (i.e. those with an uppercase extension) will be retained on the BRDB file
system as per the metadata defined in BRDB_FILES_TO_HOUSEKEEP.
3.77.1.3. Rerun Action
Correct the root cause of the failure and rerun the job.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 84 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
3.78 Schedule BRDB_PAFCD_LOAD
This schedule is run every Sunday at 13:30 only and loads the latest PAF files (postcode data files)
received from the Post Office, if available.
3.78.1 Job BRDBC038_PAF_FROM_CD
3.78.1.1 Implementation
For more on this process, please see Section 3.6.4
This job calls executable BRDBC038 in the following way: -
${BRDB_PROC}/BRDBCO38 BRDB_PAF_FROM_CD *BRDBBDAY*
BRDBCO038 will attempt to find the PAF files, of the form *compstc*.*.paf, in the
INPUTSHARE_DIR_NAME (see table below for details), register their existence within the database,
copy them to BRDB_INPUT_DIR_NAME and then calls BRDBC040, which performs validation on the
files and then calls a separate import process to load them.
The following is a list of directories used by this job: -
Note: The list is stored as values in the following table columns for the row "WHERE
ext_interface_feed_name = ‘BRDB_PAF_FROM_CD’.
Description Column Name Value
PAF (REF data) share directory INPUTSHARE_DIR_NAME Japplbrdb/trans/supportworking
BRDB input directory BRDB_INPUT_DIR_NAME Jappibrdb/trans/externalinterface/input
BRDB audit directory AUDIT_DIR_NAME NIA
BRDB PAF load directory BRDB_LOAD_DIR_NAME Japp/brdb/trans/externalinterface/loaddir
3.78.1.2 File Retention Periods
Processed PAF files - those with an uppercase extension, e.g. *.PAF - will be retained on the BRDB file
system as per the metadata defined in BRDB_FILES_TO_HOUSEKEEP.
3.78.1.3. Failure Action
Determine the root cause and notify Support teams. Possible failures could include corrupt files, or
spurious data, lack of disk space or other similar problems.
3.78.1.4 Rerun Action
None. The schedule will not need to be held.
3.79 Schedule BRDB_PAFADD_LOAD
This schedule is run every day, including Sundays and loads a PAF file received from the Post Office,
which is different from the files delivered for the full PAF load (See Section 3.78).
3.79.1 Job BRDBC038_PAF_ADD_LOAD
3.79.1.1 Implementation
For more on this process, please see Section 3.6.5
This job calls executable BRDBC038 in the following way: -
${BRDB_PROC}/BRDBCO38 BRDB_PAF ADD LOAD “*BRDBBDAY~
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 85 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
BRDBC038 will attempt to find the “additional data” PAF file, of the form *compstd*.*.paf, in the
INPUTSHARE_DIR_NAME (see table below for details), register its existence within the database, copy
it to BRDB_INPUT_DIR_NAME, also copy it to AUDIT_DIR_LNAME and then calls BRDBC040, which
performs validation on it and then calls a separate import process to load it.
The following is a list of directories used by this job: -
Note: The list is stored as values in the following table columns for the row "WHERE
ext_interface_feed_name = ‘BRDB_PAF_ADD_LOAD’.
Description Column Name Value
PAF (REF data) share directory INPUTSHARE_DIR_NAME Japplbrdb/trans/supportworking
BRDB input directory BRDB_INPUT_DIR_NAME Japp/brdb/trans/externalinterface/input
BRDB audit directory AUDIT_DIR_NAME Japp/brdb/trans/audit/externalinterfaceauditipat
BRDB PAF load directory BRDB_LOAD_DIR_NAME Japplbrdb/trans/externalinterface/loaddir
3.79.1.2 File Retention Periods
Processed PAF files - those with an uppercase extension, e.g. *.PAF - will be retained on the BRDB file
system as per the metadata defined in BRDB_FILES_TO_HOUSEKEEP.
3.79.1.3. Failure Action
Determine the root cause and notify Support teams. Possible failures could include a corrupt file, or
spurious data within the file, lack of disk space or other similar problems.
3.79.1.4 Rerun Action
None. The schedule will not need to be held.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 86 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
4 Backup and Recovery
The Branch Database and Branch Support Database are both backed up using Oracle RMAN. The
frequency of the backups, the type of backup, the backup location and retention periods are detailed in
the Branch Database High Level Design (See Section 0.4).
4.1 BRDB & BRSS Backups
4.1.1 Backup Duration
The Oracle RMAN backups, when run, tend to do so for different durations. The factors that will affect
run-time could be: -
e Activity on the node executing the backup, e.g. CPU, disk, etc.
« The type of backup being run, e.g. a full backup (incremental level 0) or an incremental level 1
backup.
e The amount of archivelogs generated since the last backup (relevant to any backup level).
It is therefore important that when backups are not run for whatever reason, that they are re-scheduled to
run as soon as possible.
4.1.1.1. RMAN & Streams
RMAN, by default, is configured to remove any archivelogs after a successful backup. Streams has a
direct impact on whether or not RMAN is able to remove an archivelog or not. This criterion is
determined by whether the archivelog is or is not needed by the Streams Capture process.
If Streams does require the archivelog, RMAN is not “allowed” to remove it and the archivelog will
remain in +BRDB_FLASH/arch. An RMAN-08137 message will be reported when this is the case. It is a
warning message and not a failure.
Any subsequent backups will skip each archivelog as each one already has a successful copy in a
previous backup. When attempting to drop the archivelog again, the same check is made and if
Streams no longer need the archivelog, it will be released for deletion by RMAN.
4.2 Restoring files with RMAN
DBAs in Ireland have standard support procedures for dealing with restores and recovery after differing
failures, e.g. restoring SPFiles, controlfiles, archivelogs, datafiles, et cetera. These scripts and
procedures will be used by the DBA Support Team in a recovery scenario in conjunction with this guide
and support from technical leads and possibly vendor specialists, e.g. EMC, Oracle, et cetera.
WARNING: As with any activity relating to the physical dimension of restoring activities, keeping the
high importance of these types of activities at the back of one’s mind is of paramount
importance! Restoring datafiles or redologs using RMAN, for instance, could cause the
crash of the entire Branch Database if performed in a non-disaster scenario and without
the proper authorisation!
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 87 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
4.3 Failure and Recovery
Failures should be detected by SMC and then escalated to the UNIX/DBA teams who, in turn where
appropriate, will escalate to CS, SSC and development.
Recovery actions will be performed by the UNIX/DBA teams with the agreement of CS, SSC and
development.
Business escalation should be handled by SMC.
4.3.1 Escalation and Notification
NB: _ In the event of a failure and subsequent recovery, the relevant Post Office Disaster Recovery
escalation procedures need to be followed in conjunction with the relevant Business Continuity personnel
and Fujitsu Support Teams.
The Business Continuity function along with the relevant management team(s) will have to consider the
facts, weigh up the current threats and decide whether to authorise the failover to Standby or not.
In general, the hierarchy in which support teams are contacted is as follows: -
e SMC will typically coordinate all types of failures and will also be the first point of contact in most
types of problems, application, networks, etc.; Responsible for monitoring Tivoli.
e SSC is responsible for supporting the application. DBA, UNIX and Network Support Teams are
also responsible for support at this level
e Finally, the development teams would support all other teams in their respective areas of
expertise.
4.3.2 Media Failure and Recovery
4.3.2.1 A Corrupt or Damaged Redolog Group
If an online redolog group has all of it's members damaged - regardless of how this came to be - the
recovery solution will change depending on the ‘state’ of the online redolog group.
4.3.2.1.1 Scenario and Recovery Solution
Scenario: This failure scenario involves having all redologs of a particular redo log group,
corrupted or damaged.
Solution: Redolog Group is INACTIVE
This redolog group will not be required for crash recovery.
Action > Clear the logfile group.
Redolog Group is ACTIVE
This redolog group is required for crash recovery.
Action > (i.) Issue a checkpoint and (ii.) clear the damaged redolog(s). If performing
(i.) and (ii.) prove unsuccessful, then the database must be restored and recovered
(incomplete recovery) to a point-in-time before the redolog(s) were damaged (to the
most recent available group prior to damage).
Redolog Group is CURRENT
This redolog group is required for crash recovery.
Action > Clear the damaged redolog(s) (do not attempt a checkpoint). If performing (i.)
is unsuccessful, then the database must be restored and recovered (incomplete
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 88 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
recovery) to a point-in-time before the redolog(s) were damaged (to the most recent
available group prior to damage).
Note:
e Depending on our SLA with the customer (in terms of time-to-recover), it may be more
advantageous to either complete the restore and recovery.
Or if the corruption is localised, i.e. only present on the hardware of the current site (e.g. IRE11),
then failing the Data Centre over (e.g. to IRE19) may be a faster (less troublesome) route to
take.
¢ The Database failover from PRIMARY to STANDBY is not recommended in this scenario.
4.3.3. Instance/Node Failure and Recovery
4.3.3.1 Working Assumptions
The guidance in the following sections assumes that every effort to resolve a failure — be that failure due
to software, hardware, network or failures of greater magnitude — has been taken. For hardware failures
this can include checking Oracle CRS logs or Linux system logs and in the case of database instance
failures, alert_BRDB[1I2/3I4].log, trace files, application and process log files, CRS logs, dump files and
Grid Control alert messages. This is by no means an exhaustive list.
The recovery of an Oracle Database instance is essentially automatic as Oracle provides internal
mechanisms which perform instance recovery on startup.
The recovery of a pBlade within the BRDB BladeFrame is similarly automatic, in that the BladeFrame
will attempt to bring the failed pBlade back online; but if unsuccessful, a replacement of the pBlade with
an operational “spare”, while not automatic is fairly trouble-free
There is a Very notable Caveat to this in that if all 4 nodes go down, and that is that OCFS2 must be
checked and verified as started (on each node). If not, the clusters will not be able to correctly
communicate with each other and it has been noted (during periods of testing) that cluster timeouts can
occur in this scenario and cause further cluster failures. Verifying the status of OCFS2 is therefore
critical.
To check that OCFS2 is started and has the correct configuration information, perform the following
check on all affected nodes as the root user (the output should show “configured” matching “active”): -
$> /etc/init.d/ocfs2 status
Configured OCFS2 mountpoints: /u02/oradata /u03/oradata /u04/oradata
Active OCFS2 mountpoints: /u02/oradata /u03/oradata /u04/oradata
Oracle Cluster Ready Services (CRS), in normal operation will automatically restart any database
instance on a node that is being restarted (for whatever reason). This will always include the grid contro!
agent(s), the Oracle listener and the local ASM instance. However, the starting of the database instance
- which is dependant on the ASM instance having started — will be disabled for all Branch Database
Cluster Ready Services. That is, upon restart, all components required by the database instance will be
restarted except for the instance itself.
What is important to note, is that within BRDB, database instances are as much a logical part of the
application DNA as they are literal entities of an Oracle RAC database. Therefore when an instance or
node fails, its recovery will always represent a two-fold process, logically within the application and the
actual node/instance itself.
4.3.3.2 I Single BRDB Instance Crash
The instance will automatically be removed from BRDB_OPERATIONAL_INSTANCES by BRDBX010
which is invoked by the Fast Application Notification (FAN) mechanism at the time of the instance
failure. Note that BRDBX010 is only executed by the FAN event and not by any other means.
The failed instance will need to be started manually via Grid Control or SQL*Plus. Starting the instance
is an activity that needs to be thought through. The reason for this is that once the failed instance has
been started manually, the cluster will once again show the full complement of instances and the listener
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 89 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
can begin accepting connections for that instance. However the ‘logical’ view represented in
BRDB_OPERATIONAL_INSTANCES will show that the instance in question is not available for requests
from the Branch Access Layer (BAL). At this point, therefore, the physical database instance has been
started, but the application will not be aware of that fact. This is done by stopping and starting, in a
sequential manner, each Online Service Router (OSR) in turn (of which there are 20).
Please note: The instructions that follow, detail the updating of BRDB_OPERATIONAL_INSTANCES
using BRDBX013 or by a manual update. It is particularly important to note that this
should be done prior to “making the application aware”, i.e. stopping and starting each
OSR to reflect the change.
At the end of the online-day (after 18:00 and preferably before the overnight schedules start, but not
essential), the recommended approach is to make the instances logically available, manually. This is
done by either executing BRDBX013 (BRDBX013 will check the state of each instance, whether up or
down, and update BRDB_OPERATIONAL_INSTANCES accordingly) or by following the instructions in the
table (Table 2) below. This is especially relevant if one wants granular control of what is represented in
that table, as BRDBX013 will update all rows if necessary in order to ensure that the table represents the
actual state of the cluster and this may not be required in every case.
BRDBX013 is executed as follows: -
$> cd /app_sw/brdb/sh
$> BRDBX013.sh
Finally, at the end of the Business day, the “End Of Day” process, namely BRDBCO009, will check that all
available instances are logically and correctly represented in BRDB_OPERATIONAL_INSTANCES and if
not, will update the table to reflect the correct real-world representation. Having BRDBC009 perform this
task is not necessarily the best course of action as the BAL needs to be made aware that the instance
mapping has changed (this is done as detailed above). Therefore, BRDBCO09 should be seen as a
backup action rather than the preferred.
If, for whatever reason, the failed instance, once started and open, needs to be made available to the
BAL and before the end of the day, then the following must be followed. Using meaningful and accurate
values for the following values, e.g.: -
SFANEVént Stings: Manual recovery by Andrew Aylward for fast recovery of
instance due to unexpected node failure. Authorisation
given by Graham Allen.
<Host Name>: 11tpbdb001 (obtain by typing hostname Or uname -n on the relevant node).
Step Description Server Execution
. i. User is logged onto any node of the BRDB cluster as the brdb user.
SS ,
um ii. \t is imperative that there are no schedule related processes running when this manual
Ptio operation is performed. There are many schedule related jobs which are fad-
ns hash/branch code dependant and if these mappings are changed mid-schedule,
significant problems could occur!
Logon to SQL*Plus command-line I $> . oraenv
interface as OPS$BRDB, but first set
the correct Oracle SID. [now type in BRDB1 (assuming you're on node 1)]
1.
This will connect you to the BRDB I $> sqlplus /
database.
Double-check that you are on the right I SQL> SELECT * FROM v$instance;
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 90 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
instance, noting in particular the
values for instance_name, host_name
and status.
Execute this DML to re-instate the
availability of the instance in question.
Commit your change
Table 2: BRDB_OPERATIONAL_INSTANCES Update Instructions
4.3.3.3. Single BRDB Node Crash and Restart
Failure notification will occur via the ITM Tivoli agent and will also be visible via Grid Control in terms of
instance availability notification. FAN will update logical instance availability upon failure.
PAN Manager (BladeFrame operational software) will attempt to automatically restart the failed pServer.
Once the pServer is initialised, the node has started, and with it the listener and ASM. The instance
must be manually started.
See section 4.3.3.2 for more on re-instating logical instance availability.
4.3.3.4 Single BRDB Instance Crash - Fails to Start
See section 4.3.3.8.
4.3.3.5 Single BRDB Node Crash - Fails to Restart
Failure notification will occur via the ITM Tivoli agent and will also be visible via Grid Control in terms of
instance availability notification. FAN will update logical instance unavailability upon failure.
If the BladeFrame cannot automatically restart the failed pServer, the PAN manager will flag an error.
An attempt will be made at restarting the pServer on the spare pBlade. If unsuccessful, Support will then
need to follow it up and resolve accordingly. Either solving the problem or replacing the pBlade and
attempting another restart.
The BAL will not have “use” of the now unavailable instance until such time as the node's failure has
been resolved and the instance is made available on the new/repaired node, by Support. As well as the
instance being logically made available by either the EOD process (BRDBCO09) or through manual
intervention (described in section 4.3.3.2). BRDBCO09 will continue to report in
BRDB_OPERATIONAL_EXCEPTIONS, that the instance is unavailable.
4.3.3.6 Two or More BRDB Instances Crash
As mentioned in section 4.3.3.2, the BAL will not have “use” of the now unavailable instances until such
time as each instance is available and either the EOD process (BRDBC009) has run or through manual
intervention.
Each failed instance will need to be started manually via Grid Control or SQL*Plus.
If the instances restart successfully, then Support must make the instances “logically” available by the
manual process specified in section 4.3.3.2, for each instance.
Depending on the consensus of Support personnel, making “logically” available the newly started
instances can be done at this point. The reason for either making the instances available or not is
simply to do with the load on the remaining nodes and whether it is perceived that they are able to cope.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 91 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
If, however, the instances are unable to restart or do restart but have further problems presenting
themselves, e.g. they aren't accepting requests, there are network issues, loss of ASM diskgroups, et
cetera, then the instances should be treated as non-restartable and the relevant escalation process
should be followed (see Section 4.3.1).
4.3.3.7 Two or More BRDB Nodes Crash and Restart
Failure notification will occur via the ITM Tivoli agent and will also be visible via Grid Control in terms of
instance availability notification. FAN will update logical instance unavailability upon failure.
The BladeFrame will attempt to automatically restart the failed pServers (on related pBlades) as defined
by the LPAN configuration. Once the blades are initialised and the nodes have restarted, normal
behaviour would dictate that the related database instances are started again automatically. As with the
scenario presented in section 4.3.3.3, the instances must be manually started and then made available
to the BAL as the cluster will not bring them up automatically.
Depending on the consensus of Support personnel, making “logically” available the newly started
instances can be done at this point. The reason for either making the instances available or not is
simply to do with the load on the remaining nodes and whether it is perceived that they are able to cope.
See section 4.3.3.2 for more on re-instating logical instance availability. This applies for every instance.
4.3.3.8 I Two or More BRDB Instances Crash — Fail to Restart
It must be assumed that every effort has been employed in restarting the instance(s) within the agreed
SLA. If this two-or-more-instance-failure persists, then the following logic in determining an outcome
should apply.
Has the problem occurred outside core business hours?
If yes, and there are at least two RAC instance(s) in full operation, then there may be sufficient
throughput available for the effective servicing of reduced business traffic. In such cases, it is often
more beneficial to continue to use BRDB (the primary database), rather than initiate the failover
procedure (see Section 0) which details the failing over of all users to SBRDB (the standby database) as
this involves a coordinated, multi-team effort (for escalation see Section 4.3.1). In addition it will also
allow more time for the resolution of the main reason for failure, be it software or hardware related.
If no or there are more than two instance failures, then the very real possibility that severe degradation
in transaction throughput will present itself. At this point then the instances should be treated as non-
restartable and the relevant escalation process should be followed (see Section 4.3.1).
4.3.3.9 I Two or More BRDB Nodes Crash — Fail to Restart
Similar in resolution to section 4.3.3.8
It must be assumed that every effort has been employed in restarting the failed pBlades and have them
correctly integrated into the cluster within the agreed SLA. If this two-or-more-node-failure persists, then
the following logic in determining an outcome should apply.
Has the problem occurred outside core business hours?
If yes, and there are at least two nodes of the RAC cluster still in full operation, then there may be
sufficient throughput available for the reduced business traffic. In such cases, it is often more beneficial
to continue to use the BRDB (primary database) cluster, rather than initiate the failover procedure (See
Appendix A) which details the failing over of all users to the SBRDB (standby database) cluster as this
involves a coordinated, multi-team effort. In addition it will also allow the resolution of the main reason
for failure, be it hardware related or not.
If no or there is only a single node available, then the very real possibility that severe degradation in
transaction throughput will present itself. The Business Continuity function along with the relevant
management team will have to consider the facts, weigh up the current threats and decide whether to
authorise the failover to the Standby cluster or not.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 92 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
See section 4.3.1 for the service team/support team contact and escalation hierarchy.
Complete failover could be manually initiated and if so will need to follow the steps outlined in Section 6.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 93 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
5 General and Troubleshooting Notes
5.1 Database
5.1.1 Oracle Database Listeners
The database listeners on all branch database nodes have been set up in the following way. This
section provides a short explanation of how they are set up, how to interact with them and the expected
status outputs.
The listeners are configured as follows: -
« The name of the listener will be of the form LISTENER_<node name>, e.g.
LISTENER_LPRPBDBOO1
« The port the listener has been configured to use is 1529
e The node (and in turn the IP) the listener has been configured to accept connections for is
Iprp<type>00[1234]-vip, e.g. for BDB node 1 the node name is Iprpbdb001-vip
In terms of Oracle Net and it’s configuration files, there should always be one of each on every node,
namely sqlnet.ora, tnsnames.ora and the listener.ora (found in
$ORACLE_HOME/network/admin)
5.1.1.1. Oracle Net Config. Files
The files have been formerly delivered by Tivoli Provisioning Manager and won't be needed to be
changed unless there is a specific problem. The following, shows a few excerpts of what the files could
look like as of October 2009 (note that these values are not representative of those in the LIVE
environment and are merely for reference): -
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 94 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
sqlnet.ora
SQLNET . INBOUND_CONNECT_TIMEOUT=15 performs the same function and behaves in the same way
as the parameter configured for the listener, only waits longer.
SQLNET.EXPIRE_TIME=5 determines the number of minutes that Oracle will allow connections which
are not in use, to exist, before terminating the process. This normally applies to connections which have
abnormally ended.
if* aaylw01@lprpbdb001:~
.lprpbdboo. C nfiguration
Oracle nf a’ on to
NET.
QLNET. EXPIRE TINE
tnsnames.ora
The tnsnames.ora would ordinarily only have entries that are applicable to the instance(s) which exist on
that node alone. However, the build process uses a single tnsnames.ora for all nodes. This is not ideal,
but is how it has been delivered.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 95 of 183
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE @
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
listener.ora
INBOUND_CONNECT_TIMEOUT_LISTENER_LPRPBDBOO1 = 10 determines the number of seconds
Oracle will wait to receive authentication from the client making the connection. Otherwise denies the
request.
ADMIN_RESTRICTIONS LISTENER_LPRPBDBOO1 = ON enforces the administration of the listener to
an authorised user only, i.e. oracle
oepbdl
LprpbdbO01) (POR’
5.1.1.2 Interaction with the Listener
Starting and stopping the listener is done via Oracle CRS as follows: -
$> srvctl stop listener -n lprpbdb001
$> srvctl start listener -n lprpbdb001
Checking the status of the listener and it’s services is done as follows: -
$> Isnrcetl status LISTENER_LPRPBDBOO1
/10.2.0/db_1/network/admin/listener.ora
Listener Parameter /201/app/ora
/10..2.0/db_1/network/1og/listener_1prpbdb001.1og
Listener Log File /401/app/ora
Listening Endpoints Summary
(DESCRIPTION= (ADDRESS= (PROTOCO!
(DESCRIPTION= (ADDRESS= (PROTOCOL:
Services Summary
Service "+ASMI" has 1 insta
op) (HOST=172. 18.50.16) (PORT=1529) ))
cp) (HOST=172.18.50.1) (PORT=1529) ))
(s)
Instance M1", status UNKNOWN, has 1 handler(s) for this service
Service "BRDB" has 1 nce (s)
Instance "BRDB1", status READY, has 1 handler(s) for this service
Service "BRDB_DGB" has 1 instance (s)
Instance "BRDB1", status READY, has 1 handler(s) for this service.
Service "BRDB_XPT" has 1 instance (s)
Instance "BRDB1", status READY, has 1 handler(s) for this service
Service "SYS$STRADMIN.BRDB_CAPTQ.BRDB" has 1 instance(s)
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 96 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
Instance "BRDB1", status READY, has 1 handler(s) for this service
The command completed successfully
Executing lsnrctl services LISTENER_LPRPBDBOO1 will show a little more information for each
service than the status command.
The important services used are listed as follows: -
+ASM[1234] This service is required for Grid Control and allows access to ASM.
BRDB This service is generally required for the BAL and TWS and allows those applications to
connect without specifying an individual instance.
BRDB_DGB and SYS$STRADMIN.BRDB_CAPTQ.BRDB are Oracle defined services and relate to and are
used by Data Guard and Streams respectively.
If any services are not created, then client connections which use those services will be unable to
connect. This is similar to the status of the listener itself in that unless it is continually being monitored,
the only way one will really know there is an issue, is with the inability to connect.
5.1.2. General Recommendations
5.1.2.1. Logs and Trace Files.
From time to time there will be important log files, trace files and background process dump files that will
be needed for support purposes and would have been explicitly renamed and “saved” by support
personnel. These files, if found in a “house kept” directory, will be removed by the housekeeping
processes after the retention period has been exceeded. For quick reference those directories are: -
- /u01/admin/<DB>/adump
- /u01/admin/<DB>/bdump
- /u01/admin/<DB>/cdump
- — /u01/admin/<DB>/udump
- $ORACLE_HOME/network/log
- $ORACLE_HOME/rdbms/log
The database alert log and the listener log files are always being written to and are important files. It is
highly recommended that these files are kept manageable. A good way of doing this would be to copy
the files every month or fortnightly in order to keep a history and keep their sizes at a manageable level.
5.1.3 Password Management
In general all Branch Database and Branch Support Database passwords fall into one of three
categories: -
e The users are locked (within the database) and even if the password is known, logging on is not
a possibility.
e The passwords are managed by Microsoft Active Directory. This is possible because the users
that this applies to are “externally identified” and in order to logon, one must be logged onto the
server as an OS user and then log onto the database, thereby relying on OS authentication.
« The passwords are set by privileged users and known to only secure/trusted personnel. This can
only apply to privileged users, e.g. SYSTEM, SYS, DBSNMP, etc. The following table shows
interdependencies of database users of this type: -
Interdependencies Risk If Changed
sys Oracle Grid Control Grid Contro! Agents will be unable to logon
(See Section Oracle Data Guard Standby Database log shipping and coordination
5.1.3.1) will fail
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref: DES/APP/SPG/0001
CONFIDENCE) Version 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 97 of 183
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE @
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
SYSTEM None None
DBSNMP Oracle Grid Control Grid Control Agents will be unable to logon
AUDITUSER The Audit Server Audit Server will fail to logon
BMC_USERLV
BMC Patrol None
BMC_USERTR
BRDBRDDS RDDS Feeds None
BRDBRDMC RDMC Feeds None
DELTRUSER Counter Training None
EMDB_SUP The EMDB Interface EMDB Interface will fail to refresh branch info
OMDBUSER The OMDB Interface OMDB Interface will fail to refresh branch info
LVBALUSER[1-4] Live Counter Connections The BAL OSR will fail to startup correctly
ORAEXCPLV BRDB Exception logging In the event of a failure, BRDB processes will not
be able to log exceptions
REP_GEN Generic Reporting Reports will fail to generate
STRADMIN Oracle Streams Streams administration will not be possible after
change
TRBALUSER[1-4] I Training Counter Connections I Counter training will not be possible
TWSs
The TWS Scheduler All schedules will fail to run
TWSSUP
5.1.3.1. Changing the SYSDBA Password
The SYS passwords have related §¥sdba password files for both the main application instance and ASM
instance on 4 HOdéS of any Online RAC Cluster. The significance of the password file is that the
password internal to the database (for the SYS user) must match the password with which the password
file was created. If either of them changes without the other, all remote logons will fail with an
“Insufficient Privileges” ORA- error.
When these passwords are changes, for whatever reason, the changes when done, must be done in
sync with the respective password file(s). Oracle Grid Control and Oracle Data Guard rely on being able
to logon remotely as privileged users.
The instances affected on BDB are as follows: -
BRDB[1I2I3/4] and +ASM[1I2/3/4]
The instances affected on BDS are as follows: -
SBRDB[1I2I3/4] and +ASM[1I2I3]/4]
The instances affected on BRS are as follows: -
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 98 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
BRSS[1] and +ASM[1]
Then on every node a password file will exist in <ORACLE_HOME>/dbs of the form
orapw<ORACLE_SID>, for each instance above.
For example should one wish to change the ‘SYS’ password on BDB node 3 to ‘bObsyOuruncl3’, one
would perform the following tasks as the oracle user logged onto node 3: -
Logon to BRDB3 and change the password:
$> sqlplus ‘/as sysdba’
SQL> ALTER USER SYS IDENTIFIED BY bObsyOuruncl3;
SQL> EXIT;
Recreate the password file:
$> cd $ORACLE_HOME/dbs
$> orapwd file=/u01/app/oracle/product/10.2.0/db_1/dbs/orapwBRDB3
password=b0bsyOuruncl3 entries=5
Note: The process for changing the ASM password is the same as that for the database instance.
5.1.3.2 Listener Password
The database listeners (one on each node) have their access restricted by privileged users only, e.g. root
or oracle. The listeners are not password protected.
5.2 Backups
5.2.1. Database Backups
See Section 4 for more detail.
5.2.2. Disk Backups
Most disks in the BladeFrame(s) are protected by either being mirrored or the disks will be replicated via
SRDF (EMC? DMXs only).
5.3 Partition Management
5.3.1 Introduction
This section does not detail specific functionality but is intended to provide an overview of how the use
of physical partitions works and to handle the partition creation failure. The partition management
describes in this section applied to both the BRDB and BRSS.
Note this section does NOT include how partitions are created and archived off though where
appropriate, reference is made to interactions.
5.3.2. Assumptions
It is assumed physical partitions exist for each partitioned table for the desired processing date.
5.3.3. Overview
The creation and removed physical partitions for each partitioned table is performed by start of day job;
i.e. BRDBCO01 and BRSSC001.
The operation of the start of day process is defined in LLD.
5.3.3.1 Partition Metadata
The operation of the partition table is driven by the following metadata:
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 99 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
5.3.3.1.1_ <BRDB/BRSS>_PARTITION_CREATES
This table is used to record the creation, status change and removal of partitions by the Start of Day
housekeeping for support and audit purposes.
5.3.3.1.2 <BRDB/BRSS>_PARTITION_STATUS_HISTORY
This table is used to record the history of the created partition. The entry is inserted by Start of the Day
process ( <BRDB/BRSS>C0001 ).
5.3.3.1.3_ <BRDB/BRSS>_SUBPARTITION_RANGES
The entry in this table will contain the next partition (range value) that will be created by Start of Day
process (<BRDB/BRSS>C0001). The partition range value will be increment by 1 at the end of the
process.
5.3.3.1.4 <BRDB/BRSS>_PROCESS_CONTROL
This table holds process run information and in this case it contains the partition creation information for
each table. This table is used for re-run of the Start of the day process for the failure partition.
5.3.4 Troubleshooting
The Start of Day (BRDBC0001/BRSSC001) process creates physical partitions for the next day,
therefore this process would have to fail twice in succession and not have been corrected in order for the
partitions to be missing for the current day. This most likely occurs due to insufficient space available in
the corresponding tablespace for which an Operational Exception would be generated. The process can
be restarted after rectifying the cause of failure.
Note that it is possible due to the unavoidable implicit database commit performed when
adding/dropping table partitions that, in some esoteric failure scenarios, the partition metadata will be out
of sync with the actual partitions. In this situation, re-running the SOD process will potentially fail.
In this scenario it will be necessary to confirm whether the metadata/partitions are inconsistent by
running a script provided by development (see further sections).
If the partitions/metadata is inconsistent it will be necessary to manipulate either to remedy the situation.
Given that the remedial activity will be dependent on a number of variables including whether any data
has been written to the new partitions etc, a call should be raised with 4" line support.
In some situations, typically in test, it is desirable to run BRDBC001/BRSSC001 more than once in a
calendar day. The default (build) value of the PROCESS_DAY_MULTIPLE_RUNS_YN flag in the
<BRDB/BRSS>_PROCESSES table for the <BRDB/BRSS>C001 process is ‘N’ so would prevent this.
Therefore the PROCESS_DAY_MULTIPLE_RUNS_YN flag should be changed to ‘Y’ to allow this if
required.
WARNING - This should only be done in Live at the guidance of development.
The following is a checklist in the event the <BRDB/BRSS>C001 job fails (to be done before re-running
the job): -
i. Check the entry in <BRDB/BRSS>_OPERATIONAL_EXCEPTONS anid this will show the
error(s) that cause the job to failure.
ii. Check the ‘parameter value for ‘BRDB SYSTEM DATE’ from
‘<BRDB/BRSS>_SYSTEM_PARAMETERS table. It should set to (N — 1 ) where N is current
system date.
iii. Check the column ‘SYSTEM_DATE’,,START_DATE’ and ‘END_DATE’ in the
<BRDB/BRSS>_PROCESS_CONTROL table. This table is used to control the process for each
Table-Group and table affected.
SYSTEM_DATE should equal to the ‘<BRDB/BRSS> SYSTEM DATE’ from the
<BRDB/BRSS>_SYSTEM_PARAMETER table
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 100 of 183,
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
END_DATE should have the NULL value for the failure partition table.
iv. Check the ‘RANGE_VALUE’ from the <BRDB/BRSS>_SUBPARTITION_RANGES table. This
value should equal to ‘<BRDB/BRSS> SYSTEM DATE’ + 2 in the format of ‘'YYYYMMDD’
v. Check the table <BRDB/BRSS>_PARTITION_CREATES and
<BRDB/BRSS>_PARITION_STATUS_HISTORY. The failure partition_range_value for the
partition table must not exist in the above tables.
vi. Check the value of the PROCESS_DAY_MULTIPLE_RUNS_YN flag in the
<BRDB/BRSS>_PROCESSES table for the <BRDB/BRSS>C001 process is ‘N’.
There is another option to fix a single partition by passing the parameters to the Start of Day; i.e.
<BRDB/BRSS>C001 [<Table-Group> <Table-Name> <Partition-Date (YYYYMMDD)>]
<SYSTEM_DATE (YYYYMMDD) >
Where SYSTEM_DATE is optional when exist and this value will set in ‘<BRDB/BRSS> SYSTEM
DATE’.
5.3.4.1. Determining Exception Information
As it is entirely possible for SOD (<BRDBIBRSS>C001) to fail during the normal day-to-day overnight
run, the following query will help in diagnosing problems and give greater detail as to the reason(s) for
failure. This query will show all exceptions for the last 24 hours, the last of which will be displayed first: -
set lines 200 pages 90
col exception_detail FOR a70
col exception object FOR a20
col process_name FOR a20
SELECT exception_timestamp, exception_detail, exception_object, process_name
FROM brdb operational_exceptions
WHERE exception timestamp >= SYSDATE - 1
ORDER BY exception_timestamp DESC;
5.3.4.2 Useful Queries
The below scripts reconcile differences between physical partitions and partition metadata maintained by
the BRDB/BRSS application.
These scripts should not be run unless directed by Development support staff.
Updates status for records in <BRDB/BRSS>_PARTITION_CREATES table to 'ARCH' where the Status
is set to 'DEL' and the partition exists in the database: -
UPDATE <brdb/brss>_partition_creates bpc
SET bpe.status = 'ARCH'
WHERE bpe.status = 'DEL'
AND EXISTS (SELECT 'x'
FROM all_tab partitions atp,
<brdb/brss>_partitioned_tables bpt
WHERE atp.table_ owner = 'OPS$<BRDB/BRSS>’
AND atp. table_name = bpc.table_name
AND atp.table name = bpt.table name
AND atp.partition_name = bpt.partition_root_name II '_'
II bpc.partition_range_value) ;
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 101 of 183,
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
Updates status for records in <BRDB/BRSS>_PARTITION_STATUS_HISTORY table to 'ARCH' where
the Status is set to 'DEL' and the partition exists in the database: -
UPDATE <brdb/brss>_partition_status_history bpsh
SET bpsh.status = 'ARCH'
WHERE bpsh.status = 'DEL'
AND EXISTS (SELECT 'x!
FROM all_tab partitions atp
WHERE atp.table owner = 'OPS$<BRDB/BRSS>'
AND = atp.table_name = bpsh.table_name
AND — atp.partition_name = bpsh.partition_name) ;
Updates mismatched records in <BRDB/BRSS>_PARTITION_CREATES table to 'DEL': -
UPDATE <brdb/brss>_partition_creates bpc
SET bpe.status = 'DEL'
WHERE bpe.status != 'DEL'
AND NOT EXISTS (SELECT 'x!
FROM all_tab_partitions atp,
<brdb/brss>_partitioned_tables bpt
WHERE atp.table owner = 'OPS$<BRDB/BRSS>'
AND atp.table_name = bpc.table_name
AND = atp.table_name = bpt.table_name
AND —atp.partition_name = bpt.partition_root_name II
'_' II bpe.partition_range_value);
Updates mismatched records in <BRDB/BRSS>_PARTITION_STATUS_HISTORY table to 'DEL'
UPDATE <brdb/brss>_partition_status_history bpsh
SET bpsh.status = 'DEL'
WHERE bpsh.status != 'DEL'
AND NOT EXISTS (SELECT 'x!
FROM all_tab partitions atp
WHERE atp.table_owner = 'OPS$<BRDB/BRSS>'
AND atp.table_name = bpsh.table_name
AND atp.partition_name = bpsh.partition_name)
AND create_date = (SELECT MAX (bpshl.create_date)
FROM <brdb/brss>_partition_status_history bpsh1
WHERE = bpsh1.table_name = bpsh.table_name
AND bpshl.partition_name = bpsh.partition_name) ;
Inserts missing records into <BRDB/BRSS>_PARTITION_CREATES table: -
INSERT INTO <brdb/brss>_partition_creates
(table_name,
partition_range_value,
status,
status_date)
SELECT atp.table_name,
substr(atp.partition_name,
LENGTH (npt .partition_root_name) + 2) partition_range_value,
'NEW', _ _ _ _
SYSDATE
FROM all_tab partitions atp,
<brdb/brss>_partitioned_tables bpt
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 102 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
WHERE atp.table_ owner = 'OPS$<BRDB/BRSS>'
AND = atp. table_name = bpt.table_name
AND NOT EXISTS (SELECT 'x'
FROM <brdb/brss>_partition_creates bpc
WHERE bpc.table_name = atp.table_name
AND bpc.partition_range_value =
SUBSTR(atp.partition_name,
~ LENGTH (bpt .partition_root_name) + 2));
Inserts missing records into <BRDB/BRSS>_PARTITION_STATUS_HISTORY table: -
INSERT INTO <brdb/brss>_partition_status_history (
table_name, partition_name,
create_date, status, sql_statement)
SELECT atp.table_name,
atp.partition_name,
SYSDATE,
"NEW",
"METADATA CORRECTION UTILITY FROM SUPPORT GUIDE!
FROM all_tab_partitions atp,
<brdb/brss>_partitioned_tables bpt
WHERE atp.table_owner = 'OPS$<BRDB/BRSS>'
AND = atp.table_name = bpt.table_name
AND NOT EXISTS (SELECT ‘x!
FROM <brdb/brss>_partition_status_history bpsh
WHERE bpsh.table_name - atp.table_name
AND bpsh.partition_name = atp.partition_name) ;
Check the partition that will .be created when BRDBC001/BRSSCO01 next run: -
SELECT table_name,
range_value
FROM <brdb/brss>_subpartition_ranges;
Check the latest partition created in the system: -
SELECT table_name,
max (partition_range_value) ,
FROM <brdb/brss>_partition_creates
GROUP BY table_name
ORDER BY table_name;
“pt_clean.sh” shell script can be used to rebuilt the meta partition tables (<brdb/brss>_partition_creates,
<brdb/brss>_subpartition_ranges and <brdb/brss>_partition_status_history ) from the database. This
shell script can be found in /app_sw/brdb/build/schema or /app_sw/brss/build/schema.
NB. This script will set status to ‘ARCH’ in the <brdb/brss>_partition_creates and all the partitions will
be deleted when the <BRDB/BRSS>C0001 next run.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 103 of 183,
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
5.4 Standby Database
5.4.1 Introduction
The build of and theory surrounding the BRDB Standby database (SBRDB) is detailed extensively in the
Standby Database Low Level Design [DES/APP/LLD/0152].
5.4.2. Assumptions
The Primary Database BRDB will be running on a 4-node cluster and the Standby Database on a 1-node
cluster configuration.
The Data Guard Configuration has been successfully built and running without errors.
5.4.3. Troubleshooting
The very first thing one should consider when troubleshooting is to consider the status of the
architectural components surrounding the solution, e.g. the network, the SAN, the BladeFrame, etc. (see
Section 5.4.3.1)
Oracle has a number of processes on both the Primary database and the Standby database monitoring
the sending, the transportation and the receiving of replicated redo from source to the destination.
It is important to note that the Data Guard Broker is key to the monitoring of the solution without which,
the seamless failover to Standby from Primary would not be possible nor would the trouble free
monitoring through Grid Control be possible.
5.4.3.1. Checklist
Is the database in recovery mode or is it down (all nodes)?
Is there enough storage space, e.g. check +SBRDB_FLASH, /archredo? Do all the file systems have
sufficient free space?
Is the network up?
Have you checked the Data Guard Monitor status, e.g. dgmgr1 show configuration? Is it
showing succESs (see Section 5.4.3.3)?
Have you checked the Data Guard logs on _ Standby and Primary, e.g.,
/u01/admin/SBRDB/bdump/drcSBRDB1.1log?
5.4.3.2 Useful Queries
This query will help identify Data Guard problems (On BRDB or SBRDB).
‘T lines 100
fe}
D-
This query will help with determining if any Standby Logs are not in use when they should be (On
SBRDB). It does not matter what group the standby logs belong to, but one should see 1 log for every
primary instance, e.g. 1, 2, 3 and 4 in LIVE.
T lines 1
MAT=' DD-MON
124:MI:SS
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 104 of 183,
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
@
tatus <> 'UNA
ndby_log WHE
5.4.3.3 Useful Tools
Data Guard Monitor is very important for monitoring the status of the Data Guard Configuration and is
not possible without the Data Guard Broker. The broker is started automatically — at instance startup -
by setting the database initialisation parameter dg_broker_start to TRUE. The broker is in essence
the DMON process and writes information to a log called
/a01/admin/[SIB]BRDB/bdump/dre[SI]BRDB[1-4] .1og in which all status and error information
can be monitored/viewed.
The Data Guard Monitor Command-line Utility or DGMGRL can be used to get useful feedback from the
configuration, e.g. ...
$> dgmgrl
DGMGRL for Linux: Version 10.2.0.4.0 - 64bit Production
Copyright (c) 2000, 2005, Oracle. All rights reserved.
Welcome to DGMGRL, type "help" for information.
DGMGRL> connect /
Connected.
DGMGRL> show configuration
Configuration
Name: BRDB_DATAGUARD_CFG
Enabled: YES
Protection Mode: MaxPerformance
Fast-Start Failover: DISABLED
Databases:
BRDB - Primary database
SBRDB - Physical standby database
Current status for "BRDB_DATAGUARD_CFG":
SUCCESS
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 105 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
5.5 Oracle Streams
5.5.1 Introduction
Streams configuration and activation for BRDB and BRSS are detailed extensively in the BRDB High
Level Design [DES/APP/HLD/0020], BRSS High Level Design [DES/APP/HLD/0023] and Low Level
Design [DES/APP/LLD/0151].
5.5.2 Assumptions
A single-source replication environment is configured and has the following characteristics:
e One Capture process named BRDB_CAPTURE located in BRDB node 1
* One Propagation process named BRDB_PROPG located in BRDB node 1.
« One Apply process named BRSS_APPY located in BRSS node 1
5.5.3 Overview
This section only cover the diagnostics are required by the support persons when troubleshooting the
Streams processes.
5.5.4 Troubleshooting
The following is a list of tables and views that are useful, in troubleshooting Streams issues. This is
basically “reference” information (more detailed information can be found in the Oracle Streams
administration guide):
Capture Process
dba_capture: basic status, error info
v$streams_capture: detailed current status info
dba_capture_parameters: configuration information
Propagate Process
dba_propagation: basic status, error info
v$propagation_sender. detailed current status
Apply Process
dba_apply: basic status, error info
v$streams_apply_reader. status of the current apply reader
v$streams_apply_server. status of current apply server(s)
v$streams_apply_coordinator. overall status, latency info
dba_apply_progress
dba_apply_parameters: configuration information
“Miscellaneous” Tables and Views
v$buffered_queues: view that displays the current and cumulative number of messages enqueued and
spilled, for each buffered queue.
sys.streams$_apply_spill_msgs_part: table that the apply process uses, to “spill” messages from large
transactions to disk.
system.logmnr_restart_ckpt$: table that holds capture process “checkpoint” information.
Whenever there is an error or performance problems present themselves, the first place to check will be
to note any Grid Control errors, then the alert_<SID>.log and finally the relevant database instance trace
files. Messages about each capture process, propagation, and apply process are recorded in trace files
for the database in which the processes are running.
« Capture - BRDB1
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 106 of 183,
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
alert files = /u01/admin/BRDB/bdump/alert_BRDB1.log
trace files = /u01/admin/BRDB/bdump/brdb1_cnnn_iiiii.tre
e Propagation - BRDB1
alert files = /u01/admin/BRDB/bdump/alert.log
* Apply - BRSS1
alert files = /u01/admin/BRSS/bdump/alert.log
trace files = /u01/admin/BRSS/bdump/brdb1_pnnn_iiiii.tre
NOTE
nnnn: refers to the Oracle process.
iii: refers to the UNIX process number.
€.g. brdb1_cjq0_14883.tre
5.5.4.1. Troubleshooting Capture Problems
The Oracle Streams Capture process resides within the Branch Database. It captures all data
(DML/DDL) changes to objects own by OPS$BRDB. The Capture process may stop capturing changes
or start running very slowly on an intermittent basis. Some of the useful methods describes in this
section can use to diagnose the problem and resolve them: -
Check capture process status:
The Capture Process captures changes only when it is ENABLED. One can check whether the process
is enabled, disabled, or aborted by querying the DBA_CAPTURE data dictionary view:
SELECT status FROM dba_capture WHERE capture _name = ‘BRDB_ CAPTURE’ ;
If the capture process is disabled, then try restarting it.
If the capture is aborted, then it needs to correct an error before restarting it. The following query shows
when the capture process aborted and the error that caused it to abort:
SELECT status_change_time, error_message
FROM dba_capture
WHERE status = ‘ABORTED’ AND capture_name = ‘BRDB_CAPTURE’ ;
Check Capture current status: -
The state of a capture process describes what the capture process is doing currently. One can view the
state of a capture process by querying the STATE column in the V$STREAMS_CAPTURE dynamic
performance view.
SELECT state
FROM v$streams_capture
WHERE capture_name = 'BRDB_CAPTURE';
The following capture process states are possible: -
INITIALIZING: Starting up.
CAPTURING CHANGES: Scanning the redo log for changes that evaluate to TRUE against the
capture process rule sets.
EVALUATING RULE: Evaluating a change against a capture process rule set.
CREATING LCR: Converting a change into an LCR.
ENQUEUING MESSAGE: Enqueuing an LCR that satisfies the capture process rule sets into the
capture process queue.
SHUTTING DOWN: Stopping.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 107 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
WAITING FOR DICTIONARY REDO: Waiting for redo log files containing the dictionary build
related to the first SCN to be added to the capture process session. A capture process cannot
begin to scan the redo log files until all of the log files containing the dictionary build have been
added.
DICTONARY INITIALIZATION: Processing a dictionary build.
MINING (PROCESSED SCN = scn_value): Mining a dictionary build at the SCN scn_value.
LOAD (step X Of Y): Processing information from a dictionary build and currently at
step X in a process that involves Y steps, where X and Y are number.
PAUSED FOR FLOW CONTROL: Unable to enqueue LCRs either because of low memory or
because propagations and apply processes are consuming messages slower than the capture
process is creating them. This state indicates flow control that is used to reduce spilling of
captured messages when propagation or apply has fallen behind or is unavailable.
Common capture issues: -
1. ORA-01291: missing logfile.
A missing redo is possible when a logfile is dropped for any administrative reasons. The
v$logmnr_logs can be checked to determine the missing SCN range and add the relevant redo
log files
Query the REQUIRED_CHECKPOINT_SCN column in the DBA_CAPTURE to determine the
required checkpoint SCN for a captured. Then restore the redo log file that includes the required
checkpoint SCN and all subsequent redo log files.
2. Capture process loops on startup.
This may be a missing logfile which cannot be opened. All logs from the BRDB nodes (1[2/3)4 )
have to be present with respect to the required_checkpoint_scn.
3. Capture process is in “PAUSED FOR FLOW CONTROL’ or “ENQUEUING MESSAGE’ status.
e Check the source queue, as there is probably a large amount of LCRs being spilled to
disk.
* Check if the destination site is down.
eCheck the propagation and apply status’.
5.5.4.2 Troubleshooting Propagation Problems
The Oracle Streams propagation process resides within the Branch Database. It propagates the captured
events to the destination queue in the target database (BRSS). Some of the useful methods describes in
this section can use to diagnose the propagation problem and resolve them.
eCheck the database link is configured correctly and active.
e Check whether a propagation job status is enabled, disabled or aborted by querying the
DBA_PROPAGATION dictionary view:
select status from dba propagation where propation_name =
‘BRDB_PROPG'! ;
If status is ‘disabled’ or ‘aborted’ then check the error message:
select status,destination_dblink,error_date, error_message
from dba_propagation
where propation_name = ‘BRDB_PROPG'’ ;
Correct the error and restart the propagation process
begin
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 108 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
DBMS_PROPAGATION_ADM.START_PROPAGATION( 'BRDB_PROPG') ;
end;
e If the propagation job is enabled, but is not propagating messages, then try stopping and
restarting the propagation.
e Check propagation schedule is enabled and associated with a job queue process, any failures or
errors received, the date and time the propagation schedule will be started
select SCHEDULE_DISABLED, PROCESS NAME,
LAST_RUN_DATE ,NEXT_RUN_DATE,
TOTAL NUMBER, FAILURES,
LAST_ERROR_TIME, LAST_ERROR_MSG,
from dba_queue_schedules s, dba_propagation p
where s.DESTINATION = p.destination_dblink ;
SCHEDULE_DISABLE = ‘N’ Enable
SCHEDULE_DISABLE = ‘Y’ Disable
e Determine the number of messages sent and the number of messages that have been
acknowledged.
select queue_name,schedule_status,
high_water_mark, acknowledgement
from v$propagation_sender;
5.5.4.3 Troubleshooting Apply Problems
The Oracle Streams Apply process resides within the Branch Support Database. It dequeues all the
captured data and applies them to the OPS$BRDB schema. Some of the useful methods describes in
this section can use to diagnose the apply problem and resolve them.
Check apply process status:
An apply process applies changes only when it is enabled. Query the STATUS column in DBA_LAPPLY
to determine the state of the apply process: -
SELECT status
FROM dba_apply
WHERE apply name = ‘BRSS_APPY';
The possible values are ENABLED, DISABLED and ABORTED.
If the apply process is disabled, then try restarting it: -
DBMS_APPLY_ADM.START_APPLY( apply name => 'BRSS_APPY') ;
If the apply process is aborted, then correct an error before restart the apply process. The following
query shows when the apply process and the error that caused it to abort: -
SELECT status_change_time,error_ message
FROM dba_apply ~ ~
WHERE status = ‘ABORTED’
AND apply_name = ‘BRSS_APPY';
If the apply process is enabled, but changes are not applied: -
Check that the apply process queue is receiving the messages to be applied using v$buffered_queues: -
SELECT queue_id, (num_msgs - spill_msgs) mem_msgs,
spill_msgs
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 109 of 183,
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
@
FROM vS$buffered_queues
WHERE queue _name = ‘BRSS_APPLQ’ ;
Or using the v$streams_apply_coordinator view: -
SELECT total_received,
total_applied,
total_errors, (total_assigned - total_rollbacks + total_applied))
being_applied
FROM v$streams_apply coordinator
WHERE apply name = 'BRSS_APPY';
Check the Error Queue
When an apply process cannot apply a message, it moves the event and all the other events in the
same transaction into the error queue.
Query DBA_APPLY_ERROR to determine if there are errors in the error queue.
SELECT local_transaction_id,
meesage_number,error_message
FROM §_ dba_apply_error
WHERE apply name = ‘BRSS_APPY’ ;
If there are apply errors, then you can either try to re-execute the transactions that encountered the
errors, or you can delete the transactions. If you want to re-execute a transaction that encountered an
error, then first correct the condition that caused the transaction to raise an error.
The execute_error procedure re-executes the specified error transaction.
Syntax:
DBMS_APPLY_ADM.EXECUTE_ERROR(local_transaction_id IN VARCHAR2,
execute_as_user IN BOOLEAN DEFAULT false) ;
The error transaction can be dropped from the from the error queue by invoking the delete_error
procedure. The delete_error procedure deletes the specified error transaction.
Syntax:
DBMS_APPLY_ADM.DELETE ERROR(local_transaction_id IN VARCHAR2) ;
5.5.4.4 Working with Apply Errors
The Oracle Streams Apply process will store all rows of every transaction that fails to apply. These
failed transactions with their associated errors are available to query from DBA_APPLY_ERROR.
Errorred transactions can be applied once the problem that caused the failure has been rectified (if
possible). The following is a little help in finding them and extracting them to help with their resolution: -
set lines 140
set pages 45
SELECT TO_CHAR(error_creation_time,'YYYY/MM/DD') created,
error_number,
COUNT (1)
FROM dba_apply error
GROUP BY error number, TO_CHAR(error_creation_time, 'Y¥YY/MM/DD')
ORDER BY 1, 2;
CREATED ERROR_NUMBER COUNT (1)
2009/06/23 1403 6
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN Ref: DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 110 of 183
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE @
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
2009/06/24 1403 6
Once you have the list of errors, choose whichever error you may be interested in and use it in the
following query, which will provide you with the error_message and the local_transaction_id: -
SET lines 140
SET pages 80
col error_message FOR a70
SELECT local_transaction_id, error_message
FROM dba apply error
WHERE error_number = 1403;
LOCAL_TRANSACTION_ID ERROR_MESSAGE
23.28.244
3.33.140771
17.30.370
Using the local_transaction_id you can perform the following procedure execution of
PKG_BRSS_STREAMS.PR_PRINT_TRANSACTION, which will show every record affected by the error
in question, information such as the table, the columns affected the before and after values, etc. are
shown: -
SET SERVEROUTPUT ON
BEGIN
stradmin.pkg_brss_streams.pr_print_transaction('3.33.140771');
END;
/
Local Transaction ID: 3.33.140771
Source Database: BRDB
-Error in Message: 1
-Error Number: 1403
-Message Text: ORA-01403: no data found
--message: 1
type name: SYS.LCR$_ROW_RECORD.
source database: BRDB
owner: OPS$BRDB
object: BRDB_SYSTEM PARAMETERS
is tag null: ¥
command_type: UPDATE
old(1) : PARAMETER NAME
BRDB_PDEL_TO LFS _STOP_YN
old(2): VERSTON_NUMBER
1
o1d(3): UPDATE_TIMESTAMP
typename is SYS.TIMESTAMP
old(4): PARAMETER_TEXT
N
new(1): UPDATE_TIMESTAMP
typename is SYS.TIMESTAMP
new(2): PARAMETER TEXT
Y
transaction name:
~-message: 2
type name: SYS.LCR$_ROW_RECORD
source database: BRDB
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 111 of 183,
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE @
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
owner: OPS$BRDB
object: BRDB_PROCESS_CONTROL
is tag null: ¥
command_type: UPDATE
old (1): INSTANCE_NAME
BRDB_PDEL_TO LFS _STOP_YN
o1d(2): RUN_NUMBER
2
old(3): PROCESS_NAME
BRDBXO11
old(4): SYSTEM DATE
23-JUN-09
old (5): END_DATE
new(1): END_DATE
23-JUN-09
transaction name:
--message: 3
type name: SYS.LCR$_ROW_RECORD
source database: BRDB
owner: OPS$BRDB
object: BRDB_PROCESS_AUDIT
is tag null: ¥
command_type: INSERT
new (1): PROCESS_SEQUENCE_NUMBER
432918
new(2): INSTANCE_NAME
BRDB_PDEL_TO_LFS_STOP_YN
new (3): PROCESS_NAME
BRDBXO11
new (4): BLOCK_CHANGES
new(5): BLOCK_GETS
new (6): CONSISTENT_CHANGES
new(7): CONSISTENT_GETS
new(8): ORACLE_SERTAL#
new (9): ORACLE_SID
new(10): RUN_NUMBER
new(11): PHYSICAL READ
new(12): UNIX_PID
new(13): INSERT_DATE
23-JUN-09
new(14): SYSTEM DATE
23-JUN-09
new(15): START_FINISH_INDICATOR
F
new (16): USER_NAME
OPS$BRDBBLV2
transaction name:
Once again, the errorred transaction can be applied once the cause of the failed apply has been
identified.
exec dbms_apply_adm.execute_error (’3.33.140771');
One could also print all the current errors by executing PKG_BRSS_STREAMS.PR_PRINT_ERRORS.
This procedure prints @ll errorred transactions regardless of transaction_id of error_number.
BEGIN
stradmin.pkg_brss_streams.print_all_errors;
END;
/
© Copyright Fujitsu Ltd 2017 FUJITSU RESTRICTED (COMMERCIAL IN _ Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 112 of 183,
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE ®
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN Ref: DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 113 of 183
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE @
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
5.5.4.5 Useful Queries
i. The following query displays the current status of the capture process
set lines 100
column capture_name heading
process name heading
sid heading nIID' format 999999
column serial# heading nISerialINumber' format 9999999
column state heading 'State' format A27
column total_messages_captured heading 'RedoIEntriesIEvaluatedIIn Detail’
format 9999999
column total_messages_enqueued heading ‘Tot
Name' format A12
eIProcessINumber' format A7
LCRsIEnqueued' format 999999
SELECT
;capture_name,
substr (s.program, instr (s.program, ' (
sid,
serial#,
state,
total_messages_captured,
total_messages_enqueued
$streams_capture c, v$session s
sid = s.sid
serial# = s.serial#;
+1,4) process_name,
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 114 of 183
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE @
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
ji. Minimum Archive Log Necessary to Restart Capture
set lines 300
set pages 9999
set serveroutput on
DECLARE
hScn number
1Scn number
sScn number;
ascn number;
alog varchar2 (1000);
BEGIN
select min(start_scn), min(applied_scn) i
from dba_capture
where capture_name = ‘BRDB_CAPTURE’;
to sScn, ascn
DBMS_OUTPUT.ENABLE (2000) ;
for cr in (select distinct (a.ckpt_scn)
from system. logm restart_ckpt$ a
where a.ckpt_scn <= ascn and a.valid = 1
and exists (select * from system.logmnr_log$ 1
where a.ckpt_scn between 1.firs
change# and
l.next nge#)
order by a.ckpt_scn desc)
loop
if (hScn = 0) then
hScn := cr.ckpt_sen;
else -
sen := cr.ckpt_scn;
exit;
end if;
end loop;
if 1Scn = 0 then
1sen := sScn;
end if;
dbms_output.put_line('Capture will restart from SCN ' II 1Scn II' in the
following file:');
for cr in (select name, first_time
from DBA_REGISTERED_ARCHIVED_LOG
where 1Scn between first_scn and next_scn order by thread#)
loop
dbms_output.put_line(cr.nameII' ("I Icr.first_timeII')');
end loop;
end;
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref: DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 115 of 183
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE @
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
Display Capture Status Error Message
set serveroutput on size 950000
set verify off
set feedback off
set lines 180
set pages 9999
prompt +
prompt IDisplay Capture Status Error Message I
prompt +=
column capture_name heading 'CaptureIProcessIName' format Al0
column status change time heading ‘Abort Time!
column heading 'Error Number' format 99999999
heading ‘Error Message' format A40 wrap
SELECT capture_name, status_change_time , error_number, error_message
FROM dba_captur
WHERE status="ABORTED'
AND capture_name = 'BRDB_CAPTURE' ;
iv. This query will help to Display Information Reader ¢r for Each Apply Pre
column apply_name heading 'Apply ProcessIName' format A15
column apply_captured heading 'Dequeues CapturedIMessages?' format
Al7
column process_name heading 'ProcessIName' format A7
column state heading 'State' format A17
column total_messages_dequeued heading ‘Total MessagesIDequeued' format
99999999
SELECT r.apply_name,
ap.apply_captured,
substr(s.program, instr (s.program, ' (
r.state,
r. total_messages_dequeued
FROM v$streams_apply reader r, v$session s, dba_apply ap
WHERE r.sid = s.sid
AND r.serial# = s.serial#
AND r.apply_name = ap.apply_name;
)+1,4) process_name,
v. The following query displays the information about each transaction currently being applied for which
the apply process has spilled messages:
column apply_name heading 'Apply Name' format A20
column ‘Transaction ID' heading ‘Transaction ID' format A15
column first_scn heading 'First SCN' format 99999999
column message_count heading 'Message Count' format 99999999
SELECT apply_name,
xidusn II'.'/I
xidslt [I'.'/1
xidsqn "Transaction ID",
first_scn,
message_count
FROM dba_apply_spill_txn;
vi. The following query displays information about the transactions received, applied, and being applied
by the apply proces:
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref: DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 116 of 183
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE ®
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
apply_name heading 'Apply Process Name' format A25
in total_received heading 'TotalITransIReceived' format 99999999
"TotalITransIApplied' format 9999 9
heading 'TotalIApplyIErrors' format 9999
heading 'TotalITr BeingIApplied' format 9 99
heading 'TotalITransIIgnored' format 9 99
ned - (total_rollbacks + total_applied)) being_applied,
FROM v$streams_apply_coordinator;
5.6 SCC Transaction Correction Tools
5.6.1 BRDBX015 — Transaction Correction Tool
The transaction correction tool module BRDBX015.sh will allow Support Service Centre to correct
transactions by inserting balancing records to transactional/accounting/stock tables in the BRDB system.
It takes two parameters, the file name containing the insert statement and the branch code. If the
process completes successfully, the insert statement is audited in
BRDB_TXN_CORR_TOOL_JOURNAL. If there is an error, the entire transaction is rolled back and
nothing is written to the database. The module uses process audit and does not use process control
(allows multiple runs).
The file containing the insert statement must be copied to the /app/brdb/trans/support/brdbx015/input
directory on the Linux box, and the module run from the directory /app/brdb/trans/support/brdbx015. If
the module completes successfully, the file will be moved to /app/brdb/trans/support/brdbx015/output. A
log file will be written to /bvnw01/brdb/brdbx015/log using the file name template
<transaction_file>_<CCYYMMDDHHMISS>.log
This module can be run only by the Linux user “supporttooluser” which has only the necessary privileges
required to run the module. The module will call a package procedure which runs under Oracle user
‘OPS$SUPPORTTOOLUSER’ which allows inserts only into selected tables. This is a powerful tool
which has inherent risks and care must be taken when constructing the insert statement. The SQL
statement must begin with an ‘INSERT INTO’ clause and can only insert one row into the corresponding
transactional table. This is validated in the tool and will raise an error if the condition is not met.
The format of the SQL statement should be based on the templates supplied in Appendix C.
The following tables have been granted insert privileges to OPS$SUPPORTTOOLUSER:-
BRDB_RX_APS_TRANSACTIONS
BRDB_RX_BUREAU_TRANSACTIONS.
BRDB_RX_CUT_OFF_SUMMARIES
BRDB_RX_DCS_TRANSACTIONS
BRDB_RX_EPOSS_EVENTS
BRDB_RX_EPOSS_TRANSACTIONS
BRDB_RX_NWB_TRANSACTIONS
BRDB_RX_REP_EVENT_DATA
BRDB_RX_REP_SESSION_DATA
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 117 of 183
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE @
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
5.6.1.1 Parameters
The tool must be supplied with 2 parameters:
e Transaction File Name (not including path) e.g. t1.sql
e Branch Code (numeric) e.g. 8009
5.6.1.2 Scheduling
This task is scheduled on an ad hoc basis, as and when transaction corrections need to be applied.
5.6.1.3 Sample output
This is an example of the output written to standard output and the log file when the module is
successful:
Wed 21-Oct-2009 14:35:08 Starting BRDBX015.sh
Wed 2 009 14 BRDBXO15. Debug message level for this program is 0
Wed 14: BRDBXO1
Wed 2 14: BRDBX015.sh: In check parameters ()
Wed 14 BRDBXO15 ORACLE_HOME = /u01/app/oracle/product/10.2.0/db_1
Wed 2 1a: } BRDBXO15.sh: ORACLE SID = BRDBL
Wed 2 14: BRDBXO015.sh: TRANS FILE = ./input/t1.sql
Wed 2
Wed
Wed
BRDBXO15.<)
BRDBXO1
BRDBXO15
BRAI
CODE = 9004
Script <BRDBX015.sh> started on Wed Oct 21 14:35:08
2009
Wed 21-0¢ 14: BRDBXO15.sh:
Wed 21-0ct 14: .150 Started PKG BRDB_TXN CORRECTION.LOAD DATA
14:35:08.750 Version information: $Logfile: /HNG-
-90]/BRDB/Database and Schema Build/PLSOL Objects/pkg_brdb_txn_correction_body.sql
009 14:35:08.750 This Feed does not use process control
2009 14:35:08.995 Number of rows inserted = 1
009 14:35:09.011 Completed PKG_BRDB_TXN CORRECTION.LOAD_DATA
sfully completed.
Wed 2
Wed 2
Wed 2
Wed
Wed 21-Oct-2009
Wed 21-Oct-2009
Wed 21-Oct-2009
5:08 Return code is 0
8 BRDBXO15.s!
08 BRDBXO15.s1
108 exit 0
5:08 BRDBX015
208 BRDBXO15
5:08
BRDBXO15.sh ran successfully
Finished on Wed Oct 21 14:35:09 BST 2009
5.6.1.4 Diagnostics
The module may fail for one of the following reasons
e May not be logged in as the SSC user.
e Transaction file containing SQL statement is not present in
/app/brdb/trans/support/brdbx015/input directory.
e The Oracle directory name ‘BRDBX015_DIR’ must be mapped to physical directory
/app/brdb/trans/support/brdbx015/input. Connect to the Linux box as user
‘supporttooluser’. Login to SQL and run the following command:-
SELECT owner, substr(directory_path,1,40) directory_path
FROM all_directories
WHERE directory_name = 'BRDBX015_DIR'
/
The above statement must return 1 row.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 118 of 183,
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
e SQL statement does not begin with ‘INSERT INTO’ statement or contains more than one insert
statement.
e The following in-line select statement has not been added to the end of the insert statement
(SELECT fhom.branch_accounting_code,
fhom.fad_hash,
tetc.current_jsn
FROM _ ops$brdb.brdb_fad_hash_outlet_mapping fhom,
ops$brdb.brdb_txn_corr_tool_ctl tte
WHERE fhom.branch_accounting_code = :bind_branch_code
AND _ tctc.branch_accounting_code = fhom.branch_accounting_code) A;
e Check the insert statement for any syntax errors.
5.6.2 BRDB Clear Stock Unit Lock (clear_su_lock.sh)
This tool allows members of the SSC group to unlock stock units for any given branch accounting code,
locking user and stock unit. Any attempt to run the tool will be audited as well as the actual changes
made and running user. The SSC user must have been granted the SSC role within the BRDB database
prior to running this tool.
Validation and processing occurs in an Oracle package
(OPS$SUPPORTTOOLUSER.PKG_BRDB_CLEAR_SU_LOCK) while the package is initially called by a
shell script (clear_su_lock.sh) on the BRDB server.
The script is located in /app/brdb/trans/support/brdbx015/clear_su_lock.sh
See DEV/APP/LLD/0202 for more information
5.6.2.1 Parameters
The tool must be supplied with 3 switches, each with a parameter:
Parameter Parameter Name Datatype Example __ Valid Inpu
-b Branch Accounting Code Number 999999 1-999999
-u Lock Holder Username STRING USR123 A[1-15 chr]
Ss ‘Stock Unit STRING DEF 0O-zz
5.6.2.2 Executing
-/clear_su_lock.sh -b <BRANCH_CODE> -u <LOCK_USER> -s <STOCK_UNIT>
5.6.2.3 Scheduling
This task is scheduled on an ad hoc basis, as and when stock units need to be unlocked.
5.6.2.4 Audit Records/Logging
Start and finish records are inserted into OPS$BRDB.BRDB_PROCESS_AUDIT with a process_name of
‘BRDB_CLEAR_SU_LOCK’.
Each update to OPS$BRDB.BRDB_BRANCH_STOCK_UNITS is audited in
OPS$BRDB.BRDB_TXN_CORR_TOOL_JOURNAL.
Any exceptions will be logged to OPS$BRDB.BRDB_OPERATIONAL_EXCEPTIONS with an exception
code of ‘BRDB_SU_LOCK’ and process_name (package name) of ‘PKG_BRDB_CLEAR_SU_LOCK’.
The script verbosity level is controlled by BRDB system parameter
BRDB_CLEAR_SU_LOCK_DEBUG_LEVEL (parameter_number set to 1 initially), set
parameter_number to 2 in order to view the SQL update statement as well as the XML string.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 119 of 183
Fe)
FUJITSU
HOST BRANCH DATABASE SUPPORT GUIDE
FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
FUJ00088797
FUJ00088797
Log files from each run are stored in /app/brdb/trans/support/brdbx015/log.
5.6.2.5 Sample output
This is an example of the output written to standard output and the log file when the module is
successful:
01 Dec 14:54:10 writelock...:
01 Dec 14:54:10 writelock...:
01 Dec 14:54:10 writelock...:
01 Dec 14:54:10
O01 Dec 14:54:10 check_env.
01 Dec 14:54:10 check_env.
01 Dec 14:54:10
01 Dec 14:54:10 Main.. :
O01 Dec 14:54:10
01 Dec 14:54:10 view_gvar.
O01 Dec 14:54:10 view gvar.
01 Dec 14:54:10 view gvar.
01 Dec 14:54:10 view gvar.
01 Dec 14:54:10 view gvar...:
01 Dec 14:54:10 view gvar...
Starting
Lock file /tmp/clear_su_lock.run.lock created
Complete.
Starting
Complete.
Environment OK
Starting
WHOAML. .
PROGNAME,
SCRIPT. .
THISDIR.
LOG_FILE.....5
gseem01
clear_su_lock.sh
clear_su_lock
/app/brdb/trans/support/brdbx015
/app/brdb/trans/support /brdbx015/log/clear_su_lock_20091201_145410.1log
01 Dec 14:54:10 view_gvar.
01 Dec 14:54:10 view gvar.
01 Dec 14:54:10 view gvar.
01 Dec 14:54:10 view gvar...:
01 Dec 14:54:10 view gvar...:
01 Dec 14:54:10 view _gvar...
01 Dec 14:54:10 view _qvar...:
01 Dec 14:54:10 view gvar...:
01 Dec 14:54:10 view gvar...:
01 Dec 0
01 Dec 0 unlock......:
01 Dec 0
Enabling ssc role
LOCK FILE...
TSTMB..
VERBOSE. :
ORACLE Sib...
BRANCH_cODs.
LOCK _USER....--.
STOCK_UNIT......000005
completes
Starting
Set DEBUG LEVEL to 1
/tmp/clea
20091201_145410
ON
BRDB
BRDBAL
2007
x
DEF
0.619 Starting pkg_brdb_clear_su_lock.update data
Starting pkg_brdb_clear_su_lock.process_audit
0.620 Completed pkg_brdb clear_su_lock.process_audit
INFO:
INFO:
Parameter p branch code = 2007
Parameter p rollover lock_user = X
INFO: Parameter p_stock_unit:
DEF
su_lock.run.lock
14:54:10,620 Starting pkg_brdb clear_su_lock.validate parameters
: Validating branch_accounting_code
Tue 01-Dec-2009 14:54:10.619
Tue 01-Dec-2009 14:5
Tue 01-Dec-2009 0.619
Tue 01-Dec-2009
Tue 01-Dec-2009 0.620
Tue 01-Dec-2009 0.620
Tue 01-Dec-2009 0.620
Tue 01-Dec-2009
Tue 01-Dec-2009 14:54:10,621
Tue 01-Dec-2009 14:54:10.623
OPS$BRDB.BRDB_BRANCH_INFO
Tue 01-Dec-2009 14:54:
OPS$BRDB.BRDB_TXN_CORR_TOOL CTL
0.623
Tue 01-Dec-2009 14:54:10.628
Tue 01-Dec-2009 14:54:10.628
Tue 01-Dec-2009 14:54:10.629
BRDB_BRANCH STOCK UNITS
Tue 01-Dec-2009 0.629
Tue 01-Dec-2009 0.629
Tue 01-Dec-2009 0.629
Tue 01-Dec-2009 0.629
Tue 01-Dec-2009 0.629
Tue 01-Dec-2009 0.630
Tue 01-Dec-2009 14:54:10,630
Tue 01-Dec-2009 14:54:10.630
Tue 01-Dec-2009 14:54:10.630
Tue 01-Dec-2009 14:54:10.630
Tue 01-Dec-2009 14:54:10.630
Tue 01-Dec-2009 14:54:10,630
Tue 01-Dec-2009 14:54:10.630
Tue 01-Dec-2009 14:54:10.630
Tue 01-Dec-2009 14:54:10.631
Tue 01-Dec-2009 14:5
01 Dec 14:54:
OL
Dec 14:54
INFO:
OK:
ol
ol
OK:
ol
OK:
Starting pkg_brdb_
Derived FAD_HASH for branch accounting code 2007 is:
table OPS$BRDB.BRDB_BRANC!
ol
Ok:
Completed pkg _brdb clear
Branch Accounting Code:
Branch Accounting Code:
2007
2007 exists in
is open and exists in
Stock unit DEF is locked for branch accounting code 2007
Stock unit DEF is locked by X
OPS$SUPPORTTOOLUSER is allowed to update
Input parameters validated successfully
Completed pkg_brdb_clear_su_lock.validate_parameters
Updated 1
‘lear_su_lock.reset_lock
le
su update data
Starting pkg_brdb_clear_su_lock.audit_update
INFO:
INFO:
INFO:
INFO:
OK:
Starting pkg_brdb_clear_su_lock.process_audit
Completed pkg_brdb clear_su_
0.631 Completed pkg_brdb clear_su_lock.update_data
UNIX USEI
CURRENT.
BRDB INSTANCE NAME:
ORACLE USER
JSN
BRDBAL
gseem01
SUPPORTTOOLUSER
48 for
96
I STOCK_UNITS
branch accounting code 2007
Inserted 1 row into OPS$BRDB.BRDB_TXN_CORR_TOOL JOURNAL
Completed pkg_brdb_clear_su_lock.audit_update
unlock......: Complete
lock.process_audit
UNCONTROLLED IF PRINTED
FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
Ref:
Version:
Date
Page No:
DES/APP/SPG/0001
3.0
21-Sep-11
120 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
01D
01
01
01
o1
o1
01
4e 2007
>ck unit DEF for branch ¢
:54:10 cleanup...
1:54:10 cleanup...
154210
:54:10 cleanup..... : Processing Complete
caning up ...
Lock file /tmp/clear_su_lock.run.lock freed.
5.6.2.6 Diagnostics
The module may fail for one of the following reasons
«The SSC user may not be logged in with their SSC unix login.
e The SSC user's Oracle login may not have been granted the SSC role.
e One or more of the parameters are invalid
5.6.3 BRDB Clear Rollover Lock (clear_ro_lock.sh)
This tool allows members of the SSC group to clear branch rollover locks for any given branch
accounting code and locking user. Any attempt to run the tool will be audited as well as the actual
changes made and running user.
Validation and processing occurs in an Oracle package
(OPS$SUPPORTTOOLUSER.PKG_BRDB_CLEAR_RO_LOCK) while the package is initially called by
a shell script (clear_ro_lock.sh) on the BRDB server.
The script is located in /app/brdb/trans/support/brdbx015/clear_ro_lock.sh
See DEV/APP/LLD/0203 for more information.
5.6.3.1 Parameters
The tool must be supplied with 2 switches, each with a parameter:
Parameter Parameter Name Script Variable Name Datatyps Valid Input
-b Branch Accounting Code BRANCH_CODE Number 1 -999999
-u Lock Holder Username LOCK_USER STRING A[1-15 chr]
5.6.3.2
5.6.3.3 Executing
«/clear_ro_lock.sh -b <BRANCH_CODE> -u <LOCK_USER>
5.6.3.4 Scheduling
This task is scheduled on an ad hoc basis, as and when branch rollover locks need to be unlocked.
5.6.3.5 Audit Records/Logging
Start and finish records are inserted into OPS$BRDB.BRDB_PROCESS_AUDIT with a process_name of
‘BRDB_CLEAR_RO_LOCK’.
Each update to OPS$BRDB.BRDB_BRANCH_INFO is audited in
OPS$BRDB.BRDB_TXN_CORR_TOOL_JOURNAL.
Any exceptions will be logged to OPS$BRDB.BRDB_OPERATIONAL_EXCEPTIONS with an exception
code of ‘BRDB_RO_LOCK'’ and process_name (package name) of ‘PKG_BRDB_CLEAR_RO_LOCK’.
The script verbosity level is controlled by BRDB system parameter
BRDB_CLEAR_RO_LOCK_DEBUG_LEVEL (parameter_number set to 11 __ initially), set
parameter_number to 2 in order to view the SQL update statement as well as the XML string.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 121 of 183,
Fe)
FUJITSU
HOST BRANCH DATABASE SUPPORT GUIDE
FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
FUJ00088797
FUJ00088797
Log files from each run are stored in /app/brdb/trans/support/brdbx015/log.
5.6.3.6 Sample output
This is an example of the output written to standard output and the log file when the module is
successful:
03 Dec 15:06:55 writelock...:
03 Dec 15:06:55 writelock...:
/app/brdb/trans/support/brdbx015/log/clear_ro_lock.run.lock created
03
03
03
03
03
03
03
03
03
03
03
03
03
/app/brdb/ trans /aupport /ezdbn015/log/elear ro.
Dec 15:06
Dec
Dec 15:06
Dec 15:06:
Dec 15:06
Dec 15:06:55
Dec 15:06
Dec
Dec
Dec
Dec 15:06:55
Dec 5
Dec 15:06:55
5
5
5
writelock...:
check_env...:
check_env...:
view gvar.
view_gvar.
view _gvar.
view gvar...:
view _gvar...
view gvar...:
03 Dec 15:06:55 view _qvar...: sees
/app/brdb/trans/support /brdbx015/1og/élear_ro_lock. run. lock
03
Dec 15:06:55
view_gvar...3
Starting
Lock file
complete.
Starting
Complete.
Environment OK
Starting
WHOAML. . gseem01
PROGNAME, clear_ro_lock.sh
SCRIPT. clear_ro_lock
THISDIR. . /app/brdb/trans/ support /brdbx015
LOG_FILE
lock_20091203_150655.1og
LOCK FILE...
TMP_FILE....++-5
/app/brab/trans/support/brdbx015/1og/clear__ ro_lock. tmp
03
03
03
03
03
03
03
03
03
03
Dec 15:06:
Dee 15:06:55
Dec
Dec
Dec
Dec
Dec
Dec
Dec 15:06:55
Dec 15:06:55
SeSaas
view gvar...:
view _gvar...:
view gvar...:
view _gvar...:
view _gvar...:
unlock......1
Enabling ssc role
Thu
Thu
Thu
Thu
Thu
Thu
Thu
Thu
Thu
OPS$BRDB.BRDB_BRANCH_INFO
Thu 03-Dec-2009 15:0
03-Dec-2009
03-Dec-2009
03-Dec-2009
03-Dec-2009
03-Dec-2009
03-Dec-2009
03-Dec-2009
03-Dec-2009
03-Dec-2009
5.541.
5.541
5.541
5.542.
15:06:55.542
5.542
15:06:55.542
5.542
5.546
5.547
TSTMP... ++ 20091203 150655
VERBOSE. +. ON
APP.....secssceecveees BRDB
ORACLE_SID...... BRDBAL
BRANCH_CODE....+..+-.. 2007
LOCK USER... es
Complete.
Starting
Set DEBUG LEVEL to 1
Starting BRDB_CLEAR_RO_LOCK.update_data
Starting BRDB_CLEAR_RO_LOCK.process_audit
Completed BRDB_CLEAR_RO_LOCK.process_audit
INFO: Parameter p_bac = 2007
INFO: Parameter p_rollover_lock_user = X
Starting BRDB_CLEAR_RO_LOCK.validate_parameters
INFO: Validating branch_accounting_code
oO
OK: Branch Accounting Code: 2007 exists in
OPS$BRDB.BRDB_TXN_CORR_TOOL_ CTI.
Thu
Thu
Thu
Thu
Thu
Thu
Thu
Thu
Thu
Thu
Thu
Thu
Thu
Thu
Thu
Thu
Thu
Thu
Thu
03
03-Dec-2009
03-Dec-2009
03-Dec-2009
03-Dec-2009
03-Dec-2009
03-Dec-2009
03-Dec-2009
03-D.
03-Dec-2009
03-Dec-2009
03-Dec-2009
03-Dec-2009
03-Dec-2009
03-Dec-2009
03-Dec-2009
15:06:55.545
15:06:55.549
03-Dec-2009 15
03-Dec-2009
03-Dec-2009
03-Dec-2009
Dec 15:06:55
ol
Branch Accounting Code 2007 is locked
Lock on Branch Ac
OPS$SUPPORTTOOLUS!
Input parameters validated successfully
Completed BRDB_CLEAR_RO_LOCK.validate parameters
Starting BRDB CLEAR RO_LOCK.reset_lock
OK: Derived FAD HASH for Branch Ac
oO
Completed BRDB_CLEAR_RO_LOCK.update_data
Starting BRDB_CLEAR_RO_LOCK.audit_update
INFO: BRDB INSTANCE NAME: BRDBAT
INFO: UNIX USER: gseem01
INFO: ORACLE USER: SUPPORTTOOLUSER
: Branch Accounting Code: 2007 is open and exists in
counting Code 2007 is locked by X
R is allowed to update BRDB_BRANCH_INFO
7 counting Code 2007 is
Updated 1 row in table OPS$BRDB.BRDB_BRANCH_INFO
INFO: CURRENT_JSN: 67 for branch accounting code 2007
OK: Inserted I row into OPS$BRDB.BRDB_TXN CORR
Completed BRDB_CLEAR_RO_LOCK.audit_update
Starting BRDB_CLEAR_RO_LOCK.process_audit
Completed BRDB_CLEAR_RO_LOCK.process_audit
Completed BRDB_CLEAR RO_LOCK.update data
TOOL_JOURNAL
© Copyright Fuji
UNCONTROLLED IF PRINTED
FUJITSU RESTRICTED (COMMERCIAL IN Ref:
CONFIDENCE) Version:
Date:
Page No:
DES/APP/SPG/0001
21-Sep-11
122 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
complete
Unlocked branch rollover for branch
eaning up ...
ock file
/app/b: 015/log/clear_ro_lock.run.lock freed.
03 De
03 Dec Processing Complete
5.6.3.7. Diagnostics
The module may fail for one of the following reasons
e The SSC user may not be logged in with their SSC unix login.
« The SSC user’s Oracle login may not have been granted the SSC role.
e One or more of the parameters are invalid
5.6.4 BRDB Update Outstanding Recovery Transaction Tool
(upd_rvy_txn.sh)
This tool allows members of the SSC group to mark outstanding recovery transactions as processed in
table OPS$BRDB.BRDB_RX_RECOVERY_TRANSACTIONS for any given branch accounting code,
node ID, transaction start date and unique sequence number (USN). Any attempt to run the tool will be
audited as well as the actual changes made and running user.
Validation and processing occurs in an Oracle package
(OPS$SUPPORTTOOLUSER.PKG_BRDB_UPD_RVY_TXN) while the package is initially called by a
shell script (upd_rvy_txn.sh) on the BRDB server.
The script is located in /app/brdb/trans/support/brdbx015/upd_rvy_txn.sh
See DES/APP/LLD/0204 for more information.
5.6.4.1 Parameters
The tool must be supplied with 4 switches, each with a parameter:
Parameter Parameter Name Script Variable Name Datatyp Valid
Input/Format
-b Branch Accounting Code BRANCH_CODE NUMBER 1-999999
n Node 1D NODE_ID NUMBER 1-99
+t Transaction Start Date TXN_STRT_DATE STRING DD/MMIYYYY
“u Unique Sequence Number SEQ_NUM NUMBER >0
5.6.4.2 Executing
./upd_rvy_txn.sh -b <BRANCH_CODE> -n <NODE_ID> -t <DD/MM/YYYY> -u <SEQ_NUM>
5.6.4.3 Scheduling
This task is scheduled on an ad hoc basis, as and when recovery transactions need to be marked as
processed.
5.6.4.4 Audit Records/Logging
Start and finish records are inserted into OPS$BRDB.BRDB_PROCESS_AUDIT with a process_name of
‘BRDB_UPD_RVY_TXN’.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 123 of 183,
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE @
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
Each update to OPS$BRDB.BRDB_RX_RECOVERY_TRANSACTIONS is audited in
OPS$BRDB.BRDB_TXN_CORR_TOOL_JOURNAL.
Any exceptions will be logged to OPS$BRDB.BRDB_OPERATIONAL_EXCEPTIONS with an exception
code of ‘UPD_RVY_TXN’ and process_name (package name) of ‘PKG_BRDB_UPD_RVY_TXN’.
The script verbosity level is controlled by BRDB system parameter
BRDB_UPD_RVY_TXN_DEBUG_LEVEL (parameter_number set to 1 initially), set parameter_number
to 2 in order to view the SQL update statement as well as the XML string.
Log files from each run are stored in /app/brdb/trans/support/brdbx015/log.
5.6.4.5 Sample output
This is an example of the output written to standard output and the log file when the module is
successful:
02 Dec 14:30:44 writelock
02 Dec 14:30:44 writelock
02 Dec 14:30:44 writelock
02 Dec 14:30:44
02 Dec 14:30:44 check env
02 Dec 14:30:44 check env
02 Dec 14:30:44
02 Dec 14:30:44 Main........: Environment OK
Starting
Lock file /tmp/upd_rvy_txn.run.lock created
complete.
Starting
Complete.
02 Dec 14:30:44
02 Dec 14:30:44 Starting
02 Dec WHOAMI... gseem01
02 Dec PROGNAME.. upd_rvy_txn.sh
02 Dec view _gvar SCRIPT. upd_rvy_txn
02 Dec view gvar...: THISDIR...++++++++++++ /app/brdb/trans/support /brdbx015
02 Dec 14:30:44 view gvar...: I LOG_FILE....
/app/brdb/trans/support /brdbx015/log/upd_rvy_txn_20091202_143044.1og
02 Dec LOCK_FILE 7tmp/upd_rvy_txn.run. lock
02 Dec TSTMP.... 20091202_143044
02 Dec VERBOSE ON
02 Dec view _gvar... BPP. ese eee noes sees BRDB
02 Dec view_gvar ORACLE_SID.. BRDBAL
02 Dec view_gvar BRANCH_CODE. 2007
02 Dec view_gvar NODE_ID..... 1
02 Dec view_qvar TXN_STRT_DATE. 06/10/2009
02 Dec view gvar...: 0 USNeesseeeeeeeeeeeeees 123
02 Dec view gvar...: Complete.
02 Dec 244
02 Dec 14:30:44 unlock..... : Starting
02 Dec 14:30:44
Enabling ssc role
Wed 02-Dec-2009 14:30:44.809 Set DEBUG LEVEL to 1
Wed 02-Dec-2009 14:30:44.809 Starting pkg_brdb upd _rvy txn.update data
Wed 02-Dec-2009 14:30:44.809 Starting pkg_brdb_clear_su_lock.process_audit
Wed 02-Dec-2009 14:30:44.810 Completed pkq_brdb clear_su_lock.process_audit
Wed 02-Dec-2009 14:30:44.810 INFO: Parameter p_bac = 2007
Wed 02-Dec-2009 14:30:44.810 INFO: Parameter p_node_id =
14:30:44,810 INFO: Parameter p_txn_strt_date: 06-0
Dec-2009 14:30:44,810 INFO: Parameter p usn: 123
Wed 02-Dec-2009 14:30:44.810 Starting pkg_brdb_upd_rvy txn.validate parameters
Wed 02-Dec-2009 14:30:44.810 INFO: Validating branch_accounting_code
Wed 02-Dec-2009 14:30:44.812 OK: Branch Accounting Code: 2007 is open and exis
OPS$BRDB.BRDB_BRANCH_INFO
Wed 02-Dec-2009 14:30:44.813 OK: Branch Accounting Code: 2007 exists in
OPS$BRDB.BRDB_TXN_CORR_TOOL CTL
Wed 02-Dec-2009 14:30:44.813 OK: USN 123 is outstanding for branch accounting code 2007
Wed 02-Dec-2009 14:30:44.813 OK: OPS$SUPPORTTOOLUSER is allowed to update
BRDB_RX_RECOVERY_TRANSACTIONS
Wed 02-Dec-2009 44.813 OK: Input parameters validated successfully
Wed 02-Dec-2009 14:30:44.813 Completed pkg brdb upd _rvy txn.validate parameters
Wed 02-Dec-2009 14:30:44,813 Starting pkg_brdb upd rvy txn.reset_outstanding
Wed 02-Dec-2009 14:30:44.813 OK: Derived FAD HASH for branch accounting code 2007 i,
2009
96
Wed 02-Dec-2009 14:30:44.814 OK: Updated 1 row in table OPS$BRDB.BRDB_RX RECOVERY TRANSACTIONS
Wed 02-Dec-2009 14:30:44.814 Completed pkg_brdb_rvy_txn.update data
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN _ Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 124 of 183
FUJ00088797
FUJ00088797
o HOST BRANCH DATABASE SUPPORT GUIDE ®
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
wea 02-Dec-2009 veld starting pkg_brdb Lock.audit_update
fea 020m soe
Wed 02-De 814 UPPORTTOOLUSER
med 02-Dec “es ceeed Troy inte OPssanoU-BROB TKN CORR TOOL, JOURNAL
ned 02-0 “eis eudit Gedate
Wed 02-Dec-2009
02 Dec 14:30:44
+815 k.update_data
02 Dec 14:30:44 unlock......:
02 Dec 14:30:44
02 Dec 14:30:44 Main........: Unlocked stock unit for branch code 2007
02 Dec 14:30:44
02 Dec 14:30:44 cleanup..
02 Dec 14:30:44 cleanup...
02 Dec 14:30:44
02 Dec 14:30:44 cleanup..... : Processing Complete
Cleaning up ...
Lock file /tmp/upd_rvy_txn.run.lock freed.
5.6.1.6 Diagnostics
The module may fail for one of the following reasons
e The SSC user may not be logged in with their SSC unix login.
e The SSC user's Oracle login may not have been granted the SSC role.
e One or more of the parameters are invalid
5.7 BRDBC004 Archival/Purge Logic
Starting with HNG-X Release 4.39, the replication of DELETE SQL statements to BRSS will be
controlled by a new flag named ‘ALLOW_REPLICATION’, introduced as a column in the table
BRDB_ARCHIVED_TABLES. A value of ‘N’ against a particular table indicates that DELETES against
that table, will not get replicated across to BRSS by Oracle Streams and a ‘Y’ indicates otherwise.
BRDBC004 has been enhanced to use this new flag and allows or blocks the replication of DELETEs
against a particular table, accordingly.
At the time of implementation all, archive/purge metadata records will be set to a default value of ‘N’.
This change to Branch Database archive metadata was made, firstly because local maintenance of
purging OPS$BRDB tables in BRSS was required. Hence, archive metadata for all OPS$BRDB tables
that were not already managed locally in BRSS were added to BRSS_ARCHIVED_TABLES in order to
enable BRSSC004 to purge the respective local tables based on corresponding retention periods.
Secondly, making the necessary changes to the archive processes on both BRDB and BRSS became
critical as the large volume of transaction records being purged overnight in BRDB caused load stress on
Oracle Streams.
An associated benefit of making this fix is that all data records in BRDB, which are replicated across to
BRSS can be retained locally in BRSS with differing retention periods to that of BRDB without having to
manually create Streams mechanisims for every transaction table in BRDB that needed a higher
retention period in BRSS.
As a result of this enhancement, any new table introduced into the Branch Database, must have the
requisite ‘archive metadata’ added to both BRDB_ARCHIVED_TABLES (in BRDB) and
BRSS_ARCHIVED_TABLES (in BRSS) in order for BRDBC004 and BRSSC004 to perform their
respective purge functions effectively
5.8 BRDB Software Updates/Installation
re to BRDB (whether that software is
If a total service outage is possible due to the application of sot
a Ly e e fo ould b
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN Ref: DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 125 of 183,
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
Ensure the delivery handover notes clearly state a system outage is required
CS should communicate the date/time of the planned service outage to POL (and hence the
branches)
Access to the BRDB database should be controlled by disabling/re-enabling access via the ACE.
The OSR instances may need to be restarted if there are changes that have a direct impact on
the OSRs (for example a change to BAL SQL statements)
5.9 Querying/Updating BRDB/BRSS during the
online day
Any database query that could be considered to be ‘large’ should, in general, be kept outside the
accepted online day operating hours.
The following is a guide to which queries (SELECTs, UPDATEs, DELETEs) might turn out to be ‘large’ or
over-utilise resource unnecessarily (and should therefore not be executed): -
The query involves more than one date partition (or does not even have a date restriction in the
WHERE clause) as per those tables present in BRDB_PARTITIONED_TABLES
The query features a function around the partitioned key column in the WHERE clause -
preventing Oracle from utilising partition pruning
Transactions that run for more than 5 minutes or consist of more than 500,000 rows may stress
the Streams implementation, with the result that Streams fails to keep BRSS up to date
Any query which that does not utilise the localisation of data to the instance from which the
query is executed. In other words, if a set of data relating to a branch whose natural/defined
instance is BRDB2 (for example according to the defined fad_hash-mappings) should not be
queried from BRDB3. The localisation of every query should always be a consideration!
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 126 of 183,
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
6 Appendix A — Standby Database
The build of and theory surrounding the BRDB Standby database (SBRDB) is detailed extensively in the
Standby Database Low Level Design [DES/APP/LLD/0152]. This section details the failover procedures
in changing the role of a database, in our case BRDB or SBRDB. The method described in sections 6.1
and 6.2, is known as complete failover and must be executed as described in order to ensure no data
loss.
It is very important to note - as detailed in the Branch Database High Level Design
[DES/APP/HLD/0020] — that the changing of roles of the Standby to Primary is utterly irreversible! The
term “switchover”, which is a temporary role change is not supported. Section 6.3.1 therefore, details the
temporary opening of the Standby Database for read-only purposes.
Without the broker, you perform role transitions by first determining if a role transition is necessary and
then issuing a series of SQL statements (as described later in this section). After failover to a physical
standby database, the original primary database must be re-enabled to act as a standby database for the
new primary database.
Note: The procedure described in section 6.1 is the recommended course of action. Section 6.2 has
been provided for, in the event that the Data Guard Broker is unavailable.
6.1 Oracle Data Guard Broker (DGMGRL)
Failover
The broker simplifies failovers by allowing you to invoke them using a single command in the DGMGRL
command-line interface, e.g. a manual failover. The method described in this manual procedure is
known as complete failover and must be executed as described in order to ensure no data loss.
Server Execution
Step Descript
i. User is logged onto the Standby Database Server as oracle.
Ass ii. If sufficient time is available prior to failover, it is assumed that the Grid Control “Blackout”
um of BRDB database instances BRDB2 to BRDB4 has been completed.
ptio
iii. After determining that there is no possibility of recovering the primary database in a timely
We manner, ensure that the primary database is'shut down (if not already) and then begin the
failover operation.
[Who: DBA] >
Logon to DGMGRL command-line » oRaeDY
interface. [now type in SBRDB1]
1. $> dgmgrl
<sys password> is always required as I nevcrn> CONNECT sys/<sys password>
this is a “sysdba” connection. This will
connect you via the Data Guard Broker
to the Standby Database.
[Who: DBA]
On the target standby database, issue
the FAILOVER command to invoke a
2. complete failover, specifying the name
of the standby database that you want
to change into the primary role.
DGMGRL> FAILOVER TO ‘SBRDB’ ;
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 127 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE ®
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
How the Broker Performs a Complete Failover Operation
Once you start a complete failover, the broker:
i. Checks to see if the primary database is still available and, if so, issues a warning message
asking whether you want to continue with the failover operation.
ii. Verifies that the target standby database is enabled. If the database is not enabled, you will not
be able to perform a failover to this database. The broker shuts down all RAC instances except the apply
instance assuming they are up. This is unlikely in Branch Standby Database as _ only one node is
configured to be active at any one time.
iii. Waits for the target standby database to finish applying any remaining archived redo logs before
stopping Redo Apply or SQL Apply.
iv. Transitions the target standby database into the primary database role by opening the new
primary database SBRDB, in read/write mode.
[Who: DBA]
Issue the SHOW CONFIGURATION
command to verify the failover.
DGMGRL> SHOW CONFIGURATION;
You should see ...
Configuration
Name: BRDB_DATAGUARD_CFG
Enabled: YES
3. Protection Mode: MaxPerformance
Fast-Start Failover: DISABLED
Databases:
BRDB - Physical standby database (disabled)
SBRDB - Primary database
Current status for "BRDB_DATAGUARD_CFG":
SUCCESS
[Who: DBA]
Issue the SHOW DATABASE
command to see that the former (failed)
primary database was disabled by the
4. broker as a consequence of the
DGMGRL> SHOW DATABASE ‘BRDB’ ;
You should see “Enabled: NO” and a message
returned, similar to “current status for "BRDB":
Error: ORA-16661: the standby database needs
Breve Remember, it must be re-I oy. pcinstatea”
Who: DBA] . SQL> SELECT owner, index_name
Check that all the indexes — database FROM dba_indexes
wide — are available for use. WHERE status = 'UNUSABLE';
5.
If any indexes are marked as I S9L> ALTER INDEX <OWNER>.<index> REBUILD
‘UNUSABLE’ they need to be rebuilt, I ONPTN® { PARADUEL <# cpu’ s> J;
See example to the right of this cell.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 128 of 183,
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE @
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
[Who: DBA]
Depending on the timing of the failover to Standby, there is a possibility that the BladeFrame may
not have the full complement of four pBlades configured and started for the Standby cluster.
If not already done, add the 3 additional pBlades into the BladeFrame: -
6a. 1. Physically “plug” the pBlades into the frame.
2. Configure and start the pServers.
3. Once you're able to log on as oracle, bring up the remaining database instances
starting with SBRDB2, e.g. srvctl start instance -d SBRDB -i SBRDB2
4. In Grid Control (OEM) remove the “Blackout” previously placed on instance SBRDB2
to SBRDB4.
[Who: DBA]
The following SBRDB database initialisation parameters need to be checked and if not correct,
need to be set correctly after the Standby database (SBRDB) has been successfully transitioned
from Standby to it’s new role as Primary.
This information can be double-checked by comparing the initialisation parameters from Primary
with those of Standby. The comparison can be done against pfiles generated from both nodes.
Follow Step [6b.] to accomplish this.
The following parameters should be checked and the values shown below should be reflected in
SBRDB for all new instances. This is done by executing a statement of the form “ALTER SYSTEM
SET <parameter>=<value> SCOPE=<scope> SID='*’ ;”.
Parameter Future Value Likely Current Value
audit_trail DB NONE
cluster_database_instances 4 1
control_file record keep time 21 NULL
instance_number [1] to [4] <See action 1 below>
6b. I instance_name NULL <See action 2 below>
local_listener LISTENER_<node> <See action 3 below>
log_archive_dest_3 NULL ‘LOCATION=/archredo/<DB> OPTIONAL!
log_archive_dest_state_3 NULL 'ENABLE'
sessions 2205 610
thread [1] to [4] <See action 4 below>
[1] An “aurER sysTEM .. STD=’ SBRDB2’” statement required on 6@GA instance, e.g.
instance_number=2 for node 2, 3 for node 3, et cetera.
[2] An “ALTER SYSTEM ... SID=’ SBRDB2’” statement required on @@éH instance, e.g.
instance_name=’ SBRDB2’ for node 2, ’ sBRDB3’ for node 3, et cetera.
[3] An “ALTER SYSTEM ... SID=’ SBRDB2’” statement required on 6aGH instance, e.g.
local_listener=’LISTENER_<node002>’ for node 2, ‘LISTENER_<node003>’ for node 3, etc.
[4] An “auTER SYSTEM .. SID=’ SBRDB2’” statement required on @aeNh instance, e.g. thread=2 for
node 2, 3 for node 3, et cetera.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 129 of 183
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE @
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
[Who: DBA]
In the same way as the parameters were added above, the following parameters delivered in
various releases from Release 0108 up to and including Release 0500, should also be added: -
Parameter Future Value Likely Current Value
“Red Alert” parameter changes
_kghdsidx_count 2 NULL
“library cache _advice FALSE NULL
_object_statistics FALSE NULL
db_cache_advice OFF oN
event='14532 trace name context forever, level 1' Currently unset, simply set the
value on any instance.
pga_aggregate_target 4294967296 5368709120
Oracle Resource Manager parameter changes
resource_limit TRUE NULL
resource_manager_plan HNGX_PLAN NULL
_low_server_threshold 16 7 ox NULL
“high_server_threshold 32 12 or NULL
6b shared_pool_size 4311744512 2256M
cont. Instance Group parameter changes
instance_groups 'BRDB_GRP1' ...GRP2 etc.
‘BRDB_GRP4' <See action 5 below>
parallel_instance_group 'BRDB_GRP1' ...GRP2 etc.
‘BRDB_GRP4' <See action 6 below>
parallel_max_servers 64 NULL
PAF parameter changes
sga_target 24534581248 21474836480
db_keep_cache_size 5637144576 NULL
Other Oracle Bug parameter changes
memory broker_stat_interval 60 NULL
[5] An “ALTER SYSTEM .. SID=’ SBRDB2’” statement required on 6a@h instance, e.g.
instance_groups='BRDB_GRP2' for node 2, 'BRDB_GRP3' for node 3, et cetera.
[6] An “ALTER SYSTEM .. SID=’ SBRDB2’” statement required on @a@h instance, e.g.
parallel_instance_group='BRDB_GRP2' for node 2, 'BRDB_GRP3' for node 3, et cetera.
[Who: DBA] ; I oraenv
Create a text file “copy” of the current
spfile (server parameter file) on both I I Now type BRDB1 (on node’) ]
the Primary (BRDB) and the Standby
SBRDB nodes. sqlplus ‘/as sysdba’
SQL> CREATE
PFILE='<some_dir>/pfile<DATABASE>.ora’ FROM
SPFILE;
6c.
[ Now do the same for SBRDB on the Standby node. ]
diff pfileBRDB. £ileSBRDB.
Copy the files to a location where they wee peewee one pews’ ene
can be compared and compare them
either by using the UNIX diff command
or a Windows compare tool, e.g.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 130 of 183
Fe)
FUJITSU
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
[Who: DBA]
After failover, the “new” Primary
database cluster (Iprpbds001 - 4) and
database, SBRDB, must accept
connections from all applications
without changing any application
connection properties. Therefore, in
order to accomplish this, a new
database service must be created for
BRDB.
On the first node: -
The service should already be enabled,
so all that needs to be done is to start
the service.
If starting the service is unsuccessful
for some reason, then try enabling the
service.
Once again, after enabling the service,
try starting the service again.
With the service having been correctly
created, check the CRS status to see
the state of the services as well as the
listener control utility.
Syntax
srvctl add service
-d <db_unique_name>
-s <service_name>
-x <preferred_list>
Command
srvctl add service -d SBRDB
SBRDB1, SBRDB2 , SBRDB3 , SBRDB4
-s BRDB -r
srvetl start service -d SBRDB -s BRDB
srvectl enable service -d SBRDB -s BRDB
[A.] crs_stat -t
[B.] 1snrctl status listener_lprpbds001
(Al
application
application
application
application
application
(B.]
Connecting to
vip) (PORT=1529) (IP=FIRST) ))
STATUS of the LISTENER
Services Summary
Service "BRDB"
Instance "SBRDB1",
The correct output seen, should be similar to the following:
LSNRCTL for Linux: Version 10.2.0.4.0 - Production on
Copyright (c) 1991, 2007, Oracle.
(DESCRIPTION= (ADDRESS= (PROTOCOL=TCP) (HOST=1prpbds001-
LISTENER_LPRPBDSOO1
TNSLSNR for Linux: Version 10.2.0.4.0 - Production
has 1 instance(s) .
status READY, has 1 handler(s) for this service
Target State Host
ONLINE ONLINE 1prpbds001
ONLINE ONLINE 1prpbds002
ONLINE ONLINE 1prpbds003
ONLINE ONLINE 1prpbds004
ONLINE ONLINE 1prpbds001
All rights reserved.
© Copyright Fujitsu Ltd 2011
UNCONTROLLED IF PRINTED
FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
Ref: DES/APP/SPG/0001
Version: 3.0
Date: 21-Sep-11
Page No: 131 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
[Who: UNIX ADMIN]
The Primary database cluster (Iprpbdb001 — 4) after failover will be the former Standby database
cluster (Iprpbds001 - 4), so as a result of the BRDB failover to the BDS cluster, it will be necessary
to re-configure DNS to seamlessly make this change, thereby allowing all applications that
reference the Primary database cluster to instead reference the Standby database cluster.
In order to accomplish this, the following should be followed: -
[1.] Update ACD001 to change the PBDBOOX-VIP alias to point to associated BDS servers, e.g.
(Iprpbds001 - 4)
[2.] Flush DNS cache on all Linux DNS servers (DNP and DNS)
/usr/sbin/rnde flush
[8.] Clear the DNS cache on all servers that address BDB on VIP alias
/usr/sbin/nscd --invalidate=hosts
[4.] Once the DNS switch is complete perform a set of ‘ping’ sanity checks to ensure that client
applications (DAT, BAL/OSR, etc) are referencing the “new” Primary server IP addresses.
[5.] In addition to [4.] above, perform a quick test to ensure that one is connecting to the correct
database and that the newly created service (Step 7. above) is accepting connections.
sqlplus lvbaluserl/<lvbaluserl password>@BRDB
[6.] To allow TWS to access and run schedules on the new Primary nodes: -
Update vws . cpu to point AGBRDBI1234] to PBDS00[1234]
Update DNS to point PBDB00[1234] to LPRPBDS00[1234]
WARNING
Any subsequent DNS deliveries may reset the IP addresses back to the original BRDB1..4 servers.
It may be necessary to raise an OCP along with a DNS delivery to set the IP addresses back to the
fail-over servers.
[Who: DBA or UNIX ADMIN]
On the new Primary server, e.g. the BDS Cluster (on
each node, e.g. 1 — 4), the cron jobs which run on these $> crontab -e
nodes in the absence of any TWS schedules need to be
9. stopped.
Edit the crontab.
Once the crontab has loaded (output should reflect
schedule shown below).
As the oracle user ...
Use vi commands to add a “#” in front of every line
where one does not exist. Then save and quit the file.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 132 of 183,
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
FUJ00088797
FUJ00088797
HouseKeeping
# RMANBackup
#0 4 * * Sun /app_sw/brdb/sh/RMANBackupWrapper.sh SBRDB 0 >
2>81
#0 4 * * Mon /app_sw/brdb/sh/RMANBackupWrapper.sh SBRDB 1 >
2>61
#0 4 * * Tue /app_sw/brdb/sh/RMANBackupWrapper.sh SBRDB 1 >
2>61
#0 4 * * Wed /app_sw/brdb/sh/RMANBackupWrapper.sh SBRDB 0 >
2>61
#0 4 * * Thu /app_sw/brdb/sh/RMANBackupWrapper.sh SBRDB 1 >
2>61
#0 4 * * Fri /app_sw/brdb/sh/RMANBackupWrapper.sh SBRDB 1 >
2>61
#0 4 * * Sat /app_sw/brdb/sh/RMANBackupWrapper.sh SBRDB 1 >
2>61
#
#
06 * * * /app_sw/brdb/sh/HousekeepWrapper.sh SBRDB > cron.
56 * * * /app _sw/brdb/sh/HousekeepWrapper.sh +ASM > cron.
#
sbrdb.out 2>&1
2>61
asm.out
eron.rb.
cron.rb.
cron. rb.
eron.rb.
cron.rb.
cron. rb.
cron.rb.
sun.
mon.
tue.
wed.
thu.
fri
sat.
sout
out
out
out
out
out
out
from a copy of the new primary database.
To maintain a viable disaster-recovery solution in the event of another disaster you must reinstate
10 the original primary database to act as a standby database in the new configuration. This can be
. accomplished by following the notes in Section 6.3, as one must re-create the primary database
Manual Complete Failover through DGMGRL is complete.
Table 3: Data Guard Failover Procedure.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN Ref:
CONFIDENCE) Version:
Date:
UNCONTROLLED IF PRINTED Page No’
DES/APP/SPG/0001
3.0
21-Sep-11
133 of 183,
Fe)
FUJITSU
HOST BRANCH DATABASE SUPPORT GUIDE
FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
FUJ00088797
FUJ00088797
6.2 SQL*Plus Failover
Perform role transitions by first determining if a role transition is necessary and then issuing the following
series of SQL statements. The method described in this procedure is also known as complete failover
and must be executed as described in order to ensure no data loss.
Step Description Server Executit
> i: User is logged onto the Standby Database Server as oracle.
SS
um ii. After determining that there is no possibility of recovering the primary database in a
ptio timely manner, ensure that the primary database is shut down (if fot already) and any
ns other standby database instances that may be started, then begin the failover
Operation.
(Who: DBA] $> . oraenv
Logon to SQL*Plus command-line
interface as SYSDBA, but first set the I [now type in SBRDB1]
correct Oracle SID.
This will connect you to the Standby I $> sqlplus ‘/as sysdba’
1. Database.
SQL> SELECT * FROM v$instance;
Double-check that you are on the right
instance, noting in particular the
values for instance_name, host_name.
and status.
[Who: DBA] SQL> ALTER DATABASE RECOVER MANAGED
Initiate the failover by issuing the STANDBY DATABASE FINISH FORCE;
following.
Note: Include the FORCE keyword
to ensure that the RFS
2. processes on the standby
database will fail over
without waiting for the
network connections to time
out through normal TCP
timeout processing before
shutting down.
[Who: DBA] SQL> ALTER DATABASE COMMIT TO SWITCHOVER
Convert the physical standby TO PRIMARY;
database to the production role.
3. Note: Don’t get confused by the
word “Swit¢hoVer” as this
command is part of a
complete manual primary
failover and not a role switch
as may be interpreted by this
© Copyright Fujitsu Ltd 2011
UNCONTROLLED IF PRINTED
FUJITSU RESTRICTED (COMMERCIAL IN Ref:
DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
Page No: 134 of 183
Fe)
FUJITSU
HOST BRANCH DATABASE SUPPORT GUIDE
FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
FUJ00088797
FUJ00088797
word.
4a.
[Who: DBA]
Open the new production (primary)
database by issuing the following
statement.
SQL> ALTER DATABASE OPEN;
4b.
[Who: DBA]
Only complete the following if the
condition below is met! Otherwise do
not. This can be verified by checking
for this value. Run the following SQL
to do so.
Condition:
If the physical standby database has
been opened in read-only mode since
the last time it was started, shut down
the standby database (now primary
database) and restart it.
SQL> SELECT value
FROM v$dataguard stats
WHERE name = ‘standby has been open';
SQL> SHUTDOWN IMMEDIATE;
SQL> STARTUP;
[Who: DBA]
Check that all the indexes — database
wide — are available for use.
(SUT of Section 6.1
[Who: DBA]
The database initialisation parameters
need to be checked and if not correct,
need to be set correctly after the
Standby database has been
successfully transitioned from Standby
to it's new role as Primary.
See Step [f@M of section 6.1
[Who: DBA]
After failover, the “new” Primary
database cluster (Iprpbds001 - 4) and
database, SBRDB, must accept
connections from all applications
without changing any application
connection properties.
Therefore, in order to accomplish this,
[i] a new database service must be
created for BRDB and [ii.] the DNS
settings for both servers need to be
reconfigured.
See Steps [f#Mj and (BM of section 6.1
To maintain a viable disaster-recovery solution in the event of another disaster you must
reinstate the original primary database to act as a standby database in the new configuration.
This can be accomplished by following the original Standby Database deployment handover
© Copyright Fujitsu Lid 2011
UNCONTROLLED IF PRINTED
FUJITSU RESTRICTED (COMMERCIAL IN __ Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
Page No: 135 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
notes as one must re-create the primary database from a copy of the new primary database.
Manual Complete Failover through SQL*Plus is complete.
Table 4: SQL*Plus Failover Procedure.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 136 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
6.3 Standby Database Re-instantiation
As explained and demonstrated in the preceding sections of this chapter, the original primary database,
namely BRDB, would have now failed over to the standby database, namely SBRDB. Therefore in order
to ensure a viable and highly available configuration once again, the old primary must be re-instated as
the new standby.
The database is setup correctly. All that is required is getting a duplicate of the new primary database
back onto the server in order to start the new standby in managed recovery mode. This is that process.
Step Description Server Execution
acs b User is logged onto the Standby Database Server as oracle.
um ii. This procedure is only applicable after having completed a failover of Primary (BRDB)
ptio to Standby (SBRDB) as detailed in sections 5.1 and 5.3.
us iii. Only one node should be used as the new standby database node.
[Who: DBA]
New Prim Server
$> . oraenv
Backup the new primary (SBRDB)
database using RMAN. Ensure there I [now type in SBRDB1]
is sufficient space on the device you
specify as <RMAN DIR>.
$> $ORACLE_HOME/bin/rman NOCATALOG TARGET /
Logon to RMAN.
RMAN> run
{
Execute the backup commands asI P@ckup .
they appear, e.g. run { .. } current controlfile
, for standby
format '<RMAN DIR>/%d_%u';
1. backup
format '<RMAN DIR>/%d_%U'
database;
backup
format '<RMAN DIR>/%d_%U'
archivelog all
not backed up 1 times;
}
> cd <RMAN DIR>
Exit RMAN and change directory to g S “1
the <RMAN DIR> and make sure the
backup is as you expect. This can be
confirmed by listing the backup in
RMAN, €.g. list backup summary;
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref: DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 137 of 183
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE @
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
[Who: DBA]
New Prim Server
$> scp <RMAN DIR>/* pbdb001<RMAN DIR>
Ensure the entire backup, which will
2 consist of a number of files is copied
. across from this server to the new
standby server.
Note: The backup must exist in the
same location on both servers!
[Who: DBA]
Old Prim Server (node 1
$> . oraenv
Cleanup the old archive directory as it
would be full of files that are no longer I [now type in +ASM1]
needed. Type ves, if prompted.
$> asmemd -p
3. ASMCMD [+] > cd BRDB_FLASH/arch
ASMCMD [+BRDB_FLASH/arch] > rm -r brdb*.arc
[Who: DBA]
Old Prim Server (node 1
$> . oraenv
Set the environment for the new .
standby database. [now type in BRDB1]
$> $ORACLE_HOME/bin/rman
4. Log onto RMAN and execute the TARGET=sys/<SYS_PASSWD>@sbrdb AUXILIARY /
Sen the new primary as the NeW I ans duplicate target database for standby;
Ensure there are no errors in this
restore. Otherwise, fix the errors and
run again.
[Who: DBA]
Old Prim Server (node 1
5. The standby database should I SOL> ALTER DATABASE MOUNT STANDBY DATABASE;
already be mounted, but if not,
mount the new standby database.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref: DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 138 of 183,
Fe)
FUJITSU
HOST BRANCH DATABASE SUPPORT GUIDE
FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
FUJ00088797
FUJ00088797
[Who: DBA]
Old Prim Server (node 1
Start the standby database in
managed recovery mode.
This must have completed
SQL> ALTER DATABASE RECOVER MANAGED
STANDBY DATABASE USING CURRENT LOGFILE
DISCONNECT FROM SESSION PARALLEL 8;
8. successfully. To check that it has, I SQL> SELECT * FROM v$dataguard_status
query v$dataguard_status. ORDER BY message_num DESC;
Also, check that the application of logs I S°4> SELECT * FROM v$dataguard_stats;
is performing well, query
v$dataguard_stats.
[Who: DBA]
New Prim Server
$> cd /app_sw/sbrdb/standby
7. Ensure that the tnsnames.ora has an I $> SBRDB_edit_tnsnames.sh -v ~s lprpbds001
entry for the new primary.
Note: The following files should already be available and configured correctly from the
previous installation of the old Primary database. If for whatever reason, they are not,
configure accordingly: -
$ORACLE_HOME/dbs/orapwBRDB
8. SORACLE_HOME/network/admin/tnsnames.ora
$ORACLE_HOME/network/admin/listener.ora
/102/oradata/BRDB/spfileBRDB.ora
/u02/oradata/BRDB/dr1BRDB.dat
/302/oradata/BRDB/dr2BRDB.dat
Manual re-instantiation of Standby Database Complete.
Table 5: Primary Re-instatiation Procedure.
6.3.1 Tripwire Configuration
The Tripwire targeting on the EMS platform needs to be edited to:
Comment out the BDB platforms
Uncomment the BDS platforms
© Copyright Fujitsu Ltd 2011
FUJITSU RESTRICTED (COMMERCIAL IN Ref:
CONFIDENCE) Version:
UNCONTROLLED IF PRINTED
Date:
Page No:
DES/APP/SPG/0001
3.0
21-Sep-11
139 of 183
Fe)
FUJITSU
HOST BRANCH DATABASE SUPPORT GUIDE
FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
FUJ00088797
FUJ00088797
6.4 Opening Standby Database “READ ONLY”
As explained earlier, the changing of roles of the Standby to Primary is irreversible, so in a scenario
where, for whatever reason, the Standby database needs to be read to check data available only in an
“OPEN’” state then the steps in the following table will apply.
There is a (an extremely critical) caveat to performing this operation. This operation opens the SBRDB
database in “READ ONLY” mode. Oracle Data Guard is aware that this operation would have taken
place and keeps a record of each time this is done. It is important to note that performing this operation
changes the way the database is recovered if it ever becomes Primary. This is noted by the step
detailed in [4b.] of Section 6.2.
Step Description Server Execution
i. User is logged onto the Standby Database Server as oracle.
ii. The application of archivelogs/redo on the Standby Database will be cancelled and will
Ass therefore fall behind Primary for the period the database is in a “READ ONLY” state and
um that the relevant authority to perform this task has been sought, being completely aware of
Ptio the consequences detailed above.
ns
Recommendation:
Open a second terminal session to the Standby Server and “tail -£” the SBRDB alert log.
Who: DBA} $> . oraenv
Logon to the database as SYSDBA.
[now type in SBRDB1]
1 $> sqlplus ‘/as sysdba’
. Check that the standby database is I SQL> set lines 140
applying logs at good rate, i.e. ensure I SQL> set pages 45
that the lag isn't too great. SQL> col value for a20
SQL> select * from v$dataguard_stats;
{Who: DBA] SQL> ALTER DATABASE RECOVER MANAGED
Cancel the real-time apply of I STANDBY DATABASE CANCEL;
archivelogs to standby.
2. The alert log will show the following
error which can safely be ignored: -
‘ORA-16037: user requested
cancel of managed recovery
operation”
(Who: DBA] SQL> ALTER DATABASE OPEN READ ONLY;
Open the Standby database in “READ
ONLY” mode.
3.
The database is now open for any
“SELECT” queries that may be run
against it.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED
Page No: 140 of 183,
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
[Who: DBA]
4 After having queried the information
° required, shutdown the database once
again.
SQL> SHUTDOWN IMMEDIATE;
[Who: DBA]
At this point it is possible that you may
experience symptoms of the following
bugs for Oracle 10.2.0.4:
- 6737051
- 7677840 (base bug is 6737051)
- 8533262 (base bug is 7677840)
SQL> SHUTDOWN IMMEDIATE;
The ORA-00600 messages (see 4b.
below) in the alert log can be ignored at
this point, as at the time of the
authoring of this document (June 2009)
a fix for these bugs has not yet been
released. In addition continuing with
step 5. below does not reveal any
additional “ORA-“ errors and real-time
apply continues as expected without
hiccups.
4a.
The following are the messages you may see in the alert log: -
4b, I 0RA-00600: internal error code, arguments: [kjevg04], [], (1, 0, U1, U1, 1, 0
ORA-00600: internal error code, arguments: [kjr_pool_free_all:resp],
[0x0B4E44B98], [0x0BF73B138], [0x0BF73B138], [0], [8], [1], []
[Who: DBA] SQL> CONNECT /as sysdba
Bring the Standby database back up I SQL> STARTUP NOMOUNT
into a “MOUNTED*” state. SQL> ALTER DATABASE MOUNT STANDBY DATABASE;
The Data Guard broker should take
over at this point and start the real-time
apply automatically. To check that this
has indeed taken place, check that the
following is evident in the alert log: -
5. “Completed: ALTER DATABASE RECOVER
MANAGED STANDBY DATABASE THROUGH ALL
SWITCHOVER DISCONNECT USING CURRENT
LOGFILE”
$> dgmgrl
DGMGRL> connect /
DGMGRL> show configuration;
Double-check that the database is
correctly applying as expected. Using
the dataguard broker check the status
of the configuration. You should see: -
Current status for
"“BRDB_DATAGUARD_CFG": SUCCESS
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 141 of 183,
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
(Who: DBA] SQL> ALTER DATABASE RECOVER MANAGED
If the above checks are inconclusive, I STANDBY DATABASE USING CURRENT LOGFILE
start the real-time apply by manually. DISCONNECT FROM SESSION PARALLEL 8;
6. Check the alert log that messages
similar to the following are being
successfully logged: -
‘Media Recovery Log
+SBRDB_FLASH/arch/brdb_ 000...”
Manual opening of Standby in READ ONLY mode is complete.
Table 6: Opening Standby Database Read-only
6.5 Standby Cluster — Software Installation
The Standby Database BladeFrame has been configured (for the first release) to make use of a single
active pServer and 3 inactive (i.e. 1 pBlade plugged in and active with the remaining pBlades utilized
elsewhere). This setup effectively makes the cluster run as single-node RAC cluster, but at the point
where a failover is required, the remaining pBlades are activated allowing the cluster the full compliment
of nodes.
Having this configuration is sufficient for running in an environment where there is no need for software
updates. However, software installations, UNIX patches, database software upgrades, database
patches, etc. is an ongoing required activity.
Therefore the following describes a means of accomplishing a software update across all standby nodes
in order to keep them functionally in sync with Iprpbds001 (node 1).
There are two possibilities, both of which will require a period of downtime, so ideally this would be after
working hours each day or on the weekend. The first, “Alternative A’, will allow the software update to
be accomplished fairly quickly but renders the primary cluster without throughput, which may be
considered a problem if batch schedules run at the same time. The second possibility will be
accomplished a lot slower, but leaves the primary cluster with the ability to carry most of the operational
workload.
Both possibilities are in essence the same set of steps, just executed in differing combinations of
pBlades.
Alternative Implementation Description
NOTE
These alternatives are presented at a high level and the level of detail required, is beyond the scope of this
document.
The steps mentioned below will need to be coordinated by more than one team; At first glance, those teams
would likely be UNIX Support, DBA Support and cooperation from Tivoli/Schedule Support (SMC).
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 142 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
Primary Cluster
i. SMC: Give the go-ahead that all schedules are held for the affected node(s)
ii. DBA: Using Grid Control, initiate a blackout of all components on the affected
nodes, e.g. agents, listeners, database instances, etc.
iii. I UNIX: Using the BladeFrame PAN Manager, shutdown the pServers which
correspond to nodes 2, 3 and 4, €.g. lprpbdb002 - 004
Standby Cluster
iv. UNIX: Using the BladeFrame PAN Manager, startup (logically switch) the pServers
which correspond to nodes 2, 3 and 4, e.g. lprpbds002 - 004
v. DBA/UNIX/3" Party: Perform the required change, installation, patch, etc.
Once the required changes are complete, reverse the process of implementation and restore
the pBlades to their original BladeFrame, thereby returning the Primary Cluster to it’s former,
fully operational state, including all Grid Control blackouts and notification to SMC. There
must be no unresolved Grid Control alerts or exceptions in
Alternati BRDB_OPERATIONAL_EXCEPTIONS at the end of this process. The BAL OSR’s need to
ve A. also be recycled at this point.
Finally, if the “End of Day” process is not, for whatever reason, going to be run by the time all
nodes will be required, then one needs to use the process defined in Section 4.3.3 to logically
bring the nodes/instances back into operation.
NOTE
When restoring (whether in a failover scenario or general build maintenance) the Standby
nodes iprpbds002 - 004 Oracle CRS will not know that the instances SBRDB2 - 4 are
standby instances and will therefore behave as configured and attempt to start them. This
behaviour is correct and must not be changed.
Because of the way this is configured, this should always be a manual task, i.e. make sure
that the apply instance (SBRDB1) has been started and mounted “as standby”; this will “kick
off’ the recovery process. Even though the other instances are not up, they will be in
“nomount” mode, so bring and keep them down.
No viable alternatives have currently been agreed upon.
Alternati
veB
Table 7: Alternatives for Managing Software Installations on BDS nodes.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 143 of 183,
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
7 Appendix B - Branch Support
The Branch Support Database is a database used in supporting the main BRDB application by providing
access to all data found in the main database but without having access to it. The means by which the
data is replicated from BRDB to BRSS is via Oracle Streams. Oracle Streams is inherently complex and
therefore has multiple facets to consider when supporting it day-to-day and troubleshooting any problems
that arise.
The following procedures detail the rather destructive process of cleaning out all the Streams queue
Lu les and configuration and then re-creating it. This is extremely destructive and
be recovered without restoring both the Branch Database and the Support Database, unless this
is an intended action (see Section 7.1.2).
Branch Database Cluster Branch Support Cluster
Ine Ons38RD8 I 2
Exclude’ WKG_* . ia
t I BRSS_APPLQ
BRDB_CAPTURE) [-] } >
BRDB_CAPTQ
“Redo Log I Discarded changes
Miner I
BRDB I BRSS
7.1 Cleanup and Re-instantiation of Oracle
Streams
7.1.1. Assumptions
Oracle Streams, as mentioned before, is an extremely effective replication technology but is rather
complex. The cleaning up of streams is an action of Jastifesort! Therefore you must have exhausted
every other means of resolution before attempting the steps in the following procedure.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 144 of 183,
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE ®
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
7.1.2 Cleanup and Re-instantiation Procedure
The procedure ...
Step Description Server Exec
Assumptions
User is logged on as 6faelé, on both the BRDB and BRSS servers, i.e. 2 sessions.
ion
Sections and Scenarios
Steps 1-3 Pre-build Section: This section is related to a scenario where BRDB has been built to phase x and a period of 12 hours or more occurs,
such that the partitions are moved forward out-of-sync with BRSS. BRSS will also need to be built to phase x, but with
the correct dates.
Steps4-18 Main: This section is related to general cleanup and re-creation of the Streams configuration and associated objects.
Step/Section A: Specific cleanup scenario related to Step 5b.
Step/Section B: Streams recovery section with resolution of the following scenarios:-
Scenario 1 - BDB has “moved on” in terms of partition management and BRS has fallen behind. There will be a
number of partitions that exist in BDB for a particular table, but not in BRS. As a result data inserted into that table will
fail in BRS as the required partition is missing.
Section 7.2.4: Scenario 2 - Scenario 1 is a confirmed case and required archivelogs, i.e. any single archivelog or a number of, or all
archivelogs created after required_checkpoint_scn are permanently missing and hence Streams cannot recover.
Scenario 3 - It has been determined that Streams has fallen behind to an unacceptable level and a decision has been
made to re-instantiate the Streams configuration.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN Ref: DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 145 of 183
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE ®
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
BDB:
oraenv
Login into the database (as oracle)
Run the following SQL.
Find the oldest partition date which exists in the
output as “MIN_PARTS”. In this example it is
[now type in BRDB1]
sqlplus ‘/as sysdba’
SET lines 120
SET pages 45
SELECT table_name,
MIN (SUBSTR (partitio:
MAX (SUBSTR (partitio:
all tab partitions
table owner "OPS$BRDB'
composite = 'YES'
GROUP BY table name
ORDER BY table_name;
ame, LENGTH (part
ame, LENGTH (partition_name)-7,9)) min_parts,
tion name)-7,9)) max parts,
COUNT (SUBSTR(partition_name, LENGTH (partition_name)-7,9))
count_parts
described in the software handover note, but
using the oldest partition date from (1.) as the
input.
“90080803” TA NAME MAX_PARTS COUNT_PARTS
BRDB_DAILY_CUMULATIVE_SUMMARY 080803 20080927 56
BRDB_DAILY_SUMMARY 20080927 20080927 1
BRDB_RX_REP_EVENT_DATA 20080923 20080927 5
BRDB_RX_REP_SESSION_DATA 20080803 20080927 56
BRDB_TX_TRANSACTION_TOTALS 20080923 20080927 5
BRS:
Now continue with the build instructions as I BRSSSchemaBuild.sh ~p 20080803
© Copyright Fujitsu Ltd 2011
CONFIDENCE)
UNCONTROLLED IF PRINTED
FUJITSU RESTRICTED (COMMERCIAL IN. Ref:
DES/APP/SPG/0001
Version: 3.0
Date: 21-Sep-11
Page No: 146 of 183
2
FUJITSU
HOST BRANCH DATABASE SUPPORT GUIDE ®
FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
FUJ00088797
FUJ00088797
BRS:
Using “BRSS Start of Day” BRSSC001, roll the
database forward to the present day, i.e. the
same date known to be configured for BRDB.
This is most likely the greatest value of the
“MAX_PARTS” column from the output shown
in (1.). In this example it is “20080927”.
Note: BRDB and BRSS should now both be
in sync.
TABLE_NAME
I DAILY_CUMULATIVE_SUMMARY
MIN_PARTS
MAX PARTS COUNT PARTS
4a.
BRS:
If at this point, Streams exists and you're
unable to continue with [4b.], skip to [5a.].
4b.
BRS:
Now continue with the build instructions as
described in the handover note and run the
BRSS Streams installation.
From /app_sw/brss/build/streams
brdbStreams
1 -a BRB -o +BRBS Frias
run .
-s <Streams
Tablespace Size>
© Copyright Fujitsu Ltd 2011
CONFIDENCE)
UNCONTROLLED IF PRINTED
FUJITSU RESTRICTED (COMMERCIAL IN Ref:
DES/APP/SPG/0001
Version: 3.0
Date: 21-Sep-11
Page No: 147 of 183
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE ®
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
BDB:
oracny
As in [4b.] complete the configuration of
streams on BRDB. However, if Streams has
already been configured, then the old
components need to be removed first.
[now type BRDB1]
sqlplus stradmin/<stradmin_password>
[at the “SQL>” prompt, run the following]
SELECT propagation_name FROM dba_propagation;
[BRDB_PROPG should be returned]
exec dbms_propagation_adm.drop propagation('BRDB_PROPG');
exec dbms_streams_adm.remove_streams_configuration;
col object_name for a40
_name, object_type FROM user_objects WHERE object_name like
5a.
[If any queue tables exist, remove them by running the following]
exec dbms_agadm.drop_queue_table (queue_table
'STRADMIN.BRDB_CAPT ,
[Then run this again]
exec dbms_streams_adm.remove_streams_configuration;
CT object_name, object_type FROM user_objects WHERE object_name like
'SBRDBS';
[Then repeat the process .. if any queue tables exist, remove them. However, if
any other objects exist, then follow [5b.] and the Oracle Metalink Note:
356323.1]
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN Ref: DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED
Page No: 148 of 183
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE ®
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
BDB:
Oracle Metalink Note 356323.1 > Oracle Metalink Note: 356323.1 leads to --> OM Note: 203225.1 wh explains
5b. [Shown in item “A” at the end of this I more on the symptoms and how to resolve issues for 10g (else OM N 236898.1
procedure.] for 9.2).
BDB:
[Now logout and log back in as sysdba]
5c Continue with the cleanup, e.g. drop the
° STRADMIN user and then drop the old I DROP USER stradmin CASCADE;
tablespace. DROP TABLESPACE brdb_streams_data INCLUDING CONTENTS AND DATAFILES;
BRS:
5d. I At this point [4b.] needs to be complete before I Complete [4b.]
continuing. Once complete, continue with [5e.]
BDB:
From /app_sw/brdb/build/streams/ .. run
5e. Once the cleanup is 100% complete. Run the - :
BRDB Streams installation. brdbStreams.sh -a BRDB -d +BRBB FLASH -s <Streams Tablespace Size>
BDB: . su back to oracle ..
Check that Streams was successfully installed. + oraenv
{now type BRDB1]
6 sqlplus stradmin/<stradmin_password>
[at the “SQL>” prompt, run the following]
3 SYSDATE FROM dual@BRSS;
[If you don’t see the system time,
the installation failed.
© Copyright Fujitsu Ltd 2011
CONFIDENCE)
UNCONTROLLED IF PRINTED
FUJITSU RESTRICTED (COMMERCIAL IN
Retry.]
Ref: DES/APP/SPG/0001
Version: 3.0
Date: 21-Sep-11
Page No: 149 of 183
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE ®
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
BDB:
From /app_sw/brdb/build/streams/ ... run
7. Activate the Streams configuration.
streams
BDB:
Check that Streams has activated.
8. STATUS should be “ENABLED”
Continue to Step 9 ...
sqlplus '/as sysdba'
SELECT capture_name, queue_name, status, logminer_id, error_number,
message
dba
err
The following steps (Steps 9 — 18), are required to “install” releases of Streams which were delivered subsequent to the original build.
These releases were made in order that capture and apply processes perform better as well as introduce additional DML and DDL filters.
BRS:
Execute the following DML in order to “reset”
the patch release data. This will enable the re-
running of the required database patch,
BRSSPatch0035.sh, in order to install the
change.
One should only see 7 rows deleted.
As oracle execute the script ...
DELETE
FROM OPS$BRSS.pat
WHERE patch_name
AND patch_step IN (
release_management
BRSSPatch0035.sh'
4,20,21,22,23,24,25,26);
-- Commit when you’re ready
--COMMIT;
BRSSPatch0035.sh -v -i BRSS1 -s OPS\$BRSS
© Copyright Fujitsu Ltd 2011
CONFIDENCE)
UNCONTROLLED IF PRINTED
FUJITSU RESTRICTED (COMMERCIAL IN Ref:
DES/APP/SPG/0001
Version: 3.0
Date: 21-Sep-11
Page No: 150 of 183
2
FUJITSU
HOST BRANCH DATABASE SUPPORT GUIDE ®
FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
FUJ00088797
FUJ00088797
BRS:
Execute the following DML. The patch,
BRSSPatch0029.sh will install the change.
One should only see 14 rows deleted.
ROM OPS$BRSS.pa
WHERE patch_name =
release_management
BRSSPatch0029.sh';
-- Commit when you're ready
10. --COMMIT;
As oracle execute the script ... BRSSPatch0029.sh -v -i BRSS1 -s OPS\$BRSS
After the patch is complete, execute the I S=LECT
following SQL. One should see a count of 45. ROM dml_handlers
WHERE c = 'OPS$BRDB';
BRS:
Execute the following DML. The patch,
BRSSPatch0037.sh will install the change. nagement
WHERE ssh!
-- Commit when you're ready
"4 ~-COMMIT ;
As oracle execute the script ...
After the patch is complete, execute the
following SQL. One should see a count of 27.
BRSSPatch0037.sh -v -i BRSS1 -s OPS\$BRSS
SELECT COUNT (1)
FROM dml_handlers
WHERE ¢ = 'OPSSBRDB'
AND "Y';
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN Ref: DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date 21-Sep-11
UNCONTROLLED IF PRINTED
Page No: 151 of 183
Fe) HOST BRANCH DATABASE SUPPORT GUIDE ®
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
FUJ00088797
FUJ00088797
BDB:
Execute the following DML. The patch,
BRDBPatch0208.sh will install the change.
DELETE
OM OPSS$BRDB.pa
SRE patch_name
12.
One should only see 5 rows deleted -- Commit when you're ready
~-COMMIT;
As oracle execute the script ... BRDBPatch0208.sh -v -i BRDBl -s OPS\$BRDB
DELETE
OM OPSS$BRSS.patch_release_management
Execute the following DML. The patch, WHERE patch_name "BRSSPa 0043.sh';
BRSSPatch0043.sh will install the change.
mmit when you're ready
COMMI
As oracle execute the script ... BRSSPatch0043.sh -v -i BRSS1 -s OPS\$BRSS
13.
After the patch is complete, execute the
following SQL. One should see a count of 66.
Now, execute the following SQL. One should
see a count of 27.
COUNT (1)
dba apply dml_handlers
= object owner = 'OPS$BRDB'
user_procedure LIKE '%PKG_BRSS_APPLY_HANDLER:
COUNT (1)
dba_apply_dml_handlers
WHERE object_owner "OPS$BRDB'
AND assemble_lobs = 'Y';
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED
Page No: 152 of 183
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE ®
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
BDB:
Execute the following DML. The patch,
BRDBPatch0307.sh will install the change.
14.
As oracle execute the script ...
ROM OPS$BRI
WHERE patch_name =
release_management
BRDBPatch0307.sh';
-- Commit when you're ready
COMMIT;
BRDBPatch0307.sh -v -i BRDB1 -s OPS\$BRDB
BRS:
Execute the following DML. The patch,
15a. I BRSSPatch0054.sh will install the change.
One should only see 3 rows deleted
As oracle execute the script ...
15b. I The patch, BRSSPatch0055.sh should not need
to be executed, but execute it in any case.
As oracle execute the script ...
DELETE
FROM OPS$BRSS.patch_release_management
WHERE patch_name "BRSSPatch0054.sh'
AND patch_step IN (1,3,4);
- Commit when you're ready
COMMIT
BRSSPatch0054.sh -v -i BRSS1 -s OPS\$BRSS
BRSSPatch0055.sh -v -i BRSS1 -s OPS\$BI
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN Ref. DES/APPISPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED
Page No: 153 of 183
Fe) HOST BRANCH DATABASE SUPPORT GUIDE ®
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
FUJ00088797
FUJ00088797
BRS:
Execute the following DML. The patch,
BRSSPatch0061.sh will install the change.
16. I One should only see 3 rows deleted
As oracle execute the script ...
FROM OPS$BRSS.patch_release_management
WHERE patch_name = 'BRSSPatch0061.sh'
AND patch_step IN (1,2,3);
-- Commit when you're ready
COMMIT;
BRSSPatch0061.sh -v -i BRSS1 -s OPS\$BRSS
BRS:
Execute the following DML. The patch,
BRSSPatch0062.sh will install the change.
17. I One should only see 3 rows deleted
As oracle execute the script ...
DELETE
FROM OPS$BRSS.pa release_management
WHERE patch_name BRSSPatch0062.sh'
AND patch step IN (1,2,3);
-- Commit when you’re ready
COMMIT;
BRSSPatch0062.sh -v -i BRSS1 -s OPS\SBRSS
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN Ref. DES/APPISPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED
Page No: 154 of 183
Fe) HOST BRANCH DATABASE SUPPORT GUIDE ®
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
FUJ00088797
FUJ00088797
BDB:
Execute the following DML. The patch,
BRDBPatch0335.sh will install the change.
18.
As oracle execute the script ...
FROM OPS$BRDB.patch_release_management
WHERE patch_name = 'BRDBPatch0335.sh'
AND patch_step = 2;
-- Commit when you're ready
COMMIT;
BRDBPatch0335.sh -v -i BRDB1 -s OPS\$BRDB
BRS:
Execute the following DML. The patch,
BRSSPatch0073.sh will install the change.
19. I One should only see 3 rows deleted
As oracle execute the script ...
DELETE
FROM OPS$BRSS.patch_release_management
WHERE patch_name "BRSSPatch0073.sh'
AND patch step IN (7,8,9);
-- Commit when you’re ready
COMMIT;
BRSSPatch0073.sh -v -i BRSS1 -s OPS\SBRSS
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN Ref. DES/APPISPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED
Page No: 155 of 183
Fe) HOST BRANCH DATABASE SUPPORT GUIDE ®
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
FUJ00088797
FUJ00088797
Following Métalink Note! 20322574, here is an example of the sequence of commands used in a previous resolution to a streams-cleanup problem: -
SQL> col OBJECT_NAME format a30
SQL> SELECT object_name, object_type FROM user objects WHERE object name like '%BRDB%';
OBJECT_NAME OBJECT _TYPE
AQ$_BRDB_CAPT QUEUE TABLE C TABLE
AQ$_BRDB_CAPT QUEUE TABLE Y INDEX
SQL> ALTER SESSION SET EVENTS '10851 trace name context forever, level 2';
Session altered.
A. I SQL> drop table AQ$ BRDB CAPT QUEUE _TABLE_C cascade constraints;
Table dropped.
SQL> SELECT object_name, object type FROM user objects WHERE object name like '%BRDB%';
no rows selected
SQL> select object_name, object_type from dba objects where owner = 'STRADMIN';
OBJECT _NAME OBJECT _TYPE
BRSS DATABASE LINK
SQL> exit
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN Ref: DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 156 of 183
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE ®
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
B. Item B is a section of this document which deals with a few scenarios of Streams Failure and Recovery.
BDB and BRS:
The OPS$BRDB schema on both BDB and I Export the OPS$BRDB schema on both databases:
BRS are not the same, in terms of structure, I exp system@brdb OWNER=ops\$brdb FILE=<BRDB Export>.dmp LOG=<BRDB Export>.log
for whatever reason and need to beI ROWS=N BUFFER=10485760 COMPRESS=N GRANTS=N STATIS? NONE
synchronised.
exp system@brss OWN!
ROWS=N_ BU:
ops\$brdb FILE=<BRSS Export>
=10485760 COMPRESS=N GRANTS=N STATIST
dmp LOG=<BRSS Export
S=NONE
Bt Repeat all steps until there are no structural I Create two users, e.g. BDB_COMP and BRS_COMP, on some other temporary database where it is safe to
. differences. When Streams is re-activated I perform an investigation. Then import the OPS$BRDB schema exports into the new schemas:
there can be no differences which will cause I imp system@brdb FILE=<BRDB Export>.dmp LOG=<BRDB_COMP Import>.log
data to error on insertion to BRS. FROMUSER=OPS\$BRDB TOUSER=BDB_COMP BUFFER=10485760 IGNORE=Y
t>.dmp LOG=<BRSS
COMP Import>.log
BUFFER=10 0
IGNORE=Y
6!
Now perform an investigation/comparison of both schemas. Produce a script to run, in order to synchronise
OPS$BRDB on BRSS with that of BRDB.
Be Complete Steps [5a.], [5b.], [Sc.J, [Sd.] and [Se.]
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN Ref. DES/APPISPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 157 of 183
(e)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
HOST BRANCH DATABASE SUPPORT GUIDE
CONFIDENCE)
FUJ00088797
FUJ00088797
BRS:
same.
WARNING!!!
STEP.
B3. continue...
Clean out all
The schemas should structurally now be the
DATA LOSS WILL OCCUR DURING THIS
lf the current failure of Streams was caused
by the schedules not being run and you're
sure that cleaning out partition and schedule
related data won't be a problem, then
partitions in OPS$BRDB, stored in the
OPS$BRSS schema, and correctly populate
the metadata, including setting the business
date to SYSDATE. su = brss
The relevant metadata tables are shown
below. Check them to see how the scripts
affect the metadata.
metadata related to the
From /app_sw/brss/build/schema/ ... run
pt_cleanup.sh BRSS
Query the tables below
© Copyright Fujitsu Ltd 2011
UNCONTROLLED IF PRINTED
FUJITSU RESTRICTED (COMMERCIAL IN Ref:
CONFIDENCE) Version:
Date:
Page No:
DES/APP/SPG/0001
3.0
21-Sep-11
158 of 183
2
FUJITSU
HOST BRANCH DATABASE SUPPORT GUIDE ®
FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
FUJ00088797
FUJ00088797
These tables include amongst others: -
BRSS_OPERATIONAL_EXCEPTIONS:
BRSS_PARTITION_CREATES:
BRSS_PARTITION_STATUS_HISTORY:
BRSS_SUBPARTITION_RANGES:
This shows the exceptions that may have occurred.
Shows partitions created and when. Order by table_name, partition_range_value.
Shows the history of partition maintenance. Order by table_name, partition_name.
Shows the range upper bound required at next run. Relevant column is range_value.
B4.
BRS:
Now, envoke the Start of Day process to
create the required partitions for “today” and
“tomorrow”, correctly updating all partition
metadata.
Check the metadata tables again, ensuring
the metadata is as you expect it to be.
su b:
From /app_sw/brss/build/schema/ .. run ..
B5.
BDB:
WARNINGIIT
DATA LOSS WILL OCCUR DURING THIS
STEP.
If and only if you're sure that there is a major
problem with the schedules or there is no
other way around the cleanup of partition
metadata, then do the same to BDB as in
[B3.].
From /app_sw/brdb/build/schema/ ... run
pt_cleanup.sh BRB
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN Ref. DES/APPISPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED
Page No: 159 of 183
HOST BRANCH DATABASE SUPPORT GUIDE ®
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
FUJ00088797
FUJ00088797
BDB:
BG Do the same for BDB as in [B4.]. su - brdb
From /app_sw/brdb/build/schema/ ... run .
BDB:
87. I Complete steps [6.] and [7] Complete Steps [6.] and [7.]
All done.
These are not acceptable and must be rectified.
See Section 5.5.4.4 for additional information on working with and resolving these errors.
Note: There may be errors detected on the application of Streams data on the “apply” queue [D
partitions. A very good example is the fact that a DELETE may occur on BDB, but the DML fails because the associated row does not exist on BRS.
], due to the mismatch of data and
Table 8: BRSS Support Procedure
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN Ref. DES/APPISPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 160 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
7.2 Managing Streams Lag
7.2.1. Context and Assumptions
Oracle Streams is in essence, a set of components which capture changes at a source database,
propagate those changes and then apply them on a target database. Oracle Streams, in most cases,
manages the capture-propagate-apply flow of changes fairly well, provided the flow of changes is applied
in good time.
Oracle Streams attempts to manage the flow of changes by putting the capture process into a state of
“flow control” or by getting the apply process to “spill” changes to disk and in most cases this ends up to
be a combination of both.
Flow control, simply means that the capture process stops capturing and put into a “paused” state until
the message which caused the pause is either applied or is spilled.
Problems with the rate that the changes are applied comes to bear if transaction sizes are: -
i. Too large, i.e. greater than 500,000 records.
ii. Last too long, i.e. longer than 5 minutes.
In these cases, the apply process will spill the LCRs (Logical Change Records) to disk, and will continue
to do so for that entire transaction, however long that might take. In addition the apply server (an apply
sub-process) that begins the spill and apply will have to complete it before any of the other processes
may continue — at great cost.
The assumptions for the rest of this section are therefore that the Streams process has had to either spill
transactions or is in a continual state of “PAUSED FOR FLOW CONTROL”, or both. In addition the
apply process has reached a period of unacceptable lag (see Section 7.2.2) and a decision has been
made to re-instantiate Streams.
This re-instantiation is assumed to save all data and the only part of the solution being recreated is
Streams and it's constituent components.
7.2.2 Lag Evaluation and Escalation
There are a number of factors to consider when deciding whether or not to re-instantiate Streams. The
escalation procedures will need to be followed in the first instance and a decision can be made,
considering the facts and reasons for the lag.
Our recommendation is that at the following periods the appropriate action is performed, bearing in mind
that as the solution matures, the responses might change or even the periods at which
escalation/investigation begins, might change: -
Lag Period Action
DBA Support notified in order to understand the transactions
4 hrs. responsible and continue to monitor apply progress.
DBA Support notified.
8 hrs. Are the original problems reoccurring? Is it the same or a
similar transaction?
DBA Support notified.
12 hrs Are the original problems reoccurring?
4'"-Line Support notified of the cause and progress.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 161 of 183,
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
16-20 DBA Support notified.
hrs. Are the original problems reoccurring?
DBA Support notified.
4"-Line Support notified.
Appropriate business owner notified.
At this point, there are x number of days (currently 4) which
remain in which to continue the investigation or to put in place
a fix and prepare for Streams re-instantiation, should the
24 hrs. decision be made to do so.
Note that x is defined as the lowest number of days for
data retention of any table on BRDB. The following query
shows the result:
SELECT MIN(retention_period)
FROM brdb_archived_tables
WHERE retention _period <> 0
AND additional_criteria IS NULL;
Re-evaluate the situation and prepare for re-instantiation
46 hs. providing all the approvals have been received.
7.2.3 Lag Assessment and Action Procedure
Step Descriptio Server Execution
Assumptions
User is logged onto the respective Database Server(s) as orac/e and into the database as SYSDBA.
BDB
After logging on to the database as SYSDBA.
Use Section 5.5.4.1 to check the state of the capture process.
[iii.] of the same section.
For capture latency with mining the logs then run [A.] below.
For capture latency with enqueuing messages then run [B.] below.
For capture memory usage then run [C.] below.
For long running transactions (greater than 10 mins), run [D.] below.
Ignore the capture of a particular transaction by following Step 4.
For a concise capture summary use Query [i.] of Section 5.5.4.5. For capture errors use Query
[A] Likely to change with every transaction ..
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 162 of 183,
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE @
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
CAPTURE_NAME,
((SYSDATE - CAPTURE_MESSAGE_CREATE TIME) *86400) LATENCY_SECONDS,
((SYSDATE - CAPTURE_TIME)*86400) LAST_STATUS,
TO_CHAR(CAPTURE_TIME, 'HH24:MI:SS MM/DD/Y¥Y') CAPTURE_TIME,
TO_CHAR(CAPTURE_MESSAGE_CREATE_TIME, 'HH24:MI:SS MM/DD/YY') CREATE_TIME
FROM VSSTREAMS CAPTURE
[B]
COLUMN JRE_NAME HEADING 'CaptureIProcessIName' FORMAT Al2
COLUMN ICY_SECONDS HEADING 'LatencyIinISeconds' FORMAT 999999
COLUMN CREATE _TIME HEADING ‘Message CreationITime' FORMAT A20
COLUMN ENQUEUE_TIME HEADING 'Enqueue Time’ FORMAT A20
COLUMN ENQUEUE_MESSAGE_NUMBER HEADING 'MessageINumber' FORMAT 999999999999
SELECT
CAPTURE_NAME,
(ENQUEUE_TIME-ENQUEUE_MESSAGE_CREATE_TIME) *86400 LATENCY_SECONDS
TO_CHAR (ENQUEUE_MESSAGE_CREATE_ 'HH24:MI:SS MM/DD/YY") CREA’
TO_CHAR(ENQUEUE_TIME, 'HH24:MI MM/DD/YY') ENQUEUE_TIME,
ENQUEUE_MESSAGE_NUMBER
FROM V$STREAMS_CAPTURE;
(c]
set pages 100
col “capture Process" for a25
col "capture Name" for a25
TIME,
SELECT
p.spid Spid,
*cOO"I Ic.capture#II' 'IIUPPER(1p.ROLE) "Capture Process",
c.capture_name "Capture Name",
p.pga_used_mem "PGA Memory Used",
p-pga_alloc_mem "PGA Memory Allocated",
p-pga_max_mem "PGA Maximum Memory"
FROM v$streams_capture c, v$logmnr_process lp, v$session s, v$process P
WHERE c.logminer_id = lp.session_id
AND lp.ROLE IN ('reader','preparer’, 'builder')
AND 1p.SID = s.SID
AND lp.serial# = s.serial#¥
AND s.paddr = P.addr
UNION
SELECT p.spid,
"cO0'IIc.capture#II' Coordinator’,
c.capture_name,
p.pga_used_mem,
p.pga_alloc_mem,
p-pga_max_mem
FROM v$streams_capture c, v$session s, vSprocess P
WHERE c.SID = s.SID
AND c.serial# = s.serial#
AND s.paddr = P.addr
ORDER BY 6,5;
DI
pl runlength HEAD 'Txns OpenIMinutes' format 9999.99
col sid HEAD 'Session' format al3
col xid HEAD 'TransactionIID' format al8
col terminal HEAD 'Terminal' format al0
col program ‘program’ format a27 wrap
SELECT t.inst_id,
sid II ',' II serial# sid,
xidusn II '.' II xidslot II '.' II xidsgn xid,
(SYSDATE - start_date) * 1440 runlength,
terminal,
program
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN DES/APP/SPG/0001
CONFIDENCE) r
21-Sep-11
UNCONTROLLED IF PRINTED 163 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
FROM gvStransaction t, gv$session 5
t.addr = s.taddr AND (SYSDATE - start_date) *
BDB
2. Use Section 5.5.4.2 to check the state of the propagation process.
For latency in propagation of new messages then run [E.] below.
Maximum Wait Time to Propagate a New Mess
MN START_DATE HEADING 'Start Date!
INDOW HEADING 'DurationIin Seconds’ FORMAT
iG ‘Next ITime!
'LatencyIlin Seconds' FORMAT 99999
HEADING 'Status' FORMAT A8
IAME HEADING 'Process' FORMAT AS
HEADING 'Number ofIFailures' FORMAT 99
Latency
(s.START_DATE, 'HH2
WINDOW,
:MI:SS MM/DD/YY") START_DAT!
. LATENCY,
ECODE (s . SCHE: E_DISABLED,
"y', "Disabl
D.
8
s.NEXT
DI
SCHEDU
HEDULES s, DBA_PROP
ATION NAME = 'BRDB_PROPG'
.DESTINATION_DBLINK DESTINATI
.SCHEMA = p.SOURCE_QUEUE_OWNER
-ONAME = p.SOURCE_QUEUE_NAME;
“Ay
i}
BRS
Use Section 5.5.4.3 to check the state of the apply process.
For Apply dequeuing info use Query [iv.] of Section 5.5.4.5.
3. For tracking the applying of transactions use Query [v.] of Section 5.5.4.5. And Use [vi.] for more
° detail of a particular transaction.
For latency from capture to dequeue at the apply, then run [F.] below.
For latency from capture to actual message apply, then run [G.] below.
For a better idea of the types and number of apply errors occurring, run [H.] below.
y ProcessIName' FORMAT Al7
IinISeconds' FORMAT 9999
ge Creation’ FORMAT Al7
"Last Dequeue Time! FORMAT A20
SAGE NUMBER HEADING 'DequeuedIMessage Number' FORMAT
999999999999
SELECT
APPLY_NAME,
(DEQUEUE_TIME-DEQUEUED_MESSAG!
TO_CHAR (DEQUEUED_MESS
TIME,
NUM
I CREATE_TIME)*86400 LATENCY,
I CREATE_TIME, 'HH24:MI MM/DD/YY¥') CREATION,
24:MI MM/DD/YY") LAST _DEQUEUE
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 164 of 183
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE @
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
ING 'Apply ProcessIName' FORMAT A17
in Seconds' FORMA 999
IADING 'AppliedIMessageINumber' FORMAT 999999
6400 "Latency in S
S MM/DD/YY") "Message
Apply Time",
applied_me
FROM dba_apply I
TO_CHAR (error_crea’
n_time, 'YYYY/MM/DD')
error_number,
1) error_count
FROM dba_apply_e
GROUP BY error_number,
TO_CHAR(error_creation_time, 'YYYY/MM/DD')
R BY 1, 2;
BDB
Once a transaction has been identified as problematic and can safely be assumed to not be a
problem if ignored, then the following steps will help in allowing the capture process to ignore that
transaction. This is providing of course that the transaction has not already begun to be mined by
4. the capture process.
Identifying the problematic/long running transaction can be done by running query [D.] of step 1 of
this procedure.
Then run the steps which follow as STRADMIN. Note that <rtxn_ID> below refers to the actual
transaction id of the transaction you wish to ignore.
EXECUTE dbms_capture_adm.stop_capture('BRDB_CAPTURE') ;
EXECUTE dbms_capture_adm.set_parameter('BRDB_CAPTURE',
'_ignore_transaction', '<TXN_ID>');
EXECUTE dbms_capture_adm.start_capture ('BRDB_CAPTURE') ;
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 165 of 183,
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
7.2.4 Post Lag Action Procedure
Step Description Server Execution
Assumptions
- User is logged onto the respective Database Server(s) as oracle and where possible as SYSDBA
- The TWS schedules can continue to run and won't need to be stopped during this procedure.
- “Partition Managed Tables” simply means those tables managed by BR/DB I SS/C001.
Applicable Scenario
The Branch Support Database has been determined to have unacceptable levels of lag and a decision has
been made to re-instantiate Streams.
The scenario is as follows
- The decision to re-instantiate Streams was made “TODAY”
- Streams has been lagging for less than 24 hrs.
- The data for BRSS partitions of date “TODAY” are lagging.
- The data for BRSS partitions of date “YESTERDAY” have replicated without issue.
This procedure should therefore be executed approximately 30 — 15 minutes before midnight “TODAY”
(before."TOMORROW’") and Step’ 4a MUST be complete before 00700 (i.e. before ‘TOMORROW’).
NOTE
Throughout this procedure, “YESTERDAY”, “TODAY” and “TOMORROW’ will remain specific dates in order
to avoid confusion, e.g. 28", 29" and 30" even though this procedure, in reality, will be used before and
after midnight and therefore create confusion..
BDB and BRS
Re-instantiate Streams by following the methodology presented in Section 7.1.2, executing Steps
fa. 5a.-7
This will recreate the Streams Configuration and will not affect the data that has already been
mined.
1b. Following from 1a. above, Steps 8 — 11 of Section 7.1.2, must follow as soon as possible!
BDB
Determine whether the partitions created on BRDB are the same as those on BRSS. This should
2. always be the case, but double-check.
There should be no rows returned ... (i.e. the output to the first sub-query should be the same as
the second).
Please run just these queries as STRADMIN: -
T , MAX(dtp.partition_name) max_part_name
(+)
‘© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN Ref: DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 166 of 183,
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE @
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
sdtp.table_name, MAX(sdtp.partition_name) max_part_name
dba_tab_partitions@brss sdtp,
ops$brss.brss_pa:
ss.brss_pa
.table_name
.table_name
-table_owner = 'OPS$BRDB'
1. Determine the list of partitions to export from BRDB (for a later import into BRSS)
2. Perform the export(s) — ensure you have the password for SYSTEM
3. Transport the export dump files to /prpbrs001
Find a directory that has enough space to store the export dump files. The create a directory
object as follows (SQL*Plus): -
CREATE OR REPLACE DIRECTORY <dump_directory name> AS '/<dump_directory>';
Create a parfile (vi <parfile_name>.par) with the output from the following query (SQL*Plus): -
set pages 0
set trimspool ON
set feedback off
set echo off
SELECT DECODE (ROWNUM, 1, 'CONTENT=ALL
TABLES=(' I ICHR(39) I Idtp.table_ownerII'.'IIdtp.table_nameII'.'IIdtp.partition_nam
eI ICHR(39)
,18,','IICHR(39) I Idtp.table_ownerII'.'IIdtp.table_nameII'.'I Idtp.partition_nameI
ICHR(39) II") *
,',' I {CHR (39) I Idtp.table_ownerII'.'IIdtp.table_nameII'.'IIdtp.partition_nameI ICH
R(39)) ~ ~ ~
FROM dba_tab_partitions dtp,
brdb_partition_creates bpc,
brdb_partitioned_tables brt
WHERE dtp.table_name = bpc.table_name (+)
AND dtp.table_name = brt.table_name (+)
AND dtp.table_owner = 'OPS$BRDB*
AND bpc.partition_range_value = TO_CHAR(TRUNC (SYSDATE) , 'YYYYMMDD')
AND dtp.partition_name =
brt.partition_root_nameII‘_'IIbpc.partition_range_value;
Export the tables by using the following datapump command, substituting the required values: -
expdp system DIRECTORY=<dump directory name> PARFILE=<parfile_name>.par
DUMPFILE=<dumpfile name>.dmp LOGFILE=<logfile_name>.log
Remote copy or FTP the <dumpfile_name>.dmp dump file to the BRS server. The quickest way
to accomplish this may be to create an NFS or NAS mount between the servers.
BRS - Partition Managed Tables ONLY
4. 1. Truncate the partitions for “TODAY” only, using the SQL provided.
2. Import the partitions exported from BRDB (see Step 3 above) for “TODAY” only.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 167 of 183,
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
defined in later version
command to k d in later version
BDB - Non-partition Managed Tables ONLY
1. Merge the data from the Branch Database into those of the Branch Support Database.
BDB
1. Run a comparison between the two database schemas.
table name, partition_name, num_rows,
M dba_tab_partitions W
7.3 Streams DML Behaviour on OPS$BRDB
Tables
The diagram in Section 7 shows an overview of the Oracle Streams technology by which the data is
replicated from BRDB to BRSS, the processes involved and the action performed by each.
The table below is for informational purposes and has been included to aid in determining why certain
problems with data in BRSS might occur, e.g. data seems to have “disappeared” or is the cause of
Streams errors theoretically based on an assumption by a user that the deletion of data in BRDB
succeeded therefore it must also have succeeded in BRSS. This will not be the case for tables which
Streams is configured to discard deletes for (see below).
The tables below have Streams DML Handlers defined for them. DML Handlers are created for a
number of reasons, however the main reasons used in our solution are for the discarding of DELETE
LCRs and in addition, dealing with LOB data. The processing of LOB data is a complex task due to the
way LOBs are stored (e.g. LOB locators, etc.) and therefore, to make this processing more efficient LOB
Assembly is employed. If a table has a LOB defined, it should have a DML Handler that allows all DML
to be processed and if in addition DELETEs are “not allowed” on BRSS, then the Handler will be
configured to discard the DELETEs.
Note: This table applies to OPS$BRDB tables in BRSS.
Table Name DML Apply Action Perform LOB Assembly
BRDB_BRANCH_NODE_INFO. Allows All DML. Yes
BRDB_EXT_FEED_REPORTS Allows All DML Yes
BRDB_HOST_INTERFACE_FEED_EXCP Discards Deletes Yes
BRDB_HYDRA_EXCEPTIONS. Discards Deletes No
BRDB_LAST_POST_INDICATOR Discards Deletes No
BRDB_OPERATIONAL_EXCEPTIONS Discards Deletes No
BRDB_PARTITION_CREATES Discards Deletes No
BRDB_PARTITION_STATUS_HISTORY Discards Deletes No
BRDB_PROCESS_AUDIT Discards Deletes No
BRDB_PROCESS_CONTROL Discards Deletes No
BRDB_RX_GUARANTEED_REVERSALS Allows All DML. No
BRDB_RX_MESSAGE_JOURNAL Allows All DML. Yes
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref: DES/APP/SPG/0001
CONFIDENCE) Version 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 168 of 183
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
FUJ00088797
FUJ00088797
BRDB_RX_NRT_TRANSACTIONS Discards Deletes
BRDB_RX_RECOVERY_TRANSACTIONS Allows All DML.
BRDB_RX_REPORT_REPRINTS Allows All DML.
BRDB_TXN_CORR_TOOL_JOURNAL Allows All DML
LFS_PLO_DETAILS Discards Deletes
LFS_PLO_HEADER Discards Deletes
LFS_RDC_DETAILS Discards Deletes
LFS_RDC_HEADER Discards Deletes
RDDS_CHECKSUM Discards Deletes
RDDS_CHECKSUM_HIST Discards Deletes
RDDS_DELIVERY Allows All DML.
TPS_HYDRA_INDAY_DATA Allows All DML.
TPS_HYDRA_RECON_TOTALS. Allows All DML
Yes
Yes
Yes
Yes
No
No
No
No
No
No
Yes
Yes
Yes
An exception to the above “rule” that a// tables owned by OPS$BRDB in the BRSS database which have
LOB columns should have a DML Handler defined is untrue for the following tables because they belong
to Branch Cleardown functionality and are not replicated from BRDB but contain local content: -
C_BRDB_HOST_INTERFACE_FEED_EXC
C_BRDB_RX_MESSAGE_JOURNAL
C_BRDB_RX_NRT_TRANSACTIONS
C_BRDB_RX_RECOVERY_TRANSACTION
C_BRDB_RX_REPORT_REPRINTS
C_BRDB_TXN_CORR_TOOL_JOURNAL
7.4 Data Aggregations
Host Data Aggregation modules in the Branch Database have been cloned and implemented in BRSS as
part of HNG-X Release 5 CP0639 — Capacity Management Reporting. This is the first step in moving
towards the product steer and roadmap of having all management reporting information generated out of
BRSS and, thereby, doing away with the need for Data Warehouse Host system.
Except for minor customisations done to localise the modules in BRSS Database, the data aggregation
related database objects, LINUX shell scripts and TWS schedule job definitions will almost entirely
resemble their counterparts in BRDB. It has to be noted that the Data Aggregation processes in BRSS
will not perform Instance ID/Fad Hash based processing as it is not applicable to BRSS, which is a single
instance Database as of HNG-X Release 5 implementation.
The following tables have been created in BRSS Database to contain aggregation metadata and report
statistics for Capacity Management Reporting:
- BRSS_HOST_AGGREGATIONS
- BRSS_HOST_AGGREGATION_CTL
- BRSS_CAPMGMT_SMIN_STATS
- BRSS_CAPMGMT_HOURLY_STATS
- BRSS_CAPMGMT_DAILY_STATS
7.5 Table of BRSS Host Processes
The following table lists the current BRSS Host processes, a brief description of each and the names of
the executables used to run them. The process name corresponds to the name that is registered in table
BRSS_PROCESSES and, where applicable, the name that is used to control processing via table
BRSS_PROCESS_CONTROL.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIAL IN Ref:
CONFIDENCE) Version:
Date:
UNCONTROLLED IF PRINTED Page No’
DES/APP/SPG/0001
21-Sep-11
169 of 183
FUJ00088797
FUJ00088797
Fe) HOST BRANCH DATABASE SUPPORT GUIDE @
FUIITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
No. Executable Description
1 BRSSCO01 BRSSCO001 Start of Day
2 BRSSC004 BRSSC004 Audit, Archive, Purge
3 BRSSX002.sh BRSSX002 BRSS Message Journal Auditing
4 BRSSX005.sh BRSSX005.sh Gather Optimiser Statistics
5 BRSSX006.sh BRSSX006 File Housekeeping
6 BRSSX007.sh SLT_TO_5MIN_STATS Data aggregation for Cap Mgmt Reporting -
Peak 5-Minute Stats for HNG-X RAW SLT
STATS.
7 BRSSX007.sh SETTLEMENT_TO_5MIN_STATS. Data aggregation for Cap Mgmt Reporting -
Peak 5-Minute Stats for Settlement
transactions
8 I BRSSXOO07.sh NRT_TO_5MIN_STATS Data aggregation for Cap Mgmt Reporting -
Peak 5-Minute Stats for NRT transactions
9 BRSSX007.sh 5MIN_TO_HOURLY_STATS Data aggregation for Cap Mgmt Reporting -
Peak Hourly Stats
10 I BRSSX007.sh HOURLY_TO_DAILY_STATS Data aggregation for Cap Mgmt Reporting -
Peak Daily Stats
1 BRSSX021.sh BRSSX021 Streams Pause, Start
12 BRSSX022.sh BRSSX022 Daily copy of DBA_HIST tables from
BRDB into BRSS
13, BRSSX023.sh BRSSX023 Pre-processor job for GREPX001
14 GREPX001.sh GREPX001 Generic Reporting Mechanism - view
creation
15 GREPX002.sh GREPX002 Generic Reporting Mechanism - report
extraction
16 BRSSX037.sh BRSS_CLR_BRANCH_DATA. BRSS Branch closure clear down
7.6 BRSS Scheduling
7.6.1 Schedule BRSS_TRACE_STOP1
This schedule is run daily (07:30 a.m.).
7.6.1.1 Dependencies
None.
7.6.1.2 Job BRSSX011_TRACE_PAUSE_1
Updates the BRSS_SYSTEM_PARAMETERS table, sets parameter BRSS_C002_STOP_YN flag to
7.6.1.2.1 Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name
and parameter.
7.6.1.2.2 Rerun Action
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 170 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
7.6.2 Schedule BRSS_SOD
This schedule is run daily (08:00 a.m.).
7.6.2.1 Dependencies
Flag in " /opt/tws/FLAGS/BRSS_COMPLETE. flag" present.
7.6.2.2 Job BRSS_RM_COMPLETE_FLAG
Removes BRSS_COMPLETE. flag.
7.6.2.2.1 Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name
and parameter.
7.6.2.2.2 Rerun Action
#* Prompts for rerun = action?
7.6.3 Schedule BRSS_CLR_BRANCH
This schedule runs from 9pm but only after BRSS_SOD and BRDB_FROM_EMDB complete and is
stopped at 01:05. The called job archives and then deletes transactions for all closed branches. This
schedule is run on 1 instance at any one time.
7.6.3.1 Dependencies
Schedule BRSS_CLR_BRANCH depends on the completion of schedules BRSS_SOD and
BRDB_FROM_EMDB. This job is stopped at 01:05 irrespective of whether it has completed already
(outstanding transactions will be rolled back and picked up the following night).
7.6.3.2 Job BRSSX037_CLEAR_BRDATA
This job runs the BRSS automated closure process (BRSSX037.sh).
7.6.3.2.1 Implementation
This job is implemented by a call to the shell script BRSSX037.sh, along with the TWS business date
and instance number.
The process identifies all branches to be cleared by the following query
de
T branch_ ting,
closure data
cleare
HERE cleared_date IS NULL
All transactions for those closed branches in a number of tables (identified in column
BRDB_CLEARED_CONTROL_DATA.source_table) are loaded into archive tables (identified in column
BRDB_CLEARED_CONTROL_DATA.target_table) and then deleted from the original tables.
Closed, cleared and archived branches are recorded in table BRDB_CLEARED_CLOSURE_DATA, with
column brss_cleared_date identifying when the branch was cleared on BRSS.
7.6.3.2.2 Rerun Action
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 171 of 183,
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
7.6.4 Schedule BRSS_TRACE_STRT1
This schedule is run daily (at 8:10). Allows BRSSC002 to restart by resetting the start/stop flag.
7.6.4.1 Dependencies
Schedule BRSS_TRACE_STRT1 depends on the completion of schedule BRSS_SOD.
7.6.4.2 Job BRSSX011_TRACE_RESUME
Updates the BRSS_SYSTEM_PARAMETERS table, sets parameter BRSS_C002_STOP_YN flag to 'N'.
7.6.4.2.1 Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name
and parameter.
7.6.4.2.2 Rerun Action
Prompts for rerun’ = action?
7.6.5 Schedule BRSS_JRNL_TRACE1
This schedule is run daily.
7.6.5.1 Dependencies
Schedule BRSS_JRNL_TRACE1 depends on the completion of schedule BRSS_TRACE_STRT1.
7.6.5.2 Job BRSSC002_JRNL_TRACE1
The message journal tracing process (BRSSC002) will generate text files for a given day's journalised
messages by reading records from the message journal table (BRDB_RX_MESSAGE_JOURNAL). The
process will run throughout the day as a Unix daemon. This process is essentially a clone of BRDBC002
without the check that sequence numbers are a dense set.
7.6.5.2.1 Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name
and date.
Outputs files to the following directory below.
Usage Environment Variable
BRSS output directory
7.6.5.2.2 Rerun Action
5 Prompts for rerun = action?
7.6.6 Schedule BRSS_DXC
This schedule is run daily (??:??).
7.6.6.1 Dependencies
Schedule BRSS_DXC depends on the completion of schedule DW_EOD.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 172 of 183,
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
7.6.6.2 Job BRSS_DXC_RUN
This job is used to transfer Reporting information from the BRSS environment (specifically a NAS share
named, /app/brss/trans/support/sltreports) to the “Corporate” environment. This is
accomplished by executing a DXC java client which invokes a “transfer plan”, allowing the contents of
the above directory to be copied to “Corporate” via the DXC.
Please note that this job is not owned by Host development.
7.6.6.2.1 Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name
and input parameters as shown: /app_sw/dxc/executedxc.sh upload BRSSMSUOUTPUT
7.6.6.2.2 Rerun Action
#% Prompts for rerun’= action?
7.6.7 Schedule BRSS_GEN_REP
This schedule is run daily. Every 5 hours until 0700 hrs.
7.6.7.1 Dependencies
Schedule BRSS_GEN_REP depends on the completion of schedule BRSS_SOD.
7.6.7.2 Job GENERIC_CREATE_REPORT_VIEWS
Calls shell script BRSSX023.sh with the TWS business date.
7.6.7.2.1 Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name
and date. The shell script BRSSX023.sh will in turn, call shell script GREPX001.sh (and subsequent
aggregation jobs) depending upon the outcome of the validation performed between
BRSS_C002_JOURNAL_DATE and REP_EFFECTIVE_DATE. This validation ensures that if the date
values of these two parameters are equal, that the chain iof dependent jobs is executed, otherwise
BRSSX023.sh does not run..
7.6.7.2.2 Rerun Action
# Prompts for rerun = action?
7.6.7.3 Job BRSSX007_SLT_TO_5MIN_STATS
Calls shell script BRSSX007.sh with aggregation name ‘SLT_TO_5MIN_STATS’ and the TWS business
date. Data aggregation performed by this job will be used for Capacity Management Reporting
requirements of Customer Services.
7.6.7.3.1 Dependencies
Job BRSSX007_SLT_TO_5MIN_STATS. depends on the completion of job
GENERIC_CREATE_REPORT_VIEWS.
7.6.7.3.2 Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name,
aggregation name and date.
7.6.7.3.3 Rerun Action
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 173 of 183,
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
7.6.7.4 Job BRSSX007_SETTLEMENT_TO_5MIN_STATS
Calls shell script BRSSX007.sh with aggregation name ‘SETTLEMENT_TO_5MIN_STATS' and the TWS
business date. Data aggregation performed by this job will be used for Capacity Management Reporting
requirements of Customer Services.
7.6.7.4.1 Dependencies
Job BRSSX007_SETTLEMENT_TO_5MIN_STATS depends on _ the completion of job
BRSSX007_SLT_TO_5MIN_STATS.
7.6.7.4.2 Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name,
aggregation name and date.
7.6.7.4,.3 Rerun Action
#8 Prompts for rerun = action? 4
7.6.7.5 Job BRSSX007_NRT_TO_5MIN_STATS
Calls shell script BRSSX007.sh with aggregation name ‘NRT_TO_5MIN_STATS’ and the TWS business
date. Data aggregation performed by this job will be used for Capacity Management Reporting
requirements of Customer Services.
7.6.7.5.1 Dependencies
Job BRSSX007_NRT_TO_5MIN_STATS depends on the completion of job
BRSSX007_SETTLEMENT_TO_5MIN_STATS.
7.6.7.5.2 Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name,
aggregation name and date.
7.6.7.5.3 Rerun Action
8 Prompts for renin = action? 4
7.6.7.6 Job BRSSX007_5MIN_TO_HOURLY_STATS
Calls shell script BRSSX007.sh with aggregation name ‘SMIN_TO_HOURLY_STATS’ and the TWS
business date. Data aggregation performed by this job will be used for Capacity Management Reporting
requirements of Customer Services.
7.6.7.6.1 Dependencies
Job BRSSX007_5MIN_TO_HOURLY_STATS depends on the completion of job
BRSSX007_NRT_TO_SMIN_STATS.
7.6.7.6.2 Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name,
aggregation name and date.
7.6.7.6.3 Rerun Action
#9 Prompts for terun’= action?
7.6.7.7 Job BRSSX007_HOURLY_TO_DAILY_STATS
Calls shell script BRSSX007.sh with aggregation name ‘HOURLY_TO_DAILY_STATS’ and the TWS
business date. Data aggregation performed by this job will be used for Capacity Management Reporting
requirements of Customer Services.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 174 of 183,
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
7.6.7.7.1_ Dependencies
Job BRSSX007_HOURLY_TO_DAILY_STATS depends on the completion of job
BRSSX007_5MIN_TO_HOURLY_STATS.
7.6.7.7.2 Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name,
aggregation name and date.
7.6.7.7.3 Rerun Action
SP) Prompts for'reruin = action?
7.6.7.8 Job GENERIC_CREATE_REPORTS
Calls shell script grepx002.sh with the system name (BRSS), outputs text based report files.
Outputs files to the following directories below.
Usage BRDBBLV1 Environment Variable
Working directory BRSS_MSU_WORKING
BRSS reports directory BRSS_MSU_OUTPUT
7.6.7.8.1 Dependencies
Job GENERIC_CREATE_REPORTS. depends on the completion of job
BRSSX007_HOURLY_TO_DAILY_STATS.
7.6.7.8.2 Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name
and parameter.
7.6.7.8.3 Rerun Action
2 Prompts for rerun = action?
7.6.8 Schedule BRSS_ORA_STATS
This schedule is run daily (01:05).
7.6.8.1 Dependencies
Schedule BRSS_ORA_STATS depends on the completion of schedule BRSS_SOD.
7.6.8.2 Job BRSSX005_SCHEMA
Gathers statistics on all objects within the OPS$BRSS and OPS$BRDB schemas.
7.6.8.1.1 Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name
and date.
7.6.8.1.2 Rerun Action
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 175 of 183,
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
7.6.9 Schedule BRSS_ADMIN
This schedule is run daily (01:15).
7.6.9.1 Dependencies
Schedule BRSS_ADMIN depends on the completion of schedule BRSS_SOD.
7.6.9.2 Job BRSSC004
Calls binary BRSSC004 to housekeep BRSS.
7.6.9.2.1 Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name
and date.
7.6.9.2.2 Rerun Action
He Prompts for rerun = action? 4
7.6.9.3 Job BRSSX022
Calls shell script BRSSX022.sh to copy AWR statistics from BRDB into BRSS tables with names starting
“HIST_BRDB_’. BRDBX022.sh then calls the procedure ops$brss.hist_brdb_ refresh, which
copies the tables in the order specified below: -
1] HIST_BRDB_SYS_TIME_MODEL
2] HIST_BRDB_SYSSTAT
3] HIST_BRDB_SYSTEM_EVENT
4] HIST _BRDB_SQLSTAT
6] HIST _BRDB_SNAPSHOT
5] HIST_BRDB_SQLTEXT
7] HIST_BRDB_ACTIVE_SESS_HISTORY
8] HIST_BRDB_SGASTAT
{ 9] HIST_BRDB_SQL PLAN
(10] HIST_BRDB_OPTSTAT_HSTHEAD_HST
(11] HIST_BRDB_OPTSTAT_TAB_HISTORY
[12] HIST_BRDB_OPTSTAT_IND_HISTORY
[13] HIST_BRDB_OPTSTAT_HISTGRM_HST
The stats have the potential to be copies the tables in the order specified below: -
7.6.9.3.1 Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name
and date.
7.6.9.3.2 Rerun Action
A re-run is not required, nor recommended. Mark job complete. Work will complete next time job is run.
7.6.9.4 Job BRSSX006
Calls binary BRSSX006 to housekeep BRSS directories.
7.6.9.4.1 Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name
and date.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 176 of 183,
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
7.6.9.4.2 Rerun Action
#9? Prompts for renin = action? 4
7.6.9.5 Job BRSS_HkP_Orafiles1
Calls script HousekeepOrafiles.sh to housekeep Oracle files.
7.6.9.5.1 Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name
and parameter.
7.6.9.5.2 Rerun Action
Pe Prompts for rerun = action? 4
7.6.9.6 Job BRSS_HkP_Orafiles2
Calls script HousekeepOrafiles.sh to housekeep Oracle ASM files.
7.6.9.6.1 Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name
and parameter.
7.6.9.6.2 Rerun Action
#8 Prompts for rerun = action?
7.6.10 Schedule BRSS_START_BKP
This schedule is run daily (with an alert if not started by 04:00).
7.6.10.1 Dependencies
Schedule BRSS_START_BKP depends on the completion of schedule BRSS_ADMIN.
7.6.10.2 Job MARKER
Writes marker.
7.6.10.2.1Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name
and date.
7.6.10.2.2Rerun Action
CONTINUE
7.6.11 Schedule BRSS_BACKUP_0
This schedule is run every 4th Sunday.
7.6.11.1 Dependencies
Schedule BRSS_BACKUP_0 depends on the completion of schedule BRSS_START_BKP.
7.6.11.2 Job BRSS_LVLO_BACKUP
Carries out level O RMAN backup.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 177 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
7.6.11.2.1Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name
and parameters.
7.6.11.2.2Rerun Action
28 Prompts for rerun ~ action?
7.6.12 Schedule BRSS_BACKUP_1
This schedule is run daily except 4th Sunday.
7.6.12.1 Dependencies
Schedule BRSS_BACKUP_1 depends on the completion of schedule BRSS_START_BKP.
7.6.12.2 Job BRSS_LVL1_BACKUP
Carries out level 1 RMAN backup.
7.6.12.2.1Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name
and parameters.
7.6.12.2.2Rerun Action
Prompts for rerun = action?
7.6.13 Schedule BRSS_STARTUP
This schedule is run daily (raises alert if not started by 06:00).
7.6.13.1 Dependencies
Schedule BRSS_STARTUP depends on the completion of schedule BRSS_BACKUP_O or
BRSS_BACKUP_1.
7.6.13.2 Job BRSSC001
Calls start of day process BRSSC001 to generate the next day's partitions.
7.6.13.2.1Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name
and parameters.
7.6.13.2.2Rerun Action
2 Prompts for rerun = action?
7.6.14 Schedule BRSS_COMPLETE
This schedule is run daily.
7.6.14.1 Dependencies
Schedule BRSS_COMPLETE depends on the completion of schedules BRSS_STARTUP,
BRSS_TRACE_STOP1 and BRSS_GEN_REP.
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 178 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
7.1.14.2 I Job BRSS_COMPLETE_FLAG
Creates complete flag.
7.1.14.2.1Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name
and parameters.
7.1.14.2.2Job Dependency
This job is dependent on job BRSSC001.
7.1.14.2.3Rerun Action
Prompts for terun = action?
7.1.15 Schedule BRSS_MONITOR
This schedule is run daily.
7.1.1.1 Dependencies
None
7.1.1.2 I Job BRSS_MON_STARTUP
Calls maestro script monitor_schedule.sh
7.1.1.2.1 Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name
and parameters.
7.1.1.2.2 Rerun Action
CONTINUE
7.1.1.3 Job BRSS_MON_BKP
Calls maestro script monitor_schedule.sh
7.1.1.1.1 Implementation
This job is implemented by a call to the Maestro monitor schedule command with the relevant job name
and parameters.
7.1.1.1.2 Rerun Action
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 179 of 183
FUJ00088797
FUJ00088797
HOST BRANCH DATABASE SUPPORT GUIDE @
Fe)
FUJITSU FUJITSU RESTRICTED (COMMERCIAL IN
CONFIDENCE)
8 Appendix C — Transaction Correction Templates
Section 5.6.1 describes the use of the transaction correction tool BRDBX015.sh. This is used by SSC to
correct transactions by inserting balancing records to transactional/accounting/stock tables in the BRDB
system. The tool must be supplied with a file containing a SQL statement that performs the required
insert. This statement must be of a particular form, and should be based on one of the templates listed
here.
Separate templates are given for each given target table, which reflect the columns of the target table.
8.1 Templates
The following templates are available on the live estate in /app/brdb/trans/support/brdbx015/input
Table to Correct Template File
BRDB_RX_REP_SESSION_DATA brdb_rx_rep_session_data.file
BRDB_RX_REP_EVENT_DATA brdb_r_rep_event_data file
BRDB_RX_NWB_TRANSACTIONS brdb_rx_nwb_transactions.file
BRDB_RX_EPOSS_TRANSACTIONS I brdb_rx_eposs_transactions.file
BRDB_RX_EPOSS_EVENTS brdb_rmx_eposs_events. file
BRDB_RX_DCS_TRANSACTIONS brdb_rm_des_transactions.file
BRDB_RX_CUT_OFF_SUMMARIES I brdb_rx_cut_off_summaries.file
BRDB_RX_BUREAU_TRANSACTIONS I brdb_rx_bureau_transactions file
BRDB_RX_APS_TRANSACTIONS brdb_rx_aps_transactions.file
<End of document>
© Copyright Fujitsu Ltd 2011 FUJITSU RESTRICTED (COMMERCIALIN Ref. DES/APP/SPG/0001
CONFIDENCE) Version: 3.0
Date: 21-Sep-11
UNCONTROLLED IF PRINTED Page No: 180 of 183,