FUJ00117504
FUJ00117504
co VOLUME AND PERFORMANCE TEST REPORT - PHASE 1
FUJITSU COMMERCIAL IN CONFIDENCE
Document Title: VOLUME AND PERFORMANCE TEST REPORT - PHASE 1
Document Reference: TST/SOT/REP/0008
Release: INT14
Abstract: This is the Volume and Performance Test Report for Phase 1
conducted on the V&l rig up to release 14.15.03
Document Status: APPROVED
Author & Dept: Michael Welch
External Distribution: None
Security Risk YES — No Risk
Assessment Confirmed
Approval Authorities:
lame Role Date
Alan D'Alvarez Programme Manager See Dimensions for record
Debbie Richardson Test & Integration Manager See Dimensions for record
Lee Farman Test Manager (POL JTT) See Dimensions for record
See HNG-X Reviewers/Approvers Matrix (PGM/DCM/ION/0001) for guidance on who should approve.
Ref: TST/SOT/REP/0008
© Copyright Fujitsu Services Ltd 2009 COMMERCIAL IN CONFIDENCE Version, 1.0
Date: 16-Nov-2009
UNCONTROLLED IF PRINTED Page No: 1 of 20
FUJ00117504
FUJ00117504
oO VOLUME AND PERFORMANCE TEST REPORT - PHASE 1
FUJITSU COMMERCIAL IN CONFIDENCE
0 Document Control
0.1 Table of Contents
DOCUMENT CONTROL
Table of Contents
Document History
Review Details ..
Associated Documents (Internal & External)
Abbreviations and Terms
Glossary.
Changes Expected ..
Accuracy ....
Security Risk Assessment
INTRODUCTION...
ie Io
SISISISISISISE
SISISISISIEIS
ic
Ko
I=
In
3 MANAGEMENT SUMMARY..
3.
3.1 Test Case Coverage
3. Defects
Defects Raise
Open Defects
4 TEST CASE DETAILG...........
4,
4.4 Test Group H01 — Horizon Volume...
Horizon Volume (Monday) — State 1
4
-2 Horizon Volume (Monday) — State 2
4.2 Test Group H10 — Horizon T&T......
Horizon T&T Contracted Peak Hour / Peak 5 Minutes
43 Test Group H13 — PAF......
PAF Contracted / Design Peak Hour / Peak 5 Minutes .
Test Group HCB — Horizon Combined Bankins
4.4.4 Horizon Combined Banking Contracted Peak Hour
4.4.2 Horizon Combined Banking Contracted Peak 5 Minutes
4.5 Test Group HPC — Horizon PCI Combined Banking & Debit Card
4.5.1 Horizon PC! Combined Banking & Debit Card Design Peak Hour ..
4.5.2 Horizon PC! Combined Banking & Debit Card Design Peak 5 Minutes .
4.6 Test Group XCAL —- HNG-X Calibration Test
4.6.1 HNG-X Calibration — Simple EPOSS
4.6.2 HNG-X Calibration — Basket Mix...
109 1p
ip
ne
i?
we
ip
hal tad
5 RECOMMENDATIONS 20
6 CONCLUSIONS... +20
Ref: TST/SOT/REP/0008
© Copyright Fujitsu Services Ltd 2009 COMMERCIAL IN CONFIDENCE Veniom 40
Date: 16-Nov-2009
UNCONTROLLED IF PRINTED Page No: 2 of 20
FUJ00117504
FUJ00117504
co VOLUME AND PERFORMANCE TEST REPORT - PHASE 1
FUJITSU COMMERCIAL IN CONFIDENCE
0.2 Document History
Version No. Date Summary of Changes and Reason for Iss Associated Change -
CP/PEAK/PPRR
Reference
0.1 29-Sep-2009 _I First draft for review by Joint Test Team
1.0 16-Nov-2009 Final draft for Approval.
Changes since last revision:
3.0 — Correction to table for Test Grp MCB
3.2 — Updated table of Open Defects following Peak re-tests
since previous draft
4.5 ~ Clarification of the number of active OSR services used
for the tests.
0.3 Review Details
See HNG-X Reviewers/Approvers Matrix (PGM/DCM/ION/0001) for guidance on completing the lists below. You
may include additional reviewers if necessary, but you should generally not exclude any of the mandatory reviewers
shown in the matrix for the document type you are authoring.
Review Comments by
Review Comments to
Mandatory Review
Role Name
Programme Manager Alan D’Alvarez
Test & Integration Manager Debbie Richardson
Test Manager (POL JTT) Lee Farman
Solution Design/Development Adam Cousins (Counter & BALIOSR)
Solution Design/Development David Harrison (Legacy Web Svcs, Agents & Audit, Crypto Host &
Ref Data)
Role Name
Test Manager Chris Young (POL JTT)
RV Manager James Brett (POL JTT)
VI & TE Manager Mark Ascott
Systems Qualities Architect David Chapman
Lead Architect David Johns
SV&l Manager Sheila Bamber
Issued for Information — Please restrict this
distribution list to a minimum
Position/Role Name
Ref: TST/SOT/REP/0008
© Copyright Fujitsu Services Ltd 2009 COMMERCIAL IN CONFIDENCE Version. 1.0
Date 16-Nov-2009
UNCONTROLLED IF PRINTED Page No: 3 of 20
FUJ00117504
FUJ00117504
oO VOLUME AND PERFORMANCE TEST REPORT - PHASE 1
FUJITSU COMMERCIAL IN CONFIDENCE
(*) = Reviewers that returned comments
0.4 Associated Documents (Internal & External)
Reference Version Date Title Source
[1] 5.0 03 June 2009 RMGA HNG-X Generic Document Dimensions
PGM/DCM/TEM/0001 Template
(DO NOT REMOVE)
[2] 1.0 12-September-2008 I HNG-X Volume and Performance Dimensions
TST/SOT/HTP/0003 High Level Test Plan
13] Horizon Capacity Management and PVCS
PA/PER/O33 Business Volumes
(4) ‘System Qualities Architecture Dimensions
ARC/PER/ARC/0001
Unless a specific version is referred to above, reference should be made to the current approved
versions of the documents.
0.5 Abbreviations and Terms
Abbreviation Definition
BRDB Branch Database
DAT Main Host Database
DC Data Centre
Dcs Debit Card Service
DRS Data Reconciliation Service
DVLA Driver Vehicle Licensing Authority
EOD BLOB Attachment to End Of Day messages generated at a Horizon counter containing data
pertaining to stock unit and product information
EPOSS Electronic Point Of Sale Service
ETU Electronic Top-Up
HLTP. High Level Test Plan
LREC Link Reconciliation File
NTSS Non Test Stream Specific
PAF Postal Address File
PCI The Payment Card Industry Data Security Standard
T&T Track & Trace
TPS Transaction Processing System
© Copyright Fujitsu Services Ltd 2009 COMMERCIAL IN CONFIDENCE er on Pe SOTIRER O08
UNCONTROLLED IF PRINTED pate 16-Nov-2009
PageNo: 4 of 20
FUJ00117504
FUJ00117504
co VOLUME AND PERFORMANCE TEST REPORT - PHASE 1
FUJITSU
COMMERCIAL IN CONFIDENCE
Abbreviation
tps
Definition
transactions per second
V&l
Volume & Integrity
0.6 Glossary
Performance Testing or
Load Testing
Aseries of tests to measure the performance characteristics and behaviour
of specific components of the solution under load to demonstrate their ability
to achieve defined performance requirements. For example, to test that a
particular web service is capable of supporting the relevant contractual peak
rate of transactions per second (the defined load).
Volume Testing
A series of tests to demonstrate that the whole solution is able to support a
typical daily workload. For example, to test that a peak weekday transaction
profile can be successfully harvested and that all scheduled batch
processing can be completed within expected timescales.
0.7 Changes Expected
Accepted comments following document review.
0.8 Accuracy
Fujitsu Services endeavours to ensure that the information contained in this document is correct but, whilst every
effort is made to ensure the accuracy of such information, it accepts no liability for any loss (however caused)
sustained as a result of any error or omission in the same.
0.9 Security Risk Assessment
Security risks have been assessed and it is considered that there are no security risks relating specifically to this
document
Ref: TST/SOT/REP/0008
© Copyright Fujitsu Services Ltd 2009 COMMERCIAL IN CONFIDENCE Version, 1.0
Date 46-Nov-2009
UNCONTROLLED IF PRINTED Page No: Sof 20
FUJ00117504
FUJ00117504
oO VOLUME AND PERFORMANCE TEST REPORT - PHASE 1
FUJITSU COMMERCIAL IN CONFIDENCE
1 = Introduction
The Volume and Performance High Level Test Plan [2] outlined a 2-phase approach using different test
environments:
>» Phase 1 on the VI (Live rig) in IRE11 prior to pilot.
>» Phase 2 on the Vol rig in IRE19 post pilot
The V&l rig used for Phase 1 was also used by other test streams; Integrity, Disaster Recovery and
NTSS. This is the Test Report for Volume and Performance testing only, as executed during Phase 1
Phase 1 Test Cycles were periods of testing activity (by all test streams) interspersed with rig build
activity to introduce new Data Centre and Counter releases. There was no availability to repeat Test
Cases within each cycle as originally intended and outlined in the HLTP.
This report provides a factual statement on the test status metrics at completion of Phase 1. The status of
particular Test Cases and any associated defects can be ascertained from the Quality Centre system.
2 Scope
This report is solely concerned with the Phase 1 Volume and Performance test activities and is not
reporting on the rig build activities that occurred before and during the period, or any other test stream
activities conducted in the V&l test environment.
The main objectives of Phase 1 Volume and Performance testing focussed on those test areas that must
be completed prior to live pilot. In essence, this concentrated on Horizon testing against a post-migration
Data Centre, as there will be no opportunity to perform any Horizon testing on the Vol rig during Phase 2.
In addition, some HNG-X ‘calibration’ testing was performed so as to provide benchmark performance
characteristics to be used to validate the Phase 2 test environment on the Vol rig.
3 Management Summary
Use of the V&l test environment for other streams of testing (Integrity, Disaster Recovery, NTSS)
severely restricted availability of the rig for volume and performance testing. Such testing could generally
not be run in parallel with other test streams due to their intrusive nature. As a consequence of
increasingly constrained rig availability, the list of tests identified in the HLTP were reviewed and re-
prioritised. The following table shows a revised list of each test category identifying which Phase each
would be tested in, or not tested at all. This differs from the table originally published in the HLTP. Those
tests marked for Phase 2 will be further reviewed prior to commencement of Phase 2 to re-assess
priorities in the light of any findings and considerations arising from Phase 1.
Green = tests conducted during Phase 1 and which are the focus of this report.
Yellow = tests earmarked for Phase 2 on the Vol rig
Pink = will not be tested. These are either subsumed in other tests or considered to be low risk.
Phase I Phase
1 2
Test No. on on
Grp__I Description Tests vai_I VoL
cot Counter performance tests 40 Y
Ref: TST/SOT/REP/0008
© Copyright Fujitsu Services Ltd 2009 COMMERCIAL IN CONFIDENCE Version: 1.0
Date: 16-Nov-2009
UNCONTROLLED IF PRINTED Page No: 6 of 20
FUJ00117504
FUJ00117504
o VOLUME AND PERFORMANCE TEST REPORT - PHASE 4 .
FUJITSU COMMERCIAL IN CONFIDENCE
Phase I Phase
1 2
Test No. on I on
Grp Description Tests val I VOL .
HOt _ I Horizon volume - Monday 2I_y
Ho4 glume 2 day harvest 4
HOS
Ho6 ctval its. 2
Ho7 rizon Link to contractual linuits -
Hos, rizon no
Hoo TU El
H10 I Horizon T&T to contractual limits 2I_y¥
Hit I APOP to contractual & design limits 2 Y
H12__I DVLA to contractual & design limits 4 Y
H12b DVLA stress test 1 Y
H13 PAF to contractual & design limits. 4 Y
H13b PAF stress test 1 Y
HCB Horizon combined banking to contractual limits - non-PCI 2 Y
HPC _I Horizon PCI combined banking & Debit Card to contractual limits 2I_y
Mot
Moz vol
Mo3 mixed I
Mo4
MOS
Mog. & ye
M10 _I Hydra mixed al 8
MCB med 4 MI
R01 __I Branch router tests 4 Y
X01 HNG-X volume Monday 6 Y
X05 _I HNG-X Debit Card to contractual & design limits 4 Y
X05 _I HNG-X Debit Card stress test 1 Y
X06__I HNG-X CAPO to contractual & design limits 4 Y
X06b HNG-X CAPO stress test 1 Y
x07 I HNG-X Link to contractual & design limits
X07b HNG-X Link stress test 1 Y
x08 tractua c
X08b HNG-X A&L stress test 1 Y
X09_I HNG-X ETU to contractual & design limits 4 Y
X09 _I HNG-X ETU stress test 1 Y
X10__I HNG-x T&T to contractual & design limits 4 Y
X10b_ I HNG-X T&T stress test 1 Y
X11__ I BAL and BRDB to contractual & design limits 2 Y
X12__I BAL and BRDB stress test 2 Y
X13 I Logon /logotf to contractual & design limits 2 Y
X13 I Logon /logoff stress test 1 Y
X13¢_ I Maximum concurrent session 1 Y
X15 HNG-X Reports to contractual & design limits 2 Y
X15b HNG-X Reports stress test 1 Y
© Copyright Fujitsu Services Ltd 2009 COMMERCIAL IN CONFIDENCE Boom TaTSOTIREPIONNS
UNCONTROLLED IF PRINTED Pegs No: ene 2008
FUJ00117504
FUJ00117504
oO VOLUME AND PERFORMANCE TEST REPORT - PHASE 1
FUJITSU COMMERCIAL IN CONFIDENCE
Phase
1
Test No. on
Grp Description Tests val
XCB___I HNG-X combined banking to contractual & design limits 4
XCBb_I HNG-X combined banking stress test 1 y
XCAL_I HNG-X calibration test 1, ¥ y
3.1 Test Case Coverage
The following table provides an overview of the final test status at the end of Phase 1 for each Test Case
within those test groups highlighted in green above.
‘est Description tatus
Hot Horizon Volume Monday Schedule - State 1 ¥ Passed
Horizon Volume Monday Schedule - State 2 # Failed
H10 Horizon T&T Contracted Peak Hour ¥ Passed
Horizon T&T Contracted Peak 5 minutes ¥ Passed
PAF Contracted Peak Hour v Passed
H13. PAF Contracted Peak 5 minutes ¥ Passed
PAF Design Limit Peak Hour ¥ Passed
PAF Design Limit Peak 5 minutes ¥ Passed
HcB Horizon Combined Banking Contracted Peak Hour v Passed
Horizon Combined Banking Contracted Peak 5 minutes ¥ Passed
HPC Horizon PCI Combined Banking and DCS Contracted Peak Hour ¥ Passed
Horizon PCI Combined Banking and DCS Contracted Peak 5 minutes _I “ Passed
XCAL HNG-X Calibration Test * Failed
Full results for each test are detailed in Section 4 of this report.
3.2 Defects
3.2.1 Defects Raised
A total of 221 defects were raised by the Volume and Performance test team up to the conclusion of
Phase 1 testing. A further 2 Peaks have been raised by the Performance Architect in relation to
observations made during the HNG-X Calibration test (these are not associated with a defect reference in
Quality Centre).
The following table shows the priority and status breakdown for these defects as at the date of this report.
Ref: TST/SOT/REP/0008
© Copyright Fujitsu Services Ltd 2009 COMMERCIAL IN CONFIDENCE Version; 1.0
Date: 16-Nov-2009
UNCONTROLLED IF PRINTED PageNo: 8 of 20
FUJ00117504
FUJ00117504
VOLUME AND PERFORMANCE TEST REPORT - PHASE 1
oO
FUJITSU COMMERCIAL IN CONFIDENCE
Current Priority I Open I Awaiting I Awaiting I Closed
PEAK (QC) Test Closure
A (4 - Very High) ie) ie) 0 116
B (3 - High) 2 ie) 0 67
C (2 - Medium i*) i'} 0 38
D(1-Low) 0 0 0 0
Open Calls are with a development or support team
Awaiting Test Calls have been returned to V&I awaiting retest or some other action
Awaiting Closure Calls are with V&l awaiting formal closure
Closed Calls are closed, no further actions required
3.2.2. Open Defects
The following table shows the priority and call summary for all defects that currently remain Open.
9100 I PCO180780 I B I VI- Audit gatherer errors accessing platforms pcon001- I tfs- Unix 20091105
Iprpcond0’ & \\SWICONO'\audit
77949 I PC0188321 I B I Volume Integrity - Final - EDS_BACKUP, ERP_BACKUP I HNG-back- I 20091772
dependency issue dev
4 Test Case Details
This section provides a more detailed view of each Test Case executed for each of the Test Groups
during Phase 1.
4.1 Test Group H01 — Horizon Volume
4.1.1 Horizon Volume (Monday) — State 1
The objective of this test was to exercise data centre processing for a peak day volume of representative
Horizon transactions. State 1 represents all outlets in pure Horizon mode without the enabling of pre-
migration reference data for generation of EOD BLOBs. As such, there was no workload for BRDB Hydra
batch scheduling, but all batch scheduling was performed against a migrated Data Centre. Effectively,
therefore, this test represents existing Horizon performed against new Data Centre.
Early iterations of this test identified a number of defects whereby critical schedules failed or took an
inordinate amount of time to complete. The suite of batch schedules was unable to complete in the time
available before next business day. However, subsequent fixes delivered enabled successful completion
of the test. Whilst some issues remained with specific jobs, they were not of a nature that inhibited
performance and the full suite of batch schedules was achieved within expected timescales to enable
completion in time for rollover to the next business day at 8:00hrs
Ref: TST/SOT/REP/0008
© Copyright Fujitsu Services Ltd 2009 COMMERCIAL IN CONFIDENCE Version; 1.0
Date 16-Nov-2009
UNCONTROLLED IF PRINTED Page No: 9 of 20
FUJ00117504
FUJ00117504
oO VOLUME AND PERFORMANCE TEST REPORT - PHASE 1
FUJITSU COMMERCIAL IN CONFIDENCE
Test Status: Passed
Date of Final Run: 19/08/09
Build Release: 14.14.01
4.1.2 Horizon Volume (Monday) — State 2
The objective of this test was to exercise data centre processing for a peak day volume of representative
Horizon transactions. State 2 represents all outlets in Horizon mode but with pre-migration reference data
enabled for generation of EOD BLOBs. This generates a workload for the BRDB Hydra batch processing.
The Volume State 2 test (which comprised 9597 outlets and represents circa 75% of live) failed to
complete running of the overnight schedule within the allotted timeframe. Three main issues were
observed during the run, all of which occur during the Hydra operation:
1. The transfer of the files from TPS to the BRDB caused the DAT to run at maximum utilisation for
periods. This operation is scheduled pre midnight and despite the high utilisation the schedules
completed in time. Notwithstanding this, the condition is being investigated to satisfy ourselves
that there are no underlying issues that may cause problems in live operation.
2. BRDB Hydra Proc took 4hrs 1min to complete. This process kicks off at around 2am and
currently does not complete in sufficient time to allow following process to complete prior to
branch trading.
3. BRDB Statistics Process takes longer the first time this is run. Subsequent running of this
process is complete in a much reduced timeline.
Issue [2] above is considered a ‘stopper’ and is required to be addressed prior to Weekend B/C. A
number of options for addressing this have been considered. The general consensus was that a
technical solution could be found to decreasing the time taken for the schedule to run. However, this will
require thorough analyses and the development of a fix which is likely to be a combination of tuning,
configuration and code changes. This will likely take a number of weeks to deliver and would require
validation in an environment able to test under representative volumes. To follow this course of action
alone would have resulted in a delay to handing over the V&lI rig to DC Prep and slip the Weekend B/C
Migration weekend.
A procedural solution was also considered and this was accepted to have less technical risk but would
create significant additional work for Fujitsu, Post Office Operations and the post masters during branch
migration. The procedural solution requires us to demonstrate that BRDB Hydra Proc would complete
before 4am with 20% of branches set within the database at ‘HydraT’, thus reducing the processing load
for this part of the schedule. To follow this course of action alone would require the additional processes
required to support the implementation of the pilot sites to continue through the full roll out (for example,
to create the Post Masters Remuneration Report the post master will need to produce both a pre
migration and post migration report and manually reconcile the two). This would significantly increase the
risk to completing the Branch Migration within the current scheduled timeframe.
The focus was on maintaining the current date for starting the Weekend B/C Migration, (30/10) whilst
minimising the risk to the Branch Migration. To achieve this, a hybrid way forward was agreed:
1. To validate that the BRDB Hydra Proc will complete before 4am with 20% of branches set within
database to HydraT. If successful;
2. Hand over the V&l rig to DC Prep. In parallel;
3. Development work on a technical solution
4. B/C Weekend will complete with 10% - 20% of branches set within the BRDB to HydraT (the 10%
- 20% will need to contain the candidate Pilot sites and preferable those branches likely to be at
the beginning of the roll out schedule)
Ref: TST/SOT/REP/0008
© Copyright Fujitsu Services Ltd 2009 COMMERCIAL IN CONFIDENCE Version: 10
Date 18-Nov-2008
UNCONTROLLED IF PRINTED Page No: 10 of 20
FUJ00117504
FUJ00117504
oO VOLUME AND PERFORMANCE TEST REPORT - PHASE 1
FUJITSU COMMERCIAL IN CONFIDENCE
5. Technical fixes to be validated in the first instance on SV&l to validate correct operation of the
BRDB and schedules
6. Once validated on SV&I and LST, the fixes to be delivered to Live and performance monitored
with the ‘20% processing load’.
7. Once satisfied that the fixes improve the performance of the Schedule the number of the
branches set within the BRDB to HydraT will be increased in increments (e.g. 30%, 40%, 50%).
At each increment the performance of the Schedule will be monitored for a period of time
requiring achieving set criteria prior to moving to the next increment.
[Note: Once the numbers of branches set to HydraT in the BRDB is incremented this action
cannot be reversed. Therefore it is important to increment the numbers in a controlled way to
ensure that the overnight schedule completes within the allocated time]
8. Should the fixes prove successful, and the branches set within the BRDB reach 100% prior to 84
days before end of Pilot, the system will be able to support branch roll out without the need to
continue with the additional Pilot processes.
9. Should the fixes not prove successful the Programme could choose to:
a. Not exit Pilot until a technical solution has been implemented [Note: in this scenario the
Pilot could be extended to the 1250 - 2500 branches set within the BRDB to HydraT], or
b. Exit pilot and proceed with the roll out continuing to use the additional Pilot processes
A retest of Horizon Volume State 2 with only 2500 outlets (circa 20% of live) as per [1] above was
therefore executed. Whilst BRDB Hydra Proc still did not complete before 4am as performance
improvement was not linear (2 hours 26 minutes for 2500 outlets versus 4 hours 1 minute for 9597
outlets), the full suite of BRDB schedules did complete before next busing day and start of branch
trading.
Further changes to BRDB Hydra Proc are now also being developed so as to split the workload into pre-
processor tasks that can be run before midnight (and make use of a current ‘quiet’ period in BRDB
scheduling), and compare modules that will continue to run at around 2am. This will enable the entire
process to complete much earlier. This change (which also requires amendment to the batch scheduling)
is to be delivered prior to Weekend B/C Migration
Investigation into the DAT utilisation will continue and any tuning/fixes will be introduced as part of the
actions described in [5] and [6] above. Meanwhile, a DRS Schedule amendment will be made to add
dependency on BRDB_XFR_COMPL for night running of DRSC310 in schedules
DRS_NWB_MS_NIGHT and DRS_EFT_MS_NIGHT in order to reduce load on the DAT server and to
even out the load. This is possible as the DRS LREC file doesn't need to be processed before 02:00am
Test Status: Failed — further fixes to be delivered in accordance with the way forward outlined above.
Date of Final Run: 21/09/09
Build Release: 14.15.00
4.2. Test Group H10 — Horizon T&T
4.2.1 Horizon T&T Contracted Peak Hour / Peak 5 Minutes
The contracted peak hourly and peak 5 minute transaction rates for Horizon T&T equate to the same rate
of tps. Effectively, therefore, they comprise a single test. The transaction rates initially run for this test
were in accordance with those defined in the Horizon Capacity Management and Business Volumes [3]
Namely;
Ref: TST/SOT/REP/0008
© Copyright Fujitsu Services Ltd 2009 COMMERCIAL IN CONFIDENCE Version. 1.0
Date 16-Nov-2008
UNCONTROLLED IF PRINTED Page No: 11 of 20
FUJ00117504
FUJ00117504
oO VOLUME AND PERFORMANCE TEST REPORT - PHASE 1
FUJITSU COMMERCIAL IN CONFIDENCE
Peak Hour Contracted Volume = 42,500 = 11.81 tps
Peak 5 Minutes Contracted Volume = 11.66 tps.
The test was therefore run at a rate of 12 tps for a period exceeding 1 hour, with the load balanced
evenly across all 4 clusters. This rate was successfully maintained with no transaction failures.
Performance monitoring (from analysis of Windows Perfmon logs and Metron Athene data) indicated no
performance issues on any of the key platforms involved in processing T&T transactions (NRA, FLG and
NPS platforms).
Subsequent to the above test, at the request of the Performance Architect in consideration of increasing
live volumes, the test was re-run with a significantly increased rate of 52 tps. Again, this rate was
successfully sustained for a period exceeding 1 hour with no indication of any performance concerns on
any of the key platforms involved.
Test Status: Passed
Date of Final Run: 15/06/09
Build Release: 14.10.00
4.3 Test Group H13 — PAF
4.3.1. PAF Contracted / Design Peak Hour / Peak 5 Minutes
The Peak Hour and Peak 5 Minute ratess for PAF are similar and targets are as follows:
Peak Hour Contracted Volume = 162,246 = 45 tps
Peak 5 Minutes Contracted Volume = 46 tps.
Peak Hour Design Limit volume = 194,695 = 54 tps
Peak 5 Minutes Design Limit Volume = 56 tps
Asingle test was run to demonstrate the maximum sustainable rate for a period exceeding 1 hour against
a single PWS server (i.e. the maximum load supported by a single server).
First iteration of the test failed to achieve contracted volumes and only sustained 33 tps. Investigations
identified incorrect configuration of the PWS server. Specifically, the expected configuration is:
a) 4 EJBs for the PAFEnglishSession
b) English.ini should contain Total_Result_Limit = 600
The actual configuration only had 3 EJB's and the English.ini did not contain the Total_Result_Limit = 600
(JVM was correct at 256MB and thread concurrency of 1 was also correct).
Second iteration of the test with correct configuration of the PWS server again failed to achieve
contracted volumes and was only able to support 37 tps. This was with the PWS server located in an
area with 2x vCPU vBLADES as per the original server layout plan.
The PWS server was relocated to 4 x vCPU vBLADES (it was swapped with a BWS server which has
much lower throughput and for which 2 x vCPU would suffice).
Third iteration of the test demonstrate that the achievable successful rate of transactions per second
peaked at 54tps. This rate was sustainable for a period well in excess of 1 hour and therefore achieves
the Design Limit Peak Hourly rate of 54 tps. A Change Proposal was raised to formalise relocation of the
servers.
Ref: TST/SOT/REP/0008
© Copyright Fujitsu Services Ltd 2009 COMMERCIAL IN CONFIDENCE Version: 10
Date 18-Nov-2008
UNCONTROLLED IF PRINTED Page No: 12 0f 20
FUJ00117504
FUJ00117504
oO VOLUME AND PERFORMANCE TEST REPORT - PHASE 1
FUJITSU COMMERCIAL IN CONFIDENCE
Test Status: Passed
Date of Final Run: 07/05/09
Build Release: 14.07.03
4.4 Test Group HCB — Horizon Combined Banking
4.4.1. Horizon Combined Banking Contracted Peak Hour
The target contracted peak hour rate for Horizon combined banking is as follows:
Contracted: Peak Hour Volume:
CAPO = 484,174 = 134 tps
Link = 97,750 = 27 tps
A&L = 36,316 = 10 tps
TOTAL = 171 tps
First iteration of the test was only able to support a combined total of 171 tps for a few minutes before a
significant number of transaction timeouts began to occur. Subsequent investigations identified the cause
to be a time synchronisation issue between the Routing Agent and Authorisation Agents, which resulted
in the Authorisation Agent regarding transaction requests as ‘stale’
Having resolved the time synchronisation issue, the second iteration of the test successfully
demonstrated that the contracted peak hourly rate of 171 tps was sustainable for a period exceeding 1
hour.with no transaction failures. Performance monitoring (from analysis of Metron Athene data) indicated
no performance issues on any of the key platforms involved in processing Horizon Banking transactions
(NRA, NAC, NAL, NAA and NPS platforms).
Test Status: Passed
Date of Final Run: 15/06/09
Build Release: 14.10.00
4.4.2. Horizon Combined Banking Contracted Peak 5 Minutes
The target contracted peak 5 minute rate for Horizon combined banking is as follows:
Contracted: Peak Hour Volume:
CAPO = 160 tps
Link = 32 tps
A&L = 12 tps
TOTAL = 204 tps
The test began at a rate of 120 total tps increasing by 50 tps every 5 minutes up to a maximum total of
320 tps. The maximum rate of 320 tps was successfully sustained without any transaction failures for a
period exceeding 20 minutes. This therefore demonstrates that a rate well in excess of contracted peak 5
minute volumes can be sustained.
Test Status: Passed
Ref: TST/SOT/REP/0008
© Copyright Fujitsu Services Ltd 2009 COMMERCIAL IN CONFIDENCE Version. 1.0
Date: 16-Nov-2009
UNCONTROLLED IF PRINTED Page No: 13 of 20
FUJ00117504
FUJ00117504
oO VOLUME AND PERFORMANCE TEST REPORT - PHASE 1
FUJITSU COMMERCIAL IN CONFIDENCE
Date of Final Run: 15/06/09
Build Release: 14.10.00
4.5 Test Group HPC — Horizon PCI Combined Banking &
Debit Card
The following tests were conducted with 8 active Horizon OSR services, 1 on each of servers BALO01 to
BALOO8.
4.5.1 Horizon PCI Combined Banking & Debit Card Design Peak
Hour
Current contracted and design limit peak hour tps rates for Horizon PCI are as follows:
CAPO Link A&L DCS TOTAL
Contracted Peak Hour 134 27 10 22 193
Design Limit Peak Hour 161 33 12 26 232
The actual test run was at / above the design limit rates (thus effectively proving the contracted rates
anyway), and successfully sustained the following rates for more than an hour with no transaction
failures.
CAPO Link A&L DCS TOTAL
Average tps sustained 177 33 13 29 252
Test Status: Passed
Date of Final Run: 12/07/09
Build Release: 14.11.04
4.5.2 Horizon PCI Combined Banking & Debit Card Design Peak 5
Minutes
Current contracted and design limit peak 5 minute tps rates for Horizon PCI are as follows
CAPO Link A&L DCS TOTAL
Contracted Peak 5 Minutes 160 32 12 22 226
Design Limit Peak 5 Minutes 250 51 19 34 354
However, with Debit Card transaction rates being subject to renegotiation, the target rate for DCS was set
at 57 tps in line with future expectations.
The actual test run was at / above the design limit rates (thus effectively proving the contracted rates
anyway), and successfully sustained the following rates for more than 5 minutes with no transaction
failures.
CAPO Link A&L DCs TOTAL
Average tps sustained 256 50 19 57 382
Ref. TST/SOTIREP/0008
© Copyright Fujitsu Services Ltd 2009 COMMERCIAL IN CONFIDENCE Version: 1.0
Date: 46-Nov-2009
UNCONTROLLED IF PRINTED Page No: 14 0f 20
FUJ00117504
FUJ00117504
oO VOLUME AND PERFORMANCE TEST REPORT - PHASE 1
FUJITSU COMMERCIAL IN CONFIDENCE
Test Status: Passed
Date of Final Run: 12/07/09
Build Release: 14.11.04
4.6 Test Group XCAL —- HNG-X Calibration Test
The objective of the ‘calibration’ test was to run a reduced basket mix of HNG-X transactions, but at
reasonable volumes, in order to capture performance metrics of the V&l rig using Metron Athene Data
gathered for the period of the test run. The same test would then be performed on the Vol rig to help
understand any environment differences and how this could affect interpretation of performance results
An initial test was performed for EPOSS transactions only, before later attempting a more representative
basket mix.
4.6.1. HNG-X Calibration - Simple EPOSS
This test comprised of simple EPOSS transactions (selling a stamp) only and the conditions under which
the test was run are as follows:
There were 16 OSR instances in total, with 2 OSRs running on each of servers BALOO1 —
BALO08.
There were 4 BRDB nodes available.
The bandwidth was throttled to 256Kbps from the load generator to the ACE Blade.
All messages were compressed.
The 400 branches used in the test were spread across the fad hashes to generate an even load
across the 4 BRDB Nodes.
The incremental rate of transaction volumes during the run was as follows:
150 tps (across 150 branches) for 15 minutes, then increased to
200 tps (across 200 branches) for further 15 minutes, then increased to
300 tps (across 300 branches) for further 15 minutes, then increased to
400 tps (across 400 branches) until the end of the test.
Throughout this test run, transaction response times were good
Ref: TST/SOT/REP/0008
© Copyright Fujitsu Services Ltd 2009 COMMERCIAL IN CONFIDENCE Version. 1.0
Date 16-Nov-2008
UNCONTROLLED IF PRINTED Page No: 15 of 20
FUJ00117504
FUJ00117504
Ge) VOLUME AND PERFORMANCE TEST REPORT - PHASE 4
FUJITSU COMMERCIAL IN CONFIDENCE
Average Settlement Response Time Settlements per Second
400
l 380
300 O«&E
[ 250 3
20g
op I 10g
I 100 «=
H ll :
04 a
0.35 ‘
03
025
02
005
" il
ll
Average Response Time (seconds)
00:10 00:20 00:30 00:40 00:50 01:00 01:40 04:20 01:30 01:40
Elapsed scenario time hhimm.
[Ei Settlements response tine [2] Settlements per second
Test Status: Passed
Date of Final Run: 6th August 2009
Build Release: 14.12.01
4.6.2 HNG-X Calibration - Basket Mix
Several attempts at running a calibration test for a mixed basket containing EPOSS, Banking, Debit Card,
ETU, DVLA, Logon and Logoff transactions were run on the 18" and 19!" September. All iterations were
run under the following conditions:
There were 16 OSR instances in total, with 2 OSRs running on each of servers BALOO1 —
BALO007 and BALOOS.
There were 4 BRDB nodes available.
The bandwidth was set to maximum between the load generator and the ACE Blade.
All messages were compressed.
All branches used in the test were spread across the fad hashes to generate an even load across
the 4 BRDB Nodes.
All banking transactions were submitted for agent hash value 1, meaning that only the capo_b,
al_b and link_b services were handling transaction load.
Whilst this has generated some data that can be used for comparison purposes when run under similar
conditions on the Vol rig, issues were encountered during each run and not all transaction requests were
Ref: TST/SOT/REP/0008
© Copyright Fujitsu Services Ltd 2009 COMMERCIAL IN CONFIDENCE Version; 1.0
Date: 16-Nov-2009
UNCONTROLLED IF PRINTED Page No: 16 of 20
FUJ00117504
FUJ00117504
oO VOLUME AND PERFORMANCE TEST REPORT - PHASE 1
FUJITSU COMMERCIAL IN CONFIDENCE
successfully authorised. As at the date of this report, the reason for transaction failures is not yet known
and investigations are continuing.
Test Status: Failed
Date of Final Run: 19/09/09
Build Release: 14.15.00
An outline of each iteration is shown in the following sections. In each table, the value shown for Txn/sec
is the total number of transaction records per second for each service type, including settlement
transaction (i.e. the number of transactions addressed to the BAL layer). The values shown for Auth/sec
is the number of actual requests that would be forwarded to each relevant Authorisation service. The end
column shows the median number of transactions per second successfully authororised for banking,
debit card and ETU services.
4.6.2.1 Mix of EPOSS/Banking/Debit/E-TopUps/DVLA/Login/Logoffs @ 100tps
0700_CAPO_CAL 37 37 13 11.047
0800_AL_CAL 3 3 1 0.766
0900_LINK_CAL 7 8 3 25
1000_EPOSS_CAL 23 24 24
1100_DEBIT_CAL 7 6 3 2.047
1200_LoginLogoff_CAL 8 4 41
7300_DVLA_CAL 7 Ti 6
1400_ETU_CAL 8 8 3 2.641
100 101 54
After a 5 minute ramp up 103 tps was reached. After 25 minutes there was a sudden drop in successful
authorisation rate. After 40 minutes the rate of successful transactions deteriorated further.
4.6.2.2 Mix of EPOSS/Banking/Debit/E-TopUps/DVLA/Login/Logoffs @ 200tps
0700_CAPO_CAL 75 75 25 16.078
0800_AL_CAL 7 7 2.33 1.547
0900_LINK_CAL 13 14 4.68 4.609
1000_EPOSS_CAL 47 49 49
1100_DEBIT_CAL 13 13 417 3.531
7200_LoginLogoff_CAL 17 9 2.91
7300_DVLA_CAL 13 20 6.76
1400_ETU_CAL 15 15 5 3.906
200 202 100
© Copyright Fujitsu Services Ltd 2009 COMMERCIAL IN CONFIDENCE ae on ry ouREPionne
Date: 16-Nov-2009
UNCONTROLLED IF PRINTED Page No: 17 of 20
FUJ00117504
FUJ00117504
oO VOLUME AND PERFORMANCE TEST REPORT - PHASE 1
FUJITSU COMMERCIAL IN CONFIDENCE
4.6.2.3 Mix of EPOSS/Banking/Debit/E-TopUps/DVLA/Login/Logoffs @ 300tps
0700_CAPO_CAL. 112 112 37.33 22.297
0800_AL_CAL 7 17 3.66 2.504
0900_LINK_CAL 19 21 6.84 6.656
7000_EPOSS_CAL 71 75 75
7700_DEBIT_CAL 19 19 6.16 5.063
1200_LoginLogoff_CAL 26 13 445
7300_DVLA_CAL 79 30 9.88
7400_ETU_CAL 23 23 7.66 5.984
300 303 151
46.2.4 Mix of EPOSS/Banking/Debit/E-TopUps/DVLA/Login/Logoffs @ 400tps
0700_CAPO_CAL 150 150 50 10.576
0800_AL_CAL 15 15 5 3.984
0900_LINK_CAL 25 27 9 5.719
1000_EPOSS_CAL 95 100 100
1100_DEBIT_CAL 25 22 8 0
1200_LoginLogoff_CAL 35 30 6
1300_DVLA_CAL 25 26 13
1400_ETU_CAL 30 30 10 7
400 400 201
The CAPO service was initially authorising at a rate of approximately 26 tps, but saw a significant
reduction to approximately 10 tps for a period of 20 minutes before the NAC004 service then failed over
to NAC003 and the authorisation rate recovered to around 36 tps. The average Pass rate was 17tps but
the median value above is somewhat ‘skewed’ by the 20 minute period of low authorisation rates prior to
the agent failover.
The LINK service was initially authorising at a rate of approximately 9 tps before a sudden reduction to
5.7 tps t 10 minutes into the run.
The Debit Card service failed after only a few minutes into the run.
46.2.5 Simple EPOSS transaction (selling a stamp) @ 646tps
1000_EPOSS_CAL 646 646
Ref: TST/SOT/REP/0008
© Copyright Fujitsu Services Ltd 2009 COMMERCIAL IN CONFIDENCE vesion 10
Date 16-Nov-2009
UNCONTROLLED IF PRINTED Page No: 18 of 20
FUJ00117504
FUJ00117504
oO VOLUME AND PERFORMANCE TEST REPORT - PHASE 1
FUJITSU COMMERCIAL IN CONFIDENCE
Hits per Second - Average Transaction Response Time Z
4
H
043 8
£
O12 9
E
O41 3
5
I Oo &
j 8
j ©
} 009 A
é L Ll Te
- nae = ~- o-§ 4
600 ; J
sof
zg i
8 4004-4 ‘
8 ! \
ao i
®
z mys
!
1004 :
1500 1505 1510 1815 1520 1525 1830 1535 1540 15451550 18.55
Absolute Time hh:hhimm
EPOSS only transactions produced similar successful results as previously demonstrated.
46.2.6 Mix of EPOSS/Banking/Debit @ 888tps
0700_CAPO_CAL 600 525 175 (e)
0800_AL_CAL 50 36 12 5.531
0900_LINK_CAL 100 99 33 2.78
1000_EPOSS_CAL 150 150 150
1100_DEBIT_CAL 100 78 26 6.703
1000 888 396
The CAPO service failed after only a few minutes into the run.
The LINK service failed approximately 25 minutes into the run.
The Debit Card service failed for a period of 15 minutes mid-way through the run.
Refi TST/SOT/REP/0008
© Copyright Fujitsu Services Ltd 2009 COMMERCIAL IN CONFIDENCE Version; 1.0
UNCONTROLLED IF PRINTED Pegs No: pNezone
FUJ00117504
FUJ00117504
oO VOLUME AND PERFORMANCE TEST REPORT - PHASE 1
FUJITSU COMMERCIAL IN CONFIDENCE
5 Recommendations
o Continue with development fixes for BRDB processing improvements in accordance with the way
forward outlined in section 4.1.2
o Continue investigations as to the cause of issues encountered during HNG-X Calibraton test
runs.
o Review planned activities for Phase 2 testing on Vol rig in light of any considerations arising from
Phase 1 testing.
6 Conclusions
Whilst the originally intending level of testing for Phase 1 as outlined in the HLTP was significantly
reduced, the revised objectives pertaining to Horizon testing have been achieved subject to the
introduction of further performance improvement fixes for BRDB Hydra batch schedule processing.
As regards HNG-X Calibration testing, whilst this has enabled Metron Athene data to be captured for
comparison with results to be achieved on the Vol rig, concerns remain as to the transaction failures
encuntered during these tests. It is not currently understood whether such failures were due to issues
with the test data itself, environment issues caused by other activities on the rig, or regression in
behaviour of the Authorisation Agents
Ref: TST/SOT/REP/0008
© Copyright Fujitsu Services Ltd 2009 COMMERCIAL IN CONFIDENCE Version: 10
Date 16-Nov-2008
UNCONTROLLED IF PRINTED Page No: 20 0f 20