FUJ00232610
FUJ00232610
(oe) Transaction Time Benchmarking Process
FUJITSU Commercial in Confidence
Document Title: Transaction Time Benchmarking Process
Document Reference: SVM/SDM/PRO/0017
Release: Release Independent
Abstract: This document describes the process that will be used to conduct
the HNG-X Counter Performance assessment as described in
Schedule B6.1. It also describes how the basis for assessment has
changed from video benchmark analysis to system derived
transaction time data.
Further, the basis for the ongoing Transaction Time Benchmarking
Service is described.
Document Status: FOR APPROVAL
Author & Dept: Victor Gough / Dave Cooke
External Distribution: (Specify those individuals outside of the Royal Mail Group Account
who required approved version only. For RMGA Document to
distribute following approval)
Security Risk NO
Assessment Confirmed
Approval Authorities:
Narr Role Signature Date
Sarah Bull Manager — Branch Services and See Dimensions for record
Release Management
See Dimensions for record
See HNG-X Reviewers/Approvers Matrix (PGM/DCM/ION/0001) for guidance on who should approve.
sapyright Fujitsu Services Limited Commercial in Confidence Ref: SVM/SDM/PRO/0017
Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY Date: 17-JUL-2009
STORED
Page No: 1 of 24
FUJ00232610
FUJ00232610
Transaction Time Benchmarking Process
he)
FUJITSU Commercial in Confidence
0 Document Control
0.1 Table of Contents
) DOCUMENT CONTROL.
0.1 Table of Contents.
0.2 Document History.
0.3 Review Details.
0.4 Associated Documen
0.5 Acceptance by Document Review.
0.6 Abbreviations.
0.7 Glossary..
0.8 Changes Expected.
0.9
0.10
1 INTRODUCTION
1.1. Scope...
1.2 Document Hierarchy
1.3. HNG-X Contractual Requirements. 8
1.4 Assessment Principles.....
2 COUNTER PERFORMANCE ASSESSMENT PROCESS.
2.1 Process Overview...... 1
3 ‘STREAM 1’ ACTIVITIES... 12
3.1. Define Sample Transact
3.2. Obtain Timing Data.
3.3. Verification...
4 ‘STREAM 2’ ACTIVITIES..
4.1 Create Representative Basket Mi
4.1.1 Benchmark Period,
4.1.2 Transaction Types.
4.2 Create HNG-X Timings..
4.2.1 Product Frequency.
4.2.2 Adjustments...
4.3 Create Horizon timings.
4.4 Assessment Results...
5 FUTURE TRANSACTION TIME BENCHMARKING
A EXTRACT FROM SCHEDULE B6.1.......
sapyright Fujitsu Services Limited Commercial in Confidence Ref: SVM/SDM/PRO/0017
Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY Date: 17-JUL-2009
STORED
Page No: 2 of 24
FUJ00232610
FUJ00232610
Transaction Time Benchmarking Process
he)
FUJITSU Commercial in Confidence
B APPLICABLE POL REQUIREMENTS.
0.2 Document History
Version No. Date Summary of Changes and Reason for Issue Associated Change -
CP/PEAK/PPRR
Reference
04 29 May 2009 First draft
0.2 (01 Jun 2008 Revisions following initial internal review
0.3 05 Jun 2009 Revisions following internal review
0.4 26 Jun 2009 Revised to address internal and initial POL comments.
1.0 17 Jul 2009 Version For Approval
0.3 Review Details
See HNG-X Reviewers/Approvers Matrix (PGM/DCM/ION/0001) for guidance on completing the lists below. You
may include additional reviewers if necessary, but you should generally not exclude any of the mandatory
reviewers shown in the matrix for the document type you are authoring.
Review Comments by 17/07/09
Review Comments to David.cooke¢
Mandatory Review
Role Name
FS Service Manager Sarah Bull
HNG-X and Non Functional Test Manager Lee Farman
POL Design Authority lan Trundell
Role Name
POL Acceptance Manager Neil Williams
POL Noel Beaton
FS Design/ Development Manager ‘Adam Cousins
Joint Test Team Peter Langford
FS Systems Qualities Architect Dave Chapman
FS SV&I Test Manager Sheila Bamber
CS Migration Governance Manager Graham Welsh
Issued for Information lease restrict this
distribution list to a minimum
Position/Role Name
CS Service Definition Manager ‘Adam Bowe
sapyright Fujitsu Services Limited Commercial in Confidence Ref: SVM/SDM/PRO/0017
Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY Date: 17-JUL-2009
STORED Page No: 3 of 24
FUJ00232610
FUJ00232610
(oe) Transaction Time Benchmarking Process
FUJITSU Commercial in Confidence
(* ) = Reviewers that returned comments
0.4 Associated Documents (Internal & External)
Reference Versiot Date Title Source
PGM/DCM/TEM/0001 I 5.0 03 June 2009 RMGA HNG-X Generic Document Dimensions
(DO NOT REMOVE) Template
SVM/SDM/SD/0016 Management Information Service: Dimensions
Service Description
SVM/SDM/REP/0007 Validation of HNG-X System Derived I Dimensions
Transaction Component Times
SVM/SDM/REP/0009 HNG-X Representative Basket mix Dimensions
SVM/SDM/TP/0002 HNG-X Representative Transaction Dimensions
Scripts
SVM/SDM/REP/0008 HNG-X Counter Performance Dimensions
Assessment Results
Unless a specific version is referred to above, reference should be made to the current approved
versions of the documents.
0.5 Acceptance by Document Review
The sections in this document that have been identified to POL as comprising evidence to support
Acceptance by Document review (DR) are listed below for the relevant Requirements:
POL NFR DR Internal FS PO! Document Document Section Heading
Acceptance Ref NFR Reference Section Number
SVC-842 SVC-795 444 Benchmark Period
SVC-799 SVC-799 41.24 Back Office / Admin transactions
SVC-805 SVC-805 Whole Document
SVC-806 SVC-806 Whole Document
SVC-855 SVC-808 3 ‘Stream 1' activities
0.6 Abbreviations
Abbreviation Definiti
sapyright Fujitsu Services Limited Commercial in Confidence Ref: SVM/SDM/PRO/0017
Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY Date: 17-JUL-2009
STORED Page No: 4 of 24
FUJ00232610
FUJ00232610
(oe) Transaction Time Benchmarking Process
FUJITSU Commercial in Confidence
0.7 Glossary
Term Definition
Benchmark transaction
set
Basket mix A set of Basket / Customer Sessions containing one or more Transaction Types
ranked by volume of occurrence.
Benchmark Period The period of time from which transaction data will be taken. The dates are
specified in section 4.1.1
‘System components Discrete periods of counter processing activity that will be measured to support the
counter performance assessment.
Transaction Type A generalised instance of an actual transaction - e.g. ‘AP-InPay’ representing all
utility bill payment transactions
0.8 Changes Expected
None
0.9 Accuracy
Fujitsu Services endeavours to ensure that the information contained in this document is correct but, whilst every
effort is made to ensure the accuracy of such information, it accepts no liability for any loss (however caused)
sustained as a result of any error or omission in the same.
0.10 Security Risk Assessment
Security risks have been assessed and it is considered that there are no security risks relating specifically to this
document.
sapyright Fujitsu Services Limited Commercial in Confidence Ref: SVM/SDM/PRO/0017
Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY Date: 17-JUL-2009
STORED Page No: 5 of 24
FUJ00232610
FUJ00232610
Transaction Time Benchmarking Process
fee)
FUJITSU Commercial in Confidence
1 ‘Introduction
This document describes the process that will be used by Fujitsu to conduct the HNG-X Counter
Performance assessment as described in Schedule B6.1. This activity (also known as Transaction Time
Benchmarking) is concerned with a comparison of applicable sets of Horizon and HNG-X representative
transaction types and comprises the formal basis for the acceptance of the HNG-X counter performance.
This document describes the key stages of this assessment process, the analysis method that is to be
used, the sources and usage of the underlying transaction component time data and the basis for any
adjustments to these component timings.
A key change from the previous Horizon Transaction Benchmarking process is the introduction of
system generated transaction component times to replace Video Benchmarking (Subject to!
Gonfitmation)'. Validating that this approach provides accurate and comparable timings is a necessary
stage in ensuring both POL and FS can have confidence in this approach going forward, and this aspect
is covered in this document.
As noted above, the overall objective of this process is to satisfy the contractual obligations and
acceptance criteria contained in Appendix A & B. Separate from these specific activities is the ongoing
Transaction Time Benchmarking Service which is described in SVM/SDM/SD/0016 - Management
Information Service — Service Description. This service will utilise some of the deliverables from the
counter performance assessment and this aspect is also described in this document.
This document and the associated assessment activities are intended to be completed and assessed as
part of Acceptance Gateway 3 (Readiness for Pilot), and so will act as one of the entry criteria into Pilot.
1.1 Scope
This document is organised as follows:-
Section 1 — Introduction - provides an overview of the scope of the process and the contractual
background.
Section 2 — Counter Performance Assessment Process — describes the tasks within the two main
streams of activity, namely
e The validation of the system derived transaction component times
e The analysis, calculation and comparison of the HNG-X Counter Performance
Section 3 —- Representative Basket Mix — describes the approach to establishing the representative
workload profile
Section 4 — Assessing the HNG-X Counter Performance — describes the analysis and calculations that
underpin the comparison calculations.
It should be noted that the calculations and performance comparisons that underpin this process are
based on averaged representative transaction types, and that various adjustments will be applied to take
account of agreed differences between Horizon and HNG-X transactions, typically revised user
interactions or UI functionality. Accordingly no direct inference can be drawn from these averaged
figures to the typical transaction times that will be experienced by Branches.
Other performance assessment activities, such as the outcomes of the testing carried out via
TST/SOT/HTP/0008 HNG-X Counter Performance High Level Test Plan, may provide information in this
area.
* The introduction of system generated transaction component times is subject to the successful completion of the Steam 1 activities
described in section 3.
sapyright Fujitsu Services Limited Commercial in Confidence Ref: SVM/SDM/PRO/0017
Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY Date: 17-JUL-2009
STORED PageNo: 6 of 24
2
FUJITSU
FUJ00232610
FUJ00232610
Transaction Time Benchmarking Process
Commercial in Confidence
1.2 Document Hierarchy
This document forms part of a set of documents that will collectively provide the descriptions and results
associated with HNG-X Counter Performance assessment. They are as follows:-
SVM/SDM/PRO/0017 — this document
SVM/SDM/REP/0007 — Validation of HNG-X System Derived Transaction Component Times
o This document describes how the system generated transaction component times have
been validated as accurate against the Winrunner tool.
SVM/SDM/REP/0009 — HNG-X Representative Basket mix.
o This document defines the representative workload profile from data gathered during
the Benchmark Period. It describes the set of representative baskets (customer
sessions) ranked by frequency of occurrence.
SVM/SDM/TP/0002 — HNG-X Representative Transaction Scripts
o This document describes the user and system interaction sequences that make up each
of the representative transaction types within the Representative Basket mix. It also
contains details of the Winrunner scripts that will be used to run these transactions.
SVM/SDM/REP/0008 — HNG-X Counter Performance Assessment Results
o This document describes the calculations and data that have been used to generate the
Horizon / HNG-X counter performance assessment. This includes the Horizon and
HNG-X system component times and any adjustments that arise from changed UI
features or transaction sequences. It includes results of the Basket mix aggregation
and the overall performance comparison.
Various ‘SVC’ Schedule B6.1
POL Annex 3
Requirements
defines acceptance a sets formal
criteria for context for
~~,
SVM/SDM/PRO/0017
‘Transaction Time
Benchmarking Process’
defines defines
basis for besisfor I SVM/SDM/REP/0009
a eines. ‘HNG-X Representative
SVM/SDM/REP/0007 defines Basket mix’
‘Validation of HNG-X System basis for ¥
Derived Transaction Component derived from
Times SVM/SDM/REP/0008 ;
SVMISDM/TP/0002
‘HING-X Count
Performance. ‘HNG-X Representative
Assessment Results’ Transaction Scripts’
°
sapyright Fujitsu Services Limited Commercial in Confidence Ref: SVM/SDM/PRO/0017
Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY Date: 17-JUL-2009
STORED PageNo: 7 of 24
FUJ00232610
FUJ00232610
Transaction Time Benchmarking Process
2
FUJITSU
Commercial in Confidence
1.3 HNG-X Contractual Requirements
As indicated above the formal basis for the HNG-X Counter Performance Assessment is Schedule B6.1,
Annex 3 para 4 — reproduced in Appendix A, together with various SVC requirements — reproduced in
Appendix B.
The criteria that this document specifically addresses are referenced in section 0.5 —- Acceptance by
Document Review.
1.4 Assessment Principles
From the Schedule specification and the POL requirements a number of key assessment principles and
definitions are either explicitly specified or can be derived. These are shown below: -
Assessment principle / Derived from
id definition
Comment
Schedule B3.1 /
Annex 3
1. I The performance
characteristics of the HNG-X
Counter applications must be
acceptable to POL, and that
this will be demonstrated by a
jointly agreed process
POL will review and approve this
document and the assessment
results
The document SVM/SDM/REP/0008
— HNG-X Counter Performance
Assessment Results will
demonstrate this
2. I The basis for assessing the Schedule B3.1 /
HNG-X counter performance is I Annex 3
that it delivers equivalent or
better performance
characteristics than Horizon,
based on principles described
in this document
The document SVM/SDM/REP/0009
Schedule B3.1 / — HNG-X Representative Basket
3. I The assessment will be based
Customer Sessions.
Types and their associated SVC-798. customer session data (transactions
and settlement) are translated into
the various combinations of
Transaction Types.
These are then ranked by volume to
enable the most frequently occurring
to be used as the basis for the
performance assessment.
4. I The assessment will be based
Schedule B3.1 /
on an agreed measurement Annex 3.
period - The Benchmark
Period SVC-842.
Section 4.1.1 defines this period
Copyright Fujitsu Services Limited
Commercial in Confidence Ref: SVM/SDM/PRO/0017
2009
Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY Date: 17-JUL-2009
STORED PageNo: 8 of 24
fee)
FUJITSU
FUJ00232610
FUJ00232610
Transaction Time Benchmarking Process
Commercial in Confidence
5. I The basis of the assessment is
on the system components of
the counter application
Schedule B3.1 /
Annex 3
As with standard benchmarking
activities, user interactions are
excluded. In addition on-line
component times that are subject to
separate SLTs will be recorded but
are excluded from the comparison
analysis, since achievement targets
for these have been agreed and are
included in the relevant SLA.
6. I The composition of each
Transaction Type may be
adjusted to take account of
any changed processes or
changed UI characteristics
Schedule B3.1 /
Annex 3
HNG-X has enabled POL to
introduce various operational
enhancements to the user dialogues
of some transactions. In addition the
new UI introduces some new display
and interaction characteristics that
will also change the user actions for
some transactions from their Horizon
equivalents.
The document SVM/SDM/REP/0008
— HNG-X Counter Performance
Assessment Results will specify
which transactions have been
adjusted such that a fair comparison
of Horizon and HNG-X timings can
be made.
7. I POL may request the
Schedule B3.1 /
The Transaction Time Benchmarking
introduction of additional Annex 3 Service and/or the Capacity
Transaction Types if Management Service will assess any
anomalous performance such anomalies and will agree with
characteristics are observed. POL what effect or remediation is
required.
8. I Design Targets may beI Schedule B3.1 /I The document SVM/SDM/REP/0008
substituted for actual I Annex 3 - HNG-X Counter Performance
measurements Assessment Results will specify
where design targets or derived
values have been used in place of
actual measurements
9. I The basis for the assessment I Schedule B3.1 / I Section 4.4 specifies the basis for
will be based on the average I Annex 3 calculating these average figures for
of the measurements for the
sets of representative
Transaction Types.
Horizon and HNG-X.
Copyright Fujitsu Services Limited
2009
UNCONTROLLED IF PRINTED OR LOCALLY
‘Commercial in Confidence
STORED
Ref: SVM/SDM/PRO/0017
Version: 1.0
Date: 17-JUL-2009
Page No: 9 of 24
fee)
FUJITSU
FUJ00232610
FUJ00232610
Transaction Time Benchmarking Process
Commercial in Confidence
10.I Video benchmarking shall be I SVC-809 Section 3 and the document
replaced by system generated SVM/SDM/REP/0007 — Validation of
transaction component times SVC-810 HNG-X System Derived Transaction
Component Times describe how the
1) change from video benchmarking
has been validated.
11.I Video Benchmarking shall be I SVC-807 Existing Horizon video benchmark
used to derive transaction data from S92 will be used a part of
component times for HNG-X I SVC-855 the counter performance
and Horizon transactions assessment.
For the HNG-X transactions it is
proposed that the system generated
transaction times are used as the
basis for deriving the component
time data as an alternative to Video
Benchmarking.
This approach is described in section
3 and the validation of the system
generated data is described in
SVM/SDM/REP/0007 — Validation of
HNG-X System Derived Transaction
Component Times
Copyright Fujitsu Services Limited
2009
Commercial in Confidence Ref: SVM/SDM/PRO/0017
Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY Date: 17-JUL-2009
STORED
PageNo: 10 of 24
FUJ00232610
FUJ00232610
Transaction Time Benchmarking Process
2
FUJITSU
Commercial in Confidence
2 Counter Performance Assessment Process
The counter performance assessment process comprises two main streams of activities: -
e Stream 1 - Verify the system generated transaction component times
e Stream 2 - Create and compare the transaction times of the Horizon & HNG-X
representative basket mix.
The objective of Stream 1 is to establish that the system generated transaction component times are
accurate. This will be achieved by using a proven measurement technique — Winrunner — to obtain
timing data for defined transaction sequences and then validating these values against the equivalent
system generated component time data.
The objective of Stream 2 is to establish the representative basket mix and the associated transaction
types, obtain the component time data, and generate the comparison data.
2.1 Process Overview
The main stages of each of the two processes are shown below: -
Stream 1 I Stream 2
Establish Benchmark period
+ Establish
transactions that
include a range of
measurement points
‘Obtain Horizon transaction /
settlement & session data
Create the
representative
basket mix
Identify sample
transactions
Transform actual transactions
into transaction type data
Rank by occurrence
© Create Winrunner
Establish scripts scripts and establish
and common Apply $92 component time
measurement measurement points Create the data to transaction types
points average basket (including any derived values)
times for Calculate Horizon transaction
Horizon times by type, and average
i basket times.
Execute
transactions and I + — Obtain both sets of Apply system derived
obtain timing timing data Create the ‘component time data to
data (Winrunner transaction types
and system)
average basket
times for HNG-X Calculate HNG-x transaction
times by type, and average
basket times.
* Analyse and compare
results to verify
Analyse data accuracy of system
and conduct generated data
comparison
Analyse and Compare data for agreed % of
overall basket mix
compare the
results
sapyright Fujitsu Services Limited Commercial in Confidence Ref: SVM/SDM/PRO/0017
Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY Date: 17-JUL-2009
STORED PageNo: 11 of 24
FUJ00232610
FUJ00232610
Transaction Time Benchmarking Process
fee)
FUJITSU Commercial in Confidence
3 ‘Stream 1’ activities
The POL Requirements SVC-808 and SVC-809 state that Video Benchmarking should be applied to both
Horizon and HNG-X transactions (for the purposes of comparison) and that subsequently transaction
component times should be derived directly from the counter application. SVC-810 states that this
transition requires that the results must be calibrated (to verify that the system derived times are
comparable to previously acquired values).
Fujitsu have assumed that the requirement to employ video benchmarking for HNG-X was intended
solely to acquire transaction time data using an existing and established technique.
Fujitsu believe that a more flexible approach can be employed using Winrunner, a standard transaction
scripting tool used by Joint Test Team, rather than Video Benchmarking. This will enable the accurate
execution of multiple scripts to be run and modified as necessary for a range of transactions. Winrunner
scripts also provide transaction component times.
Accordingly Stream 1 will define a range of HNG-X transactions that will be executed by Winrunner with
timing data being obtained from both Winruner and the counter application. Demonstrating that these
times are comparable will allow system generated timings to be introduced and used in the main counter
performance assessment, and remove the need to conduct any HNG-X Video Benchmarks.
3.1 Define Sample Transactions
Stream 1 will define a sample set of transactions that include a range of measurement points including: -
e Arrange of different user input and output UI constructs
e Transaction steps involving the operation of all counter peripherals
e Initiation, wait and respond to on-line steps
e Report production
e The use of local (e.g. cash) and on-line settlement
The number of transactions is expected to be relatively small (<20) since this activity is only concerned
with verifying that a representative selection of measurement points has been included.
The specific transactions and their measurement points will be described in SVM/SDM/REP/0007 —
Validation of HNG-X System Derived Transaction Component Times.
3.2 Obtain Timing Data
The set of transactions identified above will be run a number of times and the component times from
both sets of measurements averaged.
The types of component time data that will be obtained includes: -
« Counter processing time
e User waiting time
e Peripheral time
¢ ‘On-line’ communication time
as well as the total duration of the transaction.
It will be important to verify that the measurement points for both sets of timings are identical, such that
accurate comparisons can be made. The granularity of all of the measurement points may be different
sapyright Fujitsu Services Limited Commercial in Confidence Ref: SVM/SDM/PRO/0017
Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY Date: 17-JUL-2009
STORED PageNo: 12 of 24
FUJ00232610
FUJ00232610
(oe) Transaction Time Benchmarking Process
FUJITSU Commercial in Confidence
between the two sets of timings but there will always be a set of common start and stop points from
which the data will be drawn.
3.3 Verification
The use of system generated component time data will be proven once the average of both sets of
common measurement point data are identical to within agreed tolerances.
sapyright Fujitsu Services Limited ‘Commercial in Confidence Ref: SVM/SDM/PRO/0017
Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY __ Date: 17-JUL-2009
STORED PageNo: 13 of 24
FUJ00232610
FUJ00232610
Transaction Time Benchmarking Process
fee)
FUJITSU Commercial in Confidence
4 ‘Stream 2’ activities
Stream 2 is concerned with delivering the counter performance assessment described in Schedule B6.1
and in meeting the various POL requirements contained in Annex A. The principal activities of this
stream are to establish the Basket Mix, obtain the HNG-X and Horizon timing data and to establish the
applicable transaction steps, taking account of any adjustments arising as a consequence of introducing
HNG-X.
4.1 Create Representative Basket Mix
In order that the ‘agreed sets of representative transactions’ (Schedule B3.1 / Annex 3) can be
established it is necessary to analyse data from the ‘Benchmark Period’ in order to ascertain the most
frequently occurring transactions. In order that this reflects the actual business throughput of a Branch
the approach to determining this representative workload will be on the basis of Baskets or Customer
Sessions. These represent the actual units of work undertaken by Branches rather then just establishing
a ranking of discrete transactions.
The representative basket mix will be established by taking the transaction data associated with all
customer sessions for all Branches during the Benchmark period. The source of this data is intended to
be the POL-MI data for this period.
This data will be analysed to establish the number of customer baskets containing one or more ‘logical’
transaction types, these being determined from the actual discrete transactions, and in particular the
product ID of the item being transacted.
For example three separate baskets each containing a single bill payment involving a different utility
company will count as three instances of a basket with a single AP In-Pay transaction.
The results of this analysis will be ranked by occurrence such that a workload profile or Basket Mix
analysis is established identifying the most commonly occurring baskets through to the most infrequent.
An analysis of this type conducted on data approx two years ago showed that whilst there were approx
10,000 different basket / transaction combinations, the top 20 entries accounted for approx 80% of all
baskets. It is expected that a similar workload profile will result from this updated analysis and if this is
the case it is proposed that the top 20 transactions are used as the set for which transaction time
analysis is done.
The results of this analysis will be recorded in SVM/SDM/REP/0009 — HNG-X Representative Basket
mix.
4.1.1 Benchmark Period
The Benchmark Period is the period from which the Horizon transaction data will be drawn to establish
the average workload profile. This period should be representative of normal Branch trading, and should
therefore exclude seasonal periods but should include normal cyclical activities such as month end
trading.
It is proposed that the Benchmark Period will be the two weeks straddling the end of March / start of April
— 23/03/09 to 05/04/09.
This period has been verified as representative by Dave Chapman — architect for HNG-X System
Qualities & Capacity Management.
sapyright Fujitsu Services Limited Commercial in Confidence Ref: SVM/SDM/PRO/0017
Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY Date: 17-JUL-2009
STORED PageNo: 14 of 24
FUJ00232610
FUJ00232610
Transaction Time Benchmarking Process
fee)
FUJITSU Commercial in Confidence
4.1.2 Transaction Types
Each actual transaction will be counted as an instance of the logical transaction type that it relates to.
This essentially ensures that discrete transactions that are realised through the same set of user / system
activities (i.e. realised through the same Use Case) are counted as the same. The set of Transaction
Types will be:-
e Retail — all simple ‘EPOSS' style transactions
e AP In-Pay — all basic ‘AP’ style transactions
e Bureau — all sell and buy Bureau de change transactions
e Banking — all Banking deposit and withdrawal transactions
« ETU -—all types of E-Top up transactions
« Back Office / Admin transactions — see section 4.1.2.1
e Other — all other transactions (excluding those in section 4.1.2.2). Additional Transaction Types
will be introduced as necessary if the volume of transactions / baskets cause them to appear in
the ‘top 20’.
In addition a settlement transaction will be associated with each basket. This will either be a Debit Card
transaction or a local settlement item (i.e. Cash / Cheque etc.).
4.1.1.1 Back Office / Admin transactions
Where possible the Transaction Type analysis will also aim to identify Back Office / Admin transactions
although these may not always be derivable from the overall transaction data.
The volume of such transactions will be much smaller than the customer facing transactions that form
the vast majority of the Basket Mix analysis, and so their overall effect on an average basket is expected
to be quite limited. In addition although the actual transaction content will be different, most Back Office
/ Admin transactions will involve many common transaction components (data input response, peripheral
operation, screen updates etc) that will have been included in the main customer facing analysis.
The results and contribution of any Back Office / Admin transactions will be included in
SVM/SDM/REP/0008 — HNG-X Counter Performance Assessment Results.
4.1.1.2 Candidates for Exclusion
Horizon Mails transactions may be excluded from this analysis as they are wholly replaced by HNG-X
Postal Services which has different user process and system interactions. Only those transaction steps
that are comparable in terms of user interactions will be included in the comparative timing analysis.
See also section 5.
4.2 Create HNG-X Timings
Specific transactions will be identified that represent each of the logical transaction types in the basket
mix analysis. If the most frequently occurring basket is a Banking Cash Withdrawal, then a specific
instance of this transaction will be scripted (using the most frequently occurring product from the product
frequency analysis) and used as the basis for gathering transaction component time data.
Each transaction (representing a transaction type) will be decomposed down into its constituent
component parts these representing the measurement points for which timing data will be obtained.
sapyright Fujitsu Services Limited Commercial in Confidence Ref: SVM/SDM/PRO/0017
Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY Date: 17-JUL-2009
STORED PageNo: 15 of 24
FUJ00232610
FUJ00232610
Transaction Time Benchmarking Process
2
FUJITSU
Commercial in Confidence
These measurement points will separate out user activity (i.e. awaiting user input, entering data), local
counter / peripheral processing, and external / on-line activities.
Transaction scripts will be created for each transaction and recorded in SVM/SDM/SSD/0003 — HNG-X
Representative Transaction Scripts. They will be executed on a suitable test environment.
The applicable component times (see section 4.2.2) taken from the system generated timing data will be
summed for each transaction type within each basket to give an overall duration for each basket entry in
the ‘top 20’.
4.2.1 Product Frequency
The data used to create the basket mix analysis will also be used to ascertain the volume and frequency
of specific products being transacted. This ranking will be used to select the specific product to be used
when identifying a specific transaction to represent a transaction type.
4.2.2 Adjustments
Certain adjustments will be made to the transaction component analysis to allow for those items that are
not applicable to the counter performance assessment calculations. These are:-
e Steps involving User actions — i.e. screen observation, user input or action. For scripted
transactions this will only affect the data input steps since there is negligible ‘wait’ time between
script steps.
e Steps that have changed as a result of any changed processes or UI interactions that have been
introduced in HNG-X. Examples include the introduction of any new message dialogues that
require user confirmation, any changes to the sequence of user actions where a comparable
Horizon sequence does not exist, any changes arising from UI Constructs that require additional
user actions to perform or complete data input.
e Steps that are waiting for an external response. This assessment is concerned with the
performance of the HNG-X counter and so external dialogues will be excluded. In live operation
FS and POL will be monitoring the key online interactions since they are subject to Service
Level Targets and their assessment will take place once a representative workload is in place.
This will not be the case at the point at which this counter performance assessment takes place,
just prior to Pilot.
4.2.1.1 Menu Hierarchy
The Menu Hierarchy for HNG-X has been redesigned and as such there can be no comparison with the
menu navigation undertaken on Horizon. Accordingly for any transactions that are selected from menu
buttons, all user selections and timings associated with menu navigation will (where possible) be
reported but excluded from the comparative timing analysis, since the measurement point for all such
transactions will commence with the final button selection that initiates the transaction.
4.3 Create Horizon timings
Horizon component timings will be applied to each of the transactions that make up the set of
transactions for each entry in the basket mix analysis, taking account of any adjustments. These
component times will be taken from the data gathered as part of the Horizon Video Benchmark analysis
of $92.
sapyright Fujitsu Services Limited Commercial in Confidence Ref: SVM/SDM/PRO/0017
Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY Date: 17-JUL-2009
STORED PageNo: 16 of 24
FUJ00232610
FUJ00232610
Transaction Time Benchmarking Process
fee)
FUJITSU Commercial in Confidence
It is expected that the range of transactions for this assessment will have increased from the set used for
previous Horizon transaction time benchmarking, and so it may be necessary to extrapolate from
existing timing data to establish values for previously unmeasured transaction steps.
The number of these will be reviewed to ascertain whether any other method of obtaining any missing
Horizon component time data is practical and necessary.
4.4 Assessment Results
With the Horizon and HNG-X component time data applied to each of the transaction types for each of
the entries in the ‘top 20’ basket mix analysis, an overall transaction time for Horizon and HNG-X will be
obtained.
Comparing each basket line entry will give a view as the relative performance of HNG-X. The overall
comparison is based on the timing for an average customer basket, and so the set of all entries in the
basket mix analysis is adjusted by the volume of occurrences.
The average Horizon basket time is as follows: -
Basket entry 20
> (Horizon basket time) * (Volume of baskets per entry)
Basket entry 1
(Total volume of baskets for top 20 entries)
The same calculation is done for the HNG-X entries and the HNG-X counter performance assessment is
met if the average basket time for HNG-X is less than or equal to the average basket time for Horizon.
The results of this analysis will be recorded in SVM/SDM/REP/0008 — HNG-X Counter Performance
Assessment Results.
sapyright Fujitsu Services Limited Commercial in Confidence Ref: SVM/SDM/PRO/0017
Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY Date: 17-JUL-2009
STORED PageNo: 17 of 24
FUJ00232610
FUJ00232610
Transaction Time Benchmarking Process
2
FUJITSU
Commercial in Confidence
5 Future Transaction Time benchmarking
As well as meeting the formal counter performance assessment obligation, this process establishes the
foundations on which the on-going Transaction Time benchmarking Service will be based. This service
will support the introduction of future HNG-X releases and will enable the effect of change, either at
transaction or workload profile to be assessed.
The Basket Mix analysis and the HNG-X timings will act as the starting point from which future HNG-X
Release assessments are made. In order that an accurate Release 1 position is established it will be
necessary to substitute the Horizon Mail transactions with the associated Postal Services transactions in
the relevant Basket entries. The HNG-X timings can then be added to complete the starting position for
HNG-X Released 1.
For future releases it is expected that, subject to agreement, the same approach will be used to assess
the impact of changes on the previous baselined position. This revised approach to Transaction Time
benchmarking is expected to deliver the following benefits: -
e Speedier execution of the benchmark transaction set
e Speedier analysis
« More timely availability of results to facilitate feedback prior to deployment to the live estate
e Larger sample size to improve statistical confidence
e Reduced scope for human error
e More flexibility in the benchmark transaction set in response to changing customer needs
The operation of the Transaction Time Benchmarking Service is described in SVM/SDM/SD/0016 -
Management Information Service: Service Description - Annex A.
sapyright Fujitsu Services Limited Commercial in Confidence Ref: SVM/SDM/PRO/0017
Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY Date: 17-JUL-2009
STORED PageNo: 18 of 24
2
FUJITSU
FUJ00232610
FUJ00232610
Transaction Time Benchmarking Process
Commercial in Confidence
A
Extract from Schedule B6.1
The following text is reproduced from Schedule B6.1, Annex 3 version 2.0
The Business Equivalence Principles do not apply to the System Capacity and Performance
Requirements or associated Service Levels, but it is accepted that the performance
characteristics of the HNG-X System counter applications and associated infrastructure must be
acceptable to the Post Office. This specific aspect of the Solution Baseline will be assessed to
determine that the HNG-X System counter performance delivers equivalent or better
performance characteristics to the Applicable Horizon Baseline, based on the following
principles:
4.1 Post Office and Fujitsu Services will jointly agree an assessment process that will:
« define and agree sets of representative Transaction types and a measurement period;
e establish system component benchmark measurements for these Transaction types on
the Horizon system and on the HNG-X System (taking account of any changed
processes or UI characteristics that may be present in the HNG-X Transaction); and
« allow Post Office to request that the set of Transaction types is augmented (agreement
to such requests not to be unreasonably withheld by Fujitsu Services) if anomalous HNG-
X Transaction performance characteristics are identified prior to the commencement of
the HNG-X Project Workstream X4 (HNG-X Application Rollout).
4.2 Acceptance by Post Office of the HNG-X System counter performance will be based on the
average system component benchmark measurements for each of the agreed sets of
representative Transaction types for the HNG-X System being no worse than the comparable
measurements for each of the Transaction types within the Applicable Horizon Baseline or where
comparison is not applicable, to their agreed design targets.
Copyright Fujitsu Services Limited
2009
Commercial in Confidence Ref: SVM/SDM/PRO/0017
Version: 1.0
UNCONTROLLED IF PRINTED OR LOCALLY Date: 17-JUL-2009
STORED PageNo: 19 of 24
Fe)
FUJITSU
Transaction Time Benchmarking Process
Commercial in Confidence
B- Applicable POL Requirements
The following requirements have been identified as being applicable to the HNG-X Counter Performance assessment. Note that the details
below are taken from the DOORS system as at the date of this document. The current version of all POL requirements are held in the POL
DOORS system.
Original Original Object text Acceptance Criteria Verification I Parent/ I Acceptance
Object Id Method Child Gateway
SVC-795 I A Benchmark Period shall be agreed Parent
following contract during which
current business volumes and
measurements of Baseline Horizon
performance of daily Post Office
Business and Operations, will be
used to update Capacity Model peak
loading and confirm other capacity
assumptions for Capacity and
Performance Models.
SVC-842 I A Benchmark Period shall be agreed There will be an appropriate section in DR Child AG3
following contract during which current aCCD or a document referred to by a
business volumes and measurements of I CCD which contains a clause
Baseline Horizon performance of daily confirming this obligation. An artefact
Post Office Business and Operations, within the Solution Baseline
will be used to update Capacity Model Documentation Set will be supplied
peak loading and confirm other capacity I which cross references this requirement
assumptions for Capacity and with the appropriate clause will be
Performance Models. supplied.
Copyright Fujitsu Services Limited Commercial in Confidence Ref: SVM/SDM/PRO/0017
2008 Version: 0.4
UNCONTROLLED IF PRINTED OR LOCALLY STORED Date: 26-JUN-2009
Page No:
20 of 24
FUJ00232610
FUJ00232610
Fe)
FUJITSU
Transaction Time Benchmarking Process
Commercial in Confidence
SVC-843 I A Benchmark Period shall be agreed There will be an appropriate section in SOF Child AG3
following contract during which current a CCD or a document referred to by a
business volumes and measurements of I CCD which contains a clause
Baseline Horizon performance of daily confirming this statement of fact. An
Post Office Business and Operations, artefact within the Solution Baseline
will be used to update Capacity Model Documentation Set will be supplied
peak loading and confirm other capacity I which cross references this requirement
assumptions for Capacity and with the appropriate clause will be
Performance Models. supplied
SVC-798 I Analysis of data in the Benchmark There will be an appropriate section in soo AG3
Period shall be used to validate the aCCD or a document referred to by a
assumptions and modelling CCD which contains a clause
parameters for volume of Customer confirming this obligation. An artefact
Sessions that require Settlement and _ I within the Solution Baseline
the number and most frequent Documentation Set will be supplied
occurring combination of products in I which cross references this requirement
a Session Basket. with the appropriate clause will be
supplied.
SVC-799 I The volume of Administrative and There will be an appropriate section in I DR AG3
Back Office functions occurring a CCD or a document referred to by a
across the Estate during the CCD which contains a clause
Benchmark Period shall be measured I confirming this obligation. An artefact
on Baseline Horizon and these within the Solution Baseline
adjusted according to any modified Documentation Set will be supplied
User behaviour or Business which cross references this requirement
Processes expected by Post Office as I with the appropriate clause will be
a result of the move to the new supplied.
System, for use in Capacity and
Performance Models.
Copyright Fujitsu Services Limited Commercial in Confidence Ref: SVM/SDM/PRO/0017
2008 Version 0.4
UNCONTROLLED IF PRINTED OR LOCALLY STORED Date: 26-JUN-2009
PageNo: 21 of 24
FUJ00232610
FUJ00232610
Fe)
FUJITSU
Transaction Time Benchmarking Process
Commercial in Confidence
SVC-805 I Transaction Time Benchmark An entry in an artefact within the DR AG3
measurement shall be carried out in Solution Baseline Documentation Set
accordance with the Service will be supplied which cross references
Management Service and the method I this requirement with the appropriate
described in the Transaction Time document and the relevant clause
Benchmarking, Joint Working within the document will be supplied.
Document (CS ). The document will describe how this
requirement will be catered for in the
design.
SVC-806 I Transaction Time Benchmark An entry in an artefact within the DR AG3
evaluation shall be conducted in Solution Baseline Documentation Set
accordance with the process will be supplied which cross references
described in the document ‘Counter this requirement with the appropriate
Transaction Time Performance - document and the relevant clause
measurement and results within the document will be supplied.
(CS/PER/046)'. The document will describe how this
requirement will be catered for in the
design. branches.
SVC-807 I New Video Benchmarks shall be There will be an appropriate section in I SOO AG3
defined for a limited number (to be a CCD or a document referred to by a
agreed) of representative Single and I CCD which contains a clause
multi-product Baskets that will at confirming this obligation. An artefact
least include an example of each within the Solution Baseline
Transaction Type. Documentation Set will be supplied
which cross references this requirement
with the appropriate clause will be
supplied.
Copyright Fujitsu Services Limited Commercial in Confidence Ref: SVM/SDM/PRO/0017
2008 Version 0.4
UNCONTROLLED IF PRINTED OR LOCALLY STORED Date: 26-JUN-2009
PageNo: 22 of 24
FUJ00232610
FUJ00232610
Fe)
FUJITSU
Transaction Time Benchmarking Process
Commercial in Confidence
SVC-808
During testing New Video
Benchmarks shall be conducted on
Baseline Horizon and equivalent
Video Benchmarks shall be
conducted on the System, in
accordance with the process
described in HNG-X Test Strategy -
HX/STR/001 and assessment made
according to the Video
Benchmarking Tolerances agreed
therein.
Parent
SVC-855
During testing New Video Benchmarks
shall be conducted on Baseline Horizon
and equivalent Video Benchmarks shall
be conducted on the System, in
accordance with the process described
in HNG-X Test Strategy - HX/STR/001
and assessment made according to the
Video Benchmarking Tolerances agreed
therein.
An entry in an artefact within the
Solution Baseline Documentation Set
will be supplied which cross references
this requirement with the appropriate
document and the relevant clause
within the document will be supplied.
The document will describe how this
requirement will be catered for in the
design.
DR
Child AG3
SVC-856
During testing New Video Benchmarks
shall be conducted on Baseline Horizon
and equivalent Video Benchmarks shall
be conducted on the System, in
accordance with the process described
in HNG-X Test Strategy - HX/STR/001
and assessment made according to the
Video Benchmarking Tolerances agreed
therein.
Performance testing during ST will
demonstrate capabilities in accordance
with HNG-X Test Strategy.
ST
Child AG3
Copyright Fujitsu Services Limited
2009
Commercial in Confidence
UNCONTROLLED IF PRINTED OR LOCALLY STORED
Ref: SVM/SDM/PRO/0017
Version: 04
Date: 26-JUN-2009
Page No: 23 of 24
FUJ00232610
FUJ00232610
Fe)
FUJITSU
Transaction Time Benchmarking Process
Commercial in Confidence
SVC-809 I System Measurement of Counter There will be an appropriate section in Soo AG3
Performance will be introduced in the I a CCD or a document referred to by a
System as a future alternative to CCD which contains a clause
Video Benchmarking. confirming this obligation. An artefact
within the Solution Baseline
Documentation Set will be supplied
which cross references this requirement
with the appropriate clause will be
supplied.
SVC-810 I System Measurement of Counter There will be an appropriate section in SOF AG3
Performance may be calibrated with aCCD or a document referred to by a
the Video Benchmark results CCD which contains a clause
obtained for the System for confirming this statement of fact. An
equivalent sequences of operation. artefact within the Solution Baseline
Documentation Set will be supplied
which cross references this requirement
with the appropriate clause will be
supplied
---- End of document ----
Copyright Fujitsu Services Limited Commercial in Confidence Ref: SVM/SDM/PRO/0017
2008 Version 0.4
UNCONTROLLED IF PRINTED OR LOCALLY STORED Date: 26-JUN-2009
Page No: 24 of 24
FUJ00232610
FUJ00232610