POL00337565
POL00337565
POL00337565
POL00337565
1 Document Management
Delivery ID D: 09-019
Workstream ID WS09 - Testing
About this Document POL organisation quality assurance strategy
Document Author Rohit Gogna, Horizon Remediation Programme Test Lead
Referenced Documents D: 09-028 Defect Triage Meeting
D: 09-005 POL Test Policy
POL IT Landscape v1.81
D: 09-21 Test Environments Analysis Deck
1.1. Disclaimer
This document is for use by Post Office Limited (POL) staff and other audiences under NDA
only. This document is confidential and is not to be copied, distributed, or reproduced in
whole or in part, nor passed to any third party, without the written consent of the Head of
QA & Test Management and the Post Office Limited.
1.2 Version History
Version __ Change Reason By Date
Vo.1d Initial Draft Rohit Gogna 07/12/21
V1.0 Signed off Rohit 24/05/22
1.3 Document Approvers (and Reviewers)
Name Title Approval Date
Simon Oldnall Horizon IT Director Yes
Dan Addy Chief Architect No
Dean Bessell Head of Security and Risk No
Ajay Patel Head of Enterprise IT Demand No
Martin Godbold Head of Horizon Live Services No
Katrina Holmes Head of Branch Operations No
1.4 Document Reviewers
Name Title Version Date
Stuart Beech Change Management Lead VO.2d TBC
Harshwardhan Somon Senior Test Manager Vo.2d TBC
Sarah Birch Test Architect Vo.2d TBC
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
2
aw
x
©
©
Table of Contents
Document Management .
1.1 Disclaimer
1.2 Version History ......
1.3. Document Approvers (and Reviewers)...
1.4 Document Reviewer:
Table of Contents
Glossary of Terms.
Introduction ....
4.1 Document Objective
4.2 Scope of Document ..
4.3 Intended Audience .......
4.4 Overview of Document...
45 Document hierarachy.
Post Office Overview ...
Horizon Solution Overview.
6.1 POLIT Landscape.
6.2 POLIT Landscape Summary...
Post Office Test Approach
7.1 POLTest Centre of Excellence...
7.2 POL Quality Assurance Framework
7.3. Testing Principals...
74 Testing Objectives at POL.
7.4.1. Service Virtualisation.
7.42 Automation Testing...
7.43 Quality Standards and Goals..
7.4.4 Risk-Based Testing (RB7)......
Test Activities Summary
8.1 Delivery Test activities workflow.
Test Activities ...
9.1 Design phasi
9.1.1 Programme Test Strategy...
9.1.2 Static Testing... sine
9.1.3 Automation Framework Implementation...
9.2 Development phase...
9.2.1 Unit Testing and Test-Driven Development.
9.3 Testing phase.
9.3.1 POL System Integration Testing .
9.3.2 POL System Acceptance Testing
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
9.4 Quality Assurance Phase
9.4.1 Non-functional Testing...
9.4.2 POL Business Acceptance Testing...
9.4.3 User Acceptance Testing...
9.5 Live phase s.cscs scam neTTTCT TTT
9.5.1 Problems and Incidents in Live.
10 POL Quality Gates........
11 Entry, Exit and Suspension Criteria
111 _ Entry Criteria. .
11.1.1 Sprint Testing (commencement only)
11.1.2 System Integration Testing
11.1.3 System Acceptance Testing
11.1.4 Performance Testing.
1115 Penetration Testing......
11.1.6 Business Acceptance Testing.
11.1.7 User Acceptance Testing,
11.1.8 Operational Acceptance Testing...
112 — Exit Crit
11.2.1 Sprint Testing
11.2.2 System Integration Testing
11.2.3 System Acceptance Testing
11.2.4 Performance Testing.
11.2.5 Penetration Testing...
11.2.6 Business Acceptance Testing .
11.2.7 User Acceptance Testing.
11.2.8 Operational Acceptance Testing
11.3 Defect Ackowledgement Times..
11.4 Suspension of Testing..
11.4.1 Test Resumption Criteria...
12 Scope of Testing:
12.1, Out of Scope for POL Test Team.
13 Defect Management.....
13.1 Defect Classification
13.1.1 Severity Defi
13.1.2 Priority Definitions
ions.
13.2 Defect Process wou .
13.2.1 Defect Triage Meeting (DTM)
13.3. Defect Statuses...
13.4 Defect Workflow.
14 ‘Test Data.....
14.1 Pre-test execution
14.2 Obtaining Test Data..
15 Test Organisation ws.
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
POL00337565
POL00337565
15.1 Horizon Quality and Test Management ......
16 Test Roles and Responsibilities.
16.1 POLTest Leadership
16.2 POL Test Management...
16.3 POL Test Team .ecnsnn
17 Envirionments.
17.1 Current environments
17.2 Proposed environments
18 Test Deliverables...
18.1 Supplier Test Deliverables...
18.2 _ Deliverables to POL Testing...
18,3 Deliverables from POL Testing s.r...
19 POL Test Reporting ..
19.1 Portfolio Reports
19.2 Programme Reports
20 Assumptions and Dependencies.
20.1 Assumptions......
20.2 Dependencies...
21 Appendix...
21.1 — Reference Data Test Readiness Checklist.
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
3 Glossary of Terms
Term Definition
POL Post Office Limited
sit System Integration Testing
SAT System Acceptance Testing
BAT Business Acceptance Testing
UAT User Acceptance Testing
NFT Non-functional Testing
OAT Operational Acceptance Testing
RBT Risk Based Testing
SLA Service Level Agreement
TDD Test Driven Development
BDD Behavior Driven Development
cR Change Request
sc Source Control
TooE Test Centre of Excellence
QA Quality Assurance
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
4 Introduction
4.1 Document Objectives
The objectives of this document are:
* To ensure that there is a collective understanding of the test activities, terms, and
responsibilities across the Post Office Limited.
* Provide details to key POL stakeholders of POLs test strategy (per Programme/Project)
which will provide confidence to the business. This will enable a robust process to ensure
Horizon Changes are deployed into the Production environment, as part of the route to
live.
* To obtain sign-off from key Business stakeholders thatthe testing approach/plan is
achievable and responsibilities of testing activities are understood.
4.2. Scope of Document
The document provides an overview of the testing to.be undertaken by the POL Test
Capability in the implementation and the deployment of Horizon solutions. Therefore, the
Test strategy will consider both continuous minor changes such as Operational and Service
enhancements alongside major changes in the form of POL programmes and/or projects.
The scope of this document:
* Describes the scope of testing to be carried out
Identifies POL test processes
Identifies the various test environments
Identifies key testing deliverables
Identifies test data requirements
Details the POL entry/exit per test phase and suspension criteria
Details how defects are to be managed at POL
Identifies test tools that will be used at POL
Identifies POL test resource and responsibilities
4.3 Intended Audience
The intended audience for this document are:
* POL stakeholders to provide sign-off and agreement of this document
© Test resource for information purposes to enable them to understand _ their
responsibilities’
* Development for information on the approach and their responsibilities
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
POL00337565
POL00337565
4.4 Overview of Document
This is a Quality Assurance Strategy document for POL. This document is to be used by POL
delivery teams, Partners, Suppliers, and stakeholders, as a guide of the quality assurance and
testing activities at POL.
It should be noted that Programme Test Strategies are to be delivered by Programmes of
work at POL and this this document sits at an organisational level and therefore covers how
QA is to be delivered at POL as a whole. It is envisaged that Programmes will dovetail their
Programme strategies in line with this document to ensure that robust testing has been
carried out
Testing efforts will be prioritised based on Horizon Change business priorities, which are
defined in the POL Demand Tracker, in addition to the functional and technical specifications.
This document is deemed to be a ‘living document’ that may be re-defined by the Head of
QA, as the POL test capability evolves and progresses. New versions of this document will be
tributed to all interested parties.
The following approver list have reviewed and agreed to POL QA Strategy.
a
Simon Oldnall Horizon IT Director
Dan Addy Chief Architect
Dean Bessell Head of Horizon Security and Risk
Ajay Patel Head of IT Enterprise Demand Management
Martin Godbold Head of Horizon Live Services
Katrina Holmes Head of Branch Operations
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
4.5 Document hierarchy
The illustration in this section conveys the test document hierarchy at POL. IThis will be
landscape mode in version 0.2d
‘raison Test Process
riaton POL Test, poLgasraeey
Policy drives the POL QA ans into the POL Bleach
ww ne “standardise the test
‘Srmary gn Seay oes
opens
wee
bose th etn
"ecu ne GA
Sracay
“entinaoement reas
wv
: 09-019: POL Test Strategy - Confidential
‘Copyright 2022 Post Office Limited
POL00337565
POL00337565
5 Post Office Overview
POL has thrived at the heart of high streets and local communities across the UK for over 370
years. As one of the country’s most trusted brands, POL takes its commitment to providing
essential services to customers across the UK very seriously.
POL is the UK's largest retail network, as well as the largest financial services provider in the
UK, with over 11,600 branches nationwide — more than all the UK’s banks and building
societies put together.
The POL organisation understands that the best way to provide a great'service for customers
is to evolve the business and adapt to the customers changing needs. That is why POL have a
range of over 170 products and services, from personal financial services such as, banking,
insurance, payments and travel money, to telecoms and, of course, mails.
Working at the Post Office:
Post Office colleagues are the driving force behind the POL business. Whether they are in POL
branches or supporting from the POL offices, the business is proud of the energy,
commitment and customer focus our people, all have in common.
All Post Office people are guided by three vallles.and behaviours, see Code of Business
Standards
© We care by always thinking customer
* We strive to make'things ever better through honest challenge
+ We commit to decisive delivery
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
6 Horizon Solution Overview
6.1 POLIT Landscape
—--II
_
6.2, POLIT Landscape Summary
For avoidance of duplication and for synchroneity purposes, please refer to the POL IT
Landscape document (a ‘living document’) by clicking here. If you require access to this
document, please contact your line manager.
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
7 Post Office Test Approach
7.1 POL Test Centre of Excellence
A Testing Centre of Excellence (TCoE) is to be set up at POL. The TCoE capability is to be
developed by the TCoE Manager, alongside the Head of QA. The TCoE will function at the POL
organisational level. This will enable it to govern, measure and implement uniformity on POL
programmes. The TCoE will include the following goals:
Commitment to quality through t.
‘admap of achievable testing related goals
Automation of all repeatable t developed by
election correlating to the TCoE objectives
3
@
3
g
5
8
=
2
2
Training of the POL Test Team in relation to the TCoE object
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
The illustration below conveys the remit of the TCoE and its level of test governance at the
organisational level. It should be noted that the said remit is only applicable at a
7.2 POL QA Governance Framework.
‘The POL QA Governance Framework is to be developed and implemented by the Head of QA
and POL TCoE Team, and.will be designed to function as an organisational level testing
approach, with a shift-left mentality. It will seek to work in a collaborative fashion with
supplier and internal development teams. The illustration below conveys the [ x ] steps that
are to be repeated throughout the processes, giving POL, a repeatable testing model. The
steps are to be implemented by the POL Test Management layer, with support from the POL
Test Leadership layers
{enter illustration here]
The aim of the POL QA Framework is to ensure swiftness, as well as nimbleness is possible,
whilst meeting the quality goals, driven by KPI's set out by the TCoE.
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
7.3. Testing Principals
The following test principles are to be adopted at POL:
ingand
of change
function
wuirements and
ting are to be
sTesting is to be
considered early in the
development lifecycle
integrations are to be
considered when
impacting change environment
Note: These test principals are captured in the POL Test Policy please refer to the POL Test Policy document for
further de!
7.4 Testing Objectives at POL
The objectives of the POL Test Capability are to confirm the solution meets the signed-off
functional and non-functional»requirements, through verification, evidenced by test
reporting. Failure to meet said requirements are to be captured in the central Test repository
- Jira Software tool. The POL Test Capability are to address deviations from requirements, as
early as possible in the software development life cycle and help facilitate thought leadership,
to the resolution of said deviations. This is to include static testing of pre-signed off
requirements.
To meet these objectives, QA processes and activities will include:
«The POL Test Team being provided ample time to perform static testing and to feedback
on requirements
Determining and reporting the quality standards of the solution are being met by
implementing key quality gates throughout the delivery life cycle (including supplier
quality gates into POL)
* Providing support to the business/representative and end-user teams in validating that
the overall solution meets the described requirements, (both functionally and non-
functionally)
* Providing the business confidence through evidence-based metrics that the solution is fit
for purpose to move to the next quality gate and through to live. Where this is not the
case assist in mitigating the risk through issue management
* Supporting Operational Acceptance Testing (OAT) to make sure the exit criteria is met to
successfully support the live operation of a component and overall solution.
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
7.4.1 Service Virtualisation
In scenarios where a software dependency has been identified by the POL Test Team, the
objective of the team is to add this dependency to the POL Test Services RAID log and raise
this to the appropriate development team, as a requirement to add ‘Service virtualisation’ so
the dependency can be simulated. This will enable the team to provide testing services and
reports, in line with testing objectives prior to a solution being built.
For avoidance of doubt, this approach is deemed to be applicable at both an organisational
level and at a programme level. Service virtualisation at a programme level is considered a
service that is applicable to only the single programme. In this scenario, the internal
programme teams are to manage service virtualisation development. Where service
virtualisation is beneficial at a portfolio level (org level), the service is to be available to all
programmes for test purposes.
7.4.2 Automation Testing
To achieve the objectives above, an automated regression suite is to be created by the POL
Automation Test Team, that will enable all regression testing to run every time code is
released to an environment, whilst providing the POL Test Team with enough time to honour
the testing objectives outlined above. Both manual and automation test scripts are to be run
in environments, which are representative test environment and are as close as possible to
live.
Like service virtualisation, an automation framework is tobe developed at an organisational
level. At this level, the framework is considered an internal POL product that is useable by all
programmes. Just like a software. product, it will have its own source control and releases,
that will be distributed internally.
Scenarios where a standalone automation framework is to be created fora single programme,
is to be discussed and approved by the POL Automation Manager. It should be noted that not
everything is automatable, and) in some cases, automation candidates may be executed
better, as manual test cases due to complexities or other reasons.
The POL Automation Team are to set up a scoring system, in cases where this is questionable
and to display that the decision was metric driven, should there be an audit.
A QA Automation Approach document is to be created by the POL Automation Manager,
which will be a detailed extension of the POL QA Strategy.
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
7.4.3 Quality Standards and Goals
This section is intentionally left blank and will be worked on and populated with the TCoE
Manager.
7.4.4 Risk-Based Testing (RBT)
Risk Based Testing (RBT) is a method in software testing in which features, or discreet
components are prioritised by forming a Risk Rating against each feature or component. The
formula to calculate the risk, is as follows — should there be a failure, what is the:
Impact to business x Likelihood of failing = Risk Rating
The POL Test Team on any given programme of work, are in the first instance to focus testing
efforts on features and components that are deemed to be of high priority. This will start with
a review of the requirements for areas that are deemed high priority.
All risk analysis is to be added to a dedicated folder within a programmes folder structure, in
the POL SharePoint instance. This is to empower all resources that are tagged to working on
the programme to contest a risk rating for a given feature or component via a review process
that is to be set up by the testing authority on the said programme.
Test cases are to be development by the POL Test Team thereafter. The POL Test Team are to
develop in the first instance test cases for all high-priority features and components and then
medium and low-priority. Within priority ‘batch’ the test cases are to similarly be prioritised
and follow a high, medium, and low approach. The test cases are to be tagged with a priority
within the POL TestRail instance.
Per a functional area, the above distribution is expected, in which a small number of high-
priority test cases are developed, a high number of medium-priority test cases and finally a
small number of low-priority test cases are created, for execution purposes
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
8 Test Activities Summary
8.1 Delivery Test activities workflow
In Progress.
A
<
al
Sy
: 09-019: POL Test Strategy - Confidential
‘Copyright 2022 Post Office Limited
POL00337565
POL00337565
POL00337565
POL00337565
9 Test Activities
The high-level phases illustrated below are applicable to both changes made to the solution
through the demand process, as well as other programmes of work within the Post Office.
9.1. Design phase
9.1.1 Programme Test Strategy
Where the POL Test Team are part of a programme, a Programme Test Strategy document is
to be produced, prior to development starting. The Programme Test Strategies at the Post
Office are to be templated and are not to deviate from. the QA Strategy (this document),
without approval from the Head of QA. All Programme Test Strategy documents, along with
the programme stakeholders are to be reviewed and signed-off by the Head of QA.
Where the POL Test Team are testing changes which are deemed to be BAU operational
demand, the QA Strategy (this document) is to be followed to ensure the correct quality gates
are met for change to be signed off by the POL Test Services Team.
9.1.2 Static Testing
9.1.2.1 Design Feedback
Testing is to start as early as the design phase. The POL Test Team are to work with the
Architect Team to review the design and potential weaknesses. Early feedback of the designs
is to be completed post-test review, through review meetings. The POL Test Team are to set
up meetings with the Architect, as part of the static testing process. The meetings are to help
identify and feedback on risks which are required to be discussed and potentially resolved
Following this approach will assist with the overall quality of design and requirements, which
delivery will rely on as a base for the construction of the solution.
In parallel, [POL Security Services] are to perform a review of the design from a security
perspective, to provide feedback on the design and mitigate security vulnerabilities caused
by design issues.
Performing static testing early in the development cycle, will enable a shift-left approach and
enables the Post Office to move from a defect detection to defect prevention approach.
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
9.1.2.2 Functional and Technical Review
Prior to requirements being sent to the business or a supplier to be signed-off, the POL Test
Team are provided ample time in which to perform a review of said requirements, whereby
the POL Test Team are to challenge and attempt to ‘break’ the requirements through static
testing. If discrepancies are found, these are to be listed and discussed with the [POL Design
Team].
The aim of this approach is to reduce misunderstandings of detailed requirements both
internally and externally (where Post Office suppliers are involved). Furthermore, this
approach will specifically highlight requirements that compel deeper specifications. The
outcome is to have the audience to have a clearer and concise requirement.
The review process is also considered a time for a POL Tester to understand what needs to be
tested and how the process and flow of the overall solution has the potential to break. The
review process, therefore, builds upon the creation of developing both manual test cases and
automation test scripts.
Under no circumstances are the POL Test Team to review or accept requirements for test
development which are in emails, instant messages or similar. Only requirements in the POL
Jira Software instance are applicable for test reviews and thereafter development activities.
9.1.2.3, Requirements Sign-Off
Prior to any development activities taking place, the requirements are to be reviewed and
signed off by the business, This will build upon an initial quality gate for both Post Office and
supplier development activities:
Under _no circumstances are the Post Office or Post Office suppliers to commence
development activities with requirements that are not signed-off.
9.1.3 Automation Framework Implementation
The goal of the POL Test Team is to adopt a shift-left approach to software testing. To support
this goal, the POL Automation Test Manager is to create and implement an automation testing
framework, A three-tier approach is to be taken to automation.
Tier 1: Automation of unit tests
© Tier 2: Automation of APIs (Application Programming Interface)
© Tier 3: Automation of Ul — E2E scenarios
Where automation is required on a programme outside of the two-tiers described above, the
approach to automation is to be guided by the programmes, Programme Test Strategy.
APOL automation approach is to be created by the POL Automation Test Manager, which will
be a detailed extension of the POL QA Strategy.
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
9.2 Development phase
9.2.1 Unit Testing and Test-Driven Development
Unit testing is a software development process that relies on the repetition of short
development cycles where a requirement is turned into a small, well defined and discreet
‘test’ script written in code. The software delivered is then developed to solve the discreet
‘test’ script written in code. Further test scripts and working software are added, until the
requirement is fully realised as a delivered feature.
This promotes a lean approach to development, specifically coding. Developers’at the Post
Office are to only write enough code to resolve the resolution of the unit test and to create a
function that helps that unit test to pass. As a result, the code is frequently smaller, more
efficient and of a better design and quality.
During a development phase, the Post Office Developer is to create a unit test for a discreet
function following the method described. A unit test is thereafter to be executed prior to the
code being committed to the source control and as part of the build process.
Scenarios where unit tests do not obtain a 100% pass rate, the Test-Driven Development
approach is to be followed to ensure that the failing test(s) are investigated. The diagram
below illustrates the approach in whith'tthis is to be accomplished.
9.2.1.1 Iterative Testing
Where agile delivery methodologies are implemented on programmes of work (such as
Scrum), during the development phase, the developer and testers are to work collaboratively
on the delivery of requirements. Tester(s) are to work on the creation of test scenarios, cases
and scripts, which will be executed once a ‘Task’ is in a ‘Ready for Testing’ status. It should be
noted that the accumulation of Tasks, would therefore make up a ‘Story’. Requirements are
to be gathered in the POL Jira Software instance.
During a Sprint, the POL Development Team will test at a Task and User Story level to verify
that the solution meets the acceptance criteria (and moves us closer to meeting the Definition
of Done (DoD)). A Product Owner (PO) is to validate the Story meets the requirements.
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
9.3 Testing phase
9.3.1 POL System Integration Testing
System Integration Testing (SIT) will start once the entry criteria into SIT is met. This will be
the first instance in which the solution will go through a formal entry criterion. Where
possible, the environment in which SIT is conducted is to point'to live or test integrations.
Where this is not possible, virtualised stubs are to be used.
The POL SIT phase will validate that the solution components and interfaces integrate with
other components correctly and support the fundamental end-to-end business processes.
The SIT phase is to be scenario based and event driven using a risk-based testing approach.
Where appropriate and to minimise risk, the POL Test Team are to test against non-human
created data i.e. machine-driven data, that is as close as possible to the expected live data.
This is particularly important for regression, testing activities, where consistency will be
paramount.
The approach to POL SIT will be to perform both positive and negative scenarios to observe
and report how the system behaves to happy paths and to errors and exceptions. The POL
Test Team are to develop POL SIT suite of test cases via the data journeys. These will be part
of an overall regression test suite and therefore scripts will be marked with a priority. All test
cases are to be held in the Gurock TestRail POL instance.
ASIT Test Plan.document will be created for each release. The SIT Test Plan will be a detailed
extension of a POL test strategy
9.3.2 POL System Acceptance Testing
System Acceptance Testing (SAT) will start once the entry criteria for SAT is met. Where
possible, the environment in which SAT is conducted is to point to live or test integrations.
Where this is not possible, virtualised stubs are to be used.
System Acceptance Testing is to consist of functional testing of the Core Business Application.
The aim of this approach is to assure the solution has been delivered to requirements and to
test with an aim to break the application. Scenarios identified where interfaces are required
but are directly not available to the POL Test Team will constitute as a candidate interface
that requires the development team to help mimic the flow of data, through a stub.
Although SAT will primarily be executed manually, the SAT regression suite is to be
automated. It should be noted, the second time a script is executed, it is considered a
regression test. To this end, the aim of the POL Automation Team will be to continuously
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
automate SAT scripts, so the POL Test Team are eventually able to focus on only manual SAT
execution for new changes. All test cases are to be held in the Gurock TestRail POL instance.
ASAT Test Plan document will be created for each release. The SAT Test Plan will be a detailed
extension of a POL test strategy
9.3.2.1 Smoke Test
During the creation of test cases, the POL Test Team are to identify and create a standalone
smoke test suite for the core business application. A smoke test is a technique to assess if a
build is in a ‘ready’ status for further testing. A smoke test is to test only critical paths and is
to be executed as soon as and every time new release is available to the POL Test Team.
Were smoke tests are automated, the scripts are to be tagged as “@smoke”. This is to enable
the identification and execution of automated scripts, as a standalone activity as and when
required by the business. Automated smoke tests are to be flexible enough for them to be
executed on multiple environments, as required (this includes environments.in which the
business and users will perform Business and User Acceptance Testing).
‘Smoke tests are to be executed every time a new build is deployed to an environment under
test, regardless of what phase in the SDLC (Software Development Life Cycle) or test phase. A
smoke test suite is therefore deemed universally applicable to all code releases of testing.
9.3.2.2 POL Regression Testing
Additional development or code enhancements to resolve defects create a likelihood of an
issue to occur in other areas of a Solution. To manage this risk, the Post Office are to periodical
execute regression tests,
A strategic approach is to be developed in'the identification of test cases, that are deemed to
be candidates for automation. A n-1 approach is to be taken i.e. test cases applicable for
automation from'the previous releases will be automated. This strategic approach will enable
an entire automation suite to be executed frequently (and save on execution costs). In turn,
this will provide the benefit of enabling the POL Test Team to focus efforts on only new change
and perform exploratory testing.
Where automation testing is not possible and the test cases are deemed to be of high priority,
the POL test team are to execute these test cases manually, as a regression cycle. The POL
Manual and Automation Team will tactically work together to ensure duplication is not taking
place.
The POL Automation Test Team are to communicate the results of an automated regression
run to the POL Manual Test Team. All failures in the automated regression run are to be
verified manually by the POL Manual Test Team. If a failure is valid a bug is to be raised by the
POL Manual Test Team. If the failure is not valid, the POL Manual Test Team are to
communicate this to the POL Automation Test Team for investigation and maintenance.
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
9.3.2.3 Exploratory Testing
Each time a test script is executed, its probability of finding a defect over time deteriorates.
However, consistency is required to ensure that the solution has not regressed, a level of
randomness will equally prove useful when testing. The POL Test Team are therefore to
perform exploratory testing to compliment the (automated) regression testing cycle but
without the constraints of a test case.
Value per a test
script
The ‘exploratory tester’ is to use the Jira Software tool to create a task issue type, which is
labelled as ‘exploratory’. The task is to outline the area of testing and the time constraint set
against the task. The tester is to then exploré/and test the ‘fenced! areas of the system to find
defects. All findings and recommendations are to be added to Jira Software, defects are to be
linked back to the task issue type and in cases where defects have been found, a test case is
to be created.
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
9.4 Quality Assurance Phase
9.4.1 Non-functional Testing
Non-functional Testing (NFT) will start once the entry criteria for the applicable NFT type of
testing is met. The NFT test phase verifies that non-functional features have been developed
to perform as described by the NFT requirements and design specifications. Testing is to
include validation activities of the procedures, operational processes and applications.
Three types of Non-functional testing are applicable at the Post Office:
* Performance Testing
© Security Testing
© Operational Acceptance Testing
9.4.1.1 Performance Testing
Performance testing is to be performed on a dedicated NFT environment that is
representative of the production environment. This is so the dedicated NFT environment is
subject to a representative load to validate the performance of the applications. Distinct types
of loads are to be applied to verify scenarios, which will include spikes and gradual increases
in load caused by more traffic over time.
Although it may be deemed less of an importance to perform performance testing on
solutions hosted ifi the cloud, the applications themselves that are hosted in the cloud remain
candidates for performance optimisation activities.
Performance testing activities are to start early, in which the Performance Test Manager will
conduct an initial evaluation to understand the NFR (Non-Functional Requirements)
requirements or to put forward NFR requirements, as part of the demand process, to help
guide change requests further with Non-functional Requirement (NFR) considerations.
Detailed test scenarios and test data models are to be created during the design phase as
preparation to assure maximum system and application coverage is achieved by the Post
Office.
A Performance Testing Approach document is to_be created by the Performance Test
Manager, which will be a detailed extension of the POL QA Strategy.
9.4.1.2. Security Testing
Please refer to D: 07-010 - HZ Security Testing Strategy and Governance Framework
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
9.4.1.3 Operational Acceptance Testing
OAT is to validate that the applications and business operational processes, process and
operational readiness of the solution. OAT is to be executed mostly independently of other
test phases, although overlaps with penetration and performance testing are likely to occur.
OAT is to be managed by POL Services Team with support from the Head of Testing and QA.
The POL Services Team will plan and execute OAT, ensuring it has alignment and minimal
impact with test delivery activities, whilst maintaining the full, planned OAT test coverage.
Where applicable, suppliers will be responsible for supporting the POL Services Team in test
planning, environmental set-up, and test execution activities, which will include but are not
limited to defect diagnosis and change management. The POL Services Team, along with the
Head of Test and QA are to arrange and manage conversations with the suppliers.
Where OAT test activities are applicable for a release, an OAT Test Plan document is to be
created. The OAT Test Plan will be a detailed extension of the POL QA Strategy (org level)
9.4.2 POL Business Acceptance Testing
Business Acceptance Testing (BAT) will start once the entry criteria into BAT is met. The BAT
phase will validate that the agreed requirements have been met. POL Business Managers are
to performs checks on core business processes & flows, as well as ensure POL meets (where
applicable) its compliance and legislative requirements.
Business test scenarios are to be created by the POL Business Managers. Scenarios which are
to be executed to verify business processes and flows are to be behaviour driven and are to
correlate and with different end-user personas. Scenarios which are compliance or legislative,
are to be validated by executing a separate test suite, named “Compliance”. This approach
will enable the test suites to be exectited and reported independently of one another.
Although the POL Test Team will not execute BAT test scenarios, testing support will be
provided for test management tools, defect triage meetings and exit / entry criteria evidence
required by the Business Managers. All test cases are to be held in the Gurock TestRail POL
instance. All BAT bugs are to be tagged with BAT in the Jira Software tool and are to be
traceable back to the requirements.
ABAT Test Plan document will be created for each release. The BAT Test Plan will be a detailed
extension of a POL test strategy
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
9.4.3 User Acceptance Testing
User Acceptance Testing (UAT) will start once the entry criteria into UAT is met. The UAT
phase will validate that the agreed requirements fit the needs of the end-users, when the
solution is integrated with all interfaces, both internally and externally.
Identification of persona-driven test scenarios are to be created by the business managers
and end-users as a collaborative effort. The POL Test Team are to support this activity through
training of testing best standards, test tooling and bug reporting.
AIl UAT bugs are to be tagged with UAT in the Jira Software tool and are to be traceable back
to the requirements.
A UAT Test Plan document will be created for each release. The UAT Test Plan will be a
detailed extension of a POL test strategy
9.4.3.1 UAT in Model Office vs Non-Model Office
The Model Office is the first instance in which people, process and technology come together
in a staged setting. The technology at this point will have gone through several testing and
quality cycles to ensure that the end-users are testing against proven technology. Therefore,
the aim of UAT in the model office will be to ensure that processes outside of the technology
are efficiently workable with the technology, by end-users.
To this end, UAT is to be executed by end-users, with the Support of both the POL Test Team
and POL Business Managers. At the Post Office UAT will have two approaches (1) UAT (non-
model office) and (2) UAT (model office). Where, major changes are required to be tested
UAT is to be conducted in a Model Office setting to bring together the people, process and
technology.
Where minor changes are made to the baselined core business application through reference
data changes, then UAT can be conducted virtually. All test cases are to be held in the Gurock
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
9.5 Live phase
Testing activities are not applicable in the live phase; however, the)POL Test Team can
perform non-intrusive checks of the application, in live.
9.5.1 Problems and Incidents in Live
Once a release is in live, issues are no longer to be referred to as defects but instead incidents.
These incidents are to be identified, logged in the ServiceNow application and prioritised by
a business representative.
The POL Service Test Team are to investigate the issue and provide their input. Where
applicable, a test case is to be created. As part of the investigation, the POL Service Test Team
are to refer to the original requirements and put forward a case of it being a Change Request
(CR) where the requirement is fully met. This is to follow the demand process.
Scenarios where the requirement is not fully met, will require the incident to be recorded in
the Jira Software tool and prioritised in correlation with the business priority. This incident is
to be tracked to resolution, by the POL Service Test team, including a retest of the resolution,
to provide the business confidence it can be deployed to live.
Problems are considered the root cause of many incidents. In other words, multiple incidents
have the same pointer or symptom. The same process described above applies for problems.
9.5.1.1) Incident Management Process
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
10 POL Quality Gates
The following table conveys the official POL quality gates that will be implemented on all
programmes of work. Under no circumstances do the POL Test Capability accept a quality
gate not being met. IN PROGRESS
Name Criteria Owner Sponsoi
ff requirements Supplier Head of QA,
Programme
Test Manager,
IT Director
POL Supplier quality
gate
POL Testing quality gate
POL QA quality gate
Production quality gate
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
11 Entry, Exit and Suspension Criteria
11.1 Entry Criteria
The following entry criteria will be applicable for all programmes and projects across the Post
Office Limited. Programme Test Strategies are to follow the entry criteria below, as a
minimum standardised baseline. behaviours
11.1.1 Sprint Testing (commencement only)
Identifier Criteria
IT_ECO1 Task or Story in the POL Jira Software instance POL Development
havechanged status to indicate it is ready for Team
testing
IT_ECo2 Testing resource support is available and ready POL Test Capability
IT_ECo3 Test data has been uploaded and made available POL Development
to test resources Team
IT_ECO4 If available, test automation scripts are ready to be POL Test Automation
executed Team
IT_ECOS Where automation is not possible, test caseshave POL Test Team
been uploaded to the POL TestRail instance and
are in a ready to execute state
IT_ECO6 The test environment is)available to the test POL Development
resources and ready for the test team to execute Team
‘against
\T_£CO7 The correct build has been deployed to the test POL Development
environment Team
IT_ECOB Proven service virtualisation is developed for test POL Development
purposes and has been integrated to the test Team
environment
IT_ECo9 Definition of Done (DoD) has been created and POL Product Owner,
agreed by stakeholders POL Scrum Master,
POL Development
Team
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
11.1.2 System Integration Testing
Identifier _Criteri
SITECO1 Development activities are completed and thus
construction is deemed complete
SIT_ECO2 Login and access details have been created and
have been provided to test resources
SIT_ECO3 ASIT Test Plan has been reviewed and signed-off.
by stakeholders
SIT_ECO4 —_ Testing resource support is available and ready
SIT_ECOS Test data has been uploaded and made available
to test resources
SIT_ECO6 If available, test automation scripts are ready to be
executed
SIT_ECO7 —_ASIT cycle is ready to be executed
SIT_ECO8 The test environment is available to the test
resources and ready for the test team to execute
against
SITECO9 ‘The correct build has been deployed to the test
environment
SITEC10 —_Proven service virtualisation is developed for test
purposes and has been integrated to the test
environment
SIT_EC11 If applicable, supplier testing exit criteria is met
SITEC12 Testing risk assesement is complete, all risks to
test execution have been mitigated or have been
accepted
SITEC13. _SIT kick-off meeting is completed
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
Responsibility
POL Development
team
POL Development
Team, Supplier
Development Team
POL Test Team
POL Test Team
POL Development
Team, Supplier
Development Team
POL Test Automation
Team
POL Test Team
POL Development
Team, Supplier
Development Team
POL Development
Team, Supplier
Development Team
POL Development
Team, Supplier
Development Team
All suppliers
POL Test Team, POL
stakeholders, POL
Development Team,
Suppliers
POL Test Team
POL00337565
POL00337565
11.1.3 System Acceptance Testing
Identifier I Criteria
SAT_ECO1 Development activities are completed and thus
construction is deemed complete
SAT_ECO2 Login and access details have been created and
have been provided to test resources
SAT_ECO3 ASAT Test Plan has been reviewed and signed-off
by stakeholders
SAT_ECO4 _Testing resource support is available and ready
SAT_ECOS. Test data has been uploaded and made available
to test resources
SAT_ECO6 If available, test automation scripts are ready to
be executed
SAT_ECO7 ASAT cycle is ready to be executed
SAT_ECO8 The test environment is available to the test
resources and ready for the test team to execute
against
SAT_ECO9 The correct build has been deployed to the test
environment
SAT_ECLO Proven service virtualisation is developed for test
purposes and has been integrated to the test
environment
SAT_EC11 If applicable, supplier testing exit criteria is met
SAT_EC12 Testing risk assesement is complete, all risks to
test execution have been mitigated or have been
accepted
SAT_EC13. SAT kick-off meeting is completed
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
Responsibility
POL Development
team
POL Development
Team, Supplier
Development Team
POL Test Team
POL Test Team
POL > Development
Team, Supplier
Development Team
POL Test Automation
Team
POL Test Team
POL Development
Team, Supplier
Development Team
POL Development
Team, Supplier
Development Team
POL Development
Team, Supplier
Development Team
All suppliers
POL Test Team, POL
stakeholders, POL
Development Team,
Suppliers
POL Test Team
POL00337565
POL00337565
11.1.4 Performance Testing
(ze Ie
PER_ECO1
PER_ECO2
PER_ECO3
PER_ECO4
PER_ECOS:
PER_ECO6
PER_ECO7
ia
A Performance Test Plan has been reviewed and
signed-off by stakeholders
A dedicated and scaled environment is available
for NFT execution
Environment monitoring tools are integrated and
have been proven to be working as expected
Work Load Models (WLM's) have been developed
and mapped to the performance requirements
Performance testing scenarios have been
socialised and approved
NFRs (Non Functional Requirements) and
volumetric data has been agreed and signed-off
by a POL business representative
Performance testing resource is ready to monitor
test execution
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
Responsi
ity
POL Performance
Manager
POL Development
Team
POL Performance
Manager, POL
Development Team
POL Performance
Manager
POL Performance
Manager
POL stakeholders, POL
Performance Manager
POL Performance
Manager
POL00337565
POL00337565
11.1.5 Penetration Testing
or
PEN_ECO1
PEN_ECO2
PEN_ECO3
PEN_ECO4
PEN_ECOS
PEN_ECO6
PEN_ECO7
PEN_ECO8
ia
SoW (Statement of Works) agreed with
penetration testing supplier
Authorisation form has been completed and
signed-off by IT Director
If applicable, penetration testing supplier
questionaire or entry criteria has been fufilled
System owners (including cloud providers), have
been informed and permissions have been
obtained and stored
Access have been provided to pentration testing
supplier to the POL Jira Software instance so bugs
can be logged in a standalone project (access only
to relevant project(s) applicable to supplier)
Other NFT is not being conducted in parallel on
the same environment
Test data has been uploaded to a NFT
environment and is available to the penetration
testing supplier
All planned development and testing activities are
complete
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
Penentration supplier
POL Security
Archietect
POL Security
Archietect
POL Security
Archietect
POL Security
Archietect
POL Head of QA, POL
Security Archietect
POL Security
Archietect
POL Security
Archietect
POL00337565
POL00337565
11.1.6 Business Acceptance Testing
BAT_ECO1 —_—SAT exit criteria is honored
ia
BAT_ECO2 WaT environment is in a ready status
BAT_ECO3_BAT scripts are in a ready to be executed
BAT_ECO4 _Live integrations are pointing to the UAT test
environment and stubs are set up where this is
not possible
BAT_ECOS _If applicable, the business has set up any required
content in the application
BAT_ECO6 —_Alll required configurations have been) set up by
the business
BAT_ECO7 Training required for the application, or any
tooling has been completed
BAT_ECO8 Login details for the application have been
provided to the POL BAT Test Team
BAT_ECO9 —_—_BAT test execution support is ready
BAT_EC1O _ Risks have been examined and been mitigated or
removed
BAT_£C11 Login details to test tool and bug tracking tools
have been provided to the POL BAT Test Team
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
Responsi
ity
POL Test Team
POL Development
Team/Partner
POL BAT Lead
POL Development
Team/Partner
POL BAT Lead
POL BAT Lead
POL BAT Lead/POL
Test Team
POL BAT Lead
POL BAT Lead
POL BAT Lead
POL Test Team
POL00337565
POL00337565
11.1.7 User Acceptance Testing
Pte
UAT_ECO1
UAT_ECO2
UAT_ECO3
UAT_ECO4
UAT_ECOS
UAT_ECO6
UAT_ECO7
UAT_ECO8
UAT_ECO9
UAT_EC10
UAT_ECI1
UAT_EC12
ia
BAT exit criteria is honored
UAT environment is in a ready status
UAT test plan has been signed-off
UAT scripts are in a ready to be executed
Live integrations are pointing to the UAT test
environment and stubs are set up where this is
not possible
If applicable, the business has set up any required
content in the application
All required configurations have been set up by
the business
Training required for the application, or any
tooling has been completed
Login details for the application have been
provided to the POL UAT Test Team
UAT test execution support is ready
Risks have been examined and been mitigated or
removed
Login details to test tool and bug tracking tools
have been provided to the POL UAT Test Team
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL Test Team
POL Development
Team/Partner
POL UAT Lead
POL UAT Team
POL Development
Team/Partner
POL BAT Lead
POL BAT Lead
POL BAT Lead/POL
Test Team
POL UAT Lead
POL UAT Lead
POL UAT Lead
POL Test Team
POL00337565
POL00337565
POL00337565
POL00337565
11.1.8 Operational Acceptance Testing
Identifier Crit
Responsibility
OAT_ECO1 —_—To be confirmed To be confirmed
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
11.2 Exit Criteria
The following exit criteria will be applicable for all programmes and projects across the Post
Office Limited. Programme Test Strategies are to follow the exit criteria below, as a minimum
standardised baseline.
11.2.1 Sprint Testing
Sprint exit criteria is not applicable in an agile methodology. Instead, please refer to the
Definition of Done on the programme.
The remainder of this page is intentionally blank.
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
11.2.2 System Integration Testing
Identifier Criteria
SIT_EXO1 _All tests have been documented and results have
been recorded
SIT_EXO2 All open defects have a resoluton plan that has
been agreed with stakeholders
SIT_EXO3 _Defect levels are within stated thresholds
SIT_EXO4 —_ All planned test scripts have been executed at
least once
SIT_EXOS. The test phase QA completion report has been
completed signed-off by the POL Head of QA
SITEXO6 SIT test phase closure meeting has been
completed
Responsibility
POL Test Team
POL Test Team, POL
Head of Demand
Supplier, POL Test
Team
POL Test Team
POL Test Team, POL
Head of QA
POL Test Team
Blocker
Critical
Major
Minor
Trivial
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
°
25
30
35
POL00337565
POL00337565
11.2.3 System Acceptance Testing
Identifier Criteria
Responsibility
SAT_EXO1 _Alll tests have been documented and results have POL Test Team
been recorded
SAT_EXO2 All open defects have a resoluton plan that has POL Test Team, POL
been agreed with stakeholders Head of Demand
SAT_EX03._ Defect levels are within stated thresholds Supplier, POL Test
Team
SAT_EXO4 _ All planned test scripts have been executed at POL Test Team
least once
SAT_EXO5S. The test phase QA completion report has been POL Test Team, POL
completed signed-off by the POL Head of QA Head of QA
SAT_EXO6 SAT test phase closure meeting has been — POL Test Team
completed
Blocker
Critical
Major
Minor
Trivial
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
0
20
25
30
POL00337565
POL00337565
POL00337565
POL00337565
11.2.4 Performance Testing
Identifier Cr ia Responsibility
PER_EXO1 All tests have been documented and results have POL Performance Test
been recorded Manager
PER_EXO2 All open defects have a resoluton plan that has POL Performance Test
been agreed with stakeholders Manager
PER_EXO3 _Allhigh priority test scenarios have passed or have POL Performance Test
been accepted by the business as a risk with a’ Manager, Head of QA
resolution plan
PER_EXO4 —_All high and medium prioirty scenarios have been _ POL Performance Test
executed at least once Manager
PER_EXOS. Performance test QA report has been published POL Performance Test
and signed-off by the Head of QA. Manager, Head of QA
PER_EXO6 —The performance test closure meeting has been _POL Performance Test
completed and the performance QA report have Manager
been reviewed
Blocker 0
Critical 0
Major 10
Minor 15
Trivial 20
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
11.2.5 Penetration Testing
PEN_EX01 _Alll tests have been documented and results have Supplier, POL Security
Identifier
been recorded Archietect, POL Head
of QA
PEN_EXO2 All open defects have a resoluton plan that has POL Security
been agreed with stakeholders Archietect, POL Head
of QA
PEN_EX03 _Re-testing activities are completed and set toa Supplier, POL Security
[ready for testing] status have been verified by Archietect, POL Head
the penetration testing supplier of QA
PEN_EX04 —_ Defect levels are within stated thresholds ‘Supplier, POL Security
Archietect, POL Head
of QA
PEN_EX05 —_Alll planned test scriptsIhave been executed at Supplier, POL Security
least once Archietect, POL Head
of QA
PEN_EXO6 —The test phase QA completion report has been —_—_ Supplier, POL Security
completed by the penetration testing supplier Archietect, POL Head
and signed-offthe POL Security Architectand POL of QA
Head of QA
Blocker 0
Critical oO
Major 5
Minor 10
Trivial 15
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
11.2.6 Business Acceptance Testing
BAT_EXO1
BAT_EXO2
BAT_EX03
BAT_EX04
BAT_EXO5
BAT_EX06
BAT_EX07
All test scripts have been executed at least once
Test phase completion report is distributed and
signed-off
Defects are within stated thresholds
Test phase closure meeting has been completed
Any outstanding defects are agreed with
stakeholders and have a resolution plan
All test documentation and results have been
recorded and filed
All compliance or legistative related test scripts
have been executed with 100% pass rate
Responsi
ity
POL BAT Team
POL BAT Lead
POL Development
Team/Partner
POL BAT Lead
POL UAT Lead/POL
Test Team
POL BAT Lead
POL BAT Lead
Blocker
Critical
Major
Minor
Trivial
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
°
15
25
35
POL00337565
POL00337565
POL00337565
POL00337565
11.2.7 User Acceptance Testing
UAT_EXO1 —_Alll test scripts have been executed at least once POL UAT Team
Identifier Cri
UAT_EXO2 Test phase completion report is distributed and 44) Wat Lead
signed-off
+ {
UAT_EX03 Defects are within stated thresholds POL Development Formatted Table
Team/Partner
UAT_EX04 _Test phase closure meeting has been completed POL UAT Lead
UAT_EXOS Any outstanding defects are agreed with POL UAT Lead/POL
stakeholders and have a resolution plan Test Team
UAT_EXO6 —_— All test documentation and results have been
recorded and filed
POL UAT Lead
Blocker 0
Critical °
Major 20
Minor 30
Trivial 40
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
11.2.8 Operational Acceptance Testing
Identifier Criteria Responsibility
OAT_EXO1 To be confirmed To be confirmed
Blocker Oo
Critical 0
Major 10
Minor 15
Trivial 20
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
11.3 Defect Acknowledgement Times
Blocker
Critical
Major
Minor
Trivial
Defect is to be acknowledged within 2 hours of initial assignments.
Notification of review and predicted resolution time is to be
communicated within 4 hours
Defect is to be acknowledged within 2 hours. of initial assignments.
Notification of review and predicted resolution time is to be
communicated within 4 hours
Defect is to be acknowledged within 24 hours of initial assignments.
Notification of review and, predicted resolution time is to be
communicated within 48 hours
Defect is to be acknowledged within 48 hours of initial assignments.
Notification of review and. predicted resolution time is to be
communicated within 1 week
Defect is to be acknowledged within 48 hours of initial assignments.
Notification of review and predicted resolution time is to be
communicated within 1 week
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
11.4 Suspension of Testing
The following suspension criteria is applicable across the POL portfolio, programmes of work
and during the execution of the
a
Smoke tests fail for a new build
A significant amount of test data is not
available or is incorrect such that the quality
of testing is compromised
Test Readiness Checklist has not been met
Smoke tests fail for a new build
More than [359%] of the test cases that are to
be executed have been proven to be flawed
Test execution is no longer possible or
meaningful due to unresolved failures
At least 35% of planned test execution has
failed within a single testing cycle or where
the number of defects encountered at a
testing stage, would result in 35% of planned
tests to fail
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
Any test cases marked as Smoke Test fail, as
evidenced in the [Test Management Tool]
If 40% of test’ cases planned are not
executable over a [1 day} period, as evidenced
in the [Test Management Tool]
[25%] of the criteria have not been honoured,
evidenced by the [test readiness health check
document]
Any scriptithat has been tagged as a smoke
test both manual or automated fail, as
evidenced in either the manual test
management tool or automation reports
[35%] of test cases that were planned to be
executed by the POL Test Team have failed
within a single day, as evidenced in the POL
TestRail instance
Resolution of blocker or critical defects have
not occurred within the agreed target
resolution times; therefore, it is not possible
for testing to progress to a reasonable level
Evidenced within the test automation report
or within the POL TestRail instance, in which
over 35% of test scripts have failed or
evidenced by defects in the POL Jira Instance
resulting 35% of test scripts failing
POL00337565
POL00337565
POL00337565
POL00337565
Further to the suspension of testing, testing will be resumed when the above conditions have
been resolved. Where the suspension criteria are required to be exercised, the test authority
‘on a programme is to seek execute the suspension of testing. Test suspension is to be
comprehensively communicated by the test authority, to the business stakeholders (including
the Head of QA) with evidence.
11.4.1 Test Resumption Criteria
Where the test authority has decided to suspend testing, the criteria for resuming testing is
outlined, as follows:
* Areasonable caveat is in place to enable testing to continue. The risk has been mitigated
or resolved
‘Anew code release is made available for testing
Environmental issues have been addressed and resolved
Test cases have been corrected and baselined
Test data issues have been resolved and corrected
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
12 Scope of Testing
The scope of testing is to be determined at a programme level. However, per the principals
of testing, found in the POL Test Policy ‘all changes are to be tested’. This will include the any
changes that are part of the current ‘three routes to live’:
keII
TTS
Counter business Reference data / AP-ADC Data centre service
application changes changes changes
12.1 Out of Scope for POL Test Team
The following areas of testing are out of scope for the POL Test Team:
* Where new compliance is introduced, it is the responsibility of the POL business
representative to author and maintain compliance related test scripts and to execute
them in the first instance. As there is currently not a POL compliance test suite at POL, it
is to be created by the POL Business Managers. Test scripts are to be uploaded to the POL
TestRail instance and are to be executed from this instance, thereafter.
* Where new legislation by Government.is introduced, it will be the responsibility of the
POL business representative to author and execute the test cases in the first instance. This
is to assure legislation has been met and can be evidenced by the business. As there is
currently not a POL legislation test suite, it will need to be created by the POL Business
Managers. Test scripts are to be uploaded to the POL TestRail instance and are to be
executed from this instance, thereafter.
In both cases above, test cases will automatically deemed as high priority. Test cases,
wherever possible are therefore to be automated for regression testing purposes. Any test
cases that are not automatable are to be executed manually during the POL BAT phase.
Further to the above, where change has not been agreed with the Head of IT Enterprise
Demand Management via the Demand process, testing will be considered out of scope for
the POL Test Team.
Where an impact assessment (IA) is conducted by the POL Test Team and a feature is
untestable, the POL Test Team are to articulate this during the IA phase. Once agreed, these
items will be considered out of scope for testing purposes. Lastly, all standalone out of the
box (OTB) software and hardware is considered out of scope of testing, for example anti-virus
software or barcode reader.
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
13 Defect Management
This section of the document will cover the pre-production defect management process that
is to be followed within the Horizon Improvements Programme. Post-live, the process is to
follow the incident management process, which is governed by the POL Services.
For avoidance of doubt, defects are to be managed through Jira Software and Incidents are
to be managed through ServiceNow.
At the time of writing this document, an integration between Jira Software and ServiceNow is
being explored to help harmonise the information flow between the two platforms and to
make sure information on incidents is synchronised.
At POL, a defect is to be defined as a state in which the software developed during SDLC, does
not meet a requirement or as per POL business or end-user expectation which is non-
functional but can be considered a reasonable cosmetic change, with minimumimpact to the
underlying solution. A defect is therefore an error in the coding or logic of said solution, which
will cause the software to malfunction or produce incorrect/unexpected results.
13.1 Defect Classification
POL will delivery operations are to apply on the agreed definition of severity and priority,
which are described below. Defects are to be assigned a severity by the reporter of the defect.
A priority is to be assigned to a defect only during a Defect Triage Meeting (DTM) bya business
representative. Dependent on the stage of testing, the business representative can be
considered (1) a Product Owner (2) Business Manager (3) end-user (via Business Managers).
Defects are to be initially fixed based on severity ratings. Whilst assigning of a priority remains
important, the process calls for firstly focusing on technical factors that help the system reach
a level of stability, prior to priority being set on outstanding defects. This enables the delivery
to ensure that the Solutions core functionality is working prior to business validation or
‘showcasing’.
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
13.1.1 Severity Definitions
At POL, the defect severity represents the technical product impact, or in other words how
badly is the technical solution broken. To enable this to be communicated clearly for triage,
the following severity levels and definitions are applicable, at POL.
Severity Evidence
Waiting on confirmation from the Defect Triage Meeting PPT. Once
Block
ocker agreed, the definition here will be populated.
rites! Waiting on confirmation from the Defect Triage Meeting PPT. Once
z agreed, the definition here will be populated,
von Waiting on confirmation from the Defect Triage Meeting PPT. Once
i agreed, the definition here will be populated.
i Waiting on confirmation from the Defect Triage Meeting PPT. Once
agreed, the definition here will be populated.
Waiting on confirmation from the Defect Triage Meeting PPT. Once
Trivial agreed, the definition here will be populated.
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
13.1.2 Priority Definitions
At POL, the defect priority represents the impact to the business. Priority can be assigned as
early as the POL testing phase, but only by Product Owners, Business Managers, or end-users
(via Business Managers).
To enable this to be communicated clearly for triage, the following priority levels and
definitions are applicable, at POL.
Evidence
Waiting on confirmation from the Defect Triage Meeting PPT. Once
Urgent agreed, the definition here will be populated.
wah Waiting on confirmation from the Defect Triage Meeting PPT. Once
a agreed, the definition here will be populated.
edu Waiting on confirmation from the Defect Triage Meeting PPT. Once
ia agreed, the definition here will be populated.
_ Waiting on confirmation from the Defect Triage Meeting PPT. Once
agreed, the definition here will be populated.
On a given programme or project, the Development Lead is responsible for assigning the
defects to a developer for further investigation and resolution (where a defect has been
raised during BAT:/,UAT, the POL test team are responsible for validating the defect, prior to
it being assigned to the Development Lead.
The following should be noted during a quality gate (including within test phase exits):
* The POL Test Team accept zero Blocker or Critical severity defects
* The POL Test Team accept zero Urgent or High priority defects
In exceptional circumstances, where the above statements are not met, the POL Test Team
are to articulate the findings via a report. The report is to highlight mitigations for each issue,
where applicable. The report is to be sent to [role/department] to determine if the risks are
acceptable to proceed to the next stage.
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
13.2 Defect Process
The processes for Test Incident Management are not to vary across the programmes, test
phases and quality gates. The processes are to have the following cohesions:
* A defect manager is to review outstanding defects and incidents to ensure that a
resolution plan is firmly in place and that this plan is tracked and outcomes are reported
to relevant stakeholders
* All defects are required to be logged against a requirement. Else, they are to be
considered candidates for the change request processes
© Adaily defect triage meeting (DTM) is to be conducted as soon as a phase of testing starts.
The Defect Triage Team are to ensure defects are classified correctly (per the definitions
outlined in the ‘Defect Classifications’ section). Priority is to be set only by the business or
end-users at the appropriate time. This is outlined in the Defect Triage Meeting document.
© The detail of a single issue is to be recorded and tracked in a centralised repository, so
there isa single source of truth and to enable quicker triaging and resolution. Defects are
to have traceability to a design requirement, a build or other test artefacts that are
relevant to the defect and resolution.
© Defects are to be logged by the person that has discovered the defect in the POL Jira
Software instance, following the standardised approach to defect creation. Please refer
to the POL Defect Standardisation approach.
13.2.1 Defect Triage Meeting (DTM)
In line with the POL Test Capability objectives, the POL Test Managers are to ensure that
defects are reviewed, resolved and retested swiftly. In recognition of this, the POL Test
Managers are to review the Severity and priority of defects, with the Defect Triage Team
{0TT).
‘A Defect Triage Meeting (DTM) is to be setup by the POL Test Managers and is to invite
business, project management, development and test representation. In this meeting
participants are to go, through bugs in severity order (pre-POL BAT and UAT) and then in
priority order (during POL BAT and UAT).
The following format therefore applies:
* Review all newly raised defects in the past 24 hours
Confirm the severity and priority ratings are correct (dependent on phase)
Review all Blocker and Critical defects and confirm an estimated fix time
The meeting is to take place daily for 20 minutes and is deemed standard practice for all POL
programmes of work, regardless of delivery methodology. The aim of this meeting is to
provide 24-hour windows in which defects are reviewed and updated on consistently.
POL staff are encouraged to read D: 09-028 - Defect Triage Meetings, to understand how the
DTM process and resolution sequences are to be implemented as standard practice on
programmes.
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
13.3 Defect Statuses
seas mney
The backlog is the accumulation of all issue types in the POL Jira Software
Backlog Instance and form part of the Product Backlog Items (PBI’s) — which include
bugs.
Selected for
i, The bug has been selected to be resolved as part of the next build.
resolution
In Dev The bug has been picked up by a POL developer andiis being investigated
3" Party Fix POL have established that the bug requires supplier to fix the issue
Fixed The bug is deemed fix by the POL developer or by a supplier
The bug fix has been deployed to the SV&l environment and is ready for
On sv&l retesting
Sva verified T@ Bug has been confirmed as fixed by the POL Test Team or supplier
team, The fix is ready to be deployed to the LST environment
yA The bug has been confirmed as fixed by the POL Test Team or supplier
team. The fix is ready to be deployed to the model office environment
Ready for POL
Testing The bug has been deployed to the model office and is ready for retesting
Closed The bug has been deemed closed and no further action is required
When future state environments are available, this flow will be simplified further. A flow has
been created as a future state and is ready for this purpose.
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
13.4 Defect Workflow
To be confirmed as part of the POL Jira Software bug issue type standardisation. Temporary
flow added in the interim. Page will be landscape in the next version
State flows for Bugs
=
aecrantors =
Te accion of lisse pes, cea the Product
Scio (8 nace pes FB ee
2s Predict Balog ms (PES)
Toe quay deb hasbeen seco deveprent A {
spe vauofrpeced everson fas beer adoedw the
‘The ase i ith a td pany provider at
request ror fhe. To stats eres
(Sto wack tos rqureaby sal pay
“Te sues consdeedo be resched y
Sicko 2 he St tA
Teles has ben as The }
Te big hs ben dope Sh
} Tetg hs ben vere on SV by POL Ting
— a
: 09-019: POL Test Strategy - Confidential
‘Copyright 2022 Post Office Limited
POL00337565
POL00337565
POL00337565
POL00337565
14 Test Data
14.1 Pre-test execution
Pre-POL test execution on an environment, the requirement to identify appropriate test data
for test execution will be deemed a compulsory activity by the team performing the test
execution. Data is to be made available to the test execution team, at least 7 days prior to the
planned test execution activities starting, per the test plan and test entry criteria.
14.2 Obtaining Test Data
The types of data required to support the testing of each deliverable at both a programme
level and organisational level and how it is to be generated, is required to be agreed at an
organisational level for the core regression testing suite and at a programme level, per a test
phase fora given release.
The following stages are to be considered when discussing the generation of test data:
‘Establish the scope of the inputs and outputs that
Establish the scope are required to reach a level of confidence that test,
scripts can be executed smoothly (including any gaps)
*Perform an analysis of the requirements as part of a
Confirmation of data inputs ‘static testing’ approach and confirm the data inputs
required to enable test execution
Check if usable data is already present and how it
can be sourced to populate a test environment for:
testing activities (including the volume of test data
required)
Analysis of available data
“Where no data sources are available to the POL Test
Team, discuss this with the design/development
teams to identify alternative data sources that could
be used
Identification of alternative data
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
15 Test Organisation
15.1 Horizon Quality and Test Management
This sub-section is to be put in landscape view in the next version and synchronised with the
look and feel of the rest of the document
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
16 Test Roles and Responsibi
16.1 POL Test Leadership
Role Reasonability
POL Head of QA and This role will be responsible for:
Test Management sae
POL TCoE Test This role will be responsible for:
Manager 3 tHe
POL Automation Test This role will be responsible for:
Manager * THe
POL Performance Test This role will be responsible for:
Manager . Tec
POL Senior Test This role will be responsible for:
Manager y
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
16.2 POL Test Management
Role
POL Head of QA and
Test Management
Horizon Senior Test
Manager
Test Architect
Horizon Test Manager
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
Reasonal
y
This role will be responsible for:
«TBC
This role will be responsible for:
« TBC
This role will be responsible for:
° TBC
This role will be responsible for:
° TBC
POL00337565
POL00337565
16.3 POL Test Team
Role
POL Test Lead
POL Senior Test
Analyst
POL Test Analyst
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
Reasonal
y
This role will be responsible for:
Day-to-day responsibility for test delivery
Maintainer of test schedule
Maintainer and executor of test plan
Co-Ordinator of test case/script creation and test
execution
Responsible for providing a progress {and test report
Plan and manage test data
Co-Ordinator of communication between cross-functional
teams
Peer reviewer of test assets
Participator of Defect Triage Meetings
This role will be responsible for:
Following and understanding the POL Test Delivery
Strategy
Reviewing functional and technical requirements
Development of Test scripts
Execution of Test Scripts
Raise, retest and close defects
Perform regression testing following defect resolution
Informing the POL Test Lead of any issues that may affect
the schedule, budget, or quality of the product or the
testing process
This role will be responsible for:
Following and understanding the POL Test Delivery
Strategy
Reviewing functional and technical requirements
Development of Test scripts
Execution of Test Scripts
Raise, retest and close defects
Perform regression testing following defect resolution
POL00337565
POL00337565
POL Junior Tester
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
Escalators of issues to POL Test Lead
This role will be responsible for:
Following and understanding the POL Test Delivery
Strategy
Development of Test scripts
Execution of Test Scripts
Raise, retest and close defects
Perform regression testing following defect resolution
This role will require a quick progression based on a
competency framework to POL Test Analyst in 6 months or
less, Therefore, these positions are considered the same
grade.
POL00337565
POL00337565
17 Environments
17.1 Current environments
Current environment available for testing
SV&l
LsT
ROT
Des
Respons!
The Solution Validation and Integration
environment has been specifically tailored /
designed to support functional testing to
validate customer business requirements for
project related change. It is representative of
LIVE in that it has many of the capabilities
that operate in the LIVE environment: “Real”
Counter and peripheral hardware supplied
and managed by Computacenter, with
Verizon managed Branch networking
integrated with a fully managed Data Centre
and real interfaces to third parties — albeit
with test end points. It is used by many user
groups and is seen as the “busiest” of test
environments servicing multiple requests on
a regular basis.
The Live System Test environment isin place
to support the LIVE environment, focussing
on the patching and maintenance of the
individual components that make up the
Horizon solution. For Project change, LST is
used primarily to rehearse the
deployment/migration activities with
the same tooling/teams and process that
would be used for LIVE. However, the POL
Test Team believe this is not only limited to
the said activities but also to some further
changes and testing from a partner
perspective.
RDT is not a full-blown replication of the
production Horizon environment. It only has
elements required to support operation of
branch counters, and the delivery of
reference data to said counters. This means it
has several counters (spread across Fujitsu
Bracknell, Future Walk, Finsbury Dials, and
some home locations). Within the Horizon
data centre, it has a BAL (Branch Access
Layer) server, connecting to the BRDB
(Branch Database), connections to RDMC
(Reference Data Management Centre) and
Fujitsu
Fujitsu
POL/Fujitsu
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
RDDS (Reference Data Distribution Service).
It does not have connections or processes to
support many Back Office processes
(i.e. connections to Credence for MI data or
CFS for finance reconciliation and
settlement). It also does not have any
connectivity to external systems, so it is not
possible to test transactions that interface
with third party systems. Some internal stubs
and emulators are provided by Fujitsu, but
the scope and effectiveness of these is
diminishing and will be completely redundant.
once the Belfast Exit Program completes, This
negates RDT as a meaningful option to
replicate E2E business transaction flows.
17.2 Proposed environments
Proposed environments as a future state view
Environment
Manual
Automation
The purpose of this environment is to
provide the manual test team/a dedicated
environment from which to test the latest
build. An array of manual test activities can
take place iin this environment. Overall, this
environment will serve as an environment
from which both manual SAT and SIT test
phases can be executed from. This
environment is virtual and should
be accessable via a PC/laptop.
The purpose of this environment is to
provide the automation test team a
dedicated environment from which to run
the automation test suite with the latest
build. Where possible, both the manual
and automation test environments should
be coordinated during an official test
phase. The purpose of having a dedicated
environment is to ensure that automation
tests run based on consistency and
uninterrupted. This environment is virtual.
The purpose of the QA environment is to
enable Product Owner Checks, Business
Acceptance and User Accetance Testing
activities. This environment will be used by
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
Responsibility
POL Test Team
POL Automation Team
POL Business Team
POL00337565
POL00337565
NFT
Model Office
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
business representative, business
managers and users to validate that the
product meets the business needs.
This environment will be virtual with
release candidates deployed.
The purpose of the NFT environment is to
provide a dedicated environment for
performance and penetration testing
activities. This environment will be virtual
with release candidates deployed.
The model office will knit together the
people, process and technology. The
software deployed to the staged Post
Office counters will have beenquality
checked and signed-off software. This will
enable the checking of ‘real-world’
scenarios related to different real-world
personas. Where the releases versions are
coordinated with production, the.
environment ¢an be used to check live
incidents.
POL Test Team
POL Business Team
POL00337565
POL00337565
POL00337565
POL00337565
18 Test Deliverables
18.1 Supplier Test Deliverables
Test Strategy or A document that provides the test All suppliers
Approach approach that the supplier is to undertake
in delivering a tested solution to the client
Test Reports A summary of test activities completed (or All suppliers
not completed) and the results achieved
during each phase of testing including test
cycles executed and associated bugs
QA Sign-off A detailed report of the overarching testing _ All suppliers
Reports completed with insights to! ALL open bugs
in the system. This is to be used to form an
initial acceptance to POL test activities
Release Notes Documents that are tobe distributed with All suppliers
the software release to» POL. This
document also. builds upon’ » initial
acceptance by the POL Test Team
18.2 Deliverables to POL Testing
Asset Responsi
Functional requirements POL Design Team
Non-functional requirements POL Design Team
Epics, Features, Stories, Tasks POL Design Team
Technical architecture
Project, Programme or Release plans
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL Chief Architect
POL Project, Programme or
Manager
Release
18.3 Deliverables from POL Testing.
POLQAStrategy The organisational test strategy (this
document)
Test Approach A single testing approach across the POL
Landscape for the following areas:
Performance test approach
Automation test approach
Security test approach
Operational acceptance > testing
approach
Programme Test Within the framework of the QA Strategy,
Strategy a Programme Test Strategy is to becreated
when a programme is created
Defect ‘A document providing a the detail of how
Standardisation defects are to be written by»anyone
creating a bug in the POL Jira Software
instance
Core Regression A living test suite that is to be maintained
Pack by the POL Test.Team. Used for Horizon
related testing and considered a strong
automation candidate
Weekly status Aweekly status report of test activities at a
reports portfolio level, including any areas of
concern
Other deliverables are applicable; however, these are considered programme level
deliverables, which will include and are to be added to the Programme Test Strategy:
© Requirements risk analysis,
© QAsign-off documents,
© High-level estimates,
* Daily test reports and;
Inputs to portfolio reporting
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
PDF
POF
POF
PDF
TestRail
PPT
POL00337565
POL00337565
POL00337565
POL00337565
19 POL Test Reporting
This section of the POL QA Strategy will outline the reports that are to be generated by the
POL Test Team at all stages of the System Development Lifecycle and Release Lifecycle.
Reporting is an important aspect of testing, as it will enable the POL Test Team to
communicate testing outcomes to relevant stakeholders and therefore afford constructive
discussions and metric-driven decision making. All reporting is considered a snapshot and any
changes made to the solution are therefore subject to further testing.
At a programme level, reports will be added to the relevant folder structure within the
programme itself. At an organisational level, particularly when the core regression suite is
executed reporting is to be stored in a dedicated location on SharePoint.
19.1 Portfolio Reports
Weekly portfolio Aweekly status report of test activities at a PPT
test reports portfolio level aross all programmes and
demand, including-any areas of concern
with portfolio level RAID items
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
19.2 Programme Reports
The following reports are required across all programmes or projects
Smoke test
report
Entry criteria
report
Exit critera
report
Daily test
progress report
QA sign-off
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
A ‘quick fire’ report that provides the
development team insights in a
deployment to a given environment. The
report will be used to provide evidence
that the test team are in a ‘ready’ position
to proceed with further tests
Initially presented as a health check at least
‘one month prior to planed test activities,
this report will communicate if an entry
criteria has passed or failed to be met. This
is produced per a test phase and is based
on the relevant entry criteria for a given
test phase
Presented as a health check everytime a
daily test report is published. The report is
to have a RAG status with a passed or failed
status of if the»exit has been met. The
report is based on the relevant exit criteria
fora given test phase
A detailed daily report of the test activities
progress for each test cycle (which is
related to a test phase), along with the
scripts planned, how many have been
executed, metrics, RAID items and
downtime metrics
A overall detailed report of the overarching
testing completed with insights to ALL
open bugs in the system. This is to be used
to form acceptance to the service team.
Everytime a new
deployement occurs
for an environement
under test. Where
smoke tests. are
automated, this report
is to be automatically
generated.
One month prior to
planned test activities
starting. The first three
weeks this entry
criteria will be
managed weekly the
last week as a run up
to test activities start,
changed to daily
Embedded withing the
daily test progress
report
Daily
Part of the UAT exit
criteria
POL00337565
POL00337565
20 Assumpti
ions and Dependencies
20.1 Assumptions
Identifier
AS_001
AS_002
AS_003
AS_004
AS_005
AS_006
AS_007
AS_008
AS_009
AS_010
Assumption
Suppliers will have resources available to execute internal system testing
Suppliers will have resources available that will enable the communication
of bug resolution times and bug fixing
Commercial contracts with suppliers will have a standardised set of test
clauses in the contract that will enable quality gates, test resolution times
and entry criteria into POL testing to be honored, to be contractual
Suppliers, the business and end-users will use the POL Jira software
instance
The POL Jira software instance will be security protected, so suppliers or
end-users do not view data that isinot relevantto them. In each case, NDAs
are to be signed to enable the access of the POL Jira Software instance
The POL Horizon Quality and Test Management organisation does not own
security testing, but are to/assist with the facilitation and co-ordination of
such activity
Business and end-user representatives willbe made available to perform
Business Acceptance Testing and User Acceptance Testing
Where it is wholly applicable, suppliers will develop or assist in the
development of stubs, emulators, or simulators for testing purposes
Enough time will be provided for core regression testing to take place
Programme Test Managers will have a level of autonomy within the
framework of the QA Strategy to form their own Programme Test Strategy
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
POL00337565
POL00337565
20.2 Dependencie:
Identifier
DEP_001
DEP_002
DEP_003
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
Jependency
Target resolution times for defects are to
be agreed and followed
Signed-off requirements are to be provided
to the POL Test Team in a timely manner
Security strategy document is to be made
available for review by the Head of QA
Responsi
ity
Suppliers, POL
Development Team
Suppliers, POL Design
Team
Security Architect
POL00337565
POL00337565
21 Appendix
21.1 Reference Data Test Readiness Checklist
The following criteria is to be met for reference data changes, prior to any test activities
taking place.
REF_0O1 POL Tester has been tagged to a project time
code in Snow
REF_002 Requirements have been signed off by a POL
Business Manager
REF_003 Requirements have been signed off by'the POL
Design Representative
REF_004 Solution design is to be signed off by the Chief
Architect
REF_OOS A detailed project’ plany which contains an
implementation plan is made available to the
POL Test Team
REF_006 Defect ‘Target resolution times’ are to be
agreed with the POL Reference Data Team
REF_OO7 Defect, ‘Target resolution times’ are to be
agreed with the 3 party suppliers that are
required to fix bugs
REF_008 ‘A formal release is to be provided to the POL
Test Team, no new development scope is
permitted thereafter (except for defect fixes)
REF_009 Test data is uploaded and available to the POL
Test Team at least 24 hours prior to the test
execution start date
REF_010 Where applicable, Release Notes are to be
provided to the POL Test Team
REF_011 Where applicable, Unit Testing has passed with
an outcome of 100% pass rate
: 09-019: POL Test Strategy - Confidential
Copyright 2022 Post Office Limited
Responsibility
POL Project Manager
POL Business Manager
POL Design Team
POL Chief Architect
POL Project Manager
POL Reference
Team
Data
POL Project Manager
POL Reference
Team
POL Reference
Team
POL Reference
Team
POL Reference
Team
Data
Data
Data
Data
POL00337565
POL00337565