POL00028297
POL00028297
a8 NOU’ '95 83:53 FROM BA-POCL TERMINAL HSE PAGE .8027089
PWKP4-3 RESTRICTED CONTRACTS
Bringing Technology to Post Offices and Benefit Payments
FROM DEMONSTRATOR TO SELECTION
Author: MR Purchase Version: Issue 1.0
Authority: I D Selwood 20 October 1995
Reference: PWKP4-3
Contents Page
2. THE EVALUATION METHOD ........
2.1. Financial vs, Non-Financial Impact
2.2. Assessment Results
2.3. PFI Perspectives ......
3. THE EVALUATION PROCESS
3.1, Overview
3.2. Stage 3 Stream:
3.3. Assessment Review:
3.4, The ITT Issue Decisio
DIA A WHE
1. PURPOSE
1A. This paper provides an overview of the evaluation process from Stage 3
“Demonstrator and Negotiation” to the final selection decision in Stage 4
“Evaluation and Selection”. The aim is to provide a common understanding of the
evaluation framework within which further work will be undertaken.
£2. The paper builds on the concept of using financial and non-financial information to
aid the selection of the winning service provider and on the value factors for the non-
financial assessment put forward in the “Evaluation Post Shortlist” paper (PWKP3-
18). The ideas outlined in that paper for the evaluation process and for measuring the
value factors are developed by this paper.
13. Section 2 considers the outputs needed from the evaluation to support the selection
decision, while section 3 looks at the activities to produce them.
2. THE EVALUATION METHOD
21. Financial vs. Non-Financial Impact
2.1.1. The basis for financial business decisions is to consider the cost, benefits and risks of
each option (i.e. risks of the costs not being contained and of the benefits not being
20 October 1995 Page 1 of 8 Issue 1.0
a8 Nou '35 @9:53 FROM BA-POCL TERMINAL HSE
PWKP4-3
TO
RESTRICTED CONTRACTS
GRO
POL00028297
POL00028297
PAGE .8837289
delivered). The tendency in Government procurements has been to express these
three factors solely in monetary terms. An overall cost of ownership can then be
calculated for each option and the lowest selected as the winner. It has becn decided
that this programme is to take a wider view by also considering the impact of non-
financial aspects. This is shown in context in Figure 1, where the “Financial” impact
column shows the main items contributing to the total cost of ownership and the
“Non-Financial” column shows the areas of non-financial impact.
Impact
Financial Non-Financial
Cost Charges x
Internal costs
Quantifiable Qualitative
Benefit benefits benefits
Risk Attributable costs Issues
Total Cost
Figure I - Financial and Non-Financial Impact
Key:
. Charges refer to the service provider’s charges;
. Internal costs refer to the contracting authorities own additional costs (e.g. in
providing accommodation for the service provider, in making staff available to
attend training);
+ Attributable costs are what may have to be spent to off-set the risk of poor
service provider performance;
. Issues covers the various risks and issues to be managed in addition to those
quantified as Attributable costs.
2.2. Assessment Results
2.2.1. Unlike the items for financial impact, it is not possible to add the non-financial
benefit and risk items to give a single value. The benefit and risk headings are
equivalent to the axes uscd in evaluating responses to the SSR, i.c.:
(a) Characteristics — how well the characteristics of the proposed services assist
BA and POCL in meeting their business objectives and the users in performing
their operational roles. Benefits, or the lack of them, for a particular option are
identified in relation to the baseline requirement.
20 October 1995
Page 2 of 8
Issue 1.0
POL00028297
POL00028297
GRO PAGE .0847009
@8 NOV '95 18:88 FROM BA-POCL TERMINAL HSE TO
PWKP4-3 RESTRICTED CONTRACTS
(6) Viability — issues or concerns about the viability of the proposed services,
taking account of how the underlying designs, methods, systems, procedures,
tools, plans and resources enable the services to be delivered and function.
2.2.2. Thus the non-financial “value” view will be shown on a two-dimensional grid in a
similar manner to the proposal evaluation (see Figure 2). The grid showing the
overall view of the value assessment will be derived from the assessment of service
providers’ proposed solutions and capability against each value factor, combined
using agreed weights.
Good match to
business objectivesI
and requirements x x
Characteristics “Reservations about
Genefits) Viabilility Viable contender
Poor match to
business objectivesI Unacceptable
and requirements
Reservations about
Solution
Poor: ————————————_» _ Good:
High Risk Viability (Risks) Low Risk
Figure 2 - Value Assessment Grid
2.2.3. The results from the financial evaluation will be reviewed against the Value
Assessment Grid to support the decision process. This will draw out potential points
such as the lowest cost solution falling into the “Reservations about viability” box
from the grid in the ‘Evaluation PostShortlist’ paper (i.e. “cheap but ...”), or possibly
the solution with the best characteristics and viability in the “Viable contender” box
also being the most expensive (i.e. is it “Affordable?”).
2.2.4. (A future paper will identify how best to undertake the financial evaluation and
present the results. It will address the method for taking account of the uncertainty in
predicting business workloads and the speed of implementation.
23. PFI Perspectives
2.3.1. In addition to the financial and non-financial views, there are two specific
perspectives that need to be considered due to the PFI / Programme objectives. These
are:
(a) Partnership — the capability and business compatibility of the service providers
to make a good business partner (particularly for POCL) in ensuring the
success of the initial services and helping to generate new business;
20 October 1995 Page 3 of 8 Issue 1.0
POL00028297
POL00028297
@8 NOV ’95 18:88 FROM BA-POCL TERMINAL HSE TO
PAGE .@85/7809
PWKP4-3 RESTRICTED CONTRACTS
(b) Risk Transfer — the extent to which risk is reduced and transferred to the
Service Provider in accordance with the aim to minimise risk and place it
where it can most effectively be managed. A major element of this will be the
fraud risks associated with the benefit method of payment, but it also covers
the other service delivery and business workload risks.
2.3.2. These commercial perspectives plus the value assessments and financial evaluation
will form the information on which to base the selection decision.
3 THE EVALUATION PROCESS
3.1. Overview
3.1.1. The main activities and milestones for the Stage 4 evaluation process are shown in
relation to the main Stage 3 activities in Figure 3 below, with the arrows indicating
dependencies.
Stage 3 streams:
¢ Requirements definition
Ie Demonstrator events
le Contract production
supported by:
* Service Provider Risk
Register (SPRR)
od ———+@ Assessment Reviews
4 pe iis 4 ITT issue decision
@ Issue ITT
Commercial:
je Partnership) =f .- eof ev ee eee Agree Value Assessments
je Risk Transfer
@ Receive Best Tenders
F Financial Evaluation
(including Risk Transfer)
Financial Preparation
a
Post Tender Negotiation
Evaluation & Selection > Selection Decision
}Decision Endorsement
Business Case analysis
c3
Business Case Approval
Figure 3 - Activity & Milestone Dependencies
20 October 1995 Page 4 of 8 Issue 1.0
POL00028297
POL00028297
PAGE .@86/089
@8 NOV '95 18:81 FROM BA-POCL TERMINAL HSE
PWKP4-3 RESTRICTED CONTRACTS
3.1.2. The general process is described as follows, with cross-references to the headings in
Figure 3 shown in bold:
(a) outputs from the Stage 3 streams of work (discussed later at 3.2) feed into
regular Assessment Reviews (discussed further at 3.3);
(b) the Commercial assessments, Partnership and Risk Transfer, feed into the
Assessment Reviews and where necessary into the Service Provider Risk
Register (SPRR);
(©) _ the Assessment Reviews assess the service providers against the value factors,
agree appropriate feedback to service providers where there are specific issues,
and provide direction to the Stage 3 work and the commercial assessments;
(4d) Financial Preparation work provides the groundwork for the financial
evaluation and business case submission after receipt of best tenders. The
format and content of information required in the tenders is needed prior to
Issue of ITT;
(e) the SPRR analysis quantifies the outstanding risks on the SPRR at the end of
Stage 3. These, and the outcome of the contract negotiations, then provide the
basis for the ITT issue decision;
(f) the Value Assessments (based on the Assessment Reviews) need to be agreed
by the Evaluation Board and lodged with the lawyers prior to receipt of Best
Tenders;
(g) following receipt of Best Tenders, work commences on the Financial
Evaluation and Business Case Analysis;
(h) during the Financial Evaluation, the commercial assessment of Risk Transfer
will be completed;
(i) _ initial results from the Financial Evaluation will provide input for any Post
Tender Negotiation;
Gj) the outcomes from the Value Assessment and Financial Evaluation will be
analysed to aid the selection of the winner (Evaluation & Selection, leading to
the Selection Decision);
(k) following the Selection Decision, the sponsor’s organisations will need to
Endorse the Decision and the Business Cases will need to be put through the
final approval processes.
3.1.3. Other work will be progressing to initiate Stage 5 “Operational Trial” in addition to
the Stage 4 activities covered in the above diagram and description. This is outside
the scope of Stage 4.
20 October 1995 Page 5 of 8 Issue 1.0
POL00028297
POL00028297
PAGE.@077883
as NOU "95 18:81 FROM BA-POCL TERMINAL HSE
PWKP4-3 RESTRICTED CONTRACTS
3.2. Stage 3 Streams
3.2.1. The evaluation model for shortlisting proposals contained a detailed hierarchy of
evaluation criteria. At first sight, the use of the Value Factors may appear to be
superficial by comparison. However it is important to understand that the Stage 3
work provides for a far more detailed and thorough evaluation of proposed solutions
and service providers than was possible during shortlisting. The three main streams
of work for Stage 3 are:
(a) Requirements definition
The requirements from the SSR are refined and evolved through feedback from
service providers (to confirm that the requirements are clear and reasonable)
and from input from the sponsors’ organisations. Service providers’ responses
to the requirements are assessed and issues or concerns raised via the SPRR.
(b) Demonstrator Events
The aim of the Demonstrator is to reduce risk to the Programme and its
sponsors by improving the understanding of and assessing the service
providers’ proposed solutions. A variety of events, including demonstrations,
meetings and working papers, will be used to examine solutions from the
perspectives of:
i. Fraud-free Method of Payment
ii. Infrastructure for POCL Automation
iii. Applications for POCL Automation
iv. Operational Trial and Implementation
v. Service Management
vi. End to End Service Delivery
Demonstrator events will involve sponsor representation (either directly or
indirectly). Concerns and issues arising from events will be recorded on the
SPRR, and the service providers will be given the opportunity to address these
risks. The results from assessments will also be recorded to provide input to
Assessment Reviews.
(c) Contract Production
The contract production stream is responsible for the production, negotiation
and agreement of the contractual documentation, ensuring that it properly
Teflects:
i. the sponsors’ commercial needs for the overall business agreement;
ii. the requirements identified under (a) and key aspects of service provider
solutions.
20 October 1995 Page 6 of 8 Issue 1.0
POL00028297
POL00028297
@8 NOV '95 18:82 FROM BA-POCL TERMINAL HSE TO GRO PAGE .@087009
PWKP4-3 RESTRICTED CONTRACTS
3.2.2. It is expected that the service providers will address many of the risks raised on the
SPRR during Stage 3 by clarifying or improving their proposals. As a consequence
the Stage 4 evaluation approach is to look for the differentiators, on the basis that the
detailed requirements and solutions have been examined with the service providers
during Stage 3. The combination of Stage 3 and 4 work provides for a far more
detailed evaluation than the paper-evaluation undertaken to shortlist proposals.
33. Assessment Reviews
3.3.1. Assessment Review meetings will be held to assess the service providers’
performance against the value factors. These will take into account the views shown
in Figure 4.
Requirements ™ Demonstrator
Demonstrator Zz Views
Contract ™ Negotiation Assessment
Negotiation we View ae Reviews
° score factors
le feedback issues to
Sponsors — Business a service providers
a View le focus remaining
Stage 3 work
SPRR ba ad
Figure 4 - Assessment Reviews
3.3.2. Where issues are identified with a service provider’s performance, the Assessment
Review will agree actions to resolve them as appropriate. Normally this will lead to a
tisk being raised against the SPRR, and/or the issue being addressed directly with the
service provider.
3.3.3. If an Assessment Review has insufficient information on which to assess a value
factor for a particular service provider, then it may commission work designed to
provide suitable details.
3.3.4. It is intended that the Assessment Reviews will be undertaken by the BA and POCL
representatives on the Programme Management Team and include representation
from the six Demonstrator perspectives listed under 3.2.1(b) above. Sponsors will be
able to present their views via their respective Programme Requirements Authority
who will attend the reviews. Appropriate mechanisms for any feedback will need to
be agreed.
3.3.5. While the final value assessments need not be agreed by the Evaluation Board until
after issue of ITT but prior to receipt of best tenders, an Assessment Review must be
held before the SPRR is closed to new risks (expected one to two weeks before the
20 October 1995 Page 7 of 8 Issue 1.0
POL00028297
POL00028297
@8- NOU "95 18:82 FROM BA-POCL TERMINAL HSE PAGE .@89/7089
PWKP4-3 RESTRICTED CONTRACTS
end of the Demonstrator). This is to ensure that any significant issues are identified
on the SPRR prior to the ITT issue decision.
3.4. The ITT Issue Decision
3.4.1. Any service provider invited to submit a best tender must have cleared a number of
“hurdles”. The Stage 3 work and the commercial assessments will assess whether
each service provider meets or exceeds:
(a) the minimum service requirements acceptable to the sponsors;
(b) the minimum requirements for partnership with POCL and the development of
new business opportunities (as described in the Prospectus and the SSR);
(c) sufficient transfer of risk for an acceptable risk profile, in particular for the
fraud risks but also the general service development and business volume risks;
(d) an acceptable funding method and financial structure.
3.4.2. The mechanisms by which failure to clear the hurdles will be detected are the SPRR
and agreement to final draft contracts. A service provider will not be invited to tender
(a) he has any “Critical” (category A) risks or an unacceptable profile of other
tisks outstanding at the time the SPRR is finalised; and/or
(b) it has proved impossible to agree with him a contract acceptable to BA and
POCL.
3.4.3. The final SPRR from Stage 3 will analysed to identify the impact (both financial and
non-financial) of the outstanding risks. This analysis and the outcome of the contract
negotiations will be put to the Evaluation Board to decide if any service provider
should not be invited to tender.
20 October 1995 Page 8 of 8 Issue 1.0