PWKP4-25 RestricreD CONTRACTS
POL00031154
POL00031154
Bringing Technology to Post Offices and Benefit Payments
SUPPLIER SCORES IN RESPECT OF VALUE FACTORS
Author: Derek Selwood
Authority: Derek Selwood
Reference: PWKP4-25
Contents
2. BACKGROUND
3. THE PILOT REVIEWS .......
4, CONDUCT OF THE REVIEWS ...
3. RESULTS OF THE PINAL PROGRAMME REVIEW
fi
6, USE OF THE RESULTS IN EVALUATION & SELECTION
7, RECOMMENDATION
ANNEX A- VALUE FACTORS
ANNEX B - MEMBERSHIP OF REVIEW PANBLS..WW...
ANNEX C - TOPICS
LL PURPOSE
/S, VALUE FACTORS MATRIX (reproduced from PWKP4-12
Version: Issue 1.0
11 March 1996
Ld. This paper sets out for Evaluation Board consideration the suppliers’ Value Factor
scores arising from the Programme Review held on 7 March 1996. The Board is
invited to endorse the scores, which will be lodged with the Programme lawyers prior
to receipt of tenders and used in the final evaluation and selection process.
2 BACKGROUND
2.1. The concept of evaluating on a combination of monetary and non-monetary (value)
factors is described in PWKP3-18 ‘Evaluation Post Shortlist’.
The factors were
notified to suppliers by letter from the Programme on 6 November (995, and are
reproduced for convenience at Annex A.
Tl March 1996 Page 1 of 12
Issue 1.0
POL00031154
POL00031154
PWKP4-25 RESTRICTED CONTRACTS.
2.2, Paper PWKP4-5 “Use of Value Factors in final evaluation & selection’ described
how suppliers would be assessed against the value factors, and PWKP4-12 ‘Value
Assessment Model’ set out how the scores would be built up from an individual level
to a Programme view.
2.3 BA, SSA and POCL each nominated a member of staff to perform a quality
assurance check on the results of the Programme reviews.
24, The names of those participating at the various levels are shown at Annex B.
3. THE PILOT REVIEWS
3.4, It was thought prudent to hold a ‘pilot’ review at cach level to test the effectiveness
of the process. The subsequent ‘live’ reviews took account of feedback from the
pilots in general, but also of the specific points noted below.
3.2. At the pilot Demo Stream review (5 December 1995) it was realised that fraud-free
services for POCL were not covered by the Value Factors, factor 3 covering only the
benefit payment service. By that time the factors had been notified to suppliers. It
was decided that this area should nevertheless be included in the Demo Stream
reviews as ‘Factor 3A: Fraud-Free services for POCL’, and views and scores in
respect of supplier performance were subsequently recorded. The handling of this is
dealt with at para 5.5. The Contracts Stream had no evidence on which to base any
marking for this factor.
3.3, The shortage of evidence available to the Contracts Stream with respect to some of
the factors led its pilot review to conclude that it should apply a validity marking to
its scores, to reflect the amount of evidence on which the scores were based. The
validity markings and meanings adopted were:
. A Assessors totally in the picture regarding supplier's capability, full
confidence in seore
* B Very well qualified to comment, with a few gaps
° Cc Reasonable amount of evidence on which to assess
. dD Some evidence, adequate for an indicative assessment only
. E No evidence at all - no score can be given
This system was adopted for all subsequent Contracts, Demo and Programme
reviews.
we
-
At the pilot Contracts Stream review it was recognised that provision should be made
for taking account of the assessment of suppliers’ Partership capability, on which
David Miller, the head of CNT, had taken the senior role within the Programme on a
personal basis, It was thought inappropriate to consider {it at Contracts Stream level,
given that the other two members of that stream had had no involvement in the
L1 March 1996 Page 2 of 12 issue 1.0
POL00031154
POL00031154
PWKP4-25 RESTRICTED CONTRACTS
partnership discussions. It was therefore decided to consider it at the Programme
reviews.
4, CONDUCT OF THE REVIEWS
4). Each review took the output from the previous one as its starting point and
considered the evidence that had become available since then. It highlighted any
areas where further evidence was required, so that this could be specifically pursued
prior to the next review.
4.2, In the case of the Demo Stream the basis of assessment was the evidence from the
interaction of the team with suppliers up to early January, from the work of
colleagues in the Requirements/Solutions Stream thereafter and from the work
involved in the latter stages of ITT drafting. The Contract Stream evidence derived
from the Core Negotiating Team meetings with suppliers up to the end of February,
4.3, Each Group Leader within the Demo Stream identified a number of lower-level
topics and their relationship to each factor; the matrix of topics to value factors is
shown at Annex C. Prior to each Demo Stream assessment review, the six Group
Leaders scored each of their topics in relation to each relevant value factor. Pre-
agreed weightings were applied to these to produce an initial score for each factor for
each Group Leader and the six resultant scores for each factor were averaged to
produce a provisional Stream score. At the review the markings of each Group
Leader were subjected to quality assurance by the others and any resultant changes
were incorporated into the model. The amended factor scores were then reviewed by
the Stream to take account of the varying amount of evidence brought by each Group
Leader to each factor and final scores agreed together with the supporting rationale.
44. At the Contracts reviews each attendee put forward their mark for each factor (where
appropriate - see below) and a discussion then ensued, leading to production of an
agreed score for the Stream with accompanying rationale. Members felt they had no
evidence on which to mark factors 1, 2, 3A and 5. Validity markings of ‘E’ (= no
evidence at all - no score can be given) were therefore recorded in respect of these
factors.
>
ln
At the Programme reviews the Stream which had most evidence relating to a
particular factor was the first to put forward its scores and supporting rationale. The
other Stream then either put forward its own score and rationale or performed a
purely QA function (factors 1,2, 3A and 5 - see para 4.4). After this, the attendees as
a whole agreed a score and associated rationale that took account of the relative
weight of evidence from each stream (including the partnership stream -see para 3.4).
11 March 1996 Page 3 of 12 Issue 1.0
PWKP4-25
RESTRICTED CONTRACTS
POL00031154
POL00031154
4.6,
47.
SA,
The full list of reviews held was as follows:
es
§ December 1995 (P) ~ -
18-19 January 30 January (P) 2 February (P)
7-9 February 19 February 20 February
I 4-3 March 1 Match 7 March
{P) signifies pilot review
The Quality Assurance team received the output from the Programme review of 20
February and their comments were considered during the review of 7 March, Two of
the team attended the 7 March review and exercised a QA role during the meeting.
RESULTS OF THE FINAL PROGRAMME REVIEW
The final Programme Review (7 March) considered:
{a) the output from the Contracts Review of 1 March and the Demo Review of 4/5
March;
(b) _ the results of the review of suppliers’ Partnership capability:
(ce) the feedback from the QA team on the results of the Programme Review of 20
February.
The results of the review are attached as PWKP4-22.
February.
The presentation of the
material takes account of comments on the report of the 20 February results, an
expurgated edition of which members saw following the Board mecting of 26
The results of the lower-level assessments within the Demo Stream review of 4/5
March were made available to the QA team. Further detail below that level is being
retained for subsequent perusal by audit teams etc. if required.
A number of points need to be borne in mind when considering the results:
@
the pre-ITT hurdles process established that all three suppliers meet the
minimum requirements set by the sponsors:
11 March 1996
Page 4 of 12 Issue 1.0
PWKP4-25
RESTRICTED CONTRACTS
POL00031154
POL00031154
wa
a
6.2,
63,
{b) this process also identified a number of issues relating to some of the hurdles,
none of which were regarded by the Board at its meeting on 26 February or by
Messrs Peaple and Brown at their subsequent ‘empowered’ meeting on 28
February as sufficiently serious to disqualify any of the suppliers;
(c) the Demo Stream breaks down into six ‘strands’ and 44 topics, many of which
relate to more than one value factor (see Topics vs. Value Factors matrix
reproduced at Annex C). This means that, within a value factor, low scores on
some topics may well be balanced by high scores on others and result in an
overall mark at Programme level (which also takes account of input from the
Contracts Stream) that is not particularly high or low.
(d) it follows from the above that members should not necessarily expect a low or
high score on factors where they have pre-knowledge that a supplier was
regarded badly or well on a particular topic. They should take comfort from
the fact that none of the value factor scores emanating from the Programme
Review is inconsistent with the hurdles clearances already given.
Para 3.1 referred to the belated realisation that the Value Factors should ideally have
included Fraud-Free Services for POCL. The scores from the Programme review
are:
. Cardlink 3.7
. IBM 53
. Pathway 3.
Since this factor has not been notified to suppliers it is considered that the scores
cannot be used in the evaluation; the Board’s endorsement of them is not, therefore,
being sought. However, the Programme review felt that the fact that all suppliers
were considered satisfactory in this area should be brought to the Board’s attention.
USE OF THE RESULTS IN EVALUATION & SELECTION
The scores emerging from the Board’s consideration of this paper will be lodged with
the Programme’s lawyers before receipt of tenders.
Once the various assurance reviews of the tenders (e.g. technical, contract) have
taken place the reviewers will consider whether there is reason to reappraise the
Value Factor scores. If there is, review will take place at succeeding levels of
cumulation. Ifa Programme review is necessary, and results in any changed scores,
then the outcome will be put to the Evaluation Board for endorsement.
‘The results emerging from the process at para 6.2 will be subjected to weighting and
sensitivity analysis and the outcome presented to the Board in the Evaluation Team
report as outlined in PWKP4-5.
11 March 1996
Page $ of 12 Issue 1.0
POL00031154
POL00031154
PWKP4-25 RESTRICTED CONTRACTS
7. RECOMMENDATION
7A. The Board is recommended to endorse the scores for the value factors for the three
suppliers as set out in section 2 of PWKP4-22 (except factor 3A - see para 5.6) and
their lodgement with the Programme lawyers.
7.2, The Board is also asked to note the scores in respect of value factor 3A recorded in
para 5.5,
1] March 1996 Page 6 of 12 Issue 1.0
POLO0031154
POL00031154
PWKP4-25 RESTRICTED CONTRACTS
ANNEX A- VALUE FACTORS
1. Customer the service provider needs to ensure that the services are viewed
Acceptability favourably by customers at post offices, and that the benefit payment
services are acceptable to all benefit customers choosing to be paid at
post offices.
2. Staff/Agent The service provider's services need to be Post Office and BA local
Acceptability I Office staff friendly, for example by being easy to use, responsive and
supportive of their job functions.
3. Fraud-Free The measures proposed to make the service for benefit payment fraud
Method of free and to maintain that fraud free level.
Payment
4, Credibility of The service provider’s designs, procedures, tools, methods, resources
Delivery _and organisation need to ensure that the steady-state services will be
_ delivered to time and quality, showing understanding of and empathy
with the BA and POCL requirements and objectives.
5. Start-up The service provider's design, development, acceptance and initial
implementation services need to be credible, showing appropriate
controls, management interfaces and capability for managing,
controlling and delivering the start-up of the services.
6, Innovation The service provider needs to be pro-active, change-orientated and
demonstrate a genuine “can do” attitude.
He should generate creative ideas and understand how to apply
technology profitably building upon his initial services, complementing
rather than competing with POCL’s core competencies.
He should seek to do existing things better and better new things.
7, Flexibility The service provider and his services need the ability to react to external
change and to meet a diverse range of existing and potential needs,
thereby maximising value for money and faster delivery of new
products and services,
8. Management The service provider management needs to be competent, customer
Capability focused, accountable, with evidence of well defined internal controls
enabling a proper external focus. Suitable quality processes must be in
place, and key resources must have skills and experience appropriate to
their roles.
11 March 1996 Page 7 of 12 Issue 1.0
POL00031154
POL00031154
PWKP4-25 RESTRICTED CONTRACTS
9, Reliability and I The service provider needs to anticipate and prevent problems, with
Support robust fallback procedures for benefit payments and other customer
services in the event of system failure. He should regard his first priority
as maintaining continuity of service rather than referring to the contract
in the event of unforeseen problems.
10. Stability / ‘The prime service provider and his associated consortium members
Coherence and/or main subcontractors need to have a stable relationship from
which to provide the services. The member organisations should
balance each others’ skills and resources to match those needed to
deliver the services. The prime service provider should show his
competence to successfully lead such a group of organisations to deliver
similar services.
11 March 1996 Page 8 of 12 issue 1.0
POL00031154
POL00031154
PWKP4-25 RESTRICTED CONTRACTS
ANNEX B - MEMBERSHIP OF REVIEW PANELS
Programme Review Panel
Andrew Stott Chairman
Robert Albright Contract Negotiation Team
Dave Miller Contract Negotiation Team
Pat Kelsey Contract Negotiation Team
Tony Johnson Demo/Requirements/Solutions
Michael Berg Demo/Requirements/Solutions
Bob King Demo/Requirements/Solutions
Sean Johnston Quality Assurance - SSA.
Wayne Stephens Quality Assurance - BA
Tim Brown Quality Assurance - POCL
Derek Selwood Facilitator
Michael Purchase Facilitator
Nick Richardson Secretary
Contracts Stream
Dave Miller Chairman
Pat Kelsey BA Chief Negotiator
Robert Albright POCL Chief Negotiator
Derek Selwood Facilitator
Nick Richardson Secretary
{cont}
11 March 1996 Page 9 of 12 Issue 1.0
POL00031154
POL00031154
PWKP4-25 RESTRICTED CONTRACTS
Demonstrator Stream
Tony Johnson Chairman
Michael Berg Stream Leader
Jeremy Folkes Group Leader - POCL Infrastructure
John Meagher Group Leader - POCL Applications
Colin Oudot Group Leader - Benefits Payment Service
Janis Hatcheil Group Leader - Implementation & Management
Naresh Mohindra Group Leader - End to End Solution
Gareth Lewis Group Leader - Security
Michael Purchase Facilitator/Secretary
Derek Selwood Facilitator/Secretary
11 March 1996 Page 10 of [2 issue 1.0
POL00031154
POL00031154
PWKP4-25 RESTRICTED CONTRACTS
ANNEX C - TOPICS VS, VALUE FACTORS MATRIX (reproduced from PWKP4-12)
In the table below ticks (“) indicate the primary relationships between the Demonstrator
topics and the Value Factors, while the gaps indicate that issues arising from these topics are
unlikely to affect the corresponding Valne Factors.
Value Factor
iP2[3{4[s le[7[s 9 [ie
BPS
Card characteristics v viv
Card distribution ¥
Card usage v
<
SES
Card status monitoring
SIAESIALS
SISESIATS
IENENEN
<i is
SISSS
Card support services v
POCL Infrastructure
OP Hardware v
OP Middleware
WAN
TMS
Integrated Infrastructure v
SISESESIESES
System Management
RISERS SSS
RISERS RESTS
SERESTRAE SESE
Technical Support Services v
SAISIAISISISTS
SIAISIAISISIS TS
SIAISIAI SISSIES
SIAISIAISIRISSS
HCL
6
POCL Applications:
(a) EPOS
Transaction processing ve
Accounting
<<
NINES
ai<I<
Robust application
Introduction of new products i I i
AIS
“
aLRA
Generic approach fot ft I
(b} Automated Payments
Transaction processing - local vivi
Transaction processing - central Yio
AESER
SISTA
Data Integrity ¥
4
Introduction of new clients
SINSISIS
<
SASS
Introduction of new products
implementation and Management
Roll-out
SIS
SAN
SN
<
<
Service Management
AIAISIS
aps
Card Marketing & Education v
SESS S
Operational Trial
11 March 1996 Page 11 of 12 Issue 1.0
POL00031154
POL00031154
PWKP4-25 RESTRICTED CONTRACTS
Value Factor
iT 273 Tse sel 7 ere tie
End to End
Design Assurance viv Yiviviv¥iv¥ vv v¥
Financial reconciliation & settlem’t v viviv viv ive
Transaction timings viv ae
External interfaces (incl CAPS) v v Yitiv vive
OBCS viv iv ¥
MIS v viv
Security
Card characteristics
Card Production & personalisation
Card and PUN Distribution
Card Usage
Card Support Services
Contingency
Interfaces
Service development
Service transfer
SERISESRESIAESE SSE
Fraud & risk management
{1 March 1996 Page 12 of 12 Issue 1.0