POL00028435
POL00028435
Proving the TIP Interface
(and Implications for Horizon testing)
Draft Report
POL00028435
POL00028435
Terms of reference
1. review testing plans and procedures
2. assess issues and problems
3. remedial action, taken and required
4. reporting and control arrangements
5. further actions to ensure success
6. identify risks which need management to achieve successful outcome
TIP
¢ David Parnell
¢ Peter Jones
Pathway
¢ Steve Doyle
¢ Roy Smethurst
¢ Nikki O’Sullivan
POL00028435
POL00028435
Meetings
Horizon
¢ Simon Rilot
* Chris Young
Model Office
¢ James Brett
French Thornton
* Andrew Simpkins
* Naresh Mohindra
¢ Tim O'Leary
POL00028435
POLO0028435
Structure of Report
1. Perceptions from the main parties.
2. Assessment and technical actions
3. Reporting, communications and control actions
4. Summary
POL00028435
POL00028435
The Chesterfield View
DIT was problematic for TP; DIT 1 and the first DIT 2 reducing
confidence (but the second run of DIT was reported as ‘successful’)
TP believe that 33 days is needed to be certain of migration; MOT is heed Udens.
only 21 days, E2E has been extended to 33 days, but is deficient in NG wes
other areas
The weekly testing meeting is “very politically driven...issues don’t
get aired”
Concern that serious problems get watered down at each transmission
stage e.g. “Is everyone clear that we have not yet done a cash account”
a)
Concern that some problems from MOR 1 were never addressed, e.g. Juggs.
22 files outstanding for delivery to TIP
POL00028435
POL00028435
TIP Concerns
Cash accounts do not balance
Mismatch between Reference data at TIP and at Pathway
Files rejected by TIP
Script errors at Model office
Operator errors at Model office
Problems with environment causing tests to be run out of sequence
Unuvnes I
“Cannot take risks on this ...it’s a showstopper” - Dave P
POL00028435
__P0L00028435
The Feltham View
The Interface Specification concentrated on the syntax not semantics. Va
(i.e. it specified the layout, but not the meaning of particular fields) ie al
Horizon take too long to sign off AIS revisions (5.3) - why? hore oxedad
TIP rejects whole files too readily Ray ea bans
— the scales example I . we
— the timestamp
Concerned POCL do not understand scale of business issues i
n
reference data shy aw unferdiig, S50 .
— controlling which non core products in which offices
the menu hierarchy is important to Pathway commercially —~»>
°
ae —
Model Office Environment
Simulating 7 offices gives sufficient range and volume
Inspection of the scripts shows them to be thorough and well specified
The control of the office environment is good
The rate of script and operator errors is no worse than should be
expected, and does not detract from the primary purpose of MOT
The timespan of 21 days is adequate for the primary purpose
The Model Office Environment is well planned and executed; fit for 4
purpose and to best industry practice
(so long as it sticks to its purpose)
POL00028435
POL00028435
POL00028435
POL00028435
Model Office Progress
¢ Environmental issues cause severe problems, e.g.
— on 21/10 the wrong date was entered, loosing one day
— on22/10aBA file failed, loosing a further 1/2 day
— no interface files sent to TIP so far due to dates synchronisation
problem)
POL00028435
POL00028435
what found / evidence
2nd pass of E2E shows as many errors as first, yet we should see
significant reduction
at least 150 repairs to go into final MOT
at least further 30 outstanding problems
Sica
At this rate, we will have at least 40 problems (many more incidents) w+}.
4a_the final MOT /E2E
the likely number of problems outstanding at the end of testing is higher
than the business may accept; or worse, live trial is so bad that rollout
delayed
Gk L erme tewte Grays te yt fd,
POL00028435
POLO0028435
Inodenks
Analysis of Errors:
Ef Script
Lai Data
Al System error
O other
Problems associated with running tests overwhelming real
errors
Assessment of TIP Concerns
‘Cash accounts do not balance
Must be fixed and proven
Mismatch between Reference data
at TIP and at Pathway
Causes 10% of all errors, but
number of differences unknown.
Should be analysed
Files rejected by TIP
having stripped out reference data
problems, resolve any underlying
errors
Script errors at Model office
Operator errors at Model office
i
Problems with environment causing
60% of all errors
Need to understand whether have
realistic aims here
Nie wad bok fe Agar I eperoly evr
\s cteage. UW Sleve's exgerence .
oh
tests to be run out of sequence
POL00028435
POL00028435
Assessment of Pathway Concerns
POL00028435
POLO0028435
Interface specification concentrates
on syntax .
Inspection of document shows
concern justified, but this is not a
unique issue for a design team
\
ogee bak fo lk -— oh Guin s
AIS revisions
There is a local agreement, but
Horizon should quickly accept or
reject
Whole files rejected for 1 error
Not accepted; can understand reason
for rigour at TP interface (though it
may be expedient to take some risk )
Ne
Scale of reference data issues
Experience suggests concern
probably valid, but nevertheless, it
has to be made to work technically
Menu hierarchy
The design should separate this and
reference data; fix on fail
unattractive
N&
POL00028435
POL00028435
Reference Data
Reference Data is Central to many of the issues
* the current design of 3 independent feeds is messy
* Noevidence of cash account data mapping
— TP believe this was agreed by all parties (and is now planned to be
done)
* how can we verify that each time fresh data is dropped, the reference
data is correctly replicated
* balancing accounts could be due to a double error
Experience at other sites suggests there will be considerable I bes.
operational problems with reference data; vital to be confident I Tyg -
that IT part is reliable (to get through testing, ... and beyond... )
POL00028435
POL00028435
Working Together - TIP, Testing
and Pathway
Issues are not getting resolved
* .correspondence evidences problems in working together
—. useful to state issues once in writing
— not useful to have stream of correspondence
‘ e partial answers given in meetings with Pathway
— e.g. “All Cash Accounts Balance”, but this is not the full story, © ecawomet whe
they only balance in a special Pathway environment
— Pathway may feel under significant contractual pressure
¢ documents and letters all ‘on message’
— atest report stating “all cash accounts have been produced” would
have been clearer if it stated “but we have still to get them to
balance” :
Points to an underlying problem of confidence ... or fear of creating
a lack of confidence
POL00028435
POL00028435
Diagnosis
TP are (largely) justified in their concerns toler Tes is Cong corte .
the test strategy has becometoo.complicated is abr ngir babs exearhd hae moewert
objectives for each test are unclear be on planta «
lack of transparency, leading to suspicion that all is not well 0”
premature integration, hope triumphs over realism (Le. integration was Have Pit
started before components were proven leading to more complicated ctor .
debugging)
inappropriate testing methods chosen, leading to more work (in
particular, MO and E2E are wrong forIfinancial validation)
vv
awd
ferns, eal haan = =
bptaneen [ges valk, Unni mse on
POL00028435
POL00028435
Fitting Test Styles to Need
-THCI & ‘Soft’ Issues IData/ Financial Dan ELE Arama
; Integrity ay ds bis
‘Data Input Real Operators 100% Consistent
‘Data Volumes IMedium Low
[No Offices Range of Types (7) [1 or2
Coverage All transactions with IAll transactions with
input permutations I data permutations
Timing I day/ day many simulated days
I per actual day
Equipment. Identical Same functions
Frequency Once per build Repeatable, until 3 tote reoR ?
correct Ly » “seg . ;
Timespan 2idays 33 days ae
The Model Office is ideal for first column, and poor for second
POL00028435
POL00028435
Bringing Testing to a Successful
Conclusion
¢ Because this is a PFI initiative, it was decided there was no place for
conventional UAT, ...
BUT - the service must be acceptable to POCL
We must find a way of addressing the issues on the TIP Interface
¢ Recommended actions need to take account of where we are, ...
BUT - The present course has a high risk of failure
(Le. the current process is wrong for the type of testing proof we need
for TIP)
An extra run of MOR does not focus on areas of uncertainty
POL00028435
POLO0028435
Actions to Reduce Risk
1. Confirm the primary purpose of MOR’T is about office working and
procedural issues. In discussion with the parties, allow the Model
Office team to drop tasks concerned with financial integrity IF a
necessary to secure primary purpose. \lagesed ws
Sx Hare we 2. Construct a different test to demonstrate financial integrity (possibly x ie x
it fine? ? based on DIT). The cost of this would be significantly less than ee
another MOR run, and could proceed in parallel. & for Imthdagoah Hn
3. Review with Pathway whether they are getting value from current run
* of E2E. A quicker route to the end goal might be to stop, give Pathway
time to rebuild the system and then restart
Wank Oupenk rom_tI E2E - PR
POL00028435
POL00028435
[inn opt
Contingency Actions
4. Drop automatic interfacing of Cash accounts (TIP), but any underlying
problems would remain.
5. Offer only one migration option (either manual or 1c) to reduce the xX ast bed oy
complexity of testing.
6. Reduce the number of offices in.live trial so that the anomalies can be I
coped with
These actions are all unattractive in that they descope the programme.
POL00028435
POL00028435
Reporting
¢ Allanomalies found in MOT noted on an incident report
¢ After removing incidents which are not product defects, PinICLs are .
raised for rest.
* Daily reports are written by the MOT, showing Incidents and PinICLs
raised
BUT
¢ PinICLs are closed once Pathway believe they have solved the
problem
* Itis believed that errors may reappear in later releases, but there is no
system for tracking -
The data collected does not give Horizon a view on system stability
Resch. nracde te Lase an amesmenk §\ Oro
Needs for Test Reporting
* The daily reports demand much effort yet cast little light on trends and
stability
* Pathway needs to own an independent view of system stability
* This can be achieved using Incident Reports
[ Reports should not be closed until Horizon have verified the I
solution
¢ Aweekly testing report should show,
— Outstanding brought forward, raised, solved, Outstanding carried
“>a. forward,
we re — This should be analysed by business impact, and occasionally by
oe) area affected (e.g. TIP) and product area
— The graph of outstanding each week is the most useful indicator of
stability.
¢ Pathways view will often be different; occasionally the two views
should be reconciled.
POL00028435
POL00028435
Some Detailed Actions to Improve
Reporting and Control .
Horizon maintains its own log of defects
This log is analysed weekly to meet the specified needs (outstanding,
trend, etc.)
Reduce daily reporting to a simpler content
Institute weekly reporting giving the information specified above
POL00028435
POL00028435
Communications within the
programme
© “Our views are sought, but appear to be ignored if not on message”
© “Don’t get feedback on PinICLs”, but this may be changing
These concerns, together with the previously mentioned problems of
working together, draw us to consider the extent to which the roles
and responsibilities of the Testing Group are serving the Programme.
POL00028435
POL00028435
POL00028435
POL00028435
Roles and Responsibilities
(Which side is Horizon Testing on?)
TP worry they have to be the ‘conscience’ of POCL, isn’t that Test _
Manager’s job?
SR thinks testing “in the middle” Tie [0 mo. Lo:
TP worried that the Test Manager doesn’t have the [perspective’ ~ dege Sd lgvaee «
What is the mechanism for recommending acceptance?
— Oncurrent evidence, would TP feel the system was acceptable?
It may be better to position Horizon Testing as independent
of Pathway