POL00039735
POL00039735
To
From
Bruce McNiven. Copy John Main
Andy Radka
Mike Granville.
. Peter Restarick
Clare Dryhurst
Trevor Rollason
Dave McLaughlin
Kathryn Cook 9 April 1999
HORIZON TRAINING: COMPETENCY
1
When we spoke last week I promised to let you have a progress report
on the Horizon competency work which you commissioned in January.
As you know I've had a small group working to understand three
things: oo :
the extent to which individual competency now (i.e. pre Horizon)
affects the ability of that individual to successfully undertake
Horizon training and operate successfully in live state;
e the extent and nature of any gaps in competency;
¢ our ability as a business to fill any gaps identified and the impact of
doing so (and of not doing so). .
To understand these issues we've had the advantage of a fairly large
amount of data that has been collected already by various people
across the business (including the Horizon team and BSM ) looking ata
range of issues (including feedback from those offices which already
have a form of Horizon). We have also commissioned some additional
work from Research Services to fill in some of our remaining data gaps.
The attached report is structured to address three main areas:
the background issues; why competency is: important etc.;
¢ what the various pieces of research that have been carried out are
telling us about front line competency now and in an automated
environment;
¢ what we think we need to do prior to the commencement of live
trial; what needs to be measured as part of the trial and, finally,
issues that we'll need to track/measure/address further
downstream. ~.
POL00039735
POL00039735
4. Let me know if you've any queries and if you're content that we
proceed as recommended (section three of the report): there are of
course budgetary considerations which we'll need to clear and it'd be -,
helpful to bottom these with you ASAP.
KF Cook
HO Network Resourcing Team
POL00039735
POL00039735
HORIZON COMPETENCY REPORT
Section One: Background issues
1.
The business’ current focus of allocating training resource in the retail
line to new entrants has been under review for some time. Concerns
have centred primarily around the limited numbers of levers that are
currently available to support on going development of peoples skills
once they have joined the business (either as a member of staff or as a
subpostmaster). The recent review of TK and its reallocation of training
time towards improving the performance of ‘poor’ offices has gone
some way to enabling us positively to affect competence after .
recruitment but there is still some way to go (and this is something that
the Network Team will be looking at in 1999/2000. Clearly its
important that we get better at understanding and managing ongoing
individual competence in order to support the introduction of new
products and services and to deploy developing service standards in
an increasingly competitive environment. The introduction of Horizon
(and the ongoing work to develop a conforming culture within the
business) has made the need to address this more pressing. This report
addresses some practical competency support mechanisms and also
covers steady state training issues.
The business has never had a detailed definition - other than the
recently introduced one for new entrant counter clerks which is
applied at the end of their trial - of what we mean by ‘competency’ for
all of those who are customer facing in post offices (employees, agents
and assistants). Nor do we have any specific mechanisms to ‘audit’ or
assess competency levels of those serving our customers either
individually or collectively (this is particularly. true of agents and their
assistants). Our first task as a group has therefore been to try to'define
what we mean by competency. Attached as annex A is the competency
statement we, and those who have seen it (including HoRNS, several
members of the HO Network Team and the People Programme Board)
feel is a credible first draft of what we require now and how it will
change when Horizon is in operation: it deliberately highlights areas of
competence, not absolute competency levels although, when we
understand this issue better in an automated environment, this may be
something that the business wants us to address at a later stage.
Why do we need to bother about competency?
We know already that there are varying degrees of individual
competency in the network and RNMs know only too well which are
their ‘problem’ offices. The benefits though from us developing a
better understanding and managing competency exist on a number of
levels and will include:
POL00039735
POL00039735
¢ Horizon contract: although there are no provisions in the contract
for POCL to pay remedies for a lack of competency by those using
the system, the business does have an obligation to ensure that only
competent people use the system and Keith Baines has confirmed
that we would be liable - in the event that they , or we, could
measure it - to compensate Pathway for their costs of rectifying any
damage which resulted from us failing to meet that obligation or if
lack of competency of those using the system was a contributory
factor to any service shortfall e.g. if transaction times exceeded
contractual limits and inadequate competency levels were one
reason for this. (In roll out ICL are carrying out the training and are
measuring competency so the penalty risk to the business is clearly
lower but in steady state training (where the subpostmaster will
play a major role in ensuring competency) the risk of ICL claiming
compensation from POCL is clearly higher;
the individual: we know already that some prospective system users
are nervous about the introduction of Horizon and indeed we've
already had at least one (well publicised) pre-emptive resignation. A
‘fear of failure’ on the part of prospective users - particularly where
they have already have had to transfer over to cash account could
lead to adverse impacts both on the size and reach of the network
but also on publicity and customer service. Similarly, in some parts
of the rural network - particularly where volumes are low (and pay
levels consequently modest) - the ‘bother’ factor of transferring over
to the Horizon system could make things difficult. Whatever can be
done to ease people through the training process should pay
dividends;
* the business: the business will bear the cost of putting things right
both because it is obliged by the Pathway contract and because it
obviously anyway has to provide extra resource e.g. fix errors
retrospectively or to offer additional help which is required by the
individual such as additional training, RNM or field support or
‘helpline assistance. Individual support needs are likely to vary
considerably and meeting this variation will be one of the key
challenges for the business;
the customer: the effects on customer service of a lack of competency
using the Horizon system will range from elongated transaction
times and QofS problems as users try to navigate their way through
the system or call a helpline, to loss of business as customers lose
confidence in our ability to provide effective service. Its important
that we understand the competency ‘learning curve’ and hence the
likely duration of impacted customer service so that the effects on
customers can be minimised.
POL00039735
POL00039735
When starting this work the key issue we were asked to address was
the extent to which a lack of competency on the part of the individual
prior to attending Horizon training would prejudice their ability to
make the most of that training and to operate successfully in a live
environment. The report addresses this issue but also has to tackle the
concern that is clearly present in the business about the effectiveness
of the ICL Pathway’s training solution as a mechanism for delivering
competence in using the system in a live environment.
Section two - Research findings:
5.
We have had the advantage of accessing quite substantial amounts of
research that has been commissioned by various people since Horizon.
kit first appeared in NE and SWSW. Many of these reports make
reference to competency issues albeit that none of them has specifically
addressed the issue of pre entry level competence and its effect on-post
Horizon training competency. We therefore commissioned additional
research to start to address this using as our target audience the 100
people (a mixture of subpostmasters, counter clerks and assistants)
who were trained on the newly baselined Horizon courses in March.
This research looked at a number of aspects:
eattitude/ expectation to the training;
previous exposure to and use of computers;
eindividual learning styles;
performance pre training (i.e. a measure of their competency now)
and performance using the Horizon competency test at the end of
training.
In addition Research Services was also able to carry out-a literature
search on competency issues addressed/ encountered in. other major.
(external) automation programmes.
Asummary of the findings from the detailed competency research is
at Annex B. The key issues to note are:
Pre course competency:
* pre course competency measures indicated an overall competency
level of 79% (those tested were given the option of looking the
answer up before answering the questions which were based on.
them demonstrating their transactional knowledge) but there were
no ‘patterns’ of failure (e.g. by age, status etc.);
« individual competency scores ranged from 33% to 100%;
© acompetency score range of between 24% and 100% was achieved
for specific questions;
POL00039735
POL00039735
© .scores for attendees from sub post offices were lower than those
from branch offices (77% compared to 84%);
Pre course attitudes/ experience:
¢ 65% had used a computer before; 83% said that they were familiar
with using a computer key board;
© 98% were looking forward to using Horizon but 13% were worried
that it would be difficult to learn and-only 49% thought that it
would be easy to change from current methods to using Horizon;
e 95% felt that Horizon was essential for their office to succeed;
Post course competency assessment/views:
¢ 3 delegates would have failed the training provided by ICL - 2.37%
of the total (i.e. well within the contractual target. Regrettably we
don’t have any detailed data to get underneath this e.g. whether
they were assistants or their demographic considerations and this
data needs to be collected so that we are in a position to understand
the detailed links to pre course competency);
¢ 97.6% rated the course as ‘satisfactory’ or better;
¢ improvements requested centred around the need for more training
time; more training time spent balancing and practice on using the
system. .
Research results from other major automation programmes re
competency:
¢ older people are less likely to ‘problem solve’ using a computer ie
they will ask for help from a helpline (as opposed to colleagues)
earlier on and will request help 2-3 times more often than younger
staff;
¢ older people take twice as long to learn but reach equal
performance;
¢ previous computing experience is not strongly related to learning
performance.
As mentioned earlier a number of existing studies of Horizon and
conformance have commented on competency issues: the key ones are
summarised at Annex C - the main issues that need to be noted from.
these studies are:
¢ Horizon training isn’t comprehensive i.e. it’s designed to teach
people how to use the system, not to fill in any competency gaps or
to deliver conformance per se;
post-training support has been defined as a period of time rather
than with specific learning goals; [however a training evaluation
POL00039735
questionnaire will be completed for each individual and used by the
HESO);
¢ outlets will be provided with excessive information i in the user
guides and will need to search for information that is relevant to
them;
¢ most problems in subpostoffices are to do with the preparation of
the cash account rather than with customer facing activities;
- the cost of non conformance is s already high and may increase with
automation;
.® current arrangements whereby subpostmasters are responsible for
training their assistants may be a significant area of risk under an
automated environment;
¢ POCL helpline staffing assumptions are based on 5 calls per office
per week in the first month and one per office per month thereafter;
¢ both the End to End Testing Evaluation report and the Model Office
-testing Report for Nile 2.0 raise their concerns about the adequacy
of the time allowed for the basic face to face training Horizon
training. (Provision ranges from one day for subpostmasters
assistants to a day and a half for those who prepare the office
balance e.g. subpostmasters). (Note: additional support provisions
e.g. remedial training provision and training mode will shortly be
available and were not factored into these assessments but it is still
true.to say that the training is not in itself flexible enough to
recognise individual training needs).
Section three: What's all this telling us about competency and what should
we do about it?
8.
That there is evidence of competency gaps isn’t surprising: many of
our current systems and processes are dedicated to handling errors
and_ we know already that we do not have a conformance culture. It
could be thought that one of the things that the data collected is telling
us is that although there was a pre course competency of only 79%, the
fact that only 3 people would have failed the Horizon training (2.37%)
could be taken as an encouraging sign that a lack of competency
currently would not be an inhibitor to using the Horizon system. There
may however be other explanations e.g. that the Horizon competency
test is ‘too easy’ or that it isn’t predictive of live operation (e.g. that its
measuring the wrong things).
We know a lot about competency but does the data collected thus far
allow us to answer two key questions?:
1. does a lack of competency in current operational procedures
prejudice an individual's ability to make the most of Horizon
training?
2. will Horizon training deliver competence i inlive operation?
POL00039735
POL00039735
POL00039735
10.
Although we have a lot of data we can’t conclusively answer either
until we have data which tracks people before during and after
training and measures their competency at each stage. We propose that
this work is commissioned from Research Services ASAP to allow the
offices (and the people in thern) receiving kit for the first time in live
trial to be tracked over time and competency assessments made. If we
were able to do this then we would have data available by the end of
May which would confirm the definitive answer to question one. We
also propose that we track competency over time in live operation to
enable us to answer the second question (and indeed to assess the
effect in steady state of varying approaches to turn over training).
Actions to be addressed now: The data collected already will allow us
however to start to develop our responses to known areas of concern
(this is however a ‘best judgement’ issue and some areas of concern
may not be borne out during live trial) and to measure their
effectiveness as supports to improving competency as part of live trial.
The areas we recommend that we concentrate on are all ones which
directly link to supporting competency and which are largely ‘free’ or
low cost to develop. (See paragraph 11 for our recommendations for
more resource intensive competency support). We recommend that we
develop enablers to address the following issues:
* Pre training competency support - expectations about
Horizon are high and people expect it to be easy to use but
only 49% felt that it would be easy to change their current
methods. One option for the business is to consider clarifying
in a more direct way for those about to undergo Horizon
training what will change and what won't and (as a way of
starting to plug current competency gaps) we could help
people to think about their current competency gaps in the
run up to training (e.g. by offering some form of self score
sheet to remind them of what signs there may be already that
would indicate a current competency failure) There are
already vehicles which we could use for this such as the
Horizon Implementation Guide or the ‘Countdown to Go-
live’ documents which have already been produced. We
could also back this up by ensuring easy access to distance
learning and other training opportunities (e.g. access to their
Transactional Knowledge Manager).
¢ additional balancing/accounting support - concerns about
balancing appear to be pretty universal (both in an automated
and non automated context). One key activity is to better
equip people before they attend training to understand what
changes there will be to the balancing process and to expose
POL00039735
POL00039735
11.
12,
them to the new terminology e.g. ‘individual/shared
balancing and stock unit balancing
¢ structuring of post training support - post training support
(via HFSOs etc.) can be tightly structured to meet the learning
needs of the individual and this should be developed along
with their existing materials. In some instances, the level of
HEFSO support may need to be flexed to meet individual
needs and resource might be available from within existing
templates e.g. through using TKMs to support this project;
° providing support materials - materials to support in office
preparation sessions prior to attending training could be
developed to reinforce the disciplines necessary to both use
the system and to use reference materials;
increasing kit access - opening up the option of visiting offices
which have already gone live or to open up counter training
schools to offer access to kit should also be explored;
¢ spread best practice - we already have experience of what can
go-wrong (and how to put it right) in live operation, this
information should be available to those who are newly
trained to reinforce their learning (via HFSOs and from
encouraging networking amongst subpostmasters)
In order to assess the effectiveness of these measures we propose that
half of the offices receiving Horizon kit for the first time (c52 offices) in
live trial are offered these solutions (and their take up monitored) and
half not. Tracking of HFSO support and measures of help required e.g.
of helpline calls to support first balance will allow us to understand
very quickly the need to support first and subsequent balances (e.g.
with HFSOs). Measures similar to those detailed in paragraph 7 should
also be measured for all of these offices (all 104) and, additionally,
much more detailed competency data collected from the Horizon
training so that we understand individual competency gaps better.
Finally, to allow us to track the appropriateness of our competency
tests and measures we should continue tracking performance every
three months. Detailed research proposals are currently being drawn
up and will be available shortly.
Solutions which involve the deployment of large scale resource (e.g.
more in-office support post go live) and feel instinctively right (and
supporting all first balances for example feels instinctively right) will
need to be“proved’ by the measures outlined in paragraph 9 above.
Such support could be very expensive for the business (e.g. it’s costing
c £6m for the current HFSO resource) and we need to understand the
POL00039735
POL00039735
cost to the business of not not supporting offices will become clear from
this research.
In conclusion it clear that there is an urgent need to better understand
the operational effect of the introduction of Horizon on our front line
capability, particularly on our ability to meet business KPIs. We also
need to understand the effects for the individual (and consequential
knock on effects back on the business e.g. demand for helplines. The
proposals outlined above are designed to both collect the data we need
to understand these impacts and to develop short term enablers to .
address those:issues which can already be identified as areas for
improvement.
POL00039735
POL00039735
Annex A
Competency statement
Now
Post Horizon?
Complete transactions accurately
e Customer is ‘happy’
* no errors :
transaction completed as fast as
possible ‘
e keep client ‘happy’
sell appropriately
e in your own preferred style
* complete documentation as
necessary
Use reference materials
e haveallrelevant (uptodate
materials in the office) and know
where it is
© Know how to use the
documentation
* actually use the relevant
documentation (i.e. don’t guess)
¢ know how to get help if stuck and
do so .
° Yes °
¢ Yes: transactions will need to be
completed in a specified way or
there could be systems difficulties;
transaction may be longer at first
and some transactions will require
authorisation;
errors will be more transparent
and to reduce these we will need to
standardise how. we do
transactions ©
e Yes :
transactions will need to be
‘ ‘completed in amore standard
way
e Yes
Additional Horizon requirements:
¢ use Human Computer Interface
effectively with customer
¢ reengineered transactions will
need to be completed : some will
require transactional procedures
‘which are very different to now
« Yes
* reference materials will be
different and user will need to be
able to use the new versions
° Yes
© Yes, but help facilities will be
different and user will need to
know which form of help to.use
when
POL00039735
POL00039735
Balance Stock Unit
e declare cash and stock onhand _ .
¢ feed into receipts and payments
any vouchers on hand
¢ bring items to account e.g. error
notices
¢ identify discrepancies
Office Balance
¢ amalgamate all stock balances
e Prepare office balance
Cash Account
take office balance and put into
c/a. Know reporting requirements
and retention periods :
When it all goes wrong
¢ Manual procedures (if ECCO) if
equipment fails
Using equipment :
¢ H&S/Cleaning/Security/Consum
ables
e Yes, need to use Horizon
effectively as part of this
© Yes, enter into Horizon correctly
and adjust as necessary. System
then produces report
e Yes, need to enter into Horizon
correctly and adjust as necessary.
¢ Yes, system does this, based on
what user has entered _
Additional Horizon requirements:
¢ need t understand what a stock
unit is and understand the whole
process using Horizon e.g.
including balancing periods
* Yes, automatic with Horizon (need
to know about inactive SU). Need
to complete any corrections
required to SU’s using correct
procedures
e Yes, need to understand
uncharged receipts and unclaimed
payments and how suspense
account works
¢ Yes, Hit a key correctly! Send to.
Chesterfield, know what/how to
retain
e* Yes, need to know what to do if
“equipment fails, how to recover
transactions (fallback and recovery
procedures); manually balance?;
revert to manual transactions?
e Yes, but the new versions
Additional Horizon requirements:
¢ Key board skills
¢ use of consumables
POL00039735
POL00039735
Annex B
Research findings: summary
Research parameters:
¢ target population: 100 attendees at baselined ICL training courses in March
1999
pre course competency measures -
self assessment of previous exposure to computer systems;
attitude/ expectations measured;
learning styles questionnaire;
post ICL training competency test results;
post ICL training comments/ views.
qualitative and quantitative data from other competency studies for major
automation programmes.
Research results (headline):
¢ Pre course competency measures:
overall competency score: 79%
range from 33% to 100% for individuals
range of 24% to 100% for specific questions
BO score: 84%; subs 77%
no gender, ethnic or language differences apparent (group
too small to differentiate) .
¢ no significant correlation between previous computer use and
competency (which would be expected)
e Previous exposure to automation: ‘
* 83% were familiar with using a-computer key board;
-e 65% currently used a computer system at work;
¢ 61% had previous experience of using a computer at work;
¢ 87% said that they felt comfortable using a computer.
e Attitudes/ expectations:
© 98% were looking forward to learning how to use Horizon
¢ 13% were worried that they would find it difficult to learn to
use the Horizon system
¢ 49% expected that it would be easy to change from current
methods to using Horizon
¢ 95% felt that Horizon was essential for their office to succeed
in the future
POL00039735
POL00039735
¢ 80% felt confident that Horizon would be easy to use
¢ 5% felt that there wouldn’t be much for them to learn to use
Horizon. (65% disagreed or tended to disagree with this
statement)
¢ Learning styles:
- the range of learning styles generally matched the population at
large with no real preferences coming through. (this could have
implications for Horizon training which as currently designed will
appeal more to activists and pragmatists than to reflectors and
. theorists.
¢ Post ICL training competency test results:
¢ 3 delegates would have failed the course i.e. 2.37% of the total
(well within the contractual target)
¢ Post ICL training comments/views:
¢ 97.6% of those trained rated the course as ‘satisfactory’ or
better
* several comments about the need for more time/too much
compressed into the course
¢ more time needed on balancing
¢ much better than on site training :
* would have been useful to have had more practice time
¢ Qualitative and quantitative results from other companies:
older people less likely to problem solve with computers (i.e.
will seek help earlier e.g. by calling helplines);
¢ older people take twice as long to learn but reach equal
performance; :
¢ older people request help 2-3 times more often than younger
staff; .
© computer aptitude and prior level of achievement are most
important in computer based learning;
© computer anxiety is related to slower completion of simple
computer tasks;
¢ previous computing experience is not strongly related to
learning performance;
© computing experience in males is a greater predictor of
performance than for females;
POL00039735
POL00039735
, Annex C
Findings from other research
Report:
Key competency issues:
Before and After group
Horizon training isn’t comprehensive t e. its not
designed.to do any more then teach people how to
use the system) and it doesn’t ensure that people
’I are prepared for training.
Post training support has been defined as a period
of time rather than with specific learning
objectives; it has no specific tools;
Concerns about user documentation: particularly
links to operations manuals;
outlets will be provided with ‘excessive’
information much of which is irrelevant to them;
Pathway user instructions only cover automated
activities not manual ones which are required to
support the ‘whole’ activity;
Levels of conformance have three basic causes:
¢ awareness of correct procedure;
¢ ability of individual (and availability of
correct aids);
© process: does it build in compliance?
Horizon CSF workshop
ICL have expectations about our level of
performance which will need to be met
HEB Study
Most problems in subpostoffices are to do with the
preparation of the cash account rather than in
customer serving activities
Levels of errors in POCL is low already; fine
tuning is of particular importance
Low error offices have a structured way of
completing back office procedures (e.g. tick lists)
CEC conformance paper
(March 1999)
Cost of non conformance to outlet transaction and
processing standards c£31m pa
automation is not tolerant of non conformance
transaction errors cost TP £4.5m now; an
additional c£1m with Horizon?
Regions spend £10m now handling errors
Clients have penalty clauses in their contracts
which are expensive for the business -
Under Horizon errors will be more visible to -
clients ;
Not everyone gets trained (assistants)
POL00039735
POL00039735
Business case for
conformance Strand 3
Horizon transaction time is slightly longer than the
manual process but benchmark timings are
achievable
Steady state training arrangements for assistants
need to be addressed: -
50% of offices didn’t follow standard
methodology and the number of non conformance
incidents has increased since December (people
finding ways around the system?)
Helpline staffing has assumed that there will be 5
calls per office per week in the first month and one
call per month subsequently. An additional call per
office after the first month adds £3.9 m in cost
End to End testing
Evaluation Report for
Nile 2.0
‘Unanimous view from the End-to End team is that
the training event will not be sufficient to equip
‘real’ end users in readiness for their offices
receiving the Horizon system. The course was
thought to be too short with too little time to
reflect.’ More time was felt to be needed on
balancing and accounting.
Model Office testing
Evaluation report for
Nile Release 2.0
‘A day and a half is not long enough to cover the
Horizon system (particularly for PO staff not used
to an automated working system’.