POL00039648
POL00039648
ICL Pathway Training Evaluation Paper Refi SU/TRN/OOS
Version: 2.0 08/7/99
Date:
Document Title: Training Evaluation Paper
Document Type: Reference
Abstract: This document defines the processes applicable to the Peritas
activities undertaken to discharge the contractual
responsibilities for the measurement of the effectiveness of the
user training programme in a manner consistent with the
Kirkpatrick model.
Status: Draft
Distribution: Stuart Kearns ICL Training Services
Richard Kelly ICL Training Services
Steve Lovegrove ICL Pathway
Satish Mistry ICL Pathway
Robert McDermott ICL Pathway
Jim Flynn ICL Pathway
Liam Foley ICL Pathway
Paul Underwood ICL Pathway
Barry Hancill ICL Pathway
Bill Herd ICL Pathway
David Howells ICL Pathway
Dean Felix ICL Pathway
Andy Barkham ICL Pathway
Simon Robertson ICL Pathway
Bruce McNiven POCL
Douglas Craik POCL
Trevor Rollason POCL
Anne Cocker POCL
Author: Andy Barkham
File ref Evaluation.doc COMMERCIAL IN CONFIDENCE Page 1 of 22,
© Peritas Ltd
POL00039648
POL00039648
ICL Pathway Training Evaluation Paper Refi SU/TRN/005
Version: 2.0 08/7/99
Date:
0 Document control
0.1 Document history
Reference Version Date Title
PATH/22/047 * 0.1 22/8/97 Training Evaluation Paper
SU/TRN/005 1.0 15/9/97 ICL Pathway version of the above
SU/TRN/005 2.0 08/7/99 Changed as a result of DSS contract
. termination and removal of the Benefit
Payment Card.
0.2 Approval authorities
Name Position Signature Date
S. L. Kearns Training Project Manager
S Lovegrove User Implementation
Manager
0.3 Associated documents
Reference Version Date Title . Source
1 SU/REP/021 1.0 22/08/97 Standard Reports and Formats ICL
Pathway
2 BP/TRN/001 7.0 6/11/97 Training and User Awareness ICL
Baseline Document Pathway
0.4 Abbreviations
MTC Minimum Training Compliance
RODB Rollout Database
TAC Training Administration Centre
File ref: Evaluation.doc COMMERCIAL IN CONFIDENCE Page 2 of 22
© Peritas Ltd
POL00039648
POL00039648
ICL Pathway Training Evaluation Paper Refi SU/TRN/OOS
Version: 2.0 08/7/99
Date:
0.5 Changes in this version
Changes made to remove references to the Benefit Encashment Service following the
Treasury Review.
File ref: Evaluation.doc COMMERCIAL IN CONFIDENCE Page 3 of 22
© Peritas Ltd
POL00039648
POL00039648
ICL Pathway Training Evaluation Paper Ref: SU/TRN/O0S
Version: 2.0 08/7/99
Date:
0.6 Table of content
3. Principles of Training Evaluation.
3.1 Introduction ..
3.2 The Four Level Approach to Evaluatio:
3.2.1 Level 1 csesssssesssssecssneceensessnsecsenes
3.2.2 Level 2
3.2.3 Level 3
3.2.4 Level Four Evaluation
4, Contractual Obligation:
4.1 Venue Assessment
4.2 Training Assessment.
5. Proposed Procedure
5.1 Timing of Measurement
5.2 Course Appraisal
5.3 Venue Assessment
5.4 Level 1 Assessment
5.5 Level 2 Assessment
5.5.1 Measurement method
6. Remedial Training
6.1 Criteria for live operation...
6.2 Remedial Training Provision.
6.3 Remedial Training Timetable.
6.3.1 Types of Event
6.3.2 Scheduling ........0..
6.3.3 Result notification.....ssssessssessseeessseessssseeessneeesn
File ref; Evaluation.doc COMMERCIAL IN CONFIDENCE Page 4 of 22
© Peritas Ltd
POL00039648
POL00039648
ICL Pathway Training Evaluation Paper Refi SU/TRN/00S
Version: 2.0 08/7/99
Date:
- 7. Appendices
7.1 Appendix A1: Delegate User Training Course Appraisal f0rM .....sssssssssseeere
7.2 Appendix A.2 Trainer Course Appraisal (User Training) ......ssssssssessssssseesseee
7.3 Appendix B: Sample IGL Competence Test...
7.4 Appendix C: Sample Horizon Achievement Certificate
File ref: Evaluation.doc COMMERCIAL IN CONFIDENCE . Page 5 of 22
© Peritas Ltd
fo
POL00039648
POL00039648
ICL Pathway Training Evaluation Paper Ref: SU/TRN/OOS
Version: 2.0 ~ 08/7/99
Date:
1. Introduction
This document defines the processes applicable to the Peritas activities undertaken to
discharge the contractual responsibilities for measurement of the effectiveness of the user
training programme in a manner consistent with the Kirkpatrick model.
2. Scope
This paper is intended to provide a definition of the evaluation process required to
satisfy Requirement 915, Evaluation of User Training.
Although User Awareness programmes will have an appraisal process, this is outside the
scope of this paper.
3. Principles of Training Evaluation
3.1 Introduction
Training evaluation is an essential process in developing and delivering effective
training programmes. It is vital that effective evaluation is built into the development
as well as the delivery stages of training projects, so that milestones are set and
achieved. In this way, the training products are assessed throughout their lifecycle
against a known set of criteria, based upon the goals of the programme itself. Should
these change, the products can change as well. Thus, the evaluation process becomes
the controlling element that ensures the products finally delivered into the training
cycle meet all the needs and goals set for them.
It is clear that evaluation must follow a systematic approach which is commonly
understood so that it encourages a partnership between the training function and the
managers and staff to be developed. The advantage of obtaining commitment from the
management at the start of the programme is that the overall needs are clearly
identified and joint ownership of the solution exists.
Within the framework of such a partnership it is possible to document an evaluation
strategy. This strategy enables the development of tactics and measurements by which
to judge the effectiveness of the training itself.
File ref: Evaluation.doc COMMERCIAL IN CONFIDENCE Page 6 of 22
© Peritas Ltd
POL00039648
POL00039648
mae
ICL Pathway Training Evaluation Paper Refi SU/TRN/O0S
Version: 2.0 08/7/99
Date:
3.2 The Four Level Approach to Evaluation
To ensure that the training delivered is effective, both Peritas and POCL apply a Four
Level Approach based closely on the well-known Kirkpatrick Model. Levels One to
Three are those most commonly used; Level Four, which complements the
methodology, is used less often.
The first three levels of the approach can be encapsulated within these questions:
Level One— Reaction
“Were the participants pleased with the training?”
“Did it meet their expectations?”
Level Two — Learning
“What did the participants learn?”
Level Three — Application
“Were the participants able to apply what they learnt during training in the work
place?”
One can see from the questions that levels 1 & 2 may be measured.
3.2.1 Level 1
Level One looks at the reaction of the participants and what they thought of the
training. This includes tutors, materials, facilities, methodology, content and similar
factors. Initial receptivity provides a good atmosphere for learning, but does not mean
high levels of learning will automatically occur.
3.2.2 Level 2
File ref: Evaluation.doc COMMERCIAL IN CONFIDENCE Page 7 of 22
© Peritas Ltd .
POL00039648
POL00039648
ICL Pathway Training Evaluation Paper Refi SU/TRN/005
Version: 2.0 08/7/99
Date:
Level Two concems learning or knowledge gain. It measures the retention of
principles, facts, techniques and skills presented in the training. The measures must
be objective and quantifiable to be good indicators of what knowledge has been learnt.
3.2.3 Level 3
Level Three concerns measurement of the extent of application of knowledge. This
level is progressively harder to measure than the previous levels as there are many
factors which can affect on-the-job performance. Techniques include before and after
comparisons, observations by superiors and subordinates, statistical comparisons and
long range follow-ups.
3.2.4 Level Four Evaluation
Level Four evaluation attempts to measure the impact of training on overall business
performance. Typical success measures are reduced costs or increased output. This
would mean objective collection of detailed data before and after training to provide
benchmarks against which to analyse the changes. Many other variables exist which
can affect results, so attempts are made to isolate these.
Level Four evaluation is extremely complex and very difficult to measure in practical
terms, due to the many variables which can affect the enterprise during the period in
which a learning programme is being evaluated.
4. Contractual Obligations
4.1 Venue Assessment
Peritas are required to obtain an assessment of whether each venue used for user
training is satisfactory for that purpose. ICL Pathway have a contractual threshold
level which requires 85% of all events to be delivered within a satisfactory venue.
(See Appendix Al)
4.2 Training Assessment
Peritas are required to provide a procedure for measuring the effectiveness of the
training programme at levels 1 and 2 of the Kirkpatrick model. Both these measures
File ref: Evaluation.doc COMMERCIAL IN CONFIDENCE Page 8 of 22
© Peritas Ltd
POL00039648
POL00039648
ICL Pathway Training Evaluation Paper Refi SU/TRN/00S
Version: 2.0 08/7/99
Date:
are to be taken when the delegate attends any user training event. Level 1 assessment
occurring when the training phase of the event is completed, subsequently a
competence test is used to secure the level 2 evaluation. (See Appendix A2)
The higher levels of the model, i.e. levels 3 and 4 are addressed by the “Management
of Change” programmes undertaken by ICL Pathway and POCL.
-5. Proposed Procedure
5.1 Timing of Measurement
On completion of each training event each delegate is required to provide an
assessment of the event in a manner consistent with level 1 of the Kirkpatrick model.
Such a process is consistent with the traditional end of course appraisal process used
in most commercial training events.
Following the course appraisal and review, each delegate is required to demonstrate
operational competence in a manner consistent with level 2 of the Kirkpatrick model.
Such a procedure requires the individual to complete a number of operational
processes both accurately and within agreed time constraints.
5.2 Course Appraisal
A course appraisal form will be the basis of the level one assessment and also the
venue assessment.
The appraisal form is completed by the trainee, subsequently the trainer’s comment
code is added to the document.
The form is a two sided document, side 1 is entered into the Peritas system (TPAS).
Should further analysis be required the reverse side of the form is used to capture the
detailed comments of each trainee.
5.3 Venue Assessment
Section 1 of the end of course appraisal will contain the venue assessment, the
delegate being requested to tick one of three statements which best describes the
File ref: Evaluation.doc COMMERCIAL IN CONFIDENCE Page 9 of 22
© Peritas Ltd
POL00039648
POL00039648
ICL Pathway Training Evaluation Paper Refi SU/TRN/00S
Version: 2.0 08/7/99
Date:
contribution of the venue to the overall quality of the training event (See Appendix
Al). The three descriptions proposed are:
1, The venue helped the learning process.
2. The venue made no difference to the ability to leam.
3. The venue made learning more difficult.
The training administration system is used to record the comments and aggregate the
number of satisfactory responses i.e. the summation of questions 1 and 2, and the
number of unsatisfactory responses, i.e. question 3, selected.
The venue assessment is complemented by the trainer’s perception of the individual
attributes of the venue,
The attributes assessed are:
e Ease of access
e Extraneous Noise
© Comfort
« HASAW Conformance
¢ Refreshments
© General Decor
© Overall Venue Assessment
5.4 Level 1 Assessment
On completion of each training event each delegate is required to provide an
assessment of the event in a manner consistent with level 1 of the Kirkpatrick model.
Such a process is consistent with the traditional end of course appraisal process used
in most commercial training events.
Following the course appraisal and review, each delegate is required to demonstrate
operational competence in a manner consistent with level 2 of the Kirkpatrick model.
Such a procedure requires the individual to complete a number of operational
processes both accurately and within agree time constraints.
File ref: Evaluation.doc COMMERCIAL IN CONFIDENCE Page 10 of 22
© Peritas Ltd
POL00039648
POL00039648
ICL Pathway Training Evaluation Paper Refi SU/TRN/O0S
Version: 2.0 08/7/99
Date:
The appraisal requests the delegate to comment on the 5 key attributes of the training
event which are:
1, Achievement of course objectives,
2. Quality of course presentation.
3. Skill and knowledge of trainer.
4. Relevance of the course workbook.
5. Overall level of satisfaction.
The contractual requirement is based on a target percentage of the delegates regarding
the event as “satisfactory”, Rather than rely on secondary translation to determine
whether the event is satisfactory the contractual term is used within the appraisal.
For each attribute there are 4 possible ratings:
1, Excellent
2. Good
3. Satisfactory
4. Unsatisfactory
The rationale which results in the additional positive rating in this case “Excellent”
arises from the findings of the majority of research studies in that delegates have a
basic aversion to marking the highest positive category and therefore the need for
balance is achieved by inclusion of the extra category.
The subsequent processing of the appraisal form within TPAS, the training project
administration system, determines the “predominant perception” of each delegate as
follows:
The categories 1 to 4 are assigned a score in the range from 4 to 1 respectively, this
creates a maximum score of 20 for each delegate.
The “predominant perception” is calculated as follows:
Score <8 the course classification is unsatisfactory
File ref: Evaluation.doc COMMERCIAL IN CONFIDENCE Page 11 of 22
© Peritas Ltd
POL00039648
POL00039648
ICL Pathway Training Evaluation Paper Ref: SU/TRN/00S
Version: 2.0 08/7/99
Date:
7<score <13 the course classification is satisfactory
12 <score <18 the course classification is good
17< score <21 the course classification is excellent
The system aggregates the total number of each category in order to determine
performance against the contractual target on a cumulative and period basis, the
period being a calendar month.
The above banding ensures that the course is only considered as acceptable if the
majority of attributes are scored as satisfactory or better.
On the reverse side of the appraisal form detailed comments may be added concerning
the specific attributes of the event as measured on Page 1 of the appraisal form. In
addition, examples of the course appraisal are shown in Appendix A.
5.5 Level 2 Assessment
5.5.1 Measurement method
This assessment is based on a test of user competence which occurs on completion of
the training event (See Appendix B). The competence test content and critical
elements are based on the key competencies defined for each job category by POCL.
The categories identified are:
Trainer
Auditor
Postmaster
Counter Assistant
The successful completion of the competence test results in the award of a “Horizon
Achievement Certificate” for the appropriate job category as listed above (See
Appendix C). In the event of failure the trainee is invited to attend a remedial training
event.
File ref: Evaluation.doc COMMERCIAL IN CONFIDENCE Page 12 of 22
© Peritas Ltd
POL00039648
POL00039648
7
ICL Pathway Training Evaluation Paper Refi SU/TRN/O0S
Version: 2.0 08/7/99
Date:
6. Remedial Training
6.1 Criteria for live operation
All training for essential staff must be completed with not more than four working
days remaining prior to the installation date for the office.
Additional staff, above the level of Minimum Training Compliance (MTC), are
targeted for training within the same window. However, should they be unavailable
then they must be trained not more than two working days after the office installation
date.
6.2 Remedial Training Provision
Remedial Training is provided in two forms:-
1, Where a critical area of competence fails to meet the required standard the delegate
is invited to attend a remedial training event.
2. Where multiple areas of competency fail to meet the required standard the delegate
is invited to another training event.
Remedial events are provided during the afternoon or evening and are of 2.5 hours
tuition duration plus competence test re-sit.
The delegate will be offered a place on the first available local event, which will be on
the same day or the following day. Subject to the constraint that the remedial activity
must be completed prior to installation, then further alternative dates may be offered.
6.3 Remedial Training Timetable
6.3.1 Types of Event
File ref: Evaluation.doc COMMERCIAL IN CONFIDENCE Page 13 of 22
© Peritas Ltd
POL00039648
POL00039648
ICL Pathway Training Evaluation Paper Refi SU/TRN/005
Version: 2.0 08/7/99
Date:
There are two variants of events ie. Manager’s remedial and Counter Assistant’s
remedial.
6.3.2 Scheduling
Manager’s remedial events are scheduled within each geographic area for the
Tuesday, Friday and Saturday (afternoon).
Counter Assistant’s events are scheduled within the same are for Monday, Wednesday
and Thursday of each week.
Typically, there will be up to 8 events on each day of the week covered by the
implementation programme.
6.3.3 Result notification
Competence test results for remedial events are known by 22.00 on any day. During
the period 22.00 to 23.00 the trainer will brief delegates on the outcome of their tests,
arrange any further remedial action should that be necessary and finally communicate
the results to the ICL Pathway Training Administration Centre (TAC).
The TAC processes the test results and transmits any changes to the status of the MTC
to the Roll Out Database (RoDB), in the event of a competence test failure which
relates to an office due for installation on the following working day, the information
is also transmitted by electronic mail to the appropriate implementation group
manager.
File ref: Evaluation.doc COMMERCIAL IN CONFIDENCE . Page 14 of 22
© Peritas Ltd
POL00039648
POL00039648
5]
ICL Pathway Training Evaluation Paper Ref: SU/TRN/00S
Version: 2.0 08/7/99
Date:
7. Appendices
7.1 Appendix A1: Delegate User Training Course Appraisal form
Venue Assessment
Please tick hichi . his trant
The venue helped the learning process
The venue made no difference
The venue made learning more difficult
OOO
Comments:
Training
Please tick one box for each question to indi he followi
Unsatisfactory 1 Satisfactory2 Good 3 Excellent 4
Achievement of course objectives oO oO oO oO
Comments:
Quality of Course presentation
Comments:
‘Skill and knowledge of the trainer
Comments:
Quality of course workbook [
Comments:
File ref: Evaluation.doc COMMERCIAL IN CONFIDENCE Page 15 of 22
© Peritas Ltd
POL00039648
POL00039648
ICL Pathway Training Evaluation Paper Ref: SU/TRN/00S
Version: 2.0 08/7/99
Date:
Your overall level of satisfaction
Comments:
Additional Comments
rs Are there any topics you would like to be added to the training session?
2 Are there any topics you would omit from the training session?
3. Was enough time allocated to each topic? Yost ] Not }
If your answer to the above question is no, which topic/s needed more time?
4 Did you have enough time to complete the exercises? Yes[ ] Nof J]
If your answer to the above question is no, which exercises needed more time?
5. Was there enough time allocated to role play? Yes ] Nof }
6 How confident do you feel in using the Horizon System?
Any further comments :
Delegate’s name: FAD code:
(please print)
Delegate's signature:
File ref: Evaluation.doc COMMERGIAL IN CONFIDENCE Page 16 of 22
© Peritas Ltd
POL00039648
POL00039648
I
ICL Pathway Training Evaluation Paper Refi SU/TRN/00S
Version: 2.0 08/7/99
I Date:
Trainer's signature: Comment Code
Date:
I
I File ref: Evaluation.doc COMMERCIAL IN CONFIDENCE Page 17 of 22
I © Peritas Ltd
POL00039648
POL00039648
ICL Pathway Training Evaluation Paper Refi SU/TRN/O0S
Version: 2.0 08/7/99
Date:
7.2 Appendix A.2 Trainer Course Appraisal (User Training)
Course Summary
Event Detait ‘Summary of the Statisties
of Achievement
[Trainer Name
Venue Name ‘Number expected
[Room Name ‘Number attended
[Start Time Number passed
Date Number remedials
Event 1D
Venue Assessment
‘Attribute Unacceptable 3 ‘Acceptable2 Goods
Ease of access
Extrancous noise
Comfort
HASAW conformance
Refreshments
General decor
‘Overall Venue assessment
Ifunacceptable, please comment
This section should be completed at the end of each event. Detail any incidents/opinions which may have influenced the
answers given on each delegate appraisal.
Questions
Details
‘Attainment of course objectives
‘Quality of course presentation
‘Skills and knowledge of the trais
iner
‘Quality ifcourse work book
Relevance of the training to your job
‘Confidence in using the Horizon system
‘Overall level of satisfaction
Trainer Comments
File ref: Evaluation.doc
© Peritas Ltd
COMMERCIAL IN CONFIDENCE
Page 18 of 22
POL00039648
POL00039648
ICL Pathway Training Evaluation Paper Refi SU/TRN/00S
Version: 2.0 08/7/99
Date:
Training Equipment Problems
Equipment Type ‘Serial Number Crate Number Urgent Repair Non Urgent Repair
Entries into this table will prompt a call from Logistics during the next working day
Please continue on a separate shect Ifnecessary.
File ref: Evaluation.doc
© Peritas Ltd
COMMERCIAL IN CONFIDENCE
Page 19 of 22
POL00039648
POL00039648
ICL Pathway Training Evaluation Paper Refi SU/TRN/O0S
Version: 2.0 08/7/99
Date:
7.3 Appendix B: Sample IGL Competence Test
Ask the delegate to complete the five practical exercises notified from the selection below:-
Start Time: End Time:
1 A customer requires 10 1* class stamps, 3 2 class stamps and 2 Postal Orders,1 for £5 and 1 for £12.75, the customer fs paying by
cash.
What Is the value of the transaction ?
‘The customer tenders £40, what change Is duc?
‘The customer also requires a receipt. Print and attach.
2. A customer requires to cash a Green Giro valued at £97.50, the customer also requires to purchase a £5 BT stamp,2 £1 TV stamps
and £5 of Quantum Gas.
‘What is the value due to the customer ?.
3.A customer requires to pay a BT bill. The bill has a value of £57.87. The customer pays £20 in BT stamps and the £37.87 In cash.
The customer when asked for the balance suddenly find he does not have enough cash and decides not to pay the bill, Now
vold/reverse the transaction?
4. A customer wants to make a payment Into hls NS account by warrant of £75. Complete the transaction.
5. You need to transfer Into your stock unit 100 1* class stamps and £500 cash. Complete the transaction on your counter system.
6. You need to transfer to supply division 100 2™ class stamps (surplus duc to increase in price). Complete the transaction.
7. A customer requires a renewal of a 10 year passport with SO pages and wants to Include a child, also the customer requires to
send a large parcel to Hong Kong. Complete the transaction, the customer Is paylng by cheque.
8. A customer wants to cash a National Westminster Banks cheque at your Post Office valued at £128, complete the transaction.
How much is due to the customer?
Is commission automatically deducted from the balance due to the customer?
9. A customer requires to pay an electricity bill of £67.20, the customer Is paying by cheque and £10 In stamps. Complete the
transaction.
10. A customer has Just received a Pickup Notice to collect her new benefit card.
Go through the procedures to activate the card for payment.
File ref: Evaluation.doc COMMERCIAL IN CONFIDENCE Page 20 of 22
© Peritas Ltd
POL00039648
POL00039648
ICL Pathway Training Evaluation Paper Refi SU/TRN/00S
Version: 2.0 08/7/99
Date:
Pass Marginal Fall Fall
(Support Required) (Extra Training Required)
Trainers Signature:
Delegates Signature:
File ref: Evaluation.doc
© Peritas Ltd
COMMERCIAL IN CONFIDENCE
Page 21 of 22
POL00039648
POL00039648
ICL Pathway Training Evaluation Paper Refi SU/TRN/O0S
Version: 2.0 08/7/99
Date:
7.4 Appendix C: Sample Horizon Achievement Certificate
To be approved
File ref: Evaluation.doc COMMERCIAL IN CONFIDENCE Page 22 of 22
© Peritas Ltd