POL00033398
POL00033398
Assurance Review: Quality of Auditing
February 2011
WELCOME
Martin Ferlinc
Risk & Compliance Team
Contents & Distribution
POL00033398
POL00033398
Contents
Introduction & Background
Headline Findings
Detailed Findings
Recommendations
Appendix A- Audit Activity Period 1-9
Appendix B - Supporting Documents
ae
5-16
47
18
19
Distribution List
For Action (All working in Network Support):
Sue Richardson — Project & Standards Manager
Sandra Lewis - Field Support Change Advisor
Regional Network Managers
Network Support Team Leaders
For Information:
Sue Huggins — Head of Network Development
Paul Meadows — Head of Risk & Compliance
Angela Van-Den-Bogerd — Head of Network Services
Nigel Viles — Enterprise Risk & Assurance Manager
Introduction & Background
POL00033398
POL00033398
Introduction
The purpose of this report is to document the findings, conclusions and recommendations in respect of an annual review that
sought to independently assure the quality of branch ausiting within Post Office.
Background
The Network Support Team is responsible for the delivery of the Annual Audit Plan, which covers financial and compliance
audits of Post Office® branches and cash centres.
The reporting line of personnel performing auditing activities moved to the Network at the beginning of 2009/10, as part of a
move towards developing a pool of multi-skilled resource that was able to perform audit, training and other intervention
activities
The Business, clients (e.g. NS&I & DVLA) and other stakeholders (e.g. Bank of Ireland and POFTS) rely on the outputs of
auditing activities either as part of their own monitoring or as a means of assurance. Given concerns raised by external
stakeholders about the independence of Post Office's auciting activity (ie. the teams performing audits of the branches in the
Network report within the same directorate as the management structure of those branches), an annual assurance activity is
performed by the Post Office Risk & Compliance Team in order to provided a level of independent assurance that branch
auditing activities are being undertaken in accordance with laid down procedures and, more broadly, that audits are being
conducted to expected internal auditing standards.
The key risk exposure to Post Office Limited includes stakeholders (internal and external) not being assured by audit findings
and branch staff being incorrectly advised of correct procedures.
Headline Findings
The key positive findings from this review are:
The quantity of audit activity performed to date is in excess of that planned (as of Period 9)
There is a generally good standard of audit papersifiles retained for financial audit activity
Quality Assurance Reviews (QARs) have been introduced since the last assurance review
Field Support Team conduct audits in a professional manner
Some good examples of scenario based compliance questioning techniques in operation
An Audit Process Manual exists for all auciting process, with evidence of most chapters being maintained
There has been progress since the last review, with attention given to the majority of previous issues identified
The key findings from this review that require further review or attention relate to the following issues-
Branches selected randomly for audits are not done on the basis of random sampling methods
Team members seem to lack awareness of how the Branch Profile works, and therefore, why they are auditing a branch
* Inconsistent use of version controlling for the Audit Process Manual, with the annual review of one chapter overdue
Incomplete documentation retained for compliance reviews, including file housekeeping
* Accuracy of inputs to SharePoint (21% of all SharePoint surveys contain errors)
Frequency/Quantity of quality assurance reviews (QARs)
No QAR levelling performed in the last 12 months
Length of time branches are kept closed during financial audits, impacting on Post Office customers
Verification of financial discrepancies with subpostmaster or officer in charge
Positioning of the Compliance Audit and inconsistent use of formal compliance audit questioning techniques
+ Communication of audit findings (close of meeting)
POL00033398
POL00033398
Detailed Findings — Audit Plan, Scheduling & Planning
POL00033398
POL00033398
Audit Plan
« The Network Audit Plan was presented to Post Office's Risk & Compliance Committee (R&CC) in April 2010, by Lynn Hobbs.
Lynn had committed to presenting a review of the plan to the committee in August (given the impact of rolling out Horizon
Online on auditing activities) but this was never done. However, achievement against the plan is reported to the R&CC ona
monthly basis and, therefore, the committee has the opportunity to discuss at each meeting.
+ Areview of audit activity delivered by the end of Period 9 (shown at Appendix A) revealed that, overall, audit activity is in
excess of that planned to date and is on target for delivery by the end of the year. The only exceptions to this are in respect of
cash centre audits (only 1 of 11 had been completed by the end of December) and T&D aucit activity at crown offices, where
71 activities had been completed (against 125 planned). The shortfall in T&D audits is considered to be outside of the control
of Network Support, primarily as a result of delays in the migration of financial specialists and the associated revised T&D
scheme. Overall, given the use of resource to support Horizon Online roll out during the year, it is @ tremendous achievement
to be on track to Celver in excess of that planned.
Scheduling & Planning
* Scheduling of aucit activity is undertaken centrally (by Lee Heil). It appeared, from discussion with Lee that there is a lack of
understancing and purpose of random audits or true random sampling principles and, as such, random audit selections have
been done from a biased sample (e.g. taking into account geographic considerations), sample rather than using true random
sampling methods.
* Discussions with field support team member at aucit reinforced a lack of understanding of random audits and, more
importantly, the rationale for audit selection, Although most team members at audit attended were aware of branch profile,
there was a lack of awareness of how the profile worked or, for the audits being performed, what aspect of the profile had
prompted the audit. Greater awareness would improve preparation for audit assignments
POL00033398
POL00033398
Detailed Findings — Audit Process
The Audit Process Manual (Volume 4) was examined to confirm that it is fit for purpose. There was evidence that many of the
chapters were subject to annual reviews and, where appropriate, interim reviews, to reflect any operational changes that
impact on the audit process. The findings of each chapter are detailed as follows —
Chapter 1: Audit Pian & Scheduling: Current Version: 8.0
Last Annual Review: January 2610: Next Scheduled Review: January 2611
Author: Alan Stuart
Comments: Annual reviews completed. Correct version control used,
Chapter 2a: Working Papers
A full set of working papers provided evidence of a recent review (dated February 2011)
Chapter 3: Performing a Branch Audit: Current Version: 5.4
Last Annual Review: Aprii 2010: Next Scheduled Review: March 2047
Author: Dave Ogleby; Peter Jackson; Linda McLaughlin
Comments: Annual reviews completed. Correct version control used.
Chapter 4: Transfers & Conversions: Current Version: 9.4
Last Annual Review: April 2010: Next Scheduled Review: April 2011
Author: Rita Kendellen
Comments: Annual reviews completed. Correct version contro! used
Detailed Findings — Audit Process
POL00033398
POL00033398
Chapter 5: Closures: Current Version: 10.6
Last Annual Review: Unclear: Next Scheduled Review: May 2011
Author: Peter Jackson
Comments: Unclear if annual reviews are completed as these are not clearly documented and incorrect version contro! used.
Chapter 6: Robbery & Burglary: Current Version: 8.0
Last Annual Review: Unclear: Next Scheduled Review: June 2011
Author: David Patrick
Comments: Unclear if annual reviews are completed as these are not clearly documented and incorrect version control used
Chapter 7: Performing a Cash Centre Audit: Current Version: 2.1
Last Annual Review: July 2009: Next Scheduled Review: July 2010
Author: Bob Collins; Chris Fayers
Comments: No evidence of annual review due in July 2010 having been completed
Chapter 9: Retention of Audit Papers: Current Version: 5.2
Last Annual Review: Unclear: Next Scheduled Review: September 2011
Author: Frank Martin
Comments: Unciear if annual reviews are completed as these are not clearly documented and incorrect version control used.
Detailed Findings — Audit Process
POL00033398
POL00033398
Chapter 11: Quality Assurance: Current Version: 8.1
Last Annual Review: Unclear: Next Scheduled Review: November 2011
Author: Paul Humber
Comments: Unciear if annual reviews are completed as these are not clearly documented and incorrect version contro! used,
Chapter 12: Continuity Planning: Current Version: 6.0
Last Reviewed: January 2011: Next Scheduled Review: December 2011
Author: Julia Mann
Comments: Annual reviews compieted. Correct version control used
Although it was clear that most chapters were well maintained, the version numbers of the chapters were found to be
inconsistent making it in some instances difficult to determine if annual reviews had been completed. An example is Chapter 5
Closures, where the first recorded version in August 2009 was 9.0 which is the assumed annual review date. A number of
interim amendments were made between September 2009 — March 2010, the version numbers of which were correctly
documented (the last of which being V9.6). Numerous changes were subsequently made in May 2010 under V10.1 and it is
assumed that this was the annual review although not identified as such
Another example is Chapter 6 Robbery & Burglary Audits which starts as V7 in July 2009. Between this date and January 2011,
ten amendments were made with the current version being V8. Assuming June is the annual review date, no annual review
was undertaken in June 2010 and version numbers have continued to increase 7.1;7.2; 7.3 ete without being renumbered 8.0
following the annual review.
Detailed Findings — Standard of Audit Documentation
POL00033398
POL00033398
A sample of twenty P32 Financial and Compliance audits, which had been carried out in November and December 2010,
were randomly selected for review. All associated paperwork was requested from the lead auditors for examination. Of these
requests, only three had not been received at the time of writing this report due to ~
* Lead auditor on leave until 18" February 2071 (with papers discovered to be with the line manager, who is now on leave)
* Working papers destroyed early in error
* One not yet received after posting on the 2r¢ February 2011
Of the twenty requests, a further five had had the paperwork destroyed as the request was mace afier the 60 day retention
period, so the QAR could only be carried out on the electronic files provided.
Completion of P32 financial aucits was found to be generally of a good standard with planning, on site and post audit activity
fully documented. Most failures (summarised below) were around reporting and the printing and retention of relevant
documents within the file
P32 Financial Audits - Summarised Findings
* Copies of reports were not in the file (38%)
P32 incorrectly named (5%)
Reports not in zip file on Lotus Notes Library (10%)
Reports - Grammatical errors (15%), formatting (15%) and errors relating to registered staff at the branch (10%)
No record that Cash Management had been contacted for code 100 audits (25%)
Detailed Findings — Standard of Audit Documentation
POL00033398
POL00033398
A greater number of errors were found in the compliance aucits (summarised below}. Some of these can be attributed to
careless/human error whereas some would suggest lack of clear understanding of what is required. With the current working
papers there is no provision for sample sizes or conclusions and it is therefore impossible to determine if minimum auditing
samples of 1 or 50% (whichever is the greatest) is adhered to. Some compliance working papers were found to contain
minimal input (ust questions with control gaps endorsed with ‘1') with little or no supporting narrative. Some instances were
found where the compliance questions were input direct to the laptop.
CATs ~ Summarised Findings
* CAT reporting too! not in zip file on Lotus Notes Library (10%)
© Copy of Appendix A&B (Action Plan) and Compliance Certificate not in file (50%)
Full electronic documentation not in zip file on Lotus Notes library (15%)
Cell G17 (Previous aucit findings) in planning tab not completed yes/no (82%)
Previous control gaps not recorded/left blank with no explanation (30%
Oniy controi gaps identified on working papers therefore unable to confirm if ali questions asked (44%)
Control gaps incorrectly reported (12%)
Compliance Certificate incorrectly formatted (12%)
10
Detailed Findings — Quality Assurance Process
Significant progress has been made regarding the completion of QARs since the previous Quality of Auditing Review
completed in 2009 where it was found that QAR activity was not undertaken.
The minimum number of QARs that are due to be completed, as defined in Chapter 11 of the Audit Process Manual, is six
reviews per direct report, per appraisal year. Based on the current staffing levels of 120 Field Advisors, there is an expectation
of a minimum of 720 QARs being completed in 2010-11.The figures below highlight the number of QARs performed, against
that expected as of Period 9 (allowing for the fact that audit activity did not take place for around three months during the roll
out of Horizon Online).
P32 Financial Audit QARs Due to be Completed P1- P9: 360: Actual: 203
Compliance QARs Due to be Completed P1- P9: 360 Actual: 184
Observation Audits Due to be Completed P1- P9: 120 Actual: 57
Clearly the figures above are based on minimum requirements and do not take into account that QARs are due to be
compieted on each direct report monthly and would only moved to bi monthly if the score met the minimum standard of 95%
Field Support Managers should also attend an audit led by their Field Support Team Leaders at least once in a twelve-month
period. Evidence could only be found of one observational audit which was completed in November 2010 on Chris Gilding and
a P32/CAT QAR completed on Rita Kendellen in April 2010,
POL00033398
POL00033398
Detailed Findings — Quality Assurance Process
POL00033398
POL00033398
During the period April ~ December 2010 a total of 204 Quality Assurance Reviews (QARs) were completed on P32s,185
on Compiiance Audit Tests (CATs) and 58 audit observations. Results are detailed in the table below which are split into
the three audit team areas,
Gara Herth [Average I Total ‘Average
E 204 7
5 ve I 186 Fa 6AE
Observation z ines [58 97418
A sample of ten P32 financial and compliance audits (one per field team leader) was selected which had been subject to
the QAR process during periods 4 - 9 in 2010/11. Two had not been received! at the time of writing this report. The aim was
to test the quality of the complete QARs to ensure that there was a consistency of marking and that all errors were
detailed. The following paragraphs summarise the findings:-
P32 Financial Audits:
The standard of completion was consistent across the field team leaders. There was a degree of consistency of marking
across the range of questions with themes identified mainly around standards of reports. It was noted however that, when
completing QARS, field team leaders request all paperwork and electronic files from the field support advisors. Whilst there
is no major issue with this process, it was found, when completing the sample of QARs, that a number of P32 files were
noi on the P32 library on Lotus Notes, indicating that the field support advisor may have overlooked transferrin:
completion of the audit.
Compliance Audits
It is apparent that there were inconsistencies with scoring across the range of QARs examined. The following common
themes were identified which were incorrectly marked —
Detailed Findings — Quality Assurance Process
POL00033398
POL00033398
+ Planning
Previous audit control gaps not recorded or eft blank with no explanation
Cell G17 not completed yes/no to confirm if compliance was tested at previous aucit.
- Working Papers
No provision for test population/sample sizes/periods on working papers, therefore - test population/sample sizes/periods not
annotated on Working Papers
No provision for conclusions and recommendations on the working papers, therefore - Conclusions and recommendations
not fully explained on the working papers
incomplete paperwork retained in files (Compliance working papers/Appendix A&B/Compliance Certificate)
Compliance questions input to laptops direct on audit
From the sample of QARs undertaken, as part of this review, it is evident that there is a degree of inconsistency of marking
across the field team leaders. No evidence was found that the levelling activity, as detailed in Chapter 11 of the Audit Process
Manual, is currently being undertaken to ensure a consistent approach across the teams.
SharePoint Surveys
Error rates for SharePoint surveys for Core CATs, Government Services and Procedural Security were examined for the
periods 1 ~ 9 in 2010/11, to determine the level of accuracy of inputting. Results are detailed in the following slide which
shows the average number of errors per survey type. The level of errors per survey completed is currently averaging around
21%. Input errors can impact on the accuracy of data provided to stakeholders and clients, and also results in considerable
time consuming data cleansing activity taking place each period to correct errors.
POL00033398
POL00033398
Detailed Findings — Quality Assurance Process
‘Number of Surveys
Number of Errots
Evror Rate.
Core CATS
Number of Questions in Survey I 60. ao ia Ea @
Number of Surveys 305 I 494 [217 a
‘Humber of Errors 93 34 ra EA 36
&. Exot Rate.
Humber of Surveys
Number of Errors
umber of Questions in Surveys 94 a Sa
Number of
Number of Errors
®. Ertot Rate.
POL00033398
POL00033398
Detailed Findings —- Observed Audit Practices
Four audits were attending during the course of the review, to observe the audit process being deployed. The four audits
selected included a crown office, an MSPO, a large SPSO and a small SPSO, The key findings are detailed below, against
the main themes of an audit
Planning/Preparation
In pre-audit phone calls (with the lead auditors), preparation guidance focused primarily on meeting point/meeting time
information. There was limited information provided on the reason for the audit or on the history of the branch (e.g. key
issues revealed at the last aucit). When on-site, it became apparent that, while most field support advisors were aware of the
branch profile, they did not have a great understanding of how the branch profile worked or what aspect of the branch profile
had triggered a reason to aucit the branch (and so tailor their own preparations). At the audit of the crown office, the AEI test
was allocated to a field support advisor who had never performing the test before and was not aware that he would be doing
this until the day. Not having undertaken the test before meant that that he minimal opportunity to familiarise himself with the
testing approach.
Financial Audit
At all the audits attended, there appeared to be a lack of urgency in getting the branch open or any concern expressed at the
impact that a closed branch had on customers. At one audit (where the branch was due to open at 8.45am and did not open
until 10.35am), the field support advisor was counting foreign currency (which could have been counted and agreed after the
branch had opened) before sterling currency had been agreed.
There were two instances at different audits where a minor discrepancy was revealed by the audit but the
subpostmaster/officer in charge was not invited or encouraged to agree this discrepancy. In one case, where the discrepancy
was challenged, the error was found to be an error by the field support advisor and, therefore, there was no discrepancy.
Detailed Findings —- Observed Audit Practices
POL00033398
POL00033398
Compliance Audit
At two of the audits attended, the compliance audit testing was introduced in a fairly apologetic manner (e.g. by stating, “?m
afraid that we now have to ask a number of compliance questions”), rather than taking the opportunity to emphasise the
importance of compliance for the Business and for customers.
There was some evidence of some team members adopting some good use of scenario based compliance questioning but
there appeared to be a limited use of the breadth of available compliance audit techniques on display, including corroborative
testing. Compliance questions are designed for the field support advisor to answer yet the tendency was to read out the
question (word for word) to the person being tested. This sometimes created leading questions being asked and there were
examples of providing unnecessary prompts (e.g. by stating, “I'll give you a clue, Marvin Gaye heard it through this’)
At one of the audits, not all the core questions were asked and the results of those that were, were input directly to the laptop
without including any comments and the subpostmaster did not have the opportunity to agree the fincings.
At the crown office attended, it was noted that only the branch manager was tested against the core compliance questions
i.e. with no attempt to confirm that ‘text book’ process answers given by the branch manager reflected practical deployment
in the branch and no counter clerks were tested, either to give a wider view of compliance in the branch or corroborate
answers given by the branch manager.
Communication
Informal rather than formal opening meetings were held at all audits attended. They did not always set out the plan for the
aucit (e.g. how long the audit would be expected to take, that the financial audit would not involve counting all the stock or
that a compliance review would be performed (and how). Closing meetings were used well to highlight and summarise
findings although it was not evident that comments (including mitigating remarks) made by the auditee were captured to be
included in the report. In one case, the findings were discussed with a counter clerk rather than the subpostmaster (even
though he was on site) and, in other case, the findings were ciecussed with a relief subposimaster. It was mentioned to the
relief that the report would be issued in the next couple of days (before the subpostmaster had returned from his holiday) and
there was no mention of plans to contact the subpostmaster before issuing the report. Indeed, the field support advisor did
not take the opportunity to speak to the subpostmaster who had called the branch on the day of the audit
Conduct
It was considered that field support team members conducted themselves professionally while on site and displayed a
pleasant manner throughout the audit.
Recommendations — Next Steps
POL00033398
POL00033398
Issue
Action
[Action Owner
Timescale
™
‘Audit Preparation -
Team awareness of
Branch Profile
Incorrect use of random
auditing sampling
methods
Audit Process Manual —
Version Control
Completion standard of
compliance audit
reporting
Deployment of QAR
process
Financial Auditing
(Impact of customers)
Compliance Ausiting
(Questioning
Techniques)
[Shaun & Alan asked to deliver a
lsession at TL WTLS
Conference call to discuss Pure
Random sampling between
Martin, Sue & Lee.
[Version issue now addressed
Re-communicate the standards in
regard to the completion of both
lelectronic and hard copy reports
and files.
Introduce 2 levelling sessions in
2011/12 to ensure consistency is
lembedded across the team — to
be led by The Project & Standards
Manager
Review & refresh Chapter 3 to.
provide clarity & consistency,
maintaining an awareness of
lcustomer impact on audit
Reposition how the FSA obtains
the answers to the question being
asked in an audit
Regional Managers (RNMs)
Sue Richardson
Sue Richardson/Sandra Lewis
IRNMs
IRNMs ,TL & Project &
Standards Manager
IRNM's & Team Leaders
IRNM's & Team Leaders
By end of G2 (2011/12)
March 2011
Current & on-going
By end of Q1 (2011/12)
31% March 2011
By end of Q1 (2011/12)
By end of Q1 (2011/12)
POL00033398
POL00033398
Appendix A - Network Support Audit Activity Period 9
ts Stock checks all ranches
ompliance tenting “Agency branche
oven Compliance texting
Fandom each stockchecks
Fallowap each eockand
ah Cant UTR Snow
co las yea).
had discrepancy of over £10k Total netdscrepancies ID £2.790519
POL00033398
POL00033398
Appendix B — Supporting Documents
QAR Sample Audit Sample QARs
Results