RESTRICTED - COMMERCIAL
Product Assurance - A Re-Think
1. Assurance vs Issues Management
Our role is assurance, but up to now we've had very little opportunity to do more
than manage issues as they come up. _ In the absence of Pathway’s co-operation on
assurance (across the board), and the ability to engage in proper review activities -
the V&V - we are not going to be able to able to provide the “assurance” that
sponsors expect.
We end up giving Pathway false credibility and doing ourselves no favours if we
give the impression of assurance. Pathway see little benefit in overall assurance,
they choose to involve us when it suits them but consistently fail to provide long
term access. If they refuse us access, what happens - do we have any leverage? In
effect we have no clout - what does “no assurance” mean in the end?
So either we have to achieve the coverage/depth that we require to provide a
credible level of assurance, or we need to stop pretending that we can do it, and
revert to “issues management” as our prime role.
Decent level of assurance is only going to come from high level (Dave Miller and
above) pressure on Pathway, as a condition of signing up to yet-another-plan, as a
risk-minimisation action - the little local agreement approach is doomed.
2. Assurance vs Product Management
Despite being called “Product Assurance”, we seem to have two distinct areas of
operations:
* assurance (of which we've not managed to be very effective yet)
© product management (in both business and fraud risk management areas) -
fettling release contents, availability dates - almost a development management
role, etc
Having assurance and product management puts together gamekeeper and poacher
- is this appropriate? We have products like EPOSS where the level of assurance is
still very low, despite fairly fruitless long term PM involvement (no criticism of PM).
If we separate we can be measure “assurance” without it being clouded by PM
activities.
Current arrangement of Product Management being joined with Business Assurance
and not the other Assurance areas gives uneven communication and timing - and
tisks gives impression that a product is “ok” when it’s not been looked at
“technically”.
Is PM carrying out User Assurance as well as Business Assurance?
WITNO5970118
WITN05970118
WITNO5970118
WITN05970118
RESTRICTED - COMMERCIAL,
3._Assurance Viewpoints vs Products
Up to now the three Assurance teams have had their own “products”, with very
little overlap - so PM have looked at the business products, Technical at the
infrastructure, Security at security functionality, with little crossover/interworking.
Technical have taken on certain aspects of “business” around interfaces and data,
but with slightly unclear responsibility between the teams.
Where there has been crossover it’s been awkward - eg access control has sat
between security and PM (EPOSS), reference data, although being a key driver for
EPOSS, has fallen between PM and TA.
In other areas there’s been no sensible cross over and problems has occurred as a
result - eg BES fallback where FSG (and since, accounting/ reconciliation) discover
serious gaps in functionality but business side haven’t flagged risk).
For FSG products (eg FRM), are they being Technically Assured?
CURRENT
whereas what we need to achieve is coverage of the entire service from each of the
viewpoints - from user, business, technical and security.
PROPOSED
(Infrastructure may not have user or business angles ~although usability of hardware, for instance, could be user angle)
Note this is far nearer the original PDA grid model of having product managers who
facilitate a product through the various “ gates” of user, business, security, etc - that a
product manager would have to get a “tick in the box” for their product.
We need to present a single “assurance view”, get away from the “we think it’s ok, it’s
just those buggers in <insert name here>” attitude - or at least the presentation of that
viewpoint to outside.
Need to break down team boundaries - either shape into “multi-disciplined product
teams” (and include a “product” which covers end-to-end), on some matrix model,
or have a single product manager per product who has to get their product through
each of the 3/4 assurance areas.
WITNO5970118
WITN05970118
RESTRICTED - COMMERCIAL
4. Applications
Our most legitimate area of concern, and the one of greatest risk (based on
experience to date), lies with the Business Applications. Currently these are
“owned” by product management (apart from FRMS), but they need to be assured
from a number of angles - eg integrity is a business issue but needs deep technical
understanding to flush out the issues (we know they will not be raised by Pathway).
We've had experience of the problems from not having visibility of the application
design, with the Rellc duplicate payment problems (which as a result only emerged
late in the day in testing and caused a significant hassle at all levels - not all of it well
informed) and more recently with EPOSS issues.
We need to assure the applications from all three (four) viewpoints, in a joined up
manner - it’s no good Tech and Sec being “after the event” and after PM have given
the product some form of ok. Separating PM from Business Assurance will help.
5. Operational Roles
FSG has an operational element - analysis of FRM outputs, some analysis of
suspected frauds; is this rightly part of assurance, or should it be elevated to a
“Fraud Management” role alongside Service Management.
However, we need greater visibility of the live service, to learn/build on the
experience of the “trials” - otherwise why have trials?
6. Assurance vs Testing
Is it really sensible for testing to be outwith assurance -why is it done if not for
assurance? It can be managed outwith assurance, as a “bought in” service, but it’s
not primarily being done for assurance - it’s got a life/ purpose of its own?
The “joint testing” (esp in technical areas, where the “injection of business
knowledge” isn’t valid) is effectively another form of unmanaged assurance, and
one which is getting visibility of Pathway in a way that we do not? Is it tenable to
have a TA team with little visibility of the design when we've got a team of testers
testing it!
7. Issues Management
Generally a weak area across the board, esp programme wide. Product Assurance
Issues were a start, but restricting to team leaders and running separately to
individual team issues means that we don’t have the benefits of single view and
communication.
WITNO5970118
WITN05970118
RESTRICTED - COMMERCIAL
We need to get to a scheme where someone shouldn't be working on an issue unless
it's logged - and shouldn't be working on something unless it’s an issue or an
otherwise recognised activity ona plan.
We need a means of giving advanced warning to RAB etc of the issues - and this
includes “lack of visibility”. The previous RABs have all had a presumption of
authorisation, and Pathway know it - and there is apparent use of issues as advance
warning of gating items on authorisation. In other words, we use “issues
management” as a means of sharing risks and concerns leading up to RAB (and
“national rollout” RAB, assuming there’s a concept) - rather than as a totally
separate process.
Re-instigate the “risk register” as a means of communicating risks/issues with
Pathway?
8. Locations/Management
We need to get a single, coherent culture across the Product Assurance team, and get
away froma London/Feltham culture. Need far better communication between
the two sites - from mixing, from holding team events, whatever.
Suggestions:
¢ Move entire assurance team to Feltham, on the grounds we are assuring Pathway
and it’s managed from Feltham (albeit with work on different sites). Probably the
best solution but maybe impractical on space grounds
« Have all 3/4 teams represented (and not just one person) on each site - maybe
need to move some back to TH to fit, if so look at which people have least contact
- or need for contact - with Pathway’s Feltham staff. Ensure there is Business
representation at TH, and Technical/Security at Feltham. If there are other
sensible sites, think about putting people there...
BUT do not fall into mistake of putting Technical at Bracknell, this falls back into
the Infrastructure vs Business trap!