POL00002192
POL00002192
HNG -> EBT risks and issues
Meeting: 20/06/2013
Invitee: Peter R Laycock
Willie Hughes
Antonio Jamasb
lan Trundell
Apologies Ghulam Hussain
Agenda and Outcomes:
1. JP Morgan, via HP has reported for some time now(well over one year) the issues of POca
transaction drops followed by peaks of up to 350TPS.
This is happening every day, but is more noticeable on heavy POca days (Monday and Tuesday).
The contracted number taken from the TIS is 180TPS with the system tested to 240TPS, but this is
being regularly burst
Risk: (Commercial; Operational; Customer Experience)
Could impact JP Morgan’s ability to satisfy the SLA requirement and worst case impact POca
counter transactions and lead to transaction queuing and possible timeouts
Embedded doc provides original stats from HP/JPM and dialogue
je!
latest stats from HP
at 170613 showing PC
Embedded doc provides stats from Fujitsu but averaged over 5-minutes
capoSmmin.xisx
Note: HP/JPM has monitored at their side of the fence and have stated that the problem is not with
EBT. Moreover we have discounted the cable and wireless link that was experiencing issues, but
monitoring after the fix identified that the peaks and troughs still exist
The missing piece is that traffic monitoring from HNG has not been performed by Fujitsu and they
delivered only stats averaged over 5-minutes so not possible to compare like for like
It must be borne in mind that there is a mismatch between the HP/JPM service targets and those of
Fujitsu, but equally the Transaction Interface Specification, which details the appropriate numbers
should have been agreed by all during the POca 2 project which went live in 2010. This document is
stored in the POca Infosec SharePoint site under “Peter Laycock batons”. Dave and lan have access
POSSIBLE CAUSES
F/1089/1
POL00002192
POL00002192
Although nothing definite, due to the obvious nature of the problem and a noticeable a frequency of
4-minutes, or multiples thereof, discussions considered possible causes in the HNG space. These are:
a) The Oracle database
b) Network Persistent Store
ACTIONS:
2.
a) Tony to raise a problem record with Fujitsu
b) Tony to ask Capacity teams to check if other products may be experiencing similar issues —
especially with a cycle of 4-minutes, or multiples thereof
c) Tony to allocate this to the Problem Management Team
d) Peter to ask HP to document what they believe the actual risk to both service levels and
customer could be
e) Fujitsu to investigate. Note: it is true that if this just happens to be a mismatch of service
targets, albeit in the agreed TIS that a cost would be incurred for any change, but if this is a
defect in the NPS, Oracle database or elsewhere in the HNG space and requires a fix, cost
should be zero
HNG > EBT “rehomes”
This is expected at 02:00hrs each day but EBT has identified this happening at random times
Risk: (Commercial; Operational; Customer Experience)
If this happens during trading hours EBT will be managing these reconnections which could impact
on POca counter transactions and lead to transaction queuing and possible timeouts
ACTIONS:
3.
a) Tony to raise a problem record with Fujitsu
b) Peter was to ask HP for details going back 6-months, but HP has made numerous incident
reports for some considerable time to the Duty Manager on this. Mark Geldart (HP) will
provide problem record numbers for analysis by POL Service Management.
c) Tony to allocate this to the Problem Management Team
d) Results to be provided to Fujitsu
e) Fujitsu to provide a rationale, and hopefully a proposed fix
Horizon terminal time offsets:
“Summary: JP Morgan are reporting that out of a sample of 865,731 transactions from 10,643 FAD
codes, 1,640 FAD codes had at least one terminal with “its time out by more than 1 minute (+1.0
F/1089/2
POL00002192
POL00002192
second). And of those, 36 FAD codes had at least one terminal with its time out by more than 5
minutes, with the worst one being out by nearly 21 minutes.
This indicates that clock synchronisation at the counter is not always what it should be. Of course for
banking transactions, the time stamp on the receipt is important in relation to disputes, but if this
differs considerably to the time of the transaction recorded by the bank could complicate any
dispute”
Embedded doc provides original stats and dialogue
"2
POca - transaction
time offsets. pdf
Discussion took place on this and there was debate on the actual time stamping of transactions, but
it should be borne in mind that the time on the receipt is provided by the terminal clock
See above embedded document but important to note that HP/JPM are receiving the 1S08583
message at EBT, generated by HNG NBX, which is where they are measuring the discrepancy
Risk: (Regulatory; Customer Experience; Fraud; Transaction Disputes)
It should be noted that for POca and other banking transactions disputes do occur on a regular basis
and must follow the Financial Ombudsman Service criteria for resolution, so discrepancies in the
time on the receipt to the actual time of the transaction could have an impact. Moreover, where
fraud is concerned, data is required for evidential purposes so accuracy is paramount
ACTIONS:
a) Tony to raise a problem record with Fujitsu
b) Peter has asked HP for details more detail; see below
c) Tony to allocate this to the Problem Management Team
d) Results to be provided to Fujitsu
e) Fujitsu to provide a rationale, and hopefully a proposed fix
Email from Mark Geldart in response to Peter laycock’s questions (21/06/13)
Q1—how does the counter time get sent to EBT —is this part of the ISO8583 message?
It is part of the iSO 8583 message.
Q2 —is the counter time “used” for transaction processing or is that only between HNG and EBT (i.e.
The EBT transaction processing SLA is to return success of failure to HNG and what happens beyond
IRE11 is not “our” concern)
F/1089/3
POL00002192
POL00002192
We store the counter time and the switch time from the ISO 8583 message and add our own
timestamp. The counter time is what is displayed when transactions are displayed on a screen or
returned to CRM. This is done as these enquires would usually be done in response to dealing with a
customer, and it’s the terminal time that is printed on the customer receipt. However we do
display/retura the transactions in our internal timestamp order.
Q3 — if the times all stayed offset by 20 minutes or more, would it actually matter (other than a
customer having a receipt with potentially the incorrect transaction processing time on it?
It doesn’t matter to EBT. However, if a customer rapidly visited two branches and performed
transactions it could lead to a customer attempting @ fraud. Le. A customer performs @ balance
enquiry in a +30 minute branch showing £200. They dash to a nearby -30 minutes branch and
withdraw the £200. They are left with two receipts that show o withdrawal of £200 (and a batance of
zero} and an “hour” iater a receipt showing @ balance of £200. This should be caught if they tried to
claim the £200 as having gone missing, but there's always a possibility that it woufdn’t. There are
probably other fraudulent / criminal activities that could use an incorrectly timed receipt to their
advantage. But I'll leave those to your own imagination.
Peter Reece Laycock I Information Security Consultant
Admin block, Leeds Mail Centre, Leodis House, Leodis Way, Leeds, LS10 1AZ
F/1089/4