FUJ00118171
FUJ00118171
Post-Acceptance TIP Incident Status Report —18/11/99
TIP INCIDENT DESCRIPTIONS
TIP CA outs
reference Details
PinICL
HSH call
reference
932 25 1INo Primary Mappings provided in Reference Data for Southend Bus/Rail
Tickets.
9909210308 29966 IIt was discovered on 23rd September that there were no primary mappings
30043 Ipresent in reference data for these products which had been on the system for
Jany months (from MOT). Being local products, the absence had not affected
isystem operation until roll-out caused outlets selling those products to come on-
line. Absence of primary mappings causes products to fail to come to account
lat the outlet. A detailed check of all product reference data revealed a total of
39 for which mappings were missing. The missing reference data was provided)
lon_24'" September.
934 26 85IReference Data Error on product 196.
(On September 7th POCL sent a large number (c 550) of ‘no-change” Ref Data
9909300551 changes which were sent to POCL OSG on 16" September and were sent
down to counters on the 227. One of the items updated by these changes
was item 196 (Giro Change Giving), and this had the effect of bringing into use
lan accounting sense change sent by POCL on June 15" as a change to Item
Transaction Mode zero. [As no regression testing was performed by either
IPOCL or Pathway the problem was not noticed until the changes went live on
22" September.]
The 15" June change had been held up by Pathway with a query as to the
lreason for the change, and a request for a corresponding change to product
197. The effect of the change to product196 Session Effect, without the
jolanned corresponding changes to product 197 was to introduce inconsistency
jbetween products 196 and 197 when transacted by Horizon offices, resulting in
the transaction volumes of the two products being netted instead of added on
line 9015 of the Cash Account.
When first reported the issue was ascribed to TIP having actioned the related
change to product 197 issued by POCL RDP on the 23% September. This
\delayed diagnosis of the real cause.
There was no fault in the Horizon software. The incident was caused by the
application of POCL reference data which was inconsistent. The transaction
stream sent to TIP was consistent with the cash accounts produced in the
joutlets but, as TIP construct their cash account without reference to the Mode
\Zero accounting sense, they have being reporting a difference on line 9015
since the change in Reference Data.
Reference Data to cause consistent accounting between products 196 and 197
lwas delivered on Wednesday 20" October, along with an associated change to
ithe MiECCO migration agent to maintain consistency of the sign of migrated
ltransactions.
Printed 18:03 18/11/99
Page 1 of 10
FUJ00118171
FUJ00118171
Post-Acceptance TIP Incident Status Report -18/11/99
935 127 1\Negative sign on txn numbers. This was caused by an error in the migration
lagent which only shows up when there is an ECCO disc failure which fails to
9909290396 I 30493 Imigrate or there is a comms failure between outlet and data centre during
\migration. After rejection of the file by TIP action was taken to remove the -ve
signs from the numbers. However the numbers had previously been truncated
Ito a fixed length, and the presence of a leading negative sign had caused loss
lof the final digit. The numbers generated by removing the -ve signs were in
some cases, because of loss of the last digit, duplicate numbers, which lead to
ithe erroneous conclusion that there were duplicate transactions in the file re-
sent to TIP (and rejected by TIP as having duplicate records, see 937). These
transactions were deleted from the file and the file re-re-sent.
The effect of this has been that 8 migrated AP transactions, for one outlet, were
not been sent electronically to TIP, and have subsequently been sent via a RED)
report.
The agent bug is now understood and will be fixed. It cannot be fixed before
\Nov 5", when roll out ceases for Christmas, but will be fixed before roll-out
recommences in January.( CP2186)
936 26 2\No Primary Mappings provided in Reference Data
See 932
9909300542
937 27 Duplicate records in file transferred. See 935 above.
1
9909300555
938 27 Negative sign on Txn Numbers — see 935
1
9909300543
30493
939 28 1 IInvalid ITMs,
1. ECCO Migrated remittance where a product has failed to be mapped to an
9910050085 equivalent Horizon product and has therefore been migrated as ‘Cash’ in
accordance with POCL's instructions on the handling of ‘contingency’
products at migration.
2. ECCO Migrated discrepancy transaction carried out in ECCO as a ‘Serve
Customer’ transaction.
940 27 3IReversal of the settlement of a stock adjustment.
The set of rules agreed with TIP for signage adjustments in the host before
9910070592 I 30919 Itransactions are sent to TIP was incomplete, in that it did not provide for
settlement of existing reversals of stock adjustments. TIP therefore calculated
their values in a manner inconsistent with Horizon. This had no business impact
at outlets, and the outlet cash account was correct. Rule extension to cover all
circumstances went live on 22.10.99 .
941 27 1 [Reversal of the settlement of a stock adjustment.
\See 940
9910070586
942 27. 270\Reference data Error on product 196.
\See 934
9910070616 (A duplicate entry has been created for the 2 occurrences within the Al
onitoring period.)
(Migration of erroneous ECCO Transfer transaction.
Cheques remmed out of one SU, but remmed into the other as cash caused
9910070597 I 30923 Icompensating differences on the cash account between Cash and Cheques
943 27
Printed 18:03 18/11/99 Page 2 of 10
FUJ00118171
FUJ00118171
Post-Acceptance TIP Incident Status Report —18/11/99
944 27
(Migration of erroneous ECCO Transfer transactions.
(Cheques for £1555.00 remmed out of one SU, but remmed into the other as
9910070604 30917 Icash; Cash of £216.25 remmed out but remmed in as Vouchers (Product 4),
(caused compensating differences on the cash account between Cash and
(Cheques. Items remmed out of one stock unit as Product 763 (self adhesive
il 2nd x100), remmed in to the receiving stock unit as Product 2869 (special
2nd class stamps).
945 27 No Primary Mappings provided in Reference Data
12 See 932
99100706-
06/14/19/24/2
6/27/28/29/31
[36/37/39
942 27 Reference data Error on product 196.
2 See 934
9910070616 (Duplicate entry created for 2 occurrences within Al monitoring period
31033
946 28 367/Reference Data Error on product 196
See 934
9910130571 31302
948 n/a__ IThe reported Invalid ITM in this call was not, in fact, invalid. An error in TIP’s
processing of reference data changes caused this incident to be reported. Call
9910130584 has now been closed.
949 28 1\Cash Account received by TIP with zero values in all lines.
[This is unrelated to previous incidents involving other missing CAC records. In
9910130849 31315 this case the transaction summary data used to construct the cash account was
present at the outlet but the CAC records (constructed during cash account trial)
{do not exist. This means that the trial cash account would also have been
printed with blank information, although there is no record of a helpdesk call
rom the outlet concerned on 6" or 7" October and the final cash account
lappears to have been printed with content. Detailed investigations continue.
[Reversal of the settlement of a stock adjustment.
ISee 940/941
950 28
9910130596 I 31277
951 28 1 IIncorrect recording of Stock Transfer Transactions
[On the evening of 30" September 1999 a release of reference data was made
9910130623 31304 which included the ‘end-dating’ of all the Pathway Mode Parameter reference
\data and the ‘Pathway Product’ reference data (the Pathway Products are used
fo settle Transfers/Remittances/Parcel Traffic/Non-Accounting Data sessions).
IAs a result, transfer out transactions carried out from 1° October to 4"° October
(when the reference data was re-instated) were settled against ‘Cash’ and
lcarried none of the attributes necessary to cause the transactions to report to
the ‘Transfers OUT’ line on the balance report. Effectively these transactions
lachieved a stock adjustment (down) of the transferred item(s). When these
ransfer sessions were received into the destination stock units, the system
ent through the process of inserting a negative sign into the values for each
ransaction in the session (stock transfers out can only ever be positive values
lin Horizon) and adding the attributes to cause the transactions to report to the
(Transfer IN’ line of the balance report. Because the original Transfer Out
jsettlement had been recorded erroneously as ‘Cash’, and the transaction
lalready contained a negative value, the insertion of the negative sign into the
(Cash transaction (the system did not recognise that this was the settlement
rather than another item being transferred) caused the value for the settlement
ransaction to be recorded with two negative signs (e.g. - -1000.00).
[The effects of this problem were as follows:
Printed 18:03 18/11/99 Page 3 of 10
FUJ00118171
FUJ00118171
Post-Acceptance TIP Incident Status Report -18/11/99
1. The TPS harvester rejected the transactions with double negatives as being}
invalid;
The EPOSS processes rejected the double negative transactions;
The sending Stock Unit still retained the overall value of the transfer (as
part of the Cash holding) but the receiving stock unit correctly showed the
transfer in against the correct products with no compensating reduction in
Cash (since the double negative settlement was ignored). At this point, the
office was now out of balance.
oT
As soon as Pathway were alerted to the TPS Harvester rejections a plan was
put in place to identify, correct and re-insert the faulty records in order to restoreI
the state of balance at the affected outlets. This correctional activity took the
form of replacing the faulty double negative signs on the Transfer IN Cash
‘Settlement transactions with a positive sign, thereby generating a Cash
reduction to offset the previous increase in stock. Following this action, each ofI
the stock units affected by either the transfer in or the transfer out were in the
position of holding compensating losses and gains. These could have been
dealt with at the outlet by adjusting the stock in each of the affected stock units
land then re-entering the failed transfers. The correctional transactions inserted
lby Pathway also ensured that the ‘Transfers IN’ line of the balance report was
reduced to compensate for the correction and retain the Stock Unit Balance.
IThe effect of these events on the Cash Account in CAP 28 was as follows:
1. The lines on Table 5 for Cash and Stock were calculated from the stock
declared for each stock unit, plus the transfers in and out of the stock units
(this the normal process for Cash Account production)
[2. Because the original transfers out did not contain the attributes to cause the
transaction to be treated as a transfer out for balancing purposes, only the
transfer IN transactions (including the Pathway correctional transactions)
were added to the Cash and Stock lines;
3. Adding only the transfer IN values caused the Cash Account stock level for
a product to be increased by the value of the transfers in and the Cash level
to be decreased by the same amount (from the corrected Cash settlement
transaction), appearing therefore as a compensating adjustment.
4. Receipts and Payments totals remained equal.
952 28
Incorrect recording of Stock Transfer Transactions
See 951 above
9910130723 31310
953 28 1IIncorrect recording of Stock Transfer Transactions
See 951 above
9910130699 31308
954 28 3IIncorrect recording of Stock Transfer Transactions
See 951 above
9910130819 31312
955 28 1 Reversal of the settlement of a stock adjustment
See 940
9910130841 31316
956 28 1 IIncorrect recording of Stock Transfer Transactions
See 951 above
9910130683 _I 31306
957 28 21IIncorrect recording of Stock Transfer Transactions
See 951 above
9910130669 31305
Printed 18:03 18/11/99 Page 4 of 10
FUJ00118171
FUJ00118171
Post-Acceptance TIP Incident Status Report —18/11/99
958 28
9910130798 I 31311
Incorrect recording of Stock Transfer Transactions
See 951 above for a general description of the problems caused by transfers
between 1* and 4" October.
In a small number of outlets (c. 8) the correctional transactions inserted by
[Pathway on the evening of 5" October arrived at the outlet after some stock
units which had transferred IN products from the faulty transfer sessions had
already rolled over the stock unit into CAP 29. These stock units would have
lappeared to balance normally but would have caused the Receipts and
[Payments on the Cash Account to have been different. In this particular office
(006329) 3 stock units rolled over during the day on 5.10.99 and these stock
units contained a total of £18,620.00 of faulty transfer transactions. Because
the declared stock levels in the office were then inflated by this amount and
lbecause the Transfer IN values were then further added to the Table 5 values
(see explanation against 951 above) this resulted in the office reporting
£37 ,240.00 difference between Receipts and Payments ( the other £20.00
difference reported by TIP was due to a transfer transaction for that amount
(not normally passed to TIP) being recorded with a ‘null’ mode which was
jsubsequently modified to ‘Serve Customer’ and forwarded to TIP).
959 28
9910130709 I 31307
Incorrect recording of Stock Transfer Transactions
See 951 above
960 28
9910130716 I 31309
Incorrect recording of Stock Transfer Transactions
See 951 above
961 29
9910150098 I 31417
Missing Cash Account records because of null Cash Account id.
Similar to TIP 916. The diagnostic version of software to trap and fix cases
here the Cash Account Id was being written as a ‘null’ value failed to trap this
joccurrence. The code has been carefully examined and it has been discovered
that it was possible that the Cash Account Id had been partially formed in
lattribute grammar and contained only the attribute grammar character that
\denotes the start of an attribute — the ‘<’ symbol. The presence of this
Icharacter caused the diagnostic software to believe that the Id was ‘not null’ but
hen the value was passed to Riposte for committal to the message store,
IRiposte ignores a single ‘<’ character which is not matched by a ‘>’ character
land therefore wrote a null value against the attribute for the Cash Account Id.
[An improved version of the diagnostic software has been prepared and will be
distributed to the Live Offices from 29.10.99. (WP6068)
IN/A to Al 376 - no cash account was sent. (Al378)
962 29417
9910200585
[Reference Data Error on product 196
see 934
Printed 18:03 18/11/99
Page 5 of 10
FUJ00118171
FUJ00118171
Post-Acceptance TIP Incident Status Report —18/11/99
963 29 18ICash Account reconciliation problem - under investigation
IThis has been investigated and the 18 affected outlets were subject to one of
9910200570 31713 = Ithree problems:
IThe majority (14) were the result of TPS generating the wrong sign on stock
ladjustment reversal transactions:
(confirmed at Outlets 025511, 162116, 173511, 177704, 257523, 265611,
349704, 358329, 267511; identical symptoms at Outlets 010511, 081116,
153611, 286329, 302523 for which no message stores were available at the
‘ime of diagnosis). These are duplicate occurrences of the problem identified in
[TIP 940.
(3 were associated with a known APS problem, where a reversal fails to
correctly record the mode of the original transaction (Outlets 002704, 006329,
(008323). Work is in hand to identify and fix the root cause.
IThe last was an apparent cash account misbalance, (Outlet 162504). This
incident has now been traced to an erroneous stock unit transfer in CAP 28 for
la value of £10 (see also TIP 965 below).
965 29
o
[Apparent discrepancy due to TIP basing calculation on incorrect bought
forwards figure.
9910200639 I 31712 ICAP 29 Consequence of the CAP 28 Transfer Failures
[For a general explanation of the transfer failures in CAP 28 see 951 above.
IAs a result of the incorrect reporting of stock levels on the CAP 28 Cash
(Account for products affected by the stock transfer problems in CAP 28, the
starting figures used by TIP to calculate the values for the CAP 29 Cash
(Account were incorrect. The result is that there were apparent discrepancies onI
the week 29 cash account despite there being no errors in the week 29
ransaction stream or cash account.
966 29 27 /Apparent discrepancy due to TIP basing calculation on incorrect bought
forwards figure.
9910200598 I 31907 [As for 965 above.
[This has been confirmed for 18 outlets, checking continues on the remaining 9.
[Apparent discrepancy due to TIP basing calculation on incorrect bought
forwards figure.
9910200573 I 31884 [As for 965 above.
967 29
ES
968 29 4ICAP 29 Consequence of the CAP 28 Transfer Failures - See 951 above
9910200623 I 31908
969 29 1ICAP 29 Consequence of the CAP 28 Transfer Failures
[For a general explanation of the transfer failures in CAP 28 see 951 above.
9910200613 I 31714
970 29 7\Reference Data Error on Product 196
9910200585 _I 31906 ISee 934
971 30 388/Reference Data Error on Product 196
991026033 See 934
972 30 2\Migration of erroneous ECCO Transfer transactions
9910260379 I 32090 IFAD 010704 - Cheques transferred between Stock Units D1 and BM in the
IECCO+ system for £935.00. This value was transferred in to BM as cash,
therefore generating the compensating discrepancy reported by TIP.
IFAD 011323 - Three separate transfers in the ECCO system between stock
lunits BM and AA where the value was transferred out as cheques and in as
(Cash. The transaction values were £248.85, £2049.66 and £221.44.See 943,
[944
Printed 18:03 18/11/99 Page 6 of 10
FUJ00118171
FUJ00118171
Post-Acceptance TIP Incident Status Report —18/11/99
973 30.021
9910260358
32099
Reversal of the settlement of a stock adjustment.
As 940
(On further investigation an additional problem, common with one under 963,
as identified. Current status:
Stock adjustment problem (as 963 and 940)- Outlets: 0145041, 1491164,
1551167, 1593064, 1793233, 2393069, 2576112, 2665115, 2836114, 2865238,
12933233, 3053237, 4363299. The fix (WP6101) was applied (TPS host) on
22/10/99, i.e. later than the end of CAP 30.
(Outlet 390329X also had a SAN reversal (£600) but this does match the TIP
reported discrepancy — further investigation is needed.
IThere was one occurrence of the APS problem identified above, where the
incorrect original TM was recorded for an APS reversal (outlet 2627043)
IThe error in outlets 2555239, 2863294, 3025233 was related to 951 see above.
IThe occurrence at outlet 0136425 was due to an ECCO transfer migration.
IThe reported occurrences at following outlets are still under investigation:
1625047, 171130X
974 30 4
9910260368 _I 32100
\Under investigation
(Compensating cash account differences on three lines (investigating with 972)
975 30 1
9910260384 I 32102
Migration of erroneous ECCO Transfer transactions
Prior to migration, a transfer of £2.00 of Product 766 (TV Licence Savings
Stamps) was made from Stock Unit 04 to Stock Unit PC and a further transfer
lof £3.38 was also made against Product 21 (Stamps). The transfer in to Stock
IUnit PC was recorded as a transfer in of £5.38 of Stamps. No separate transfer,
lin of the £2.00 of TV Licence Stamps was made. No fault in Horizon system.
976 126 4\Missing primary data mapping for Southend bus/rail tickets.
9911010686 _I 32552 ISee 932
977 N/A I There is no cash a/c inconsistency. The outlets from which multiple
9910290301 32335
jsubfiles were received were entered into the Horizon automated system
jas a result of previous BES (help desk authorised) foreign encashments.
When BES was running an EOD Marker was generated (at the
Icorrespondence server) to enable BES Help Desk transactions to be
lharvested to TPS. BES stopped running on 10/7/99 (when we upgraded
jto LT2). When an outlet is automated, the first time EOD runs it searched
ifor any previous EOD Markers and generates EOD events for any
"missing" days. As a result of this it generates EOD markers for each day
lsince the Foreign Encashment. The problem is now not occurring, as rollout
lhas stopped. Before Rollout re-commences the old foreign encashments will
lhave been archived off the correspondence server, and so the problem will not
Ire-occur.
978 3 4
9910290307 I 32330
Missing CAC records following STX (PINICL 31417 /WP6068) provides for
Kdiagnostics; distribution started 29/11, 90% complete at 05/11
(see 961)
IN/A to Al 376 (AI378)
979 31 2 IMissing CAC records following STX
9910290299 I 32336 \(See 961)
IN/A to Al 376 (AI378)
980 INot used
981 31 5/Re-occurrences of ref data error on product 196 See 934.
9911020537_I 32645 I(PC32090 refers)
982 31 1 ILikely to be incorrect recording of stock unit transfer in ECCO. Investigation
9911020536 I 32644
continues.
Printed 18:03 18/11/99
Page 7 of 10
FUJ00118171
FUJ00118171
Post-Acceptance TIP Incident Status Report —18/11/99
983 31 1IThe problem appears to have been caused by a Transfer Out transaction for a
9911020555 32649 Ivalue of £2441.00 being recorded in the same 'session' as discrepancy
transactions (Mode:DDN). The settlement transaction for the session was
recorded as Mode:DDN and would have been sent to TIP, the transfer out
ransaction was recorded as Mode:TO and would therefore not have been sent
fo TIP, causing TIP's view of the value of Cash to differ from that perceived at
he outlet.
Investigation has shown that it is possible to start a transfer out, not complete it,
jsuspend the session and then enter the balancing process (which will commit
lany discrepancies arising from declarations made). Investigations are
ontinuing to determine whether this would then cause the suspended transfer
jout transaction(s) to also be committed.
984 131 18IThis incident is the result of an update of the persistent object containing the
9911020545 32648 Irevised Cash Account mappings for product 21 failing to update the persistent
jobject index at the counter. This was caused by a fault in Riposte which was
fixed in update 22, now down at counters. There now needs to be an exercise
fo rebuild the corrupted indexes on the affected counters, which is in progress.
985 31 1 [Reported unrecognised item \ transaction mode combination: (184\8). Under
9911020529 I 32643 Iinvestigation
986 30 1\Cash Account not received for FAD 476611; office not reported as non-polled
9911020258 I 32634 IN/A to Al 376 (AI378)
987 31 1IAPS problem, where a reversal fails to correctly record the mode of the original
9911020528 I 32647 Itransaction
See 963
988 32 1 Negative sign on transaction numbers
9911040231 32769 ISee 935
989 31 2IThis appears to have been the result of an update of the persistent object
9911040243 32827 Icontaining the revised Cash Account mappings for product 21 failing to update
the persistent object index at the counter.
990 34 1\See 984
9911040390 I 32828
991 32 1 [Root Cause - Negative sign on transaction numbers (See 935). In this case
9911040417 I 32779 whilst correcting the file (arising from incident 988) the elimination of the -ve
sign results in dropping the least significant digit in the 4 char transaction id
field which can cause a duplicate.
992 32 1\Recovery Mode transactions against product 2289 (an APS product) mapped
9911080659 I 33058 {correctly to line 0009. TIP mapped these (incorrectly) to line 0059.
993 32 4 Awaiting details
9911080646 I 33059
994 32 6IApparent discrepancy due to TIP basing calculation on incorrect bought
9911080664 I 32980 forwards figure.
Problem associated with revaluation of Euro stamps. (see 984)
IThe transactions and Cash Account for week 32 are correct. The apparent
\discrepancy reported by TIP is due to TIP using the incorrect cash account sent
iby Pathway in week 31 as the starting position for their reconciliation, thus
leffectively reporting for a second time the original incident.
[Fad 211328 is knock on from 990;465329, 338523,412523 from 984; 467329,
173328 from 989.
995 32 1INon polled outlet (FAD399420)
9911080370 I 33051 INot relevant to Al376
996 32 [TIP error.
9911100279 I 33100 [See 992
997 32 1
9911120504 I 33265
Printed 18:03 18/11/99 Page 8 of 10
FUJ00118171
FUJ00118171
Post-Acceptance TIP Incident Status Report —18/11/99
998 33.~«2
9911120507 I 33266
999 33. Od
9911120518 I 33268
(33333)
7000 33. —~C«T
9911120527 I 33269
1001 33°~C«*d
9911120533 I 33271
1002 33°~«*d
9911120536 I 33272
1003 33 1
9911120543 I 33273
Printed 18:03 18/11/99 Page 9 of 10
FUJ00118171
FUJ00118171
Post-Acceptance TIP Incident Status Report -18/11/99
CAUSES OF CASH ACCOUNT DISCREPANCIES
Cause First or I No of Cash IIdentifying part of AdditionalI CAD? I Ref
only refI Accounts Reconciliation Data?
A.
2. IReference Data Error on 934 1190 IN/A - not Cash Account N Y
roduct 196 (TIP942 >) [Discrepancy
3. (Migration of erroneous 943 3 IN/A - not Cash Account N N
lECCO transfer transaction (TIP972->) IDiscrepancy
[But will be detected by weekly
(CA check at counter
4. INot Pathway error 948 IN/A - Not Cash Account N N
IDiscrepancy
5. (Cash account with zero 949 1 ICA weekly reconciliation Y N
values
6. __IReversal of the settlement 940 30 IN/A - Not Cash Account N N
lof a stock adjustment (TIP950->) IDiscrepancy
(error in TIP calculation of CA)
7. Incorrect recording of stock 951 72 Daily harvester activity Y Y
ransfer transactions (and
consequences)
8. [Missing cash account 961 2 leekly CA check at counter Y N
because of null cash
laccount ID
9. APS reversal txns with 963 5 under investigation
missing O-mode (other
items dealt with under
940 or 951)
10. I Transfer Out transaction 983 1 Weekly control total check Y N
within a DDN session on CA
11. I Product incorrectly 992 2 N/A no error in Horizon N N
mapped by TIP
(Other under investigation / ?
not categorised
Printed 18:03 18/11/99
Page 10 of 10