anna-dsb / totv-utotv Goto Github PK

Once FCA Transparency system (FCA FITRS) is available and the DSB is able to access and download transparency data from FCA, FCA Transparency data will be made part of ToTV v2 records

In the following scenario only:
• DSB ISIN is created with a non-EZ underlying
• The underlying has not been reported to FIRDS by a venue at the time of creation (hence uToTV=False)
• The underlying gets reported to FIRDS by a venue at a later date
• The Instrument ISIN does not get reported to FIRDS/Transparency and is not updated by DSB (a possible reason for an update will be ISIN expiration)

After Underlying ISIN is reported by a TV to FIRDS, ToTV record for the Instrument ISIN enquired by API’s/GUI will show uToTV=True, whereas the File Download for the Instrument ISIN will continue to hold uToTV=False until there is a change to Instrument ISIN itself and its updated ToTV record is included in ToTV File Download.

This affects ToTV File Download only and does not affect ToTV request by ISIN via API’s/GUI

On 23 September 2019 ESMA changed their xml schema to new version (1.1.0) for FIRDS Ref Data.

The ‘First Trading Date’ component in the new schema now includes the symbol for UTC time, e.g. “2018-09-21T08:00:00Z”.

As a direct result of the date format change, this component is being incorrectly parsed.

This means that in some rare use cases the ToTV Effective Date will be missing or incorrect.

Environments Affected:
UAT, Production

Functions Affected:

• ToTV searches via APIs/GUI from the 23 September 2019 for the 2 scenarios set out below
• ToTV File Downloads dated from the 24 September 2019 for the 2 scenarios set out below

Instruments Impacted:
Only those instruments that have ToTV flag set (ie. non-EZ ISINs traded by at least one trading venue) will be impacted in the 2 scenarios detailed below:

1. A TV submission is in a FULINS/DLTINS file dated from 23 September and it has an earlier ‘First Trading Date’ than TV submissions of the same instrument coming from an earlier FULINS/DLTINS file (dated before 23 September)
   -> In this instance ToTV flag will be incorrect as the earlier ‘First Trading Date’ would not be parsed successfully and ToTV Effective Date would not be updated.

2. All TV submissions are new, i.e. coming from a FULINS/DLTINS file dated from 23 September
   -> In this instance ToTV flag will be missing ‘ToTV Effective Date’

The DSB is currently conducting analysis and an update will be provided once it becomes available. Once the DSB has applied a fix for this issue, all ToTV File Download folders since 24 September will be re-loaded

If an instrument is in FIRDS Reference Data FULL FILE but has never appeared in FIRDS Reference Data DELTA FILE – it is missing from DSB ToTV records until it is in FIRDS Reference Data DELTA FILE.
E.g. ISIN AT0000490610 first appears in FIRDS Reference Data Full file (FULINS_D_20180106_01of02). But not in any of the Delta files prior 20180116. As DSB’s only processing Delta passed 15/10/2017, the ISIN is missing from ToTV records until 20180116.

Some ToTV records in EoD files between 15-Oct-17 and 31-Dec-17, DSB-ISIN component might be missing for DSB-issued ISINs in cases when the ISIN was created/last updated prior T-1.

Workaround:
Please use the ISIN reference data from the DSB end of day data instead.

If FIRDS Reference Data or Transparency records for T, are not available by 9:00 CET at T, they will get included in T+1 ToTV files.
Data that were available at T by the cut off time of 9:00 CET, will also get included in T+1 ToTV files.

Workaround:
None needed. The same information is simply available in the T+1 files as already exists in the day before.

According to FIRDS Transparency System Instructions on download and use of full and delta transparency results, 2016-01-01:2016-12-31 Transparency period is valid from 01/06/2017 to 31/05/2018, and from 01/06/2018 2017-01-01:2017-12-31 Transparency period becomes current.

2016-01-01:2016-12-31 Transparency period is not currently filtered out and both yearly periods are shown on a ToTV record. This issue is only in June’s data and come July, 2016-01-01:2016-12-31 will get filtered out and the only yearly period will be 2017-01-01:2017-12-31.

User impact: When using Transparency data, the users will need to use 01/01/2017-31/12/2017 data from 01/06/2018.

ToTV-EffectiveDate was wrongly named as ToTV-ErliestEffectiveDate on the JSON Schema
uToTV-EffectiveDate was wrongly named as uToTV-ErliestEffectiveDate on the JSON Schema

Change the JSON schema to say: ToTV-EffectiveDate and uToTV-EffectiveDate.

Response message for version 2 ToTV request via FIX is missing a tag that indicates ToTV version 2: SecurityGroup(1151)=2.

If no FIRDS Reference Data are published for the day, there will be no File Download for that day. Any Transparency Data from that day /DSB Production ISINs from the previous day will be missed and will not form part of ToTV File Download on the same or different day. Such ISINs will still be available via ToTV GUI Search and API on a per ISIN basis.

E.g. there were no FIRDS Reference Data files published on 22/03/2018, but FIRDS Transparency Data were published, yet ToTV File Download has no files for 22/03/2018.

Following FCA clarification on Concept of Traded on a Trading Venue in paragraph (6) in FCA Supervisory Statement on the Operation of the MiFID Transparency Regime post-Brexit published here, the DSB will be using Relevant Competent Authority field instead of MIC Country in order to determine which instruments are ToTV in the UK from FCA FIRDS.

File Download for ToTV is currently incorrect and will be reloaded:

1. FIRDS records that appear in 5/Dec files will be extracted in the same date folder and the same date files, e.g. /file-download/totv/20171205/ToTV-ST-20171205.zip
2. DSB ISIN records that have Last Update DateTime: 5/Dec will be extracted on 6/Dec, e.g. /file-download/totv/20171206/ToTV-CG-20171206.ZIP

Impact: UAT

If DSB-issued ISIN created with an underlying is not reported in ESMA/FCA, but its underlying has been reported to ESMA/FCA, the version 2 ToTV record will not have ToTV/uToTV flags as EU and UK blocks will be missing (the instrument ISIN has not been reported to FIRDS). So there will be no indication that instrument ISIN is uToTV in ToTV version 2, only in ToTV version 1 where ToTV/uToTV flags are always part of any ToTV record.

For the 25-Jun-2018 ToTV Production release, DSB is uploading historical FIRDS Reference and Transparency data for the months of Oct 2017 -June 2018 to create EoD ToTV files for all historical dates FIRDS published Reference and Transparency data for.
From 4-Jan-2018, when FIRDS first started publishing Transparency, some ISINs have been reported multiple times for the same periods.
DSB has uploaded historical FIRDS data month by month, resulting in only the latest transparency record for an <ISIN,period> for every month being stored in the database.
This means that for an <ISIN, period> reported on several days in the same month, the ISIN’s transparency record will only be available in the latest ToTV file it was reported on for that month. The ISIN might still be present in the ToTV files on the earlier days it was reported in FIRDS Transparency but not have a Transparency component. This will be the case if there are FIRDS Reference Data for that ISIN available. Or it might be missing altogether from the ToTV files on the earlier days it was reported in FIRDS Transparency but no FIRDS Reference Data was published for that ISIN.

User impact:
There is no impact to current users of the DSB’s ToTV/uToTV service, as the current/last updated transparency record for all <ISIN,period> will be correct and in line with Transparency Data published by FIRDS.

Problem:
If for any reason (e.g. due to a large number of files published by ESMA/FCA), the process of ToTV File Download generation does not complete within the same date and T data published on T+1, the system does not generate T+1 ToTV File Download on T+1, and any data for T+1 are then included in T+2 ToTV File Download.

Impact:
UAT ToTV File Download is not available for June 3, 2019. Data for June 3 has been published in June 4, 2019 ToTV File Download in UAT. There is no impact on enquiring ToTV records via an API/GUI as it always returns the latest ToTV record containing all information published by ESMA/FCA.

Workaround:
Any data not made available in ToTV File Download on T (due late ESMA/FCA publication or large number of files made available on the day), are made available in T+1 ToTV File Download as part of normal process.

FIRDS data could be published days later. The cut off time for FIRDS data is 9 am CET, if no FIRDS data available by then, the DSB continues to search FIRDS until 23:00. Any files published after that time are never picked up.
E.g. FIRDS Transparency files for 2018-03-17 were published only on 2018-03-19:

The DSB will be amending its ToTV/uToTV codebase to address any changes in ESMA publication schedule of transparency calculations. In the event of delayed publication, the MiFID II reporting period that was last available for an instrument will remain on the DSB ToTV record until a new reporting period is available and the published record becomes valid.

ESMA’s statement on the Use of UK data in ESMA databases and performance of MiFID II calculations in case of a no-deal Brexit, published on 5 February 2019, set out the following changes to the publication schedule of transparency calculations for non-equity instruments:

• Quarterly liquidity determination for bonds – 1st Jan 2019 – 31st Mar 2019 data might not be published on 1st May 2019
• Annual transparency calculations for all non-equity instruments other than bonds (liquidity status, LIS, SSTI) – 1st Jan 2018 – 31st Dec 2018 data might not be published on 30th Apr 2019

The DSB is amending its underlying functionality such that any publication delays of MiFID II transparency data will result in data for the most recently published reporting period remaining available on the DSB ToTV record until a new reporting period becomes valid.

Some ToTV records have an incorrect LastModifiedDate.
E.g. these ISINs were reported in FIRDS on 2017/11/03 but have LastModifiedDate of 2017/10/15 in the ToTV-RW-20171103.records file in UAT:
− DE000HW7VBX5
− DE000HW7VBN6
− DE000HW7VBT3
− DE000HW7VBR7
− DE000HW7VBV9

Such ISINs have an incorrect LastModifiedDate in ToTV File Download as well as via ToTV GUI Search or API if queried on the same day.

Some ISINs are missing from the DSB TOTV db when DSB Production ISINs are loaded overnight into UAT.
The content of ToTV files in UAT File Download shows less ToTV records than created in DSB Production the day before:

The missing ISINs will not be available in ToTV File Download or ToTV GUI Search/API on the same days.

If DSB was to generate EoD ToTV files in the historical mode as opposed the normal daily mode, there could be a potential issue with the filtering out of no longer current reporting periods as the current process would use the Today’s date rather than the date of the ToTV file being created. Essentially transparency periods that are no longer current today could be filtered out, even if they were still current historically.

Workaround: None needed. The most up-to-date transparency information will be made available in historical files as well as in the latest.

Some ISINs are truncated to an invalid length due to a bug when parsing FIRDS Transparency xml files.
− 381 instead of GB00H22N3381
− BM1 instead of DE000DB8SBM1
− D3 instead of DE000VL4R7D3
− 3697 instead of GB00H1GQ3697

Those ISINs are then missing from UAT ToTV File Download but can be seen via GUI ToTV Search/API.

Problem:
FIX API users sending ToTV FIX messages which coincides with the daily ToTV file generation processing time (09:00 am UTC to 12:00 pm UTC) may experience a delay in receiving responses as the database is busy during the generation of the files and there is a threshold limitation that may be exceeded that will result in throttling of requests.

Impact:
If you are utilizing the same FIX session for both ToTV and DSB ISIN requests and sending requests during the ToTV file generation processing time, responses may be delayed until the ToTV file generation has finished processing for the day. The DSB is putting in place a timeout for ToTV requests. Once the request’s idle timeout is reached a response will be sent back to the FIX API user. Please note that there is no impact if you are using a separate FIX session for ToTV requests and DSB ISIN requests.

Workaround:
The DSB recommends FIX users to avoid sending ToTV requests during the ToTV file generation processing time (09:00 am UTC to 12:00 pm UTC) which will minimize delays in ToTV responses.
If possible, DSB also recommends that DSB ISIN requests are sent on different FIX sessions to ToTV requests.

Some ISINs are missing from the DSB ToTV db when FIRDS data are processed.
E.g. the content of ToTV files in UAT File Download for 2018/01/25 shows less records than published in FIRDS Reference Data for the same day:

The missing ISINs will not be available in ToTV File Download or ToTV GUI Search/API on the same days.

Problem:
Minor display issue for Transparency Data component for anyone looking at ToTV records via GUI: Transparency Data component does not only have attribute names and values but also data type of the attributes (e.g. object/string).

Impact:
GUI users only

Environment:
UAT

Problem:
There are instances when an instrument submitted to FIRDS first reported with one or more Underlying Instrument ISINs, and later the same instrument is reported with no Underlying Instrument ISIN at all.
In this scenario, the DSB could have identified such ISIN as uToTV (if one or more of its Underlying Instrument ISIN is traded by at least one trading venue), and it then remains uToTV even if the instrument no longer have Underlying Instrument ISIN).

Impact:
There will be instruments that have uToTV set to True, but have no Underlying Instrument ISIN.

Environment:
UAT

Some ISINs are truncated to an invalid length due to a bug when parsing FIRDS Reference Data xml files.
E.g.
− 2 instead of DE000HV1XXH2
− 3 instead of DE000CD5NYK3
− 4 instead of DE000C165084
− 5 instead of DE000C164KD5

Such ISINs can be seen both via UAT ToTV File Download and GUI ToTV Search/API.

Problem:
The following ToTV ISINs are not searchable via GUI/REST/FIX due to an invalid date format in the "period" attribute.

DE000A289FK7
DE000HB12N76
NO0010780463
XS1752915808
DE000VQ7XXV9
USQ04578AG72
US26442CAA27
US45818WCP95
XS2101121205
XS2209052765
SE0018260440

Impact:
Users who try to retrieve the mentioned ToTV ISINs will receive "Internal Server Error" message.

Workaround:
Users who wish to retrieve the details of the ToTV ISINs may use the respective file in the File-download directories below

DE000A289FK7 https://prod.anna-dsb.com/file-download/totv/20220730/ToTV-DB-20220730.zip
DE000HB12N76 https://prod.anna-dsb.com/file-download/totv/20220730/ToTV-DY-20220730.zip
NO0010780463 https://prod.anna-dsb.com/file-download/totv/20220403/ToTV-DB-20220403.zip
XS1752915808 https://prod.anna-dsb.com/file-download/totv/20211231/ToTV-DT-20211231.zip
DE000VQ7XXV9 https://prod.anna-dsb.com/file-download/totv/20210522/ToTV-DY-20210522.zip
USQ04578AG72 https://prod.anna-dsb.com/file-download/totv/20220730/ToTV-DB-20220730.zip
US26442CAA27 https://prod.anna-dsb.com/file-download/totv/20220730/ToTV-DB-20220730.zip
US45818WCP95 https://prod.anna-dsb.com/file-download/totv/20220730/ToTV-DT-20220730.zip
XS2101121205 https://prod.anna-dsb.com/file-download/totv/20211106/ToTV-DT-20211106.zip
XS2209052765 https://prod.anna-dsb.com/file-download/totv/20220830/ToTV-DB-20220830.zip
SE0018260440 https://prod.anna-dsb.com/file-download/totv/20220712/ToTV-FF-20220712.zip

LastCompleteProcessingDate to be updated to be the last date when ALL FIRDS files have been processed. Currently shows 2/Dec, although 5/Dec have been last processed.

Impact: ToTV records
Environment: UAT

If DSB-issued ISIN created with no underlying is reported in ESMA/FCA with an Underlying and that Underlying is ToTV, the instrument ISIN is not marked as uToTV even though according to ESMA/FCA data it is.

Problem:
From 26-Feb-2018, all DSB ISINs in UAT have ToTV set to False when queried via ToTV GUI Search or an API.
ToTV file Download has ToTV set to False for all DSB ISINs from 27-Feb-2018 onwards, but historical records still need to be updated.

Action:
ToTV File Download records will be reloaded from 15-Oct-2017 to 26-Feb-2018 inclusive to set ToTV flag to False for all DSB ISINs in all historical records.

Impact:
ToTV File Download users only

Environment:
UAT

If a non-DSB ISIN is reported in ESMA Transparency only (CFI code is unknown), only v1 ToTV record is available in v1 File Download (/file-download/totv/) in an XX file, but v2 ToTV record is not available in v2 File Download (/file-download/totv-v2/).

If an instrument is published in FIRDS Transparency Data prior to FIRDS Reference Data – it is missing from the ToTV File Download until it is published in FIRDS Reference Data.
E.g. ISIN=GB00F77GRH10 is published in Transparency on 24-01-2018 (DLTNCR_20180124_1of2.xml), but it appears in FIRDS Reference Data only on 27-01-2018 (FULINS_O_20180127_01of02.xml). Such ISIN will not be in the ToTV File Download until published in FIRDS Reference Data.