Solution to frequently missing taxonomy specifications in UK submissions about py-xbrl HOT 21 OPEN

Hey, no worries.
Yes, it's true that these errors still occur often on UK submissions because many companies do not properly define their schema_urls.
I searched the internet specifically for this "dpl-frs" taxonomy. However, I could not find an online entry point (only .zip folders containing the taxonomy).

But finding some solution to this problem is in the top 3 list of fixes I want to implement in the next month. Will keep you updated.

from py-xbrl.

#76 is related

from py-xbrl.

Hi there - apologies for lack of knowledge of this, but for UK data i am frequently unable to pass statements because of:

• 'LargeMedium-sizedCompaniesRegimeForAccounts'
• 'dpl-frs'
• 'core'
• 'ThirdPartyAgentTypeDimension'

Was this fixed? I assume its a problem with the UK data.

Thanks very much

from py-xbrl.

@manusimidt I realise that this is provided for free and its an excellent solution, but i wonder if the UK aspect is any closer to completion? I am close to completing a research report and would ideally like to use this (with attribution!)

from py-xbrl.

The issue I still have with UK submissions is that many of them do not properly import their underlying taxonomies.
Additionally, the taxonomy endpoints are extremely hard to find and sometimes even not reachable.

If I would find the taxonomy schema files on the web it would be pretty easy to implement.

For example, the taxonomy endpoints for us submissions are pretty straightforward and easy to find:

If I had the same table for the UK taxonomies, I could really easy fix this issue. However, I can't find proper taxonomy endpoints for the taxonomies where the parser fails:

So basically if I find the missing taxonomy schema files on the web (and basically complete the table for all the years), it would be relatively easy to fix this issue.

from py-xbrl.

@manusimidt does this page contain what you need? https://www.frc.org.uk/accountants/frc-taxonomies#current-taxonomies-downloads

or

https://uk-taxonomies-tdp.corefiling.com/yeti/resources/yeti-gwt/Yeti.jsp#tax~(id~9*v~150)!net~(a~160*l~32)!lang~(code~en)!rg~(rg~4*p~2)

Realise they're not end points but have the xsd files

from py-xbrl.

https://uk-taxonomies-tdp.corefiling.com/yeti/resources/yeti-gwt/Yeti.jsp
Is a great way to visualize the taxonomy structure. They also have the FRC 2022 - 2024 Taxonomies, however just visualized for the user and not machine readable.

https://www.frc.org.uk/accountants/frc-taxonomies#current-taxonomies-downloads
Is the correct location for downloading the FRC taxonomy. But as far as I can see they only provide .zip folder downloads containing the entire taxonomy and supporting documents. For py-xbrl I ideally would need a direct endpoint (.xds file on server) to easily fetch it. There are basically two workarounds:

1. Upload the extracted .zip folder to some server and map the namespace to this url
2. Manually implement functionality to download the .zip folder, extract the necessary documents and map the prefixes, namespaces and (local) schema URLs.

But this would not even solve the full problem, since there site provides only the FRC Taxonomies (as far as I can see). I still don't know to which the other prefixes and namespaces belong.. (i.e.: "dpl-countries", " dpl-frc"...)

from py-xbrl.

On your last point, I think (but admit i dont know much about this) countries and frc are in the Zip folders, but named slightly differently. But yeah i see your point. How frustrating! I can try and raise this with companies house

from py-xbrl.

Was there any progress on this with companies house? Currently running in to the same issue
I have some of found the dpl files here : https://www.frc.org.uk/library/standards-codes-policy/accounting-and-reporting/frc-taxonomies/

They are in zip format but could we put them in a public s3 bucket or a dedicated github project and map them to the right namespace?

from py-xbrl.

Are these any of the missing Schema URLs?

• All content: "https://xbrl.frc.org.uk/fr/2023-01-01/core/frc-core-full-2023-01-01.xsd"
• FRS-101: "https://xbrl.frc.org.uk/FRS-101/2023-01-01/FRS-101-2023-01-01.xsd"
• FRS-102: "https://xbrl.frc.org.uk/FRS-102/2023-01-01/FRS-102-2023-01-01.xsd"
• IFRS: "https://xbrl.frc.org.uk/IFRS/2023-01-01/IFRS-2023-01-01.xsd"
• FRS-102-UKSEF: "https://xbrl.frc.org.uk/FRS-102/2023-01-01/UKSEF/FRS-102-2023-01-01.xsd"
• IFRS-UKSEF: "https://xbrl.frc.org.uk/IFRS/2023-01-01/UKSEF/IFRS-2023-01-01.xsd"
• Core: "https://xbrl.frc.org.uk/fr/2023-01-01/core/frc-core-2023-01-01.xsd"
• DPL: "https://xbrl.frc.org.uk/dpl/2023-01-01/dpl-2023-01-01.xsd"

This also works for previous years ie
https://xbrl.frc.org.uk/fr/2022-01-01/core/frc-core-full-2022-01-01.xsd

from py-xbrl.

hi @manusimidt - what is needed for me to create a PR with these additional taxonimies? Are you able to link to a similar PR demonstrating whats needed? Happy to have a go at getting these added (and anymore i can find)

from py-xbrl.

@Cave-Johnson thanks for researching and providing the links. They indeed look promising! I am not aware of any PR which addressed this problem.

Keep in mind that most of the errors are caused not because the schema URL of the taxonomy is missing but because both the schema URL and the namespace of a given taxonomy are missing. This creates for example the rather frequent error KeyError: 'dpl-frs'.

Here a simple map between the namespace and schema URL is not enough since the namespace dpl-frs could refer to multiple taxonomies. In this case we basically have two options:

1. Always map the prefix of the taxonomy to the newest taxonomy version that is available and hope for compatibility
2. Map the prefix of the taxonomy to the most probable taxonomy (i.e.: filing submitted on 31.10.2019 will most probably use the taxonomy version that was released in January 2019).

from py-xbrl.

Another option would be that the parser just throws a warning and just ignores all facts that were tagged with the taxonomy which could not be imported. However, this could lead to cascading problems since the parser for example also parses XBRL footnotes. If the fact which the footnote references is not present due to missing taxonomy then the assigning of the footnote to the fact will also fail.
This case could be avoided by a simple if-check, but must still be taken into account if such a solution were to be implemented.

from py-xbrl.

Probably the last option (to just ignore facts tagged with not locatable taxonomies) would be the best one. In the last two years 90% of my time I invested in py-xbrl was to implement a workaround because some submissions from some country doesn't strictly follow the XBRL standard. However, all these exceptions really bloat the library and make the code harder and harder to read and maintain.

I also tried to formulate this in #84. I hope that I have some time over Christmas to dive into this. In the meantime any PRs or suggestions are highly appreciated :)

from py-xbrl.

I just ran a test with new UK submissions and noticed that the schema URL for the following namespace was often missing:

The taxonomy with namespace **http://www.hmrc.gov.uk/schemas/ct/dpl/2021-01-01** could not be found

@Cave-Johnson already found the DPL taxonomy for 2023: https://xbrl.frc.org.uk/dpl/2023-01-01/dpl-2023-01-01.xsd
If we find a 2021 version these errors simply could be resolved for now by adding it to the mapping

from py-xbrl.

from py-xbrl.

A bit more digging i have found the following https://www.hmrc.gov.uk/schemas/ct/combined/2021-01-01/dpl-2021-frs-102-2021-v1.0.0.xsd which lead me to

• https://www.hmrc.gov.uk/schemas/ct/dpl/2021-01-01/dpl-2021.xsd
• https://xbrl.frc.org.uk/FRS-102/2021-01-01/FRS-102-2021-01-01.xsd

Are these what was missing?

from py-xbrl.

Been having the same issue recently when parsing companies house docs, was wondering if there was any update on this please?

from py-xbrl.

I have tried it with the taxonomies provided by @Cave-Johnson, however, unfortunately, they did not work for all of the failed submissions I tested on.
The instance documents were using tags like dpl-countries:CountriesDimension which are not present in the taxonomy schema.

from py-xbrl.

I think the only viable option is to build in some "fail gracefully" mode into py-xbrl in which the parser returns a partially parsed xbrl instance document and just omits every fact/context that is labeled/uses a concept from a taxonomy which could not be located.

from py-xbrl.

understand your point RE fail gracefully logic and that makes sense. It would still be good to get the taxonomies linked where possible.

Ive done a bit more digging and come across the attached that references dpl countries in the dpl-2021-presentation.xml file - is this of any use?
DPL-2021-v1.1.0.zip

from py-xbrl.