Comments (12)
SEC replied:
All XBRL taxonomies currently accepted in EDGAR filings are posted on https://www.sec.gov/info/edgar/edgartaxonomies.shtml. An XML version can be found at https://www.sec.gov/info/edgar/edgartaxonomies.xml. We do not maintain anything similar for what has historically been accepted, but you can find the information in the latest Release Notes, for example Figure 1 in https://xbrl.sec.gov/doc/releasenotes-2022-draft.pdf
Please note that STPR-2011 is no longer in use. STPR-2018 and 2021 versions are available in the first link mentioned above.
So does this mean we can poll a website once after startup to get a list, instead of hardcoding? (maybe on demand for non SEC users)
from py-xbrl.
It would be possible to query the XML Version of the Edgar Taxonomies list instead of hardcoding it into the libary.
Keep in mind, however, that this only applies to SEC Edgar submissions. I deliberately did not include this functionality because I did not want to optimize py-xbrl for a specific XBRL source, but wanted to keep it as general as possible for all XBRL documents. If I were to include such functionality, I would decouple it modularly from the xbrl parser core modules (the core modules being instance.py
, taxonomy.py
and linkbase.py
).
from py-xbrl.
Unfortunately SEC drops old items from xml, but this would still give us 10years to add anything in xml to github, which is better than doing it every year.
My proposal would be to keep existing items in taxonomy.ns_schema_map but provide a utility function that grabs the latest ones from SEC and user can extend the dict (also with custom ones).
taxonomy.ns_schema_map # contains historical/common items
# extend base schemas, download can fail, new ones override old ones
ns_schema_map.update(get_SEC_schemas())
This eliminates user having to write the download/parsing, and for us having the update the lib every year.
We could put get_SEC_schemas into taxonomy.py, or util_sec.py
I will chase SEC up to get a list with older items, so there might be more urls in get_SEC_schemas.
My implementation: grab.txt
from py-xbrl.
SEC also suggested this file, contains all the old schemas, we should add these to the hard coded ones, and poll the new ones less frequently.
from py-xbrl.
Yes that's a good resource. The xml file also contains the namespace-schema mapping. It is probably the best to create an abstraction layer above the core parsing modules and implement both ways simutainiously.
So basically:
- First look into the taxonomy schema for the schema location
- If schema url could not be located look into the a hardcoded mapping in the parser
- If the namespace could still not be resolved query the mapping provided by the SEC
from py-xbrl.
Sounds good, I would still use my API suggestion:
- hardcoded is default, unless user clears the dictionary
- user can add custom (not SEC) ones
- latest SEC ones can be loaded on demand (not automatically), since it can be used for non SEC too, maybe a mode='sec' for automatic fetching
I parsed arelle hardcoded ones into python dict format:
hardcoded_arelle.txt
from py-xbrl.
I extended with these hard coded ones in #77 for first step.
from py-xbrl.
Thank you! I will check and merge it in the following days.
from py-xbrl.
I only left https protocol to ones which would auto redirect from http->https anyway, otherwise it wastes time with redirection or certificate checking.
from py-xbrl.
Now that we're in 2022, filings are using the 2022 taxonomies. This means the library may fail to parse filings that assume presence of those common taxonomies. @mrx23dot would you suggest using the function you wrote above (get_SEC_schemas) to fill the gaps?
from py-xbrl.
@Ajmed I think the suggestion of @mrx23dot is quite good and i am planning to implement a similar function that queries the common taxonomy file for SEC submissions.
Normally, if every submission would strictly follow the XBRL Standard, we wouldn't need this function. But i understand that it is really helpful if you want to parse SEC submissions from all companies (also those that fail to completely comply to the standards). Also it's probably better than keeping a static list that does the namespace-to-schemaUrl mapping.
The reason why i haven't implemented it yet is that I want to implement an advanced caching for this file. The libary should not automatically download this file from the SEC Servers whenever the taxonomy module is imported. On the other hand the libary should also not download it once and never update the local copy.
It should only be downloaded when really needed and maybe cached for an hour or something like that before it gets updated by a new version from the SEC servers.
from py-xbrl.
@manusimidt makes sense. No rush, wouldn't want extra things entering the library that become cruft. In the meantime I've used the above code and it's successfully parsing the ill-formed filings.
from py-xbrl.
Related Issues (20)
- Need path or reference to source file of a Linkbase HOT 2
- Standardised Financial Data HOT 5
- Equals method for all fact classes HOT 1
- Solution to frequently missing taxonomy specifications in UK submissions HOT 21
- Parsing Failures for Empty Fact Values and 'nil' Text in XBRL Documents HOT 2
- Support a New Taxonomy? HOT 1
- Parsing filings with empty imports HOT 3
- Issue with dateMonthDayYearEN function in Transformation __init__ file HOT 1
- xbrl.TaxonomyNotFound HOT 2
- parse_ixbrl does not close the file it opens HOT 4
- No support for embedded/inline schemas
- Elements missing after parsing HOT 3
- maximum recursion depth exceeded in __instancecheck__ HOT 4
- Difference between pypi and github releases? HOT 2
- Infinite recursion loops with cyclic taxonomy imports. HOT 2
- Check differences between transformation registry version 4 and 5
- TaxonomyNotFound: The taxonomy with namespace http://xbrl.sec.gov/dei/2024 could not be found. Please check if it is imported in the schema file
- Unclear +/- sign of some facts HOT 17
- Space in url creates issues when requesting a taxonomy
- Potential arg bug in transformations __init__ HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from py-xbrl.