How to use
First of all, set the mongodb credentials in the danish_cbr_spider/settings.py
Then run scrapy crawl companies
This should scrape the data into the mongodb database
How I did it
CVR generation
The companies in the registry are identified using CVR numbers.
I found the specs for the CVR number verification algorithm (along with implementation using ruby) and wrote an algorithm that generates them (spec, ruby impl)
It can be found in danish_cbr_spider/cvr.py
It seems that success rate (probability of the generated CVR being assigned to any company) is around 80%
The code was tested using pytest
(run pytest danish_cbr_spider/cvr.py
)
The scraper
Currently, the scraper goes over randomly generated 2000 cvr numbers (not all generated CVRs have companies associated with them so a bit more is generated)
This results in 1,578 company entries
Using JSON data
Even though the website uses web technologies to display the data, it appears to be fetching the actual data from the API located at https://datacvr.virk.dk/gateway/virksomhed/hentVirksomhed?cvrnummer={cvr_number}&locale=en
.
The data seems to be a well-structured JSON, so I decided not to parse any HTML and use the JSON
Data Exploration
Before writing most of the conversion code and designing schema I desided to look at the data itself
First I scraped the raw data
scrapy crawl companies -O anal/raw_companies.json
(disabling all the pipelines in settings.py
)
Then I loaded it into the jupyter notebook and started exploring (see anal/schema_devel.ipynb
)
Json Key Translation
First of all, it seems that most of the json keys are in danish which I don't really understand.
Many of them are domain-specific and not easy to translate.
So the most of the notebook is concerned by this task.
It appears that the website loads some translations from the following URL: https://datacvr.virk.dk/gateway/tekst
I saved it to anal/raw_translations.json
Overall, my process for finding a good translation of the JSON consisted of search through en_translations.txt, raw_companies.json, using Google Translate and, sometimes, google to try to understand the meanings of the fields
Currently, the translations are not complete, but it is quite straightforward to implement all of them
See the notebook for more info
Json schema generation
After translation of the JSON keys I decided to generate JSON schema to understand the format of the data & satisfy the "stongly-typed" requirement
I used genson project, which allows to partially automate the process
The schema can potentially be refined using SchemaStrategy. It can add more storing types like defining enums in places where currently there are just strings
The resulting schema is saved to anal/schema.json
See the notebook for more info
Person search
The data on persons connected to the company does not seem to include much
There seems to be some kind of ID field, but it seems that it is internal to the service
The only "global" ids available seem to be the name of the person and their address
Both would require some kind of fuzzy matching, which usually requires a specialized database (not a one like mongodb) to handle the load
To facilitate queries like "which companies is this person affiliated with?" one could build a person-centric view of the data
It would store the person as the document and list the companies they are affiliated with
For this one can crawl https://datacvr.virk.dk/enhed/person/{id}/deltager
pages, which seems to contain such information. Though, if a complete data set is collected, I believe it would not be necessary
One can build this view having only the companies' data.
Maintenance
In case the structure of the JSON would change one can use the generated JSON schema to detect such a change and use the jupyter notebook to find the translations for the new keys & re-generate the schema