split up results.json and move to ctree about getpapers HOT 5 OPEN

@chreman is there somewhere that ctree is properly defined?

from getpapers.

I am working on cproject and ctree and raise these issues for discussion. Some architecture is described in http://www.slideshare.net/petermurrayrust/architecture-of and was ported to Discourse http://discuss.contentmine.org/t/overall-architecture/142 with the idea we could address it there. But probably I should tackle it in https://github.com/peter/cmine and then give @Anusha a pull request to https://github.com/ContentMine/cmine (which doesn't yet exist)

from getpapers.

OK. For the moment how about I follow what quickscrape does?

for each entry, create a directory with:

• bib.json with results in bibjson format
• api_results.json with raw json results, e.g. epmc_results.json
• fulltext.{xml, pdf, html} - fulltext in whatever format
• images/*.* images in whatever format

If bib.json already exists, getpapers will merge the new results into the existing ones, unless the use specifies --overwrite, in which case it will replace the whole file. If any of the fulltext.* files already exists, they will not be downloaded again unless --overwrite is specified.

from getpapers.

looks good.
when I looked at eupmc_results.json the JSON seemed to have been created from a database with a lot of 1-element arrays. Presumably you can convert this to BibJSON ok?

from getpapers.

Yeah that is pretty common in lots of databases! Easy to convert :)

from getpapers.