Dynamic updating of datasets from the conp-datasets repo. about conp-portal HOT 8 CLOSED

If all of the datasets referred to by the portal are in the conp-dataset repo, then an automatic poll would make the most sense. However, if there are (or will be) datasets who do not reside directly in the repo, then presumably any changes to those externally-hosted datasets would involve pushing notification of the changes into the CONP portal/repo through a versioned registration process.

I think I'm not completely clear on the relationship between the portal and the repo, so perhaps my thought isn't quite worth the $0.02 I was hoping it was?

from conp-portal.

All datasets should be in the conp-pcno/conp-dataset - pointing (as submodules) to their original datasets - ie, repositories. Some but not all of these repositories are living - names are super confusing - on the github conpdatasets organization, but they can certainly live outside as well. So yes, I think you are right Tom, as long as we have access to the datasets from conp-pcno/conp-dataset then we should be able to populate the portal.

from conp-portal.

Note that it would require
1- clone conp-dataset
2- datalad install . for all datasets ... so that we get the dats.json files

from conp-portal.

Thanks @jbpoline, really all we need in the repository is the metadata, so I would qualify your previous point with that there needs to be at a minimum the dats.json with the appropriate information to populate the front page. Also there needs to be a means to access the data where it lives, whether it be through datalad or some other means. Right now, the portal is storing a url to the data, which is not a robust means to define how to access the data.

But to populate the portal page with appropriate information, the metadata that describes the dataset should be in the conp-dataset repository.

@jbpoline, have you and the team settled on a definitive list of things that are necessary in the dats.json? I think so, but I thought it would be good to provide a pointer here in case others would like to comment.

from conp-portal.

@shots47s : yes, the dats files are compulsery, tests should fail if they dont exist !
yes - there is a list - need to dig out where (in Jen's documents) but here they are

• the schema.org required fields (title + description)
• license / data usage agreement
• authors / contact
• doi
  So, 5 required fields (we intentionally kept that to the minimal)
  @[email protected] : could you point us to your document on this ?

from conp-portal.

Is there any documentation somewhere about how they're currently statically created at install time?

It sounds to me like a simple cron job that does git pull would be sufficient based on the above thread, and I don't see why they would need to be completely dynamic if it's a significant amount more work for marginal benefit.

from conp-portal.

@driusan, thank you for the clarification. Currently, the data is being populated from a manually created SQL dump file, so not dynamic at all. When I say dynamic, I mean "created from the repo" or something like that, not something that is being populated by manual entry. I would agree, some periodic updating through something like cron would be appropriate.

from conp-portal.

#120 and #115 fixes this issue.

Datalad python module is used to clone https://github.com/CONP-PCNO/conp-dataset then install, update and merge its subdatasets.

This can be done with flask update_datasets and have been wrapped in a cronjob that run every 15 minutes.

The datasets details are read from their respective DATS.json file

from conp-portal.