adafede / lotusfiller Goto Github PK

Hi @mSorok!

I thought I would manage this alone but it is probably wiser to ask you...

Now that everything is on Zenodo, the following steps:

java -Xmx16288m -jar lotusfiller-0.0.1-SNAPSHOT.jar data/platinum.tsv fragments/fragment_without_sugar.txt fragments/fragment_with_sugar.txt importLOTUSids lotus_ids_march2021.txt > latest.logs.mar12.txt

## add metadata 
java -Xmx16288m -jar lotusfiller-0.0.1-SNAPSHOT.jar addMetadata data/210223_frozen_metadata.tsv

should take directly what is on Zenodo.

I have no idea how to do it best. If you prefer to first download locally, or how java deals with loading files remotely.

Here is an easy snippet from python to get automatically the needed URLs:

from requests_html import HTMLSession

s = HTMLSession()
r = s.get("https://doi.org/10.5281/zenodo.5794106")

urls = [l for l in r.html.absolute_links if "frozen.csv.gz" in l]
if not urls:
    raise ValueError(f"can't get the link for frozen.csv.gz")
print(urls[0])

r = s.get("https://doi.org/10.5281/zenodo.6378224")

urls = [l for l in r.html.absolute_links if "organism_metadata" in l]
if not urls:
    raise ValueError(f"can't get the link for organisms")
print(urls[0])

urls = [l for l in r.html.absolute_links if "structure_metadata" in l]
if not urls:
    raise ValueError(f"can't get the link for structure")
print(urls[0])

These URLs can then be used to read the files directly or download them.

I do not know how to keep low the needed java dependencies etc. If you can guide me on how to do it best, great!

Are fragments/fragment_with_sugar.txt and fragments/fragment_without_sugar.txt supposed to be updated?

If so, how?

From my understanding, those fields are needed for the search but there are users using them for different (wrong) purposes, shall we state (stronger?) it is a bad idea?

The organism_url is null for many DBs. We could fill this

The organism Wikidata ID should be outside of the "taxonomy" table, next to the reference link to WD ideally.

https://lotus.naturalproducts.net/browser/table

Clicking on the link does not redirects to the taxonomy/reference but to the compound page

There is a deep smiles field. Shall we leave it empty? I don't personally care