biocommons / seqrepo-rest-service Goto Github PK

• create serepo w/seqs for compliance tests
• address all compliance suite tests

See ga4gh/vrs-python#20 and ga4gh/vrs-python#21 for data needs

steps to reproduce on main:

docker build -t seqrepo-rest-service:latest .

[...omitting long output...]

#10 20.25 error: Jinja2 3.0.3 is installed but Jinja2<3.0,>=2.10.1 is required by {'flask'}
------
executor failed running [/bin/sh -c python3 setup.py install]: exit code: 1

Add an argv parameter to cli.main, cli._parse_opts. Currently uses ArgumentParser.parse_args() with no args, so it only looks at sys.argv.

Parameterizing those would make it easier to run the service programmatically from another application.

Description

The same file exceptions related to seqrepo thread safety tracked here: clingen-data-model/architecture#548
occur when running seqrepo-rest-service. When there are concurrent requests, the global seqrepo object in the flask worker causes exceptions, and once it hits this exception, there will be an exception on all future requests that attempt to read from that file as well.

Stopping and starting the seqrepo-rest-service process resets it and temporarily resolves the problem. So the issue is with process state, there's no issue with the actual files on the filesystem.

The temporary solution to the above github issue that avoids modifying the seqrepo codebase was to add mutual exclusion to the object that contained the SeqRepo object. This enables the application to run concurrent threads except when they are executing code under that object, which is suboptimal because that is a fairly large critical section to make mutually exclusive.

Steps to reproduce

Run the server from one shell:

$ seqrepo-rest-service /path/to/seqrepo/2021-01-29

Send requests from a second shell:

$ bash -c 'for i in $(seq 1 100); do curl "http://127.0.0.1:5001/seqrepo/1/sequence/refseq%3ANM_000551.3" & done'

There is no search interface in the current implementations: a sequence either exists or doesn't (404).

For this issue, design and implement an endpoint to search for sequences. Examples:

• Find all versions of an unversioned sequence
• Find all sequences from a specific namespace

Because searching could return large sets, the implementation will need to consider whether/how to handle paging and state (sessions).

also, consider public deployment options

When fetching aliases, the ga4gh computed identifier has a GS prefix, which is not listed in the documentation.

This bug can be seen from the readme example:

  $ curl -f "http://0.0.0.0:5000/seqrepo/1/metadata/GRCh38:1"
    {
    "added": "2016-08-27T21:17:00Z",
    "aliases": [
        "GRCh38:1",
        "GRCh38:chr1",
        "GRCh38.p1:1",
            ... 
        "GRCh38.p9:chr1",
        "MD5:6aef897c3d6ff0c78aff06ac189178dd",
        "NCBI:NC_000001.11",
        "refseq:NC_000001.11",
        "SEGUID:FCUd6VJ6uikS/VWLbhGdVmj2rOA",
        "SHA1:14251de9527aba2912fd558b6e119d5668f6ace0",
        "VMC:GS_Ya6Rs7DHhDeg7YaOSg1EoNi3U_nQ9SvO",
        "sha512t24u:Ya6Rs7DHhDeg7YaOSg1EoNi3U_nQ9SvO",
        "ga4gh:GS.Ya6Rs7DHhDeg7YaOSg1EoNi3U_nQ9SvO"
    ],
    "alphabet": "ACGMNRT",
    "length": 248956422
    }

I might be wrong, but I believe that the prefix should instead be SQ?

connexion 3.x provides an ASGI compatible app interface that can be run in production servers like uvicorn

https://connexion.readthedocs.io/en/latest/v3.html#getting-started-with-connexion-3-0

Before the dawn of time, @korikuzma wrote in ga4gh/vrs-python#102 (comment):

I can confirm that it works! The only endpoint that doesn't work is http://localhost:5000/seqrepo/1/ping. This is the traceback:

Traceback (most recent call last):
⋮
  File "/usr/lib/python3.10/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
 

This service was working great for me for a couple of days but then it maxes out the OS allowable files open at the same time and fails.

Gives a Failed to open FASTA index ... "Too many open files"

A lsof | wc -l shows -> 65,863 files, a huge number of db.sqlite3 as well as individual fasta files.

Is there a workaround for this?