candig / candig-server Goto Github PK

Done in #43

Details TBD

This will be the master ticket to collect the technical requirements for bug fixes and performance improvement for v1.1 release.

After a user selects a new dataset, data should be reloaded on the page. Currently on the Sample Analysis page, old data persists and can be seen by the user. We should try to clear and reload all page contents after a dataset change.

This issue can be closed when the server is ready for 1.0 release

• No remaining security advisories
• Python 3
• No remaining issues labelled [science support] or [marker paper]
• Repos renamed from ga4gh-* to something else
• Clean up branches of ga4gh-*, PROFYLE_ingest, and any other needed repos
• API docs
• Documentation

It is confusing because it has list of endpoints, some of which are no longer supported.

More details TBA

July.18th:

This problem occurs for all of the file-based endpoints, including /reads, /variants, and some other ones we don't necessarily use a lot; the exception occurs after you ingest the files, and have the files somehow removed; this would raise a OSError due to the file not found for the /variants endpoint; and a custom exception class called 'FileNotFoundException' for the /reads endpoint, which does not have a toProtocolElement because it was not designed to be a runtime-level exception, but instead a ingest-level exception.

Master branch has been out of date for quite some time now. We will merge the authz into it and fix any required dependency issues

This issue can be closed when all the of the data can be ingested OK.

• Is there a test that we can run?

There's certain modules of the candig-server that is not used anymore, with the upcoming upgrade, it could be worth looking into which modules are unused, and thus can be safely removed, especially if this can help simplify the process of upgrading.

It would be great if the graph that displays the relationship between the cancer type and the treatment drugs can display the period of the usage of the drugs in the drilldown graph. This would involve the calculation of the period of the usage of the drugs.

Sometimes, users may do a pip install on the ga4gh-server by accident and run into all sorts of problems. It would be good that we have our own to reduce confusion I think.

The published candig-server package could require other candig service package as a dependency so users do not need to run multiple pip install commands.

The front-end javascript is poorly structured, and needs to be refactored to be more maintainable and readable.

Now that we have all existing tests and flake8 checks passing, and authz being merged into the master branch, it may be the time to clarify what we should do to ensure the new master branch is passing our existing/newly-added tests and flake8 checks.

Some of the things we could do is:

To push to the master branch

1. When new changes are being made, run the flake8 locally to ensure the code is following the style guide.
2. Run corresponding unit tests (if any).
3. Push the changes into a newly-created branch, make a PR to request the merge of these changes into the master branch.
4. Request a reviewer. Reviewer will review the code and give feedback.
5. Ensure Travis tests are passing, this includes both nosetests and flake8.
6. Reviewer merges the changes into the master branch.
7. Delete the branches created for the purpose of making PR. These branches are only good for one-time use.

Variant queries by annotation, return samples with given call, aggregations, etc

Create a 'no auth required' config for the flask app to run the latest versions without needing tyk or keycloak. Would be useful for testing and development cases where we don't need the gateway or want to use login tokens.

The file is too big, we are looking to serve the file from somewhere else, such as from Github.com

Possibly with https://github.com/Legrandin/pycryptodome

distribute does not seem to serve any purpose here

We have quite a few stale branches, it would be good if we can take a look at them and do a bit of clean-up.

Stale branches can be found here: https://github.com/CanDIG/candig-server/branches/stale, if you wish to look at 'your' branches, see https://github.com/CanDIG/candig-server/branches/yours

If one of you started a particular branch yourself, and is certain that this particular branch will not be of use, feel free to delete it yourself. Otherwise please comment on this issue so we can discuss.

If you think your branch could be of potential use in future, leave them as it is! :)

We weren't aware of this issue until recently we ingested ~100 variantsets, indicating a list of 100 variantsets just doesn't seem like a feasible way to do this type of query in the long run

might be helpful to just specify the datasetid in the query, which would then trigger a backend search that returns a list of variantsets which can be used for further search

some questions remained:

1. how to validate query with some or all of datasetId, variantSetId and variantSetIds present? variantSetId and variantSetIds are a one-of, while currently, including datasetId itself doesn't do anything.
2. when theres a lot of variantsets to be searched, the app just silently quits. Reasons unknown as of now.

Currently, travis only succeeds when the nosetests and the flake8 are commented out.

The server setup itself passed. Previously it didn't pass because we have out-of-date pip installed, ensure the pip install pip --upgrade is run so the latest pip in installed.

TODO:

1. Flake8 is the tool that checks the styling and the syntax of our code, and we can choose to ignore certain warning if we feel like so.

2. nosetest is the unit-test package, and currently, all tests that call one of our endpoints will fail due to the new federation. It looks like we might need to rewrite these tests for them to work with federation.

to avoid security vulnerabilities

Swagger is certainly an option, but it does introduce a bunch of unnecessary stuff into the doc (as its compiled from the proto). All of the tier information and the additional_props, if they are of no use, we can/should probably not communicate that back to the users

I suspect that a few packages may not be used anymore, either directly or indirectly. There probably won't be too many of these, however.

E.g., future.

Effectively immediately, the codebase currently sitting in master and develop branches will be freezed until further notice. During the freezing period, no new pull requests or commits will be accepted into these two branches. Security patches will continue to be added into the them, however.

Part of the codebase upgrade project

Master ticket for candig packages update

This issue can be closed when all of the candig-* dependencies, in our case, includes candig-schemas, candig-ingest, and ga4gh-common are all functioning in 3.6 or above.

This is the master ticket for the corresponding codebase upgrade tickets in candig-schemas, candig-ingest, and ga4gh-common.

This issue can be closed when:

• candig-schemas is running okay under 3.6 or above.
  Assignee: @HaoYuanLi
• ga4gh-common is running okay under 3.6 or above, and its project, namespace be changed to candig.
  Assignee: @dashaylan
• candig-ingest is running okay under 3.6 or above.
  Assignee: TBD

1. Should these parameters be mandatory?
2. If so, should we check their validity?

More details to come..

That particular IGV was the minified 2.0.0-rc3 by babel minifier

igv 2.0.0-rc3: Line 39171: "readGroupIds": [self.config.readGroupIds],

it seems to have been written to consume a string, supposedly one readGroupId, our server can certainly handle more - need to look into this

As titled, current status object can be confusing at times, but we probably need more discussion this.

This will be the ticket that will used down the road for candig-server's codebase migration.

This is because in the short-term plan, there is no plan to migrate the candig-client to python 3. In future, if time and dev resources permit, we can revisit this topic.

For now, this issue can be closed when

• All of the candig-client related tests are removed from the test suite.

This issue can be closed when complex queries can return a federated, differentially private counts

https://github.com/CanDIG/ga4gh-server/blob/5c25b18a2e2ee224c746a605d34a28f634d4db8d/ga4gh/server/static/src/custom_visualization.js#L142

Work with GenAP team to demonstrate PoC for data discovery with search API via GenAP portal. This is a stepping stone to a censored discovery API for the CHORD grant. This issue can be closed when the PoC is demonstrated.

Currently, the only querying parameters supported for most POST endpoints are the datasetId and some key (e.g., patient_id), this can be expanded by extending the related request object schema, as well as related Generator method in backend.py; the data model might need some changes as well.

To avoid duplication no further issues will be made under ga4gh-schema

Implement better support for CNV and SV handling in API

Details TBA

By occasionally, I mean around 1% of fail rate （~3/500 under python 2.7; ~5/500 under python 3; no meaningful difference)

This issue does not require any immediate attention yet, I am however still opening an issue just so that I don't forget about it.

The following issues have been reported:

1. The new table with scrolling bar doesn't work well when zoomed out
2. Sometimes the initial load the the table would have their thead and tbody misaligned.
3. The sample analysis table's column has names that are hard to read

More details to come

Related to CanDIG/candig-schemas#2

We are adding a set of new tables to support pipeline metadata, this would involve a few changes in datamodel and backend that define new data model and endpoints.

• Part of the codebase upgrade project

According to the past experience, this is the most difficult part of our upgrade project. There are quite a few things that broke on python 2 when running 0.15.0, needless to mention its compatibility when we run it on python 3.

This issue can be closed when:

• Running pysam==0.15.0

• All the of tests are passing

• Related endpoints are running, this might include some manual testing, e.g., all of the pysam related endpoints are functional

This involves the update of front-end javascript, backend.py and variants_services.proto,

1. in backend.py, a new function would be needed as an entry point to call the generator.
2. after the changes, the front end will no longer consume a stringified body, so the related JSON.parse function needs to be deleted

Update the front-end so the user is able to select two different readGroupSets for the IGV to render a comparison graph between the two.

This would involve the changes in main.js