varfish-org / varfish-cli Goto Github PK

Describe the bug
Python error when running varfish-cli
AttributeError: module 'pydantic' has no attribute 'ConfigDict'

To Reproduce
Steps to reproduce the behavior:

1. Install varfish-cli as described in the readme
2. execute conda activate varfish-cli
3. execute varfish-cli --help

Expected behavior
showing help info.

Additional context

Traceback (most recent call last):
File "/home/humgen/miniconda3/envs/varfish-cli-ex/bin/varfish-cli", line 6, in
from varfish_cli.main import main
File "/home/humgen/miniconda3/envs/varfish-cli-ex/lib/python3.6/site-packages/varfish_cli/main.py", line 3, in
from varfish_cli import cli
File "/home/humgen/miniconda3/envs/varfish-cli-ex/lib/python3.6/site-packages/varfish_cli/cli/init.py", line 10, in
from varfish_cli.cli import cases, importer, projects, tools, varannos
File "/home/humgen/miniconda3/envs/varfish-cli-ex/lib/python3.6/site-packages/varfish_cli/cli/cases.py", line 8, in
from varfish_cli import api, common
File "/home/humgen/miniconda3/envs/varfish-cli-ex/lib/python3.6/site-packages/varfish_cli/api/init.py", line 3, in
from varfish_cli.api.case import * # noqa: F403, F401
File "/home/humgen/miniconda3/envs/varfish-cli-ex/lib/python3.6/site-packages/varfish_cli/api/case.py", line 12, in
from varfish_cli.api import models
File "/home/humgen/miniconda3/envs/varfish-cli-ex/lib/python3.6/site-packages/varfish_cli/api/models.py", line 20, in
class Project(pydantic.BaseModel):
File "/home/humgen/miniconda3/envs/varfish-cli-ex/lib/python3.6/site-packages/varfish_cli/api/models.py", line 21, in Project
model_config = pydantic.ConfigDict(frozen=True)
AttributeError: module 'pydantic' has no attribute 'ConfigDict'

Describe the bug
DB info files during the import do not distinguish between smallvars and SVs import. This causes problems downstream in VarFish as it maintains the info for both.

To Reproduce
See ticket in VarFish Server repos (link below).

Expected behavior
Distinguish between smallvars and SVs database info during file guessing, or at least before triggering the import in VarFish Server.

Additional context
Related issue in VarFish Server repository: varfish-org/varfish-server#497

Is your feature request related to a problem? Please describe.
As the interface for varfish-cli is growing (and also its integration into cubi-tk and such) it could make sense to look how other projects have solved similar problems. These include composing single tools from multiple packages.

Describe the solution you'd like
Look into pbr from the OpenStack project for a potential solution that can reduce boilerplate and simplify integrating tools.

Describe alternatives you've considered
N/A

Additional context
N/A

Is your feature request related to a problem? Please describe.
#70 adds support for bootstrapping phenopackets YAML files. However, the handling of quality control files and read alignment files is still unimplemented.

Describe the solution you'd like
Add support for automatically detecting quality control file types and support for interpreting BAM files.

Describe alternatives you've considered
N/A

Additional context

• #70

Is your feature request related to a problem? Please describe.
The levenshtein library that we use is GPL licensed. polyleven is not.

Describe the solution you'd like
Switch dependencies.

Describe alternatives you've considered
N/A

Additional context

• https://github.com/fujimotos/polyleven

Is your feature request related to a problem? Please describe.
When trying to import a case to a category or listing the content of a category, VarFish throws now a 400 error. This should be handled by the varfish-cli. Currently the error message is cryptic for the user:

$ varfish-cli case list-import-info 170ed9f8-630b-4ce6-a3f5-928383a9b7f4
[I 211203 16:19:47] Configuration: CaseListImportInfoConfig(case_config=CaseConfig(global_config=CommonConfig(verbose=False, verify_ssl=True, varfish_api_token='4c2d************************************************************', varfish_server_url='http://localhost:8000/')), project_uuid=UUID('170ed9f8-630b-4ce6-a3f5-928383a9b7f4'), owner=None)
[I 211203 16:19:47] Listing cases
Traceback (most recent call last):
  File "/home/oliver/.local/bin/varfish-cli", line 11, in <module>
    load_entry_point('varfish-cli', 'console_scripts', 'varfish-cli')()
  File "/home/oliver/dev/varfish-cli/varfish_cli/__main__.py", line 109, in main
    res = cmds[args.cmd](
  File "/home/oliver/dev/varfish-cli/varfish_cli/case/__init__.py", line 31, in run
    return args.case_cmd(config, toml_config, args, parser, subparser)
  File "/home/oliver/dev/varfish-cli/varfish_cli/case/list_case_import.py", line 27, in run
    res = api.case_import_info_list(
  File "/home/oliver/dev/varfish-cli/varfish_cli/api/case.py", line 123, in case_import_info_list
    " ".join([" ".join(v) for v in result.json().values()]),
AttributeError: 'list' object has no attribute 'values'

Describe the solution you'd like
The response by the server should be caught and a proper error message should be displayed, like "Pushing case failed" (probably the reason might not be visible as VarFish itself isn't descriptive itself.

Describe alternatives you've considered
None.

Additional context
On VarFish side it looks like this:

Bad Request: /importer/api/case-import-info/170ed9f8-630b-4ce6-a3f5-928383a9b7f4/
[03/Dec/2021 15:19:47] "GET /importer/api/case-import-info/170ed9f8-630b-4ce6-a3f5-928383a9b7f4/ HTTP/1.1" 400 58

Describe the bug
Listing cases of a project results in a 404.

To Reproduce
Steps to reproduce the behavior:

1. list cases of a project
2. see error

Expected behavior
Display list of cases instead of 404.

Additional context

$ varfish-cli case list aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
[I 230403 16:39:32] Configuration: CaseListConfig(case_config=CaseConfig(global_config=CommonConfig(verbose=False, verify_ssl=True, varfish_api_token='****************************************************************', varfish_server_url='https://varfish.bihealth.org/'), output_file='-', output_format=<OutputFormat.TABLE: 'table'>, output_delimiter=',', output_fields=('sodar_uuid', 'name', 'index', 'members')), project_uuid=UUID('aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa'))
[I 230403 16:39:32] Listing cases
Traceback (most recent call last):
  File "/home/stolpeo/dev/miniconda3/envs/varfish-cli/bin/varfish-cli", line 10, in <module>
    sys.exit(main())
  File "/home/stolpeo/dev/miniconda3/envs/varfish-cli/lib/python3.8/site-packages/varfish_cli/__main__.py", line 112, in main
    res = cmds[args.cmd](
  File "/home/stolpeo/dev/miniconda3/envs/varfish-cli/lib/python3.8/site-packages/varfish_cli/case/__init__.py", line 94, in run
    return args.case_cmd(config, toml_config, args, parser, subparser)
  File "/home/stolpeo/dev/miniconda3/envs/varfish-cli/lib/python3.8/site-packages/varfish_cli/case/list_case.py", line 39, in run
    res = api.case_list(
  File "/home/stolpeo/dev/miniconda3/envs/varfish-cli/lib/python3.8/site-packages/varfish_cli/api/case.py", line 142, in case_list
    result.raise_for_status()
  File "/home/stolpeo/dev/miniconda3/envs/varfish-cli/lib/python3.8/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://varfish.bihealth.org/variants/api/case/list/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa/

Might be an VarFish issue instead.

Is your feature request related to a problem? Please describe.
The latest version of varfish-annotator needs additional comments (see varfish-org/varfish-annotator#41).

Describe the solution you'd like
Implement support for recognizing this to varfish-cli.

Describe alternatives you've considered
N/A

Additional context

• varfish-org/varfish-annotator#41

Release for version v0.3.5

• edit HISTORY.rst and ensure a proper section is added

• create a git tag v.MAJOR.MINOR.PATCH and git push --tags

• create a "Github release` based on the tag with the text

  ```
  All details can be found in the `HISTORY.rst` file.
  ```
  

The following command returns the following error:

 varfish-cli --no-verify-ssl case list f2acceb7-067d-41a4-8e39-236c022678f1

requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://varfish.bihealth.org/variants/api/case/list/f2acceb7-067d-41a4-8e39-236c022678f1/

@holtgrewe Did the Varfish API for listing cases change?

Is your feature request related to a problem? Please describe.
While attrs is a nice package for defining data classes, pydantic can perform validation and arguably plays better with the Python type (annotation) system.

Describe the solution you'd like
Migrate the code to using pydantic rather than attrs.

Describe alternatives you've considered
N/A

Additional context
N/A

• describe setup with configuration file
• point to the VarFish server API documentation
• explain how to upload files
• point to VarFish server for files to upload

Is your feature request related to a problem? Please describe.
The package varfish-cli does not use API versioning. This makes errors hard to track.

Describe the solution you'd like
Implement API versioning as exposed by the VarFish server.

Describe alternatives you've considered
N/A

Additional context

• varfish-org/varfish-server#333

Issue
Error while trying to validate empty CNV files:

[I 220517 15:24:52] ... checking genomebuild consistency ...
Traceback (most recent call last):
  File "MINICONDA/bin/cubi-tk", line 33, in <module>
    sys.exit(load_entry_point('cubi-tk', 'console_scripts', 'cubi-tk')())
  File "PATH/cubi-tk/cubi_tk/__main__.py", line 123, in main
    res = cmds[args.cmd](args, parser, subparsers.choices[args.cmd] if args.cmd else None)
  File "PATH/cubi-tk/cubi_tk/snappy/__init__.py", line 97, in run
    return args.snappy_cmd(args, parser, subparser)
  File "PATH/cubi-tk/cubi_tk/snappy/varfish_upload.py", line 263, in run
    return SnappyVarFishUploadCommand(args).execute()
  File "PATH/cubi-tk/cubi_tk/snappy/varfish_upload.py", line 180, in execute
    self._process_dataset(dataset)
  File "PATH/cubi-tk/cubi_tk/snappy/varfish_upload.py", line 254, in _process_dataset
    varfish_cli_main(args[1:])
  File "MINICONDA/lib/python3.8/site-packages/varfish_cli/__main__.py", line 109, in main
    res = cmds[args.cmd](
  File "MINICONDA/lib/python3.8/site-packages/varfish_cli/case/__init__.py", line 31, in run
    return args.case_cmd(config, toml_config, args, parser, subparser)
  File "MINICONDA/lib/python3.8/site-packages/varfish_cli/case/create_case_import.py", line 861, in run
    return CaseImporter(config).run()
  File "MINICONDA/lib/python3.8/site-packages/varfish_cli/case/create_case_import.py", line 281, in run
    self._check_genomebuild_consistency()
  File "MINICONDA/lib/python3.8/site-packages/varfish_cli/case/create_case_import.py", line 433, in _check_genomebuild_consistency
    line = inputf.readline().splitlines(keepends=False)[0].split("\t")

To Reproduce
Steps to reproduce the behavior:

1. Go to 'NAMSE' directory
2. Run cubi-tk snappy varfish-upload --min-batch 210 --yes <UUID>
   Batch contains empty file for XHMM:
   .gts.tsv.gz

release<tab>chromosome<tab>chromosome_no<tab>start<tab>end<tab>bin<tab>start_ci_left<tab>start_ci_right<tab>end_ci_left<tab>end_ci_right<tab>case_id<tab>set_id<tab>sv_uuid<tab>caller<tab>sv_type<tab>sv_sub_type<tab>info<tab>genotype

.feature-effects.tsv.gz

case_id<tab>set_id<tab>sv_uuid<tab>refseq_gene_id<tab>refseq_transcript_id<tab>refseq_transcript_coding<tab>refseq_effect<tab>ensembl_gene_id<tab>ensembl_transcript_id<tab>ensembl_transcript_coding<tab>ensembl_effect

Expected behavior
Expected to validate header as to check the pipeline ran correctly. Ignore line if file is contains no entries.

Release for version vVERSION

• edit HISTORY.rst and ensure a proper section is added

• create a git tag v.MAJOR.MINOR.PATCH and git push --tags

• create a "Github release` based on the tag with the text

  ```
  All details can be found in the `HISTORY.rst` file.
  ```
  

Describe the bug
Creating a small variant query (from a shortcut) results in an error.

To Reproduce
Steps to reproduce the behavior:

1. Use small-var-query-shortcut to create a query using recessive
2. See error

Expected behavior
A query will be created.

Additional context

[I 230404 09:35:29] Creating query
  + Exception Group Traceback (most recent call last):
  |   File "/home/stolpeo/dev/miniconda3/envs/varfish-cli/bin/varfish-cli", line 10, in <module>
  |     sys.exit(main())
  |   File "/home/stolpeo/dev/miniconda3/envs/varfish-cli/lib/python3.8/site-packages/varfish_cli/__main__.py", line 112, in main
  |     res = cmds[args.cmd](
  |   File "/home/stolpeo/dev/miniconda3/envs/varfish-cli/lib/python3.8/site-packages/varfish_cli/case/__init__.py", line 94, in run
  |     return args.case_cmd(config, toml_config, args, parser, subparser)
  |   File "/home/stolpeo/dev/miniconda3/envs/varfish-cli/lib/python3.8/site-packages/varfish_cli/case/small_var_query_settings_shortcut.py", line 36, in run
  |     res = api.small_var_query_settings_shortcut(
  |   File "/home/stolpeo/dev/miniconda3/envs/varfish-cli/lib/python3.8/site-packages/varfish_cli/api/case.py", line 977, in small_var_query_settings_shortcut
  |     return CONVERTER.structure(result.json(), QueryShortcutsResultV1)
  |   File "/home/stolpeo/dev/miniconda3/envs/varfish-cli/lib/python3.8/site-packages/cattrs/converters.py", line 309, in structure
  |     return self._structure_func.dispatch(cl)(obj, cl)
  |   File "<cattrs generated structure varfish_cli.api.models.QueryShortcutsResultV1>", line 14, in structure_QueryShortcutsResultV1
  |     if errors: raise __c_cve('While structuring ' + 'QueryShortcutsResultV1', errors, __cl)
  | cattrs.errors.ClassValidationError: While structuring QueryShortcutsResultV1 (1 sub-exception)
  +-+---------------- 1 ----------------
    | Exception Group Traceback (most recent call last):
    |   File "<cattrs generated structure varfish_cli.api.models.QueryShortcutsResultV1>", line 5, in structure_QueryShortcutsResultV1
    |     res['presets'] = __c_structure_presets(o['presets'], __c_type_presets)
    |   File "<cattrs generated structure varfish_cli.api.models.QuickPresetsV1>", line 39, in structure_QuickPresetsV1
    |     if errors: raise __c_cve('While structuring ' + 'QuickPresetsV1', errors, __cl)
    | cattrs.errors.ClassValidationError: While structuring QuickPresetsV1 (1 sub-exception)
    | Structuring class QueryShortcutsResultV1 @ attribute presets
    +-+---------------- 1 ----------------
      | Traceback (most recent call last):
      |   File "<cattrs generated structure varfish_cli.api.models.QuickPresetsV1>", line 30, in structure_QuickPresetsV1
      |     res['flags_etc'] = __c_structure_flags_etc(o['flags_etc'])
      | KeyError: 'flags_etc'
      | Structuring class QuickPresetsV1 @ attribute flags_etc
      +------------------------------------

Is your feature request related to a problem? Please describe.
The new-style imports (based on depositing files in an external storage and registering the case as phenopackets) is currently unsupported in VarFish.

Describe the solution you'd like
Implement the import.

• implement parsing of per-project TOML configuration
• implement varfish-cli projects project-load-config PROJECT_UUID (see below)
• implement validating phenopackets YAML files
• implement submission of phenopackets YAML files

Describe alternatives you've considered
N/A

Additional context
N/A

Describe the bug
When running the "dragen-to-bam-qc" tool, it writes out the wrong JSON (serialized BamQc and not serialized BamQc.sample_data.

To Reproduce
N/A

Expected behavior
It should write out sample_data.

Additional context
N/A

Is your feature request related to a problem? Please describe.
Once, there is sodar-core-cli (cf. bihealth/sodar-cli#4) we should add support to project manipulation (and others?) to varfish-cli

Describe the solution you'd like
Once there is sodar-core-cli we should give users access to project manipulation via REST API using varfish-cli.

Describe alternatives you've considered
N/A

Additional context
N/A

Is your feature request related to a problem? Please describe.
We have slightly changed the CLI interface. This is not documented in the varfish-server manual.

Describe the solution you'd like
Update the manual accordingly.

Describe alternatives you've considered
N/A

Additional context
N/A

Describe the bug
When trying to import GRCh38 data, the following error occurs:
[E 230329 14:36:33 create_case_import:312] Inconsistent genome build from file: GRCh38 and from args: GRCh37

Using varfish-cli --help only lists the following options:

positional arguments:
  {case,varannos}
    case                Work with cases.
    varannos            Work with varannos module.

optional arguments:
  -h, --help            show this help message and exit
  --verbose             Increase verbosity.
  --version             show program's version number and exit

Basic Configuration:
  --no-verify-ssl       Disable HTTPS SSL verification
  --config CONFIG       Path to configuration file.
  --varfish-server-url VARFISH_SERVER_URL
                        VarFish server URL key to use, defaults to env VARFISH_SERVER_URL.
  --varfish-api-token VARFISH_API_TOKEN
                        VarFish API token to use, defaults to env VARFISH_API_TOKEN.

The option --genomebuild can be found in the source code, but it is not listed when using the help function. Especially when more users are switching to hg38, this option is going to be needed.

To Reproduce
Steps to reproduce the behavior:

1. Use GRCh38 when annotating vcf files
2. Try to load into varfish using varfish-cli

Expected behavior
Amend help function to show all available arguments.

Is your feature request related to a problem? Please describe.
Allow users to upload the file as described in varfish-org/varfish-server#575.

Describe the solution you'd like
Implement a new client endpoint and support for uplading gene-annotations.tsv files.

Describe alternatives you've considered
N/A

Additional context

• varfish-org/varfish-server#575

Add a method for obtaining case status information to varfish cli

Is your feature request related to a problem? Please describe.
We don't use release-please for varfish-cli.

Describe the solution you'd like
Implement release-please.

Describe alternatives you've considered
N/A

Additional context
N/A

Describe the bug
The command line switch --no-verify-ssl is ignored in the query CLI.

To Reproduce
Steps to reproduce the behavior:

1. Setup server without proper SSL certificate
2. List queries for a case
3. See certificate validation error

Expected behavior
The --no-verify-ssl flag should be interpreted.

Additional context
N/A

Is your feature request related to a problem? Please describe.
While argparse is good for simple CLIs, for multi-command projects, approaches like click or typer work better.

Describe the solution you'd like
Migrate the code to typer as this gives better composition of APIs plus type safety.

Describe alternatives you've considered
N/A

Additional context
N/A

Is your feature request related to a problem? Please describe.
We currently cannot import DRAGEN QC files.

Describe the solution you'd like
We need a way to convert from the DRAGEN QC TSV files to the legacy *.bam-qc.tsv format. Input should be the necessary Dragen QC files:

• <sample>.qc-coverage-region-1_coverage_metrics.csv
  • COVERAGE SUMMARY,,Aligned bases
  • COVERAGE SUMMARY,,Average alignment coverage over QC coverage region
  • COVERAGE SUMMARY,,Aligned bases in QC coverage region
  • both above => estimate target size
  • COVERAGE SUMMARY,,PCT of QC coverage region with coverage [$BOUNDx: inf)
• <sample>.mapping_metrics.csv
  • MAPPING/ALIGNING SUMMARY,,Total input reads
  • MAPPING/ALIGNING SUMMARY,,Number of duplicate marked reads
  • MAPPING/ALIGNING SUMMARY,,Insert length: mean
  • MAPPING/ALIGNING SUMMARY,,Insert length: standard deviation

The output will be a TSV file with postgres-read JSON, as shown below.

The tool should write out gzip-ed data if the ending .gz is used.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
TSV format is

case_id\tset_id\tbam_stats
.\t.\t.<JSON with '"' replaced by '"""' (triple-quotes)

JSON format for one sample (just add more samples by the key)

{
  "<sample>": {
    "summary": {
      "mean coverage": 144.00357104543855,
      "total target size": 39905961
    },
    "min_cov_target": {
      "5": 99.64335992491787,
      "15": 99.33895089058005,
      "20": 99.1112561973354,
      "30": 98.35921082161876,
      "50": 94.4133189153898,
      "100": 60.479872686838185,
      "200": 14.367208699732725,
      "300": 3.3456429926754128,
      "400": 1.0507416399730682
    },
    "bamstats": {
      "sequences": 96438832,
      "reads duplicated": 5501986,
      "insert size average": 197.7,
      "insert size standard deviation": 60.1,
    }
  }
}

The following two lines need to be correct to allow for a correct installation on my setup (conda with python=3.10.8):
https://github.com/bihealth/varfish-cli/blob/e713cc27b037d0d3861a37dfc22925884a529a89/requirements/base.txt#L5
https://github.com/bihealth/varfish-cli/blob/e713cc27b037d0d3861a37dfc22925884a529a89/requirements/base.txt#L25

Pull request will follow now.

Is your feature request related to a problem? Please describe.
varfish-org/varfish-server#747 defines a REST API for defining variant annotation sets and entries. There is no client yet.

Describe the solution you'd like
Add basic CLI functionality to varfish-cli.

Describe alternatives you've considered
N/A

Additional context
N/A

Is your feature request related to a problem? Please describe.
Varfish-cli does not support querying variants from VarFish via the REST API (varfish-org/varfish-server#332).

Describe the solution you'd like
Implement the REST API.

Describe alternatives you've considered
N/A

Additional context

• varfish-org/varfish-server#332

Test coverage should be increased. Requests to VarFish should be mocked out with requests_mock. The existing test code already shows how to do this for the case lists.