rtiinternational / biocloud_docker_tools Goto Github PK

Update Docker Hub repo from rticode to rtibiocloud repos in WDL workflows.

Known workflows:

• rnaseq-pipeline
• biocloud_wdl_tools
• metaxcan-pipeline
• biocloud_gwas_workflows
• local_ancestry_workflows
• ld-regression-pipeline

Create script that converts summary stats from rvtests or genesis to standard summary stat format defined here.

Build failing for dexseq/v3.9

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
The command '/bin/sh -c echo 'deb http://mirror.math.princeton.edu/pub/ubuntu/ bionic main' >> /etc/apt/sources.list     && apt-get update     && apt-get install -y --no-install-recommends         dirmngr         software-properties-common         lsb-release         ed         less         locales         vim         wget         ca-certificates         fonts-texgyre         make         libglib2.0     && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 100

Also:

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-Zzfaoz/HTSeq/
The command '/bin/sh -c apt-get update     && apt-get install -y --no-install-recommends         python-dev         python-pip     && pip install setuptools     && pip install wheel     && pip install pysam     && pip install HTSeq     && apt-get autoremove     && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 1

From https://github.com/RTIInternational/biocloud_docker_tools/runs/1875187921?check_suite_focus=true

Currently, if a file is moved from one directory to another, both files are listed in the changed files list for the GitHub action and the action fails due to multiple Dockerfiles in the list or multiple directories in the same commit.

Create branch in workflow repos and use ECR Docker image location.

Known workflows:

• rnaseq-pipeline
• biocloud_wdl_tools
• metaxcan-pipeline
• biocloud_gwas_workflows
• local_ancestry_workflows
• ld-regression-pipeline

For whatever reason genesis output files are kind of stupid and we need to account for a couple more cases than with RVTests.

1). Genesis does not include the REF/ALT allele in the output so we need to first merge the genesis output file with the info file by chr/pos to add additional information (REF/ALT). Once we have these, we need to create the variant ID like we do normally and everything can proceed like with RVTests output

2. Genesis sumstats output files apparently do not always have the Est. (beta value) columns. My genesis output from the handedness study (binary trait) only have the Score columns (z-score). Need to account for this corner case when beta-value columns are missing. Also should perhaps let the user choose whether they want to use z-score or betas. Perhaps default to beta's and only get z-scores for binary traits?

I changed the version in this repo from v1.39 to v3.9 since that's the version number used in the install command of the Dockerfile.

I moved the dexseq Dockerfile from the code_docker_lib and called it v3.11.

I'm not sure if these version numbers are correct. Does v3.9 and v3.11 refer to BiocManager or DEXSeq? If they are not correct, what are the correct version numbers?

• Archive code_docker_lib. All updates should be in this repo. ` should be used as the new source of truth.
• Add missing docker images.
• Compare existing images.
• Verify all images in rticode Docker Hub are built and exist in rtibiocloud from Dockerfile's in this repo.

Based on discussions with @ngaddis regarding relatedness filtering for GWAS, the implementation of this QC process will be refactored. Currently this process uses a combination of PLINK, an in-house R script for graph-based sample selection, and KING (for AAs only).

The new implementation will be based on recommendations by a GENESIS relatedness workflow and a GWASTools best practices guide. The main enhancements will be the simplification to only use R, and the use of the KING algorithm for all ancestries.

Previously it was discussed offline whether KING was a suitable method for admixed individuals. According to this publication, it is biased but will generally be suitable for QC purposes:

KING-robust was developed for relatedness inference in samples from populations with discrete substructure, and it is a consistent estimator of the kinship coefficient for a pair of outbred individuals from the same subpopulation. This estimator, however, will generally be negatively biased for pairs of individuals that have different ancestries. Despite this bias, the KING-robust estimator is typically able to separate close relatives with similar ancestry from unrelated individuals

Not able to build the Dockerfile dexseq/v1.30.0/Dockerfile:

Collecting pysam
  Downloading https://files.pythonhosted.org/packages/25/7e/098753acbdac54ace0c6dc1f8a74b54c8028ab73fb027f6a4215487d1fea/pysam-0.15.4.tar.gz (1.6MB)
    Complete output from command python setup.py egg_info:
    # pysam: no cython available - using pre-compiled C
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-mVBXzi/pysam/setup.py", line 193, in <module>
        .format(fn))
    ValueError: no cython installed, but can not find pysam/libchtslib.c.Make sure that cython is installed when building from the repository
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-mVBXzi/pysam/
The command '/bin/sh -c apt-get update     && apt-get install -y --no-install-recommends         python-dev         python-pip     && pip install setuptools     && pip install wheel     && pip install pysam     && pip install HTSeq     && apt-get autoremove     && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 1

See GitHub Actions run for more info: https://github.com/RTIInternational/biocloud_docker_tools/runs/465555645?check_suite_focus=true

Not able to build Dockerfile plink/v2.0/Dockerfile:

Step 9/12 : RUN wget http://s3.amazonaws.com/plink2-assets/plink2_linux_x86_64_20190221.zip     && unzip plink2_linux_x86_64_20190221.zip -d /plink_tmp     && cp /plink_tmp/plink2 /opt/     && rm -rf plink2_linux_x86_64_20190221.zip plink_tmp
 ---> Running in 64f7413551da
--2020-02-24 19:57:38--  http://s3.amazonaws.com/plink2-assets/plink2_linux_x86_64_20190221.zip
Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.162.85
Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.162.85|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-02-24 19:57:38 ERROR 404: Not Found.

The command '/bin/sh -c wget http://s3.amazonaws.com/plink2-assets/plink2_linux_x86_64_20190221.zip     && unzip plink2_linux_x86_64_20190221.zip -d /plink_tmp     && cp /plink_tmp/plink2 /opt/     && rm -rf plink2_linux_x86_64_20190221.zip plink_tmp' returned a non-zero code: 8

See GitHub Action for more info: https://github.com/RTIInternational/biocloud_docker_tools/runs/465562858?check_suite_focus=true

Mirror all images created from Docker files in this repo to ECR for the WDL workflows.

Known workflows:

• rnaseq-pipeline
• biocloud_wdl_tools
• metaxcan-pipeline
• biocloud_gwas_workflows
• local_ancestry_workflows
• ld-regression-pipeline

https://github.com/RTIInternational/biocloud_docker_tools/runs/2098521249?check_suite_focus=true