geo-omics / glamr_omics_pipelines Goto Github PK

Multi-omics pipeline for the GLAMR database

Python 48.28% Perl 40.80% Jupyter Notebook 1.02% Shell 0.03% R 9.87%

glamr_omics_pipelines's Introduction

GLAMR: The Great Lakes Atlas of Multi-omics Research

This repository contains the pipleines that power the GLAMR database.

GLAMR is designed to be a centralized resource housing Great Lakes omics datasets analyzed with standardized pipelines and integrated with environmental data.

Folder structure

GLAMR
├── code
├── config
│   ├── conda_yaml (for installing software dependencies)
│   └── profiles (for snakemake)
├── data
│   ├── omics [primary data directory]
│   │   └── {sample_type}
|   |       └── {sample_directories}
│   ├── projects [links to directories in the data/omics folder]
|   |   └── project_name
|   |       └── {sample_type}
|   |           └── {sample_directories}
│   ├── reference
│   │   ├── amplicons
│   │   ├── genomes
│   │   ├── MAGs
│   │   └── UMRAD
│   |── sample_metadata
|   └──environment
└── ReadMe.md

Snakemake

Snakemake is used extensively to manage the workflows used in this database.

Pipeline overview

To Add:

Instructions for importing samples
Instructions for running individual pipelines and all pipelines together

glamr_omics_pipelines's People

Contributors

Stargazers

Watchers

glamr_omics_pipelines's Issues

Amplicon processing pipeline

Accommodate non-paired metagenomic data

Export TPM files based on UniRef100 read mappings

"checkout" for pipeline output files to support downstream file consumption

Whenever the pipline finishes successfully writing a file intended for downstream consumption, I'd like it to append a line to a special "checkout" file denoting the last modification time and the relative path, resulting in a file e.g.:

2023-11-04 06:03:40.380944-04:00 metagenomes/samp_4452/assembly/megahit_noNORM/final.contigs.renamed.fa
2024-03-01 23:29:51.935972-05:00 metagenomes/samp_4452/samp_4452_lca_abund_summarized.tsv
2023-11-03 17:59:00.514943-04:00 metagenomes/samp_4453/assembly/megahit_noNORM/final.contigs.renamed.fa
2024-03-04 20:52:55.342476-05:00 metagenomes/samp_4453/samp_4453_lca_abund_summarized.tsv
2023-11-04 07:02:04.006604-04:00 metagenomes/samp_4454/assembly/megahit_noNORM/final.contigs.renamed.fa
2024-03-04 17:52:41.123889-05:00 metagenomes/samp_4454/samp_4454_lca_abund_summarized.tsv
2023-11-04 00:01:04.829823-04:00 metagenomes/samp_4455/assembly/megahit_noNORM/final.contigs.renamed.fa
2024-03-05 19:19:19.675924-05:00 metagenomes/samp_4455/samp_4455_lca_abund_summarized.tsv
2023-11-04 00:01:09.835734-04:00 metagenomes/samp_4456/assembly/megahit_noNORM/final.contigs.renamed.fa
2024-03-04 21:53:16.752722-05:00 metagenomes/samp_4456/samp_4456_lca_abund_summarized.tsv

The timestamp should be formatted (with python) via:
str(datetime.fromtimestamp(Path('/path/to/file').stat().st_mtime).astimezone())

The path is relative to .../data/omics/

If the pipeline later overwrites a file, a new line denoting the new modtime is appended. If a file needs to be withdrawn this can be communicated by adding a line with an empty timestamp.

Motivation: allows concurrent writing/reading of the omics data

Long read metagenome pipeline

NOAA 'Omics Tags

As a part of the NOAA 'Omics Technical Portal we are trying to add relevant tags to different NOAA 'Omics related software on Github. Would you be able to the following tags to your NOAA 'Omics repositories so we can link and search them effectively: noaa-omics-software.

Thanks for your help!

Best,

Zack

Better version tracking of software deps and outputs

Currently nearly all GLAMR outputs were built with the same software versions and reference databases, however we need to improve the mechanisms for tracking versions of software and reference databases used to create output files. Particularly important as we want to update tools and reference databases.

The question will be how to do this, an initial thought:

Software versions for the GLAMR pipelines are nearly all defined in conda envs, which makes it easy to know the current version and to update versions. By updating the conda env yaml definition files to have versions numbers in their titles, we could use Snakemake's ability to use wildcards in these filenames and then write output files with the version number in the name. The most recent version could then be symlinked to a default location which is always expected to be the most up-to-date version .