We've had some issues getting our pangolin installation to update cleanly when using both the --update_pangolin and --update_pangolin_data flags.

I suspect that the problem is caused by two pip install commands being run in the same conda environment, possibly simultaneously. The processes update_pangolin and update_pangolin_data should be prevented from running in parallel.

The pipeline currently produces two output files, named pangolin_lineages_with_incomplete.csv and pangolin_lineages.csv.

Create only one output file, called pangolin_lineages.csv that includes both incomplete and complete genomes.

As of v3.x, pangolin has a new output format. Adjust this pipeline to parse the new pangolin output format.

As of v3.x, pangolin provides two separate update-related flags: --update and --update-data. The --update flag is used to update the pangolin tool, while the --update-data flag is used to update the pangoLEARN model and other datasets related to scorpio.

Change this pipeline's --update flag to --update_pangolin.

Add a --update_pangolin_data flag.

The current implementation of this pipeline has a fairly rigid dependency on the number of fields and order of fields in the pangolin output.

In particular, this command relies on the ordering and number of fields in pangolin output remaining unchanged:

This makes the pipeline overall somewhat brittle if there are changes to the pangolin output. In general it is prefereable to parse the output by field header, allow extra fields or missing fields to be handled gracefully and not to make assumptions about the order of fields in the output.

Add a script that will allow us to easily compare outputs from different pangolin/pangoLEARN/scorpio versions to identify which lineage calls have changed.

Request to include the positives in the pangolin pipeline results (for positive control QC - ensure they aren't a different lineage than expected).

Negatives probably not important since we monitor % completeness for those.

This will also allow John/end users to check positives in our merged output file with updated lineages.

Run pangolin --update on each pipeline invocation

Only include outputs when genome completeness is above a threshold, default to 85%.

The consensus sequences generated by BCCDC-PHL/ncov2019-artic-nf are deposited to two directories. The consensus sequences generated by ivar are stored to <outdir>/ncovIllumina_sequenceAnalysis_makeConsensus, while those generated by freebayes are stored to ncovIllumina_sequenceAnalysis_callConsensusFreebayes

Add a parameter --ivar_consensus that will select the ivar-generated consensus sequence. Otherwise, the freebayes-generated sequence will be used.

As we update our BCCDC-PHL/ncov2019-artic-nf pipeline, we produce output directories that are named with the pipeline version (eg. `ncov2019-artic-nf-v1.3-output).

Originally, this pipeline would use a parameter to choose the artic output directory version to use for analysis. But we are now at a point where we don't have a results for a single ARTIC analysis version across all of our runs.

Instead, choose the most recent ARTIC analysis output directory (based on version number).

Modify the pipeline to support pangolin-4.0+

Before running pangolin, we use the prepare_multi_fasta process to create a multi-fasta file for each run. This is done by looking at the run's qc.csv file that is generated by the ncov2019-artic-nf pipeline:

In a case where an analysis output directory exists, but the qc.csv file does not exist or is empty, one of the outputs (<run_id>.consensus.fa) of prepare_multi_fasta will not be created, causing the pipeline to fail.

The pangolin recipe in bioconda is up-to-date with its dependencies so it is no longer necessary to list usher and the pip-installed packages separately.

Simplify the environment.yaml file to only list pangolin as a dependency.

Pangolin allows the user to specify which directory to use as the tempdir. By default it will use the system's $TEMDIR. But on some systems that directory may be fairly low capacity, and filling it can cause problems for other processes on the system.

Use the nextflow work dir as the tempdir.

Include a field genome_completeness for each sample in the output file.

The value should be calculated as 100 - pct_N_bases (from the ARTIC qc.csv file).

Bump the version of pangolin that's installed in the pipeline's conda env.

In practice, we will also update pangolin directly within the env on a regular basis as updates to pangolin are released.

As of pangolin v2.4.2, the pangolin_version field is included in the standard pangolin output. This pipeline had been adding that field manually, but that is no longer necessary. Currently, the pangolin_version field is duplicated, and the note field is missing from the output.

Remove the duplicate pangolin_version field from the output and restore the output format.

Include records for all samples on the sequencing run in the output. This includes:

• Samples whose consensus files are below the genome completeness threshold
• Samples that failed to produce a consensus sequence

...but does not include:

• Positive and negative control samples

For samples that are below the genome completeness threshold or don't have a consensus sequence, report their lineage as NA. Also add a value to the note field to explain why the lineage is NA.

There have been a few pangolin updates recently that have introduced changes that cause this pipeline to break. It would be useful to have a flag that could control whether or not pangolin will be updated before the pipeline were run. With that flag available, we could easily temporarily turn off pangolin updates while preparing the rest of the pipeline for the update.