Whenno sample ID list is supplied via the--ids-file` flag, and a NextSeq run ID is supplied, then we run into this errror:

Traceback (most recent call last):
  File "symlink-seqs", line 627, in <module>
    main(args)
  File "symlink-seqs", line 601, in main
    fastq_paths = get_fastq_paths(config, run_dir, sample_ids)
  File "symlink-seqs", line 501, in get_fastq_paths
    for sample in selected_samples:
  File "symlink-seqs", line 480, in <lambda>
    selected_samples = filter(lambda x: x['sample_id'] in sample_ids, candidate_samples)
TypeError: argument of type 'NoneType' is not iterable

Note that when no --ids-file is supplied, the value of sample_id is None here:

...then passed to the get_fastq_paths() function here:

...then we fail to iterate over it here because None is not iterble:

Add a GitHub Actions-based testing workflow.

We're currently assuming that the fastq files for NextSeq runs are under Analysis/1/Data/fastq:

https://github.com/BCCDC-PHL/symlink-illumina-fastq-by-project-id/blob/8c4e570483054b9d8239aa9fc9b8e2b66ac42064/symlink_illumina_fastq_by_project_id.py#L158-L159

But in cases where demultiplexing has been repeated, additional Analysis sub-directories are created:

Analysis/1/Data/fastq
Analysis/2/Data/fastq
Analysis/3/Data/fastq
...etc

In those cases, the most recent Analysis sub-directory is the one that we should link to.

On NextSeq runs, we're always parsing the SampleSheet from the top-level directory:

https://github.com/BCCDC-PHL/symlink-illumina-fastq-by-project-id/blob/38f06d4384246e6009ca59203d5e806911649952/symlink_illumina_fastq_by_project_id.py#L255

...but we should be parsing the SampleSheet from the Analysis/<LATEST>/Data directory.

There are cases where the user wants a copy of the file instead of a symlink. Support that with a --copy flag.

In the new MiSeq directory structure, there are two SampleSheets:

• At the top-level of the run output directory
• For a specific Alignment (demultiplexing): Alignment_<N>/<DATE_TIMESTAMP>/SampleSheetUsed.csv

...but we're currently only looking for a SampleSheet named SampleSheet.csv in the top-level of the run dir.

We should instead be looking for the SampleSheet for the most recent demultiplexing output dir.

When a sample ID in an illumina SampleSheet includes a underscores (_), they are automatically converted to dashes (-) in the fastq filename. This causes a mismatch when trying to match up SampleSheet entries with fastq files in this tool.

We can compensate for this by automatically converting underscores to dashes in the sample IDs prior to matching.

When I run symlink-seqs on a sample list containing samples, some of which, were on a run that has been failed and then repeated, the symlink generated for those failed samples is currently to the first (failed) run. Is it possible to exclude these runs from the symlinking? At the moment the work around is to identify those samples that were on the failed run and then run a separate sample list providing the -r flag with the run that they were successfully sequenced on.

When a --run-id is supplied with no --ids-file, we get this error:

Traceback (most recent call last):
  File "symlink-seqs", line 618, in <module>
    main(args)
  File "symlink-seqs", line 577, in main
    sample_ids = parse_ids_file(args.ids_file)
  File "symlink-seqs", line 404, in parse_ids_file
    with open(ids_file_path, 'r') as f:
TypeError: expected str, bytes or os.PathLike object, not NoneType

We should support a mode where all samples on a run are symlinked when a run ID is supplied with no sample IDs file.

When running the tool like this:

symlink-seqs -r <SOME_RUN_ID> -p <SOME_PROJECT_ID>

...we get symlinks to all sequences on the run, not only those in the specified project.

Filter the sequences to be symlinked by project ID.

After upgrading to Windows 10, our MiSeq instruments are using a different output directory structure.

This tool should check which directory structure is used for the run before attempting to locate and symlink fastq files. This should be done automatically so the user doesn't need to specify which directory structure is used.

If any NextSeq run is missing an Analysis directory at the top-level of the run dir, we get an error:

FileNotFoundError: [Errno 2] No such file or directory: '/path/to/run/Analysis'

...and the script terminates. That may cause some symlinks not to be created, if the script terminates before reaching those samples.

We should skip the run if there is no Analysis dir at the top-level for NextSeq runs.

In the last pull-request (#28), a debugging print statement was left in and should have been removed before merging:

We often want to symlink all libraries for a specific project, but sometimes we just want to grab all the libraries on a run or just a list of arbitrary libraries regardless of the project. But we're currently excluding any libraries that have an empty project ID:

We shouldn't exclude those libraries.

Ran into this error recently while attempting to create symlinks:

Traceback (most recent call last):
  File "symlink-seqs", line 648, in <module>
    main(args)
  File "symlink-seqs", line 622, in main
    fastq_paths = get_fastq_paths(config, run_dir, sample_ids, args.project_id)
  File "symlink-seqs", line 459, in get_fastq_paths
    fastq_subdir = find_miseq_fastq_subdir(run_dir)
  File "symlink-seqs", line 150, in find_miseq_fastq_subdir
    fastq_dir = fastq_dirs[-1]
IndexError: list index out of range

When running symlink-seqs with no additional command line arguments, the tool will begin symlinking every FASTQ on the system to the current working directory instead of displaying a help message. Not a major issue, but can be troublesome for newer users (and experienced users who forget).

I believe this stems from having default arguments for both the --config and --outdir arguments simultaneously. I think the solution is to remove the default='.' from the --outdir argument, while addingrequired=True.

Old:

New:

 parser.add_argument('-o', '--outdir', required=True, help="Output directory, where symlinks (or copies) will be created.") 

Add a mode where instead of creating symlinks, a csv will be written to stdout with the fields:

ID
R1
R2

These csv files are compatible with several of our pipelines that take input via a --samplesheet_input mode.

Allow the user to provide a file with a list of sample IDs. Only samples matching those sample IDs will be linked.