User metadata can be validated using pandera validation. This will allow for metadata field requirements based on a schema file. This will allow seqsender to automatically detect issues with user metadata. Pandera is a better alternative than hardcoding metadata field validation into seqsender because a schema can be created for each virus with multiple valid options for each field. This can then be easily expanded to include restrictions for other viruses or to roll back restrictions.

Pandera metadata schema files:

• FLU
• COV
• POX
• ARBO
• OTHER
• No Validation

I am getting this error when I try to run the test submit command:
python seqsender.py submit --unique_name test_submission --config test_config.yaml --metadata /root/miniconda3/seqsender/test_input/test_metadata.tsv --fasta /root/miniconda3/seqsender/test_input/test_fasta.fasta --test

The output first says:
Processing test_submission.
Processing Files.
Creating GISAID files.
Creating Genbank files.
Creating BioSample files.
Creating SRA files.
test_submission complete.

Submission report exists pulling down.
Submitting to SRA/BioSample.

Followed by the FTP error described above. Does anything appear to be missing? Thank you

• Clarify in documentation the need for users to go to gisaid.org and download their CLI-API, where to install it for SeqSender to use it and requesting their token.

• Refactor seqsender internal code to import their CLI python pkg and use their commands rather than the raw API. This will assure that changes made by GISAID get inherited to SeqSender more smoothly.

cc: @leebrian @rchau88 @kristinelacek

Setup the Default behavior of submitting to all repos to be GISAID -> NCBI in order to first capture the EPI_SEQUENCE_ID assigned by GISAID and then adding to GenBank Structure Comment field like:

https://www.ncbi.nlm.nih.gov/nuccore/OP845736.1/

COMMENT     ##FluData-START##
            EPI_ISOLATE_ID   :: EPI_ISL_9631596
            NAME             :: A/Wisconsin/01/2022
            TYPE             :: H3
            Segment_name     :: HA
            HOST_GENDER      :: F
            PASSAGE          :: Original
            LOCATION         :: United States / Wisconsin
            COLLECT_DATE     :: 11-Jan-2022
            SPECIMEN_ID      :: 22VR005083 ORIGINAL
            SENDER_LAB       :: Wisconsin State Laboratory of Hygiene
            SEQLAB_SAMPLE_ID :: 3030725183
            EPI_SEQUENCE_ID  :: EPI1981213
            ##FluData-END##
        
FEATURES             Location/Qualifiers
     source          1..1737
                     /organism="Influenza A virus"
                     /mol_type="viral cRNA"
                     /strain="A/Wisconsin/01/2022"
                     /serotype="H3N2"
                     /host="Homo sapiens"
                     /db_xref="taxon:11320"
                     /segment="4"
                     /country="USA: Wisconsin"
                     /collection_date="11-Jan-2022"
                     /note="passage details:Original"

• check-submissions: Allow option to update a single submission instead of updating all submissions in log.
• other organism: Allow any organism to be used with the flag "other" .
  Other is currently added. It doesn't allow for GISAID submissions since it cannot be determined easily which epiCLI to use. The other option is a default generic submission template which will allow for any organism to be submitted to NCBI.
• gisaid: Create gisaid submission as a toggle option to be used with any organism. This will allow automated upload for NCBI but manual submission for gisaid when a CLI option doesn't exist.
  In order to support turning off GISAID submissions for other organisms, all epiCLI's have support added for them. This is to allow for any epiCLI to be connected to seqsender and used without issue. New epiCLI's can be easily added by its information to the internal metadata config file.
  • EpiArbo (Arbovirus)
  • EpiPox (Monkeypox)
• Table2asn submission validation.
  Table2asn submissions are made via email, this prevents seqsender from being able to validate a submission is correct before submitting it. Using the Table2asn validation file seqsender can now parse this file and detect issues which will then prevent submission and notify the user of what issues to correct.
• User config file validation.
  Config files are used to store user info and determine how seqsender processes their submissions. Current checks only validated that it loaded correctly as a yaml file. Now config files are checked against schema files which can determine if a user incorrectly filled out their submission file. If the user did incorrectly fill out their submission file it will now report an error message directing the user to the incorrect field and notifying them of what to change it to.

Hi,

Thanks for creating and maintaining this very useful program.

I am trying to include patient metadata for the NCBI submission part of the process. Similar to gisaid which allows gender, passage etc, NCBI allows source modifiers such as sex.

I am not 100% sure how to include such information for the NCBI part of the metadata in the config file.

genbank_src_metadata:
  column_names:
    isolate: genbank_name
    host: host
    country: location
    isolation-source: isolation_source

Lets say I wanted to include the ncbi source modifier Sex (assuming I have a column in my metadata called gender), would I add the following:

genbank_src_metadata:
  column_names:
  ....
  Sex: gender

Is that correct?

A related question, for the structured data section eg:

COMMENT     ##Assembly-Data-START##
            Assembly Method       :: CLC Genomics
            Sequencing Technology :: PacBio Sequel II
            ##Assembly-Data-END##

How can I add more information than?

My main aim is to match all the metadata that is required in gisaid to ncbi submission.

Thanks,

Ammar

To help users and potential collaborators, please update the readme to follow the flu doc style and explicitly call out how you like folks to submit issues, test issues, commit (eg, dev branches vs main vs releases).

After your changes a user or collaborator will be able to understand how you work on and make changes to seqsender and how to expect to watch for in progress work, completed work and new releases.

Will be very helpful to have step-by-step instructions with screenshots in the documentation for creating an account, highlighting which fields will then be needed later in seqsender.

Creating automated testing for script updates with drastically increase testing time. Testing is tedious when it depends on databases processing files to determine if the changes work correctly.

Environment testing:

• Automatic docker deployment to GHCR
  Master branch now automatically builds and deploys the latest docker image to github container repository.
• Automatic docker testing on pull request
• Automatic python/mamba versioning testing

Functions to be tested:

• create_ncbi_submission
• create_gisaid_submission
• create_submission_xml
• create_submission_status_csv
• create_authorset
• create_fasta
• create_genbank_files
• create_genbank_zip
• save_xml
• create_submission_log
• create_genbank_table2asn
• read_gisaid_log
• get_required_colnames
• get_metadata
• check_credentials
• check_raw_read_files
• process_fasta_samples
• update_submission_status
• update_genbank_files
• get_token
• get_ncbi_process_report
• process_biosample_sra_report
• check_submission_description
• process_genbank_report
• get_execution_time
• get_config
• start
• args_parser
• main
• create_zip_template
• authenticate
• download_table2asn
• submit_ncbi
• submit_gisaid
• sendmail

Additional templates

• New contributor
• Suggest new virus to support

Update existing templates

• Bug Report
• Feature Request
• Maintenance

Info to add to existing templates

• Virus information
• Instrument information
• Database information

• GISAID using their CLI API
• NCBI - BioSample
• NCBI - Genbank
• NCBI - SRA
• NCBI - Table2asn

cc: @leebrian @rchau88 @kristinelacek

Hello, I am trying to do our first production submission and this is the output I am getting.

Traceback (most recent call last):
File "seqsender.py", line 616, in
main()
File "seqsender.py", line 591, in main
submission_preparation.process_submission(args.unique_name, args.fasta, args.metadata, os.path.join(os.path.dirname(os.path.abspath(file)), "config_files", args.config))
File "/root/miniconda3/seqsender/submission_preparation.py", line 493, in process_submission
main_df = merge(fasta_file, metadata_file)
File "/root/miniconda3/seqsender/submission_preparation.py", line 228, in merge
main_df = fasta.merge(metadata, left_on = "fasta_name_orig", right_on = config_dict["general"]["fasta_sample_name_col"], how = "left")
File "/root/miniconda3/envs/seqsender/lib/python3.6/site-packages/pandas/core/frame.py", line 7963, in merge
validate=validate,
File "/root/miniconda3/envs/seqsender/lib/python3.6/site-packages/pandas/core/reshape/merge.py", line 87, in merge
validate=validate,
File "/root/miniconda3/envs/seqsender/lib/python3.6/site-packages/pandas/core/reshape/merge.py", line 652, in init
) = self._get_merge_keys()
File "/root/miniconda3/envs/seqsender/lib/python3.6/site-packages/pandas/core/reshape/merge.py", line 1005, in _get_merge_keys
right_keys.append(right._get_label_or_level_values(rk))
File "/root/miniconda3/envs/seqsender/lib/python3.6/site-packages/pandas/core/generic.py", line 1563, in _get_label_or_level_values
raise KeyError(key)
KeyError: 'specimen_collector_sample_id'

As part of the enhancement to support flu submissions, please use a different group to identify submissions made by flu programs (as opposed to sc2 groups using CDC_OAMD).

Is your feature request related to a problem? Please describe.
The seqsender submission configuration has fields for NCBI username and password. But NCBI accounts are created and logged into vai thied party systems (google etc). How do we obtain an NCBI username/password pair for submission to e.g. Biosample and SRA.

Describe the solution you'd like
Please provide documentation, or a pointer to documentation, detailing how to obtain an NCBI username/passowrd pair that would enable us to make submissions to Biosample, SRA, Genbank

Describe alternatives you've considered
NA

Additional context
We would need the credentials to work for submissions to Biosample and SRA.

Complete step-by-step document on the website with screenshots of how to prepare files and run SS. The users will mainly be non-CLI, so the instructions need to be overly verbose. Write them for FLU use specifically for now.

• Document for setup (define the lab’s configs)
• Documents for operating
  i. Putting metadata into template excel
  ii. SS commands and verifying successful submission.