• Change the TSV export to all available columns
• Change the instructions on the submission page:

Virus metadata is submitted as a .tsv file. Viral genome data must be submitted as a .fasta file. Up to 5000 samples can be submitted in a single submission, but note that the larger the file the longer the submission will take. FASTA files are accepted individually, or as a single concatenated FASTA containing all samples in one file.

To format your viral sequence metadata:

1. Download the metadata TSV Template for the viral sequence metadata and populate it with accepted values for each field. A reference of the accepted values can be found in this resource.
2. DataHarmonizer is a tool that can be used to help validate the accepted values for each field in your metadata TSV locally before submitting. Download the tool and follow the instructions on the Github repository to validate metadata before submission.
3. If you are using Excel or Google sheets, make sure all characters are UTF-8 encoded.

To format your viral sequence files:

1. Make sure they have the file extension .fasta, .fa, or zipped fastas in .gz format.
2. Each sequence must be preceded be a description line, beginning with a > character. The description line should include >hCoV-19/country/identifier/year sequenced. This identifier must match exactly the "fasta_header_name" column in the metadata TSV file.

• Change the wiki docs instructions to match the above: https://github.com/cancogen-virus-seq/docs/wiki/How-to-Submit-Data

• Connect the Registration button. For testing, this whole flow should be correct:
  https://wiki.oicr.on.ca/display/OV/CanCoGen+Virus+Seq+Data+Portal#CanCoGenVirusSeqDataPortal-3.0StudyandUserRegistration
• When a user logs in, and does not have permissions, the submission dashboard should show this no permission state (ie. has no scopes in JWT).
  dashboard: https://zpl.io/29mPXBp
  profile: https://zpl.io/bAR4NPo

• Identify all possible error cases in wiki.
  Categorize and map to general errors. https://wiki.oicr.on.ca/display/OV/CanCoGen+Virus+Seq+Data+Portal#CanCoGenVirusSeqDataPortal-4.3.6ErrorStates
  ---- Upload File errors
  ---- Payload Generation Errors
  ---- Song and Dance Submission Errors

• After identifying each case, implement the error casing.

Expected testable Outcomes

From the Upload Endpoint, use files to test each of these cases identified:

• Upload File errors
• Payload Generation Errors
• Song and Dance Submission Errors

• copy JWT not API Key
• when you copy, refresh he JWT

See Zeplin for changes: https://zpl.io/addyQoR

• UI bug - the user dropdown is outside of the browser width, causing a horizontal scrollbar to appear on the page.
  

Update the text to this:

To format your viral sequence metadata:

1. Download the metadata TSV Template for the viral sequence metadata and populate it with accepted values for each field. A reference of the accepted values can be found in this resource.
2. DataHarmonizer is a tool that can be used to help validate the accepted values for each field in your metadata TSV locally before submitting. Download the tool and follow the instructions on the Github repository to pre-validate each field in your metadata before submission.
3. If you are using Excel or Google sheets, make sure all characters are UTF-8 encoded.

Links:

• https://github.com/cancogen-virus-seq/metadata-schemas/blob/main/virusseq_metadata_template.tsv
• https://github.com/Public-Health-Bioinformatics/DataHarmonizer/blob/master/template/canada_covid19/SOP.pdf
• https://github.com/Public-Health-Bioinformatics/DataHarmonizer

Zeplin link: https://zpl.io/VYqrkrw

• Since we don't have a chart, please revise the stats display. If we can make the banner a bit shorter, let's do so (as long as the researcher picture isn't cut off too much)
  

• Add the COVID Cloud callout at the bottom.
  External links open in a new window:
  DNAstack links to: www.dnastack.com
  the COVID Cloud logo and Explore VirusSeq Data on COVID Cloud link to https://virusseq.covidcloud.ca/
  

• Add github link to the right of the overture logo, opens in new window and links to: https://github.com/cancogen-virus-seq
  

• Add "About the Team" button to the banner - links to new page from: #29

• @kcullion talk with Rosi to design the study/user addition

• there are 10 duplicated samples from the Jul 13 AB Data Upload

AB_47576
AB_47577
AB_47578
AB_47579
AB_47580
AB_47581
AB_47582
AB_47583
AB_47584
AB_47585

• Suppress the duplications in analysis

• Policy design uses consistent styles with the login intermediate page.

• Add link to this page from the footer

Zeplin: https://zpl.io/ad39WnL

Apologies if this isn't the correct issue tracker to report this. I just noticed that sample L00248453 has a collection date of 2011-01-01, which is very unlikely and not consistent with the year field in the sample name (isolate).

• Add the download functionality BACK on the search page (e.x #32)
• Now that the questionable data has been suppressed, we can add some columns back as visible to the platform:
  • Add the Isolate column as an option to select for columns
  • Add host age as a facet, and option for selection in the column. This should be a default visible column.

On the submission UI Dashboard, these instructions are wrong:

Change the instructions to this blurb:

Virus metadata is submitted as a .tsv file. Viral genome data must be submitted as a .fasta file. Up to 5000 samples can be submitted in a single submission, but note that the larger the file the longer the submission will take. Fasta files are accepted individually, or as a single concatenated fasta containing all samples in one file.

To format your viral sequence metadata:

• Sequence metadata must be provided in TSV format according to the accepted values for each field. A reference of the accepted values can be found in this resource.
• DataHarmonizer is a tool that can be used to help validate your metadata TSV locally before submitting. Download the tool and follow the instructions on the Github repository to pre-validate your metadata before submission.
• If you are using Excel or Google sheets, make sure all characters are UTF-8 encoded.

To format your viral sequence files:

• Make sure they have the file extension .fasta.
• Each sequence must be preceded be a description line, beginning with a >. The description line should include >hCoV-19/country/identifier/year sequenced. This identifier must match exactly the "Isolate" column in the TSV file.

Links:

no1: https://github.com/Public-Health-Bioinformatics/DataHarmonizer/blob/master/template/canada_covid19/SOP.pdf
no2: https://github.com/Public-Health-Bioinformatics/DataHarmonizer

Steps taken

This is happening in production and dev sites.

1. Register a new user through keycloak page
2. Click on Link to e-mail address verification from the email (I'm logged in to my email at this point)
3. The site opens in the browser. I first see a white page, then the pre-loader, then the login page loads (https://virusseq-dataportal.ca/login) then the dashboard loads
4. This happens when I logout and log back in as well

Expected Outcome

• When redirecting to the logged in dashboard, just display the loader then load the dashboard. Do not display that login page first

• Add the ability to add links to banners.
• Add this Banner
  Banner Type: Info and Dismissible
  Banner Title: IMPORTANT: Data Acknowledgement
  Banner Content: You may use the data from the Canadian VirusSeq Data Portal to author results obtained from data analyses, provided that your published results acknowledge the Canadian Public Health Laboratory Network (CPHLN), CanCOGeN VirusSeq, and all laboratories having contributed data. The full acknowledgement and data use policy can be found here. Please note that the data that is being shared is the work of many individuals and should be treated as unpublished data. If you wish to publish research using the data, contact us at [email protected] first to ensure that those who have generated the data can be involved in its analysis. You are responsible to make the best efforts to collaborate with representatives of the data providers responsible for obtaining the specimen(s) and involve them in such analyses and further research using such data.

links

• https://virusseq-dataportal.ca/policies
• https://virusseq-dataportal.ca/acknowledgements

• hooking up to some email and test the email verification flow
• make sure users can confirm email and get registration emails

https://virusseq-dataportal.ca/explorer

      "path": [
        "file",
        "aggregations"
      ],
      "extensions": {
        "code": "INTERNAL_SERVER_ERROR",
        "exception": {
          "name": "ResponseError",
          "meta": {
            "body": {
              "error": {
                "root_cause": [],
                "type": "search_phase_execution_exception",
                "reason": "",
                "phase": "fetch",
                "grouped": true,
                "failed_shards": [],
                "caused_by": {
                  "type": "too_many_buckets_exception",
                  "reason": "Trying to create too many buckets. Must be less than or equal to: [10000] but was [10001]. This limit can be set by changing the [search.max_buckets] cluster level setting.",
                  "max_buckets": 10000
                }
              },
              "status": 503

• basic implementation
• some components
• remove dev IPS in the filter.

• Change the TSV export to include the 2 new columns diagnostic_pcr_ct_value_null_reason and GISAID_accession

Zeplin: https://zpl.io/Vq3vKYl

• Add "About the Team" item in top navbar
• Build the team page. Content may change
• The links are all external - open in a new window and are as follows:
  

1. www.genomecanada.ca
2. www.canada.ca
3. https://www.genomecanada.ca/en/cancogen
4. https://www.canada.ca/en/public-health/news/2021/02/government-of-canada-invests-53-million-to-address-covid-19-virus-variants-of-concern.html
5. https://www.genomecanada.ca/en/cancogen/cancogen-virusseq
6. www.genomecanada.ca
7. https://www.genomecanada.ca/en/cancogen

1. https://www.computationalgenomics.ca/c3g-montreal-node/
2. https://softeng.oicr.on.ca/team/
3. www.oicr.on.ca
4. www.overture.bio.
5. www.overture.bio
6. www.oicr.on.ca
7. www.dnastack.com
8. www.ga4gh.org
9. https://www.digitalsupercluster.ca/covid-19-program-page/beacon-realtime-global-data-sharing-network/
10. https://www.digitalsupercluster.ca/
11. https://virusseq.covidcloud.ca/
12. www.dnastack.com

We have a new Song Schema: https://github.com/cancogen-virus-seq/metadata-schemas/blob/main/schemas/consensus_sequence.json

• Make sure its updated in prod cancogen song:
• Update the Maestro mapping to reflect new fields
• Update the Arranger project to read the new indexed fields, and arranger configs.
• • Default table columns to show are marked with "Y" in column C in this doc: https://docs.google.com/spreadsheets/d/1MKOEO1Wi66eXbwEQo_OKpUJ0X0mUIQ7NU7bI86gGrjA/edit
• • Every field should show as a facet
• • age/integer slider facet gets correctly initialized to work with the component

• define study level polices
• change the song.score configs to know the format of the study level policies
• update MUSE to look at study-permissions instead of egoid
• • If a submitter submits something for a study they don't have permissions for, then reject the submission and ask them to remove those payloads

• With upcoming shutdowns from Scinet and Collab maintenance, the Virus-seq portal will become unavailable for short periods of time.
• We will need to be able to communicate the outage alerts leading up to the scheduled maintenance via a banner along the top of the portal UI.
• The text will need to be easily updated via github.

There will be additional fields added to the virus-seq dataset:

• 4 new, optional fields.
• the isolate field is no longer required, thus muse cannot rely on it to associate payloads/files.
• 1 new required field fasta_header_name. This field should replace all functionality of the isolate field.

Steps to Update

• Data harmonizer template update
• Song - virusseq/metadata-schemas#4
• ES mapping - virusseq/roadmap#26
• Muse update needed
  • virusseq/muse#74 [fields]
  • virusseq/muse#76 [isolate --> header change)
• Singularity update needed - virusseq/singularity#8
• Payload migration - virusseq/roadmap#27
• At this point all deploys are done, and need to do a reindex. Portal in maintenance with down time.
• UI deployment with updated new fields - #56

For transparency let's change the wording to this

Genome Canada has also partnered with DNAstack to integrate VirusSeq data with their COVID Cloud Platform. COVID Cloud is a cloud-based platform that helps researchers find, visualize, and analyze genomics and other datasets related to COVID-19.

in these two spots

• homepage
  

• Meet the team page
  

• UI Pages only
• remove submitter login in footer
• disable the explore tab, and the explore button from homepage
• No new data can be posted.
• hosted on Netlify
• Add a warning banner to this site with the text:

Data exploration, download and upload are currently unavailable due to site maintenance.  Service should return shortly after the maintenance window completes on June 9. 

• No submissions state:
  Zeplin Link: https://app.zeplin.io/project/5a69fad2bde6330a929157fd/screen/60709965780d473c6a005059
  

• User has submissions state:

• Zeplin Link: https://app.zeplin.io/project/5a69fad2bde6330a929157fd/screen/60661ca311c5a192df146d9f
  

• Endpoint to power the UI: https://muse.virusseq-dataportal.ca/swagger-ui/#/Muse/Get%20Submissions

• Submissions are linked to a users egoid. Users should only see submissions that they have started.

• Add UI flow for users to send to curation
• Once submitter sends to curation, add to a curation bucket. the TSV/FASTA files can be named based on study_id/egoid to identify the attempted submission
• When objects are added to the bucket, notify the data curation team.

OPTION 1: use collab, deploy KAFKA write to KAFKA queue, and read send email to helpdesk.
OPTION 2: use aws, pipeline and sns publish. free tier possible can work.

Change the data ingestion process

Currently: metadata/fasta uploaded → validated → reformatted → submitted to Song/Score → indexed by Maestro → released to Portal
Modified to: metadata/fasta uploaded → validated → stored in S3 bucket → curator reviews within 24 hours
if no problem → curator approves → data reformatted → submitted to Song/Score → indexed → Portal.
Problem case 1 → curator emails submitter and recommends changes → submitter resubmits
Problem case 2 → curator changes data directly → curator approves

Please update the virus-seq portal with the first public data set

https://wiki.oicr.on.ca/display/OV/CanCoGen+Virus+Seq+Data+Portal

Policy Page Update: https://docs.google.com/document/d/1xHX7UGcPTv91cN9aNe4Y6-s7m7oE9TBuP-jpEGXWekE/edit?usp=sharing

• With upcoming shutdowns from Scinet and Collab maintenance, the Virus-seq portal will become unavailable for short periods of time.
• During the outage (scheduled or otherwise), we will need have users directed to a maintenance page. This can be manually triggered.
• Since the virus-seq portal is hosted entirely on Collab, this page will need ot be hosted externally. Netlify was suggested as an option.
• virus-seq coordinate the dns change. let's make sure the DNS TTL's (Time To Live) are low.
  • Dont' want the DNS change to take forever to propogate

Maintenance page UI

We are just going with a similar design to the login page. Ping Kim for the background image.

Zeplin: https://zpl.io/br3GAg7

• Regarding footer: keep the footer but hide the Policies and Submitter Login links

• Add login button to portal.
• When user logs in and gets navigated to the Submission Dashboard page. Right now, this page can just be blank (or put some template text or something) with the expected header/footer.
• Dashboard link appears in header and is selected for the user.

• Upload Files: started submission with file error: https://zpl.io/aNNGENv
  The submit button should be enabled iif there is one TSV file and at least one FASTA file.
• valid submission ready: https://zpl.io/amPzLrm
• Submission loader: https://zpl.io/2p8QRLN
• Validation error: https://zpl.io/V0vnrdm

https://projects.invisionapp.com/d/main#/console/21152174/453472381/preview

Search page

• Scroll bar on dropdown needs to be removed (windows/Google Chrome)
  
• make the numbers font size 12
  
• remove # files in facet
• Footer link order. Please use the following order: "About CanCOGeN • Policies • Contact Us • Submitter Login"
  

• Change "File Table" to "Metadata Table" - downloads the metadata tsv
• Change "File Manifest" to "Consensus Seq" - downloads the fasta file (GZip/XZ)

• As user rosi@oicr I have access to the MUSE-KHSC-ON project.
• if i submit payloads for studies that i don't have access to (MUSE-UHTC-ON) , this is the error that i see from the UI
  
• this is not informative to me. Looking in the response i do see this (below) that MUSE is correctly rejecting an upload for a study that I do not have permission to give access to.
  

• Once a user logs in, profile icon shows in top rioght
• When clicked profile has link to Profile page and Logout option.
• Slight restyle for virus-seq:
  Zeplin for Profile page: https://app.zeplin.io/project/5a69fad2bde6330a929157fd/screen/6064b9b3ad796803a52f8098

• Change the column names to match the Data Harmonizer
• Country --> Geo_loc_name (Country)
• Province --> Geo_loc_name (State/province/territory)
• Host --> Host (Scientific name)
• Update the download all S3 url https://object.cancercollaboratory.org:9080/swift/v1/Download/data/virusseq_metadata_consensus_all_20210608.tgz

• Update the theme to design: https://zpl.io/brqjAy5
• Add common password restrictions [at least 8 characters, one number one symbol]
• Add terms and conditions page into the flow with permission.
• Update the email copy (@rosibaj please provide copy for this) on account verification.
• Figure out how to add the Keycloak theme to the production deployment
• Change "VirusSeq Porta" to "VirusSeq Data Portal"

Header

• Change “About VirusSeq Portal” to “About VirusSeq Data Portal”
• Change “Explore COVID-19 Data” to “Explore VirusSeq Data”
• Make the fonts for both bold

Footer

• Make the text not bold
• Make the Genome Canada logo bigger. Note for @caravinci when you make this larger can you decrease the top and bottom padding just around this logo, so the footer doesn't get much bigger than it already is? if not, that's fine
• Remove Genome Canada link (the logo already links)
• Change “2021 CanCOGeN VirusSeq Data Portal” to “2021 Canadian VirusSeq Data Portal

Landing page

• Replace The goal of VirusSeq ...and ensuring data access to the research community. with The goal of the CanCOGeN VirusSeq project is to sequence up to 150,000 viral samples from Canadians testing positive for COVID-19. The VirusSeq Data Portal is an open-source and open-access data portal for all Canadian SARS-CoV-2 sequences and associated non-sensitive clinical/epidemiological data. It harmonizes, validates and automates submission to international databases.
• Make the above text not bold.
• Under “Impact on Canadians”, change “Genomics-based” to “Genomic-based”; “COVID-19 virus” to “SARS-CoV-2 virus”

Explorer

• Facet categories, eg. “Analysis Type”, “File Type”, make the fonts smaller, about the same size as “Filters” at the top.

• Facet restyling.
  

• selected values: use the darker blue from this mockup (the lighter blue isn't accessible) https://zpl.io/awzeOmK
  

• Download link to docs - @rosibaj is there a page explaining how to download data with a file manifest? I couldn't find this. Also add a period to this sentence.
  

• I think this should say "Start by selecting filters." As we call them filters in the title, and technically you are selecting "values" not fields.
  

• Make the text in the top right and y axis # Viral Genomes so # Viral Genomes by Province. The tooltip should be fine; the label Host Age is clear.
  

• I thought we were going to show age "bins"? otherwise will this chart be each individual age that's in the portal?
  

• Should we specify what this number is counting for clarity. [not urgent will save for later if desired]
  

Brand

• If there's time, use the font that CanCOGen uses: https://fonts.google.com/specimen/Source+Sans+Pro
  (this font may make the regular weight not seem as bold?)

Submitter Login page

• you could use the same font style as the paragraphs on the homepage, this seems a bit small.
• Change "COVID-19 Data is submitted" to "Viral genomes are submitted"
• the "request an account" link could link right to the CanCOGen contact form, as well as the footer "contact us" link: @rosibaj Thoughts? https://www.genomecanada.ca/en/cancogen/contact-us

-update configs to new domian

• blocked by jenkins

• Remove "As of April 27, 2021" from the landing page. This should ideally be auto updated with the latest submission date. For now, remove.

• Expose the specimen collector sample ID in the table column by default; this should be the first column in the table by default.

page: https://virusseq-dataportal.ca/policies
text updates: https://docs.google.com/document/d/1xg-U-kFGYEhJmpkwqC1tcR3nFDz4H_tFsIelIi30z3E/edit#

• can work the same, but adda drag n drop styling indicator for users
• Style the error message. ignore the part below the line for now as that is still being confirmed.
  Zeplin: https://zpl.io/2j3PggO
  

• Some small text updates:
  First paragraph:

The goal of the CanCOGeN VirusSeq project is to sequence up to 150,000 viral samples from Canadians testing positive for COVID-19. The VirusSeq Data Portal is an open-source and open-access data portal for all Canadian SARS-CoV-2 sequences and associated non-personal contextual data. It harmonizes, validates and automates submission to international databases.

later fix lab --> laboratories

among Canadian public health laboratories, researchers and other groups interested in accessing the data for research and innovation purposes.

Microbiology Laboratory and provincial public health laboratories across the country.

• DOWNLOAD ALL will be in the header, a pill, this will appear along the navigation bar on the landing page.
• Grey out other download options if no rows selected. Disable the dropdown, OR disable the options are both acceptable approaches.

Outstanding and Needed

• finalized song schema
• finalized TSV schema that maps to analysis schema
• finalized TSV that the user will submit

Scope of this ticket

• Implement TSV validation on submission

Expected Testable Outcome

• Can use the upload endpoint with a BAD file and see it rejected with an error
• Can use the upload endpoint with a GOOD file and get a submission id