I have had this issue in the past (see zenodo file) and it looks like the current PANCAN_mutation file from xena has less samples and less columns than a previous version.

One of the columns we don't have is the specific nucleotide mutation and is preventing us from completing #15

It may be good to ask a direct question to the UCSC Xena Google Group. They have been helpful in the past (see #14)

Currently, this repository is licensed under a BSD 3-Clause License, which is the default license for Project Cognoma repositories. This license was chosen for its permissive and open nature as well as it's compatibility with other Greene Lab software products. However, the BSD 3-Clause License is intended for code. Rather than referencing all content, the license specifically mentions "source code".

However, as a data repository, cognoma/cancer-data will hold much more than source code including data, visualizations, writing, documentation, and notebooks which combine all of the aforementioned content types into a single file.

As far as openly releasing content that is not only code, there's currently a short list of recommended licenses. My favorite is CC0 because it waives all copyrights and effectively places work in the public domain. This means anyone can use CC0 content without agonizing over license compatibility.

Despite being one of the most liberating licensing options available, CC0 is not an OSI-approved open source license for complicated reasons. We're still investigating the license of TCGA and Xena Browser data (see #3), but it's possible our upstream data providers may impose restrictions on our use that may require attribution, in which case CC BY is an option. However, Creative Commons recommends against using its licenses (besides CC0) for software.

Hence, switching the entire repository to a Creative Commons license doesn't make sense, since we want to remain OSI conformant. However, we also need a licensing arrangement that is appropriate for content that is not "source code". One option I can think of is multi-licensing where we apply both a CC0/CC BY License and a BSD 3-Clause License to the repository. Users would then be able to choose either license based on their use case. I want to make sure that this is a legally valid approach. What do people think (also asked on Twitter)?

An issue has been raised in the meeting today regarding visualizations of the clinical data. Other data viz are also considered. However, more immediately, we need viz schemes of the clinical data for assessments and covariate selection.

I've noticed that some gene names have been converted to dates in PANCAN_mutation (version info, Xena Browser). Here are some of the effected rows:

The gene-to-date conversion is a well documented feature of Microsoft Excel. While the number of corrupted rows in PANCAN_mutation looked minimal, it's disturbing that the data has passed through Excel, since workflows that use Excel tend be manual rather than scripted and thus error prone and irreproducible.

Currently (in Cell 11 of 2.TCGA-process.ipynb), we retain only Primary Solid Tumor and Primary Blood Derived Cancer - Peripheral Blood. In #44 it was determined that 389 samples (with mutation and gene expression data) were missing clinical annotations. It likely that many of these samples were removed from the clinical matrix by cell 11 above.

We should consider adding Metastatic and to Cell 11.

In another discussion @gwaygenomics shared acronyms for TCGA diseases as a text file (tcga_dictionary.txt). The contents are:

The PANCAN_mutation dataset (online doc) contains several types of mutations under the effect column. My processing of the dataset (notebook) yielded the following mutation effect and frequencies (as counts and percentages):

It appears that certain effects are duplicates — such as 5_prime_UTR_variant, 5'UTR, UTR_5_PRIME — which if true represents a poor case of standardization. If we want to improve the standardization, we can create our own mapping, or we can report the issue to the upstream creators (although these fixes usually take a long time).

Anyways, we'll have to decide which types of effects to consider as functionally relevant mutations. For example, a "Silent" mutation generally does not have a biological effect. We could also let users decide for themselves, but that adds complexity.

@clairemcleod, @mp8, @DCousminer, @gwaygenomics, @cgreene, @stephenshank — I thought you may have a better understanding than I do of the biology here. Can any of these categories be discarded as irrelevant to a tumor's function and classification? Are you interested in creating a consolidated set of effects with duplicates merged?

The current format of the mutation matrix leads to some complications in the core-service repository. A more desirable format to work with for the purpose of populating the core-service mutation model would be of the form:

sample_id	entrez_gene_id
TCGA-18-3406-01	1
TCGA-38-4631-01	1
...

New, publicly available dataset of 11,078 RNAseq + clinical childhood cancer tumors.

Xena data

Blog Post

This will open up a lot of analysis opportunities - exciting it is now available!

Data is currently processed in https://github.com/cognoma/cancer-data/blob/master/2.TCGA-process.ipynb and the final matrices used in downstream analyses include samples that have mutation, expression, and clinical measurements and were not filtered for other reasons.

@kurtwheeler pointed out in cognoma/core-service#99 a potential issue that the current implementation is not finding samples it should. @dhimmel discovered that this was not an issue (at least not primarily an issue) of the backend, but of the data itself.

I outlined current problems with the data in cognoma/core-service#99 (comment) but we can continue this discussion here.

The PanCanAtlas released 27 open access papers and updated data last week.

The UCSC Xena team also added this version to their database! An overview of the updated data is here.

We should update our download and the figshare so that cognoma runs with the most recent PanCanAtlas version

Stumbled upon snaptron today and eventually found my way to this resource.

There are many variables curated here measured on each sample (in samples.tsv) including treatment (both specific therapeutic agent, and class of therapy (e.g. chemotherapy, immunotherapy, etc.). I know that @yigalron was very interested in this particular data...

At the 8/23 meetup, @dhimmel expressed interest in incorporating metabolic pathway information by combining the dataset that we have and the hetnet database that was described at the first meetup. The hetnet has information on what pathways the mutated genes in the current dataset participate in.

I figured I'd open this issue to get the conversation started. Initially, I am wondering what this dataset would look like, and do we envision it being created from what we already have? And how much tweaking will the classifier of the machine learning group (for instance, that provided by @gwaygenomics) require?

Currently, the licensing of the TCGA Pan-Cancer Data available via UCSC's Xena Browser is unclear. I've messaged their listserv and will update this issue with any progress.

See cognoma/core-service#42 (comment), where @stephenshank has discovered a gene with an invalid entrez_gene_id in mutation-matrix.tsv.bz2 from https://doi.org/10.6084/m9.figshare.3487685.v5.

The gene is 117153, which is not included in genes.tsv.

Currently, we're storing our datasets (which are matrices) as compressed TSVs which are great for long-term interoperable storage. However, we'd love a way to lookup specific rows and columns without having to read the entire dataset. We began discussing options at cognoma/cognoma#17 (comment). We want a persistent storage format (i.e. file) that allows reading only specified rows and columns into a numpy array/matrix or a pandas dataframe.

A primary benchmark for judging implementations is how much time are you saving over reading in the entire bzipped TSV into python via pandas for a variety of setups.

We need to generate a comparison between the sample IDs that exist in all three data sources. It will be good to subset the clinical matrix to only samples that are measured by RNAseq and to file a pull request with this report.

Similar to how we have sample information in samples.tsv, it would be nice to create a table with gene information. The primary identifier is entrez_gene_id. Additional columns could be:

• symbol
• name
• chromosome
• n_mutations - number of mutated samples
• median_expression - median gene expression
• mad_expression - median absolute deviation of gene expression

I'm leaning towards a combined dataset for mutation and expression genes. But I could be convinced that splitting the datasets would be better.

We should probably get this information from entrez gene as @clairemcleod did in #12.

Labeling this issue a task awaiting a claimer.

There are a number of questions around how best to represent various items that are important for sample selection. Can someone help out the django-cognoma team to specify the best way to put these into the database that the front end can access?

See cognoma/core-service#2

which column in the clinical_data should i consider to know if the tumor has recurred or not?

does _RFS_IND=1 mean definitely recurred?

how do i know if the tumor is primary or second primary?

how do i know if the tumor has recurred locally or distally?

does clinical_M=M1 or pathologic_M=M1 mean definitely metastasized?

Hi @dhimmel ,

The documentation links provided for the 3 datasets did not explain the variables involved clearly. It would be great if you could share some links around that.

Thanks,
Roshan

In speaking with a cancer biologist and collaborator about cognoma it was discovered that a huge win we could relatively easily deliver is classification performance (or classification scores) across different mutation types for an input gene. This would be extremely useful for a researcher who is interested in determining the pathogenicity of a particular mutation.

I believe that cognoma is an ideal way of approaching this problem. Typically, when genes mutate there is a range of severity regarding how the particular mutation impacts downstream changes. For a particularly virulent mutation, a classifier trained to detect an inactivation signature may output a higher score for those groups of samples, than other samples with a less virulent mutation.

In my eyes, this particular issue bypasses the machine learning group - they will still work with the previously defined Y matrices. However, in order for the backend to serve the frontend information from the database about each sample's mutation so that the frontend can visualize the results we need to know how to parse this information.

I looked briefly at the information embedded in the PANCAN mutation data - particularly the columns labeled HGVSc and HGVSp. These columns hold standard ways of storing specific mutation calls. More information about these standards are provided by the HGVS website.

An issue has been raised in today's meeting.

The clinical matrix should be carefully analyzed to select a specific covariate or a set of covariates we can use for analyses.

The relevant notebook is here
tcga notebook for data download
and the dataset is named
PANCAN-clinicalMatrix

Xena datasets (as retrieved in #1) use symbols to identify genes rather than standardized identifiers, such as Entrez GeneIDs, ensembl gene IDs, HGNC IDs, or UCSC gene IDs. This has led to upstream data quality issues such as #4. Hence, I think it makes sense to code our databases using standardized identifiers.

Currently, we use the HiSeqV2 and TCGA.PANCAN.sampleMap datasets which both use symbols. Does anyone have a preferred identifier? I like Entrez GeneIDs.