How to create database for GRCh38_latest_genomic.gtf.gz? about gffutils HOT 4 CLOSED

@daler I can confirm that this seems to work for me. Thanks!

from gffutils.

The problem is caused by this line:

NC_000001.11    BestRefSeq      gene    14362   29370   .       -       .       gene_id "WASH7P"; transcript_id ""; db_xref "GeneID:653635"; db_xref "HGNC:HGNC:38034"; description "WASP family homolog 7, pseudogene"; gbkey "Gene"; gene "WASH7P"; gene_biotype "transcribed_pseudogene"; gene_synonym "FAM39F"; gene_synonym "WASH5P"; pseudo "true";

Specifically, the description attribute is "WASP family homolog 7, pseudogene".

Some GTF and GFF files store multiple values separated by commas within a single attribute. During the dialect inference step to figure out which of the MANY different format flavors is used in this particular file, gffutils inspects the first N lines. It turns out that this gene falls within the first N lines, and the description has a comma, implying that multiple values are joined by a comma therefore no repeated keys are expected. However there are lines prior to it that have repeated keys (db_xref is repeated in the first line), indicating that multiple values are NOT joined by comma.

Hence the error about being internally inconsistent.

The fix is to restrict the number of lines checked:

db = gffutils.create_db(
    'GRCh38_latest_genomic.gtf.gz',
    'GRCh38_latest_genomic.gtf.sqldb',
    keep_order=True,
    disable_infer_genes=True,
    disable_infer_transcripts=True,

    # add this argument
    checklines=1
)

from gffutils.

I'm also hitting this issue with this GTF file, and wondering if there's a downside to setting 'repeated keys': True in the dialect for GTF files without repeated keys? My issue is that I'm using gffutils to extract transcript features from arbitrary GTF/FASTA pairs (https://github.com/Novartis/pisces/blob/master/pisces/index.py#L172) defined using a configuration file (e.g. https://github.com/Novartis/pisces/blob/master/pisces/config.json). I'd like to not require a fill GTF dialect definition in the config file, but for this most recent RefSeq release I'm having a hard time figuring out the best path forward.

from gffutils.

@mdshw5, @chenzemin see #208, I think that provides a reasonable general solution (see details in that PR). It's merged into the master branch, but I haven't released it yet -- can you confirm it works for you?

from gffutils.