Thanks to the great effort of @dcjones, we're getting a powerful framework to parse various data formats at the speed of light.
But I think that the interface of the parser can be more flexible and idiomatic in the context of Julia.
The current interface is the read
method only, and its behavior is significantly different from the methods defined in the standard library.
In the Base
module, read(io, type)
always returns an object of type
but read(io, FASTA)
returns an iterator of biological sequences; this is counterintuitive to me.
I propose the following functions as public APIs on the top of the parser framework:
open(filename, format)
: open filename
and return a stream indexed by format
read(stream, type)
: read a type
object from stream
read!(stream, object)
: read a record into the preallocated object
from stream
each(stream, type)
: return an iterator of type
objects from stream
close(stream)
: close stream.
open
, read
, read!
and close
behave like methods in the standard library, respectively. each
is something like Base.eachline
.
A stream object is indexed by a file format: defined as Stream{F<:FileFromat}
, and Stream{FASTA}
is, for example, a stream type of FASTA format..
The rationale behind this idea is that file format and data type are not completely dependent.
For example, we can read DNA sequences from several formats (including FASTA, FASTQ, SAM, etc.) and a SAM file can generate several kinds of data (including DNA sequence, alignment, quality of sequence, etc.).
The next code may illustrate my idea clearly:
s = open("hg38.fa", FASTA)
while !eof(s)
seq = read(s, DNASequence)
...
end
close(s)
or
open("hg38.fa", FASTA) do s
for seq in each(s, DNASequence)
...
end
end
If there is numerous number of records in a file, you can save memory allocation using read!
as follows:
s = open("sample1234.bam", BAM)
seq = DNASequence()
while !eof(s)
read!(s, seq)
...
end
close(s)
The multiple dispatching in Julia would enable this interface very easily; a method read(::Stream{Format}, ::Type{Record})
will be defined only if a Record
object can be extracted from a Format
stream.
I'd like to know your opinions. Do you think this is reasonable?