Segmentation fault while counting human genome about squeakr HOT 7 CLOSED

Hi @Colelyman,

Could you please share a sample input file to reproduce the issue.

Thanks,
Prashant

from squeakr.

Here is a smaller example file (the first human chromosome, still produces the error), available for download (249 MB) https://drive.google.com/open?id=1Bo0sN89X1Mrnhn1tF5Zmi9CWdIW4ZzH8

Let me know if you have trouble downloading it.

from squeakr.

Any updates on this issue?

Thanks,
Cole

from squeakr.

Hi @Colelyman,

Apologies for the delay in getting around to this. We know what the cause is here. The FASTA/Q parsing strategy we use parses the file in "blocks", with the assumption that each block will contain at least one complete record. Since blocks are large (they can be MB), this is true with sequencing reads (even long reads). However, when the "reads" are a reference genome, as is the case here, that can cause a problem. One solution is to make the block size large enough to cover the biggest contig in the assembly (@prashantpandey can point you at the line of code that would need to be changed for that). A better solution would be to "fix" our parser to avoid this assumption, but we've not had a chance to do that yet.

Best,
Rob

from squeakr.

Thanks, @rob-p. @Colelyman, you can increase the block size in reader.h

diff --git a/include/reader.h b/include/reader.h
index 679bf0d..c35669e 100644
--- a/include/reader.h
+++ b/include/reader.h
@@ -99,7 +99,7 @@ bool reader::fastq_read_parts(int mode, file_pointer *fp) {
        reader& file_reader = *(fp->freader.get());
 
        uint32_t OVERHEAD_SIZE = 65535;
-       uint64_t part_size = 1ULL << 25;
+       uint64_t part_size = 1ULL << 29;
        char *part = (char *)malloc((part_size + OVERHEAD_SIZE)*sizeof(char));
        memcpy(part, part_buffer, part_filled);

I have tried the code with the above change and it didn't crash.
We will notify you when we push the new parser changes in the repo.

Thanks,
Prashant

from squeakr.

@rob-p @prashantpandey thank you so much! I will try this out.

from squeakr.

I'm experiencing the same problem for a genome dataset with commit 695edb8.

I can confirm the suggested change/workaround by prashantpandey is working.

Please reopen this ticket (so its easier to find) until you get a chance to actually fix the parsing strategy and/or add the information to the projects README.md

Sincerely tkranz

from squeakr.