Comments (7)
Hi @Colelyman,
Could you please share a sample input file to reproduce the issue.
Thanks,
Prashant
from squeakr.
Here is a smaller example file (the first human chromosome, still produces the error), available for download (249 MB) https://drive.google.com/open?id=1Bo0sN89X1Mrnhn1tF5Zmi9CWdIW4ZzH8
Let me know if you have trouble downloading it.
from squeakr.
Any updates on this issue?
Thanks,
Cole
from squeakr.
Hi @Colelyman,
Apologies for the delay in getting around to this. We know what the cause is here. The FASTA/Q parsing strategy we use parses the file in "blocks", with the assumption that each block will contain at least one complete record. Since blocks are large (they can be MB), this is true with sequencing reads (even long reads). However, when the "reads" are a reference genome, as is the case here, that can cause a problem. One solution is to make the block size large enough to cover the biggest contig in the assembly (@prashantpandey can point you at the line of code that would need to be changed for that). A better solution would be to "fix" our parser to avoid this assumption, but we've not had a chance to do that yet.
Best,
Rob
from squeakr.
Thanks, @rob-p. @Colelyman, you can increase the block size in reader.h
diff --git a/include/reader.h b/include/reader.h
index 679bf0d..c35669e 100644
--- a/include/reader.h
+++ b/include/reader.h
@@ -99,7 +99,7 @@ bool reader::fastq_read_parts(int mode, file_pointer *fp) {
reader& file_reader = *(fp->freader.get());
uint32_t OVERHEAD_SIZE = 65535;
- uint64_t part_size = 1ULL << 25;
+ uint64_t part_size = 1ULL << 29;
char *part = (char *)malloc((part_size + OVERHEAD_SIZE)*sizeof(char));
memcpy(part, part_buffer, part_filled);
I have tried the code with the above change and it didn't crash.
We will notify you when we push the new parser changes in the repo.
Thanks,
Prashant
from squeakr.
@rob-p @prashantpandey thank you so much! I will try this out.
from squeakr.
I'm experiencing the same problem for a genome dataset with commit 695edb8.
I can confirm the suggested change/workaround by prashantpandey is working.
Please reopen this ticket (so its easier to find) until you get a chance to actually fix the parsing strategy and/or add the information to the projects README.md
Sincerely tkranz
from squeakr.
Related Issues (20)
- Doc command line doesn't much help HOT 3
- Add a --version or -v flag
- Command line parsing seems to be broken in exact branch HOT 1
- squeakr-query "Not Find: 0" HOT 3
- Plans to support FASTA? HOT 2
- lognumslots.sh sometimes underestimates the required number of slots HOT 2
- Update release? HOT 6
- Can't install on OSX, expected identifier FREAD HOT 3
- boost/thread/thread.hpp error installing on macOS HOT 3
- Release 0.6 prints 1.0 with help -v HOT 1
- MacOS: use of undeclared identifier 'posix_fallocate' HOT 3
- gqf/partitioned_counter.c:15:10: fatal error: 'sys/sysinfo.h' file not found
- endless loop in /src/count.cc HOT 1
- /src/count.cc can silently drop input files HOT 2
- Illegal instruction error HOT 4
- long reads?
- Assertion `new_value < current_remainder' failed. HOT 5
- Segmentation fault
- Error opening file for serializing.: Is a directory HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from squeakr.