Comments (6)
I'm CCing also @jnalanko who might be interested. Have you also notice this?
I can confirm that different runs of themisto may report different number of color sets, e.g., 4,236,355 vs. 4,236,354, for the same issue.
from ggcat.
Hi, @jermp. No, I have not noticed this. Thankfully, the correctness of Themisto should not be affected either, as long as the unitigs cover all k-mers at least once, and the colors of the unitigs not wrong.
from ggcat.
Yes exactly, both Themisto and Fulgor are correct anyway since the num. of kmers always seem to be correct and consistent. Just letting you know.
from ggcat.
Hi @jermp, the difference in color subsets is due to a (partially wanted, for performance) race condition when updating the hash table of the colors.
For efficiency and technical reasons, a new color is first written to disk and then added to an in-memory hashmap that is used to deduplicate equal colors, without holding a global lock.
So when two kmers of the same colors (never seen before) are processed at the same time, they can get different color indexes (that are in practice referring to the same set of colors) and they can write them before noticing that the other kmer had the same color.
To avoid this problem ggcat should get a global lock each time it writes a new color to disk and hold it until the in-memory hashmap is updated (with the index returned by the disk write), slowing down parallel insertions.
In practice, this should not lead to incorrect results, unless you check for color equality only by comparing the color subset index without reading the colormap, also the increase in space for storing (few) duplicate colors is very small, and the computed compacted graph has always the same set of maximal unitigs.
The difference in unitigs counts you're seeing is also due to the problem above since when two adjacent kmers in the same maximal unitig have a different color index (even if it refers to the same color set) they are split into multiple unitigs.
Also, this problem tends to be more visible with very small datasets, where there is a high chance that two kmers processed by different threads at the exact same time share the same color.
If you need the color sets to be always distinct I can try to see if there is a way to ensure that, maybe putting this requirement under an optional feature flag if it hurts performance.
from ggcat.
Hi @Guilucand and thank you for confirming.
I do not think it is a hard requirement. Indeed correctness is not impacted if we have some duplicated color.
Any thoughts here @rob-p?
from ggcat.
I agree it's not a hard requirement, but it would be very nice to have consistency between runs and to know the true number of colors. Differences are small in our example, but maybe could grow larger for very similar pangenomes and tons of threads.
How is color equality checked, currently? As map contention is very likely to be low anyway, perhaps you could try a sharded map like DashMap to avoid almost all lock contention?
Also, thanks for the quick response!
from ggcat.
Related Issues (20)
- Compilation failed with error: `use of unstable library feature 'int_log'` HOT 1
- Plans to generate a GFA? HOT 1
- is there a way to get the kmer counts of the kmers in the produced unitigs? HOT 1
- GGCAT keeps panicking about too many open files HOT 1
- Error compiling using latest rust version HOT 2
- Using nightly Rust is problematic HOT 11
- Build fails with `error[E0635]: unknown feature proc_macro_span_shrink` HOT 1
- [Request] Can you provide binaries?
- Interaction between `-m 0` and `--prefer-memory` causes a crash HOT 1
- Installation from scratch fail with file examples HOT 2
- Using ggcat-api in an external Rust project fails to build without including `dynamic-dispatch-proc-macro` in Cargo.toml HOT 3
- Is there any way to dump the whole colormap in a readable file ? HOT 2
- GGCAT API crash when building many graphs in memory from the same instance HOT 2
- GGCAT installation fails when compiling ggcat-api HOT 2
- Error 68 : Unfinished stream HOT 2
- Same flag `-f` for both `--forward-only` and `--colored-query-output-format` options in queries HOT 1
- Problem with GGCAT when using 32 threads HOT 1
- access datasets of origin HOT 2
- Issues with -d and -c flags (ggcat doesn't find sample in large color_mapping.in file) HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ggcat.