Comments (6)
CloXD
commented on June 21, 2024
Dear Jake,
It's very strange indeed. I'm trying to reproduce the error on a local dataset but without success. The scary thing is that it's processing k-mers without finding any hits, meanwhile, in the other run, it finds many at the first log.
I'll try to reproduce the error on a bigger dataset. Could you show me the output of the main command? ( like the one here below )
Memory occupied: 10.094Mb.
Estimating mean values...done.
Using 53909 k-mers to estimate the range of means: 173.625 - 309.266
Memory occupied: 32.473Mb.
Reducing with 4 threads.
[ 2021-8-2.10:7:45 ] Thread 0 starting on cpu 0 with k-mer AAAAAAAAAAAAAGAGGTACAACCAAATGGA to k-mer CAAGAACGTCGAGAAGATCCTGGCCGAGGTG, memory occupied: 347.609Mb.
[ 2021-8-2.10:7:46 ] Thread 1 starting on cpu 1 with k-mer CAAGAACGTCTGCACAGCTTTGGAAAAATTT to k-mer CTTTGTGAGCCAGAACCCCTCTCACTCTCAC, memory occupied: 683.402Mb.
[ 2021-8-2.10:7:47 ] Thread 2 starting on cpu 2 with k-mer CTTTGTGAGCCTCAGCCTGCAGTTCTTCCAC to k-mer GTGTGGTGGCGCATGCCTGTAGCCCCAGCTA, memory occupied: 1019.234Mb.
[ 2021-8-2.10:7:48 ] Thread 3 starting on cpu 2 with k-mer GTGTGGTGGCGCATGCCTGTAGTCCCAGATA to k-mer TTTTTTTTTTTTTTTTTTTTTTTTTTTTTTT, memory occupied: 1.323Gb.
Thank you for your patience and feedback.
Cheers,
Claudio
from imoka.
CloXD
commented on June 21, 2024
I might have found the error: it was in case a sample had not the given prefix, it was going to the previous and not to the following one.
This was producing some fake lines with values in only one column till the matrix reach the target k-mer. That shouldn't affect the results since those lines are usually discarded due to the inability to classify the samples in the groups ( indeed in the example you sent, it keeps no k-mer ) but have a big impact on the running time.
If you want to try again, let me know if the new image solved the issue.
Thank you again.
Cheers,
Claudio
from imoka.
jakewendt
commented on June 21, 2024
Memory occupied: 3.773Mb.
Estimating mean values...done.
Using 3139522 k-mers to estimate the range of means: 20.4448 - 31.5689
Memory occupied: 1.247Gb.
Reducing with 64 threads.
[ 2021-7-30.23:58:44 ] Thread 0 starting on cpu 16 with k-mer AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA to k-mer AAAGAGTCAAATGGACATTTACTGTTTAATT, memory occupied: 1.248Gb.
[ 2021-7-30.23:58:45 ] Thread 1 starting on cpu 43 with k-mer AAAGAGTCAAATGGATGAAAGTCCTCATGGA to k-mer AACCAAAGCTTCCAGTATCTGAGAAGTTGTT, memory occupied: 1.267Gb.
[ 2021-7-30.23:58:46 ] Thread 2 starting on cpu 7 with k-mer AACCAAAGCTTCCAGTGTTTGAGCATATGAA to k-mer AAGCCAATCTCGTCCTTGCTTTGGCTGGTGT, memory occupied: 1.268Gb.
[ 2021-7-30.23:58:47 ] Thread 3 starting on cpu 18 with k-mer AAGCCAATCTCGTCTGCCGTCTTCTGCTTGA to k-mer AATATTTGAAAGCGAGTGATTTGTGACCAGG, memory occupied: 1.268Gb.
[ 2021-7-30.23:58:48 ] Thread 4 starting on cpu 26 with k-mer AATATTTGAAAGCGATTTTGGGGGGAGTTGT to k-mer ACAAGTTTCTTTGGATGGCCCGGATTGGAGG, memory occupied: 1.268Gb.
[ 2021-7-30.23:58:49 ] Thread 5 starting on cpu 1 with k-mer ACAAGTTTCTTTGGATGGGCCGGATTGGAGG to k-mer ACCACCATAAGATAATCATAAATACGCAAGA, memory occupied: 1.268Gb.
[ 2021-7-30.23:58:50 ] Thread 6 starting on cpu 11 with k-mer ACCACCATAAGATACCATCTCGCAACCATAC to k-mer ACGTCTGAACTCCAGTCACGTTTAACAATCG, memory occupied: 1.272Gb.
[ 2021-7-30.23:58:51 ] Thread 7 starting on cpu 6 with k-mer ACGTCTGAACTCCAGTCACGTTTAACAATCT to k-mer AGAAAAGAGGTGGGTAGGGTTACAGATCCAG, memory occupied: 1.272Gb.
[ 2021-7-30.23:58:52 ] Thread 8 starting on cpu 4 with k-mer AGAAAAGAGGTGGGTGACTGAATGTGCAGCA to k-mer AGAGTTGACCTGGCATGAATACTGTAAGTCT, memory occupied: 1.272Gb.
[ 2021-7-30.23:58:53 ] Thread 9 starting on cpu 21 with k-mer AGAGTTGACCTGGCTTAGGCCCCATAGCTAG to k-mer AGCGCTGGGAAACAGCCTGCCGAGTGCAAAA, memory occupied: 1.272Gb.
[ 2021-7-30.23:58:54 ] Thread 10 starting on cpu 26 with k-mer AGCGCTGGGAAACAGTCTTTTGCCCCACAAC to k-mer AGGCGGGGGGGGGGGGAGGGGGGGGAGGGGG, memory occupied: 1.272Gb.
[ 2021-7-30.23:58:55 ] Thread 11 starting on cpu 28 with k-mer AGGCGGGGGGGGGGGGAGGGGGGGGAGGGGT to k-mer AGTCCAACTGCATAAACTGGTTTTGGTGGGG, memory occupied: 1.272Gb.
[ 2021-7-30.23:58:56 ] Thread 12 starting on cpu 22 with k-mer AGTCCAACTGCATAAACTGGTTTTGGTGGTG to k-mer ATACTACCTGGTTATATTCAGTGTCAAGTTT, memory occupied: 1.272Gb.
[ 2021-7-30.23:58:57 ] Thread 13 starting on cpu 36 with k-mer ATACTACCTGGTTCACCCTTCATACCTGGGA to k-mer ATCTCCACGAAGCCTGCATGCTGGCTGGGCT, memory occupied: 1.272Gb.
[ 2021-7-30.23:58:58 ] Thread 14 starting on cpu 37 with k-mer ATCTCCACGAAGCTTTTCCCAATGCCTGGAG to k-mer ATGTAGACACATGGTGAGGTGCTGAAGACAG, memory occupied: 1.272Gb.
[ 2021-7-30.23:58:59 ] Thread 15 starting on cpu 33 with k-mer ATGTAGACACATGTAACTAACCTGCACGTTG to k-mer ATTTCTACAACCCGATGAGTCAATATTCAGA, memory occupied: 1.272Gb.
[ 2021-7-30.23:59:0 ] Thread 16 starting on cpu 25 with k-mer ATTTCTACAACCCTAGTGCTTTCATGGGACC to k-mer CAAGGCCGTCTACACCACCACCCTGGATTAC, memory occupied: 1.272Gb.
[ 2021-7-30.23:59:1 ] Thread 17 starting on cpu 23 with k-mer CAAGGCCGTCTATACACAGGATTGCCCCCTG to k-mer CACGCCAGGGTCCTTGCCTCCTTGTACAGGG, memory occupied: 1.273Gb.
[ 2021-7-30.23:59:2 ] Thread 18 starting on cpu 27 with k-mer CACGCCAGGGTCGCCCCATTATACTGTGCTC to k-mer CAGCCTCCGCCACCCTTGATGAGATTGAGAT, memory occupied: 1.273Gb.
[ 2021-7-30.23:59:3 ] Thread 19 starting on cpu 25 with k-mer CAGCCTCCGCCACCGACCCAGCGCCGCGGAC to k-mer CATAAGGCATGAAAAACATGGTAATGAACAC, memory occupied: 1.273Gb.
[ 2021-7-30.23:59:4 ] Thread 20 starting on cpu 13 with k-mer CATAAGGCATGAAAAACATGGTAATGAGATC to k-mer CCAAAATAAAAAACCAGTAATACCGGTTTTA, memory occupied: 1.273Gb.
[ 2021-7-30.23:59:5 ] Thread 21 starting on cpu 7 with k-mer CCAAAATAAAAAACCCTTCCCTTATTTTGCA to k-mer CCAGTCTCATCCTTGTCTAGATAGTAATTCA, memory occupied: 1.273Gb.
[ 2021-7-30.23:59:6 ] Thread 22 starting on cpu 14 with k-mer CCAGTCTCATCCTTTACCTTGTAGTAAGCGC to k-mer CCCCTTCCCTACAATGGGTACAAAAAGTTCC, memory occupied: 1.273Gb.
[ 2021-7-30.23:59:7 ] Thread 23 starting on cpu 34 with k-mer CCCCTTCCCTACACCAACCCTTCCCTACACC to k-mer CCTAACAGGGGAATTCTTCCCTCTTATGGCA, memory occupied: 1.277Gb.
[ 2021-7-30.23:59:8 ] Thread 24 starting on cpu 29 with k-mer CCTAACAGGGGACACTTATGGCCGTCCCATC to k-mer CCTTATTCAGAAAGCCCAGCGTTCTGACAGC, memory occupied: 1.277Gb.
[ 2021-7-30.23:59:9 ] Thread 25 starting on cpu 31 with k-mer CCTTATTCAGAAATATTTACTTAAACCATAG to k-mer CGGAAAACCGGAGAGGCTGAGTCGACTAATC, memory occupied: 1.277Gb.
[ 2021-7-30.23:59:10 ] Thread 26 starting on cpu 29 with k-mer CGGAAAACCGGAGCCCCTCCTGATGCTGCTG to k-mer CTAACAAATCTGCCCTTGGAAAAAGCGATAC, memory occupied: 1.277Gb.
[ 2021-7-30.23:59:11 ] Thread 27 starting on cpu 32 with k-mer CTAACAAATCTGCCTTTCATTACCTACAACT to k-mer CTCCATCTCACACACGATCCTCTCAGCACTG, memory occupied: 1.277Gb.
[ 2021-7-30.23:59:12 ] Thread 28 starting on cpu 3 with k-mer CTCCATCTCACACAGGAAGTTCTCTGGATCC to k-mer CTGAGCCTCTTTGATAACTTTTGTGCTATTC, memory occupied: 1.278Gb.
[ 2021-7-30.23:59:13 ] Thread 29 starting on cpu 16 with k-mer CTGAGCCTCTTTGCACCTGCAATAAGCAAAT to k-mer CTGGTGGGAGACTGCTGGGGTCACCATCCGC, memory occupied: 1.278Gb.
[ 2021-7-30.23:59:14 ] Thread 30 starting on cpu 30 with k-mer CTGGTGGGAGACTGCTGGGGTCACCATCTGC to k-mer CTTGCACAATAAACAGAAAACTTTGCTTATT, memory occupied: 1.278Gb.
[ 2021-7-30.23:59:15 ] Thread 31 starting on cpu 51 with k-mer CTTGCACAATAAACTTTTAGGAGCTGCTCAA to k-mer GAACAAGCTCCGGGATACCAGCTCCGGGCTC, memory occupied: 1.278Gb.
[ 2021-7-30.23:59:16 ] Thread 32 starting on cpu 44 with k-mer GAACAAGCTCCGGGCAGAGGAGTGGTTCTGC to k-mer GACAGGGGACTGGGGCCCATACCCTCCACCT, memory occupied: 1.286Gb.
[ 2021-7-30.23:59:17 ] Thread 33 starting on cpu 16 with k-mer GACAGGGGACTGGGGCCTAGGTAGGGCTGAA to k-mer GAGCATTATAAGAGGTGGTAGCAAGTACACG, memory occupied: 1.286Gb.
[ 2021-7-30.23:59:18 ] Thread 34 starting on cpu 28 with k-mer GAGCATTATAAGCATGCTGTCAGGCAGAGTT to k-mer GATCACTTAAGCTATCTTTTATTTAGTTTTA, memory occupied: 1.286Gb.
[ 2021-7-30.23:59:19 ] Thread 35 starting on cpu 11 with k-mer GATCACTTAAGCTATGATCATGCCACTGCAC to k-mer GCACCTGGGAAGATGGCAGCAGCTAGGAGGG, memory occupied: 1.295Gb.
[ 2021-7-30.23:59:20 ] Thread 36 starting on cpu 26 with k-mer GCACCTGGGAAGATTAGGGGAGCTGCGAAGG to k-mer GCCAGTTTTATGAAAGGACCTGTAGTTTGGG, memory occupied: 1.295Gb.
[ 2021-7-30.23:59:21 ] Thread 37 starting on cpu 43 with k-mer GCCAGTTTTATGAAATAAATAAATAAATGGT to k-mer GCGCCGTGCCTCAAGGAGGGGGAGGGAGTCC, memory occupied: 1.295Gb.
[ 2021-7-30.23:59:22 ] Thread 38 starting on cpu 31 with k-mer GCGCCGTGCCTCACACAGACGAGGCGGGCTC to k-mer GCTGGCTCCCCCGTGGTGGCTGTCTCCGCCT, memory occupied: 1.295Gb.
[ 2021-7-30.23:59:23 ] Thread 39 starting on cpu 27 with k-mer GCTGGCTCCCCCTACCCGCAACCAGTTTGCT to k-mer GGACTCGTCATGCCGGGCCGTCGCCTCCGTT, memory occupied: 1.295Gb.
[ 2021-7-30.23:59:24 ] Thread 40 starting on cpu 22 with k-mer GGACTCGTCATGCTCCTGCTTGCTGATCCAC to k-mer GGCATCCACTGGGGGGTCTATCTTGGAATGT, memory occupied: 1.295Gb.
[ 2021-7-30.23:59:25 ] Thread 41 starting on cpu 25 with k-mer GGCATCCACTGGGGGTCATGGAATGTATCCC to k-mer GGGAAGGCCACAGGATGCCATCAGAGTGTGC, memory occupied: 1.295Gb.
[ 2021-7-30.23:59:26 ] Thread 42 starting on cpu 34 with k-mer GGGAAGGCCACAGGATGGCTCTGGCGATCCA to k-mer GGGGGCCCCCGATCCTTTAAAAAAAAAACAA, memory occupied: 1.295Gb.
[ 2021-7-30.23:59:27 ] Thread 43 starting on cpu 44 with k-mer GGGGGCCCCCGATCCTTTAAAAAAAAAACAC to k-mer GGTGAGCTGAGATTGCACCACTGTACTCCAG, memory occupied: 1.295Gb.
[ 2021-7-30.23:59:28 ] Thread 44 starting on cpu 37 with k-mer GGTGAGCTGAGATTGCACCAGTGCACTCCAG to k-mer GTATGCCGTCTTCTGCTTGGAAAAAAAGGCG, memory occupied: 1.295Gb.
[ 2021-7-30.23:59:29 ] Thread 45 starting on cpu 1 with k-mer GTATGCCGTCCAAAAAAAAAAAAAAAAAAAC to k-mer GTGAGGAAGGGGTGCCTGGAGTTCTGCAGGA, memory occupied: 1.295Gb.
[ 2021-7-30.23:59:30 ] Thread 46 starting on cpu 39 with k-mer GTGAGGAAGGGGTGCCTGGAGTTCTGCAGGC to k-mer GTTATACAAACGAGGGCTTGAAAACATTTTG, memory occupied: 1.295Gb.
[ 2021-7-30.23:59:31 ] Thread 47 starting on cpu 59 with k-mer GTTATACAAACGAGTAGCCCAATGAATGACA to k-mer TAAATGGAGACACCAGATCTCCCTAACCTCT, memory occupied: 1.295Gb.
[ 2021-7-30.23:59:32 ] Thread 48 starting on cpu 48 with k-mer TAAATGGAGACACTCGCAGACCTGCAGTTCT to k-mer TACTCAGTGAGGAGATGAGACTGGCACAGGG, memory occupied: 1.295Gb.
[ 2021-7-30.23:59:33 ] Thread 49 starting on cpu 18 with k-mer TACTCAGTGAGGAGTGGGACCCGGTGCTTCT to k-mer TATCCCAAACACAAAATTCCCCTGCTTAAGG, memory occupied: 1.295Gb.
[ 2021-7-30.23:59:34 ] Thread 50 starting on cpu 55 with k-mer TATCCCAAACACAACTCTTCATTTGTGTTGT to k-mer TCACTGCAGCCGAACTCCCAGGCTCAGGTGA, memory occupied: 1.296Gb.
[ 2021-7-30.23:59:35 ] Thread 51 starting on cpu 20 with k-mer TCACTGCAGCCGAACTCCCAGGCTCGGGGGA to k-mer TCCATGTTAAAAGTGGCTTCATAGCTACTGA, memory occupied: 1.296Gb.
[ 2021-7-30.23:59:36 ] Thread 52 starting on cpu 50 with k-mer TCCATGTTAAAAGTGTTCTTGTCCATTTACC to k-mer TCGGATGCCGTCTTCTGCTTGTAAAAAACTC, memory occupied: 1.296Gb.
[ 2021-7-30.23:59:37 ] Thread 53 starting on cpu 43 with k-mer TCGGATGCCGTCTTCTGCTTGTAAAAAACTG to k-mer TCTGGTTTACACAGGGACTTCCGGACAGTCC, memory occupied: 1.296Gb.
[ 2021-7-30.23:59:38 ] Thread 54 starting on cpu 45 with k-mer TCTGGTTTACACAGTAATTTTCAACTGCAGG to k-mer TGACTCTTCTGCAGAACCCCATGTTGACCCC, memory occupied: 1.296Gb.
[ 2021-7-30.23:59:39 ] Thread 55 starting on cpu 6 with k-mer TGACTCTTCTGCAGAACTATTTTTTTAAACA to k-mer TGCCAGTGAACTGCATTTACCCAGTTTTCAT, memory occupied: 1.296Gb.
[ 2021-7-30.23:59:40 ] Thread 56 starting on cpu 58 with k-mer TGCCAGTGAACTGCCTGCTAGCCAGCCCCAG to k-mer TGGATCCCTAGCTCCAGCCTCTTCCTGGGCC, memory occupied: 1.296Gb.
[ 2021-7-30.23:59:41 ] Thread 57 starting on cpu 3 with k-mer TGGATCCCTAGCTCCTTGCACCCATGATCCT to k-mer TGGTGGCTGCTGGCCACTCGTGGTGCTCCTT, memory occupied: 1.296Gb.
[ 2021-7-30.23:59:42 ] Thread 58 starting on cpu 34 with k-mer TGGTGGCTGCTGGCCAGACCTTCCATGGCAG to k-mer TGTTCCTTTTGCTCTCACTTCTGAAACAAGA, memory occupied: 1.296Gb.
[ 2021-7-30.23:59:43 ] Thread 59 starting on cpu 4 with k-mer TGTTCCTTTTGCTCTGATAGACAAAATCAGT to k-mer TTATTGCGCAATCTCGGGTGGCTGAACGCCA, memory occupied: 1.296Gb.
[ 2021-7-30.23:59:44 ] Thread 60 starting on cpu 11 with k-mer TTATTGCGCAATCTCGGGTGGCTGAACGCCC to k-mer TTCTGCAACGTTCTTCAGTTTGGAAATCCTT, memory occupied: 1.297Gb.
[ 2021-7-30.23:59:45 ] Thread 61 starting on cpu 0 with k-mer TTCTGCAACGTTGTCATCATTCAGCTTAGAA to k-mer TTGGTTCTATTAGAGCCATCATCCTTATCCC, memory occupied: 1.297Gb.
[ 2021-7-30.23:59:46 ] Thread 62 starting on cpu 29 with k-mer TTGGTTCTATTAGAGTTAAATATGGTAAAAC to k-mer TTTGAAGCAAGTTCTCGGCCAGATGGTGCTT, memory occupied: 1.297Gb.
[ 2021-7-30.23:59:47 ] Thread 63 starting on cpu 15 with k-mer TTTGAAGCAAGTTCTCGGCCAGATGGTGGTT to k-mer TTTTTTTTTTTTTTTTTTTTTTTTTTTTTTT, memory occupied: 1.298Gb.
from imoka.
jakewendt
commented on June 21, 2024
Thank you Claudio.
It is currently still running. Aside from 45, which is currently ...
...
0.00107527 10000000 0 0 52:40:22
0.0011262 10100000 0 0 53:10:18
0.000191982 10200000 0 0 53:40:16
0.000283303 10300000 0 0 54:10:06
0.000876549 10400000 0 0 54:39:54
0.00010057 10500000 0 0 55:09:49
each thread is between 20 and 63% complete. This "failing" thread will stop the entire step from completing, won't it?
from imoka.
jakewendt
commented on June 21, 2024
I killed the job and started with the new image. Fingers crossed.
from imoka.
jakewendt
commented on June 21, 2024
All appears well now. Thank you again Claudio.
from imoka.
Related Issues (20)
- iMOKA GUI HOT 10
- Empty aggregated.sequences.bed.norep.bed causes seg fault HOT 2
- Memory control HOT 3
- Aggregate step crash because "Kept 0 alignments" HOT 2
- "{} Message" when opening K-mer list HOT 7
- Plotting legend issue with PCA HOT 4
- Sample Variation Normalization HOT 2
- the singularity image HOT 2
- Aggregate step segmentation faults HOT 6
- Confidence Interval for ROC AUC metric (Random Forest) HOT 2
- Error when starting singularity exec iMOKA preprocess.sh -i test HOT 13
- --threads does not work? HOT 2
- No error handling when there are no kmers above the thresholds counts at "aggregate" step HOT 1
- How to run it in my Mac? HOT 1
- How to incorporate paired end reads? HOT 2
- File not found error in iMOKA aggregate step HOT 6
- cant get singularity to run HOT 1
- Preprocessing not producing the "sorted.bin" files HOT 2
- aggregate produces many lines of "stdtr domain error" HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from imoka.