Comments (8)
ekg
commented on August 11, 2024
I would suggest setting -s much higher (50-100kb) and -p closer to 2X the
nucleotide divergence. You appear to be running out of memory. The
alignment step is too difficult because of the very sensitive settings you
are using.
…On Wed, May 5, 2021, 03:59 Brett Chapman ***@***.***> wrote:
Hi
I've just updated to the latest build and I'm now getting errors at the
wfmash step aligning only small gene regions of a few thousand bp. I pulled
the build from docker and set many of the parameters to the now recommended
defaults (e.g. block_id_min=0). This has happened consistently with several
different gene regions.
+ block_length=0
+ min_match_len=29
+ transclose_batch=10000000
+ block_weight_max=10000000
+ block_id_min=0
+ ratio_contain=0
+ path_jump_max=100
+ edge_jump_max=0
+ poa_params=1,4,6,2,26,1
+ segment_length=2000
+ srun -n 1 singularity exec --bind /data/pangene_graph_workflow/pangene-HORVU.MOREX.r2.3HG0222170_mmseq2pctid-0.9_mmseq2mincov-0.8_pggb-mapping-pctid-85_geneBuffer-1000bp:/data/pangene_graph_workflow/pangene-HORVU.MOREX.r2.3HG0222170_mmseq2pctid-0.9_mmseq2mincov-0.8_pggb-mapping-pctid-85_geneBuffer-1000bp /data/pggb_builds/pggb.sif pggb -i /data/pangene_graph_workflow/pangene-HORVU.MOREX.r2.3HG0222170_mmseq2pctid-0.9_mmseq2mincov-0.8_pggb-mapping-pctid-85_geneBuffer-1000bp/HORVU.MOREX.r2.3HG0222170_pan_genes_buffer-1000bp.fna -N -s 2000 -l 0 -p 85 -n 21 -k 29 -B 10000000 -w 10000000 -I 0 -R 0 -j 100 -e 0 -P 1,4,6,2,26,1 -t 16 -m -v -L -S -o pggb_output
[wfmash::map] Reference = [/data/pangene_graph_workflow/pangene-HORVU.MOREX.r2.3HG0222170_mmseq2pctid-0.9_mmseq2mincov-0.8_pggb-mapping-pctid-85_geneBuffer-1000bp/HORVU.MOREX.r2.3HG0222170_pan_genes_buffer-1000bp.fna]
[wfmash::map] Query = [/data/pangene_graph_workflow/pangene-HORVU.MOREX.r2.3HG0222170_mmseq2pctid-0.9_mmseq2mincov-0.8_pggb-mapping-pctid-85_geneBuffer-1000bp/HORVU.MOREX.r2.3HG0222170_pan_genes_buffer-1000bp.fna]
[wfmash::map] Kmer size = 16
[wfmash::map] Window size = 5
[wfmash::map] Segment length = 2000 (read split disabled)
[wfmash::map] Block length min = 0
[wfmash::map] Alphabet = DNA
[wfmash::map] Percentage identity threshold = 0.85%
[wfmash::map] Mapping output file = /data/pangene_graph_workflow/pangene-HORVU.MOREX.r2.3HG0222170_mmseq2pctid-0.9_mmseq2mincov-0.8_pggb-mapping-pctid-85_geneBuffer-1000bp/wfmash-H8TzOD
[wfmash::map] Filter mode = 1 (1 = map, 2 = one-to-one, 3 = none)
[wfmash::map] Execution threads = 16
[wfmash::skch::Sketch::build] minimizers picked from reference = 20497
[wfmash::skch::Sketch::index] unique minimizers = 1415
[wfmash::skch::Sketch::computeFreqHist] Frequency histogram of minimizers = (1, 261) ... (27, 1)
[wfmash::skch::Sketch::computeFreqHist] With threshold 0.001%, consider all minimizers during lookup.
[wfmash::map] time spent computing the reference index: 0.00455175 sec
[wfmash::skch::Map::mapQuery] mapped 100.00% @ 2.44e+05 bp/s elapsed: 00:00:00:00 remain: 00:00:00:00
[wfmash::skch::Map::mapQuery] count of mapped reads = 21, reads qualified for mapping = 21, total input reads = 21, total input bp = 60984
[wfmash::map] time spent mapping the query: 2.63e-01 sec
[wfmash::map] mapping results saved in: /data/pangene_graph_workflow/pangene-HORVU.MOREX.r2.3HG0222170_mmseq2pctid-0.9_mmseq2mincov-0.8_pggb-mapping-pctid-85_geneBuffer-1000bp/wfmash-H8TzOD
[wfmash::align] Reference = [/data/pangene_graph_workflow/pangene-HORVU.MOREX.r2.3HG0222170_mmseq2pctid-0.9_mmseq2mincov-0.8_pggb-mapping-pctid-85_geneBuffer-1000bp/HORVU.MOREX.r2.3HG0222170_pan_genes_buffer-1000bp.fna]
[wfmash::align] Query = [/data/pangene_graph_workflow/pangene-HORVU.MOREX.r2.3HG0222170_mmseq2pctid-0.9_mmseq2mincov-0.8_pggb-mapping-pctid-85_geneBuffer-1000bp/HORVU.MOREX.r2.3HG0222170_pan_genes_buffer-1000bp.fna]
[wfmash::align] Mapping file = /data/pangene_graph_workflow/pangene-HORVU.MOREX.r2.3HG0222170_mmseq2pctid-0.9_mmseq2mincov-0.8_pggb-mapping-pctid-85_geneBuffer-1000bp/wfmash-H8TzOD
[wfmash::align] Alignment identity cutoff = 8.50e-01%
[wfmash::align] Alignment output file = /dev/stdout
[wfmash::align] time spent read the reference sequences: 2.53e-04 sec
Command terminated by signal 4
wfmash -X -s 2000 -l 0 -N -p 85 -n 21 -k 16 -t 16 /data/pangene_graph_workflow/pangene-HORVU.MOREX.r2.3HG0222170_mmseq2pctid-0.9_mmseq2mincov-0.8_pggb-mapping-pctid-85_geneBuffer-1000bp/HORVU.MOREX.r2.3HG0222170_pan_genes_buffer-1000bp.fna /data/pangene_graph_workflow/pangene-HORVU.MOREX.r2.3HG0222170_mmseq2pctid-0.9_mmseq2mincov-0.8_pggb-mapping-pctid-85_geneBuffer-1000bp/HORVU.MOREX.r2.3HG0222170_pan_genes_buffer-1000bp.fna
0.34s user 0.05s system 18% cpu 2.16s total 21224Kb max memory
srun: error: node-4: task 0: Exited with exit code 132
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#95>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABDQEL2IXYSVO52A5KOEKTTMCQ6PANCNFSM44DYZ6XQ>
.
from pggb.
ekg
commented on August 11, 2024
Ah this is for just a single gene region.
Would you please share the FASTA? I can try to reproduce the issue.
…On Wed, May 5, 2021, 09:23 Erik Garrison ***@***.***> wrote:
I would suggest setting -s much higher (50-100kb) and -p closer to 2X the
nucleotide divergence. You appear to be running out of memory. The
alignment step is too difficult because of the very sensitive settings you
are using.
On Wed, May 5, 2021, 03:59 Brett Chapman ***@***.***> wrote:
> Hi
>
> I've just updated to the latest build and I'm now getting errors at the
> wfmash step aligning only small gene regions of a few thousand bp. I pulled
> the build from docker and set many of the parameters to the now recommended
> defaults (e.g. block_id_min=0). This has happened consistently with several
> different gene regions.
>
> + block_length=0
> + min_match_len=29
> + transclose_batch=10000000
> + block_weight_max=10000000
> + block_id_min=0
> + ratio_contain=0
> + path_jump_max=100
> + edge_jump_max=0
> + poa_params=1,4,6,2,26,1
> + segment_length=2000
> + srun -n 1 singularity exec --bind /data/pangene_graph_workflow/pangene-HORVU.MOREX.r2.3HG0222170_mmseq2pctid-0.9_mmseq2mincov-0.8_pggb-mapping-pctid-85_geneBuffer-1000bp:/data/pangene_graph_workflow/pangene-HORVU.MOREX.r2.3HG0222170_mmseq2pctid-0.9_mmseq2mincov-0.8_pggb-mapping-pctid-85_geneBuffer-1000bp /data/pggb_builds/pggb.sif pggb -i /data/pangene_graph_workflow/pangene-HORVU.MOREX.r2.3HG0222170_mmseq2pctid-0.9_mmseq2mincov-0.8_pggb-mapping-pctid-85_geneBuffer-1000bp/HORVU.MOREX.r2.3HG0222170_pan_genes_buffer-1000bp.fna -N -s 2000 -l 0 -p 85 -n 21 -k 29 -B 10000000 -w 10000000 -I 0 -R 0 -j 100 -e 0 -P 1,4,6,2,26,1 -t 16 -m -v -L -S -o pggb_output
> [wfmash::map] Reference = [/data/pangene_graph_workflow/pangene-HORVU.MOREX.r2.3HG0222170_mmseq2pctid-0.9_mmseq2mincov-0.8_pggb-mapping-pctid-85_geneBuffer-1000bp/HORVU.MOREX.r2.3HG0222170_pan_genes_buffer-1000bp.fna]
> [wfmash::map] Query = [/data/pangene_graph_workflow/pangene-HORVU.MOREX.r2.3HG0222170_mmseq2pctid-0.9_mmseq2mincov-0.8_pggb-mapping-pctid-85_geneBuffer-1000bp/HORVU.MOREX.r2.3HG0222170_pan_genes_buffer-1000bp.fna]
> [wfmash::map] Kmer size = 16
> [wfmash::map] Window size = 5
> [wfmash::map] Segment length = 2000 (read split disabled)
> [wfmash::map] Block length min = 0
> [wfmash::map] Alphabet = DNA
> [wfmash::map] Percentage identity threshold = 0.85%
> [wfmash::map] Mapping output file = /data/pangene_graph_workflow/pangene-HORVU.MOREX.r2.3HG0222170_mmseq2pctid-0.9_mmseq2mincov-0.8_pggb-mapping-pctid-85_geneBuffer-1000bp/wfmash-H8TzOD
> [wfmash::map] Filter mode = 1 (1 = map, 2 = one-to-one, 3 = none)
> [wfmash::map] Execution threads = 16
> [wfmash::skch::Sketch::build] minimizers picked from reference = 20497
> [wfmash::skch::Sketch::index] unique minimizers = 1415
> [wfmash::skch::Sketch::computeFreqHist] Frequency histogram of minimizers = (1, 261) ... (27, 1)
> [wfmash::skch::Sketch::computeFreqHist] With threshold 0.001%, consider all minimizers during lookup.
> [wfmash::map] time spent computing the reference index: 0.00455175 sec
> [wfmash::skch::Map::mapQuery] mapped 100.00% @ 2.44e+05 bp/s elapsed: 00:00:00:00 remain: 00:00:00:00
> [wfmash::skch::Map::mapQuery] count of mapped reads = 21, reads qualified for mapping = 21, total input reads = 21, total input bp = 60984
> [wfmash::map] time spent mapping the query: 2.63e-01 sec
> [wfmash::map] mapping results saved in: /data/pangene_graph_workflow/pangene-HORVU.MOREX.r2.3HG0222170_mmseq2pctid-0.9_mmseq2mincov-0.8_pggb-mapping-pctid-85_geneBuffer-1000bp/wfmash-H8TzOD
> [wfmash::align] Reference = [/data/pangene_graph_workflow/pangene-HORVU.MOREX.r2.3HG0222170_mmseq2pctid-0.9_mmseq2mincov-0.8_pggb-mapping-pctid-85_geneBuffer-1000bp/HORVU.MOREX.r2.3HG0222170_pan_genes_buffer-1000bp.fna]
> [wfmash::align] Query = [/data/pangene_graph_workflow/pangene-HORVU.MOREX.r2.3HG0222170_mmseq2pctid-0.9_mmseq2mincov-0.8_pggb-mapping-pctid-85_geneBuffer-1000bp/HORVU.MOREX.r2.3HG0222170_pan_genes_buffer-1000bp.fna]
> [wfmash::align] Mapping file = /data/pangene_graph_workflow/pangene-HORVU.MOREX.r2.3HG0222170_mmseq2pctid-0.9_mmseq2mincov-0.8_pggb-mapping-pctid-85_geneBuffer-1000bp/wfmash-H8TzOD
> [wfmash::align] Alignment identity cutoff = 8.50e-01%
> [wfmash::align] Alignment output file = /dev/stdout
> [wfmash::align] time spent read the reference sequences: 2.53e-04 sec
> Command terminated by signal 4
> wfmash -X -s 2000 -l 0 -N -p 85 -n 21 -k 16 -t 16 /data/pangene_graph_workflow/pangene-HORVU.MOREX.r2.3HG0222170_mmseq2pctid-0.9_mmseq2mincov-0.8_pggb-mapping-pctid-85_geneBuffer-1000bp/HORVU.MOREX.r2.3HG0222170_pan_genes_buffer-1000bp.fna /data/pangene_graph_workflow/pangene-HORVU.MOREX.r2.3HG0222170_mmseq2pctid-0.9_mmseq2mincov-0.8_pggb-mapping-pctid-85_geneBuffer-1000bp/HORVU.MOREX.r2.3HG0222170_pan_genes_buffer-1000bp.fna
> 0.34s user 0.05s system 18% cpu 2.16s total 21224Kb max memory
> srun: error: node-4: task 0: Exited with exit code 132
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <#95>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AABDQEL2IXYSVO52A5KOEKTTMCQ6PANCNFSM44DYZ6XQ>
> .
>
from pggb.
brettChapman
commented on August 11, 2024
Hi Erik
Thanks. I've attached an example of one of the fasta files. Previously I was using a pggb version from a month or two ago, and these sequences aligned using wfmash back then. I used the same wfmash parameters back then as well. The only parameters I've now changed are the smoothxg parameters (e.g. block_id_min = 0).
HORVU.MOREX.r2.5HG0400750_pan_genes_buffer-1000bp.fna.zip
from pggb.
ekg
commented on August 11, 2024
Ok I'll check. We have some suspicion that the problem is with the build.
Are you using the docker image?
…On Wed, May 5, 2021, 10:00 Brett Chapman ***@***.***> wrote:
Hi Erik
Thanks. I've attached an example of one of the fasta files. Previously I
was using a pggb version from a month or two ago, and these sequences
aligned using wfmash back then. I used the same wfmash parameters back then
as well. The only parameters I've now changed are the smoothxg parameters
(e.g. block_id_min = 0).
HORVU.MOREX.r2.5HG0400750_pan_genes_buffer-1000bp.fna.zip
<https://github.com/pangenome/pggb/files/6425690/HORVU.MOREX.r2.5HG0400750_pan_genes_buffer-1000bp.fna.zip>
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#95 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABDQEMVPBCII34ZUQOFWXLTMD3LJANCNFSM44DYZ6XQ>
.
from pggb.
brettChapman
commented on August 11, 2024
Yeah I pulled the build from docker
from pggb.
brettChapman
commented on August 11, 2024
@ekg @subwaystation I can confirm that the new docker pull works. However I'm now not getting the parameters embedded in the file names, instead what looks like the version information (e.g. ba89602 for wfmash). Is this intentional? In the pggb script it still looks like its taking each parameter and naming the files accordingly. Thanks.
ls -lhtr pangene-HORVU.MOREX.r2.5HG0400750_mmseq2pctid-0.9_mmseq2mincov-0.8_pggb-mapping-pctid-85_geneBuffer-1000bp/pggb_output/
total 2.9M
-rw-rw-r-- 1 ubuntu ubuntu 1.1K May 6 00:59 HORVU.MOREX.r2.5HG0400750_pan_genes_buffer-1000bp.fna.ba89602.2ff309f.68f4dff.05-06-2021_00:59:29.params.yml
-rw-rw-r-- 1 ubuntu ubuntu 66K May 6 00:59 HORVU.MOREX.r2.5HG0400750_pan_genes_buffer-1000bp.fna.ba89602.wfmash.paf
-rw-rw-r-- 1 ubuntu ubuntu 34K May 6 00:59 HORVU.MOREX.r2.5HG0400750_pan_genes_buffer-1000bp.fna.ba89602.2ff309f.seqwish.gfa
-rw-rw-r-- 1 ubuntu ubuntu 41K May 6 00:59 HORVU.MOREX.r2.5HG0400750_pan_genes_buffer-1000bp.fna.ba89602.2ff309f.seqwish.gfa.prep.gfa
-rw-rw-r-- 1 ubuntu ubuntu 115K May 6 00:59 HORVU.MOREX.r2.5HG0400750_pan_genes_buffer-1000bp.fna.ba89602.2ff309f.68f4dff.smooth.gfa
-rw-rw-r-- 1 ubuntu ubuntu 212K May 6 00:59 HORVU.MOREX.r2.5HG0400750_pan_genes_buffer-1000bp.fna.ba89602.2ff309f.68f4dff.smooth.og
-rw-rw-r-- 1 ubuntu ubuntu 76K May 6 00:59 HORVU.MOREX.r2.5HG0400750_pan_genes_buffer-1000bp.fna.ba89602.2ff309f.seqwish.og
-rw-rw-r-- 1 ubuntu ubuntu 330 May 6 00:59 HORVU.MOREX.r2.5HG0400750_pan_genes_buffer-1000bp.fna.ba89602.2ff309f.seqwish.og.stats
-rw-rw-r-- 1 ubuntu ubuntu 333 May 6 00:59 HORVU.MOREX.r2.5HG0400750_pan_genes_buffer-1000bp.fna.ba89602.2ff309f.68f4dff.smooth.og.stats
-rw-rw-r-- 1 ubuntu ubuntu 9.2K May 6 00:59 HORVU.MOREX.r2.5HG0400750_pan_genes_buffer-1000bp.fna.ba89602.2ff309f.68f4dff.smooth.og.viz_mqc.png
-rw-rw-r-- 1 ubuntu ubuntu 38K May 6 00:59 HORVU.MOREX.r2.5HG0400750_pan_genes_buffer-1000bp.fna.ba89602.2ff309f.68f4dff.smooth.og.lay
-rw-rw-r-- 1 ubuntu ubuntu 28K May 6 00:59 HORVU.MOREX.r2.5HG0400750_pan_genes_buffer-1000bp.fna.ba89602.2ff309f.68f4dff.smooth.og.lay.draw.png
-rw-rw-r-- 1 ubuntu ubuntu 519K May 6 00:59 HORVU.MOREX.r2.5HG0400750_pan_genes_buffer-1000bp.fna.ba89602.2ff309f.68f4dff.smooth.og.lay.draw_mqc.png
-rw-rw-r-- 1 ubuntu ubuntu 1.8M May 6 01:00 multiqc_report.html
drwxrwxr-x 2 ubuntu ubuntu 4.0K May 6 01:00 multiqc_data
-rw-rw-r-- 1 ubuntu ubuntu 15K May 6 01:00 HORVU.MOREX.r2.5HG0400750_pan_genes_buffer-1000bp.fna.ba89602.2ff309f.68f4dff.05-06-2021_00:59:29.log
from pggb.
AndreaGuarracino
commented on August 11, 2024
Hi @brettChapman, yes, that's intentional. This is a way to have a naming scheme still bound to the parameter inputs (those strings are hashes computed from the input options, they are not commit hashes) and a bit more polite with the names' length.
from pggb.
subwaystation
commented on August 11, 2024
@brettChapman got it to work ;)
So closing this one.
from pggb.
Related Issues (20)
- Joining the Chromosome Alignments？ HOT 2
- Problem with the ODGI 1D visualization by path orientation HOT 6
- Producing subgraphs of large pangenomes HOT 2
- Converting to a single connected component HOT 2
- vcfwave not found HOT 2
- Construct pangenome HOT 3
- Problems with skipping the wfmash step HOT 8
- Even if the genome is small, shouldn't too many samples be added to build the graph HOT 8
- Use of pggb to generate alignments for phylogenomics HOT 2
- Wrong setting of POA, but it still finish without error HOT 4
- Path to paf2net.py not correct in latest Docker build HOT 2
- 'SN:Z' and 'SO:i' tags for segments/nodes HOT 1
- How to handle when two contigs from the same assembly sligtly overlap HOT 7
- Construct PanGenome HOT 7
- How to understand the vcf file output by PGGB？ HOT 3
- the parameters to build pangenome for metagenomes HOT 2
- [docker] wfmash: error while loading shared libraries: libwfa2cpp.so.0 HOT 1
- --n-haplotypes option not recognized HOT 2
- vg deconstruct error HOT 5
- Command terminated by signal 8 HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pggb.