Metabuli v1.0.0 segfault about metabuli HOT 8 CLOSED

With the official binary AVX2 binaries:

$ md5sum /vol/sge-tmp/xxxy.in /vol/sge-tmp/in.fas 
27fe50b9a8a5829aefc66168ec5a783f  /vol/sge-tmp/xxxy.in
27fe50b9a8a5829aefc66168ec5a783f  /vol/sge-tmp/in.fas

~/metabuli_test/metabuli/bin/metabuli classify --seq-mode 1 /vol/sge-tmp/in.fas /vol/biodb/local_databases/MGX/metabuli/refseq/ outdir jobid works as intended, while ~/metabuli_test/metabuli/bin/metabuli classify --seq-mode 1 /vol/sge-tmp/xxxy.in /vol/biodb/local_databases/MGX/metabuli/refseq/ outdir jobid crashes (reproducibly).

Core was generated by `/homes/sjaenick/metabuli_test/metabuli/bin/metabuli classify --seq-mode 1 /vol/'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000000044d843 in kseq_buffer_reader (nbyte=16384, outBuffer=0xb0f9b90 "\320N\340\001", inBuffer=0x7ffdf3ef8140)
    at /home/vsts/work/1/s/lib/mmseqs/src/commons/KSeqBufferReader.h:30
30      /home/vsts/work/1/s/lib/mmseqs/src/commons/KSeqBufferReader.h: No such file or directory.
[Current thread is 1 (Thread 0x2597940 (LWP 3753118))]
(gdb) bt
#0  0x000000000044d843 in kseq_buffer_reader (nbyte=16384, outBuffer=0xb0f9b90 "\320N\340\001", inBuffer=0x7ffdf3ef8140)
    at /home/vsts/work/1/s/lib/mmseqs/src/commons/KSeqBufferReader.h:30
#1  ks_getc (ks=0xb0f9b50) at /home/vsts/work/1/s/src/commons/SeqIterator.h:25
#2  kseq_read (seq=seq@entry=0xb0f9ac0) at /home/vsts/work/1/s/src/commons/SeqIterator.h:25
#3  0x000000000044efbf in Classifier::_ZN10Classifier27fillQueryKmerBufferParallelER15QueryKmerBufferR10MmapedDataIcERKSt6vectorI13SequenceBlockSaIS6_EERS5_I5QuerySaISB_EERKSt4pairImmERK15LocalParameters._omp_fn.0(void) () at /home/vsts/work/1/s/src/commons/Classifier.cpp:354
#4  0x0000000000a6ff06 in GOMP_parallel ()
#5  0x000000000045b619 in Classifier::fillQueryKmerBufferParallel (par=..., currentSplit=..., queryList=..., seqs=..., seqFile=..., kmerBuffer=..., this=0x25aecf0)
    at /home/vsts/work/1/s/src/commons/Classifier.cpp:345
#6  Classifier::startClassify (this=this@entry=0x25aecf0, par=...) at /home/vsts/work/1/s/src/commons/Classifier.cpp:260
#7  0x0000000000494c7c in classify (argc=<optimized out>, argv=<optimized out>, command=...) at /home/vsts/work/1/s/src/workflow/classify.cpp:47
#8  0x000000000049d62a in runCommand (p=0x25b2f60, argc=argc@entry=6, argv=argv@entry=0x7ffdf3ef97b8) at /home/vsts/work/1/s/lib/mmseqs/src/commons/Application.cpp:40
#9  0x0000000000414c26 in main (argc=8, argv=0x7ffdf3ef97a8) at /home/vsts/work/1/s/lib/mmseqs/src/commons/Application.cpp:203
(gdb)

from metabuli.

Even more strange things:

$ id -a
uid=1096(sjaenick) gid=1000(cb) groups=1000(cb),1022(mgx),....

Running as myself (sjaenick), this works - file owned by myself:
-rw-r--r-- 1 sjaenick cb 17619148 Jun 5 19:00 /vol/sge-tmp/xxxy.in

... while this segfaults, even though the file is readable via its group (and world) permissions:
-rw-r--r-- 1 mgxserv mgx 17619148 Jun 5 19:02 /vol/sge-tmp/xxxy.in

Performing exactly the same using the mgxserv account, I get segfaults for
both input files (even though one of the input files even is owned by this account).
To be honest, I have no idea what might be going on.. .

from metabuli.

Thank you for sharing the issue with great details! It will help us provide more stable software.

1. Segfault during Comparing qeury and reference metamers stage
   Could you first try using multiple threads?
   I could reproduce the Segfault during Comparing qeury and reference metamers stage only with --threads 1.
   Using more than one thread didn't make any problems.
   It will be very helpful if you try with multiple threads and share the result.

2. Segfault during metamer extraction stage
   I couldn't reproduce Segfault during metamer extraction yet, so it may take more time.
   Please use Metabuli with working setting until we solve this issue.

from metabuli.

Thank you so much for the detailed issue description and experiments. I also tried to reproduce the issue as well but failed. Just to clarify, if you run metabuli under your own account it does not crash (at all), but if you use another user it crashes?

from metabuli.

Thank you so much for the detailed issue description and experiments. I also tried to reproduce the issue as well but failed. Just to clarify, if you run metabuli under your own account it does not crash (at all), but if you use another user it crashes?

It crashes both under my own account as well as with another one; the strange thing here is that for an input file readable by both accounts, the outcome reproducibly differs. Also, copying the input file to a different location makes a difference (and I have no idea what might cause this).

-rw-r--r-- 1 mgxserv mgx 17619148 Jun 5 19:02 /vol/sge-tmp/xxxy.in
/vol/mgx-sw/bin/metabuli classify --seq-mode 1 /vol/sge-tmp/xxxy.in /vol/biodb/local_databases/MGX/metabuli/refseq/ outdir jobid

When executed under my own account, this segfaults in the metamer extraction step; when run under the mgxserv account, it segfaults in the 'Comparing qeury and reference metamers' step.

from metabuli.

1. **Segfault during `Comparing qeury and reference metamers` stage**
   Could you first try using multiple threads?
   I could reproduce the _Segfault during `Comparing qeury and reference metamers` stage_ only with `--threads 1`.
   Using more than one thread didn't make any problems.
   It will be very helpful if you try with multiple threads and share the result.

I just tested this; indeed, singlethreaded usage leads to the segfault, while specifying more than one thread works fine.

EDIT: It still segfaults, even with more than one thread:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  Classifier::linearSearchParallel (this=0x557195984800, queryKmerList=0x154714aee010, queryKmerCnt=@0x7ffff48b4b18: 28860420, matchBuffer=..., par=...)
    at /vol/mgx-sw/src/tools/Metabuli-1.0.0/src/commons/Classifier.cpp:501
501             if (diffIdxSplits.data[i].ADkmer == 0 || diffIdxSplits.data[i].ADkmer == UINT64_MAX) {
(gdb) bt
#0  Classifier::linearSearchParallel (this=0x557195984800, queryKmerList=0x154714aee010, queryKmerCnt=@0x7ffff48b4b18: 28860420, matchBuffer=..., par=...)
    at /vol/mgx-sw/src/tools/Metabuli-1.0.0/src/commons/Classifier.cpp:501
#1  0x0000557192d0272f in Classifier::startClassify (this=0x557195984800, par=...) at /vol/mgx-sw/src/tools/Metabuli-1.0.0/src/commons/Classifier.cpp:281
#2  0x0000557192d881a7 in classify (argc=8, argv=0x7ffff48b52f8, command=...) at /vol/mgx-sw/src/tools/Metabuli-1.0.0/src/workflow/classify.cpp:47
#3  0x0000557192d95ea4 in runCommand (p=0x5571959701e0, argc=8, argv=0x7ffff48b52f8)
    at /vol/mgx-sw/src/tools/Metabuli-1.0.0/lib/mmseqs/src/commons/Application.cpp:40
#4  0x0000557192d96ecc in main (argc=10, argv=0x7ffff48b52e8) at /vol/mgx-sw/src/tools/Metabuli-1.0.0/lib/mmseqs/src/commons/Application.cpp:203
(gdb) p i
$1 = 1
(gdb) p diffIdxSplits.data[i]
Cannot access memory at address 0x17

from metabuli.

2. **Segfault during metamer extraction stage**
   I couldn't reproduce _Segfault during metamer extraction_ yet, so it may take more time.
   Please use Metabuli with working setting until we solve this issue.

I have a testcase here: If the input file is writable (e.g. permissions 644), it works. Change it to
readonly (chmod 444 file.fas), the segfault occurs.

https://github.com/steineggerlab/Metabuli/blob/master/src/commons/Classifier.cpp#L161 creates
a R/W mapping of the input file (see https://github.com/steineggerlab/Metabuli/blob/master/src/commons/Mmap.h#L30)
since no value for mode is specified. If you change this to

queryFile = mmapData<char>(queryPath_1.c_str(), 2); // mmap readonly

the segfault during metamer extraction seems to be fixed.

from metabuli.

i think the second segfault (during 'Comparing qeury and reference metamers') might be due to a similar
problem:

https://github.com/steineggerlab/Metabuli/blob/master/src/commons/Classifier.cpp#L485 creates a R/W mapping
of one of the database files and later attempts to modify its data (e.g. https://github.com/steineggerlab/Metabuli/blob/master/src/commons/Classifier.cpp#L502); in my case, the database files are not writable by the mgxserv
account, hence the segfault.

A readonly mapping, i.e. mmapData<DiffIdxSplit>(diffIdxSplitFileName.c_str(), 2); won't work here (since it's being
modified in memory). I'll see if i can add a R/W mode with MAP_PRIVATE.

from metabuli.