marginpolish docker stuck at 99% about helen HOT 13 CLOSED

Hi @MichelMoser ,

This is unexpected behavior, did you let it run to completion. Maybe it's an issue with coverage of one region that is maybe causing this. What is the coverage of your bam file?

I will discuss this with @tpesout to see if it's something we've seen before.

from helen.

Hi @kishwarshafin ,

I let it run for additional 24 hours once it reached the

> Polishing 99% complete (541585/546365).  Estimated time remaining: 17m 57s

but h5 files did not change since then.

Average coverage is about 60 x.
I thought downsampling was implemented in the .json "maxDepth" when generating images?

Best,
Michel

from helen.

@MichelMoser ,

I ran two polishing runs since last night with docker and both finished correctly. Would it be possible to prune the images and run it one more time?

If you are spending too much time on this then if you share the files I can try to see what is causing the issue.

You are right, the maxDepth controls the downsampling.

from helen.

What do you mean by pruning the images?
Yes will rerun marginpolish one more time and report back.

from helen.

docker rmi <helen_docker_image>

remove the existing docker image of and pull it again. It would be crazy if this works.

from helen.

i assume using singluarity instead of docker is not the source of error

from helen.

hmm, its running for 1.5 half days now straight and still at "Polishing 99% complete". Gave it 96 threads.

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
177576 michelmo  20   0  374.1g 369.8g   1488 S 291.8 12.2 125225:18 marginPolish

How long are marginpolish runtimes for human genomes normally?

from helen.

@MichelMoser , sorry about this. Usually, a human genome takes about 10-15 hours, on 96 threads it should take less than that. This would be your second run where it got stuck, is that right?

from helen.

yes, its the second run. still at "Polishing 99% complete". You said it might be coverage problems?

I ran polishing with PEPPER simultaneously and it ran through without a hitch within 23.5 hours (on GPUs).

from helen.

@MichelMoser , at this point, with all the improvements in the basecaller, you should be able to see similar results with PEPPER and MarginPolish-HELEN.

If it's not inconvenient for you, I'd like to keep this issue open and get back to it to see if it happens to any other assemblies. This is very unusual and should be looked into.

from helen.

Hi @MichelMoser, I'm sorry you're having issues running this. I have seen something like this happen (not ever for 24 extra hours) with human reads aligned to GRCh38 with minimap2, in a very deep region flanked by very shallow regions (generally satellite DNA). There are some ways to verify this which I'm happy to do if you're willing to share your data. Also, running MarginPolish with the -a info flag will produce log messages that can help diagnose.

from helen.

Hi @tpesout and @kishwarshafin ,
Thank you for the help and I am happy to share files. But before transferring the 180 GByte file, i could generate some coverage stats with mosdepth and send you the results if that's helpful.
Also I can rerun with the logging option and send you the output.

from helen.

@MichelMoser , coverage plots, and the log would be great!

from helen.