pod5 convert fast5 is stalling about pod5-file-format HOT 4 CLOSED

Hi @sahuno,
Can you please reduce the number of --threads set and try again 4 might be a good first attempt. This argument controls the number of processes spawned as described in the docs and it might be causing memory issues.

Kind regards,
Rich

from pod5-file-format.

thank you @HalfPhoton . successfully converted fast5 -> pod5 by reducing threads to 4. Que: indulge me, is there a reason why more threads was counterproductive even on machine with multiple cores? I'm confused

from pod5-file-format.

I'm glad that you're now unblocked.

This argument controls the number of processes (not threads) spawned as described in the docs and it might be causing memory issues.

This is the docs snippet:

The --threads argument controls the number of additional processes spawned by the tools where necessary. The default value for --threads is 4 unless the number of available cpu cores is fewer in which case this lower value is used.
Specifying a --threads value less than 1 will use the number of available cpu cores. The --help text shown for all tools should show the default value on your system.
If you experience errors which mention threading or spawning processes. Please reduce the --threads value.

Asking for 64 python processes which each have their own runtime and load each the api which itself internally spawns 10 threads can quickly cause IO issues.

Kind regards,
Rich

from pod5-file-format.

that explains!
thank you once again!

from pod5-file-format.