Usage of library with ProcessPoolExecutor raise concurrent.futures.process.BrokenProcessPool about pyrx HOT 5 CLOSED

From the error message shown, this does not look to be a pyrx problem. Please either show a more specific error message that points to pyrx as the root cause of this issue.

from pyrx.

I've identified the error, the process are killed by oom killer because the
system run out of memory during pyrx initialization ! Here the logs :

[37775.066200] python3 invoked oom-killer:
gfp_mask=0x100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0
[37775.066479]
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-29.scope,task=python3,pid=15079,uid=0
[37775.066533] Out of memory: Killed process 15079 (python3)
total-vm:2639560kB,anon-rss:624304kB, file-rss:4kB, shmem-rss:8kB, UID:0
pgtables:1336kB oom_score_adj:0

Any suggestion to only initialize pyrx one time for two process or maybe to
reduce memory usage of it ?

from pyrx.

I've solved the issue by measuring the available free memory and divid it by the number of cpu cores then i define the maximum usage of memory per process with this function :

`def initializer(self):
gc.collect()

    free_memory = 0

    with open('/proc/meminfo', 'r') as mem:
        for i in mem:
            sline = i.split()
            if str(sline[0]) in ('MemFree:', 'Buffers:', 'Cached:'):
                free_memory += int(sline[1])

    soft, hard = resource.getrlimit(resource.RLIMIT_AS)
    resource.setrlimit(resource.RLIMIT_AS, (int(free_memory * 1024 / 2), hard))`

Unfortunetly it lead to not being able to use large page i don't know why i've tried to add this :

os.system("sysctl -w vm.nr_hugepages=1280")

inside the initializer without success maybe it should not be 1280 but another value because of the number of process ?

Edit :

Pyrx is in fact initialized two times because there is two cores on the processor and Processpoolexecutor start two process. How to initialize only one time pyrx ? thanks

from pyrx.

Nobody have the solution to initialize pyrx one time and use it in two different process ? I mean with the same dataset ? Process can help to gain some power compare to thread thats why im trying this way. Im still stuck there.

from pyrx.

This is not a support forum, it's a bug tracker, and this is not a bug.

from pyrx.