jemallocator in cdylib ? about jemallocator HOT 11 CLOSED

It seems jemallocator = {version = "0.3", features = ["disable_initial_exec_tls"]} will be enough !

from jemallocator.

How are you using jemallocator ? Typically, the #[global_allocator] is something that can only be meaningfully chosen by the final binary, and when compiling a cdylib, that library should work when linked to multiple different binaries, all using different allocators.

from jemallocator.

For the moment, I just put:

#[global_allocator]
static ALLOC: jemallocator::Jemalloc = jemallocator::Jemalloc;

in my lib.rs.

Which is enough to switch allocators for cargo test.

from jemallocator.

And I build the python extension based on that example : https://github.com/PyO3/rust-numpy/tree/master/examples/simple-extension

from jemallocator.

So you probably also need to get the final binary that links to your cdylib to use the same exact version of jemalloc configured in the exact same way. That might be hard to do if the binary that you are using is python which might be using a different allocator.

Hmmm, have you tried generating a staticlib instead of a cdylib ? Pinging: @alexcrichton , they might be able to help or know who else could help. I'm not really sure what happens when you put an allocator in a cdylib - if it gets dynamically linked, it would probably be replaced by the allocator in the final library, and if it gets statically linked, then you can't transfer ownerships of allocations across the cdylib boundary (e.g. you can't allocate memory in the cdylib, and free it in the final binary somewhere else).

from jemallocator.

Generating a staticlib instead of a cdylib:

setuptools_rust forces cdylib: https://github.com/PyO3/setuptools-rust/blob/v0.10.6/setuptools_rust/build.py#L142
I changed the code to staticlib, but the lib is not picked up by setuptools_rust, because they specifically look for .so files : https://github.com/PyO3/setuptools-rust/blob/v0.10.6/setuptools_rust/build.py#L226

I'm not sure a static lib will be easy to use as a python extension anyway..

from jemallocator.

the final binary that links to your cdylib

Well, setuptools_rust just copies the .so generated by rustc, and from there, the my_lib.so can be used dynamically from python with:

import my_lib

There's no other binary than python itself..

I'll resume working on it on Monday.

from jemallocator.

I'm not sure either. Maybe you can try asking in users.rust-lang.org forums ? Other people there might be able to help :)

from jemallocator.

Thanks, I'll also ask on the pyo3 repo.

from jemallocator.

jemalloc/jemalloc#937 seems related.

from jemallocator.

Good find! Glad it works now!

from jemallocator.