jemalloc-sys fails to build with musl on Ubuntu 16.04 about jemallocator HOT 16 CLOSED

I built jemalloc with x86_64-pc-linux-musl target on Ubuntu successfully by changing /usr/bin/musl-gcc from

#!/bin/sh
exec "${REALGCC:-cc}" "$@" -specs /usr/share/dpkg/no-pie-compile.specs -specs /usr/share/dpkg/no-pie-link.specs -specs "/usr/l
ib/x86_64-linux-musl/musl-gcc.specs"

into

#!/bin/sh
exec "${REALGCC:-gcc}" "$@" -specs "/usr/lib/x86_64-linux-musl/musl-gcc.specs"

from jemallocator.

Yeah, I fixed that. You can see a passing build with cross + jemalloc + Ubuntu 16.04 + musl + x86_64 here: https://travis-ci.org/BurntSushi/ripgrep/jobs/524515554

So it sounds like cross is working OK. But the normal cargo build --target x86_64-unknown-linux-musl still fails.

from jemallocator.

Now that I have builds working again with cross, I'm not sure if or when I'll be debugging jemalloc compilation unfortunately. I'm not particularly experienced with configuring cross compilation toolchains, and I don't have an easy way to test an Ubuntu 16.04 box at the moment, so it would probably be a lot of work for me. If I get some spare time I'll see about looking into it.

I expect this is likely to bite people who are producing static binaries with musl using Travis CI if and when they notice that musl's allocator is dog slow and want to switch back to jemalloc.

from jemallocator.

cross exists to save you from having to know about this, so if that works for you I would call it a day and worry more about it when you hit "the next issue".

Emitting better diagnostic when somebody attempts to build the i686 musl target is being tracked here in case you want to follow that: #106

from jemallocator.

I think it's not a bug in jemaloc. Ubuntu package mantainer patched musl-gcc to disable PIE linking with some reason.

2017-01-03 - Reiner Herrmann [email protected]

• Disable PIE linking in musl-gcc wrapper to workaround linking issues
  (Closes: #847776).
  https://ubuntu.pkgs.org/18.04/ubuntu-universe-amd64/musl-tools_1.1.19-1_amd64.deb.html

from jemallocator.

cc @alexcrichton Is this related to rust-lang/rust#59411?

from jemallocator.

See also: BurntSushi/ripgrep#1268

from jemallocator.

@BurntSushi can you test compiling with cross build --target=x86_64-unknown-linux-musl ? (you need to cargo install cross first). I just want to figure out whether this is a ripgrep specific issue, or whether it might have to do with your C cross-compilation toolchain. If cross build works, chances are your cross compilation toolchain might not be properly set (although it could also mean that there is an issue with the musl version being used, but we can figure that out as the next step).

from jemallocator.

For comparison, our build jobs do not show the "configure: WARNING: using cross tools not prefixed with host triplet" warning: https://travis-ci.com/gnzlbg/jemallocator/jobs/195415085#L665

from jemallocator.

I tried simply removing --features pcre2 from my CI build (which avoids compiling PCRE2), but that seems to have resulted in the same error: https://travis-ci.org/BurntSushi/ripgrep/jobs/524438703

Keeping PCRE2 disabled, I then tried using cross in the CI build, and that worked: https://travis-ci.org/BurntSushi/ripgrep/jobs/524440678 But it fails on i686-unknown-linux-musl: https://travis-ci.org/BurntSushi/ripgrep/jobs/524440677

And re-enabling PCRE2 with cross fails: https://travis-ci.org/BurntSushi/ripgrep/jobs/524447301

Here is the pcre2-sys build script: https://github.com/BurntSushi/rust-pcre2/blob/master/pcre2-sys/build.rs

So this seems like quite a thorny problem. There are multiple variables at play here:

• A normal cargo build with musl and with PCRE2 fails.
• A normal cargo build with musl but without PCRE2 fails. (Seemingly same error as above.)
• A cross build with musl but without PCRE2 works on x86_64.
• A cross build with musl but without PCRE2 fails on i686 with a distinct error from anything else, i.e., undefined reference to __ffsdi2'`. This appears to be this bug: jemalloc/jemalloc#1464
• A cross build with musl but with PCRE2 fails, but with a distinct error in the build script. It looks like calling git inside cross's docker container doesn't work because git isn't available.

I've never used cross before, but the normal Cargo builds worked with musl when Rust was bundling jemalloc.

from jemallocator.

But it fails on i686-unknown-linux-musl

jemalloc does not support that target anymore, so if you want to use jemalloc there you'd have to patch jemalloc itself

from jemallocator.

And re-enabling PCRE2 with cross fails: https://travis-ci.org/BurntSushi/ripgrep/jobs/524447301

From inspecting that, it errors at line 75 of the build.rs: https://github.com/BurntSushi/rust-pcre2/blob/master/pcre2-sys/build.rs#L75

Either git is not available inside cross (IIRC this should work - or was recently added), or the command might be trying to modify the source directory, which cross forbids, e.g., if you want to checkout a git repo, cross forces doing that into the OUT_DIR (e.g. target/.../) - the source directory is read-only while building.

from jemallocator.

But the normal cargo build --target x86_64-unknown-linux-musl still fails.

So I suspect that your cross-compilation toolchain for C might not be set up appropriately for compiling jemalloc. I think it might be better to try compiling jemalloc from source first and see if that succeeds. Basically:

git clone https://github.com/jemalloc/jemalloc
cd jemalloc
autoconf
./configure
make -j3 
make test -j3

But the autoconf / makefile environment variables (CC, CXX, TARGET, HOST, ... ) need to be set properly for that to work. If that works, but cargo build --target=... does not, then we have a problem in the build.rs. If that does not work, then maybe something in the environment / cross-compilation toolchain might be making the build of jemalloc fail.

from jemallocator.

Is this a bug in the jemalloc makefiles / configure script ? If so it might be best to report this upstream.

from jemallocator.

So is this an ubuntu bug ?

from jemallocator.

Sort of, It seems like the maintainer disable PIE on purpose to fix another bug
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=847776

And the bug was fixed in 1.1.20

musl (1.1.20-1) unstable; urgency=medium

• Drop patch with musl-gcc workaround for linking with default-pie
  toolchains, as the upstream specs file now includes a better solution.

from jemallocator.