R does not work after relocation about spack HOT 6 CLOSED

Okay, this is what happens:

R stores an absolute path to <prefix>/bin/which in byte code, and that byte code is compressed. It can be retrieved like this:

$ ./bin/R --quiet -e 'readRDS("./rlib/R/library/base/R/base.rdx")$variables$Sys.which' 2>/dev/null
> readRDS("./rlib/R/library/base/R/base.rdx")$variables$Sys.which
[1] 133838   2044

meaning the serialized stuff for Sys.which is stored at offset 133838 with length 2044 in base.rdb. That itself starts with a 4 byte integer, so those have to be skipped. Then a zlib header follows:

$ dd if=./rlib/R/library/base/R/base.rdb bs=1 skip=133842 count=2044 status=none | zlib-flate -uncompress
...
/home/software/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeh/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/which-2.21-3j5ymd5gybivcdmkxy53tdgq5vzuk5ys/bin/which
...

So in short: a string we need to relocate is stored as compressed, serialized byte code in some sort of database for serialized objects. In principle it's possible to decompress the blocks in base.rdb, rewrite strings, store again, and update the offsets in base.rdx -- but that's pretty involved.

Hopefully R can use which from PATH instead of an absolute path?

from spack.

Submitted a small patch to R that uses indirection through a symlink to make relocation of which work.

from spack.

I think the issue is that R compresses the original install prefix and decompresses at runtime, strace shows it tries to execute <old prefix>/bin/which for example.

from spack.

Hopefully R can use which from PATH instead of an absolute path?

AC_PATH_PROGS(WHICH, which, which)
if test "${WHICH}" = which ; then
  ## needed to build and run R
  ## ends up hard-coded in the utils package
  AC_MSG_ERROR([[which is required but missing]])
fi

looks like we can't.

Another alternative is to look if we can disable compression of byte code. (Edit: looks like you can, but now idk where the source code for Sys.which is coming from...)

from spack.

Disabling byte compilation doesn't help, it still stores something compressed & serialized in base.rdb. The file is smaller, so I guess there's a difference between byte code and compiled byte code.

Further options:

• Easiest is to patch the R build system to allow WHICH=which and expect it to be in PATH.
• Disabling compression of rdb entries (probably a bad idea)
• Ask R people to create a symlink to which in their prefix, have them register that in their R sources, so no absolute paths are registered in bytecode.

from spack.

@haampie I can't thank you enough for getting to the root cause. I've tested installing R from a buildcache to a new location and the default packages are now successfully found at start-up. Also, with the addition of gcc-runtime, R is, as far as I can tell, fully relocatable! Thanks!

from spack.