tweag / clodl Goto Github PK
View Code? Open in Web Editor NEWTurn dynamically linked ELF binaries and libraries into self-contained closures.
License: BSD 3-Clause "New" or "Revised" License
Turn dynamically linked ELF binaries and libraries into self-contained closures.
License: BSD 3-Clause "New" or "Revised" License
I'm trying to use binary_closure
(commit ed9be32ef539ea3645847987e11609028d6491f5
, master
at time of writing) to create a closure of a haskell_binary
, but when I do I'm getting:
ERROR: /.../BUILD:43:1: Linking of rule '//engine:test-closure-closure_wrapper' failed (Exit 1)
../sysdeps/x86_64/start.S:110: error: undefined reference to 'main'
collect2: error: ld returned 1 exit status
which presumably indicates that the binary's main function isn't being picked up correctly. For reference, the haskell_binary
looks like:
haskell_binary(
name = "test-bin",
src_strip_prefix = "bin/test/test-bin/src",
srcs = glob(['bin/test/test-bin/src/**/*.hs']),
deps = [
":test-lib",
],
prebuilt_dependencies = [
"base",
],
)
haskell_library(
name = "test-lib",
src_strip_prefix = "lib/test-lib/src",
srcs = glob(['lib/test-lib/src/**/*.hs']),
deps = [
hazel_library("bifunctors"),
hazel_library("constraints"),
hazel_library("dlist"),
hazel_library("hashable"),
hazel_library("lens"),
hazel_library("mmorph"),
hazel_library("mono-traversable"),
hazel_library("protolude"),
hazel_library("tagged"),
hazel_library("text"),
],
prebuilt_dependencies = [
"base",
"bytestring",
"containers",
"transformers",
],
)
while the closure is built with:
binary_closure(
name = "test-closure",
src = "test-bin",
)
(If context needed, hazel_library
is provided by FormationAI/hazel and creates external dependencies out of Hackage packages)
The binary works when built and run with bazel run //engine:test-bin
. When I run bazel build --subcommands //engine:test-closure
I see:
SUBCOMMAND: # //engine:test-closure-closure_wrapper [action 'Linking engine/test-closure-closure_wrapper']
(cd /.../.cache/bazel/_bazel_../540...ae3a/execroot/... && \
exec env - \
PATH=/../.nix-profile/bin:/.. \
PWD=/proc/self/cwd \
/.../.cache/bazel/_bazel_.../540...ae3a/external/gcc/bin/gcc -o bazel-out/k8-fastbuild/bin/engine/test-closure-closure_wrapper '-Wl,-rpath,$ORIGIN/../_solib_k8/_U_S_Sengine_Ctest-closure-closure_Uwrapper___Uengine_Sengine_UStest-closure-closure_UUsolib' -Lbazel-out/k8-fastbuild/bin/_solib_k8/_U_S_Sengine_Ctest-closure-closure_Uwrapper___Uengine_Sengine_UStest-closure-closure_UUsolib -Lengine_Stest-closure-closure_Usolib '-Wl,-rpath=$ORIGIN' bazel-out/k8-fastbuild/genfiles/engine/test-closure-closure-params.ld -Tbazel-out/k8-fastbuild/genfiles/engine/test-closure-closure-search_dirs.ld '-fuse-ld=gold' -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/.../.cache/bazel/_bazel_.../540...ae3a/external/gcc/bin -B/usr/bin -pass-exit-codes -Wl,-S -Wl,@bazel-out/k8-fastbuild/bin/engine/test-closure-closure_wrapper-2.params)
ERROR: /.../BUILD:43:1: Linking of rule '//engine:test-closure-closure_wrapper' failed (Exit 1)
../sysdeps/x86_64/start.S:110: error: undefined reference to 'main'
collect2: error: ld returned 1 exit status
Target //engine:test-closure failed to build
test-closure_closure_wrapper-2.params
contains:
-lHSrts_thr-ghc8.2.2
-lexternal_Shaskell_Utext_Slibtext-cbits
-lexternal_Shaskell_Uhashable_Slibhashable-cbits
-lexternal_Shaskell_Uprimitive_Slibprimitive-cbits
-lexternal_Shaskell_Utext_Stext-cbits_libtext-cbits
-lexternal_Shaskell_Uhashable_Shashable-cbits_libhashable-cbits
-lexternal_Shaskell_Uprimitive_Sprimitive-cbits_libprimitive-cbits
-lHSexternalZShaskellZUbifunctorsZSPaths-bifunctors-1.0.0-ghc8.2.2
-lHSexternalZShaskellZUbaseZUorphansZSbase-orphans-0.7-ghc8.2.2
-lHSexternalZShaskellZUsemigroupsZSsemigroups-0.18.4-ghc8.2.2
-lHSexternalZShaskellZUtransformersZUcompatZSPaths-transformers-compat-1.0.0-ghc8.2.2
-lHSexternalZShaskellZUtransformersZUcompatZStransformers-compat-0.5.1.4-ghc8.2.2
-lHSexternalZShaskellZUtaggedZStagged-0.8.5-ghc8.2.2
-lHSexternalZShaskellZUstmZSstm-2.4.5.0-ghc8.2.2
-lHSexternalZShaskellZUStateVarZSStateVar-1.1.1.0-ghc8.2.2
-lHSexternalZShaskellZUcontravariantZScontravariant-1.4.1-ghc8.2.2
-lHSexternalZShaskellZUdistributiveZSdistributive-0.5.3-ghc8.2.2
-lHSexternalZShaskellZUcomonadZScomonad-5.0.3-ghc8.2.2
-lHSexternalZShaskellZUthZUabstractionZSth-abstraction-0.2.8.0-ghc8.2.2
-lHSexternalZShaskellZUbifunctorsZSbifunctors-5.5.2-ghc8.2.2
-lHSexternalZShaskellZUtextZStext-1.2.3.0-ghc8.2.2
-lHSexternalZShaskellZUhashableZShashable-1.2.7.0-ghc8.2.2
-lHSexternalZShaskellZUmtlZSmtl-2.2.2-ghc8.2.2
-lHSexternalZShaskellZUconstraintsZSconstraints-0.10-ghc8.2.2
-lHSexternalZShaskellZUdlistZSdlist-0.8.0.4-ghc8.2.2
-lHSexternalZShaskellZUlensZSPaths-lens-1.0.0-ghc8.2.2
-lHSexternalZShaskellZUcallZUstackZSPaths-call-stack-1.0.0-ghc8.2.2
-lHSexternalZShaskellZUcallZUstackZScall-stack-0.1.0-ghc8.2.2
-lHSexternalZShaskellZUprofunctorsZSprofunctors-5.2.2-ghc8.2.2
-lHSexternalZShaskellZUunorderedZUcontainersZSunordered-containers-0.2.9.0-ghc8.2.2
-lHSexternalZShaskellZUsemigroupoidsZSsemigroupoids-5.2.2-ghc8.2.2
-lHSexternalZShaskellZUtransformersZUbaseZStransformers-base-0.4.4-ghc8.2.2
-lHSexternalZShaskellZUexceptionsZSexceptions-0.8.3-ghc8.2.2
-lHSexternalZShaskellZUfreeZSfree-5.0.2-ghc8.2.2
-lHSexternalZShaskellZUvoidZSvoid-0.7.2-ghc8.2.2
-lHSexternalZShaskellZUadjunctionsZSadjunctions-4.4-ghc8.2.2
-lHSexternalZShaskellZUkanZUextensionsZSkan-extensions-5.1-ghc8.2.2
-lHSexternalZShaskellZUparallelZSparallel-3.2.1.1-ghc8.2.2
-lHSexternalZShaskellZUreflectionZSreflection-2.1.3-ghc8.2.2
-lHSexternalZShaskellZUprimitiveZSprimitive-0.6.4.0-ghc8.2.2
-lHSexternalZShaskellZUvectorZSvector-0.12.0.1-ghc8.2.2
-lHSexternalZShaskellZUlensZSlens-4.16.1-ghc8.2.2
-lHSexternalZShaskellZUmmorphZSmmorph-1.1.2-ghc8.2.2
-lHSexternalZShaskellZUmonoZUtraversableZSPaths-mono-traversable-1.0.0-ghc8.2.2
-lHSexternalZShaskellZUsplitZSsplit-0.2.3.3-ghc8.2.2
-lHSexternalZShaskellZUvectorZUalgorithmsZSvector-algorithms-0.7.0.1-ghc8.2.2
-lHSexternalZShaskellZUmonoZUtraversableZSmono-traversable-1.0.8.1-ghc8.2.2
-lHSexternalZShaskellZUasyncZSasync-2.1.1.1-ghc8.2.2
-lHSexternalZShaskellZUprotoludeZSprotolude-0.2.2-ghc8.2.2
-lHSengineZStest-lib-1.0.0-ghc8.2.2
-ltest-bin
-lstdc++
-lm
So it looks like -ltest-bin
is being linked against, but perhaps not in the right way? Not sure if there's any useful information I've omitted that might also help debug this; thanks!
The following example fails.
cc_binary(
name = "hello-cc",
srcs = ["main.c"],
linkopts = ["-shared"],
)
binary_closure(
name = "clotestbin",
src = "hello-cc",
)
A rule we have to get the runfiles of a binary complains that it produces an empty output, and the rules that use it also don't like having a label producing no files.
ERROR: /home/facundo/tweag/clodl/BUILD:73:1: in _expose_runfiles rule //:clotestbin-closure-runfiles:
Traceback (most recent call last):
File "/home/facundo/tweag/clodl/BUILD", line 73
_expose_runfiles(name = 'clotestbin-closure-runfiles')
File "/tmp/bazel/_bazel_facundo/31d856e5632fce7175cf6a803d2d4fa7/external/io_tweag_clodl/clodl/clodl.bzl", line 179, in _expose_runfiles_impl
ctx.actions.run_shell(outputs = output_libs_files, input..., <2 more arguments>)
action outputs may not be empty
And after some more hacking to workaround that
ERROR: /home/facundo/tweag/clodl/BUILD:73:1: in srcs attribute of cc_binary rule //:clotestbin-closure_wrapper: '//:clotestbin-closure-runfiles' does not produce any cc_binary srcs files (expected .cc, .cpp, .cxx, .c++, .C, .c, .h, .hh, .hpp, .ipp, .hxx, .inc, .S, .s, .asm, .a, .lib, .pic.a, .lo, .pic.lo, .so, .dylib, .dll, .o or .pic.o). Since this rule was created by the macro 'binary_closure', the error might have been caused by the macro implementation in /tmp/bazel/_bazel_facundo/31d856e5632fce7175cf6a803d2d4fa7/external/io_tweag_clodl/clodl/clodl.bzl:351:16
This likely requires using cc_haskell_import or writing some adapter rule for the purpose. As it stands, haskell_library
doesn't yield the shared library in the outputs.
cc_haskell_import was not forwarding some runfiles last time I tried it, so it may need some fixing. This issue is about coming with a reasonable story for making closures of these libraries.
The readme should comment how to modify the WORKSPACE file and how to import the rules in BUILD files.
A test should be added that demonstrates how to import. I suspect that at the moment, importing is not as straight forward as it could be because the top level BUILD file is importing Haskell rules for examples and buildtools for formatting files.
There is some earlier discussion about it in #46.
cc_library doesn't link the dependencies into the library, which makes relying on ldd insuficient to find the dependencies. We will have to see how to find them. Failing that we could ask the user to link them into a cc_binary before using library_closure.
Some related discussion here:
bazelbuild/bazel#492
Fails with bazel 0.22:
root@f19bb6a322bd:/clodl# bazel build //:clotestbin-cc
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Invocation ID: 0544d3f9-ac19-42c3-b50f-6c8ef4329a5b
ERROR: error loading package '': Encountered error while reading extension file 'haskell/repositories.bzl': no such package '@io_tweag_rules_haskell//haskell': The native http_archive rule is deprecated. load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") for a drop-in replacement.
Use --incompatible_remove_native_http_archive=false to temporarily continue using the native rule.
ERROR: error loading package '': Encountered error while reading extension file 'haskell/repositories.bzl': no such package '@io_tweag_rules_haskell//haskell': The native http_archive rule is deprecated. load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") for a drop-in replacement.
Use --incompatible_remove_native_http_archive=false to temporarily continue using the native rule.
Using the proposed workaround still fail:
root@f19bb6a322bd:/clodl# bazel build //:clotestbin-cc --incompatible_remove_native_http_archive=false
INFO: Invocation ID: 82f0e5ec-5ee5-40c7-8a41-59ca8fd03759
INFO: SHA256 (https://github.com/tweag/rules_haskell/archive/5c94b23107809026d7e6de25a891bd3874dbc522.tar.gz) = 89e312cc854251a988743f9c77c2a8a6a97a00ab73edb794bbe1f9d9217e5e50
INFO: SHA256 (https://github.com/tweag/rules_nixpkgs/archive/v0.2.1.tar.gz) = 96454359a456239f8829b8a7c71ffcb6bd27f28dd7d0c206b08f98c1281a2f0d
INFO: SHA256 (https://github.com/bazelbuild/rules_sass/archive/0.0.3.tar.gz) = 14536292b14b5d36d1d72ae68ee7384a51e304fa35a3c4e4db0f4590394f36ad
INFO: SHA256 (https://github.com/bazelbuild/skydoc/archive/f531844d137c7accc44d841c08a2a2a366688571.tar.gz) = a6f91d35beeff4b3d082349952b62a598ace855c268970a580244513bf4635ee
INFO: SHA256 (https://github.com/bazelbuild/bazel-skylib/archive/0.2.0.tar.gz) = 981624731d7371061ca56d532fa00d33bdda3e9d62e085698880346a01d0a6ef
ERROR: /clodl/BUILD:16:1: no such target '@io_tweag_rules_haskell//haskell:toolchain': target 'toolchain' not declared in package 'haskell' (did you mean 'toolchain.bzl'?) defined by /root/.cache/bazel/_bazel_root/28e33e437e789c23dd810e321f1fc417/external/io_tweag_rules_haskell/haskell/BUILD and referenced by '//:ghc'
ERROR: While resolving toolchains for target //:clotestbin-cc-closure_wrapper: invalid registered toolchain '//:ghc': Analysis failed
ERROR: Analysis of target '//:clotestbin-cc' failed; build aborted: Analysis failed
INFO: Elapsed time: 10.939s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (12 packages loaded, 47 targets configured)
When removing some library from the closure like in
library_closure(
name = "closure",
srcs = ...,
excludes = [
"libjvm\.so.*",
],
)
all the dependencies of libjvm.so
are retained. If the binary does not require those dependencies transitively through other libraries, they shouldn't be included.
It would be nice if we could use jarify to call an arbitrary function from haskell and get its return value (other than an int).
This would allow to compose haskell and java in a cleaner way.
From current master:
Build works
# in git checkout of clodl
$ nix-shell
[nix-shell:~/tweag/clodl]$ bazel build //:clotestbin-cc-pie
INFO: Analysed target //:clotestbin-cc-pie (0 packages loaded).
INFO: Found 1 target...
Target //:clotestbin-cc-pie up-to-date:
bazel-genfiles/clotestbin-cc-pie.sh
INFO: Elapsed time: 0.416s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
Execution in the shell works:
[nix-shell:~/tweag/clodl]$ bazel-genfiles/clotestbin-cc-pie.sh
hello cc
However, if I'm leaving the shell, the execution does not work anymore:
[nix-shell:~/tweag/clodl]$ exit
$ bazel-genfiles/clotestbin-cc-pie.sh
bazel-genfiles/clotestbin-cc-pie.sh: line 6: /tmp/tmp.cO8LRRZISw/clotestbin-cc-pie-closure_wrapper: No such file or directory
It may come from the behavior of mktemp -d
which is different between inside and outside of the nix shell:
# In the shell
[nix-shell:~/tweag/clodl]$ mktemp -d
/run/user/1000/tmp.AQlPgn2GIW
# Outside of the nix shell
$ mktemp -d
/tmp/tmp.EcwXFl3PpA
I'm using nixos (from nixos-unstable) as my os.
On NixOS, inside a nix-shell --pure
shell, I get:
$ bazel build //...
[...]
INFO: From Executing genrule //:clotest:
cp: cannot stat '/home/mboes/.cache/bazel/_bazel_mboes/516d4f3dd4e279fb51d72cf964cb28de/sandbox/2049516182333261116/execroot/io_tweag_clodl/bazel-out/k8-fastbuild/bin/../../../bazel-out/k8-fastbuild/bin/libbootstrap_libbootstrap.so': No such file or directory
INFO: From Executing genrule //:clotestbin-closure:
cp: cannot stat '/home/mboes/.cache/bazel/_bazel_mboes/516d4f3dd4e279fb51d72cf964cb28de/sandbox/3745048280373793804/execroot/io_tweag_clodl/bazel-out/k8-fastbuild/bin/../../../bazel-out/k8-fastbuild/bin/libbootstrap_libbootstrap.so': No such file or directory
[...]
$ echo $?
0
These errors look suspicious.
The JAR's produced by jarify
(and by extension sparkle
) are always large. Even for a small "hello world" app, on NixOS one can count on an output size of about 32 MB. Upon closer inspection, it turns out that the bulk of the space comes from stuff that could be excluded.
Here's what I did:
$ stack exec jarify <some-haskell-exe-that-uses-jni>
$ unzip <some-haskell-exe-that-uses-jni>.jar
$ unzip jarify-app.zip
$ du -sh * | sort -h
[...]
5,2M libgtk-x11-2.0.so.0
11M libHSbase-4.9.1.0-ghc8.0.2.so
15M libjvm.so
31M jarify-app.zip
So about a third of the space is eaten up by base
. Not much we can do about that. libjvm
eats up another big chunk. But why are we including libjvm.so
in the packaging? Surely that's already provided by the JVM instance that loads the JAR.
Moreover, it turns out that libjvm.so
is but the tip of the iceberg. Notice the libgtk
above. lddtree
from the pax-utils
package tells us where it comes from: it's a transitive dependency of libjvm.so
. As is in fact libX11.so
and a whole host of other graphical libs.
So if we cut out libjvm.so
and all dependencies, we could significantly reduce the number of libs in the JAR, presumably reduce the packaging time, and get a 3x reduction in size in this particular example.
cc @robinbb
Hi,
I'm new to clodl and using it to package a binary in a Bazel project for cross-platform compatibility. However, I'm facing some problems and hope to get some help.
I'm not sure where to ask for help, so I'm posting it here. If this isn't the right place, please let me know.
In my WORKSPACE file, I added the following:
http_archive(
name = "com_github_bazelbuild_buildtools",
sha256 = "ae34c344514e08c23e90da0e2d6cb700fcd28e80c02e23e4d5715dddcb42f7b3",
strip_prefix = "buildtools-4.2.2",
urls = [
"https://github.com/bazelbuild/buildtools/archive/refs/tags/4.2.2.tar.gz",
],
)
http_archive(
name = "rules_sh",
sha256 = "d668bb32f112ead69c58bde2cae62f6b8acefe759a8c95a2d80ff6a85af5ac5e",
strip_prefix = "rules_sh-0.3.0",
urls = ["https://github.com/tweag/rules_sh/archive/v0.3.0.tar.gz"],
)
http_archive(
name = "rules_haskell",
sha256 = "298f6f0db23391274b4eca215daa01797d05699469048ef94540d5829b466377",
strip_prefix = "rules_haskell-0.17",
url = "https://github.com/tweag/rules_haskell/archive/refs/tags/v0.17.tar.gz",
)
http_archive(
name = "io_tweag_clodl",
urls = ["https://github.com/tweag/clodl/archive/b75ce067f9c95546b553faac76c07e99346251e5.tar.gz"],
strip_prefix = "clodl-b75ce067f9c95546b553faac76c07e99346251e5",
)
(I'm not sure why there are so many dependencies. If there is anyway to avoid it, please let me know).
And in the BUILD file:
load(
"@io_tweag_clodl//clodl:clodl.bzl",
"binary_closure",
)
binary_closure(
name = "my_binary_closure",
src = "my_binary",
)
When I run:
bazel build //path:my_binary_closure --verbose_failures
I get this error:
ERROR: /home/user/workspace/path/BUILD.bazel:195:15: output 'path/patchelf' was not created
ERROR: /home/user/workspace/path/BUILD.bazel:195:15: output 'path/scanelf' was not created
ERROR: /home/user/workspace/path/BUILD.bazel:195:15: Action path/bash failed: not all outputs were created or valid
Target //path:my_binary_closure failed to build
ERROR: /home/user/workspace/path/BUILD.bazel:195:15 Action path/my_binary_closure.zip failed: not all outputs were created or valid
Any help is appreciated. Thanks!
Currently we require that the user do two things manually:
libHSjarify*.so
,$ORIGIN
to the RPATH
.Linking as PIE will always be the responsibility of the user. But the rest we can do ourselves, using patchelf
. An alternative is elf-edit, but the editing facilities for that lib are not powerful enough yet (can't figure out how to edit dynamic section).
We now have the patchelf
codebase imported as vendor code. We even build it from Setup.hs
. But we don't use it. Let's use it to:
$ORIGIN
into the RPATH
using --set-rpath
.The tests cover a haskell_binary with a haskell_toolchain_library dependency but not a haskell_binary with a haskell_library or even haskell_cabal_library dependency. It might be good to add such test cases as well, as there are subtle differences between these types of libraries.
ERROR: /home/centos/clodl/BUILD:54:16: Linking of rule '//:libclotest_wrapper.so' failed (Exit 1) gcc failed: error executing command /nix/store/8pbwywcj6vbswz7xmy2dh716x8blgh8w-gcc-wrapper-9.3.0/
bin/gcc @bazel-out/k8-fastbuild/bin/libclotest_wrapper.so-2.params
Use --sandbox_debug to see verbose messages from the sandbox
/nix/store/p792j5f44l3f0xi7ai5jllwnxqwnka88-binutils-2.31.1/bin/ld.gold: error: bazel-out/k8-fastbuild/bin/_solib_k8/_U_S_S_Clibclotest_Uwrapper.so___Uclotest_UUsolib/libhello-hs.so: unsupported E
LF file type 2
collect2: error: ld returned 1 exit status
Target //:clotest failed to build
Use --verbose_failures to see the command lines of failed build steps.
Seems related to this:
tweag/rules_haskell#1418
GHC is passing -no-pie
to the linker when building the haskell binary, and the subsequent attempt to link the wrapper fails probably because of that.
library_closure
fails with
ERROR: file 'clotestbin-closure_Usolib/libhello-cc.so' is generated by these conflicting actions:
Label: //:clotestbin-closure-as-libs, //:clotestbin-closure-runfiles
RuleClass: _rename_as_solib rule, _expose_runfiles rule
Configuration: b93351232485d7aae598752ea518de7e
Mnemonic: Symlink, SkylarkAction
Action key: a405cd256b658226272436ee29792619, 491ba63def9a64729d4385c6f63c0274
Progress message: Symlink clotestbin-closure_Usolib/libhello-cc.so, SkylarkAction clotestbin-closure_Usolib/libhello-cc.so
PrimaryInput: File:[[<execution_root>]bazel-out/k8-fastbuild/bin]libhello-cc.so
PrimaryOutput: File:[[<execution_root>]bazel-out/k8-fastbuild/bin]clotestbin-closure_Usolib/libhello-cc.so
Owner information: //:clotestbin-closure-as-libs com.google.devtools.build.lib.skyframe.BuildConfigurationValue$Key@99896fcf false (996475399), //:clotestbin-closure-runfiles com.google.devtools.build.lib.skyframe.BuildConfigurationValue$Key@99896fcf false (1588167226)
MandatoryInputs: are equal
Outputs: are equal
Our CI passes because we only make closures of executables or executables made libraries.
Allowing users to provide their own build specification and java shim would be usefull for many big data application.
Indeed, in many cases users submit a .jar
to be executed and this .jar
contains a class implementing a specified method and having its own build dependencies.
In the case of AWS lambda, the user is supposed to submit a .jar
, with a class implementing com.amazonaws.services.lambda.runtime.RequestHandler
and overriding the HandleRequest
method.
binary_closure should produce the same zip file as library_closure, prepended with a script that unzips to a temporary folder and executes the application.
The closures built by clodl are likely useful to anyone who needs to deliver portable binaries.
There is no technical reason that requires running our scripts from bazel rules, so they could be extracted.
This issue is about providing scripts that the user can invoke to create closures of her binaries or libraries.
Modifying the JarifyMain
class to call invokeMain
more than once causes a segfault.
Here is my modified version of JarifyMain:
package io.tweag.jarify;
public class JarifyMain extends JarifyBase {
private static native void invokeMain(String[] args);
public static void main(String[] args) {
invokeMain(args);
invokeMain(args);
}
}
I then call jarify on a dummy haskell program that simply prints hello, and get the following output.
➜ jarify git:(master) ✗ java -jar hello.jar
"hello"
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f75f9641469, pid=26688, tid=0x00007f75f9d9e700
#
# JRE version: OpenJDK Runtime Environment (8.0_121-b13) (build 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13)
# Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libc.so.6+0x7e469]# [ timer expired, abort... ]
[2] 26688 abort (core dumped) java -jar hello.jar
The desired behaviour is to be able to call invokeMain multiple times.
This is currently an issue for me when I use a jar generated by jarify in AWS lambda, which will call my haskell code multiple times in a row (and thus fail on the second call).
I had build clotestbin-cc-pie.sh
on nixos using:
nix-shell
bazel build //:clotestbin-cc-pie
After coping clotestbin-cc-pie.sh
to another computer, I realized that the closure depends on the static linker of the building environment.
For example, after patching clotest-cc-pie.sh
so that it displays the list of executed command and does not remove the temporary directory and inside a docker container running an ubuntu
root@4a9668643bbb:/chien# ./clotestbin-cc-pie.sh
+++ mktemp -d
++ tmpdir=/tmp/tmp.3emtCtrOEB
++ unzip -q ./clotestbin-cc-pie.sh -d /tmp/tmp.3emtCtrOEB
++ true
++ /tmp/tmp.3emtCtrOEB/clotestbin-cc-pie-closure_wrapper
./clotestbin-cc-pie.sh: line 6: /tmp/tmp.3emtCtrOEB/clotestbin-cc-pie-closure_wrapper: No such file or directory
root@4a9668643bbb:/chien# file /tmp/tmp.3emtCtrOEB/clotestbin-cc-pie-closure_wrapper
/tmp/tmp.3emtCtrOEB/clotestbin-cc-pie-closure_wrapper: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /nix/store/l0adcz8y05vajfi483a7qv31i03rspqq-glibc-2.27/lib/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, not stripped
I'm on the ubuntu system and it tries to load the linker from /nix/store
path.
Overriding the static linker (using patchelf
) makes it work:
root@4a9668643bbb:/chien# patchelf --set-interpreter /lib64/ld-linux-x86-64.so.2 /tmp/tmp.3emtCtrOEB/clotestbin-cc-pie-closure_wrapper
root@4a9668643bbb:/chien# /tmp/tmp.3emtCtrOEB/clotestbin-cc-pie-closure_wrapper
hello cc
Please note that's not a nix(os) specific issue, because using the current system dynamic loader may not work. For example, a closure built on a recent (2019) nixos works on a ubuntu 18.04 with overrided dynamic loader. However, on an ubuntu 14.04 it does not work and fails with the following error:
/tmp/tmp.IpokZOrdzx/clotestbin-cc-pie-closure_wrapper: relocation error: /tmp/tmp.IpokZOrdzx/libc.so.6: symbol _dl_exception_create, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference
One solution to this problem may be to bundle the dynamic loader too and use it to run the closure in the running shell as such:
/tmp/tmp.IpokZOrdzx/ld-linux-x86-64.so.2 /tmp/tmp.IpokZOrdzx/clotestbin-cc-pie-closure_wrapper
Se the discussion here
#36 (comment)
Related to this, we had a request to use tar as an archiving format in #31.
If tar files were easier to make reproducible, it could be a better solution than insisting on zip files.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.