Not bundle common libraries (e.g. for LLVM and C++) by default about packagecompiler.jl HOT 2 OPEN

The problem with not bundling libstdc++ by default has different flavors, and I am surely not an expert on all of them. What I can say, though, is this:

Before #853, PackageCompiler.jl always included the libstdc++.so that is shipped with Julia in the app/library bundle. This, however, presented problems when compiling a library with PC and three conditions were met at the same time:

a) the library links against system-local 3rd party library (i.e., not from a JLL artifact)
b) the system-local 3d party library is linked against the system-local libstdc++
c) the system-local libstdc++ is newer than the Julia-provided one

In this case, it could happen that running the PC-compiled library became impossible, since it automatically loads the bundled, older libstdc++.so provided by Julia, while the system-local 3d party library requires the use of some features only available in the newer system-local libstdc++.so.

Thus, I think just excluding the C++ library from the bundle by default is not really helpful in many cases, and I'd rather have users deal with a slightly larger bundle than getting weird, hard to understand errors.

It could be considered to be added as (yet another) kwarg option to exclude certain libraries from the bundle, at the cost of increasing the complexity of PC. The question to me would be, how much impact is actually achieved by excluding just libstdc++? In my case, for example, the PREFIX/lib/julia folder is ~220 MIB, thus removing libstdc++ even if it were 30 MB large (which it isn't), would be but a drop in the ocean...

from packagecompiler.jl.

This could shave 234 MB off the download file of your app (code not common with others), assuming more stripped. More conservative 155+ MB of what can be skipped and I find likely to already be installed on computers. One way all the stripped files could be found, even if not already installed as part of your distro, is if you previously downloaded another Julia app (you would then additionally just need to check cryptographic checksums to see if you could for sure use or copy them).

I might need to change the title, it only refers to phase 1, an easy part (that could be opt-in; and useful for proof-of-concept), but phase 2 makes this an even more promising approach, which could be default, and if download is implemented for any one then trivial to extend to other libs. You have Downloads.jl available precompiled, so I assume it doesn't need LLVM nor the C++ standard library. We would need to figure out a place to download from (Julia's infrastructure? or) from Ubuntu (mirrors) or Debian or

https://build.opensuse.org/

which I believe works across distros. Stuff would added to your (or julia's) local folder, i.e. sudo not needed unless we want to support a a global option to save space. That could be implemented later, and opt-in or just skipped.

libLLVM is largest (and likely will get larger, what would happen if you use (slightly) older version, if such is installed, would it still work, make less efficient native code, besides most code already precompiled, so for a tiny part of your app) and libjulia-codegen.so.1.10.0 is next largest (common to all apps? or not used for), except for the sysimage, sys.so, which is of course largest and no longer the default one, so will not be skipped.

libgfortran.so.5.0.0 is maybe not to be relied on installed (nor is used by most), but it and all down to at least libgomp.so.1.0.0 would potentially be installed, and could be downloaded if not, since they are the same anyway, on first use. So this adds up.

Potentially the same would also apply for OpenBLAS which is larger (and often not used, when used should also work and would be with some of the following?):

I have on my Linux Mint though not quite latest:

/snap/julia/83/lib/julia/libopenblas64_.0.3.21.so
/snap/julia/83/lib/julia/libopenblas64_.so
/snap/julia/83/lib/julia/libopenblas64_.so.0
/snap/julia/86/lib/julia/libopenblas64_.0.3.21.so

[not sure what those mean:]
/usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblas.so.0
/usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so

-rwxr-xr-x 1 pharaldsson pharaldsson 241767920 ágú 17 06:40 sys.so
-rwxr-xr-x 1 pharaldsson pharaldsson 94694152 ágú 17 06:42 libLLVM-15jl.so
-rwxr-xr-x 1 pharaldsson pharaldsson 67135792 ágú 17 06:42 libjulia-codegen.so.1.10.0
-rwxr-xr-x 1 pharaldsson pharaldsson 33314616 ágú 17 06:33 libopenblas64_.0.3.23.so
-rwxr-xr-x 1 pharaldsson pharaldsson 18763496 ágú 17 06:33 libstdc++.so.6.0.30
-rwxr-xr-x 1 pharaldsson pharaldsson 12606152 ágú 17 06:42 libjulia-internal.so.1.10.0
-rwxr-xr-x 1 pharaldsson pharaldsson 9375448 ágú 17 06:33 libgfortran.so.5.0.0
-rwxr-xr-x 1 pharaldsson pharaldsson 2796495 ágú 17 06:33 libblastrampoline.so.5
-rwxr-xr-x 1 pharaldsson pharaldsson 2518360 ágú 17 06:33 libmpfr.so.6.2.0
-rwxr-xr-x 1 pharaldsson pharaldsson 1709022 ágú 17 06:33 libgit2.so.1.6.4
-rwxr-xr-x 1 pharaldsson pharaldsson 1461288 ágú 17 06:33 libcholmod.so.4.0.4
-rwxr-xr-x 1 pharaldsson pharaldsson 1341800 ágú 17 06:33 libgomp.so.1.0.0

I think just excluding the C++ library from the bundle by default is not really helpful in many cases, and I'd rather have users deal with a slightly larger bundle than getting weird, hard to understand errors.

I'm not seeing a problem, or possible errors assuming using the download way (or if you worry about no internet, then very non-modern concern, you could always opt into full file, or if you insist have it the default).

I think even without internet (or download option) the error doesn't have to be so cryptic, something like fix your internet (or so sudo apt-get install ...).

I also think the C++ standard lib part only applies to Linux(?) so that part a limited worry to often more advanced users, that could fix, or are used to...

from packagecompiler.jl.