juliapackaging / jllwrappers.jl Goto Github PK

Would it be possible to use SetDllDirectoryW or AddDllDirectory on Windows so that Windows searches the LIBPATH_list for DLLs dynamically loaded by DLLs loaded by Julia?

https://learn.microsoft.com/en-us/windows/win32/api/libloaderapi/nf-libloaderapi-adddlldirectory
https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-setdlldirectoryw

There's been a few commits to master that are somewhat useful, for example, #52. Maybe it is time for a new release?

% julia
julia> using Libglvnd_jll

julia> libGL
"libGL.so.1"

julia> 
% julia-master
julia> using Libglvnd_jll

julia> libGL
"/home/mose/.julia/artifacts/a84cc58d5161b950f268bb562e105bbbf4d6004a/lib/libGL.so"

julia> using BinaryBuilder

julia> BinaryBuilder.Auditor.get_soname(libGL)
"libGL.so.1"

Shouldn't the libGL variable point to libGL.so.1 on Julia v1.6? For example I can't do using GR in v1.6 on a machine without a system-provided libGL.so.1:

julia> using GR
[ Info: Precompiling GR [28b8d3ca-fb5f-59d9-8090-bfdbd6d07a71]
ERROR: LoadError: LoadError: InitError: could not load library "/home/mose/.julia/artifacts/88ce5f00656a15580dd57ce06b013edab1fe74cc/lib/libQt53DAnimation.so"
libGL.so.1: cannot open shared object file: No such file or directory
Stacktrace:
  [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
    @ Base.Libc.Libdl ./libdl.jl:114
  [2] dlopen(s::String, flags::UInt32)
    @ Base.Libc.Libdl ./libdl.jl:114
  [3] macro expansion
    @ ~/.julia/packages/JLLWrappers/KuIwt/src/products/library_generators.jl:61 [inlined]
  [4] __init__()
    @ Qt_jll ~/.julia/packages/Qt_jll/L9NfG/src/wrappers/x86_64-linux-gnu-cxx11.jl:62
  [5] _include_from_serialized(path::String, depmods::Vector{Any})
    @ Base ./loading.jl:669
  [6] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String)
    @ Base ./loading.jl:755
  [7] _tryrequire_from_serialized(modkey::Base.PkgId, build_id::UInt64, modpath::String)
    @ Base ./loading.jl:684
  [8] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String)
    @ Base ./loading.jl:744
  [9] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:993
 [10] require(uuidkey::Base.PkgId)
    @ Base ./loading.jl:909
 [11] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:896
 [12] include(mod::Module, _path::String)
    @ Base ./Base.jl:386
 [13] include(x::String)
    @ GR ~/.julia/packages/GR/RlE5Y/src/GR.jl:3
 [14] top-level scope
    @ ~/.julia/packages/GR/RlE5Y/src/GR.jl:15
 [15] include
    @ ./Base.jl:386 [inlined]
 [16] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base ./loading.jl:1208
 [17] top-level scope
    @ none:1
 [18] eval
    @ ./boot.jl:360 [inlined]
 [19] eval(x::Expr)
    @ Base.MainInclude ./client.jl:446
 [20] top-level scope
    @ none:1
during initialization of module Qt_jll
in expression starting at /home/mose/.julia/packages/GR/RlE5Y/deps/deps.jl:1
in expression starting at /home/mose/.julia/packages/GR/RlE5Y/src/GR.jl:3
ERROR: Failed to precompile GR [28b8d3ca-fb5f-59d9-8090-bfdbd6d07a71] to /home/mose/.julia/compiled/v1.6/GR/jl_vxojcr.
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:33
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::Base.TTY, internal_stdout::Base.TTY)
   @ Base ./loading.jl:1355
 [3] compilecache(pkg::Base.PkgId, path::String)
   @ Base ./loading.jl:1301
 [4] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1016
 [5] require(uuidkey::Base.PkgId)
   @ Base ./loading.jl:909
 [6] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:896

julia> using Libdl

julia> filter(l -> occursin("libGL", l), dllist())
5-element Vector{String}:
 "/home/mose/.julia/artifacts/a84cc58d5161b950f268bb562e105bbbf4d6004a/lib/libGLdispatch.so.0"
 "/home/mose/.julia/artifacts/a84cc58d5161b950f268bb562e105bbbf4d6004a/lib/libGL.so"
 "/home/mose/.julia/artifacts/a84cc58d5161b950f268bb562e105bbbf4d6004a/lib/libGLX.so.0"
 "/home/mose/.julia/artifacts/a84cc58d5161b950f268bb562e105bbbf4d6004a/lib/libGLESv1_CM.so"
 "/home/mose/.julia/artifacts/a84cc58d5161b950f268bb562e105bbbf4d6004a/lib/libGLESv2.so"

CC: @staticfloat

The part here:

uses a Platform as a key in a dictionary. But a Platform is an object that AFAIU will use objectid as part of its hash computation, so for example:

julia> valid_wrappers = Dict{Platform,String}()
Dict{Platform, String}()

julia> p = Platform("x86_64", "windows"; cuda = "10.1");

julia> q = Platform("x86_64", "windows"; cuda = "10.1");

julia> valid_wrappers[p] = "foo"
"foo"

julia> valid_wrappers[p2] = "bar"
"bar"

julia> valid_wrappers
Dict{Platform, String} with 2 entries:
  Platform(Dict("arch"=>"x86_64", "os"=>"windows", "cuda"=>"10.1"), Dict{String, Function}()) => "foo"
  Platform(Dict("arch"=>"x86_64", "os"=>"windows", "cuda"=>"10.1"), Dict{String, Function}()) => "bar"

It's also a bit unclear to me what happens when you serialize such a dict. Is it still valid when you read it back? Would an IdDict be more suitable here to emphasize that the key is based on the constructed object itself and not the content?

I think there is a problem in the definition of excat:

julia> using JLLWrappers

julia> JLLWrappers.excat(:(global artifact_dir = find_artifact_dir()))
:($(Expr(:escape, quote
    artifact_dir = find_artifact_dir()
end)))

julia> @macroexpand JLLWrappers.@generate_init_header
quote
    artifact_dir = find_artifact_dir()
end

the global disappears and the wrappers don't have the artifact_dir variable.

CC: @staticfloat

The init_block argument to build_tarballs currently generates code that's executed when a specific wrapper is found. That's generally fine, but in some cases we might also want to perform an action when no wrapper is available (say, perform system discovery, or log a message). I'd therefore like to propose to add a global_init_block argument to build_tarballs, and adapt JLLWrappers to call the wrappers' init functions from a global __init__ instead.

Thoughts?

Using a JLL generated with master BB and latest Julia nightly:

julia> using Libmount_jll
[ Info: Precompiling Libmount_jll [4b2f31a3-9ecc-558c-b454-b3730dcb73e9]
ERROR: LoadError: LoadError: Cannot locate artifact 'Libmount' in '/Users/kristoffercarlsson/.julia/dev/Libmount_jll/Artifacts.toml'
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] artifact_slash_lookup(name::String, artifact_dict::Dict{String, Any}, artifacts_toml::String, platform::Base.BinaryPlatforms.Platform)
    @ Artifacts ~/julia/usr/share/julia/stdlib/v1.6/Artifacts/src/Artifacts.jl:555
  [3] @artifact_str(__source__::LineNumberNode, __module__::Module, name::Any, platform::Nothing)
    @ Artifacts ~/julia/usr/share/julia/stdlib/v1.6/Artifacts/src/Artifacts.jl:611
  [4] @artifact_str(__source__::LineNumberNode, __module__::Module, name::Any)
    @ Artifacts ~/julia/usr/share/julia/stdlib/v1.6/Artifacts/src/Artifacts.jl:584
  [5] include
    @ ./Base.jl:379 [inlined]
  [6] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base ./loading.jl:1144
  [7] top-level scope
    @ none:1
  [8] eval
    @ ./boot.jl:345 [inlined]
  [9] eval(x::Expr)
    @ Base.MainInclude ./client.jl:446
 [10] top-level scope
    @ none:1
in expression starting at /Users/kristoffercarlsson/.julia/dev/Libmount_jll/src/Libmount_jll.jl:8
in expression starting at /Users/kristoffercarlsson/.julia/dev/Libmount_jll/src/Libmount_jll.jl:2

This occurred when trying to load GTK3_jll on mac.

We should create a JLLExtras.jl package that allows for meta-inspection/management of JLLs. This package doesn't have to be as hyper-optimized as JLLWrappers so we can use it to dev out a JLL for editing, etc...

Examples of things we want to be able to do:

• collect_artifacts(jll_name::String; recursive::Bool, extra_artifacts::Vector{String}) Extract the artifacts of the given jll_name (as looked up from the current pkg environment) out to a scratch space for building/linking against. This should replace the current dev_jll() implementation, and notably should use Preferences and Scratch. We can dump the whole idea of the overrides directory now that Preferences is working properly. recursive will determine whether we will extract all dependencies into the same directory (useful for people who just want to link against a tree of dependencies with minimum hassle) and extra_artifacts will be used to merge together multiple artifacts, given that Mosè and I ever get around to splitting things up into runtime, debug, logs, etc... artifacts.

Eventually, we'll have the ability to target specific glibc/macOS SDK/etc... versions, and we'll need to be able to surface incompatibilities to users. Since Pkg doesn't allow us to declare compat constraints on OS version, we must instead do something at __init__() time.

Some of the information is already encoded into the HostTriplet, for example:

julia> using Base.BinaryPlatforms; HostPlatform()
Linux x86_64 {cxxstring_abi=cxx11, julia_version=1.7.1, libc=glibc, libgfortran_version=5.0.0, libstdcxx_version=3.4.29}

julia> using Base.BinaryPlatforms; HostPlatform()
macOS x86_64 {cxxstring_abi=cxx11, julia_version=1.7.1, libgfortran_version=5.0.0}

Notice that this information is, as of now, incomplete; the macOS triplet lacks the os_version() information that we'd need to determine SDK compatibility, the Linux triplet lacks the precise glibc version, etc... But it's a start! With complete information in the Platform object, we can fully support artifact selection, but we would still need to surface good errors to the user in the event that no artifact can be selected, as in the case where something requires macOS 12+, but the user is running 10.15.

It would be good to teach BB to pay attention to these standard limiting factors, and see if (for instance) the minimum-supported glibc version for a particular architecture is greater than Julia's minimum-supported version, then emit checks in __init__() that ensure that the HostPlatform() is compatible with those bounds. So e.g. if an x86_64 build of LibFoo_jll requires glibc 2.12.2 or greater, we can elide the check, but if it requires 2.17+, we would generate the check. That is, of course, only if the check takes a significant amount of time; if it's essentially free, then I think we can get away with just always emitting the relevant checks.

Every jll package does the following checks during precompile time

statx(AT_FDCWD, "/home/kc/.julia/packages/GR_jll/k8I6U/src/wrappers/JuliaArtifacts.toml"  ENOENT (No such file or directory)
statx(AT_FDCWD, "/home/kc/.julia/packages/GR_jll/k8I6U/src/wrappers/Artifacts.toml"  ENOENT (No such file or directory)
statx(AT_FDCWD, "/home/kc/.julia/packages/GR_jll/k8I6U/src/wrappers/JuliaProject.toml"  ENOENT (No such file or directory)
statx(AT_FDCWD, "/home/kc/.julia/packages/GR_jll/k8I6U/src/wrappers/Project.toml"  ENOENT (No such file or directory)
statx(AT_FDCWD, "/home/kc/.julia/packages/GR_jll/k8I6U/src/JuliaArtifacts.toml"  ENOENT (No such file or directory)
statx(AT_FDCWD, "/home/kc/.julia/packages/GR_jll/k8I6U/src/Artifacts.toml"  ENOENT (No such file or directory)
statx(AT_FDCWD, "/home/kc/.julia/packages/GR_jll/k8I6U/src/JuliaProject.toml"  ENOENT (No such file or directory)
statx(AT_FDCWD, "/home/kc/.julia/packages/GR_jll/k8I6U/src/Project.toml"  ENOENT (No such file or directory)
statx(AT_FDCWD, "/home/kc/.julia/packages/GR_jll/k8I6U/JuliaArtifacts.toml"  ENOENT (No such file or directory)
statx(AT_FDCWD, "/home/kc/.julia/packages/GR_jll/k8I6U/Artifacts.toml", 

Now, stat is pretty cheap on linux/windows but there have been reports about it being quite slow on Windows (JuliaLang/julia#40570).

Since we know the exact layout of a jll package, it should be no need to have to look through the filesystem for the Artifact file.

@staticfloat believes these lookups come from

Instead of generating a call to @artifact_str, perhaps we can just look up the path from the artifact file which we know where it lives?

Using JuliaLang/julia#43671

Just as an example:

could be type annotated to be a String.

AFAIU, that is the thing that sets PATH, but nothing calls it (neither in JLLWrappers, or in the test JLLs). Is there some missing coverage for this functionality?

Ref: JuliaPackaging/BinaryBuilder.jl#1242

The depwarn created here creates a log message with a malformed location

┌ Warning: ffmpeg() is deprecated, use the non-do-block form
│   caller = ip:0x0
└ @ Core :-1

Also, perhaps the message should say Warning: using executables via a function return like ffmpeg() is deprecated by JLLWrappers, use the non-do-block form

Even with --depwarn=error it appears to be coming from FFMPEG_jll but the src is JLLWrappers

julia> FFMPEG.exe("-v")
ERROR: ffmpeg() is deprecated, use the non-do-block form
Stacktrace:
 [1] depwarn(msg::String, funcsym::String; force::Bool)
   @ Base ./deprecated.jl:124
 [2] depwarn(msg::String, funcsym::String)
   @ Base ./deprecated.jl:121
 [3] ffmpeg(f::Function; adjust_PATH::Bool, adjust_LIBPATH::Bool)
   @ FFMPEG_jll ~/.julia/packages/JLLWrappers/pG9bm/src/products/executable_generators.jl:20
 [4] ffmpeg(f::Function)
   @ FFMPEG_jll ~/.julia/packages/JLLWrappers/pG9bm/src/products/executable_generators.jl:19
 [5] #exe#2
   @ ~/.julia/packages/FFMPEG/OUpap/src/FFMPEG.jl:0 [inlined]
 [6] exe(args::String; command::Function, collect::Bool)
   @ FFMPEG ~/.julia/packages/FFMPEG/OUpap/src/FFMPEG.jl:64
 [7] top-level scope
   @ REPL[9]:1

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Now that we are starting to actively use platform tags we might have the situation that someone runs
on a platform + tag combo that is not supported by the current _jll.

Situations in which that might arise:

• New architectures like riscv
• Microarchitectures
• A new CUDA version
• Library only build against some of the MPI implementations
• A user wanting to load an old jll on a newer platform

In that situation loading the jll will fail and there are no products available to override using Preferences.jl

So now a user has to go through the effort of rebuilding all the artifacts etc. in Yggdrasil just to test their code.

Right now products are platform specific (and generally that's a really good thing), but maybe we need a "generic"/"default" platform that provides a minimum set of products that the user can provide through preferences.