juliapackaging / relocatablefolders.jl Goto Github PK

License: Other

Julia 100.00%

relocatablefolders.jl's Introduction

RelocatableFolders.jl

An alternative to the @__DIR__ macro. Packages that wish to reference paths in their project directory run into issues with relocatability when used in conjunction with PackageCompiler. The @path macro provided by this package overcomes this limitation. See here and here for further details.

Usage

The package provides one export, the @path macro. It can be used to replace @__DIR__ in the following way:

module MyPackage

using RelocatableFolders

# const ASSETS = joinpath(@__DIR__, "../assets")
const ASSETS = @path joinpath(@__DIR__, "../assets")

end

At runtime the path stored in ASSETS will get resolved to either the original path, if it still exists, or to an automatically generated scratchspace containing the same folder and file structure as the original.

Limitations

This macro should only be used for reasonably small file or folder sizes. If there are very large files then it is better to make use of Julia's Artifact system instead.

Building new paths from, for example, ASSETS in the above example will return a String containing the resolved path rather than a Path object. Doing this at the module-level will result in hardcoded paths that will run into relocatability issues as discussed above. Always create a new @path for each resource you wish to reference rather than building them in parts, e.g.

module MyPackage

using RelocatableFolders

const ASSETS = @path joinpath(@__DIR__, "../assets")
const SUBDIR = @path joinpath(ASSETS, "subdir")
const FILE = @path joinpath(ASSETS, "file.txt")

end

Internals

At compile-time the @path macro will read in all the files contained in the referenced path and store them and their paths. The returned object is a Path <: AbstractString. Whenever a Path is passed to a function expecting an AbstractString (such as readdir) it will be converted to a String by looking up the stored path and returning that. When no path exists (the source tree no longer exists) then the contents of the files that were read at compile-time are written to a Scratch scratchspace and that path is returned instead.

Alternatives

The alternative approach is to use the Artifacts system to distribute the required files, which is a reasonably heavyweight solution for a simple collection of source-controlled files. It is recommended that users look to use artifacts when that file sizes are reasonably large and only use this package when the distributed files are small.

relocatablefolders.jl's People

Contributors

Stargazers

Watchers

Forkers

michaelhatherly ven-k fingolfin jackdunnnz piever fonsp ranocha connectedsystems yuliyacl sharanry

relocatablefolders.jl's Issues

Despite `safe_isfile` returning `false` on f.path the `getpath` doesn't return scratch-path

I'm working on relocatability of BinaryBuilderBase.jl
And have used @path for declaring the _artifacts_toml and subsequently have used artifacts_toml = String(_artifacts_toml) locally to get the path from scratchspaces.
Yet it fails on relocated sysimage:

The following is on a relocated sysimage with a different julia-depot-path


julia> f = BinaryBuilderBase._artifacts_toml
"<original-depot-path>/packages/BinaryBuilderBase/F4W5f/Artifacts.toml"

julia> typeof(f)
RelocatableFolders.Path

julia> f.path
"<original-depot-path>/packages/BinaryBuilderBase/F4W5f"

julia> rf = RelocatableFolders
RelocatableFolders

julia> rf.safe_isfile(joinpath(f.path, collect(keys(f.files))[1]))
false

julia> rf.safe_ispath(f.path)
true

julia> rf.getpath(f)
"<original-depot-path>/packages/BinaryBuilderBase/F4W5f/Artifacts.toml"

julia> String(f)
"<original-depot-path>/packages/BinaryBuilderBase/F4W5f/Artifacts.toml"

I expected the getpath to return the scratch-path as rf.safe_isfile(f.path) returns false; but it somehow fails

RelocatableFolders does not handle removed files?

We encountered it originally while trying to compile a server app with swagger spec, and it didn't work for some reason: GenieFramework/SwaggerMarkdown.jl#8

That is strange, because it worked in Oxygen https://github.com/ndortega/Oxygen.jl/pull/128/files

Looks like it works with entire folder but not with a single file:

import Pkg
Pkg.activate(; temp=true)
Pkg.add("RelocatableFolders")

open("test1.txt", "w") do io
    write(io, "some data")
end

mkdir("subfolder")
open("subfolder/test2.txt", "w") do io
    write(io, "another data")
end

module MyTest
    using RelocatableFolders 
    const FILE = @path joinpath(@__DIR__, "test1.txt")
    const FOLDER = @path joinpath(@__DIR__, "subfolder")
    read_from_file() = read(FILE, String)
    read_from_folder() = read(joinpath(FOLDER, "test2.txt"), String)
end

using .MyTest
MyTest.read_from_file() # ok
MyTest.read_from_folder() # ok

rm("subfolder"; recursive = true)
MyTest.read_from_folder() # ok

rm("test1.txt")
MyTest.read_from_file() # error

Define method `codeunit(s::Path)`

Hi
Thanks for you nice package - it's easy to use and helpfull!

I've encountered an error, when used JSON3.jl for file reading.

ERROR: MethodError: no method matching codeunit(::RelocatableFolders.Path)

Closest candidates are:
  codeunit(::AbstractString, ::Integer)
   @ Base strings\basic.jl:107
  codeunit(::SubString)
   @ Base strings\substring.jl:73
  codeunit(::SubString, ::Integer)
   @ Base strings\substring.jl:76
  ...

Stacktrace:
  [1] sizeof(s::RelocatableFolders.Path)
    @ Base .\strings\basic.jl:179
  [2] read_json_str(json::RelocatableFolders.Path)
    @ JSON3 C:\Users\yzh\.julia\packages\JSON3\L8Yfy\src\utils.jl:240
  [3] _prepare_read(json::RelocatableFolders.Path, #unused#::Type{Dict{String, Any}})
    @ JSON3 C:\Users\yzh\.julia\packages\JSON3\L8Yfy\src\structs.jl:17
  [4] read(str::RelocatableFolders.Path, ::Type{Dict{String, Any}}; jsonlines::Bool, kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ JSON3 C:\Users\yzh\.julia\packages\JSON3\L8Yfy\src\structs.jl:34
<....>

To reproduce an error use some JSON-file

using JSON3, RelocatableFolders
const PATH2JSON = @path joinpath(@__DIR__, "some_file.json")
content= JSON3.read(PATH2JSON , Dict{String, Any})

The problem was related with sizeof(path_to_file) call inside the JSON3.read().
I've defined the method for RelocatableFolders.Path and everything works correctly now.

Base.sizeof(s::RelocatableFolders.Path) = sizeof(String(s))

It would be great If you may add this also to the Pkg! Thank you!

Release notes for 1.0

Sorry if this wrong place, but I'm wondering what is breaking in 1.0? I couldn't grep that from the release notes and am examining a compat bump PR at MLJModels.jl. Even better, if someone is able to directly comment on JuliaAI/MLJModels.jl#477 that would be great.

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Is `getroot` part of the API?

Hello! I've been trying to use this to support making apps based on JSServe, see SimonDanisch/Bonito.jl#117.

I would like to just compute the extension of a file stored as a RelocatableFolders.Path, without triggering the string conversion logic. I've found that getroot does exactly that (see here), but it seems to be an internal function. Could it become part of the API?

Broken link in README

The README contains a link to https://julialang.github.io/PackageCompiler.jl/dev/devdocs/relocatable_part_3/#Relocatability-of-Julia-packages-1 which gives a "page not found" error.

Path separator issue on Windows

If there are mixed path separators in the path on Windows, then it can sometimes cause the specified file to not be found. For example, using the example in the test folder

using RelocatableFolders
PATH = @path "path"
FILE = joinpath("path", "subfolder/other.jl") # works
FILE = joinpath(PATH, "subfolder/other.jl") # doesn't find other.jl
FILE = joinpath(PATH, "subfolder\\other.jl") # works
FILE = joinpath(PATH, "subfolder", "other.jl") # works

The cause is that the paths may not compare equal at this line due to differences in the path separators used in each path. An easy fix is to instead compare normpath(path) == normpath(fullpath), but not sure if that's the best solution

Use `Tar` to store trees of content?

For dense trees of content that are going to be bundled into a system image, it might make sense to use Tar.jl to do the bundling and extraction, as there's a good amount of corner cases around things like permissions that can get a little hairy across operating systems. Perhaps a good architecture to make it easier to deal with these things would be:

Use Tar.create() to generate a tarball (that can then be compressed in-memory, for space savings for larger bundles), store that tarball in the precompile cache.
Use Tar.extract() to write it out to disk on-demand just like it works now.