juliapackaging / pkgserver.jl Goto Github PK

We should allow the registry preference header (Julia-Registry-Preference) to not just be a single value, but a comma-separated list. The Pkg server should attempt to serve each registry flavor, in order. The client can use the meta-flavor default to allow the PkgServer to make an informed decision, and if none of the choices can be found, /registries should return 404. If the header is missing, it is assumed to be equal to default, thereby maintaining the current behavior.

Examples:

Julia-Registry-Preference: super-conservative, conservative

Let us imagine that this is a hypothetical Julia v2.3 client, which knows about the existence of the super-conservative flavor, but allows fallback to the conservative flavor if the server doesn't serve the former. In this scenario, if the server does not serve either kind, it should return a 404.

Julia-Registry-Preference: eager, conservative, default

This will select the eager flavor first, if it exists, and if it doesn't it chooses conservative, then finally falling back to whatever flavor the PkgServer thinks is best. This is a convenient way to future-proof clients against naming changes.

In some cases when people want to set a shared version of the Julia depot path, this could trigger a permission error.

I'm wondering if it's possible to make it 0o755?

Originally reported by @yym0924 at https://discourse.juliacn.com/t/topic/3682

Problem Context

PkgServer makes requests to GitHub to validate the registry repo URL on startup and to validate tree SHAs when fetching updates from the storage server, but it does not provide a way to supply a GitHub access token or other credentials to be used when making these requests for a private repo. This prevents use of PkgServer for private registries. Use of PkgServer for private registries is helpful to simplify access for users and continuous integration.

Proposal

Based on a discussion with @staticfloat, I'd like to propose adding PkgServer support for GitHub personal access tokens and use GitHub REST API endpoints for the two repository checks that PkgServer makes:

• add an optional access_token member to RegistryMeta
• change the repository URL check to use HEAD /repos/{owner}/{repository}
• change the tree SHA check to use HEAD /repos/{owner}/{repository}/git/trees/{sha}
• if the token is not nothing, add an HTTP Authorization header to the HEAD requests

I've validated that the above HEAD requests can be made with HTTP package, using tokens for private repos and no token for a public repo such as JuliaRegistries/General.

Implementation sketch

Add new functions, potentially scoped to a PkgServer sub module named GitHub

• api_repository_url(repo_url::String)::String

  Composes the GitHub API URL based on the GitHub repository URL. For example, for the General
  registry, the repo and its API URL are:

  https://github.com/JuliaRegistries/General.git -> https://api.github.com/repos/JuliaRegistries/General

• api_tree_url(repo_base_url::String, sha::String)::String

  Composes the GitHub API URL for a tree SHA. This URL can be used to validate that a tree SHA
  exists in the upstream repository. repo_base_url is the API URL for the repository resource,
  such as composed by api_repository_url(repo_url)

• resource_exists(url::String; token::Union{Nothing,String}=nothing) -> Bool

  Returns true if a HEAD request for the GitHub url returns a 200, false otherwise.
  For non-public resources, specify a GitHub personal access token with the optional token
  parameter (only GitHub REST API requests can be authorized with a token).

Add new members to RegistryMeta:

• upstream_api_url::String - the base URL for the registry repo within the GitHub API path namespace
• access_token::Union{Nothing,String} - the optional access token

Modify:

• RegistryMeta constructor to accept an optional access token and generate/validate the upstream_api_url from the repository URL
• verify_registry_hash(uuid::AbstractString, hash::AbstractString) to use the proposed functions to validate the tree SHA

Considerations

GitHub personal access tokens potentially grant broad privileges to multiple repos

• GitHub UI does not offer a way to generate a token that grants read-only repo access to a specific repository
  • access is implicitly write if the user is authorized for write
  • access spans all repos the user has access to
• to comply with principle of least privilege, an organization/individual setting up a private PkgServer would likely need to create a "machine account" with only read access to the private registry repo

OAuth or GitHub deployment keys might allow for finer-grained access control, but would require more restructuring of PkgServer.

Unit tests that do not perform network requests should be straightforward.

Integration tests that validate functionality with a private registry repo are desirable, but will likely require some
assistance/coordination

• setup a private repo under an appropriate Julia project organization
• write an integration test
• setup and validate the CI job

After a lot of confusion, I discovered that the version of HTTP.jl in the Manifest.toml file (currently 0.8.17) is a version with a known bug in regards to proxies. This bug has been fixed in v0.8.18, as well as the newer version v0.8.19. It might be useful to others to update that in this repo's Manifest file.

Storage servers should share a timestamp of the current registry hash. One idea is to use the Last-Modified header for /registries but if that endpoint advertises multiple resources this does not really work.

The cache miss latency can be pretty bad, especially for large artifacts. In the case of the infamous 4747fe73a29e16337cb33702edae26b4e681ae52 artifact, this miss latency can be measured in minutes, which naturally times nginx out and results in a 504. I can think of a few different ways we can improve this:

One way we could reduce this is to forward users who request resources that we don't have, and that are taking more than N seconds to fetch, on to the storage server. E.g. return a 302 to go ask the storage server for the resource directly, while the Pkg server continues to fetch it in the background.

Another way we could reduce this is to immediately start streaming the response to the client, before we even finish verifying the treehash or anything. This is equivalent in terms of implicitly trusting the storage server, and has the added advantage of having the Pkg server be the one communicating with the user, which is most likely a lower-latency connection than straight to the storage server.

A final way we could reduce this is to immediately serve a 404 if the file is missing, which would cause the Pkg client to immediately fall back to asking GitHub/the original hosting provider, while the Pkg server fetches it in the background.

I consider all three of these responses valid options, it really just depends philosophically on what we want to do.

Hello

Is there a variable to make Julia ignore SSL certificate issues? I have a firewall doing MITM between the docker container and Github. I added this certificate to the docker CA store, and curl works fine with it, but Julia still gives me a chain error.

This does not help, either.

ENV GIT_SSL_NO_VERIFY=true
RUN GIT_SSL_NO_VERIFY=true julia --project=/app -e "using Pkg; Pkg.instantiate(); Pkg.precompile()"```

Thanks!

The verify_registry_hash method is used to check if registry hashes exist at the git repo as a safeguard, but from what I can tell this checking is not currently happening when using the default settings.

This line in PkgServer.jl sets the default upstream url for the General registry, without a .git on the end of it (a string that's repeated in a few other places in the code):

The verify_registry_hash method relies on Pkg.Operations.get_archive_url_for_version, which uses .git in its regex:

function get_archive_url_for_version(url::String, ref)
    if (m = match(r"https://github.com/(.*?)/(.*?).git", url)) !== nothing
        return "https://api.github.com/repos/$(m.captures[1])/$(m.captures[2])/tarball/$(ref)"
    end
    return nothing
end

I've tried running the methods with the default string and a valid hash, and confirmed that get_archive_url_for_version is just returning nothing, and then verify_registry_hash doesn't complain because it returns true if nothing was returned:

So it looks like the check was never actually happening with the default upstream url. Unless this was deliberate, this should be an easy fix.

Is there a simple way to clone the storage server?
Basically I'm trying to host a fully offline PkgServer for internal usage without online communication. It looks like the main issue is obtaining a copy of the packages stored at storage_server.

I do think that we should have an open source script that mirrors the Julia Computing storage servers and serves them statically. That will act as a backup in case the JC storage servers go down.

Originally posted by @StefanKarpinski in JuliaRegistries/General#16777 (comment)

Hi,

With your help I have the package server now running. Though Julia doesn't show we are using it:

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.5.3
 _/ |\__'_|_|_|\__'_|  |
|__/                   |

julia> ENV["JULIA_PKG_SERVER"]
"http://10.69.24.59:30025"

(@v1.5) pkg> up
 Installing known registries into `~/.julia`
######################################################################## 100.0%
      Added registry `General` to `~/.julia/registries/General`
   Updating registry at `~/.julia/registries/General`
No Changes to `~/.julia/environments/v1.5/Project.toml`
No Changes to `~/.julia/environments/v1.5/Manifest.toml`

(@v1.5) pkg> registry st
Registry Status
 [23338594] General (https://github.com/JuliaRegistries/General.git)

The server is up:

vader@deathstar[~]$ curl http://10.69.24.59:30025
<!doctype html>
<html lang=en>
<meta charset=UTF-8>
<title>Pkg Server</title>
You have reached a <a href="https://julialang.org">Julia</a> package server. It is not meant for browsing but for serving resource required by Julia clients to install packages. To discover and explore the Julia package ecosystem, see <a href="https://julialang.org/packages/">this page</a> for a list of services.
</html>

Is there a way from pkg to show that https://github.com/JuliaRegistries/General.git is being proxied through http://10.69.24.59:30025 either in the Julia client or through some debug logging on Pkgserver itself?

Hi,

This issue isn't resolved.

I added CA certificates to the docker container, while also setting

ENV JULIA_SSL_NO_VERIFY_HOSTS="**"

ENV JULIA_NO_VERIFY_HOSTS="**"

I can build the Docker image with no chain issues reported. Though when I try to docker run:

Info: Loading initial Server Config
Info: Performing initial registry update
ERROR: LoadError: IOError(MbedTLS error code -9984: X509 - Certificate verification failed, e.g. CRL, CA or signature check failed during request(https://kr.storage.juliahub.com/registries))

Stacktrace:
  [1] handshake(ctx::MbedTLS.SSLContext)
    @ MbedTLS /depot/packages/MbedTLS/4YY6E/src/ssl.jl:83
  [2] handshake!
    @ /depot/packages/MbedTLS/4YY6E/src/MbedTLS.jl:134 [inlined]
  [3] sslconnection(tcp::TCPSocket, host::SubString{String}; require_ssl_verification::Bool, sslconfig::MbedTLS.SSLConfig, kw::Base.Iterators.Pairs{Symbol, Union{Nothing, Bool}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :reached_redirect_limit, :status_exception), Tuple{Nothing, Bool, Bool}}})
    @ HTTP.ConnectionPool /depot/packages/HTTP/1dCtY/src/ConnectionPool.jl:722
  [4] #getconnection#29
    @ /depot/packages/HTTP/1dCtY/src/ConnectionPool.jl:706 [inlined]
  [5] newconnection(pod::HTTP.ConnectionPool.Pod, T::Type, host::SubString{String}, port::SubString{String}, pipeline_limit::Int64, require_ssl_verification::Bool, idle_timeout::Int64; kw::Base.Iterators.Pairs{Symbol, Union{Nothing, Bool}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :reached_redirect_limit, :status_exception), Tuple{Nothing, Bool, Bool}}})
    @ HTTP.ConnectionPool /depot/packages/HTTP/1dCtY/src/ConnectionPool.jl:618
  [6] getconnection(::Type{HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}}, host::SubString{String}, port::SubString{String}; connection_limit::Int64, pipeline_limit::Int64, idle_timeout::Int64, reuse_limit::Int64, require_ssl_verification::Bool, kw::Base.Iterators.Pairs{Symbol, Union{Nothing, Bool}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :reached_redirect_limit, :status_exception), Tuple{Nothing, Bool, Bool}}})
    @ HTTP.ConnectionPool /depot/packages/HTTP/1dCtY/src/ConnectionPool.jl:562
  [7] request(::Type{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}, url::URIs.URI, req::HTTP.Messages.Request, body::Vector{UInt8}; proxy::Nothing, socket_type::Type, reuse_limit::Int64, kw::Base.Iterators.Pairs{Symbol, Union{Nothing, Bool}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :reached_redirect_limit, :status_exception), Tuple{Nothing, Bool, Bool}}})
    @ HTTP.ConnectionRequest /depot/packages/HTTP/1dCtY/src/ConnectionRequest.jl:81
  [8] (::Base.var"#72#74"{Base.var"#72#73#75"{ExponentialBackOff, HTTP.RetryRequest.var"#2#3"{Bool, HTTP.Messages.Request}, typeof(HTTP.request)}})(::Type, ::Vararg{Any, N} where N; kwargs::Base.Iterators.Pairs{Symbol, Union{Nothing, Bool}, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :reached_redirect_limit, :status_exception), Tuple{Nothing, Bool, Bool}}})
    @ Base ./error.jl:301
    

Does HTTP.jl not honor those environmental variables?

Originally posted by @ifelsefi in #110 (comment)

We had a discussion about the us-east loadbalancer getting slow. Initial inspection showed that the network interface was rarely achieving more than 180Mbps out. Diving deeper, it was found that some nginx processes had become stuck in "uninterruptible sleep" (D in ps output). Looking in dmesg after a echo w > /proc/sysrq-trigger showed that they were stuck in the kernel during an fsync.

This is a fairly pathological failure, but there are a few things we could do to ameliorate it:

• Oversubscribe the number of nginx processes. Right now we only have 2 processes on the loadbalancer; we could probably double this and go up to 2x per core (so 4 processes total) without any harmful effects, which would at least delay the problem in the future. This needs a templating step on the "optimized" nginx config to insert $(($(nproc) * 2)) into the worker_processes directive.
• Better monitoring on the instances; our grafana server has bitrotted, we should resurrect that so that we can notice the lowered network throughput and increased CPU time as shown in this lightsail graph:

I am loathe to do something drastic like auto-reboot the loadbalancer because it is supposed to be the piece that you don't have to reboot. If this happens again, I'll consider it.

Franklin 0.10.9 was released at Otc 28 (The latest Franklin version is 0.10.17)

In a docker container:

julia> ENV["JULIA_PKG_SERVER"] = "https://cn-east.pkg.julialang.org"
"https://cn-east.pkg.julialang.org"

julia> versioninfo()
Julia Version 1.5.0
Commit 96786e22cc (2020-08-01 23:44 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-6800K CPU @ 3.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, broadwell)
Environment:
  JULIA_PATH = /usr/local/julia
  JULIA_GPG = 3673DF529D9049477F76B37566E3C7DC03D6E495
  JULIA_VERSION = 1.5.0
  JULIA_PKG_SERVER = https://cn-east.pkg.julialang.org

(@v1.5) pkg> add Franklin@0.10.17
 Installing known registries into `~/.julia`
######################################################################## 100.0%
      Added registry `General` to `~/.julia/registries/General`
  Resolving package versions...
ERROR: Unsatisfiable requirements detected for package Franklin [713c75ef]:
 Franklin [713c75ef] log:
 ├─possible versions are: [0.4.4, 0.5.0, 0.6.0-0.6.17, 0.7.0-0.7.10, 0.8.0-0.8.6, 0.9.0-0.9.16, 0.10.0-0.10.9] or uninstalled
 └─restricted to versions 0.10.17 by an explicit requirement — no versions left

Both USTC and SJTUG mirrors are up-to-date.

curl -vvv https://eu.pkg.julialang.org/registries
returns Content-Encoding: gzip header, but the response contents is /registry/23338594-aafe-5451-b93e-139f81909106/911a396527a56e9c8351bdb6d1c02c14b3873d4d
so clearly, it's not gzipped, but plaintext.

v1 isn't supported since this past summer. Looks like it is mostly enough to replace docker-compose with docker compose but got some other errors too. Needs to be synced with the upgrade on the servers, naturally.

Ref #66

Just opening this issue so we don't forget to update the status page.

@staticfloat @fredrikekre

If I wanted to use the static server option (i.e. just store everything on disk) for the entire General registry, how much disk space would I need, ballpark estimate?

Since a great use of a Package Server is to coordinate work across multiple registries — potentially a private registry in addition to the General one — it'd be helpful if the Package Server could advertise which registries are available. This could allow for two niceties on the client side:

• Adding the private registry by name — like how ] registry add General is currently special-cased.

and/or

• Automatically ensuring all advertised registries are added upon first connection

When using Docker to run this image, we occasionally run into issues where nginx creates a file with weird ownership. This is because it’s quite difficult to get nginx to run as our UID. This could be solved by using subuid mapping with docker to run everything under our own UID.

If I want a URL for the tarball of a package at a specific version, all I have to do is look up UUID for the package and look up the git-tree-sha1 for the version, and then the following URL gives a .tar.gz file for the tarball:

https://pkg.julialang.org/package/uuid/git-tree-sha1

However, this URL will give a 404 error:

https://pkg.julialang.org/package/uuid/git-tree-sha1.tar.gz

Would it be possible for the Pkg Server to allow the latter form of URL?

When I type ]update to update my packages, it takes a long time to refresh the entire registry(a long time on removing, and even more on downloading). I observed that many small files of package info are downloaded sequentially which can result in a bad performance. Actually those files stay unchanged.

The situation can be mitigated if only there is a list of registry package containing information of all package versions. While updating packages, we download this list file and find those package info which is out of date. In the best case, it changes nothing and all costs involve a single file download, while now it has to take several minutes to refresh the entire registry.

The current Julia compat entry in Project.toml is:

julia = "1.3"

But https://github.com/StefanKarpinski/PkgServer.jl/blob/master/bin/gen_static.jl#L132 has:

    for reg_dir in readdir(depot_regs, join=true)

In Julia 1.3, readdir does not accept keyword arguments, but it does on Julia master.

So I think you need Julia master or a relatively recent Julia nightly.

$ for region in eu-central us-east1 us-east2 us-west; do
> curl -fsSL https://${region}.pkg.julialang.org/package/12aac903-9f7c-5d81-afc2-d9565ea332ae/83aa52e9121f1cf30e7dd170b66889f9d37ecf61 | sha256sum 
> done
89e49debfc51ef09f27cf69ac7c2d927b98667f964db254e0dbe6c290aec2a1b  -
a373504da76c1a87631d96b466c49c150c774efe10954daf9fa15055d42fb59c  -
a373504da76c1a87631d96b466c49c150c774efe10954daf9fa15055d42fb59c  -
a373504da76c1a87631d96b466c49c150c774efe10954daf9fa15055d42fb59c  -

Relevant log extract:

┌ Info: [2021-05-16 04:32:02] downloading resource
│   server = https://us-east.storage.juliahub.com
│   resource = /package/12aac903-9f7c-5d81-afc2-d9565ea332ae/83aa52e9121f1cf30e7dd170b66889f9d37ecf61
└ @ PkgServer /app/src/resource.jl:422
┌ Info: [2021-05-16 04:32:02] download complete
│   server = https://us-east.storage.juliahub.com
│   resource = /package/12aac903-9f7c-5d81-afc2-d9565ea332ae/83aa52e9121f1cf30e7dd170b66889f9d37ecf61
│   elapsed = 0.4293828010559082
└ @ PkgServer /app/src/resource.jl:481
┌ Info: [2021-05-16 04:32:02] Moving
│   temp_file = /app/storage/temp/package/12aac903-9f7c-5d81-afc2-d9565ea332ae/83aa52e9121f1cf30e7dd170b66889f9d37ecf61.inprogress
│   new_path = /app/storage/cache/package/12aac903-9f7c-5d81-afc2-d9565ea332ae/83aa52e9121f1cf30e7dd170b66889f9d37ecf61
│   filesize(temp_file) = 50805
└ @ PkgServer /app/src/resource.jl:84
┌ Info: [2021-05-16 04:32:02] Deleting and pruning
│   temp_file = /app/storage/temp/package/12aac903-9f7c-5d81-afc2-d9565ea332ae/83aa52e9121f1cf30e7dd170b66889f9d37ecf61.inprogress
└ @ PkgServer /app/src/resource.jl:91
┌ Error: [2021-05-16 04:32:02] dl_task abnormal termination!
└ @ PkgServer /app/src/resource.jl:402
┌ Error: [2021-05-16 04:32:02] file size mismatch
│   content_length = 125646
│   actual = 50805
└ @ PkgServer /app/src/resource.jl:572

Some initial exploration of the data from #196 suggest that registries have a way to "stick" in the cache. Some datapoints (aggregation over all deployed servers):

In particular it seems wasteful to store so many registries in particular since they are quite large (7x the size of the average package). Registries are also a bit different in that all(?) requests want the latest one, so caching something like the 10 latest should be more than sufficient. This would free up space for packages and artifacts instead, which could potentially reduce network usage between package servers and storage servers.

The major reason causing this (I think) is that we now serve existing files from nginx directly but this cache hit is not recorded on the julia resulting in all resources have num_accessed = 1.

Removing the registries from the equation as suggested above should help quite a bit, but might also be worth trying some way to communicate the cache hits from nginx to julia.

Since Docker hub doesn't do automated builds for us anymore, let's do the build ourselves on CI.

@DilumAluthge can I enlist your help in setting up a job that does the following steps:

echo "${DOCKER_PASSWORD}" | docker login --username "${DOCKER_USERNAME}" --password-stdin
docker build -t juliapackaging/pkgserver.jl .

On every PR, and then also does:

docker push juliapackaging/pkgserver.jl

If running on a commit that was merged to master?

Julia like to eat up much of my M1's ram

Two days ago, after being notified that I am out of ram, I noticed four Julia processes or something were running and they were consuming a huge amount of RAM. I use Julia on VSCode with either .jl files or .ipynb Jupyter notebooks. That day, I had to kill some Julia processes and the results afterwards were that every time I try to add, update, instantiate, etc. a package I get the error

ERROR: CRC Failed : General.tar
ERROR: EOFError: read end of file

Temporary working solution has been

ENV["JULIA_PKG_SERVER"] = ""

Looking forward for a long term solution. Any help is appreciated.

An example: curl https://pkg.julialang.org/artifact/cb84c2e2544f3517847d90c13cc11ab911fdbc5c gets a 404. This artifact doesn't have download info, it is built when requested using the demo from https://julialang.org/blog/2019/11/artifacts/#working_with_artifacts

The current gen_static.jl could actually create tarballs for them, as long as they're available in $JULIA_DEPOT_PATH/artifacts/. IIUC adding a Pkg.build(pkg) could cover some of the similar cases.

This was observed today, the us-east PkgServer got stuck on treehash eb887e13993d3e904ac01fe41d68e7bc77d14277, which corresponds to JuliaRegistries/General@668e84c.

There were no logs of an error or anything. We need to investigate how this can happen. It's vital that we have the ability to run while true loops inside of tasks and not worry about them disappearing on us.

I was attempting to try this out today, but precompilation fails for me because of statsfs

Should this be running on Windows or not?

(PkgServer) pkg> st
     Project PkgServer v0.2.0
      Status `C:\Users\bernhard.konig\Documents\Repositories\PkgServer.jl\Project.toml`
  [89f0c457] FilesystemDatastructures v1.0.0
  [cd3eb016] HTTP v0.9.13
  [0f8b85d8] JSON3 v1.9.1
  [e6f89c97] LoggingExtras v0.4.7
  [777ac1f9] SimpleBufferStream v1.1.0
  [856f2bd8] StructTypes v1.7.2
  [be1be57a] Gzip_jll v1.10.0+0
  [ade2ca70] Dates
  [76f85450] LibGit2
  [8f399da3] Libdl
  [56ddb016] Logging
  [44cfe95a] Pkg
  [9a3f8284] Random
  [ea8e919c] SHA
  [6462fe0b] Sockets
  [a4e569a6] Tar

julia> using PkgServer
[ Info: Precompiling PkgServer [eac38ba3-4627-46d4-b1a1-bcb86ba22f8b]
ERROR: LoadError: Unable to statfs() on Val{:NT}()
Stacktrace:
  [1] error(s::String)
    @ Base .\error.jl:33
  [2] statfs(path::String, kernel::Val{:NT})
    @ FilesystemDatastructures C:\Users\bernhard.konig\.julia\packages\FilesystemDatastructures\D6JHz\src\DiskUtils.jl:58
  [3] get_disk_freespace(path::String)
    @ FilesystemDatastructures C:\Users\bernhard.konig\.julia\packages\FilesystemDatastructures\D6JHz\src\DiskUtils.jl:70
  [4] (::FilesystemDatastructures.var"#24#25"{Int64})(scfc::FilesystemDatastructures.SizeConstrainedFileCache)
    @ FilesystemDatastructures C:\Users\bernhard.konig\.julia\packages\FilesystemDatastructures\D6JHz\src\SizeConstrainedFileCache.jl:276
  [5] FilesystemDatastructures.SizeConstrainedFileCache(root::String, target_size::Function, discard_ordering::Function; predicate::Function)
    @ FilesystemDatastructures C:\Users\bernhard.konig\.julia\packages\FilesystemDatastructures\D6JHz\src\SizeConstrainedFileCache.jl:90
  [6] SizeConstrainedFileCache
    @ C:\Users\bernhard.konig\.julia\packages\FilesystemDatastructures\D6JHz\src\SizeConstrainedFileCache.jl:78 [inlined]
  [7] PkgServer.ServerConfig(; listen_addr::Sockets.InetAddr{Sockets.IPv4}, storage_root::String, registries::Dict{String, PkgServer.RegistryMeta}, storage_servers::Vector{String}, keep_free::Int64)
    @ PkgServer C:\Users\bernhard.konig\Documents\Repositories\PkgServer.jl\src\PkgServer.jl:69
  [8] PkgServer.ServerConfig()
    @ PkgServer C:\Users\bernhard.konig\Documents\Repositories\PkgServer.jl\src\PkgServer.jl:64
  [9] top-level scope
    @ C:\Users\bernhard.konig\Documents\Repositories\PkgServer.jl\src\PkgServer.jl:85
 [10] include
    @ .\Base.jl:386 [inlined]
 [11] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base .\loading.jl:1235
 [12] top-level scope
    @ none:1
 [13] eval
    @ .\boot.jl:360 [inlined]
 [14] eval(x::Expr)
    @ Base.MainInclude .\client.jl:446
 [15] top-level scope
    @ none:1
in expression starting at C:\Users\bernhard.konig\Documents\Repositories\PkgServer.jl\src\PkgServer.jl:1
ERROR: Failed to precompile PkgServer [eac38ba3-4627-46d4-b1a1-bcb86ba22f8b] to C:\Users\bernhard.konig\.julia\compiled\v1.6\PkgServer\jl_BA6B.tmp.
Stacktrace:
 [1] error(s::String)
   @ Base .\error.jl:33
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::Base.TTY, internal_stdout::Base.TTY, ignore_loaded_modules::Bool)
   @ Base .\loading.jl:1385
 [3] compilecache(pkg::Base.PkgId, path::String)
   @ Base .\loading.jl:1329
 [4] _require(pkg::Base.PkgId)
   @ Base .\loading.jl:1043
 [5] require(uuidkey::Base.PkgId)
   @ Base .\loading.jl:936
 [6] require(into::Module, mod::Symbol)
   @ Base .\loading.jl:923


julia> versioninfo()
Julia Version 1.6.2
Commit 1b93d53fc4 (2021-07-14 15:36 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: AMD Ryzen 9 5950X 16-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, generic)
Environment:
  JULIA_EDITOR = "C:\Program Files\Microsoft VS Code\Code.exe"

Navigating to https://kr.storage.julialang.org/ in a web browser gives the following nice message:

You have reached a Julia storage server. It is not meant for browsing but for serving resource required by Julia clients to install packages. To discover and explore the Julia package ecosystem, see this page for a list of services.

However, navigating to https://us-east.storage.julialang.org/ in a web browser gives a "403 forbidden" error.

Presumably, navigating to any of the storage servers in a web browser should give the same helpful message.

cc: @staticfloat

When I registered new version of package, Pkg Server didn't have it for few hours, which make running tests on Julia 1.5 on pckages using this new version fail.

The PkgServers are sad, specifically us-east.pkg.julialang.org fails alot with this stacktrace:

pkgserver1_1  | ┌ Error: error handling request
pkgserver1_1  | │   exception =
pkgserver1_1  | │    IOError: stream is closed or unusable
pkgserver1_1  | │    Stacktrace:
pkgserver1_1  | │     [1] check_open at ./stream.jl:328 [inlined]
pkgserver1_1  | │     [2] uv_write_async(::TCPSocket, ::Ptr{UInt8}, ::UInt64) at ./stream.jl:961
pkgserver1_1  | │     [3] uv_write(::TCPSocket, ::Ptr{UInt8}, ::UInt64) at ./stream.jl:924
pkgserver1_1  | │     [4] unsafe_write(::TCPSocket, ::Ptr{UInt8}, ::UInt64) at ./stream.jl:1007
pkgserver1_1  | │     [5] unsafe_write at /depot/packages/HTTP/atT5q/src/ConnectionPool.jl:174 [inlined]
pkgserver1_1  | │     [6] unsafe_write at ./io.jl:593 [inlined]
pkgserver1_1  | │     [7] write at ./io.jl:616 [inlined]
pkgserver1_1  | │     [8] startwrite(::HTTP.Streams.Stream{HTTP.Messages.Request,HTTP.ConnectionPool.Transaction{TCPSocket}}) at /depot/packages/HTTP/atT5q/src/Streams.jl:87
pkgserver1_1  | │     [9] serve_file(::HTTP.Streams.Stream{HTTP.Messages.Request,HTTP.ConnectionPool.Transaction{TCPSocket}}, ::IOStream, ::String, ::String; buffer::Array{UInt8,1}, content_length::Int64, dl_task::Task) at /app/src/resource.jl:526
pkgserver1_1  | │     [10] serve_file(::HTTP.Streams.Stream{HTTP.Messages.Request,HTTP.ConnectionPool.Transaction{TCPSocket}}, ::IOStream, ::String, ::String) at /app/src/resource.jl:497
pkgserver1_1  | │     [13] (::HTTP.Servers.var"#13#14"{PkgServer.var"#66#69",HTTP.ConnectionPool.Transaction{TCPSocket},HTTP.Streams.Stream{HTTP.Messages.Request,HTTP.ConnectionPool.Transaction{TCPSocket}}})() at ./task.jl:358 | │     [12] macro expansion at /depot/packages/HTTP/atT5q/src/Servers.jl:367 [inlined]
pkgserver1_1  | └ @ HTTP.Servers ~/depot/packages/HTTP/atT5q/src/Servers.jl:373

I'd like to set up a static server, but I only want to host a subset of packages. What would be the best way to accomplish this?

I guess one approach would be for me to make a fork of General, and then delete everything from the fork that I don't want to host. But that will become a real pain to maintain.

I'm wondering if there is an easier approach?