drmikehenry / romt Goto Github PK

I had errors like this

error: failed to download https://static.crates.io/crates/lancelot-flirt/lancelot-flirt-0.6.0.crate; exception details ConnectTimeout('')

and this

error: failed to download https://static.crates.io/crates/lc3tools-sys/lc3tools-sys-1.0.6-alpha4.crate; exception details ReadTimeout('')

while mirroring crates.io crate files.
Is this 403: Forbidden as mentioned in README, or something else?
The command i used:

E:\rust_crates>romt crate -v --keep-going update -j=64

The command prompt is opened as a normal user, not admin.

I'm seeing an issue where HTTP requests can take a long time to process before a response is sent, when it should be almost instant. This usually manifests when a client on a different machine has performed a successful HTTP request prior to the stuck HTTP request on the first machine.

As an example, I have two clients on an airgapped network and a server running romt. On the first client, I am able to pull crates and access the server directory through a web browser with no perceived slowness. I then go on the second client and access the server directory through a web browser with no perceived slowness. When I go back to the first client and attempt to pull crates or access the server directory, it will hang on the request (anywhere from ~90 seconds to forever) before it responds. Additionally, while the first client is stuck after a request, if the second client performs a request the server will not only respond to the second client, but it will also respond to the first client with the data it requested.

I do not believe this to be an issue with our network. When I use wireshark, I can see that the client is successfully communicating with the server. The client sends the HTTP request and the server ACKs it before it hangs.

We currently have the romt server running on a Ubuntu machine and it's configured to serve over unencrypted HTTP. We haven't tried using nginx, but we will try that configuration to see if this manifests. If anything seems unclear, I can grab screenshots or a screen capture to demonstrate.

I ran through the quick-start development-laptop server directions for release 0.1.2 binary, only configured the .cargo/config variable to redirect to localhost.

I've downloaded the entire crate library and am now running

romt serve -v

and I get the below error

Rewrite URL: /git/crates.io-index/info/refs?service=git-upload-pack -> /cgi-bin/git-http-backend.py/crates.io-index/info/refs?service=git-upload-pack
127.0.0.1 - - [23/May/2020 09:07:50] "GET /git/crates.io-index/info/refs?service=git-upload-pack HTTP/1.1" 200 -
127.0.0.1 - - [23/May/2020 09:07:50] command: /usr/bin/romt -u /opt/romt/mirror/cgi-bin/git-http-backend.py
127.0.0.1 - - [23/May/2020 09:07:50] b"usage: romt [-h] [--version] [-v] [-q] [--readme]\n            {crate,rustup,toolchain,serve} ...\nromt: error: argument subparser_name: invalid choice: '/opt/romt/mirror/cgi-bin/git-http-backend.py' (choose from 'crate', 'rustup', 'toolchain', 'serve')\n"
127.0.0.1 - - [23/May/2020 09:07:50] CGI script exit status 0x2

when I run

romt -u

I get the below error

usage: romt [-h] [--version] [-v] [-q] [--readme]
            {crate,rustup,toolchain,serve} ...
romt: error: unrecognized arguments: -u

I'm not sure if I'm doing something wrong. I wanted to post this first before digging into the code, cause usually its operator error.
My env is ubuntu 20.02

Hello,

I've followed the instructions, set environment variables, and set up my config file, but when I try to download crates, both via romt serve or with nginx, I get the following error:

spurious network error (2 tries remaining): [6] Couldn't resolve host name (Could not resolve host: crates.io)

Is it perhaps an issue with cgi delivery? I'm able to install rustup and toolchains from the local mirror, but crates haven't worked.

I am attempting to mirror crates for offline use, but I can't seem to get past downloading certain crates when there is a 403 forbidden. The only way for me to get past the affected crates is to create an empty file and do a download with --assume-ok option.

On wsl2 Ubuntu 20.04 from pip install romt:

romt crate -v --keep-going init export
...
[downloading] crates/bo/rk/bork/bork-0.0.0.crate
Traceback (most recent call last):
  File "/home/user/.local/bin/romt", line 8, in <module>
    sys.exit(main())
  File "/home/user/.local/lib/python3.8/site-packages/romt/cli.py", line 172, in main
    romt.crate.Main(args).run()
  File "/home/user/.local/lib/python3.8/site-packages/romt/base.py", line 50, in run
    self._run()
  File "/home/user/.local/lib/python3.8/site-packages/romt/crate.py", line 916, in _run
    getattr(self, cmd_func_name)()
  File "/home/user/.local/lib/python3.8/site-packages/romt/crate.py", line 831, in cmd_download
    download_crates(
  File "/home/user/.local/lib/python3.8/site-packages/romt/crate.py", line 341, in download_crates
    _process_crates(
  File "/home/user/.local/lib/python3.8/site-packages/romt/crate.py", line 317, in _process_crates
    downloader.run_job(_process_inner)
  File "/home/user/.local/lib/python3.8/site-packages/romt/download.py", line 223, in run_job
    trio.run(functools.partial(job, **kwargs), *args)
  File "/home/user/.local/lib/python3.8/site-packages/trio/_core/_run.py", line 1932, in run
    raise runner.main_task_outcome.error
  File "/home/user/.local/lib/python3.8/site-packages/romt/crate.py", line 313, in _process_inner
    nursery.start_soon(
  File "/home/user/.local/lib/python3.8/site-packages/trio/_core/_run.py", line 815, in __aexit__
    raise combined_error_from_nursery
  File "/home/user/.local/lib/python3.8/site-packages/romt/crate.py", line 285, in _process_one
    await downloader.adownload_verify_hash(
  File "/home/user/.local/lib/python3.8/site-packages/romt/download.py", line 129, in adownload_verify_hash
    await self.adownload(dest_url, dest_path)
  File "/home/user/.local/lib/python3.8/site-packages/romt/download.py", line 58, in adownload
    await self._adownload(url, tmp_dest_path)
  File "/home/user/.local/lib/python3.8/site-packages/romt/download.py", line 48, in _adownload
    await self.adownload_fileobj(url, f)
  File "/home/user/.local/lib/python3.8/site-packages/romt/download.py", line 35, in adownload_fileobj
    response.raise_for_status()
  File "/home/user/.local/lib/python3.8/site-packages/httpx/_models.py", line 1402, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: 403 Client Error: Forbidden for url: https://static.crates.io/crates/bork/bork-0.0.0.crate
For more information check: https://httpstatuses.com/403

Then I tried the following after downloading the stand alone executable:

./romt-0.3.0-x86_64-linux crate --keep-going --num-jobs 50 download
download...
Traceback (most recent call last):
  File "romt-wrapper.py", line 5, in <module>
  File "romt/cli.py", line 172, in main
  File "romt/base.py", line 50, in run
  File "romt/crate.py", line 916, in _run
  File "romt/crate.py", line 839, in cmd_download
  File "romt/crate.py", line 349, in download_crates
  File "romt/crate.py", line 317, in _process_crates
  File "romt/download.py", line 223, in run_job
  File "trio/_core/_run.py", line 1932, in run
  File "romt/crate.py", line 314, in _process_inner
  File "trio/_core/_run.py", line 815, in __aexit__
  File "romt/crate.py", line 286, in _process_one
  File "romt/download.py", line 129, in adownload_verify_hash
  File "romt/download.py", line 58, in adownload
  File "romt/download.py", line 48, in _adownload
  File "romt/download.py", line 35, in adownload_fileobj
  File "httpx/_models.py", line 1402, in raise_for_status
httpx.HTTPStatusError: 403 Client Error: Forbidden for url: https://static.crates.io/crates/bork/bork-0.2.0.crate
For more information check: https://httpstatuses.com/403
[1920] Failed to execute script 'romt-wrapper' due to unhandled exception!

i am trying to make a romt docker container base on alpine3.14, the git-http-backend is at /usr/libexec/git-core/git-http-backend instead of
/usr/lib/git-core/git-http-backend.

maybe we could add another argument --git-http-backend or lookup it in PATH environment

Hello,

Ran into a little trouble following the readme to unpack an export created on a Mac. I'm using the latest release and ran the following:

romt crate -v --keep-going init export
romt toolchain -v -s nightly -t windows download pack
romt rustup -v -s stable -t windows download pack

Everything downloaded as expected and I then went to move the resulting .tar.gz files to another laptop. Per the instructions I then ran (using the latest release romt-0.3.1 for windows):

romt.exe toolchain -v unpack 

with no errors. Next,

romt.exe rustup -v unpack 

with no errors. And finally,

romt.exe crate -v --keep-going init-import import

and get the following error:

init-import...
create index repository at 'git\\crates.io-index':
  remote add origin E:\mirror\git\crates.io-index\origin.bundle
create crates directory at 'crates':
import...
unpack...
error: INDEX remote ``origin`` must have ``url`` as a local file

Everything was normal before.
It may have been in December last year when the download was suddenly not normal.
His error message was so simple that I didn't know what went wrong:

root@szvphispre59334:~# romt toolchain -v -s nightly -t all download
Download: nightly
[downloading] dist/channel-rust-nightly.toml
error: failed to download https://static.rust-lang.org/dist/channel-rust-nightly.toml.sha256

I found that an error occurred in this code, but I still don't know why:
src/romt/download.py

async def adownload_fileobj(self, url: str, fileobj: BinaryIO) -> None:
        print ("debug----0")
        print (url)
        print (fileobj)
        try:
            async with self._client.stream("POST", url) as response:
                print ("debug----1")
                print (response)
                response.raise_for_status()
                async for chunk in response.aiter_bytes(chunk_size=16384):
                    fileobj.write(chunk)
        except httpx.HTTPError as e:
            print ("debug----2")
            raise error.DownloadError(url, e)

In this code, debug----0 and debug----2 can be printed, but debug----1 cannot be printed. I don't know what the problem is.Can you give us some ideas?

Hi,

I ran the crates download on a Windows machine, some of crates got downloaded into paths with uppercase letters.
For example aead got downloaded into crates/Ae/ad/aead/aead-x.y.z.crate (notice the Ae).

I have transferred the crates to a network drive on an offline network, and serve from that drive from a linux machine.

When serving from one machine with the drive mounted using cifs, everything seems to work fine, but when serving from another machine that has the drive mounted as nfs, all crates that have uppercase letters in their prefix seem to break (return 404).

Any ideas or help would be appreciated.

Is there anyway to mirror only a subset of crates.io and their dependencies?

When running

romt toolchain \
    -v \
    -s nightly \
    -s stable \
    -t all \
    download pack 

Everything seems to run fine.

Then after transferring the tar to the offline machine and running romt toolchain -v unpack everything seems to be working correctly until it breaks when it starts verifying the stable version, with the following error:

[verify] dist/2022-04-07/channel-rust-stable.toml
    ident: stable-2022-04-07(1.60.0)
error: target 'x86_64-unknown-none' not found in manifest

This appears to be a target that exists in the nightly release (17/04/2022) but doesn't exist in the stable release.
(The target appears in the manifest of the nightly toolchains and is also downloaded/extracted/verified but doesn't appear anywhere in the stable release)

My current workaround is downloading the nightly and stable versions separately into different archives.

I followed the README to download rust artifacts, I successfully download the toolchain. When download rustup, there is some problem.

romt rustup -v -s stable -t linux download

Fixup: stable
[read] rustup/release-stable.toml
[read] rustup/release-stable.toml
[write] rustup/release-stable.toml (version=1.23.1)
[read] rustup/release-stable.toml
[copytree] rustup/archive/1.23.1 -> rustup/dist
Traceback (most recent call last):
  File "/home/binbowang/.local/bin/romt", line 8, in <module>
    sys.exit(main())
  File "/home/binbowang/.local/lib/python3.8/site-packages/romt/cli.py", line 164, in main
    romt.rustup.Main(args).run()
  File "/home/binbowang/.local/lib/python3.8/site-packages/romt/rustup.py", line 504, in run
    self.cmd_fixup()
  File "/home/binbowang/.local/lib/python3.8/site-packages/romt/rustup.py", line 483, in cmd_fixup
    shutil.copytree(archive_version_path, dist_path)
  File "/usr/lib/python3.8/shutil.py", line 554, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
  File "/usr/lib/python3.8/shutil.py", line 510, in _copytree
    raise Error(errors)
shutil.Error: [('rustup/archive/1.23.1/x86_64-unknown-linux-gnu/rustup-init', 'rustup/dist/x86_64-unknown-linux-gnu/rustup-init', '[Errno 1] Operation not permitted'), ('rustup/archive/1.23.1/x86_64-unknown-linux-gnu/rustup-init.sha256', 'rustup/dist/x86_64-unknown-linux-gnu/rustup-init.sha256', '[Errno 1] Operation not permitted'), (<DirEntry 'x86_64-unknown-linux-gnu'>, 'rustup/dist/x86_64-unknown-linux-gnu', '[Errno 1] Operation not permitted'), (PosixPath('rustup/archive/1.23.1'), PosixPath('rustup/dist'), '[Errno 1] Operation not permitted')]