googlecloudplatform / artifact-registry-apt-transport Goto Github PK

Hi,

I tried to use google cloud artifact registry on docker ubuntu:21.10. It seems that the transport module can not determine the size of the packages in the repo and gives following error:

apt update
apt install opencv

E: Repository is broken: ar+https://europe-west3-apt.pkg.dev/projects/aisw-ww-ai stage-apt/main amd64 opencv amd64 4.5.5 has no Size information

using apt -oAcquire::AllowUnsizedPackages=1 install opencv suppressed the error but the download stucks at 0%.

With Ubuntu 20.04 there it comes to a warning but it generally works.

Apt has its own schema for separating private data such as usernames and passwords from the otherwise open to read config files. We should investigate offering support for it (for providing or referencing the auth data we already accept in our own config), as it seems it's where users would expect to configure and there may be tooling integration benefits.

https://manpages.ubuntu.com/manpages/bionic/man5/apt_auth.conf.5.html

The naming scheme for packages is recommended in sources.list(5):

  adding more recognizable URI types

        APT can be extended with more methods shipped in other optional
        packages, which should follow the naming scheme apt-transport-method.

This would imply naming the package apt-transport-ar.

GCP official repository does not provide binaries for arm64. However, you can build with GOARCH=arm64 to generate a working binary, which will also install Debian packages from the Artifact Registry successfully.

Is there any way to officially support this?

amd64 Packages: https://packages.cloud.google.com/apt/dists/apt-transport-artifact-registry-stable/main/binary-amd64/Packages
arm64 Packages: https://packages.cloud.google.com/apt/dists/apt-transport-artifact-registry-stable/main/binary-arm64/Packages

Documentations instructions are like following:

echo "deb ar+https://<location>-apt.pkg.dev/projects/<project> <repository> main" | sudo tee -a  /etc/apt/sources.list.d/artifact-registry.list
curl https://<location>-apt.pkg.dev/doc/repo-signing-key.gpg | sudo apt-key add -
sudo apt update

This would work, but managing GPG keys by apt-key is currently deprecated. Using /etc/apt/keyrings/... is recommended, but ar+https handling by this plugin looks not supporting signed-by=... option.

...
W: ar+https://<location>-apt.pkg.dev/projects/<project>/dists/<repository>/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.

Expected flow is following.

echo "deb [signed-by=/etc/apt/keyrings/<location>-artifact.gpg] ar+https://<location>-apt.pkg.dev/projects/<project> <repository> main" | sudo tee -a  /etc/apt/sources.list.d/artifact-registry.list
sudo curl https://<location>-apt.pkg.dev/doc/repo-signing-key.gpg -o /etc/apt/keyrings/<location>-artifact.gpg

But this causes error:

...
Err:5 ar+https://<location>-apt.pkg.dev/projects/<project> <repository> InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY <GPG-KEY-NAME>
...
W: GPG error: ar+https://<location>-apt.pkg.dev/projects/<project> <repository> InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY <GPG-KEY-NAME>
E: The repository 'ar+https://<location>-apt.pkg.dev/projects/<project> <repository> InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

Confirmed in Ubuntu22.04 Google Cloud Compute Engine environment. Thanks

Investigate having URI replacement use the name of the transport binary on disk rather than a hardcoded ar+https value (in method.go#169.

That is, if the file /usr/lib/apt/methods/ar+https is renamed to /usr/lib/apt/methods/myname, entries in sources.list(5) could use the myname:// scheme. This could possibly resolve issues where some tools don't handle the + character in the scheme properly.

Using exponential backoff when fetching packages would help reduce the chance of GCP quota related errors when reading from Artifact Registry APT repositories.

I believe an exponential backoff library could be used here: https://github.com/GoogleCloudPlatform/artifact-registry-apt-transport/blob/main/apt/method.go#L176.

Hi 👋

I'm experiencing an issue with apt install (same with apt-get) whereby it's painfully slow to complete a simple apt install command. So much so that I wasn't able to complete a simple installation of a single package since yesterday. I'm using this sequence of commands on a Google Cloud VM:

curl https://europe-west6-apt.pkg.dev/doc/repo-signing-key.gpg | sudo apt-key add -
echo deb https://europe-west6-apt.pkg.dev/projects/radicle-services radicle-cli main | sudo tee -a /etc/apt/sources.list.d/radicle-registry.list
sudo apt update
sudo apt install radicle-cli

From what I see from strace, apt repeatedly timeouts on select() calls for the most part, idling most of the time.

What's curious, wget 'https://europe-west6-apt.pkg.dev/projects/radicle-services/pool/radicle-cli/radicle-cli_0.6.0_amd64_3662282cc3a7626852c69a4218499ab6.deb' downloads the file instantly.

seems like armhf isn't in stable or unstable repositories.

I use Ubuntu Landscape to manage internal apt package mirroring and have had some trouble integrating with Google Cloud Artifact Registry, specifically because of the way landscape manages another tool (reprepro) to maintain those caches.

After some hacking/experimenting, I found that the reprepro's Method field inside the conf/updates file has trouble using ar+https as a valid scheme.

This ultimately results in reprepro being unable to use any ar+https based apt sources as a valid mirror.

Would it be possible to provide a redundant or alternative method program for the nicer-for-reprepro schemes?

For example; I have compiled and tested a version of this library using the scheme gcp-ar://... and reprepro + landscape are completely happy with this.

We're noticing from time to time that apt updates never complete, and remain locked forever presumably due to a missing timeout and some kind of underlying network issue. Each time we've looked at this, the AR transport binary still seems to be running, which makes me think the missing timeout is somewhere within it.

$ apt-get update
Reading package lists... Done
E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)
E: Unable to lock directory /var/lib/apt/lists/

$ ps aux | grep -i apt
root     18760  0.0  0.0  37564  7040 ?        S    Mar31   6:06 /usr/bin/apt-get update
_apt     18768  0.0  0.1  45420  9180 ?        S    Mar31   0:00 /usr/lib/apt/methods/https
_apt     18769  0.0  0.1  45420  9108 ?        S    Mar31   0:00 /usr/lib/apt/methods/https
root     18770  0.0  0.1 108624 10468 ?        Sl   Mar31   0:48 /usr/lib/apt/methods/ar+https
_apt     18774  0.0  0.0  42388  6624 ?        S    Mar31   0:00 /usr/lib/apt/methods/http
_apt     18775  0.0  0.0  42396  6596 ?        S    Mar31   0:00 /usr/lib/apt/methods/http
_apt     18780  0.0  0.0  36412  5680 ?        S    Mar31   0:00 /usr/lib/apt/methods/gpgv

$ pstree -ap 18760
apt-get,18760 update
  ├─ar+https,18770
  │   ├─{ar+https},18771
  │   ├─{ar+https},18772
  │   ├─{ar+https},18773
  │   ├─{ar+https},18776
  │   ├─{ar+https},18777
  │   └─{ar+https},18778
  ├─gpgv,18780
  ├─http,18774
  ├─http,18775
  ├─https,18768
  └─https,18769

Unfortunately don't have any logs or anything else available, as we mostly notice this when it's triggered via OSConfigAgent, which only seems to collect the resulting apt "exited uncleanly" error when you eventually kill ar+https.

I haven't had a look through the code to see if there a missing timeouts or context propagations. It might also be worth adding a retry mechanism.