Second build should have a different environment. Suggestions:

• disorderfs
• random(?) locale
• timezone
• build time, example: in the future 2045. When the phrakture have taken over the planet.

==> Finished making: grub 2:2.04-5 (Thu 30 Apr 2020 08:47:25 AM UTC)
==> Cleaning up...
  -> Delete snapshot for build...
==> Comparing hashes...
diff: ./build/grub-2_2.04-5-x86_64.pkg.tar.zst: No such file or directory
==> ERROR: Package is not reproducible

Whatever code that tries to find the package in build does not handle epoch

I tried to run buildinfo on a debian system and got this obscure error:

$ /usr/local/bin/buildinfo -ff cjdns-20.5-1-x86_64.pkg.tar.zst
/usr/local/bin/buildinfo: line 47: data["${key}"]: bad array subscript
pkgname=
pkgbase=
pkgver=
pkgbuild_sha256sum=
packager=
builddate=
builddir=
buildenv=
options=
installed=
$ 

after some troubleshooting it turned out zstd wasn't installed and tar failed. I would expect an error message that is pointing me towards tar.

These are valid semantic errors that can lead to printf doing the wrong thing when the variables used contain format strings. As it stands though, this behavior is expected for one of the variables used here.

I'd probably nest the printfs - one for the mesg formatting, and one for the rest.
Just in case some future terminal with more in-band signaling ends up using % in their escape sequences ;)

Alternatively, consider something like "%s ... ${mesg}" "${BOLD}" ... and whatnot, where everything but mesg is separated into printf formatting

This is also a valid issue. As it currently stands it looks like you may have intended for the quotes to end up in the args as literals, but right now they're all interpreted by your shell. The entire arg should be wrapped in quotes.

Should be "${packages[@]}" instead of ${packages[*]}. What it's currently doing is basically "join all the arguments by the first character of IFS (space), split them by IFS and expand each of them as globs, and use those as args for the commandline you're building"

$home is not a thing, you probably wanted $HOME

I didn't pay attention and missed the fact that the quoted line in #44 was actually from master:

https://github.com/Foxboron/archlinux-repro/blob/88253732fd408b424fa52a853690adb1069fd263/buildinfo#L104-L106

Running gpg --verify like that is exploitable given a malicious mirror.

          gpg may assume that a single argument is a file with a detached
          signature, and it will try to find a matching data file by
          stripping certain suffixes. Using this historical feature to
          verify a detached signature is strongly discouraged; you should
          always specify the data file explicitly.

Specifically, putting a inline signature in ${filename}.sig by somebody from /etc/pacman.d/gnupg/pubring.gpg would skip verification of ${filename}.

Details about this kind of bug over here: https://ctftime.org/task/6731

...
(1/1) installing devtools                                                                                                       [#############################################################################] 100%
Optional dependencies for devtools
    btrfs-progs: btrfs support
:: Running post-transaction hooks...
(1/1) Arming ConditionNeedsUpdate...
'/usr/share/devtools/makepkg-x86_64.conf' -> '/mnt/etc/makepkg.conf'
Generating locales...
  en_US.UTF-8... done
  de_DE.UTF-8... done
Generation complete.
Sorry, user root is not allowed to execute '/bin/bash -c bash -c .\ /etc/profile;\ .\ /env;\ cd\ /startdir;\ makepkg\ --syncdeps\ --clean\ --noconfirm\ --skippgpcheck' as builduser on <packagename>130297.
-> Delete snapshot for <packagename>

I get this error message when I run "repro -d packagename.tar.zst"

However the user that executed this command is not root, but instead a normal user in the wheel group (which is in the sudoers file)
I was able to successfully execute the command on a different system, but whatever the cause for this is, the users as setup the same and the error message is rather misleading.

Any clue whats broken here?

The entire things with rsync or btrfs should just be removed in favor of overlayfs. devtools also wants this, so any solutions here can be ported back into the devtools project.

Building twice with profiles should be moved to reprotest. Remove the code, files and documentation from this repository.

Since we know the fingerprint of the key that is expected to sign the bootstrap image we could automatically import it if it's not imported yet. This would remove one common error case and drop one step from the list of manual steps.

Maybe you can host a known good version of pacman.git on pkgbuild.com, which can be used instead.

We are currently discussing the cleanup of our release artifact directories (https://gitlab.archlinux.org/archlinux/releng/-/issues/11). One project affected by this change would be archlinux-repro, as the mirror and remote directory for the bootstrap image are currently only overridable via BOOTSTRAPMIRROR. While technically this suffices, the directory structure will soon be more explicit (in regards to per-type artifact directory structure).

To make this a little more flexible I propose to decouple the mirror and the directory from one another. A similar approach has been chosen for releng (which relies on the iso release artifact).

This ticket is here to track the progress for this change for archlinux-repro and to propose an implementation for this situation.

Make it possible to configure the following bash variables:

  readonly build_directory=/var/lib/repro
  readonly config_dir=/home/fox/Git/prosjekter/Bash/devtools-repro
  readonly bootstrap_mirror=https://mirror.archlinux.no/iso/latest

Features:

• Rebuild packages based on .BUILDINFO
• Build a package N times with different environments, compare builds
• Config directory with profiles/config files to change the build environment

Commands could be:
repro - Build package two times and compare (default)
<reposetory>-repro extra-repro testing-repro etc. repository to use when building
repor check *.tar.gz - Rebuild package based on .BUILDINFO

Switches:

• --random-profiles=4

The downloading of the bootstrap image and potential verification of the signature should be done as user not as root.

Not sure how easy this is, but for a CI setup it would be nice to know why reproducing fails. A recent makepkg outputs error codes when the build fails which make it easy to identify the build failure.

See man makepkg ERRORS section for the possible codes, which should be passed out as output of repro or only a subset.

I'm trying to run repro multiple times (giving each their own $BUILDDIRECTORY) for mass rebuilds but I ran into issues with:

Failed to allocate scope: Unit root.scope already exists

It seems this is because repro builds in /var/lib/repro/root, with root then becoming the machine name in nspawn, which has to be unique.

@coderobe

We should fetch the keyring, or have a list of keys we need somewhere so we can recv them.

Since this package is used exclusively on systems where makepkg/pacman is available, it makes sense to use the existing library facilities rather than duplicate code. In particular:

https://github.com/Foxboron/devtools-repro/blob/master/repro.in#L44-L89

is available in /usr/share/libmakepkg/util/util.sh

As galera uses rsync in source=() and devtools depends on rsync and downloads outside of the build chroot the build succeeds. However with repro that won't work:

==> Retrieving sources...
==> ERROR: The download program rsync is not installed.
  -> Downloading galera-26.4.4.tar.gz...
/usr/share/makepkg/source/file.sh: line 72: $'\E[1m': command not found
==> ERROR: Failure while downloading rsync://rsync.osuosl.org/mariadb/mariadb-10.4.13/galera-26.4.4/src/galera-26.4.4.tar.gz
    Aborting...
  -> Delete snapshot for galera_14838...

Create a snapshot of '/mnt/msata/repro/root' in '/mnt/msata/repro/build1/root'
ERROR: cannot snapshot '/mnt/msata/repro/root': File exists

This probably needs a bash trap because the build might fail or the script might fail and we would like to clean up nicely afterwards.

BUILDINFO contains the hash of the PKGBULD which was use to build the package, compare this two the fetched PKGBUILD by asp and stop the build if they don't match up.

Using unshare -n we can disable internet access in nspawn. build_package needs to be updated to first install dependencies and download the source, then unshare network access and build the package.

Rebuilding ant shows some GPG errors and systemd-nspawn exit's. No clear error is however shown when repro exits.

error: 'cache/java-environment-common-3-1-any.pkg.tar.xz': invalid or corrupted package (PGP signature)
error: 'cache/java-runtime-common-3-1-any.pkg.tar.xz': invalid or corrupted package (PGP signature)
warning: downgrading package json-c (0.14-2 => 0.13.1-2)

I noticed this error:

(13/21) Warn about old perl modules
(14/21) Updating fontconfig cache...
(15/21) Probing GDK-Pixbuf loader modules...
(16/21) Updating GIO module cache...
(17/21) Compiling GSettings XML schema files...
(18/21) Probing GTK3 input method modules...
(19/21) Updating icon theme caches...
(20/21) Updating the info directory file...
(21/21) Updating the desktop file MIME type cache...
Installing devtools from https://archive.archlinux.org/packages/d/devtools-20220207-2-any.pkg.tar.zst
:: Retrieving packages...
 devtools-20220207-2-any.pkg.tar.zst failed to download
error: failed retrieving file 'devtools-20220207-2-any.pkg.tar.zst' from archive.archlinux.org : Connection timeout after 10000 ms
warning: failed to retrieve some files
cp: cannot stat '/usr/share/devtools/makepkg-x86_64.conf': No such file or directory
%

The rebuild failing due to the connection timeout is expected, but it seems the error was ignored and the script continued running (attempting to copy a file).

This is because the code is running in a nspawn subshell that doesn't use -x:

I tried reproducing a package that has been built by OBS which uses /usr/src/packages/BUILD as the $BUILDDIR. This failed with

==> ERROR: Failed to create the directory $BUILDDIR (/usr/src/packages/BUILD).
    Aborting...

The build functions are very similar, they should become a bash function, cause we love less loc.

We could probably base everything off the devtools /var/lib/build/root instead of doing our own thing. Helps us save diskspace and could be faster.

Currently I still use my own patch since stuff is in a different location for me and my / is rather small.

 repro.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/repro.sh b/repro.sh
index 30628d6..23c3130 100755
--- a/repro.sh
+++ b/repro.sh
@@ -4,11 +4,11 @@
 
 set -eE -o pipefail
 
-readonly build_directory=/var/lib/repro
+readonly build_directory=/mnt/msata/repro
 readonly bootstrap_mirror=https://mirror.archlinux.no/iso/latest
 readonly bootstrap_img=archlinux-bootstrap-"$(date +%Y.%m)".01-"$(uname -m)".tar.gz
 
-readonly config_dir=/home/fox/Git/prosjekter/Bash/devtools-repro
+readonly config_dir=/home/jelle/projects/devtools-repro
 
 pacman_conf=$config_dir/pacman.conf
 makepkg_conf=$config_dir/makepkg.conf

Default multilib is not enabled, which makes reproducing qsopcast impossible.

The rebuild for glib2 is currently failing due to certificate issues:

==> Retrieving sources...
  -> Cloning glib git repo...
Cloning into bare repository '/startdir/glib'...
fatal: unable to access 'https://gitlab.gnome.org/GNOME/glib.git/': error setting certificate verify locations:  CAfile: /etc/ssl/certs/ca-certificates.crt CApath: none
==> ERROR: Failure while downloading glib git repo

earlier in the logs there are some error messages that are likely related:

(1/4) Rebuilding certificate stores...
trust: error while loading shared libraries: libffi.so.7: cannot open shared object file: No such file or directory
trust: error while loading shared libraries: libffi.so.7: cannot open shared object file: No such file or directory
trust: error while loading shared libraries: libffi.so.7: cannot open shared object file: No such file or directory
trust: error while loading shared libraries: libffi.so.7: cannot open shared object file: No such file or directory
trust: error while loading shared libraries: libffi.so.7: cannot open shared object file: No such file or directory
trust: error while loading shared libraries: libffi.so.7: cannot open shared object file: No such file or directory
trust: error while loading shared libraries: libffi.so.7: cannot open shared object file: No such file or directory
(2/4) Warn about old perl modules

I'm not sure how and why this happens, I'm wondering if this a binary from the host system trying to load a library that doesn't exist inside the container.

Either we read the mirror from /etc/pacman.d/mirrorlist or use pacconf and use this as this is what the user expects. On non-arch systems we could default to https://www.archlinux.org/mirrors/kernel.org/

The codestyles differ a bit so we should try stick to one style. I think buildinfo is a good starting ground as most of it is commented and stuff.

@coderobe could probably layout some basics.

It would be neat to directly run diffoscope if the shasum's don't match

If you are in a directory which contains no PKGBUILD, a chroot will be prepared and then an error message is shown. The flow should be changed to first check if everything is available and then prepare chroots.

https://ptpb.pw/-cxP/sh

zlib has an epoch, which seems to make repro check zlib-1:1.2.11-3-x86_64.pkg.tar.xz not find the PKGBUILD.

Currently running repro while another repro process is running causes issues because the 2nd process fails in create_snapshot but still runs remove_snapshot afterwards while the other process is running.

Preferably the 1st process would create a lock and the 2nd process would wait until it can acquire the lock.

This tool currently can detect reproducible build issue, but it does not reproducible packages from our repo yet. For that it requires either a BUILDINFO (or extract from pkg.tar.xz) and a PKGBUILD.

This means, we need to create a chroot:

• Install all old packages (installed =). pacman -U -r ? Can't use daily repo since packages can be updated twice a day!
• Modify makepkg.conf for possible passed variables (options/buildenv)
• Set builddir
• Set builddate

When not being run directly on a PKGBUILD, repro runs a short script inside an ephemeral container based on the downloaded chroot. This script involves running pacman -S asp devtools --noconfirm to install tools required to download and validate the PKGBUILD file, which in turn will fail if pacman cannot take the lock on its database.

Early in it's run, repro will also try and update the chroot by running pacman -Syu --noconfirm inside it, which also takes the pacman database lock. In order to guard against this possibility, repro locks the file root.lock before doing this, and skips this step if it's already locked. However, the step above does not try and take this lock, so two instances of repro can try and run these two steps simultaneously.

The fact that the PKGBUILD step is run in an ephemeral container prevents it from damaging the update step, but not vice versa. If the snapshot used by the ephemeral container is taken while the update is in progress, it will include the lock file, and the PKGBUILD step will fail.

This can be seen by doing something like:

for i in {1..20}
do
    CACHEDIR=/path/to/cache repro ./acl-2.2.53-3-x86_64.pkg.tar.zst 2>output-${i} >output-${i} &
    sleep 1
done

Most of the spawned repro instances will fail due to this issue.

Taking a shared lock on root.lock for this step (lines 301-318 in repro.in) solves this issue, and will also prevent issues that may arise from operating in a chroot that is mid-upgrade. This should also be done for the other ephemeral containers used to set up the build chroot, though I haven't actually seen any issues from this yet, probably because chroot updates are relatively rare.

Also related to this lock, when setting up the chroot for the first time, repro attempts to take the lock, but doesn't check if this operation succeeds (line 215). Thus, multiple instances of repro can be trying to install the chroot at the same time. The correct order of operations should be:

if [ ! -d "$BUILDDIRECTORY"/root ]; then
    # block on the lock file
    nlock 9 "$BUILDDIRECTORY"/root.lock
    if [ -d "$BUILDDIRECTORY"/root ]; then
        # chroot was created by someone else, don't need to do anything
        unlock 9
    else
        # set up chroot
    fi
fi

I recently set up a rebuilderd instance running on archlinux core, and discovered 54 packages fail to build because one or more of the packages in their build environment fails the GPG signature check. All the packages failing validation have valid signatures from the perspective of the current archlinux-keyring package, but in the old archlinux-keyring packages used in the builds, the signing keys have expired.

Specifically, the following two keys:
60411304C09D36628340EEFFCEB167EFB5722BD6 (eschwartz) expired on 2020-10-17, extended to 2022-07-10 in keyring version 20200817
E550FFFFD485E264A1717E30901C1C320EB0D45D (demize) expired on 2020-12-10, extended to 2022-02-03 in keyring version 20200422

Signed the following five packages:
asciidoc 9.0.2-1 (eschwartz)
autoconf-archive 1:2019.01.06-1 (eschwartz)
ncompress 4.2.4.6-1 (eschwartz)
pkgconf 1.6.3-2 (demize)
python2-typing 3.6.6-1 (demize)

Which are used along with pre-extension archlinux-keyring versions in the builds of the following packages:
automake 1.16.2-3
b43-fwcutter 019-3
base 2-2
ca-certificates 20181109-4
ca-certificates-utils 20181109-4
cracklib 2.9.7-2
crda 4.14-3
cronie 1.5.5-1
db 5.3.28-5
dbus 1.12.20-1
dbus-docs 1.12.20-1
diffutils 3.7-3
ding-libs 0.6.1-3
dmraid 1.0.0.rc16.3-12
dosfstools 4.1-3
findutils 4.7.0-2
flex 2.6.4-3
gdbm 1.18.1-3
gzip 1.10-3
hdparm 9.58-3
ifenslave 1.1.0-11
inetutils 1.9.4-8
ipw2100-fw 1.3-10
ipw2200-fw 3.1-8
jfsutils 1.1.15-7
ldns 1.7.1-2
libaio 0.3.112-2
libgssglue 0.4-4
libmnl 1.0.4-3
libmpc 1.1.0-2
libnl 3.5.0-2
libssh2 1.9.0-2
lzo 2.10-3
m4 1.4.18-3
minizip 1:1.2.11-4
mkinitcpio-nfs-utils 0.3-7
net-tools 1.60.20181103git-2
nilfs-utils 2.2.8-2
npth 1.6-2
patch 2.7.6-8
pinentry 1.1.0-5
pptpclient 1.10.0-2
procinfo-ng 2.0.304-8
psmisc 23.3-2
pth 2.0.7-7
reiserfsprogs 3.6.27-3
rpcbind 1.2.5-3
run-parts 4.8.6.1-2
sysfsutils 2.1.0-11
tar 1.32-3
vi 1:070224-4
which 2.21-5
wireless_tools 30.pre9-3
zlib 1:1.2.11-4

These packages don't appear broken on reproducible.archlinux.org because this issue only affects packages that were built before the developer in question updated their archlinux-keyring package to a version that includes the relevant key extension(s), that were rebuilt after the expiry. Assuming developers update reasonably frequently, that reproducible.archlinux.org rebuilds packages shortly after they are published, and that it doesn't rebuild packages after it gets a successful repro, all of these packages would have been checked within the remaining validity of the signing keys.

A related issue is that we ignore any key revocations that weren't in effect at the time the package was published. If you're hoping to get a chain of trust stretching from newer packages back to older packages, then the compromise of a signing key for an old package is a big problem. There's a compromise path like:

1. Steal developer signing key
2. Sign a compromised version of gcc at some old version which introduces a flaw into some other package
3. Push compromised gcc to package archive
4. Build compromised package using compromised gcc
5. Publish compromised package
6. The bad gcc is never checked, because it looks like an out-of-date package, and the new package reproduces because reproducers will use the bad gcc package, even if the signing key is later revoked

Always using the newest archlinux-keyring package would solve both of these issues, though it's still not clear to me how to recover from a compromised developer signing key.

repro check -l mypkg.pkg.tar generates build/mypkg.pkg.tar.xz, and of course hashes are different.

Probably low prio since all official packages use .pkg.tar.xz extension, but anyway.

For example, after running repro -ndf /var/cache/pacman/pkg/python-autobahn-20.6.2-1-x86_64.pkg.tar.zst, running the same command in the same terminal again results in

sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper

Apparently the terminal is no longer attached(?) to a pts

$ ps | grep zsh
  62675 ?        00:00:00 zsh

This starts from 7470fa8 if I understand it correctly. If I remove the call setsid with

diff --git a/repro.in b/repro.in
index 0fb6f15..b0ed006 100755
--- a/repro.in
+++ b/repro.in
@@ -208,7 +208,7 @@ install -d -o builduser -g builduser /pkgdest
 install -d -o builduser -g builduser /srcpkgdest
 install -d -o builduser -g builduser /build
 __END__
-    exec_nspawn "$build" $args setsid -f -c -w sudo -iu builduser bash -c ". /etc/profile; cd /startdir; SOURCE_DATE_EPOCH=$SOURCE_DATE_EPOCH makepkg -sc --noconfirm --skippgpcheck $cmds"
+    exec_nspawn "$build" $args sudo -iu builduser bash -c ". /etc/profile; cd /startdir; SOURCE_DATE_EPOCH=$SOURCE_DATE_EPOCH makepkg -sc --noconfirm --skippgpcheck $cmds"
     mkdir -p "./build"
     for pkgfile in "$BUILDDIRECTORY/$build"/pkgdest/*; do
         mv "$pkgfile" "./build/"

Then running repro multiple times in a terminal works fine. Is there a reason to keep terminal capabilities during the build? The current behavior is quite inconvenient for a modify-and-test workflow.

I'm not sure if this is intentional, due to makepkg or due to nspawn. Having all cores available would be great when rebuilding core since it contains multiple gcc packages. :)

Currently the tool sets up a build env but does not verify if it's 100% correct, this can be done by verifying the installed packages against the BUILDINFO required installed packages.

On our test builder we also vary:

• hostname
• umask

Not yet

• domain name
• buildpath
• env PATH
• cpu type = possible to emulate? / cores?
• env USER
• uid
• gid
• filesystem - disorderfs? Better would be xfs/ext4
• - shell - Not relevant for Arch, only option zsh?

Security is important m'kay.