Giter Club home page Giter Club logo

repose's Introduction

repose

Travis CI Status Coverity Scan Build Status

Owning more than one Archlinux machine, I operate my own repository to distribute customized and/or extra packages between the various machines. repo-add, the provided tool for repository management, is frustratingly limited. Updating the repository after building a series of packages quickly turned into a slow monstrous bash script: either I had to have rather complex logic to figure out which packages are new, or I had to do the expensive operation of rebuilding the repository each time. Surly, though, this was something that could be automated.

repose is an Archlinux repository compiler.

Generally, it operates by building up two package caches: one that represents the contents of the database and another that represents the various packages sitting in the root directory of the database. Updating, then, is simply a sync operation operation between the two.

To sync, it takes advantage of several rules of Archlinux repositories to automate as much logic as possible:

  1. Repositories typically only hold one version of a package (and we're interesting in the newest version).
  2. Repositories typically only hold only one architecture.
  3. Repositories and packages are expected to be in the same directory.

Updating/Removing

Most simplistically:

repose -z foo
  1. Parse the contents of foo.db if it exists.
  2. If we find a new package in the database's folder, add it to the database.
  3. If we can't find a corresponding package to a database entry, remove it from the database.
  4. Write out an updated database.

To explicitly remove a package:

repose -zd foo [pkgs]

Removes the specified packages from the database.

To generate a complementary foo.files file, add the -f flags

repose -zf foo

Globbing

Its possible to use globbing. repose uses the following logic for finding/filtering packages:

  1. Does it match the package's filename
  2. Does it match the package's name
  3. Does it glob pkgname-pkgver

This allows for operations like:

Add latest detected version systemd to a repo:

repose foo systemd

Add a specific version of systemd to a repo:

repose foo systemd-209-1

Drop all git packages from the repo:

repose -zd foo '*-git-*'
     __
    '. \
     '- \
      / /_         .---.
     / | \\,.\/--.//    )
     |  \//        )/  /
      \  ' ^ ^    /    )____.----..  6
       '.____.    .___/            \._)
          .\/.                      )
           '\                       /
           _/ \/    ).        )    (
          /#  .!    |        /\    /
          \  C// #  /'-----''/ #  /
       .   'C/ |    |    |   |    |mrf  ,
       \), .. .'OOO-'. ..'OOO'OOO-'. ..\(,

repose's People

Contributors

buhman avatar celti avatar falconindy avatar holomorph avatar kaisforza avatar kyrias avatar numerical avatar vodik avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

repose's Issues

Packages in subdirectories

repoman should not be adding packages from subdirectories, only things from the current directory specified. I don't believe that pacman handles things well when there are multiple directory layers:

:: Synchronizing package databases...
 WST420 is up to date
 WST420-debug is up to date
 repoman is up to date
 gtmanfred is up to date
 testing is up to date
 core is up to date
 extra is up to date
 community-testing is up to date
 community is up to date
 multilib-testing is up to date
 multilib is up to date
:: Starting full system upgrade...
resolving dependencies...
looking for inter-conflicts...
warning: insufficient columns available for table display

Packages (2): xcb-util-wm-0.3.9-1  i3-git-4.5.1.r57.gdc522b6-1

Total Download Size:    0.24 MiB
Total Installed Size:   0.77 MiB

:: Proceed with installation? [Y/n] :: Retrieving packages ...
error: failed retrieving file 'i3-git-4.5.1.r57.gdc522b6-1-x86_64.pkg.tar.gz' from disk : Couldn't open file /home/wgiokas/.cache/pacman/pkg-bu/i3-git-4.5.1.r57.gdc522b6-1-x86_64.pkg.tar.gz
warning: failed to retrieve some files
error: failed to commit transaction (download library error)
Errors occurred, no packages were upgraded.

i3-git is in a subdirectory here.

Delta support

It would be great if repose could also generate or at least add delta files to the database.

Segfault

I keep getting segfaults using repose.
Once a segfault has occured I have to rebuild the database entirely before I can continue again.

           PID: 8307 (repose)
           UID: 1000 (daurnimator)
           GID: 100 (users)
        Signal: 11 (SEGV)
     Timestamp: Tue 2017-06-06 14:13:18 AEST (2min 23s ago)
  Command Line: repose -vf custom db
    Executable: /usr/bin/repose
 Control Group: /user.slice/user-1000.slice/session-c2.scope
          Unit: session-c2.scope
         Slice: user-1000.slice
       Session: c2
     Owner UID: 1000 (daurnimator)
       Boot ID: d59b93124aef4c31aa38599df44209bb
    Machine ID: b8ea57b6dcf1481ba8900c42c12a4f82
      Hostname: daurn-m73
       Storage: /var/lib/systemd/coredump/core.repose.1000.d59b93124aef4c31aa38599df44209bb.8307.1496722398000000000000.lz4
       Message: Process 8307 (repose) of user 1000 dumped core.
                
                Stack trace of thread 8307:
                #0  0x00000000004032d5 n/a (repose)
                #1  0x00007ffba8cf443a __libc_start_main (libc.so.6)
                #2  0x0000000000403c1a n/a (repose)

_repose is missing

I tried adapting the old zsh completions to the new repose

#compdef repose

_arguments -s \
  {-h,--help}'[display this help and exit]' \
  {-v,--version}'[display version]' \
  {-f,--files}'[generate complementing files database]' \
  {-d,--drop}'[drop package from database]' \
  {-r,--root=-}'[repository root directory]' \
  {-p,--pool=-}'[set the pool to find packages in it]:pool:_directories' \
  {-a,--arch=-}'[the primary architecture of the database]:arch:(i686 x86_64)$
  {-j,--bzip2}'[compress the database with bzip2]' \
  {-J,--xz}'[compress the database with xz]' \
  {-z,--gzip}'[compress the database with gzip]' \
  {-Z,--compress}'[compress the database with LZ]' \
  '--rebuild[force rebuild the repo]' \
  '1:database:_files -g "*.db*~(.,@)"' \
  '*::package files:_files -g "*.pkg.tar*~(.,@)"'

[Question] Remove dependencies

Lets say, I use PKGDEST in /etc/makepkg.conf do export an AUR package with its dependencies to a directory and have repose to build a repo from it.
And later I want remove this package again.
Do you have a convenient way, to purge its dependencies as well?
Or is about removing every dependency manually.
I currently use a self-written python script to achieve something similar, but it has the disadvantage of having every package in repo installed.

Add a manifest file

Add support for defining a manifest file. A manifest is a whitelist describing which packages that should be added to a repository, rather than just adding everything.

"Failed to parse desc" with repo-add created databases

repo-add writes both %MD5SUM% and %SHA256SUM%. As a result repose -l foo fails:

$ repose -l lxqt
repose: failed to parse desc for compton-conf-git-0.3.0.3.g34a7cb6-1/desc

Here's a dump of the relevant desc file:

%FILENAME%
compton-conf-git-0.3.0.3.g34a7cb6-1-x86_64.pkg.tar.xz

%NAME%
compton-conf-git

%VERSION%
0.3.0.3.g34a7cb6-1

%DESC%
A graphical configuration tool for Compton X composite manager. Development version.

%GROUPS%
lxqt

%CSIZE%
38552

%ISIZE%
205824

%MD5SUM%
d955c56d020f370bf5c5ed44ab9f3ac4

%SHA256SUM%
3e7805671c6a55c597d31a23514322c696979599472b6d731df1687e1d0ff9a5

%URL%
https://github.com/lxde/compton-conf

%LICENSE%
LGPL2.1

%ARCH%
x86_64

%BUILDDATE%
1508864967

%PACKAGER%
Unknown Packager

%CONFLICTS%
compton-conf

%PROVIDES%
compton-conf=0.3.0.3.g34a7cb6

%DEPENDS%
qt5-base
libconfig

%MAKEDEPENDS%
cmake
git
qt5-tools
lxqt-build-tools-git

And I got a quick fix for this:

diff --git a/src/desc.rl b/src/desc.rl
index a60bf91..0bc4492 100644
--- a/src/desc.rl
+++ b/src/desc.rl
@@ -31,6 +31,7 @@
            | '%GROUPS%'       %{ parser->entry = PKG_GROUPS; }
            | '%CSIZE%'        %{ parser->entry = PKG_CSIZE; }
            | '%ISIZE%'        %{ parser->entry = PKG_ISIZE; }
+           | '%MD5SUM%'       %{ parser->entry = PKG_MD5SUM; }
            | '%SHA256SUM%'    %{ parser->entry = PKG_SHA256SUM; }
            | '%PGPSIG%'       %{ parser->entry = PKG_PGPSIG; }
            | '%URL%'          %{ parser->entry = PKG_URL; }
diff --git a/src/package.h b/src/package.h
index 7edd01c..040a860 100644
--- a/src/package.h
+++ b/src/package.h
@@ -16,6 +16,7 @@ enum pkg_entry {
     PKG_GROUPS,
     PKG_CSIZE,
     PKG_ISIZE,
+    PKG_MD5SUM,
     PKG_SHA256SUM,
     PKG_PGPSIG,
     PKG_URL,

Build error

Under i686 architecture:
'''
==> Making package: repose 5-1 (So 10. Apr 20:17:52 CEST 2016)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Retrieving sources...
-> Found 5.tar.gz
==> Validating source files with sha1sums...
5.tar.gz ... Passed
==> Extracting sources...
-> Extracting 5.tar.gz with bsdtar
==> Removing existing $pkgdir/ directory...
==> Starting build()...
make: Entering directory '/home/haawda/paketierung/repose/src/repose-5'
cc -std=c11 -g -Wall -Wextra -pedantic -Wshadow -Wpointer-arith -Wcast-qual -Wstrict-prototypes -Wmissing-prototypes -Wno-missing-field-initializers -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -DREPOSE_VERSION="2" -march=i686 -mtune=generic -O2 -pipe -fstack-protector-strong -D_FORTIFY_SOURCE=2 -c -o package.o src/package.c
In file included from src/package.c:14:0:
src/package.c: In function 'pkginfo_assignment':
src/util.h:36:36: error: '_Generic' selector of type 'unsigned int' is not compatible with any association
#define fromstr(str, out) _Generic((out), long: xstrtol, unsigned long: xstrtoul)(str, out)
^
src/package.c:36:9: note: in expansion of macro 'fromstr'
fromstr(value, &pkg->isize);
^
: recipe for target 'package.o' failed
make: *
* [package.o] Error 1
make: Leaving directory '/home/haawda/paketierung/repose/src/repose-5'
==> ERROR: A failure occurred in build().
Aborting...
'''

Missing depends when PKGBUILD includes backup

PKGBUILD

pkgname=example
pkgver=1
pkgrel=1
pkgdesc='missing depends'
url='https://example.org'
license=('custom')
arch=('x86_64')
depends=('bash')
backup=('etc/example/conf')

package() {
	echo test > conf
	install -Dm644 -t "$pkgdir/etc/example" conf
}

file size of depends 0:

$ makepkg
...
$ repose foo
$ tar tvf foo.db example-1-1/depends
-rw-r--r-- repose/repose     0 2017-01-08 23:41 example-1-1/depends

After remove backup:

$ makepkg
...
$ repose foo
$ tar tvf foo.db example-1-1/depends
-rw-r--r-- repose/repose    16 2017-01-08 23:44 example-1-1/depends
$ tar xOf foo.db example-1-1/depends
%DEPENDS%
bash

$
$ repose --version 
repose 7

Link signatures from pool directory to repository directory

The option should be available for signatures to be kept alongside packages at all times (as is done in the Arch mirrors). With the current configuration, if the repository is shared publicly but the pool is kept inaccessible, signatures are only available as part of the database and not when installing individual packages, e.g. with pacman -U https://repo.example.org/os/x86_64/package-1.0.0-1.pkg.tar.xz.

undocumented option: -l

repose -l lists the packages in a given database, together with their versions. While useful, this option is not documented in the man page.

performance issues after a certain amount of packages

I have a repo that contains 322 packages, some of which are large or contain many files. Output from repose -l: https://paste.xinu.at/Szqf/

The issue is that adding any package takes a long time:

[archie@thinkpad custom]$ time repose -dvf custom.db prelink-20130503-7-x86_64.pkg.tar 
dropping prelink
writing custom.db...
writing custom.files...

real    0m0.102s
user    0m0.070s
sys 0m0.020s
[archie@thinkpad custom]$ time repose -vf custom.db prelink-20130503-7-x86_64.pkg.tar 
adding prelink 20130503-7
writing custom.db...
writing custom.files...

real    0m16.820s
user    0m0.300s
sys 0m2.220s

Comparison to a database which was created with repo-add:

[archie@thinkpad custom]$ time repo-remove custom.db.tar prelink
==> Extracting database to a temporary location...
==> Extracting database to a temporary location...
==> Searching for package 'prelink'...
  -> Removing existing entry 'prelink-20130503-7'...
==> Creating updated database file 'custom.db.tar'

real    0m0.198s
user    0m0.060s
sys 0m0.050s
[archie@thinkpad custom]$ time repo-add custom.db.tar prelink-20130503-7-x86_64.pkg.tar 
==> Extracting database to a temporary location...
==> Extracting database to a temporary location...
==> Adding package 'prelink-20130503-7-x86_64.pkg.tar'
  -> Computing checksums...
  -> Creating 'desc' db entry...
  -> Creating 'files' db entry...
==> Creating updated database file 'custom.db.tar'

real    0m0.273s
user    0m0.110s
sys 0m0.050s

I can give you more detailed statistics if you want, but a 16s difference should be reproducable.

Ability to add packages to db via absolute path

Since GPG signing with both repose (#46) and makepkg (FS#49946) has issues as of writing, I've developed the following workarounds:

  1. Find the package name without extension from makepkg --packagelist
  2. Pass the name to find together with some glob patterns to find a package with valid archive extension
  3. Run gpg on the results as well as on the database

This is a rather fragile approach, since for example two packages with the same name but with a different .tar extension may be available in the package pool, or issues with gpg --batch when signature files from previous builds are existing.

An alternative is the following:

  1. Point PKGDEST to a temporary directory (mktemp) in /var/tmp; assume this directory only contains the current build products
  2. Sign the files in that directory
  3. Generate the database in the repository root, adding packages from the temporary directory
  4. Sign the database in the temporary directory
  5. Move over all files from the temporary directory (packages, package signatures, database signature) to the repository root. This also allows mv --backup to keep old files when rebuilding packages

With repo-add you'd do something like repo-add <path_to_db> <path_to_files_in_tmp> for the third step. With repose I'm not sure how to achieve this; you could point --pool to the temporary directory, but then previously built packages would be removed from the database. A possible workaround is to create symbolic links for existing packages in the temporary directory, but as with the first approach some guesses are made.

If there's a better general approach than either of the above, I'm open to suggestions.

Half written databases crash repose.

Steps to reproduce:

  1. Half way through writing a large files database, kill with SIGINT.
  2. Rerun repose, it (may) segfaults while reading the half written database.

False positives which causes an update

Recently after switching to version 7.1 (previously still using 5, because I missed the transition to [community]) I noticed some updates of packages that weren't at all updated.

A small log.
hekate update is an alias for

repose -Jfv --pool=/var/cache/pacman/hekate --root=/var/cache/pacman/hekate hekate.db
$ hekate_update 
updating xfce4-settings 4.12.1-1 [newer build]          <-- didn't rebuild it, or whatever
updating flif-git r690.6bdc5ab-1 => r698.221a503-1 <-- expected result
writing hekate.db...
writing hekate.files...
$ hekate_update
updating xdg-utils-mimeo 1.1.1-5 [newer build]          <-- didn't rebuild it, or whatever
writing hekate.db...
writing hekate.files...
$ hekate_update
updating volctl 0.4.r4.g4c6a59e-1 [newer build]          <-- didn't rebuild it, or whatever
updating linux-Hekate-headers 4.9.6-1 [newer build]  <-- didn't rebuild it, or whatever
writing hekate.db...
writing hekate.files...
$ hekate_update
updating xfce4-settings 4.12.1-1 [newer build]          <-- didn't rebuild it, or whatever
writing hekate.db...
writing hekate.files...
$ hekate_update
updating volctl 0.4.r4.g4c6a59e-1 [newer build]          <-- didn't rebuild it, or whatever
updating linux-Hekate-headers 4.9.6-1 [newer build]   <-- didn't rebuild it, or whatever
writing hekate.db...
writing hekate.files...
$ hekate_update
updating xfce4-settings 4.12.1-1 [newer build]
writing hekate.db...
writing hekate.files...
$ hekate_update
updating volctl 0.4.r4.g4c6a59e-1 [newer build]
updating linux-Hekate-headers 4.9.6-1 [newer build]
writing hekate.db...
writing hekate.files...
$ hekate_update
updating xfce4-settings 4.12.1-1 [newer build]
writing hekate.db...
writing hekate.files...

And so on.
But the timestamp of the files didn't change, nor has their checksums.
And most of the time it stops after a while doing this. But currently it switches everytime between the packages.

And idea what could be the cause? Or how I could investigate it further (bisecting it would propably be a possibility, if it could be assumed it is an issue with repose)

Create empty db

If database doesn't exist it is only created "lazily" when adding some package/s to it.
It would be nice if (when running in "add mode") it is always created even though it might be empty.

This would streamline initialization of repository (which might be empty) without need to resort to repo-add or creating empty archive manually.

extraneous newline at end of repose --elephant output

To maintain backward-compatibility with repo-elephant, repose must stop putting an extraneous newline at the end of repose-elephant(s):

% diff -u <(repo-elephant) <(repose --elephant) 
--- /proc/self/fd/11    2013-06-09 17:06:39.800401213 -0500
+++ /proc/self/fd/12    2013-06-09 17:06:39.800401213 -0500
@@ -5,3 +5,4 @@
   `\__)/__/'_\  / `
      //_|_|~|_|_|
      ^""'"' ""'"'
+
% diff -u <(repo-elephant) <(repose --elephant)
--- /proc/self/fd/11    2013-06-09 17:19:10.030396722 -0500
+++ /proc/self/fd/12    2013-06-09 17:19:10.030396722 -0500
@@ -13,3 +13,4 @@
           \  C// #  /'-----''/ #  /
        .   'C/ |    |    |   |    |mrf  ,
        \), .. .'OOO-'. ..'OOO'OOO-'. ..\(,
+

Preferred reactions

@vodik

In this case:

repose foo.db.tar.gz some.package.0.0-1.x86_64.tar.gz

If the repo doesn't exist currently repose does nothing.
Should it create a new repo?

updated package gets removed

$ PKGDEST=/srv/http/pkg makepkg --sign -f
<snip>
==> Finished making: pentadactyl-hg 1.0+261+e8b878e26ab1-1 (Thu Mar 13 08:02:11 EDT 2014)

$ ls /srv/http/pkg/pent*
/srv/http/pkg/pentadactyl-hg-1.0+260+15ef5bf0fb2f-1-x86_64.pkg.tar
/srv/http/pkg/pentadactyl-hg-1.0+260+15ef5bf0fb2f-1-x86_64.pkg.tar.sig
/srv/http/pkg/pentadactyl-hg-1.0+261+e8b878e26ab1-1-x86_64.pkg.tar
/srv/http/pkg/pentadactyl-hg-1.0+261+e8b878e26ab1-1-x86_64.pkg.tar.sig

$ repose -r /srv/http/pkg -zf depot
updating pentadactyl-hg 1.0+261+e8b878e26ab1-1 [newer build]
:: Writing databases to disk...
writing depot.db...
writing depot.files...
repo updated successfully

$ ls /srv/http/pkg/pent*
/srv/http/pkg/pentadactyl-hg-1.0+260+15ef5bf0fb2f-1-x86_64.pkg.tar
/srv/http/pkg/pentadactyl-hg-1.0+260+15ef5bf0fb2f-1-x86_64.pkg.tar.sig
/srv/http/pkg/pentadactyl-hg-1.0+261+e8b878e26ab1-1-x86_64.pkg.tar.sig

Doesn't work on xfs

Apparently failing to find packages on xfs, according to archlinux irc channel:

execve("/usr/bin/repose", ["repose", "-v", "home"], [/* 23 vars */]) = 0
...
open(".", O_RDONLY|O_DIRECTORY)         = 3
faccessat(3, "home.files", F_OK)        = -1 ENOENT (No such file or directory)
openat(3, "home.db", O_RDONLY)          = -1 ENOENT (No such file or directory)
dup(3)                                  = 4
lseek(4, 0, SEEK_SET)                   = 0
fstat(4, {st_mode=S_IFDIR|0755, st_size=120, ...}) = 0
fcntl(4, F_GETFL)                       = 0x18000 (flags O_RDONLY|O_LARGEFILE|O_DIRECTORY)
fcntl(4, F_SETFD, FD_CLOEXEC)           = 0
getdents(4, /* 5 entries */, 32768)     = 216
getdents(4, /* 0 entries */, 32768)     = 0
lseek(4, 0, SEEK_SET)                   = 0
getdents(4, /* 5 entries */, 32768)     = 216
getdents(4, /* 0 entries */, 32768)     = 0
close(4)                                = 0
fstat(1, {st_mode=S_IFREG|0644, st_size=17904, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1ecf418000
write(1, "repo empty!\n", 12repo empty!
)           = 12
exit_group(0)                           = ?
+++ exited with 0 +++

Use timestamps for detecting "new" packages and updating the database.

It is common for UNIX applications to rely on file timestamps for determining, which files are "new" and which are "old". For example make, git and rsync. I think, repose should also use this mechanism more actively.

In particular, I have 2 suggestions:

  1. Currently, if the .db and .files are already up to date, repose just prints a message and exits. I think, in this case, the files should also be touched (their timestamp should be updated). That way, you can use make to build the .db file. In the current situation, if make thinks, that the .db file is "old" and repose thinks, that it is up to date, its timestamp will never be updated and so make will still think, that the database is outdated even after "building" it.

  2. This might be a bug, or my misunderstanding of how repose determines, which packages are "new" (step 2 in README "If we find a new package in the database's folder, add it to the database.").

    Currently, if I build a packagename.pkg.xz via makepkg, then run repose and then build it again, increasing the pkgver or pkgrel, then makepkg will report, that the database is up-to-date, even though the packagename.pkg.xz file contains a newer version of the package.

    Maybe, I am misunderstanding the "intended" way to use repose. If my use case is valid, I would suggest also considering a .pkg.xz as "new" if it has a timestamp, that is newer than the timestamp of the .db file.

New release?

There's been a good number of commits since 9d3c4dd, and things have stabilized for a while. Perhaps it's time for a new release? 👍

Cheers

Compilation Failures with -g (clang and gcc)

Using make CFLAGS='-g' CC=clang

clang -g   -c -o repoman.o repoman.c
repoman.c:175:46: error: use of undeclared identifier 'FNM_CASEFOLD'
            if (fnmatch(filter, p->fts_path, FNM_CASEFOLD) == 0)
                                             ^
repoman.c:367:43: error: use of undeclared identifier 'program_invocation_short_name'
    fprintf(out, "usage: %s [options]\n", program_invocation_short_name);
                                          ^
repoman.c:406:31: error: use of undeclared identifier 'program_invocation_short_name'
            printf("%s %s\n", program_invocation_short_name, "devel");
                              ^
3 errors generated.
make: *** [repoman.o] Error 1

Using make CFLAGS='-g'

cc -g   -c -o repoman.o repoman.c
repoman.c: In function ‘find_packages’:
repoman.c:175:46: error: ‘FNM_CASEFOLD’ undeclared (first use in this function)
             if (fnmatch(filter, p->fts_path, FNM_CASEFOLD) == 0)
                                              ^
repoman.c:175:46: note: each undeclared identifier is reported only once for each function it appears in
repoman.c: In function ‘usage’:
repoman.c:367:43: error: ‘program_invocation_short_name’ undeclared (first use in this function)
     fprintf(out, "usage: %s [options]\n", program_invocation_short_name);
                                           ^
repoman.c: In function ‘main’:
repoman.c:406:31: error: ‘program_invocation_short_name’ undeclared (first use in this function)
             printf("%s %s\n", program_invocation_short_name, "devel");
                               ^
make: *** [repoman.o] Error 1

Rewrite in Rust

I've spent a day and currently at about 20% - can load the database and read through it, so looks like its going to be mostly smooth sailing. The current design of repose is already rather compatible with how Rust would want things architected.

Problems I'm hitting which is making me thing this is the right path:

  • Enhancing the parsers to deal with possible regressions (from #44). I'm not comfortable anymore with the longevity of ragel
  • Enhancing the internal data structures to support new features (#29 and #43). Doesn't help that the new hashmap I was attempting to implement is already very very similar to how Rust implements theirs.
  • Creating a library (#30). There is some subtle memory corruption going on somewhere inside repose that, while it operates correctly, prevents me from putting in inside a library. I currently take advantage of the short/one-time usage of this tool and just let the os reap the pkgcache's memory. Now while this should be fixable with some digging/valgrind (or with replacing the hashmap, see above), I think I'd rather have something up on crates.io.

pick up added signatures

repose's ability to pick up added signatures does not seem to work as expected anymore. In what follows, I would have expected the second repose invocation to result in foo.db being updated

$ ls
rtorrent-0.9.4-1-x86_64.pkg.tar
$ repose -vzf foo
adding rtorrent 0.9.4-1
writing foo.db...
writing foo.files...
$ gpg --detach-sign rtorrent-0.9.4-1-x86_64.pkg.tar

You need a passphrase to unlock the secret key for
<snip>

$ ls
foo.db  foo.files  rtorrent-0.9.4-1-x86_64.pkg.tar  rtorrent-0.9.4-1-x86_64.pkg.tar.sig
$ repose -vzf foo
repo does not need updating
$ repose -V
repose 2-38-g86b48f5

Issue with -s | --sign param

I have been using repose as part of the aurutils package, and have a local repository which is signed with my private key, said key is also the default key for pacman and has been locally signed. I had to manually sign the db for the repository with gpg, as repose -s repo_name causes a segfault.

/path/to/repo % repose -s repo 
[1]    13349 segmentation fault (core dumped)  repose -s repo

gdb output is vague, I am not sure where else to look.

(gdb) r
Starting program: /usr/bin/repose
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
repose: incorrect number of arguments provided

pacman/makepkg 6.1.0 adds new field to PKGINFO

Hello,
After the update to pacman 6.1.0 and packages created with it, repose fails for me with repose: failed to parse PKGINFO on (...).
Quick look at the PKGINFO seems to show a new field xdata (xdata = pkgtype=pkg).
There may be also changed behaviour of provides fields as pacman gained the feature of autodeps ( Replace libdepends and libprovides with autodeps )
Would it be possible that repose get an update in that regard?
Or accept a PR if I or someone else get it working (in the hopes it is only an update like done with #47 )?

Best regards

Repose failed to parse

I'm getting several issues with packages failing like repose: failed to parse PKGINFO on sdl2_mixer-2.0.1-1-x86_64.pkg.tar.xz

these are from the official repos on Arch.

Pick up package signatures

Currently the repository is signed just fine, but package signatures are not picked up along with the package when adding things.

Character Set Handling

Having repository with package shaman-git-2.0.0_α_19_gd28fd8e-1-x86_64.pkg.tar.xz (notice the α in the name) and running repose -f custom resulting files db seems to be causing trouble with pkgfile. At first I've suspected pkgfile and reported issue there falconindy/pkgfile#24.

It turns out that instead probably repose should be adjusted. AFAIK repose uses libarchive which also provides bsdtar which is in turn used by repo-add. libarchive uses user's locale to decode filenames. Since explicit setlocale() is not done, C/POSIX locale is assumed resulting in archive with hdrcharset=BINARY, which seem to later trouble pkgfile. Interestingly bsdtar --list works OK.

Nevertheless, as pointed out in the issue linked above, to do proper character set handling following should be done close to beginning of the program:

setlocale(LC_ALL, "");

Relative --root and --pool corner case?

This happens when both --root and --pool are relative paths.

$ tree
.
├── pool
│   └── tmux-2.1-2-x86_64.pkg.tar.xz
└── root

$ repose -fz -r root -p pool custom

$ tree
.
├── pool
│   └── tmux-2.1-2-x86_64.pkg.tar.xz
└── root
    ├── custom.db
    ├── custom.files
    └── tmux-2.1-2-x86_64.pkg.tar.xz -> pool/tmux-2.1-2-x86_64.pkg.tar.xz

Notice that the link generated is "broken".

Parse error while updating repo

I've created a custom repo to hold packages from the AUR. Occasionally, an update will fail, e.g.:

$ repose -vf  custom                                 
repose: failed to parse files for jekyll-3.7.0-3/files

I'm not sure if it's the package or repose itself, jekyll is just an example.

Updating from scratch (using --rebuild) works, though this is time consuming.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.