Giter Club home page Giter Club logo

repoctl's Introduction

repoctl

The repoctl program helps you manage a local repository of Pacman packages (as found on Arch Linux and derivatives).

In the following video you can see a whirlwind tour of what repoctl can do for someone who is just starting out with local repositories: We will search for an extension for the pass tool and add it to a new local repository.

asciicast

This isn't all repoctl can do; veterans might find they use the status and update commands more for day-to-day managing of local repositories.

A look at the available commands may help give an overview:

  • add - Copy and add packages to the repository.
  • conf - Create, edit, or show the repoctl configuration.
  • down - Download and extract tarballs from AUR.
  • host - Host repository on a network.
  • list - List packages that belong to the managed repository.
  • query - Query package information from AUR.
  • remove - Remove and delete packages from the database.
  • reset - (Re-)create repository database.
  • search - Search for packages on AUR.
  • status - Show pending changes and packages that can be upgraded.
  • update - Update database in repository to match filesystem.

See the NEWS for the latest changes in repoctl!

Installation

The recommended method is to install the repoctl package from AUR, as this package installs other useful files, such as the completion scripts.

Alternatively, if you have Go installed:

git clone https://github.com/cassava/repoctl.git
cd repoctl
# Install repoctl to $GOPATH/bin, or specify -o OUTPUT
go install
# Get help on where to install completion files to:
repoctl completion --help

You may want to switch to the devel branch if you want the bleeding edge.

Basic Usage

Before you can use really use repoctl, you need to create a configuration file, but there's a lot you can do without any configuring.

  1. Search AUR:

    $ repoctl search tomb-
    aur/mediatomb-git 7ab7616-1 (2)
        Free UPnP/DLNA media server mediatomb
    aur/gtomb-git 0.7.1-3 (2)
        GUI wrapper for Tomb, the crypto undertaker
    aur/tomb-git 2.6.r7.g6f2ce59-1 (6)
        simple tool to manage encrypted storage
    aur/tomb-kdf-git 2.6.r7.g6f2ce59-1 (6)
        Crypto Undertaker extensions to improve password security
    aur/mediatomb-samsung-tv 0.12.1-12 (8)
        Free UPnP/DLNA media server with Samsung TV compatibility
    aur/tomb-kdf 2.7-2 (45)
        Crypto Undertaker extensions to improve password security
  2. Query specific packages on AUR:

    $ repoctl query tomb
    aur/tomb 2.7-2 (45)
        Name: tomb
        Version: 2.7-2
        Description: Crypto Undertaker, a simple tool to manage encrypted storage
        URL: https://www.dyne.org/software/tomb/
        Licenses: GPL3
        Dependencies: bc cryptsetup gnupg sudo zsh e2fsprogs inetutils
        Optional Dependencies:
            steghide
            dcfldd
            qrencode
            swish-e
        Snapshot URL: https://aur.archlinux.org/cgit/aur.git/snapshot/tomb.tar.gz
        Maintainer: parazyd
        Votes: 45
        Popularity: 0.355983
        First Submitted: 2011-04-15 17:20:00 +0200 CEST
        Last Updated: 2020-01-03 13:57:47 +0100 CET
        Out-Of-Date: false
  3. Download packages from AUR, including their dependencies:

    $ repoctl down -r pass-tomb
    Downloading: pass-tomb
    Downloading: tomb

Configuration

Before we can actually start managing a local repository, repoctl needs to know where it is. No one really enjoys working with configuration files, so repoctl will help you out a little here.

  1. Create a new configuration, with our repo in ~/pkgs:

    $ repoctl conf new ~/pkgs/sirius.db.tar.zst
    Writing new configuration file at: /home/you/.config/repoctl/config.toml
  2. Initialize the repository:

    $ repoctl reset
    Creating database: /home/ben/pkgs/sirius.db.tar.zst

Now you should be set to start adding packages to your repository.

If you want to fine-tune the configuration values or just see what's there, repoctl will show you your configuration (repoctl conf show) as well as launch you into it with your favorite editor (repoctl conf edit), as set in the environment variable EDITOR.

  1. Inspect your configuration.

    $ repoctl conf show
    Current configuration:
        columnate = false
        color = "auto"
        quiet = false
    
        current_profile = ""
        default_profile = "default"
    
        [profiles.default]
            repo = "/home/you/pkgs/sirius.db.tar.zst"
            add_params = []
            rm_params = []
            ignore_aur = []
            require_signature = false
            backup = false
            backup_dir = ""
            interactive = false
            pre_action = ""
            post_action = ""
  2. Edit your configuration:

    $ repoctl conf edit

Managing Your Repository

Now, we can add and manipulate packages in the specified local repository.

  1. Add packages to the repository:

    $ repoctl add <tab>
    $ repoctl add pass-extension-tail-1.2.0-1-any.pkg.tar.zst
    Copying and adding to repository: pass-extension-tail-1.2.0-1-any.pkg.tar.zst
    Adding package to database: /home/you/pkgs/pass-extension-tail-1.2.0-1-any.pkg.tar.zst

    If you installed the completion, you really should take advantage of it, unless of course you are automating the procedure.

  2. Remove packages from the repository:

    $ repoctl rm <tab>
    $ repoctl rm pass-extension-tail
    Removing package from database: pass-extension-tail
    Deleting: pass-extension-tail-1.2.0-1-any.pkg.tar.zst

    Yes, package names from your repository are also completed.

Managing Updates To Your Packages

Of course, the initial compilation and adding of packages isn't the trouble, it's keeping them all up-to-date. This is what repoctl was originally made for: to tell me which packages have been updated on AUR and get them for me.

  1. Show which packages have updates on AUR:

    $ repoctl status -a
    On repo sirius
    
        krop: upgrade(0.4.11-1 -> 0.6.0-1)
        spotify: upgrade(1.0.98.78-1 -> 1:1.1.10.546-4)
        tmuxinator: upgrade(0.8.1-1 -> 2.0.1-1)
        ttf-ms-win10: upgrade(10.0.14393-3 -> 10.0.18362.116-2)
        ttf-ms-win10-japanese: upgrade(10.0.14393-3 -> 10.0.18362.116-2)
        ttf-ms-win10-korean: upgrade(10.0.14393-3 -> 10.0.18362.116-2)
        ttf-ms-win10-other: upgrade(10.0.14393-3 -> 10.0.18362.116-2)
        ttf-ms-win10-sea: upgrade(10.0.14393-3 -> 10.0.18362.116-2)
        ttf-ms-win10-thai: upgrade(10.0.14393-3 -> 10.0.18362.116-2)
        ttf-ms-win10-zh_cn: upgrade(10.0.14393-3 -> 10.0.18362.116-2)
        ttf-ms-win10-zh_tw: upgrade(10.0.14393-3 -> 10.0.18362.116-2)

    You can't see it here, but this is all nicely colored in your terminal.

  2. Download all updated packages:

    $ repoctl down -u -o build-order.txt
    Downloading: tmuxinator
    Downloading: krop
    Downloading: python-poppler-qt5
    Downloading: ttf-ms-win10
    Downloading: ruby-xdg
    Downloading: ruby-erubis
    
    $ cat build-order.txt
    ruby-erubis
    ruby-xdg
    ttf-ms-win10
    python-poppler-qt5
    krop
    tmuxinator

What's the build-order.txt file for, you say? I'm glad you asked. Some packages, such as tmuxinator up there, have dependencies on other packages (in this case, ruby-xdg and ruby-erubis). If these packages are in AUR, then we need to fetch them too. This is what the -r (--recursive) flag is good for, and if we specify the -o flag (--order) it is implied.

We can use this list to our advantage, and with some Bash fu compile the whole lot of packages and add them to the repository:

#!/bin/bash
set -e
repoctl down -u -o build-order.txt
for pkg in $(cat build-order.txt); do
    (
        cd "$pkg"
        makepkg -cs
        repoctl add *.pkg.tar.zst
        cd ..
        rm -rf "$pkg"
    )
done

Tips and Tricks

  1. Using PKGDEST in /etc/makepkg.conf

    You can configure makepkg to put all generated packages into a directory of your choosing. If you want, you can set PKGDEST to your repository directory, and then just run repoctl update to do the rest.

  2. Auto-completion for everything!

    Since version 0.21, auto-completion depends strongly on the repoctl tool itself. This lets us do some pretty wild things, like query AUR, read your configuration, or even read the repository database specified in the profile you just added to the command-line invocation.

    Make sure you install the completions for your shell if you haven't done so yet. There is a hidden command for exporting the shell completion:

    $ repoctl completion
    ...
    
  3. Configuring multiple repositories

    Configuration profiles are supported since version 0.21. These let you have more than one profile that you can then choose at runtime.

    The important configuration settings are:

    default_profile = "default"
    
    [profiles.default]
      repo = "/home/you/pkgs/sirius.db.tar.zst"
    
    [profiles.release]
      repo = "/home/you/public/pkgs/sirius-release.db.tar.zst"
      require_signature = true
      backup = true
      backup_dir = "backup/"
    

    See the conf command for more information on this.

  4. Migrating your configuration file

    The configuration file has changed significantly since version 0.21 in order to support profiles. This means that some configuration values are deprecated and no longer supported and the format in general is different.

    Fear not! Not only is your old configuration auto-migrated, but you can make this migration permanent with the conf migrate command:

    $ repo conf migrate
    Backing up current configuration to: /home/ben/.config/repoctl/config.toml.bak.3
    Writing new configuration file to: /home/ben/.config/repoctl/config.toml
  5. Caching obsolete packages

    Sometimes you might want to hold on to the obsolete packages and leave them in the directory at the same time, and use a tool like paccache to manage them. You can easily enable this in your config:

    [profiles.default]
        backup = true
        backup_dir = ""

    Now, obsolete packages will be ignored. They will also be ignored when removing packages from the database.

  6. Packages on a remote filesystem

    If you have a super fast internet connection and want your packages on a remote server, you can try to get repoctl to play along with the pre_action and post_action options in the configuration file:

    [profiles.default]
        pre_action = "sshfs server:location ~/localmnt"
        post_action = "fusermount -u ~/localmnt"

    This will definitely break auto-completion and if some error happens, the post_action might not be executed, so I don't recommend this. Instead, it's much better to simply rsync your packages at the end.

Getting Help

These are not the only things that repoctl can do, to get a fuller picture, have a look at the help, which you can always get by using the --help flag or by running:

$ repoctl help [cmd]
[...]

Chances are good you might encounter errors or have a bright idea about how to improve repoctl. If you do, I would love to hear about it!

Have a look at the existing issues or create a new issue at GitHub.

Enjoy!

repoctl's People

Contributors

cassava avatar cyrinux avatar intelfx avatar justtne avatar maximbaz avatar potatoattack avatar sosiska avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

repoctl's Issues

Panic

After running command repoctl down tango-icon-theme got this:

panic: runtime error: makeslice: cap out of range

goroutine 1 [running]:
panic(0x889e00, 0xc8203f4600)
    /usr/lib/go/src/runtime/panic.go:464 +0x3e6
github.com/goulash/pacman/aur.ReadAll(0xc8200f49c0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/ben/devel/go/src/github.com/goulash/pacman/aur/aur.go:210 +0x39e
github.com/cassava/repoctl.(*Repo).ReadAUR(0xc8201042c0, 0xc82000e420, 0xc8200f49c0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/ben/devel/go/src/github.com/cassava/repoctl/read.go:83 +0x125
github.com/cassava/repoctl.(*Repo).Download(0xc8201042c0, 0xc82000e420, 0x0, 0x0, 0x7a0001, 0xc8200f49c0, 0x1, 0x1, 0x0, 0x0)
    /home/ben/devel/go/src/github.com/cassava/repoctl/down.go:30 +0xfe
main.glob.func2(0xd6be20, 0xc8200f49c0, 0x1, 0x1, 0x0, 0x0)
    /home/ben/devel/go/src/github.com/cassava/repoctl/cmd/repoctl/down.go:52 +0x2bf
github.com/spf13/cobra.(*Command).execute(0xd6be20, 0xc8200f4980, 0x1, 0x1, 0x0, 0x0)
    /home/ben/devel/go/src/github.com/spf13/cobra/command.go:565 +0x62c
github.com/spf13/cobra.(*Command).ExecuteC(0xd6c420, 0xd6be20, 0x0, 0x0)
    /home/ben/devel/go/src/github.com/spf13/cobra/command.go:656 +0x55c
github.com/spf13/cobra.(*Command).Execute(0xd6c420, 0x0, 0x0)
    /home/ben/devel/go/src/github.com/spf13/cobra/command.go:615 +0x2d
main.main()
    /home/ben/devel/go/src/github.com/cassava/repoctl/cmd/repoctl/main.go:91 +0x17c

It was working fine until my system crashed. After that it stopped working.

New switches for 'add'

--update, only add files not already present. This allows me to add *.pkg.tar.xz
--symlink, symlink files instead of copying. This keeps me from having two copies around
--srcinfo, add all current version filenames found in .SRCINFO

With symlink you'll need an autoclean that removes broken symlinks from the repo and db.

ignore missing packages in `repoctl down`

I would like to be able to use:

repoctl down (pacman -Qmq)

For acquiring all foreign packages installed on my system. However, if I have packages installed that are not in the AUR, I get an error. A warning would suffice, or at least an option to ignore the missing packages.

Recursive download doesn't handle package names that are provided

The recursive download feature has been available since 0.20, and here's an edge case that hasn't been dealt with yet.
Sometimes a package depends on a name that doesn't actually have a package, but is provided by another. This mostly happens when a package is renamed from one to another or can be fulfilled by multiple other packages.

Currently, we just dump a warning that we can't find the package:

$ repoctl down -r firefox56
warning: unknown package ttf-font
         required by: firefox56
warning: unknown package mime-types
         required by: firefox56
downloading: firefox56

When Pacman has to deal with these situation, it gives the user the choice between all possibilities.
We should probably try to do the same, unless the user provides a package that provides something that works.

I suspect the AUR interface might actually help us here, by showing us all results that provide the search term.

Use new AUR API

When goulash/pacman has been updated, use the new API here.

Implement --ignore flag

The configuration file lets us ignore packages (we should define what that means), but we can't do this on the command line yet.

'repoctl down -u' throwing panic and no packages are being downloaded

I'm not sure when it started and I'm a little lost:

Console log:

main-builder@aur-builder /t/lab2> repoctl down -u
panic: value method github.com/cassava/repoctl/vendor/github.com/goulash/pacman/aur.NotFoundError.Error called using nil *NotFoundError pointer

goroutine 1 [running]:
github.com/cassava/repoctl/vendor/github.com/goulash/pacman/aur.(*NotFoundError).Error(0x0, 0x3fe, 0x438)
        <autogenerated>:1 +0x71
github.com/cassava/repoctl.(*Repo).FindUpgrades(0xc0001ec0b0, 0xc00000c0a0, 0xc000179a30, 0x0, 0x1, 0xc00011bbd0, 0x5453f7, 0xc0001cc280, 0xc0001b81c0,
 0xe)
        /home/main-builder/pkgwork/src/src/github.com/cassava/repoctl/find.go:81 +0x129
main.glob..func2(0xb89b60, 0xc000179a30, 0x0, 0x1, 0x0, 0x0)
        /home/main-builder/pkgwork/src/src/github.com/cassava/repoctl/cmd/repoctl/down.go:60 +0x4b4
github.com/cassava/repoctl/vendor/github.com/spf13/cobra.(*Command).execute(0xb89b60, 0xc000179a10, 0x1, 0x1, 0xb89b60, 0xc000179a10)
        /home/main-builder/pkgwork/src/src/github.com/cassava/repoctl/vendor/github.com/spf13/cobra/command.go:746 +0x473
github.com/cassava/repoctl/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xb8a220, 0x879462, 0x22, 0x2c)
        /home/main-builder/pkgwork/src/src/github.com/cassava/repoctl/vendor/github.com/spf13/cobra/command.go:831 +0x2dc
github.com/cassava/repoctl/vendor/github.com/spf13/cobra.(*Command).Execute(0xb8a220, 0x0, 0x0)
        /home/main-builder/pkgwork/src/src/github.com/cassava/repoctl/vendor/github.com/spf13/cobra/command.go:784 +0x2b
main.main()
        /home/main-builder/pkgwork/src/src/github.com/cassava/repoctl/cmd/repoctl/main.go:129 +0xdf

Database is the one available at: http://lonewolf-builder.duckdns.org/chaotic-aur/x86_64
Dotfiles are these: https://github.com/PedroHLC/chaotic-aur/tree/master/home/main-builder/.config/repoctl
Note: using 0.18 release
EDIT: Tested and throwing the same output with latest master commit
Also: repoctl status shows Everything up-to-date., and repoctl status -a works great

Implement filter command

Currently, all filter criteria are connected by an implicit AND. Let us support OR statements.

While traditionally, | is used to signifiy or, we would like to do this in a way that is command-line friendly. Two possibilities come into mind: Accept statements like and and or, or use a comma , to signify another group.

repoctl filter a.n, a.m

Support multiple repositories

It should be possible to support multiple repositories via the config file.

Something like

[atlas]
repo = "/srv/abs/atlas.db.tar.gz"

[mercury]
repo = "/nfs/archlinux/mercury.db.tar.gz"

However, I do not want to introduce extra complexity unless it is necessary.
If you want this therefore, please comment!

magic number mismatch

I'm using repoctl-devel and build zstd packages both in my chroot and sometimes on my main system.

After adding a chroot package I've just got

error: invalid input: magic number mismatch.
(0x562a3fa76ae0,0xc000066850)

which deleted all my packages in the repo. Seems to be the standard reaction if something goes wrong ... #47

Support hooks

Let's say that my repository is actually online. Then it would be handy if after every update operation I could have it synchronized.

There are several ways this can go down, but I will lean towards the git style.
In the configuration directory, there is a hooks folder. There can either be a single file, named after the operation, or there can be a directory.

~/.config/repoctl/hooks/update
~/.config/repoctl/hooks/update.d/upload.sh

All matching files will be executed. The one requirement is that the files can be run as is, otherwise they will be ignored.

Last point is how to handle script output. If the script ends with a nonzero response, execution of repoctl will continue, but the output of the script will be printed. Otherwise it depends on the verbosity settings.

Allow removing packages from DB without deleting old cache files

I'm trying to bring repoctl into my workflow, and I noticed that repoctl update is incompatible with paccache - paccache intentionally keeps 3 latest package versions in cache, but repoctl update purges everything except the newest one.

I'd like to use repoctl update to cleanup packages that repoctl status marked as removal, but do nothing with those that are marked as obsolete(N). Ideally I'd also prefer to make repoctl status also ignore obsolete packages, because I use paccache and I know that they are not obsolete.

Reference: aurutils/aurutils#208 (comment)

Create a small program to help zsh completion script

The Zsh completion script requires knowing the repository location and needs to get a list of the files.
This can be done by shell scripts, but it's not very reliable. There should be a program to help Zsh get this right. Call it repoctl-helper or something.

`repoctl list` is showing packages that are not in the repo

Add a package to the repo with repo-add, then remove the package with repo-remove, but leave the cache file in /var/cache/pacman/myrepo folder.

Then repoctl list is still showing this package in its output, although the package is not in the database anymore. I can confirm using aurutils (aur sync --list -d myrepo) that the package is not in the database. I would expect repoctl list to also not show this package.

Handle rate-limiting more gracefully

> repoctl down firefox-nightly
Error: package "firefox-nightly" could not be found on AUR
> repoctl --debug down haxm-altea-git
Error: package "haxm-altea-git" could not be found on AUR
> ping aur.archlinux.org
PING aur.archlinux.org(luna.archlinux.org (2a01:4f8:160:3033::2)) 56 data bytes
64 bytes from luna.archlinux.org (2a01:4f8:160:3033::2): icmp_seq=1 ttl=47 time=231 ms
> ping -4 aur.archlinux.org
PING aur.archlinux.org (5.9.250.164) 56(84) bytes of data.
64 bytes from luna.archlinux.org (5.9.250.164): icmp_seq=1 ttl=47 time=230 ms

I'm pretty sure firefox-nightly will always be on AUR, but on random times, repoctl is not finding ANY package at all. I do have an hourly service downloading the same packages, but didn't find anything to understand/reproduce the error. This started +/- a week ago...

Add separate configuration file support

When trying out different configs it is pretty limiting to not be able to specify a configuration file.
Either it should be documented if it is possible, or it should be made possible somehow.

This would then prevent any other configuration file from being loaded.

Build error

With the PKGBUILD in AUR I get:

package shortry: unrecognized import path "shortry".

I also tried this build function, with the same result:

build() {
  export PATH=/usr/bin/go:$PATH
  GOPATH=${srcdir}/go go get github.com/constabulary/gb/...
  GOPATH=${srcdir}/go go get github.com/goulash/osutil/...
  GOPATH=${srcdir}/go go get github.com/cassava/${pkgname%-git}/...
}

repoctl status: entry names

repoctl status lists new packages as update, which gives me the impression it needs an update. "updated" would be more clear.

Perhaps it would also be useful to distinguish between packages which are newer in the repo (older on the AUR), similar to pacman -Sl or cower -i.

I'm also having an obsolete entry, which I'm unsure on the meaning. It doesn't seem to match the out of date field from the AUR.

Output: https://paste.xinu.at/n8TA/

repoctl new config: hardcoded .tar.gz suffix

For example, I have a database file in /var/cache/pacman/custom/custom.db, and repoctl list gives an error as custom.db.tar.gz doesn't exist. It does however list all packages in the repository.

% repoctl new config /var/cache/pacman/custom/custom   
Error: repository path must be set in configuration
writing new configuration file at /home/archie/.config/repoctl/config.toml .
% repoctl list | wc -l
error: open /var/cache/pacman/custom/custom.db.tar.gz: no such file or directory.
44

Config file: https://paste.xinu.at/HtVT/

When I provide the full custom.db path to repoctl new config, it creates an entry with /var/cache/pacman/custom/custom.db.db.tar.gz.

If I correct the path in the configuration file, I get:

% repoctl list >/dev/null
error: unknown file format.

pacman's repo-add also requires an archive extension (defaulting to .tar), though repo functionality should still work without one.

repoctl update - Error: exit status 1

I've set up repoctl to be used with a "custom" repo for aurutils (release, not -git, since I keep hitting -git releases where basic functionality is broken). To the best of my knowledge, I set up both in accordance with the aurutils documentation.

I've been removing some old, redundant package files from my "custom" repo folder, and also their build files from aurutils' build folder, and my understanding is that I'm supposed to be able to use repoctl update to ensure that these are not repeatedly redownloaded and rebuild, as is the intended behaviour when the packages are still "in" the custom database.

However, when I run repoctl update, I get the following output:

[adam@rakka custom]$ repoctl update --debug
removing package from database: 
removing package from database: gitg-git
removing package from database: icedove-enigmail-bin
removing package from database: ioquake3-git
Error: exit status 1

Those 3 named packages are ones I only just deleted. I was getting the exit status error even before I removed those.

I get the same output each time, which makes it look as if it's hitting the error before actually committing any changes.

I was asking about this in #aurutils on Freenode, but we hit a bit of a dead end working out what was going on.

I'd appreciate any ideas trying to work out what's gone wrong, and if there's anything more I can provide, please just ask. By being documented, hopefully future users should ultimately be able to avoid the same hurdle.

repols: runtime error if repo not configured

If the repo variable is not set in ~/.config/repoctl/config.toml, or the file is not available, repols (from the devel branch) gives a runtime error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x4025a0]

goroutine 1 [running]:
panic(0x846ea0, 0xc82000a130)
        /usr/lib/go/src/runtime/panic.go:481 +0x3e6
main.main()
        /home/archie/go/src/github.com/cassava/repoctl/cmd/repols/main.go:27 +0xb10

In 0.15, a warning message was displayed instead:

Error: repository path must be set in configuration.

Handle signature files

For example, repoctl remove should also remove signature files together with actual package cache files (unless backup-dir is not set to empty string, in which case both files should be ignored).

Today presence of signature files causes repoctl remove to output error: unknown file format. I believe repoctl update also suffers from the presence of signature files.

P.S. In the example below you will see that there are also .sig~ files, handling them is optional (but desired), because they only appear when someone is re-signing the same file twice.

P.P.S. For me personally it's a low priority, because I don't expect repoctl to actually manage cache files (I use backup-dir="") I'm a little annoyed by the false errors error: unknown file format, but I'm actually not affected negatively by this.


$ ll /var/cache/pacman/maximbaz-aur/yay*
.rw-r--r-- 4.8M maximbaz  4 Mar 14:09 /var/cache/pacman/maximbaz-aur/yay-3.440-1-x86_64.pkg.tar
.rw-r--r--  566 maximbaz  4 Mar 14:09 /var/cache/pacman/maximbaz-aur/yay-3.440-1-x86_64.pkg.tar.sig
.rw-r--r--  566 maximbaz  4 Mar 13:56 /var/cache/pacman/maximbaz-aur/yay-3.440-1-x86_64.pkg.tar.sig~

$ repoctl remove --debug --backup=false yay
error: unknown file format.
error: unknown file format.
removing package from database: yay
deleting: /var/cache/pacman/maximbaz-aur/yay-3.440-1-x86_64.pkg.tar

$ ll /var/cache/pacman/maximbaz-aur/yay*
.rw-r--r-- 566 maximbaz  4 Mar 14:09 /var/cache/pacman/maximbaz-aur/yay-3.440-1-x86_64.pkg.tar.sig
.rw-r--r-- 566 maximbaz  4 Mar 13:56 /var/cache/pacman/maximbaz-aur/yay-3.440-1-x86_64.pkg.tar.sig~

$ repoctl remove --debug --backup=false yay
error: unknown file format.
error: unknown file format.

$ rm /var/cache/pacman/maximbaz-aur/yay-3.440-1-x86_64.pkg.tar.sig~

$ repoctl remove --debug --backup=false yay
error: unknown file format.

$ rm /var/cache/pacman/maximbaz-aur/yay-3.440-1-x86_64.pkg.tar.sig

$ repoctl remove --debug --backup=false yay

fatal error: sweep increased allocation count

$ repoctl status -a
On repo custom

runtime: nelems=16 nalloc=3 previous allocCount=2 nfreed=65535
fatal error: sweep increased allocation count

runtime stack:
runtime.throw(0x841684, 0x20)
	/usr/lib/go/src/runtime/panic.go:605 +0x95
runtime.(*mspan).sweep(0x7f2afaf089a0, 0x7f2af9dd2d00, 0x421800)
	/usr/lib/go/src/runtime/mgcsweep.go:298 +0x911
runtime.sweepone(0x0)
	/usr/lib/go/src/runtime/mgcsweep.go:113 +0x122
runtime.gosweepone.func1()
	/usr/lib/go/src/runtime/mgcsweep.go:137 +0x2b
runtime.systemstack(0x7f2af9dd2d88)
	/usr/lib/go/src/runtime/asm_amd64.s:360 +0xab
runtime.gosweepone(0x0)
	/usr/lib/go/src/runtime/mgcsweep.go:136 +0x4a
runtime.deductSweepCredit(0x8000, 0x0)
	/usr/lib/go/src/runtime/mgcsweep.go:407 +0x76
runtime.(*mcentral).cacheSpan(0xc7bbd0, 0x7f2afafb9858)
	/usr/lib/go/src/runtime/mcentral.go:43 +0x60
runtime.(*mcache).refill(0x7f2afafb4000, 0x7bf885, 0x7f2afafb9858)
	/usr/lib/go/src/runtime/mcache.go:123 +0xa4
runtime.(*mcache).nextFree.func1()
	/usr/lib/go/src/runtime/malloc.go:557 +0x32
runtime.systemstack(0xc420027300)
	/usr/lib/go/src/runtime/asm_amd64.s:344 +0x79
runtime.mstart()
	/usr/lib/go/src/runtime/proc.go:1125

goroutine 1 [running]:
runtime.systemstack_switch()
	/usr/lib/go/src/runtime/asm_amd64.s:298 fp=0xc420053058 sp=0xc420053050 pc=0x457360
runtime.(*mcache).nextFree(0x7f2afafb4000, 0x85, 0x38, 0x7fddc0, 0x7bf801)
	/usr/lib/go/src/runtime/malloc.go:556 +0xa9 fp=0xc4200530b0 sp=0xc420053058 pc=0x411ff9
runtime.mallocgc(0x8000, 0x7a53e0, 0x1, 0xc42008c140)
	/usr/lib/go/src/runtime/malloc.go:711 +0x6fa fp=0xc420053158 sp=0xc4200530b0 pc=0x4128ca
runtime.makeslice(0x7a53e0, 0x8000, 0x8000, 0x3a, 0x0, 0x0)
	/usr/lib/go/src/runtime/slice.go:54 +0x77 fp=0xc420053188 sp=0xc420053158 pc=0x442e17
github.com/cassava/repoctl/vendor/github.com/remyoudompheng/go-liblzma.NewReader(0xc3f240, 0xc420088028, 0x836337, 0x3, 0x0)
	/home/daurnimator/.cache/aursync/repoctl/src/src/github.com/cassava/repoctl/vendor/github.com/remyoudompheng/go-liblzma/reader.go:32 +0x81 fp=0xc4200531e0 sp=0xc420053188 pc=0x5cd591
github.com/cassava/repoctl/vendor/github.com/goulash/archive.NewDecompressor(0xc420116240, 0x3a, 0x7f2afafbb868, 0xc420116280, 0xc4200533e8)
	/home/daurnimator/.cache/aursync/repoctl/src/src/github.com/cassava/repoctl/vendor/github.com/goulash/archive/archive.go:84 +0x18e fp=0xc420053330 sp=0xc4200531e0 pc=0x5ce36e
github.com/cassava/repoctl/vendor/github.com/goulash/archive.ReadFileFromArchive(0xc420116240, 0x3a, 0x8373db, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0)
	/home/daurnimator/.cache/aursync/repoctl/src/src/github.com/cassava/repoctl/vendor/github.com/goulash/archive/archive.go:27 +0x75 fp=0xc420053390 sp=0xc420053330 pc=0x5cdd95
github.com/cassava/repoctl/vendor/github.com/goulash/pacman.Read(0xc420116240, 0x3a, 0xc420116201, 0x3a, 0xc420053470)
	/home/daurnimator/.cache/aursync/repoctl/src/src/github.com/cassava/repoctl/vendor/github.com/goulash/pacman/read-pkg.go:22 +0x54 fp=0xc420053410 sp=0xc420053390 pc=0x5d3784
github.com/cassava/repoctl/vendor/github.com/goulash/pacman.ReadDir.func1(0xc420116240, 0x3a, 0xc465e0, 0xc4202b6340, 0x0, 0x0, 0x0, 0x0)
	/home/daurnimator/.cache/aursync/repoctl/src/src/github.com/cassava/repoctl/vendor/github.com/goulash/pacman/read-fs.go:34 +0x1c0 fp=0xc4200534a8 sp=0xc420053410 pc=0x5d5c80
path/filepath.walk(0xc420116240, 0x3a, 0xc465e0, 0xc4202b6340, 0xc420248b40, 0x0, 0x0)
	/usr/lib/go/src/path/filepath/path.go:356 +0x81 fp=0xc420053580 sp=0xc4200534a8 pc=0x567991
path/filepath.walk(0xc4200142d0, 0x18, 0xc465e0, 0xc4202b9110, 0xc420248b40, 0x0, 0x30)
	/usr/lib/go/src/path/filepath/path.go:381 +0x3a0 fp=0xc420053658 sp=0xc420053580 pc=0x567cb0
path/filepath.Walk(0xc4200142d0, 0x18, 0xc420248b40, 0x18, 0x412c08)
	/usr/lib/go/src/path/filepath/path.go:403 +0x11d fp=0xc4200536b8 sp=0xc420053658 pc=0x567f4d
github.com/cassava/repoctl/vendor/github.com/goulash/pacman.ReadDir(0xc420096060, 0xc4200142d0, 0x18, 0x7, 0xc4202cc728, 0x0, 0x0, 0x30)
	/home/daurnimator/.cache/aursync/repoctl/src/src/github.com/cassava/repoctl/vendor/github.com/goulash/pacman/read-fs.go:22 +0x123 fp=0xc420053710 sp=0xc4200536b8 pc=0x5d2db3
github.com/cassava/repoctl/vendor/github.com/goulash/pacman/meta.Read(0xc420096060, 0xc4200142d0, 0x18, 0xc420014330, 0x26, 0xc4201700c0, 0x0, 0x0, 0xc420053a40, 0x75a5ba)
	/home/daurnimator/.cache/aursync/repoctl/src/src/github.com/cassava/repoctl/vendor/github.com/goulash/pacman/meta/read.go:83 +0x100 fp=0xc4200539a0 sp=0xc420053710 pc=0x72ecc0
github.com/cassava/repoctl.(*Repo).ReadMeta(0xc420180000, 0xc420096060, 0x0, 0x0, 0x0, 0x1, 0x1e, 0x0, 0x0, 0x7f2afaf03858)
	/home/daurnimator/.cache/aursync/repoctl/src/src/github.com/cassava/repoctl/read.go:61 +0xf4 fp=0xc420053a50 sp=0xc4200539a0 pc=0x742064
main.glob..func12(0xc39b40, 0xc420044910, 0x0, 0x1, 0x0, 0x0)
	/home/daurnimator/.cache/aursync/repoctl/src/src/github.com/cassava/repoctl/cmd/repoctl/status.go:46 +0x292 fp=0xc420053ce8 sp=0xc420053a50 pc=0x75ee02
github.com/cassava/repoctl/vendor/github.com/spf13/cobra.(*Command).execute(0xc39b40, 0xc4200448f0, 0x1, 0x1, 0xc39b40, 0xc4200448f0)
	/home/daurnimator/.cache/aursync/repoctl/src/src/github.com/cassava/repoctl/vendor/github.com/spf13/cobra/command.go:632 +0x3e8 fp=0xc420053d90 sp=0xc420053ce8 pc=0x584ed8
github.com/cassava/repoctl/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc38e80, 0x8426df, 0x22, 0x2c)
	/home/daurnimator/.cache/aursync/repoctl/src/src/github.com/cassava/repoctl/vendor/github.com/spf13/cobra/command.go:722 +0x2fe fp=0xc420053ec0 sp=0xc420053d90 pc=0x58567e
github.com/cassava/repoctl/vendor/github.com/spf13/cobra.(*Command).Execute(0xc38e80, 0x0, 0x0)
	/home/daurnimator/.cache/aursync/repoctl/src/src/github.com/cassava/repoctl/vendor/github.com/spf13/cobra/command.go:681 +0x2b fp=0xc420053ef0 sp=0xc420053ec0 pc=0x58535b
main.main()
	/home/daurnimator/.cache/aursync/repoctl/src/src/github.com/cassava/repoctl/cmd/repoctl/main.go:106 +0xdd fp=0xc420053f80 sp=0xc420053ef0 pc=0x75c8fd
runtime.main()
	/usr/lib/go/src/runtime/proc.go:185 +0x20d fp=0xc420053fe0 sp=0xc420053f80 pc=0x42da2d
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc420053fe8 sp=0xc420053fe0 pc=0x459f71

Make configuration optional for certain commands

Currently, repoctl dies whenever running without already having been configured.

$ repoctl help
error: repoctl is unconfigured, please create configuration

The following commands could be modified to allow unconfigured use:

  • help
  • version
  • new
  • search
  • download

Do not use Zstd archive format with `repoctl add` after updating pacman to 5.2.0 version

repoctl add -m command just dumps the whole repository and leaves only the added package, when it uses the new Zstd archive format.

That could give somebody hell, I guess.

$ sed '/^$/d; /^#.*/d' $XDG_CONFIG_HOME/repoctl/config.toml
repo = "/home/canalguada/builds/repo/cgrepo.db.tar.gz"
backup = false
backup_dir = "backup/"
interactive = false
columnate = false
color = "auto"
quiet = false

Share marking capabilities of list with filter and status

In general, make the output of the commands list, status and filter more consistent.

filter does not support marking the packages like list does. Neither does status. Maybe this is not necessary for status. Or for filter, but it would be useful to be able to get the packages that are older
in AUR, for example, and find out exactly what the versions are, like ls shows it.

Use cobra package to handle actions.

Instead of working through the actions yourself, consider using cobra. This will probably come at that same time that the download functionality from issue #10 is completed.

Error results in post_action not being run

The post_action from the config is not run when the down command errors out.
The action may not be run whenever any error is triggered.

$ repoctl down ugugu asdfasdf
Error: packages "ugugu" and "asdfasdf" could not be found on AUR

Additionaly, the output could be better, each package on it's own line, for example.

rpc.php limit to 250 packages ?

Hello,
I've got 250 packages in my "repoctl's repo" and when I try to repoctl down -u I've got the full URL to aur.archlinux.org/rpc.php and this return:

http2: server sent GOAWAY and closed the connection; LastStreamID=1, ErrCode=ENHANCE_YOUR_CALM, debug=""

I remove one package and now it's working. If some cleaning can be done on my repo, it will be greatful if repoctl can split request to be under this aur's 250 limit.

repoctl update not cleaning packages' signatures files

repoctl update is able to remove old versions of packages, but if they had a signature side file, that file is not getting removed at all, and the user is forced to go find and remove those left over files... It'd be great for the signature files that go with the old version packages, to be removed altogether.

Thanks !

Clean up source code organization

The organization of the source code is somewhat inconsistent. This could be cleaned up a little to make the structure of the program a bit more visible.

Support delta binaries

I'm using 0.18 release from AUR, and whenever I run "repoctl update" or "repoctl status", I get the error (at the very beginning):

error: unknown field 'deltas' in database entry

On update, that makes the command fail in the end:

Error: exit status 1

On status, at least I get the status of the packages. In case, my configuration:

repo = "<my_repo>.db.tar.xz"
add_params = [
  "-s -v"
]
rm_params = [
  "-s -v"
]
backup = false
backup_dir = "backup/"
interactive = false
columnate = true
quiet = false

repoctl status -u vs -m

Hi,
i just updated to the new version of repoctl (thx) but ;)

when i execute repoctl status -m it lists all (but 2 packages) and with status -u it lists only the 2 missing packages

python-pypdf2: !aur python-ruffus: !aur python2-imaging: !aur python2-lz4: !aur repoctl: !aur
(only a shorted list)

repoctl version 0.14 (6 October 2015)

thanks

Complete zsh completion

The current Zsh completion is better than nothing, but it's not very good. It would be nice to have something more comprehensive. Would appreciate some help here.

  • repo add should complete for files, not package names

Mark packages that are newer in local repository than AUR

As far as I can tell, this happens in two cases:

  • the package maintainer doesn't understand versioning and when to use epoch, or
  • you are updating your own packages, but haven't done so in AUR yet.

repoctl should let us know when this happens.

'repoctl update' picks the wrong file if version is the same

When repoctl update $pkgname is called it picks the wrong and outdate package instead of the newer one, that's happening with my xz->zst transactions. I think modified timestamp should be compared when pkgver and pkgrel are the same.

EDIT: Thinking better about it, modified timestamp should always be preferred, this would solve problems like 1.0.0-rc1 being bigger than 1.0.0.

Optimize list speed

Running repoctl list takes quite a long time, because it reads each package to get all the metadata from it.
This shouldn't be necessary. We should take advantage of the database and only read packages when we have to.

  1. Read database
  2. Check files that are not referenced in the database or are newer than the database

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.