ilikenwf / apt-fast Goto Github PK

When I see the following banner upon server login, I tried sudo apt-fast -y update && apt-fast -y upgrade, which does not get rid off the message upon next server login.

8 packages can be updated.
8 updates are security updates.

In this case, I have to revert back to apt-get. Is it a bug? If so, can it be fixed? Thanks.

For those not using the .deb to install apt-fast, I think the FHS conventions dictate that the apt-fast command it be placed in /usr/local/sbin, not in /bin, /sbin, /usr/bin, nor /usr/sbin (since those are for where programs from the package manager are placed; although I think apt-fast.conf still belongs in /etc, not /usr/local/etc).

I can create a pull request to have the Makefile, quick-install.sh, and README reflect this.

I thought it would be a good idea to implement an automatic mirror selection command (apt-fast best-mirrors-select for example) to add best mirrors according to the location of user like what we have for best server selection in ubuntu.

For example: apt-fast -y dist-upgrade

In the readme it states:

cp ./man/apt-fast.5 /usr/share/man/man5
gzip -f9 /usr/share/man/man5/apt-fast.5
cp ./man/apt-fast.conf.8 /usr/share/man/man8
gzip -f9 /usr/share/man/man8/apt-fast.conf.8

However, the files are

$ ls -R man/
man/:
apt-fast.8  apt-fast.conf.5

I would assume the readme would be correct and the man files should be re-named to apt-fast.5 and apt-fast.conf.8?

apt-fast is good , but it doesn't show update size in confirmation

apt-fast  upgrade 

 Working... this may take a while.
# apt-fast mirror list: Wed Aug 12 09:29:53 IST 2015

http://ftp.tu-chemnitz.de/pub/linux/ubuntu/pool/main/l/linux-firmware/linux-firmware_1.127.15_all.deb
 checksum=md5=49f2c647e288759805b6eb7457fdff54
 out=linux-firmware_1.127.15_all.deb
 If you want to download the packages on your system press Y else n to abort. [Y/n]:  

Please add new command: apt-fast update

the possible values: 1-16, but in the install dialog message this limit not shows, and i can set more then 16, and then the "apt-fast" command shows this msg, and start apt-get

Exception: [AbstractOptionHandler.cc:69] errorCode=28 We encountered a problem while processing the option '--max-connection-per-server'.
-> [OptionHandlerImpl.cc:187] errorCode=1 max-connection-per-server must be between 1 and 16.
Usage:
-x, --max-connection-per-server=NUM The maximum number of connections to one
server for each download.

                          Possible Values: 1-16
                          Default: 1
                          Tags: #basic, #http, #ftp

please parameter to use axel

thank

Failed to fetch http://ppa.launchpad.net/saiarcot895/myppa/ubuntu/dists/wily/main/binary-amd64/Packages 404 Not Found

I wrote it originally, but haven't been the man maintaining apt-fast as of late. Would it make more sense to move apt-fast into a github organization/group/whatever and just have myself and all the contributors be members?

It will be great if apt-fast can --assume-yes.
for automated script ;)

Can you please tag your releases?

If not invoked in root environment, environment variables are ignored. Example:

DOWNLOADBEFORE=true apt-fast dist-upgrade

What works is:

sudo DOWNLOADBEFORE=true apt-fast dist-upgrade

Following code line is responsible:

[ "$UID" = 0 ] || exec sudo "$0" "$@"

Hello, I have use apt-fast for a lone time, and found it's really a useful and powerful tools, thanks for you work.
Blow, I have some new ideas about it:

1. while apt-fast download fill, it doesn't check whether the file exist,
2. We can use & to run all axel simultaneously

just like below: (changed base on version 0.0.3, have not update it yet, ^_^ )

   apt-get -y --print-uris $@ | egrep -o -e "(ht|f)tp://[^\']+" > apt-fast.list;

   for uri in `cat apt-fast.list`; do 
            # get the file name from uri 
            file=${uri##*/};

            if [ -e $file ]; then 
                continue;
            fi

            axel -n 10 $uri &
    done

the only question is how to detect whether all downloads have finished

When I change the server from which I want to get the packages without reloading package lists ( Software & Updates > Ubuntu Software Tab > Download from: ) and then I run apt-get update the /etc/apt/sources.list file is updated correctly but when I run apt-fast update there are still some lines indicating previous server.
let's say if the old server is http://us.archive.ubuntu.com/ubuntu/ and the newly selected one is http://archive.ubuntu.totaal.net/ there are still deb http://us.archive.ubuntu.com/ubuntu/... lines in sources.list

apt-fast --version
apt-fast already running!
Verify that all apt-fast processes are finished then remove /tmp/apt-fast.lock and try again.

There isn't any apt-fast.lock file in the /tmp/ directory.

This is my configuration:

###################################################################
# CONFIGURATION OPTIONS
###################################################################
# Every item has a default value besides MIRRORS (which is unset).

# Use aptitude or apt-get?
# Note that for outputting the package URI list, we always use apt-get
# ...since aptitude can't do this
# Optionally add the FULLPATH to apt-get or apt-rpm or aptitude
# e.g. /usr/bin/aptitude
#
# Default: apt-get
#
_APTMGR=apt-get


# Enable DOWNLOADBEFORE to suppress apt-fast confirmation dialog and download
# packages directly.
#
# Default: dialog enabled
#
DOWNLOADBEFORE=true


# Choose mirror list to speed up downloads from same archive. To select some
# mirrors take a look at your distribution's archive mirror lists.
# Debian: http://www.debian.org/mirror/list
# Ubuntu: https://launchpad.net/ubuntu/+archivemirrors
#
# Examples:
# To use some German mirrors and official Debian and Ubuntu archives you can use:
# MIRRORS=( 'http://ftp.debian.org/debian,http://ftp2.de.debian.org/debian,http://ftp.de.debian.org/debian,ftp://ftp.uni-kl.de/debian'
#          'http://archive.ubuntu.com/ubuntu,http://de.archive.ubuntu.com/ubuntu,http://ftp.halifax.rwth-aachen.de/ubuntu,http://ftp.uni-kl.de/pub/linux/ubuntu,http://mirror.informatik.uni-mannheim.de/pub/linux/distributions/ubuntu/' )
# To use French Ubuntu mirrors you can use:
# MIRRORS=( 'http://fr.archive.ubuntu.com/ubuntu,http://bouyguestelecom.ubuntu.lafibre.info/ubuntu,http://mirror.ovh.net/ubuntu,http://ubuntu-archive.mirrors.proxad.net/ubuntu' )
#
# Default: disabled
#
#MIRRORS=( 'none' )


# Maximum number of connections
# You can use this value in _DOWNLOADER command. Escape with ${}: ${_MAXNUM}
#
# Default: 5
#
_MAXNUM=15


# Downloadmanager listfile
# You can use this value in _DOWNLOADER command. Escape with ${}: ${DLLIST}
#
# Default: /tmp/apt-fast.list
#
DLLIST='/tmp/apt-fast.list'


# Download command to use. Temporary download list is designed for aria2. But
# you can choose another download command or download manager. It has to
# support following input file syntax (\t is tab character):
#
# # Comment
# MIRROR1\tMIRROR2\tMIRROR3...
#  out=FILENAME1
# MIRROR1\tMIRROR2\tMIRROR3...
#  out=FILENAME2
# ...
#
# Examples:
# aria2c with a proxy (set username, proxy, ip and password!)
# _DOWNLOADER='aria2c -c -j ${_MAXNUM} -x ${_MAXNUM} -s ${_MAXNUM} --min-split-size=1M --http-proxy=http://username:password@proxy_ip:proxy_port -i ${DLLIST}'
#
# Default: _DOWNLOADER='aria2c -c -j ${_MAXNUM} -x ${_MAXNUM} -s ${_MAXNUM} --min-split-size=1M -i ${DLLIST} --connect-timeout=600 --timeout=600 -m0'
#
_DOWNLOADER='aria2c -c -j ${_MAXNUM} -x ${_MAXNUM} -s ${_MAXNUM} --min-split-size=1M -i ${DLLIST} --connect-timeout=600 --timeout=600 -m0'


# Download temp folder for Downloadmanager
# example /tmp/apt-fast. Standard is /var/cache/archives/apt-fast
#
# Default: /var/cache/apt/archives/apt-fast
#
DLDIR='/var/cache/apt/archives/apt-fast'


# APT archives cache directory
#
# Default /var/cache/apt/archives
# (APT configuration items Dir::Cache and Dir::Cache::archives)
#
APTCACHE='/var/cache/apt/archives'


# apt-fast colors
# Colors are disabled when not using a terminal.
#
# Default colors are:
#  cGreen='\e[0;32m'
#  cRed='\e[0;31m'
#  cBlue='\e[0;34m'
#  endColor='\e[0m'

See issue #55

Hey @saiarcot895, it would be possible you enable the new Ubuntu 16.04 on your ppa to build apt-fast package?

near line #215 in get_mirrors(), there is such a line:

echo -e "${list// /${filepath}\t}$filepath\n"

In my test, it produces such wrong output when multiple mirrors are configed:

<mirror1_url>t<mirror2_url>t...

change it to the following form solve the issue:

echo -e "${list// /${filepath}\\t}$filepath\n"

apt-fast doesn't seem to respect apt-get's quiet mode. When passing the flags "-qq" there is still verbose output.

Could the quiet flags please be included?

Could you please provide autocompletion for zsh?

Please print error if unkown option was passed. It was a bit annoying with update command - it took me a while (after looking into code) to see that this is an unkown option ;) . Perhaps getopts is the right thing to check valid commands.

Since apt-fast is mainly shellscript, would it be possible for the launchpad ppa to build apt-fast as multiarch/architecture independent, allowing for immediate use on ARM and other platforms?
(I can confirm that when installing the apt-fast:i386 package on Debian for powerpc using multiarch-configured apt, and overriding the aria2:i386 dependency for the native aria2:powerpc version, it runs as expected. I overrode the dependency by changing /var/lib/dpkg/status)

Hi,

I ran:

apt-fast install -y kde-desktop

and this returned the error shown in this screenshot

Here is my /etc/apt-fast.conf file (my mirrors were selected based on my location):

###################################################################
# CONFIGURATION OPTIONS
###################################################################
# Every item has a default value besides MIRRORS (which is unset).

# Use aptitude or apt-get?
# Note that for outputting the package URI list, we always use apt-get
# ...since aptitude can't do this
# Optionally add the FULLPATH to apt-get or apt-rpm or aptitude
# e.g. /usr/bin/aptitude
#
# Default: apt-get
#
#_APTMGR=apt-get


# Enable DOWNLOADBEFORE to suppress apt-fast confirmation dialog and download
# packages directly.
#
# Default: dialog enabled
#
#DOWNLOADBEFORE=true


# Choose mirror list to speed up downloads from same archive. To select some
# mirrors take a look at your distribution's archive mirror lists.
# Debian: http://www.debian.org/mirror/list
# Ubuntu: https://launchpad.net/ubuntu/+archivemirrors
#
# Examples:
# To use some German mirrors and official Debian and Ubuntu archives you can use:
# MIRRORS=( 'http://ftp.debian.org/debian,http://ftp2.de.debian.org/debian,http://ftp.de.debian.org/debian,http://ftp.uni-kl.de/debian'
#          'http://archive.ubuntu.com/ubuntu,http://de.archive.ubuntu.com/ubuntu,http://ftp.halifax.rwth-aachen.de/ubuntu,http://ftp.uni-kl.de/pub/linux/ubuntu,http://mirror.informatik.uni-mannheim.de/pub/linux/distributions/ubuntu/' )
# To use French Ubuntu mirrors you can use:
# MIRRORS=( 'http://fr.archive.ubuntu.com/ubuntu,http://bouyguestelecom.ubuntu.lafibre.info/ubuntu,http://mirror.ovh.net/ubuntu,http://ubuntu-archive.mirrors.proxad.net/ubuntu' )
#
# Default: disabled
#
MIRRORS=( 'http://ftp.au.debian.org/debian/, http://debian.mirror.digitalpacific.com.au/debian/, http://mirror.optus.net/debian/, http://mirror.linux.org.au/debian/' )


# Maximum number of connections
# You can use this value in _DOWNLOADER command. Escape with ${}: ${_MAXNUM}
#
# Default: 5
#
#_MAXNUM=5


# Downloadmanager listfile
# You can use this value in _DOWNLOADER command. Escape with ${}: ${DLLIST}
#
# Default: /tmp/apt-fast.list
#
#DLLIST=/tmp/apt-fast.list


# Download command to use. Temporary download list is designed for aria2. But
# you can choose another download command or download manager. It has to
# support following input file syntax (\t is tab character):
#
# # Comment
# MIRROR1\tMIRROR2\tMIRROR3...
#  out=FILENAME1
# MIRROR1\tMIRROR2\tMIRROR3...
#  out=FILENAME2
# ...
#
# Examples:
# aria2c with a proxy (set username, proxy, ip and password!)
# _DOWNLOADER='aria2c -c -j ${_MAXNUM} -x ${_MAXNUM} -s ${_MAXNUM} --min-split-size=1M --http-proxy=http://username:password@proxy_ip:proxy_port -i ${DLLIST}'
#
# Default: _DOWNLOADER='aria2c -c -j ${_MAXNUM} -x ${_MAXNUM} -s ${_MAXNUM} --min-split-size=1M -i ${DLLIST} --connect-timeout=600 --timeout=600 -m0'
#
#_DOWNLOADER='aria2c -c -j ${_MAXNUM} -x ${_MAXNUM} -s ${_MAXNUM} --min-split-size=1M -i ${DLLIST} --connect-timeout=600 --timeout=600 -m0'


# Download temp folder for Downloadmanager
# example /tmp/apt-fast. Standard is /var/cache/archives/apt-fast
#
# Default: /var/cache/apt/archives/apt-fast
#
#DLDIR=/var/cache/apt/archives/apt-fast


# APT archives cache directory
#
# Default /var/cache/apt/archives
# (APT configuration items Dir::Cache and Dir::Cache::archives)
#
#APTCACHE=/var/cache/apt/archives


# apt-fast colors
# Colors are disabled when not using a terminal.
#
# Default colors are:
#  cGreen='\e[0;32m'
#  cRed='\e[0;31m'
#  cBlue='\e[0;34m'
#  endColor='\e[0m'

any ideas? If this error can be overcome I think I will fall in love with this program as atm the best download speeds I'm getting just with the http://ftp.au.debian.org/debian/ is 100 kB/s even though from the pacman mirrors (I'm running Debian in a VirtualBox VM — my host machine is Arch Linux) I get download speeds of up to 3 MB/s. If the final / in the mirror list is a problem, well I have tried removing the final / and re-running apt-fast install kde-desktop and it returned the same error again.

In case it is relevant I am using the testing branch of Debian.

Thanks for your time,
Brenton

Does apt-fast transmit the whole traffic through Tor network via apt-transport-tor?
If so, does it require any extra work on the .conf file?

Thanks

IMO, it makes more sense in terms of both efficiency and readability to use case statements, such as in places where there are 3 or more options. See https://github.com/ilikenwf/apt-fast/blob/master/apt-fast#L37 for a place that could use such treatment. I'm sure there are others.

Ubuntu 16.04 adds apt which include a progress bar in the bottom. Please update apt-fast to use it instead of apt-get.

aptitude & apt-get uses different policies to decide the dependencies, apt-fast uses apt-get's to decide, if we let apt-fast to use aptitude, it may miss some files or download useless files.

The new behavior is ${_APTMGR} -y to suppress prompts from package manager. This is a regression because you don't see anymore which packages have been kept back, which will be upgraded and which will be removed. With a complex setup a non-interactive behavior can be dangerous.

I know why this switch was introduced but imho two prompts are better then a single less meaningful one. If you save decision of first promt (see #15) -y can easily be removed.

Hi,

Dont know if I am in the right place but...
When you run dpkg-reconfigure on apt-fast version 1.8.3137+git7b72bb7-0ubuntu1ppa3 (on willy and xenial) wich is on https://launchpad.net/~saiarcot895 ppa you have this warning at the end of the reconfiguring:
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/${ <-- HERE _MAXNUM}/ at /var/lib/dpkg/info/apt-fast.config line 205."
The original line is:
if ($downloadcmd =~ /${_MAXNUM}/){
But it can be replaced by
if ($downloadcmd =~ /${_MAXNUM}/){ and the regex warning will disapear.

Edit: I'm sorry but I don't know why but the corrected code don't appear even in the brace, so I put space to see it:
if ($downloadcmd =~ /$ \ <-- HERE {_MAXNUM}/){

As asked in iloveubuntu blog post 1, there is no support of update command yet.
Usage of apt-get --print-uris update works mostly fine but we have to save those files in specific location. They have same names like "Sources.bz2". "Packages.bz2" etc. so we have to rename them. With aria2 this works well with following download list example:

http://ftp.debian.org/debian/dists/experimental/main/i18n/Translation-en.bz2
 out=ftp.debian.org_debian_dists_experimental_main_i18n_Translation-en
http://ftp.debian.org/debian/dists/unstable/main/source/Sources.bz2
 out=ftp.debian.org_debian_dists_unstable_main_source_Sources

With axel this is not possible (yes we have --output option but only for one single file). Should we drop axel support because it already has a disadvantage: no download resuming

I'm nearly done with update command integration but at the moment it would break axel compatibility. But perhaps we could grep _DOWNLOADER for axel or aria and (if necessary) fallback to apt-get update without download acceleration.
I like the last idea to determine it on the fly.

@ilikenwf and others: What are your opinions?

Would you please add deb package for ubuntu 14.04 in ppa repo? so we can get it easily, many thanks!

http://pastebin.com/ePhPiRSH

i am trying to use apt-fast for dist upgrade from Linux mint 13 to 14 i ge the following error and apt-fase falls back to basic apt-get command, this is a 32 bit machine

If you want to download the packages on your system press Y else n to abort. [Y/n]:
Exception: [AbstractOptionHandler.cc:73] errorCode=28 We encountered a problem while processing the option '--max-concurrent-downloads'.
-> [util.cc:648] errorCode=1 Failed to convert string into 64bit signed integer. '-i'
Usage:
-j, --max-concurrent-downloads=N Set maximum number of parallel downloads for
every static (HTTP/FTP) URL, torrent and metalink.
See also --split option.

                          Possible Values: 1-*
                          Default: 5
                          Tags: #basic

I would like to speedup the build-time of our tools on travis-ci[1] which is mainly slow because of apt-get calls to preprae the environment.

could you propose how to install apt-fast on such a continous build vm?

I think it will not work atm because of the dialog which is started right after the installation... are there any flags/paramters which I could use so everything can be configured in place?

[1] https://travis-ci.org/

I was wondering if we can make other tools like synaptic or ppa-purge make use of apt-fast instead of apt-get without modifying those programs. Is apt-fast 100% compatible with apt-get? If yes, I think a user could move apt-get binary to apt-get.original and link apt-get -> apt-fast. That would work as a hack. Any proper way to do this?

Hello all,

Since #91 I broke apt-fast because lack of knowledge.
My "only" changeset was overcome limitation of aria2 because print-uris return sha256 and it is not supported. Without it, I can't use apt-fast because all packages with sha1 or sha256 are dropped and there is nothing to download :(

Changeset summary:

• #91: feature but broken, corrected at #96
• #92: completing #91 by enforce package version. Not common usage but could be.
• #96: fix #91, --print-uris + install generates human readable output.
• #97: fix #92, version is not usable in "raw" format.

I already test it with Docker to ensure that all is working without collateral false negatives.

Sorry for that.

PS: about version tagging, perhaps 1.8 should be at af73a1 (prior me), or after merge lastest fixes because add above required feature.

Does this work with my fabric script?

Installed apt-fast and did an alias apt-get="apt-fast" in my bash_profile
Looks about the same speed for me. Am I doing something wrong?

Or does it just not play well with fabric script?

How can I use apt-fast with apt-cacher-ng?

Now apt is considered default in ubuntu. Please, add support for it.

Initially when I installed and used apt-fast I dint see any difference at all. I checked the ap-fast.conf file and found this line

_DOWNLOADER='aria2c -c -j ${_MAXNUM} -i ${DLLIST} --connect-timeout=600 --timeout=600 -m0'

This command can be further improved. On my machine I did some playing with aria2c on my laptop and found some interesting options which really made a difference

aria2c -j 8 -k 2M -s 8 -x 8

-k for min split size ( its 2MB in this case )
-x for max connections per server
-s tells aria to split files in to peices

If you could include those options in the package and release a newer version it would be great.

When I try to use the command "apt-fast -y --purge autoremove" with a tab, it's gives error "_comp_dpkg_installed_packages: command not found". I've found and installed bash-completion-lib via https://code.google.com/p/bash-completion-lib/ but nothing was changed. I'm on Ubuntu 13.04 and have installed apt-fast via ppa repository.

Can I user apt-fast to download all the packages need to be upgraded? And I only want to download it not automatic install it.

Using apt-fast to fetch files results in a folder that neither
apt-fast clean
apt-get clean

can actually clean. ("Problem unlinking the file apt-fast - Clean (21: Is a directory)

My 16.04 virtual machine is fully updated; I think this is now the final beta.

apt-get install accepts globs or regexes in the form of apt-get install 'a*'. These forms overlap with some shell glob metachars, and the argument parsing for arguments in $@ in apt-fast just happily expends it for the current workdir. Using "$@" solves this problem.

PS. The if statements in the matching look a bit too heavy to me. Perhaps it's more appropriate to do the matching in a case statement?

PPS. The mirrors=( ${mirrorstr//,/ } ) thing can be improved in plain-bash with mirrors=( ${mirrorstr//*([[:space:]]),*([[:space:]])/ } ). This requires shopt -s extglob (available in bash 3.0+).

I am using vagrant ssh from a github4windows bash (not the default windows powershell, which github4windows is using).
-> Github4Windows -> Options -> DefaultSheel -> Git Bash

when installing apt-fast in a VM with a ssh session open from this mentioned Git Bash I get the following error while installing:

(Reading database ... 51168 files and directories currently installed.)
Unpacking libc-ares2 (from .../libc-ares2_1.7.5-1_amd64.deb) ...
Selecting previously unselected package aria2.
Unpacking aria2 (from .../aria2_1.13.0-1_amd64.deb) ...
Selecting previously unselected package apt-fast.
Unpacking apt-fast (from .../apt-fast_1.8.2-1~precise1_amd64.deb) ...
Processing triggers for man-db ...
Setting up libc-ares2 (1.7.5-1) ...
Setting up aria2 (1.13.0-1) ...
Setting up apt-fast (1.8.2-1~precise1) ...
Unknown terminal: msys
Check the TERM environment variable.
Also make sure that the terminal is defined in the terminfo database.
Alternatively, set the TERMCAP environment variable to the desired
termcap entry.
debconf: whiptail output the above errors, giving up!
Use of uninitialized value $ret in scalar chomp at /usr/share/perl5/Debconf/Clie
nt/ConfModule.pm line 132, <STDIN> line 3.
Use of uninitialized value $ret in split at /usr/share/perl5/Debconf/Client/Conf
Module.pm line 133, <STDIN> line 3.
Use of uninitialized value $ret[0] in string eq at /usr/share/perl5/Debconf/Clie
nt/ConfModule.pm line 134, <STDIN> line 3.
dpkg: error processing apt-fast (--configure):
 subprocess installed post-installation script returned error exit status 255
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
Errors were encountered while processing:
 apt-fast

Hi,
during instalation apt-fast asks for:

• application manager (apt-get/aptitude)
• number of connections
• shall it skip confirmation

It prevents apt-fast to be installed in non-interactive way.
I think, that it should install with defaults, and allow to change it later ("convention over configuration").

I propose:
application manager = aptitude installed ? aptitude : apt-get
connections = 5 (that's Your proposal) or 10 (my personal choice - helps with many small files)
skip = true

If You are unable to do so, for whatever reason, then maybe You are able to provide names and values for debconf-utils.
I have unsuccessfully tried:

echo 'apt-fast apt-fast/_APTMGR string aptitude' | sudo debconf-set-selections
echo 'apt-fast apt-fast/maxdownloads string 10' | sudo debconf-set-selections
echo 'apt-fast apt-fast/DOWNLOADBEFORE boolean true' | sudo debconf-set-selections

Which hangs whole installation with holding one CPU core at 100%...

P.S.
Why isn't apt-fast in default repository of Debian/Ubuntu/Linux Mint?

I know that the way to define mirrors for apt-fast is to use the apt-fast configuration file but the script should also parse mirror:// URIs and work with the selected mirror.