If more than one URL is provided by the user, axel should use at least as many connections as the number of URLs. If not, some URLs would go unused.

Maybe we should rather set num-connections as (num-connections x num of URLs) ?

Does anyone know axel can achieve HTTP? Why do it restart to download files again other than continuing to transfer the last tasks when debugging?

To reproduce:

# axel https://nm.debian.org/public/processes
Initializing download: https://nm.debian.org/public/processes
Connection gone.

Works fine with curl and wget.

Regards,

Eriberto

We don't yet support this. PR #49 doesn't handle it correctly.

This needs to be handled differently from regular HTTP proxying, we need to do a CONNECT and suppress all the other proxy-related behavior (missing in PR #49).

To reproduce do axel -a -n 4 URL wait for all 4 segments start to download, then axel -a -n 2 URL. You will notice that you can't make 2 connections, it always continues with 4.

Seeing a lot of Segmentation fault: 11 after upgrading to 2.12 on OSX 10.11.6.
Primarily when downloading 10-20 HTTPS connections.

Tends to happen most when progress is around 95-100% complete.

tried make after ./configure command and got this:

Undefined symbols for architecture x86_64:
  "_libintl_bindtextdomain", referenced from:
      _main in text.o
  "_libintl_gettext", referenced from:
      _axel_new in axel.o
      _axel_open in axel.o
      _axel_start in axel.o
      _axel_do in axel.o
      _conf_loadfile in conf.o
      _conn_info in conn.o
      _ftp_connect in ftp.o
      ...
  "_libintl_setlocale", referenced from:
      _main in text.o
  "_libintl_textdomain", referenced from:
      _main in text.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [axel] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

installed gettext via homebrew:

brew install gettext --universal; brew link gettext --force 

any idea?

On Debian Stretch, Axel 2.13.1 build produces the following warnings:

conf.c: In function 'conf_loadfile':
conf.c:77:3: warning: ignoring return value of 'fscanf', declared with attribute warn_unused_result [-Wunused-result]
   fscanf( fp, "%100[^\n#]s", s );
   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
conf.c:78:3: warning: ignoring return value of 'fscanf', declared with attribute warn_unused_result [-Wunused-result]
   fscanf( fp, "%*[^\n]s" );
   ^~~~~~~~~~~~~~~~~~~~~~~~

I'm currently running Ubuntu 15.10 and was trying to build the project in a custom user directory, but it gives me the following error:

$ ./configure --prefix=/home/mty/dev/.build/axel/ > /dev/null && make

CDPATH="${ZSH_VERSION+.}:" && cd . && /bin/bash /home/mty/dev/axel/missing aclocal-1.14
/home/mty/dev/axel/missing: line 81: aclocal-1.14: command not found
WARNING: 'aclocal-1.14' is missing on your system.
You should only need it if you modified 'acinclude.m4' or
'configure.ac' or m4 files included by 'configure.ac'.
The 'aclocal' program is part of the GNU Automake package:
<http://www.gnu.org/software/automake>
It also requires GNU Autoconf, GNU m4 and Perl in order to run:
<http://www.gnu.org/software/autoconf>
<http://www.gnu.org/software/m4/>
<http://www.perl.org/>
Makefile:378: recipe for target 'aclocal.m4' failed

I have not modified any configuration files apart from passing the parameter. Trying to find a fix, I looked around and found this suggestion from one user facing a similar issue in another project (github.com/SIPp/sipp/issues/61). The fix was simply to run:

$ touch configure.ac aclocal.m4 configure Makefile.am Makefile.in

after which compilation worked smoothly and installed properly. Why do you think this is happening?

this error happens unexpectedly, backtrace

#0  0x00007ffff7a9b515 in sk_pop_free () from /usr/lib/libcrypto.so.1.0.0
#1  0x00007ffff7af0c54 in X509_VERIFY_PARAM_free () from /usr/lib/libcrypto.so.1.0.0
#2  0x00007ffff77330f1 in SSL_CTX_free () from /usr/lib/libssl.so.1.0.0
#3  0x00007ffff7735621 in SSL_free () from /usr/lib/libssl.so.1.0.0
#4  0x00000000004089d8 in tcp_close ()
#5  0x0000000000405dfd in conn_disconnect ()
#6  0x0000000000404dcd in axel_close ()
#7  0x0000000000402509 in main ()
#8  0x00007ffff71542e1 in __libc_start_main () from /usr/lib/libc.so.6
#9  0x000000000040324a in _start ()

• Move .h .c files to src folder
• Move .po files to i18n folder
• Rename README.to-contribute to CONTRIBUTING.md, why?
• Rename COPYING to LICENSE
• remove ROADMAP and use Github's Milestones instead
• remove all text files (API, UPDATE-CHECK, ...) and use Github's Wiki instead
• add .gitignore

Things I'm not sure about:

• Update license to v3
• Use a dev branch
• update license in .h .c header
• remove CHANGES and use Github's release describe instead

I'll complete this list

Likely something is wrong with the speed check in axel_do (shown in the backtrace), the last thread gets stuck in a loop calling axel_nanosleep.

given that axel is focussed on being small in size, I think it would make sense to make it work with mbedtls and let the user choose (at compile time) which SSL library to use. This may help axel spreading around also on embedded platforms.

There are a bunch of objects in the code that are accessed both by the main thread and by the downloading ones. Although practically race conditions seem to be rare, the probability of hitting one increases with the number of connections.

Such objects should be protected by mutex/lock to avoid concurrent access.

Source code is

    if( !st )
    {
        fprintf( stderr, _("Error in %s line %i.\n"), file, line );
        return( 0 );
    }

Suggest add missing call to fclose.

Hi!

Thanks for this useful and excellent utile!

Can you implement in-memory buffer option to allow not write so often to disk?

Add Testing for different method
Same Idea as Broken v3

Hello.

I'm getting Segfault when trying to download a 221GB file from dropbox.

wget works fine but axel produces segfault at the start.

axel -n 16 https://www.dropbox.com/[removed]dl=1
Initializing download: https://www.dropbox.com/[removed]dl=1
File size: 238264867109 bytes
Opening output file [removed]
Starting download
Segmentation fault

Hi all,

Our bug scanner has reported a data race issue at text.c#L237,

in function main, search_sortlist sorts threads created by search_getspeeds,

j = search_getspeeds( search, i );
search_sortlist( search, i );

search_getspeeds returns 0 if pthread_create failed,
would it create a part of speedtest threads, then hit one pthread_create failure and returns to main function? Could it result a situation that the created threads still running and search_sortlist writes to variables used by threads in the same time?

int search_getspeeds( search_t *results, int count )
{
	.......

	while( done < count )
	{
		for( i = 0; i < count; i ++ )
		{
			if( running < results->conf->search_threads && !results[i].speed )
			{
				results[i].speed = SPEED_ACTIVE;
				results[i].speed_start_time = gettime();
				if( pthread_create( results[i].speed_thread,
					NULL, search_speedtest, &results[i] ) == 0 )
				{
					running ++;
					break;
				}
				else
				{
					return( 0 );        // hit one pthread_create failure here, but some of threads might still running
				}
			}

SourceBrella Inc.,
Alex

• Bug on Ubuntu: https://bugs.launchpad.net/ubuntu/+source/axel/+bug/922963
• Bug on Fedora: https://bugzilla.redhat.com/show_bug.cgi?id=675964
• Proposed patch: https://lists.alioth.debian.org/pipermail/axel-devel/2011-February/000198.html

See #36

If you ask axel to download a file that already exists, it will add a .0, .1, .2 suffix to the filename until an unused filename is found.

Perhaps a flag to skip already-downloaded files. This would not apply if the state file existed.

Like curl's:

       -4, --ipv4
              This option tells curl to resolve names to IPv4 addresses only, and not for example try IPv6.

       -6, --ipv6
              This option tells curl to resolve names to IPv6 addresses only, and not for example try IPv4.

The current implementation only escapes spaces. All special characters must be escaped.

I installed via brew ... but it's version 2.12

How can I upgrade to the latest?

This is wonderful to see updated Axel, currently when I wan use socks5 as proxy for example I get this error:

Invalid proxy string: socks5://127.0.0.1:1080

I'm not a very good programmer but I'd love to spend time on this, let me know if you think this is a good idea

From Ubuntu[1]:

downloaded using flashgot+aria2 got proper downloads

but tried it using axel+flashgot all i got was image which could not be read.

type of download - all jpg images
60% of them 453 bytes and remaining 456 bytes

at the same time aria went through and got all properly ranging from 41 to 188 kb

for axel some of the errors that i got were
too many redirects.
server unsupported.
starting from scratch 1 connection

flashgot

aria2 - ok
axel - ok
wget-ok

ko for everything else

[1] https://bugs.launchpad.net/ubuntu/+source/axel/+bug/899867

Regards,

Eriberto

Probably not related to #7, that was associated with no https support and closed by @std ssl inplementation on #19.
More URLs I had tryed ends up with this error.

Looking at source code, it seems that MAX_CONECTIONS is set to 20. Sadly, WGET only make 3 redirects to a final http address...

Details testing "https://osdn.net/dl/android-x86/android-x86-7.1-rc1.iso":

Axel fails with 'Too many redirects.' and exit code 1, no extra information increasing verbosity:

$ axel -a -vvv -n50 https://osdn.net/dl/android-x86/android-x86-7.1-rc1.iso
Initializing download: https://osdn.net/dl/android-x86/android-x86-7.1-rc1.iso
Too many redirects.

$ echo $?
1

Then I get the final redirect url using wget:

$ wget https://osdn.net/dl/android-x86/android-x86-7.1-rc1.iso
--2017-08-26 12:18:48--  https://osdn.net/dl/android-x86/android-x86-7.1-rc1.iso
Resolving osdn.net (osdn.net)... 202.221.179.17
Connecting to osdn.net (osdn.net)|202.221.179.17|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://osdn.net/projects/android-x86/downloads/67834/android-x86-7.1-rc1.iso/ [following]
--2017-08-26 12:18:50--  https://osdn.net/projects/android-x86/downloads/67834/android-x86-7.1-rc1.iso/
Reusing existing connection to osdn.net:443.
HTTP request sent, awaiting response... 302 Found
Location: https://osdn.net/frs/redir.php?m=c3sl&f=%2Fandroid-x86%2F67834%2Fandroid-x86-7.1-rc1.iso [following]
--2017-08-26 12:18:50--  https://osdn.net/frs/redir.php?m=c3sl&f=%2Fandroid-x86%2F67834%2Fandroid-x86-7.1-rc1.iso
Reusing existing connection to osdn.net:443.
HTTP request sent, awaiting response... 302 Found
Location: http://c3sl.dl.osdn.jp/android-x86/67834/android-x86-7.1-rc1.iso [following]
--2017-08-26 12:18:51--  http://c3sl.dl.osdn.jp/android-x86/67834/android-x86-7.1-rc1.iso
Resolving c3sl.dl.osdn.jp (c3sl.dl.osdn.jp)... 200.236.31.1, 2801:82:80ff:8000::2
Connecting to c3sl.dl.osdn.jp (c3sl.dl.osdn.jp)|200.236.31.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 699400192 (667M) [application/x-iso9660-image]
Saving to: 'android-x86-7.1-rc1.iso.2'

         android-x8   0%[                    ] 363.04K  1.77MB/s               ^C

So I can download with the final URL that axel can't resolve...

$ axel -a -n50 http://c3sl.dl.osdn.jp/android-x86/67834/android-x86-7.1-rc1.iso
Initializing download: http://c3sl.dl.osdn.jp/android-x86/67834/android-x86-7.1-rc1.iso
File size: 699400192 bytes
Opening output file android-x86-7.1-rc1.iso.0
Starting download

[  0%] [0123456789:;<=> ?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`a] [ 341.7KB/s] [33:18[  0%]

Hi all,
I seem to be experiencing a file name parsing issue.

I run
axel -a -n 6 -v -s 204800 "https://something.url.com/d/p/longalphastring/Name%20of%20File.zip"
and the file name the results in being downloaded is
"Name" rather than "Name of File.zip" which leads me to believe it's not parsing the spaces.

Other computers with axel (older OSX version and axel versions) run without this malformed name. Also browsers (Safari, firefox, chrome, etc.) all seem to be able to handle the URL and generate the full file name.

Example:

axel -a -n 6 -v -s 204800 "https://something.url.com/d/p/longalphastring/Name%20of%20File.zip"
Initializing download: https://something.url.com/d/p/longalphastring/Name%20of%20File.zip
File size: 165596 bytes
Opening output file Name
Starting download

[  0%] [0                                 1                                 .2                                 3                                  4                                 5                                 ] [  76.2KB/s] [ 1h10]

Platform data:

macOS Sierra 10.12.2 (16C68)

axel: stable 2.12, HEAD
Light UNIX download accelerator
https://github.com/eribertomota/axel
/Users/rjeong/homebrew/Cellar/axel/2.12 (15 files, 133K) *
  Built from source on 2017-01-12 at 21:05:19
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/axel.rb
==> Dependencies
Build: autoconf ✔, automake ✔
Required: gettext ✔, openssl ✔

Thanks

As I see many people have issues with axel and https.
I have similar issues due to self signed certificate.

What is the option for axcel equivalent to wget --no-check-certificate ?

Is there an option for axelrc to ignore or not verify SSL certificates by default?

The intent of this bug is help the people that wants help to solve the known bugs in Axel. Please, choose a bug and send us a pull request. Thanks!

From GitHub

Bugs:

• Implement SSL #3. This is a bug because makes SSL downloads impossible to be done and generates a 'Too many redirects' message. (work being done by @sdt)

Wishes

• Add socks5 support enhancement #2
• Add unit testing #5
• File name from Content-Disposition header #8

From Debian

Bugs:

• Segmentation fault when using user:pass in FTP (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=645234)
• Segmentation fault when we execute it with a connection negative values (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=703453)
• Segmentation fault when redirecting from HTTP to FTP (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=449513#43)

Wishes

• Support .netrc files (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=127074)

From Ubuntu

Bugs:

• Too many redirects. Segmentation fault. The same issue related at #3. (https://bugs.launchpad.net/ubuntu/+source/axel/+bug/253906)

Wishes

• Flashgot+axel not working properly (https://bugs.launchpad.net/ubuntu/+source/axel/+bug/899867)
• Proxy authentication (maybe implemented in Fedora) (https://bugs.launchpad.net/ubuntu/+source/axel/+bug/922963)

From Fedora

Wishes:

• Patch to allow authentication behind proxy (https://bugzilla.redhat.com/show_bug.cgi?id=675964)

The usage of a dynamic string library, such as SDS - Simple Dynamic Strings (https://github.com/antirez/sds), would help getting rid of buffer overflows, by using dynamic heap-allocated strings instead of fixed-sized stack-allocated ones. In the particular case of SDS, it even provides an implementation that is compatible with good old C strings, making it easier to make the transition and to interface with other C libraries whenever needed.

We are currently working on an Italian translation to be integrated with the rest.
Lang code will be: it_IT

From Debian[1]:

Axel should try to use the .netrc file when trying to login to an FTP server.

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=127074

Regards,

Eriberto

I'm working with Axel for years now, but sometimes I think Axel doesn't support HTTPS,
please current me if I'm wrong, but Axel fails to donwload over SSL sometimes
after take a quick look at source code I couldn't find any related code to it

When one of the connections stalls for any reason, Axel will take an enormous amount of time before timing it out and trying again. I usually kill axel and start it again to make the procedure faster.

I think we should tweak the TCP/connection timeout so that axel can be more reactive.

Hello,
this question probably goes to @ismaell @eribertomota : have you guys ever considered the possibility to use libcurl directly into axel? I think this would simplify the code a lot as you won't need to deal with the "low" level http/ftp connection and can focus more on handling the data at the higher level.
Any thought?

I guess, axel should prefer TLS over SSL.

• Homebrew/homebrew-core#2009

brew install axel

axel 2.5 remembers and uses the value of CFLAGS that the user set at configure time, but not the value of LDFLAGS.

This is sufficient to fix it:

--- configure.orig  2015-11-01 16:48:49.000000000 -0600
+++ configure   2015-11-13 10:11:08.000000000 -0600
@@ -236,6 +230,7 @@

 echo "CFLAGS+=${AXEL_CFLAGS}" >> Makefile.settings
 echo "LFLAGS+=${AXEL_LFLAGS}" >> Makefile.settings
+echo "LDFLAGS+=${LDFLAGS}" >> Makefile.settings

 echo 'Configuration done:'
 if [ "$i18n" = "1" ]; then

The Makefile is already written to use the LDFLAGS variable.

This happens quite randomly, but I finally found an example where this is reproducible: Downloading dietlibc-0.33.tar.bz2 with default settings fails to request the final byte of the file.

After doing some digging, I have learned that this is actually due to a bug in the webserver, which replies with a Content-Length header short by 1 from the actual reply. However, other download accelerators (like Firefox's DTA) can handle this situation, because they parse the file size from the Content-Range header rather than the Content-Length header. It would be nice to have this in axel as well, at least in the form of comparing the values from the two headers and falling back to using a single connection without Range requests should they differ.

In the first request, axel requests:

GET /dietlibc/dietlibc-0.33.tar.bz2 HTTP/1.0
Host: www.fefe.de
Range: bytes=1-
User-Agent: Axel 2.11 (Linux)

to which the server replies

HTTP/1.1 206 Partial Content
Accept-Ranges: bytes
Server: Gatling/0.14
Content-Type: application/octet-stream
Content-Length: 626883
Date: Tue, 20 Sep 2016 10:27:44 GMT
Last-Modified: Wed, 13 Mar 2013 11:32:49 GMT
Content-Range: bytes 1-626884/626885
Connection: close

Since the Content-Length is one byte short, axel doesn't request the last byte in the final request, and thus the final file ends up being one byte short.

It'd be great if axel could download and save file with the name that server sends it
currently files saved by a name guessed from URL

Hi,

After a disk full, Axel is sending the following message:

Initializing download: http://xyz/1GB.zip
File size: 1073741824 bytes
axel: axel.c:190: axel_open: Assertion `nread == sizeof( axel->conf->num_connections )' failed.
Aborted

IMO, the best way is show a 'Disk full' message.

Cheers,

Eriberto

• http://alioth.debian.org/ complains with error 406.

Considering the HTTPS and FTPS implementation[1] done by @sdt , Axel needs a special permission to linking with OpenSSL.

[1] master...sdt:https

So, I will use this issue to try formalise it.

I run ./autogen, I received:
"
configure.ac:46: warning: macro AM_GNU_GETTEXT' not found in library configure.ac:47: warning: macro AM_GNU_GETTEXT_VERSION' not found in library
autoreconf: configure.ac: AM_GNU_GETTEXT_VERSION is used, but not AM_GNU_GETTEXT
Done. You can use the 'clean' option to vanish the source code.
Example of use: $ ./autogen.sh clean
Now run ./configure, make, and make install.
"

and then, when I run ./configure, an error occurred as below.
"
checking for library containing ERR_get_error... none required
./configure: line 4316: syntax error near unexpected token external' ./configure: line 4316: AM_GNU_GETTEXT(external)'
"

I can only install gettext-0.17 by yum, so I manually installed gettext-0.18 followed github, and also autoconf-2.69.
Anyone can help me? Thanks a lot!!!

An example:

$ axel https://github.com/eribertomota/axel/archive/master.zip
Initializing download: https://github.com/eribertomota/axel/archive/master.zip
Too many redirects.

Is there someone able to help me with this issue?

Thanks in advance.

Regards,

Eriberto

seems axel 2.11 doesn't like cloudflare hosted HTTPS sites with ECDSA ciphers

axel --verbose https://centmin.sh/centminmodparts/ccache/ccache-3.3.2.tar.gz
Initializing download: https://centmin.sh/centminmodparts/ccache/ccache-3.3.2.tar.gz
SSL error: tlsv1 alert internal error

wget works fine

wget -O /dev/null https://centmin.sh/centminmodparts/ccache/ccache-3.3.2.tar.gz
--2016-10-04 21:03:18--  https://centmin.sh/centminmodparts/ccache/ccache-3.3.2.tar.gz
Resolving centmin.sh (centmin.sh)... 104.27.173.154, 104.27.172.154
Connecting to centmin.sh (centmin.sh)|104.27.173.154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 445945 (435K) [application/octet-stream]
Saving to: ‘/dev/null’

/dev/null                                                      100%[==================================================================================================================================================>] 435.49K   688KB/s    in 0.6s    

2016-10-04 21:03:20 (688 KB/s) - ‘/dev/null’ saved [445945/445945]

wget version

wget -V
GNU Wget 1.18 built on linux-gnu.

-cares +digest -gpgme +https +ipv6 +iri +large-file -metalink +nls 
+ntlm +opie -psl +ssl/openssl 

Wgetrc: 
    /usr/local/etc/wgetrc (system)
Locale: 
    /usr/local/share/locale 
Compile: 
    gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/usr/local/etc/wgetrc" 
    -DLOCALEDIR="/usr/local/share/locale" -I. -I../lib -I../lib -I 
    /usr/local/include -DHAVE_LIBSSL -DNDEBUG -O2 -g -pipe -Wall 
    -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong 
    --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic 
Link: 
    gcc -I /usr/local/include -DHAVE_LIBSSL -DNDEBUG -O2 -g -pipe -Wall 
    -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong 
    --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic 
    -L /usr/local/lib -lpcre -lssl -lcrypto -lz -lidn ftp-opie.o 
    openssl.o http-ntlm.o ../lib/libgnu.a