appimagecommunity / zsync2 Goto Github PK

View Code? Open in Web Editor NEW

126.0 126.0 23.0 412 KB

Rewrite of https://github.com/AppImage/zsync-curl, using modern C++, providing both a library and standalone tools.

License: Other

CMake 2.31% C++ 13.58% C 82.61% Shell 0.65% Python 0.63% Dockerfile 0.22%

cpp cpp11 update zsync

zsync2's People

Contributors

Stargazers

Watchers

zsync2's Issues

Hook into Open Build Service (OBS) similar as you do with Travis CI?

The owner of a GitHub repo can hook in additional external tools similar to hooking in Travis CI.
For this project it would be beneficial to hook in openSUSE's Open Build Service (OBS).
Go to the repo settings (here: https://github.com/AppImage/zsync2/settings), "Integration & services" to see what I mean.

OBS is listed as "Obs" in the drop-down list of available services when clicking the "Add service 'v'".

I cannot be of any help to actually set it up (or even to provide a pull request), because so far I failed to wrap my mind around the challenge to understand at all how OBS works and how to set it up.

But I guess you know all the beneficial features OBS can potentially provide:

Building not only .AppImage, but also RPM, DEB and other formats for lots of native distro package managers. This could be beneficial in order to increase visibility of the AppImage project and to remove some FUD which is spread by certain parties.
Building not only for x86_64, but also for i586, aarch64 and armvl7. The latter two are especially important in order to get AppImages built for ARM/Raspberry systems. (Currently there are only few ARM AppImages out there; these cannot be auto-updated, because tools like zsync2
and appimageupdatetool currently are not available for ARM.)

Current State

The README section "Current State" was committed Oct 2017:

zsync2/README.md

Lines 85 to 90 in e57e1fc

 ## Current State 

 Although zsync2 still shares a lot of the code with the original project, it is 

 as of now not functional. While debugging is ongoing, the API is somewhat ready 

 to be used by other projects. The project is therefore published in this state 

 to allow testing the integration into other projects.

I see in 2023 the Github releases are still marked with alpha, just checking in if the above statement is still unchanged since it was last updated in the README? Is the state looking better and more functional? Or is the alpha state likely to continue for N years?

Use consttime_memequal instead of memcmp.

From the legacy source files , In lib/librcksum/rsum.c at https://github.com/AppImage/zsync2/blob/f93e10ab4d6204a8ec64ca5909fcc23417dbdcf7/lib/librcksum/rsum.c#L154 and https://github.com/AppImage/zsync2/blob/f93e10ab4d6204a8ec64ca5909fcc23417dbdcf7/lib/librcksum/rsum.c#L238, You use memcmp to compare security critical data(i.e cryptographic hash ) which is not recommended.

From memcmp manual page ($ man memcmp for more information) , The manual explicitly states not to use memcmp for comparing critical data.

Do not use memcmp() to compare security critical data, such as cryptographic secrets, because the required CPU time depends on the number of equal bytes. Instead, a function that performs comparisons in constant time is required. Some operating systems provide such a function (e.g., NetBSD's consttime_memequal()), but no such function is specified in POSIX. On Linux, it may be necessary to implement such a function oneself.

Thus I think it would be nice if zsync2 uses consttime_memequal , But it is only available in NetBSD , Therefore we have to write it ourselves , this is the function from NetBSD's source files.

int
consttime_memequal(const void *b1, const void *b2, size_t len)
{
 const char *c1 = b1, *c2 = b2;
 int res = 0;

 while (len --)
  res |= *c1++ ^ *c2++;

 /*
  * If the compiler for your favourite architecture generates a
  * conditional branch for `!res', it will be a data-dependent
  * branch, in which case this should be replaced by
  *
  * return (1 - (1 & ((res - 1) >> 8)));
  *
  * or rewritten in assembly.
  */
 return !res;
}

The above function can be declared as a static function in the source file(rsum.c) and can be used.

IMPORTANT: consttime_memequal() returns 0 if the given memory contents are distinct and 1 if its the same.

consttime_memequal() manual: https://www.daemon-systems.org/man/consttime_memequal.3.html

[Performance] Honor "Connection: keep-alive"

Are we 100% sure that we honor "Connection: keep-alive"?

https://en.wikipedia.org/wiki/HTTP_persistent_connection

Installation configuration for CMake

Allow make install to be called, installing libraries and binaries into the appropriate directories. Could be used to e.g., package zsync2, or build an AppImage.

zlib1g-dev is an undocumented build dependency

Hi folks,

In my experience building this project, it fails without zlib1g-dev installed, but it succeeds once zlib1g-dev is installed. Might want to add that package to the apt-get install step in the README.

(currently, it's: sudo apt-get -y install git cmake g++ libssl-dev libssh2-1-dev libcurl4-gnutls-dev)

Happy to submit as a PR if desired.

Understand how to leverage zsync special handling of compressed files

Need to understand what http://zsync.moria.org.uk/ has implemented regarding

Handling for compressed files

rsync is ineffective on compressed files, unless they are compressed with a patched version of gzip. zsync has special handling for gzipped files, which enables update transfers of files which are distributed in compressed form.

The special handling of compressed files is, as far as I know, entirely new and unique to zsync. The combination of client-side rsync and HTTP is also unique to zsync, to the best of my knowledge.

Also need to understand if we can leverage the same for

Building the blocks for zsync2 but targeting our squashfs files plus the compression we are using there, and (informed by this understanding) potentially optimize the compression and block size used in our squashfs files
Draw conclusions for the chunking of files for peer-to-peer distributions (ideally: using the same blocks/chunking mechanism)

Action should only be taken after a discussion and ideally getting input from @whyrusleeping, too.

Provide Docker/Podman/apptainer images.

It would be nice to have zsync2 as Docker/Podman/apptainer images besides AppImage.

In an HPC environment, there is in general no option to install AppImage.

Singularity/Apptainer (http://apptainer.org/) and Podman (rootless Docker) is for example the only way to run containers on our HPC infrastructure. zsync2 is quite difficult to compile natively on old RedHat versions.

Docker/Podman and SingularityApptainer images are quite easy to make from the Apptainer image.

prepare_zsync2_rootfs.sh:

#!/bin/bash

# Ubuntu Bionic OCI core root image.
ubuntu_oci_root_image_url='https://partner-images.canonical.com/oci/bionic/current/ubuntu-bionic-oci-amd64-root.tar.gz'
ubuntu_oci_root_image_file='ubuntu-bionic-oci-amd64-root.tar.gz'

# Get version/commit from AppImage release links: https://github.com/AppImage/zsync2/releases
zsync2_version='2.0.0-alpha-1-20220602'
zsync2_commit='51-4e549b6'

zsync2_app_image_url="https://github.com/AppImage/zsync2/releases/download/${zsync2_version}/zsync2-${zsync2_commit}-x86_64.AppImage"
zsync2_app_image_file="zsync2-${zsync2_commit}-x86_64.AppImage"


# Download Ubuntu Bionic OCI core root image if not available or out of date.
if [ ! -e "${ubuntu_oci_root_image_file}" ] ; then
    curl -R -L -O "${ubuntu_oci_root_image_url}";
else
   # Only download Ubuntu OCI core root image, if there is a newer version.
   curl -R -L -O -z "${ubuntu_oci_root_image_file}" "${ubuntu_oci_root_image_url}";
fi

# Download zsync2 AppImage if not availabe.
if [ ! -e "${zsync2_app_image_file}" ] ; then
    curl -R -L -O "${zsync2_app_image_url}";
    
    chmod a+x "${zsync2_app_image_file}";
fi

rm -rf ./squashfs-root/

# Extract AppImage.
"./${zsync2_app_image_file}" --appimage-extract

# Fix symlink.
rm -r ./squashfs-root/usr/share/doc/libhogweed4
ln -s libnettle6 ./squashfs-root/usr/share/doc/libhogweed4

DockerFile

FROM scratch

ADD /squashfs-root/usr/ /usr
ADD ubuntu-bionic-oci-amd64-root.tar.gz /

CMD ["bash"]

zsync2.def : Singularity/Apptainer file.

Bootstrap: scratch

%setup
    cp -dR -n --preserve=mode,timestamps ./squashfs-root/usr "${APPTAINER_ROOTFS}/"
    tar xzf ubuntu-bionic-oci-amd64-root.tar.gz -C "${APPTAINER_ROOTFS}"

# Build with Docker.
docker build -t zsync-${zsync2_version}-x86_64 .

# Build with Podman.
podman build -t zsync-${zsync2_version}-x86_64 .

# Build with Singularity.
sudo singularity build "${zsync2_app_image_file%.AppImage}.sif" zsync2.def 

# Buld with Apptainer.
sudo apptainer build "${zsync2_app_image_file%.AppImage}.sif" zsync2.def

# Run with Docker/Podman.
podman run \
    --mount 'type=bind,source=/,destination=/local_filesystem' \
    --workdir=/local_filesystem/${PWD} \
    zsync-${zsync2_version}-x86_64 \
        zsyncmake2 -u https://test.com/bigfile.bin bigfile.bin

# Run with Singularity/Apptainer.
singularity exec \
    -B ${PWD} \
    ./zsync2-51-4e549b6-x86_64.sif \
        zsyncmake2 -u https://test.com/bigfile.bin bigfile.bin

We have files that are over 100G in some cases, so zsync2 would be nice to have to make sure the download is correct (and can be resumed).

Write rpm spec file for zsync2

We need to write a rpm spec file for libappimage so that it can be included in gnome-software CI which is based on Fedora. This issue is to keep track of the progress.

cc @Conan-Kudo

Windows support

Does it Works on windows? If not is there anything similar for windows?

Redundant HTTP requests for the .zsync file

Hi,

As mentioned in #33...

the first 1024 bits of the remote .zsyncfile are requested twice, and then the full .zsync file is potentially requested twice as well.

Parity with original zsync: "-i" flag prints an error and stops if the input file doesn't exist locally; Original zsync keeps going in this case

Steps to reproduce:

Use the -i flag with zsync2 and specify a file that does not exist on your hard-drive

Expected:

zsync2 can ignore the fact that the input file is missing and try to download anyway.

Actual:

This error message shows: Failed to open file [path-to-file/filename.extension] and zsync2 exits without downloading the file.

Note: original zsync handles this by basically ignoring the missing input, and tries to download the file anyway. So by changing zsync2's behavior, it could more-closely match the original zsync's behavior.

More details (click to expand!)

Command I ran:

zsync2 $URL -k ~/.zsyncs/$ZSYNCFILENAME -i ~/Downloads/$OUTPUTFILENAME -o ~/Downloads/$OUTPUTFILENAME

effectively:

zsync2 http://cdimage.ubuntu.com/ubuntu-mate/daily-live/pending/disco-desktop-amd64.iso.zsync -k ~/.zsyncs/disco-desktop-amd64.iso.zsync -i ~/Downloads/disco-desktop-amd64.iso -o ~/Downloads/disco-desktop-amd64.iso.desktop

It works the same if the URL for the remote .zsync file is at the end of the command...

zsync2 -k ~/.zsyncs/disco-desktop-amd64.iso.zsync -i ~/Downloads/disco-desktop-amd64.iso -o ~/Downloads/disco-desktop-amd64.iso.desktop http://cdimage.ubuntu.com/ubuntu-mate/daily-live/pending/disco-desktop-amd64.iso.zsync

Here was the full command output:

zsync2 version 2.0.0-alpha-1 (commit 7857ff1), build <local dev build> built on 2018-12-30 22:18:20 UTC
Checking for changes...
Cannot find file /home/[me]/Downloads/disco-desktop-amd64.iso, triggering full download
Storing copy of .zsync file in /home/[me]/.zsyncs/disco-desktop-amd64.iso.zsync, as requested
Target file: /home/[me]/Downloads/disco-desktop-amd64.iso
Reading seed file: /home/[me]/Downloads/disco-desktop-amd64.iso
Failed to open file /home/[me]/Downloads/disco-desktop-amd64.iso

More-elegant solution wanted for "-k" flag

Edit: The main bug has been fixed, so scroll down to here for where this is at now: #33 (comment)

(Original comment below.)

Hi folks.

Just filing a quick bug report...

Steps to Reproduce:

Have the full payload file downloaded onto your local hard-drive
- (If it isn't saved to the current directory, user can use the -i flag or -o flag to point zsync2 to where the payload is. Doesn't matter for this bug.)
Use the -k flag of zsync2, telling zsync2 to save the .zsync file locally

Expected Results

zsync2 should save the .zsync file to my local hard-drive if the -k flag is used.

Actual results

If zsync2 analyzes the payload on user's hard-drive, and sees that it is fully up-to-date, zsync2 will not save the .zsync file on the user's hard-disk.

Only 1.8% usable when updating from Firefox-63.0.3 to 64.0

To be analyzed.
Only 1.8% usable when updating from Firefox-63.0.3 to 64.0 - why?

me@host:~$ sudo Downloads/AppImageUpdate-x86_64.AppImage /isodevice/Applications/Firefox-63.0.3.glibc2.7-x86_64.AppImage 
AppImageUpdate version 1-alpha (commit 95cebb0), build 389 built on 2018-12-02 19:19:03 UTC
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
Updating from Bintray via ZSync
zsync2: Target file: /isodevice/Applications/Firefox-64.0.glibc2.3.4-x86_64.AppImage
zsync2: Reading seed file: /isodevice/Applications/Firefox-63.0.3.glibc2.7-x86_64.AppImage
zsync2: Usable data from seed files: 1.803200%

Files:
https://bintray.com/probono/AppImages/Firefox#files

support for proxy authentication

Currently, proxy authentication is not supported because of lack of support in CPR (libcpr/cpr/issues/248).

zsync2: failed to retrieve from VSCodium-1.59.0-1628208101.glibc2.17-x86_64.AppImage, status -1

Mirrored from AppImageLauncher fails to update VS Codium since the error message makes it seem like zsync2 is throwing this error:

zsync2: Downloading from https://github-releases.githubusercontent.com/144590939/293ea170-346b-48a6-8df0-d213e9e76b2c?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210810%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210810T130921Z&X-Amz-Expires=300&X-Amz-Signature=d5d54e732c72970fac41ceb266439fc51b65505af4291e4c1a60bba1df822b63&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=144590939&response-content-disposition=attachment%3B%20filename%3DVSCodium-1.59.0-1628208101.glibc2.17-x86_64.AppImage&response-content-type=application%2Foctet-stream
zsync2: optimized ranges, old requests count 78, new requests count 10

zsync2: failed to retrieve from VSCodium-1.59.0-1628208101.glibc2.17-x86_64.AppImage, status -1

Progress bar output broken

Something is wrong with the progress bar output:

Investigate zinc

I found this cool tool called zinc which is inspired from zsync but It does not have a backport to zsync , It also seems that it does not use rsum masks or reduced hash lengths of the strong hash , Which is the zsync algorithm and it is pretty efficient but not implemented. ( We could do something about that ).
But overall the author implemented everything that is needed for a delta update. So if it is possible, We could use libzinc as new specification of zsync2 and still keep the backport to the old zsync.

This is the project page: https://github.com/rokups/zinc/

Fix bundling of cURL in AppImages

Continuation of the discussion in #1.

.zs-old file left over

.zs-old file is left over when trying to download an already-downloaded file:

wget https://github.com/TheAssassin/zsync2/releases/download/continuous/zsync2-continuous-x86_64.AppImage
chmod +x zsync2-continuous-x86_64.AppImage 
./zsync2-continuous-x86_64.AppImage https://github.com/TheAssassin/zsync2/releases/download/continuous/zsync2-continuous-x86_64.AppImage.zsync
ls
# zsync2-continuous-x86_64.AppImage # good!
# zsync2-continuous-x86_64.AppImage.zs-old # bad!

Implement -u

As zsync2 is meant to be a drop-in replacement for zsync, it should support -u:

Use case (from AppImage/AppImageKit#519 (comment)):

if I'm on continuous and want to test the "dontDisconnectFacebook" channel, I could switch to that without having to download 80MB... I like that last example a lot because that's a very very frequent use case for our testers.

Test case:

# Get an AppImage
wget https://github.com/Subsurface-divelog/subsurface/releases/download/continuous/Subsurface-35c501677-x86_64.AppImage

# Say, you want to go to another (random) channel, like "travisTest2"
wget https://github.com/Subsurface-divelog/subsurface/releases/download/continuous-travisTest2/Subsurface-db64ec0af-x86_64.AppImage.zsync

# Let's see how well the original zsync fares
sudo apt -y install zsync
zsync -i Subsurface-35c501677-x86_64.AppImage Subsurface-db64ec0af-x86_64.AppImage.zsync 
URL 'Subsurface-db64ec0af-x86_64.AppImage' from the .zsync file is relative,
but I don't know the referer URL (you probably downloaded the .zsync separately
and gave it to me as a file). I need to know the referring URL (the URL of the .zsync)
in order to locate the download. You can specify this with -u
(or edit the URL line(s) in the .zsync file you have).

# But that also breaks; most likely due to zsnyc not handling GitHub's redirects properly
zsync -i Subsurface-35c501677-x86_64.AppImage Subsurface-db64ec0af-x86_64.AppImage.zsync -u https://github.com/Subsurface-divelog/subsurface/releases/download/continuous-travisTest2/
Read Subsurface-db64ec0af-x86_64.AppImage.part. Target 99.3% complete.      
failed to retrieve from Subsurface-db64ec0af-x86_64.AppImage
Aborting, download available in Subsurface-db64ec0af-x86_64.AppImage.part

# Now, let's try with zsnyc2
./zsync2-continuous-x86_64.AppImage -i Subsurface-35c501677-x86_64.AppImage Subsurface-db64ec0af-x86_64.AppImage.zsync -u https://github.com/Subsurface-divelog/subsurface/releases/download/continuous-travisTest2/
Flag could not be matched: 'u'

# Now, let's try with zsnyc2 without -u
./zsync2-continuous-x86_64.AppImage -i Subsurface-35c501677-x86_64.AppImage Subsurface-db64ec0af-x86_64.AppImage.zsync
Reading seed file: Subsurface-35c501677-x86_64.AppImage
Usable data from seed files: 99.301443%
Renaming temp file
Fetching remaining blocks
URL 'Subsurface-db64ec0af-x86_64.AppImage' from the .zsync file is relative,
but I don't know the referer URL (you probably downloaded the .zsync separately
and gave it to me as a file). I need to know the referring URL (the URL of the .zsync)
in order to locate the download. You can specify this with -u
(or edit the URL line(s) in the .zsync file you have).

Ability to check whether file requires update

Fix openGzFile

The implementation of openGzFile() is broken at the moment. Thanks to @pamarcos for bringing this up.

First of all, the use of fopencookie doesn't make sense, f isn't taken into account in the open/read/write/close functions, and if it went out of scope, there is no chance to access the file any more.

We'll have to use a few C++11 lambdas when creating the cookie_io_functions_t object, which basically call gz* appropriately, and store (copy) pointers to f in them. Then, we can clean it up properly.

Also, ideally, the function is designed so that we can easily drop in platform specific implementations for other OSes, like macOS or Windows.

EOF from serverOther error

Please see AppImageCommunity/AppImageUpdate#128.

CMAKE_DEBUG_POSTFIX leaking from lib/cpr/opt/curl

If and only if you are using the embedded curl from cpr,

these lines leak

lib/zsync2/lib/cpr/opt/curl/tests/libtest/CMakeLists.txt:    PROPERTIES DEBUG_POSTFIX "${CMAKE_DEBUG_POSTFIX}")
lib/zsync2/lib/cpr/opt/curl/src/CMakeLists.txt:  DEBUG_POSTFIX "${CMAKE_DEBUG_POSTFIX}")
lib/zsync2/lib/cpr/opt/curl/CMakeLists.txt:set(CMAKE_DEBUG_POSTFIX "-d" CACHE STRING "Set debug library postfix")
lib/zsync2/lib/cpr/opt/curl/lib/CMakeLists.txt:       DEBUG_POSTFIX "-d"

and hashlib gets suffixed with -d

if you have system curl dev package installed these never get triggered and hashlib is not suffixed with -d

slightly separate point, but why is the lib prefix removed from hashlib ? It makes it tricky to link against.

How

Okay let me put it simple

I have never heard of zsync2 but i wanted to update my AUR packages today and it said i cannot do so, because my local zsync2 is not an exact copy of this github repo.

I deleted my local zsync2 and cloned this repo, but then i saw that i cant even build it, because it's made for debian (according to the tutorial in the readme) and upon trying to build it, it failed.

I want to update my AUR through yay.
What should i do now?

==> ERROR: /home//.cache/yay/appimagelauncher/zsync2 is not a clone of https://github.com/TheAssassin/zsync2.git
Aborting...

[USER@MACHINE build]$ cmake .. -DUSE_SYSTEM_CURL=1 -DBUILD_CPR_TESTS=0
CMake Error at lib/CMakeLists.txt:27 (add_subdirectory):
  The source directory

    /home/USER/.cache/yay/appimagelauncher/zsync2/lib/cpr

  does not contain a CMakeLists.txt file.


CMake Error at lib/CMakeLists.txt:30 (set_property):
  set_property could not find TARGET cpr.  Perhaps it has not yet been
  created.


CMake Error at lib/CMakeLists.txt:42 (add_subdirectory):
  The source directory

    /home/USER/.cache/yay/appimagelauncher/zsync2/lib/gtest

  does not contain a CMakeLists.txt file.


-- Configuring incomplete, errors occurred!

zsyncmake2 should provide `--version` information (just as zsync2 already does)

zsync2 already supports returning version info:

$> ~/AppImages/zsync2-106-817978a-x86_64.AppImage --version
zsync2 version 2.0.0-alpha-1 (commit 817978a), build 106 built on 2018-01-03 05:04:22 UTC

zsyncmake2 instead says:

$> ~/AppImages/zsyncmake2-106-817978a-x86_64.AppImage --version
Flag could not be matched: version

   /home/kp/AppImages/zsyncmake2-106-817978a-x86_64.AppImage {OPTIONS} [filename]

[.... continues with general help options ....]

Oh, and once it does, it should of course also report this in its short help screen (i.e. when called with -h|--help).

zsyncmake2 is really slow (compared to zsyncmake) on a large file

Hello.

Thank you for this rewrite of zsync.

I have a large file of size 1.2GB which zsyncmake2 performs much worse than original zsyncmake on. Is this expected?

$ time zsyncmake2 FILE.tar
zsync2 version 2.0.0-alpha-1 (commit ), build <local dev build> built on 2023-03-08 11:03:16 UTC
No URL given, so I am including a relative URL in the .zsync file - you must keep the file being served and the .zsync in the same public directory. Use -u eikefet.tar to get this same result without this warning.
Warning: the given URL is relative. Please make sure the files are placed correctly on the server, otherwise zsync2 won't be able to resolve the path to the target file, requiring the user to specify this URL on the command line (using the -u flag).

real    2m56.362s
user    2m53.642s
sys     0m0.933s

$ time zsyncmake FILE.tar
No URL given, so I am including a relative URL in the .zsync file - you must keep the file being served and the .zsync in the same public directory. Use -u eikefet.tar to get this same result without this warning.

real    0m7,553s
user    0m6,375s
sys     0m1,151s

However, zsync2 is faster when redownloading the file.

$ time zsync2 URL/FILE.tar.zsync
zsync2 version 2.0.0-alpha-1 (commit ), build <local dev build> built on 2023-03-08 11:03:16 UTC
Checking for changes...
Using CA bundle found on system: /etc/ssl/certs/ca-certificates.crt
No changes detected, file is up to date.

real    0m4.886s
user    0m3.283s
sys     0m1.408s

$ time zsync URL/FILE.tar.zsync
Target FILE.tar
reading seed file FILE.tar: *************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************Read FILE.tar. Target 100.0% complete.      *************************************************************************************************************
verifying download...checksum matches OK
used 1261629440 local, fetched 0

real    0m24.847s
user    0m7.243s
sys     0m6.691s

Implement -v --verbose

Currently we can set an environment variable CURLOPT_VERBOSE=1 but one has to remember this. It would be more convenient if we could achieve the same by passing in -v or --verbose as a parameter.

Get into Debian and other distributions

Get this into Debian and other distributions since the zsync they are carrying seems to be quite outdated.

.netrc support

zsync2 does not offer .netrc support.

As a workaround, you can pass credentials directly in the URL like

zsync2 \
  --url=https://$USER:[email protected]/artifact.appimage \
  https://$USER:[email protected]/artifact.appimage.zsync

Any plans to support this?

Support for Web Linking

Part of AppImageCommunity/AppImageUpdate#57.

Some mirroring software like Mirrorbrain has support for Web Linking, specified by RFC 8288. With so-called Link: headers, a web server can provide alternative locations of a file. These headers are added to 302 responses, where the Location: header contains the primary location, and the Link: headers provide alternatives.

A header may look like this:

curl -v https://download.opensuse.org/repositories/home:/ocfreitag/AppImage/owncloud-client-latest-x86_64.AppImage
*   Trying [...]
> GET /repositories/home:/ocfreitag/AppImage/owncloud-client-latest-x86_64.AppImage HTTP/1.1
> Host: download.opensuse.org
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 302 Found
< Date: Fri, 09 Mar 2018 13:18:03 GMT
< Server: Apache/2.4.23 (Linux/SUSE)
< X-MirrorBrain-Mirror: ftp.gwdg.de
< X-MirrorBrain-Realm: country
< Link: <http://download.opensuse.org/repositories/home:/ocfreitag/AppImage/owncloud-client-latest-x86_64.AppImage.meta4>; rel=describedby; type="application/metalink4+xml"
< Link: <http://download.opensuse.org/repositories/home:/ocfreitag/AppImage/owncloud-client-latest-x86_64.AppImage.torrent>; rel=describedby; type="application/x-bittorrent"
< Link: <http://ftp.gwdg.de/pub/opensuse/repositories/home:/ocfreitag/AppImage/owncloud-client-2.3.3-6.1.Build6.3.glibc2.17-x86_64.AppImage>; rel=duplicate; pri=1; geo=de
< Link: <http://ftp2.nluug.nl/os/Linux/distr/opensuse/repositories/home:/ocfreitag/AppImage/owncloud-client-2.3.3-6.1.Build6.3.glibc2.17-x86_64.AppImage>; rel=duplicate; pri=2; geo=nl
< Link: <http://mirror.linux-ia64.org/opensuse/repositories/home:/ocfreitag/AppImage/owncloud-client-2.3.3-6.1.Build6.3.glibc2.17-x86_64.AppImage>; rel=duplicate; pri=3; geo=ru
< Link: <http://ftp.icm.edu.pl/pub/Linux/opensuse/repositories/home:/ocfreitag/AppImage/owncloud-client-2.3.3-6.1.Build6.3.glibc2.17-x86_64.AppImage>; rel=duplicate; pri=4; geo=pl
< Link: <http://ftp1.nluug.nl/os/Linux/distr/opensuse/repositories/home:/ocfreitag/AppImage/owncloud-client-2.3.3-6.1.Build6.3.glibc2.17-x86_64.AppImage>; rel=duplicate; pri=5; geo=nl
< Digest: MD5=U52XPoJzWp5XFb9yOBw7mA==
< Digest: SHA=bfnsmCC3MKDyUjfa7RdoEG0Qhbw=
< Digest: SHA-256=IkVw0N/V1hDi1HGwVLnCyFp/Fpr7mbWzTQjLgHEO8hI=
< Location: http://ftp.gwdg.de/pub/opensuse/repositories/home:/ocfreitag/AppImage/owncloud-client-2.3.3-6.1.Build6.3.glibc2.17-x86_64.AppImage
< Content-Length: 406
< Content-Type: text/html; charset=iso-8859-1
[...]

Links with rel=duplicate are the ones interesting for zsync2, since they are alternative locations (mirrors) of the file the file the Location: header redirects to.

I could imagine the following use cases in which these might come in handy:

temporary failure of the primary mirror
load balancing
- downloading parts from different servers -- less efficient, since one has to maintain different connections in parallel
- downloading from a random mirror -- probably not a good idea, one should trust Mirrorbrain to redirect to the most suitable mirror
- downloading from multiple mirrors in parallel -- at the moment, the download is performed chunk by chunk serially

Now, it's fairly easy to implement a parser for the Link: headers. The question is what to do with these data. I think as a beginning we should use them as fallbacks when an error occurs while fetching a chunk.

zsync{,make}2 updateinfo strings?

I downloaded zsyn{,make}2 AppImages from the AppImage/zsync2 GitHub realm:

https://github.com/AppImage/zsync2/releases/

The files show these updateinfo strings:

gh-releases-zsync|TheAssassin|zsync2|continuous|zsync2-*x86_64.AppImage.zsync
gh-releases-zsync|TheAssassin|zsync2|continuous|zsyncmake2-*x86_64.AppImage.zsync

Is this how it is intended? Shouldn't the two strings have TheAssassin replaced by AppImageKit?

Or is this a real-life test for the case where a seed file is on a different location as compared to the standard download location of the respective AppImage?

Investigate libp2p

Given that zsync2 is already doing hashing of blocks, how complicated would it be to add libp2p into the mix?

https://github.com/libp2p/specs

(Not asking to implement this yet, just asking to have a look at it to get a rough understanding what would be needed and what we would gain from it, especially when compared to just using ipfs as-is)

Segfaults

me@host:~$ sudo /home/me/Downloads/zsync2-137-bacc238-x86_64.AppImage -i /isodevice/boot/iso/custom-desktop-amd64.iso  https://github.com/probonopd/meilix/releases/download/continuous/custom-desktop-amd64.iso.zsync

zsync2 version 2.0.0-alpha-1 (commit bacc238), build 137 built on 2019-01-04 05:09:05 UTC
Checking for changes...

(Works for minutes, then...)

###################- 97.9% 8773.6 kBps         
###################- 97.9%Segmentation fault

Using bittorrent protocol with web seeding for possible P2P support for zsync2

conversation moved from #22 , To stay on topic.

Download from S3 link always stalled

Is this bug report : Yes

This issue is exactly same with probonopd/zsync-curl#15
Apparently that issue closed with the statement for changing zsync implementation to zsync2, but this issue still happens in this implementation

Misc

I somehow accidentally opened this issue report. Sorry for any inconvenience.

Checks for changes falsely report success

In several circumstances, checkForChanges() reports success although the update check should've had failed. Those are most likely caused due to failing/missing checks before calling the library functions from the legacy libzsync, as zsync_sha1() etc. work fine when the struct zsync_state has data set correctly.

Those issues need to be sorted out immediately.

[Suggestion] Improved code quality and feedback

Hi! I have a few suggestions for improving code quality and feedback:

Make the travis-ci badge show on README.md file;
Make coverage tests, integrate with https://codecov.io (free to opensource) and show the badge on README.md file too;
The codecov have a bot that shows up on pull requests that reduced the coverage tests, indicating issues that are hard to see on the code.
Integrate https://codacy.com (free to opensource) and show the badge on README.md file too;
The codacy will analyse the code and give a lot of information to improve the quality of the code.

The badges are just easy to see feedbacks.
If you don't like that kind of thing, then just ignore this suggestion and close this suggestion.

Replace zsync2's HTTP backend

We might have to switch away from CPR soon, as the project seems mostly dead, and doesn't compile on the latest distros any more. Also, the way it's used currently might not be very efficient, see #29.

If possible, we should try to use https://aria2.github.io/manual/en/html/libaria2.html instead. aria2 is a very intuitive tool to download files in parallel, with just a few flags, automagically. It's very configurable and flexible to use on the CLI, so perhaps its library libaria2 might be an option to implement more efficient downloading of the chunks.

rename: Invalid cross-device link

Upstream issue, continuing AppImageCommunity/AppImageUpdate#50.

The initial temporary files created by the legacy libzsync are created in the current working directory and not in the target directory (unlike the .part file which is managed by the C++ code in libzsync2). When trying to move, or actually, rename() this file, this might yield an invalid cross-device link error, unless one would copy the file and remove the old one like the mv command does in such cases.

Instead of doing this, the file should just be created in the right location, i.e., the target directory. That'll also allow for finding the temp files more easily after updates failed.

Relevant code:

https://github.com/TheAssassin/zsync2/blob/d7942eac257258d326c19fdc031f301a766abdda/lib/librcksum/state.c#L56-L57

This path needs to incorporate the target file path, determine the directory path, and prepend that. The path will have to be passed to the struct rcksum_state in the initializer, I guess.

0.0%Error while parsing headersOther error? -1

Using the latest continuous build on Deepin 15.4:

me@host:~$ chmod +x Downloads/zsync2-continuous-x86_64.AppImage 
me@host:~$ Downloads/zsync2-continuous-x86_64.AppImage https://github.com/TheAssassin/zsync2/releases/download/continuous/zsync2-continuous-x86_64.AppImage.zsync
https://github.com/TheAssassin/zsync2/releases/download/continuous/zsync2-continuous-x86_64.AppImage.zsync
Usable data from seed files: 0.000000%
Renaming temp file
Fetching remaining blocks
Downloading from https://github.com/TheAssassin/zsync2/releases/download/continuous/zsync2-continuous-x86_64.AppImage

-------------------- 0.0%Error while parsing headersOther error? -1
-1 returned
-------------------- 0.0% 0.0 kBps aborted    

failed to retrieve from zsync2-continuous-x86_64.AppImage, status -1

zsync_legacy fails on HTTP/2 response

Running under Arch Linux with zsync2 version 2.0.0-alpha-1 (commit 486fc9a), when attempting to download from an HTTP/2-enabled server zsync2 fails with the error:

zsync_legacy: got non-HTTP response 'HTTP/2 200

Verbose download output:

$ zsync2 -vvv -i archlinux-2021.03.01-x86_64.iso https://gitlab.archlinux.org/jonathon/archiso/-/jobs/16578/artifacts/raw/output/releng/archlinux-2021.03.04-x86_64.iso.zsync
zsync2 version 2.0.0-alpha-1 (commit 486fc9a), build <local dev build> built on 2021-03-02 22:59:06 UTC
Checking for changes...
Cannot find file archlinux-2021.03.04-x86_64.iso, triggering full download
/build/iso/archlinux-2021.03.04-x86_64.iso.part found, using as seed file
Target file: /build/iso/archlinux-2021.03.04-x86_64.iso
Reading seed file: /build/iso/archlinux-2021.03.04-x86_64.iso.part
Reading seed file: archlinux-2021.03.01-x86_64.iso
Usable data from seed files: 82.558900%
Renaming temp file
Fetching remaining blocks
Downloading from https://gitlab.archlinux.org/jonathon/archiso/-/jobs/16578/artifacts/raw/output/releng/archlinux-2021.03.04-x86_64.iso

################---- 82.6%*   Trying 2a01:4f8:c2c:5d2d::1:443...
* Connected to gitlab.archlinux.org (2a01:4f8:c2c:5d2d::1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=gitlab.archlinux.org
*  start date: Jan 23 16:51:10 2021 GMT
*  expire date: Apr 23 16:51:10 2021 GMT
*  subjectAltName: host "gitlab.archlinux.org" matched cert's "gitlab.archlinux.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55fe1e3cf790)
> GET /jonathon/archiso/-/jobs/16578/artifacts/raw/output/releng/archlinux-2021.03.04-x86_64.iso HTTP/2
Host: gitlab.archlinux.org
range: bytes=0-4095
accept: */*

* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
< server: nginx
< date: Fri, 05 Mar 2021 01:15:12 GMT
< content-type: application/x-iso9660-image
< content-length: 769994752
< cache-control: no-cache
< content-disposition: attachment; filename="archlinux-2021.03.04-x86_64.iso"
< set-cookie: experimentation_subject_id=eyJfcmFpbHMiOnsibWVzc2FnZSI6IklqYzVOakl6T1RNNExUUTVabUl0TkRZMlpTMWlOalEwTFRNM056aGxOak5oWVRsak5TST0iLCJleHAiOm51bGwsInB1ciI6ImNvb2tpZS5leHBlcmltZW50YXRpb25fc3ViamVjdF9pZCJ9fQ%3D%3D--8a9dd78ff3a947749b48451342aff6b2538164a0; path=/; expires=Tue, 05 Mar 2041 01:15:12 GMT; secure; HttpOnly; SameSite=None
< x-content-type-options: nosniff
< x-download-options: noopen
< x-frame-options: DENY
< x-gitlab-feature-category: continuous_integration
< x-permitted-cross-domain-policies: none
< x-request-id: 01EZZZ7VM16TTNAK13FXM97T7A
< x-runtime: 0.058740
< x-ua-compatible: IE=edge
< x-xss-protection: 1; mode=block
< strict-transport-security: max-age=31536000
< referrer-policy: strict-origin-when-cross-origin
< 
zsync_legacy: got non-HTTP response 'HTTP/2 200 
'

zsync_legacy: Other error? -1
* stopped the pause stream!
* Connection #0 to host gitlab.archlinux.org left intact
failed to retrieve from archlinux-2021.03.04-x86_64.iso, status -1

Misleading message "cannot find"

me@host:~$ Downloads/zsync2-137-bacc238-x86_64.AppImage --seed-file=/isodevice/boot/iso/custom-desktop-amd64.iso  https://github.com/probonopd/meilix/releases/download/continuous/custom-desktop-amd64.iso.zsync
zsync2 version 2.0.0-alpha-1 (commit bacc238), build 137 built on 2019-01-04 05:09:05 UTC
Checking for changes...
Cannot find file custom-desktop-amd64.iso, triggering full download
Target file: /home/me/custom-desktop-amd64.iso
Reading seed file: /isodevice/boot/iso/custom-desktop-amd64.iso

Here the message

Cannot find file custom-desktop-amd64.iso, triggering full download

is misleading. It took me a while to understand that after this message appears it still finds the seed file:

Reading seed file: /isodevice/boot/iso/custom-desktop-amd64.iso

Hence, the first message should be changed to say "Could not find pre-existing original..." or something like that. Or the message should not appear at all if at least one seed file is given.

Missing Build Dependency

When attempting to build and install this on Ubuntu 22.04 you will need to install the libgcrypt20-dev package so the build will be successful. If you could update the README.md to reflect that, it might help people in the future.

Move permissions logic from GUI to libzsync2

Continuation of AppImageCommunity/AppImageUpdate#39.

zsync2 does not support downloading large files.

zsync2 does not support downloading large files.
failed to parse content-range headerError while parsing headersOther error? -1

I patched zsync2 so it shows to and from values:

$ git diff -U10
diff --git a/src/legacy_http.c b/src/legacy_http.c
index 41310da..290603e 100644
--- a/src/legacy_http.c
+++ b/src/legacy_http.c
@@ -626,20 +626,21 @@ int range_fetch_read_http_headers(struct range_fetch *rf) {
             p[len] = 0;
         }
         /* buf is the header name (lower-cased), p the value */
         /* Switch based on header */
 
         if (status == 206 && !strcmp(buf, "content-range")) {
             /* Okay, we're getting a non-MIME block from the remote. Get the
              * range and set our state appropriately */
             int from, to;
             sscanf(p, "bytes " OFF_T_PF "-" OFF_T_PF "/", &from, &to);
+            fprintf(stderr, "content-range from: %d  to: %d\n", from, to);
             if (from <= to) {
                 rf->block_left = to + 1 - from;
                 rf->offset = from;
             } else {
                 fprintf(stderr, "failed to parse content-range header");
             }
 
             /* Can only have got one range. */
             rf->rangesdone++;
             rf->rangessent = rf->rangesdone;

$ ./zsync2 -v https://resources.aertslab.org/cistarget/databases/homo_sapiens/hg19/refseq_r45/mc8nr/region_based/hg19-regions-9species.all_regions.mc8nr.feather.zsync
zsync2 version 2.0.0-alpha-1 (commit 7857ff1), build <local dev build> built on 2018-12-21 09:58:27 UTC
Checking for changes...
Cannot find file hg19-regions-9species.all_regions.mc8nr.feather, triggering full download
/ddn1/vol1/site_scratch/leuven/303/vsc30366/hg19-regions-9species.all_regions.mc8nr.feather.part found, using as seed file
Target file: /ddn1/vol1/site_scratch/leuven/303/vsc30366/hg19-regions-9species.all_regions.mc8nr.feather
Reading seed file: /ddn1/vol1/site_scratch/leuven/303/vsc30366/hg19-regions-9species.all_regions.mc8nr.feather.part
Usable data from seed files: 0.000000%
Renaming temp file
Fetching remaining blocks
Downloading from https://resources.aertslab.org/cistarget/databases/homo_sapiens/hg19/refseq_r45/mc8nr/region_based/hg19-regions-9species.all_regions.mc8nr.feather

-------------------- 0.0%* Hostname was NOT found in DNS cache
*   Trying 134.58.50.8...
* Adding handle: conn: 0x1654a00
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 3 (0x1654a00) send_pipe: 1, recv_pipe: 0
* Connected to resources.aertslab.org (134.58.50.8) port 443 (#3)
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using ECDHE-RSA-AES256-GCM-SHA384
* Server certificate:
* 	 subject: CN=resources.aertslab.org
* 	 start date: 2018-11-25 04:49:48 GMT
* 	 expire date: 2019-02-23 04:49:48 GMT
* 	 subjectAltName: resources.aertslab.org matched
* 	 issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
* 	 SSL certificate verify ok.
> GET /cistarget/databases/homo_sapiens/hg19/refseq_r45/mc8nr/region_based/hg19-regions-9species.all_regions.mc8nr.feather HTTP/1.1
Range: bytes=0-3475369983
Host: resources.aertslab.org
Accept: */*

< HTTP/1.1 206 Partial Content
< Date: Fri, 21 Dec 2018 10:09:34 GMT
* Server Apache/2.4.37 (Ubuntu) is not blacklisted
< Server: Apache/2.4.37 (Ubuntu)
< Strict-Transport-Security: max-age=15768000
< Last-Modified: Wed, 23 May 2018 07:38:22 GMT
< ETag: "16cf25e760-56cda9e8f304e"
< Accept-Ranges: bytes
< Content-Length: 3475369984
< Content-Range: bytes 0-3475369983/97964648288
< 
content-range from: 0  to: -819597313
failed to parse content-range headerError while parsing headersOther error? -1
-1 returned
-------------------- 0.0% 0.0 kBps aborted    

* Closing connection 3
failed to retrieve from hg19-regions-9species.all_regions.mc8nr.feather, status -1

As you can see int (signed int) is not big enough, from and to should be uint (unsigned int) (at least 32 bits).

Inform about the "new" file name after updating

Use case:

me@host:~$ ls /isodevice/Applications/QMediathekView-*
/isodevice/Applications/QMediathekView-0951c25-x86_64.AppImage
/isodevice/Applications/QMediathekView-0951c25-x86_64.AppImage.part
/isodevice/Applications/QMediathekView-18dd68f-x86_64.AppImage
/isodevice/Applications/QMediathekView-18dd68f-x86_64.AppImage.part
/isodevice/Applications/QMediathekView-bdd3b30-x86_64.AppImage
/isodevice/Applications/QMediathekView-d63852a-x86_64.AppImage

When I run an update on any of these, I want to know what the new filename is...

zsync2: Verifying downloaded file
zsync2: checksum matches OK
zsync2: used 26968064 local, fetched 249416

should be changed to

zsync2: Verifying downloaded file the/path/to/the/new/file <-- so that we know what the new file is
zsync2: checksum matches OK
zsync2: used 26968064 local, fetched 249416

non-versioned continuous release

Please at least remove the commit / version info from the assets so the links are stable and do not break with every new release.

	## Current State

	Although zsync2 still shares a lot of the code with the original project, it is
	as of now not functional. While debugging is ongoing, the API is somewhat ready
	to be used by other projects. The project is therefore published in this state
	to allow testing the integration into other projects.