Tarsnap currently stores hardlinked files as "link" archive entries without data. For example, if you have

# echo foo > a
# ln a b
# tarsnap -c -f foo a b

then the archive entry for 'a' will contain "foo", while the archive entry for 'b' will simply say "this is a hardlink to 'a'". Unfortunately, this doesn't interact well with selective extracts:

# tarsnap -x -f foo b
b: Can't create 'b'

This is in fact the identical behaviour to bsdtar: By the time it gets to the archive entry for 'b', it's too late to get the data from the 'a' archive entry (because we're dealing with a streaming archive format and bsdtar/libarchive are designed around that notion).

Tarsnap however should be able to do better. Since we're deduplicating, including the file data in every hardlinked archive entry is really cheap. We should probably do this.

Not the intended behaviour:

$ tarsnap --dry-run --print-stats --humanize-numbers -c /
tarsnap: stat(/root/tarsnap.key): No such file or directory
tarsnap: Cannot read key file: /root/tarsnap.key

I think in the command line processing we should record whether a key file or a cache directory were specified on the command line; if not (e.g., we have a keyfile set, but it wasn't done on the command line) then we need to add fallback mechanisms for dry runs.

I can't remember the details of cache directories enough to say exactly what needs to be done there without looking at the code, but I think there may also need to be some special "ignore the value which was set in a configuration file" handling there too.

writetape_close: in flushtape, if chunklist_exportdup returns true, d->h.index is freed and flushtape returns -1 , after err2 it does stream_free(&d->h) which is chunklist_free(S->index)

I'm not suggesting you need to include git hashes; these are just so that you can look up the commit quickly if you need to refresh your memory

• fix misc memory leaks given unusual failure conditions (edfd943, 0df0b6f)
• warn if user attempts to place options at the end of the command line (c3c383e)
• on 32-bit and smaller machines, an attacker who can store a validly HMACed tarsnap block can make tarsnap crash upon listing archives or reading an archive the attacker created; on larger machines, that attacker can make tarsnap waste a lot of memory. (77751c1)
• add AESNI support (45efe0c)
• add better attempt at zeroing a buffer "insecure_memzero" (a847930)
• add --retry-forever (c5d4a51)
• Support listing archives with files >68 years old (ebd4ece)
  [probably not an important issue, but it would be funny to see that in the release notes!]
• fix for builds containing whitespace, and out-of-source-tree builds (a19cdf3, and later commits)
• add SSE2 support (527e7a8)
• add tarsnap bash completion (49eedde)
• allow uses to test dedupe&encryption without signing up (--dry-run compression without a keyfile 92d5c36)
• disable --aggressive-networking if --maxbw-rate-* (9478f36)
• add --creationtime (da68672)

It's just possible that somebody might be relying on the current behaviour, so this will wait for tarsnap 1.1. For now, we just print a warning.

Patch ready in 221949b.

First search result also points to non-https version of site
https://www.google.com/search?q=tarsnap

Maybe change
<link rel="canonical" href="http://www.tarsnap.com/">
to
<link rel="canonical" href="https://www.tarsnap.com/">

The INSTALL documentation (lol which ?) should say that you need

apt-get install e2fslibs-dev zlib1g-dev libssl-dev

otherwise you may get (or some such)

checking for ext2fs/ext2_fs.h... no
configure: error: *** ext2fs/ext2_fs.h missing ***

It would be nice if we could do

tarsnap --nuke
tarsnap -c -f whatever ~/whatever

without having a tarsnap --fsck in between those commands. (on a slightly related note, it would be nice if --nuke and --fsck finished quicker if there was no data in tarsnap; right now fsck takes 10 seconds. Although if fsck is no longer required, that part is moot (or at least mooter)).

../configure --prefix=$HOME/.local/ --with-bash-completion-dir

produces a Makefile containing

BASH_COMPLETION_DIR = /usr/share/bash-completion/completions

This obviously causes make install to fail. :(

If a user has to run --fsck because of local filesystem or disk corruption, there's a nontrivial chance that the chunkification cache will also be damaged. Since that cache exists purely as an optimization, and fsck isn't something people should run very often, maybe we should delete /cache just in case.

When running tarsnap --print-stats or tarsnap -d, multiple -f options can be specified. It might be a good idea to have a --keep-going option which makes tarsnap continue if one of the archives specified does not exist.

Use case: Trying to create daily archives, but sometimes the archives don't get created (e.g., laptop + cron job). Want to delete a set of old archives without worrying about not deleting the ones which aren't there in the first place.

Another use case: Deleting a list of archives, network glitches in the middle. Want to restart the deletes and skip the ones which were already deleted.

Currently the writetape_open (in tar/multitape/multitape_write.c) gets the archive creation time by calling time(NULL). For the benefit of software built around tarsnap (e.g., the GUI) we should accept a new --creationtime <posixtime> option which overrides this value.

In terms of code organization, since main() is already checking the time, it may make sense to set the creation time there (modulo override) rather than conditionally checking the time again later.

Estimate: 1 hour, 20 lines of code spread between several files.

if I'm typing

 tarsnap --dry-run --c -f test-dry-run --humaniz<TAB>

it would be nice if the shell auto-completed --humanize-numbers because I can never remember if tarsnap uses the Oxford 'z' or not. A few quick googles suggest that this is done (for bash) by adding a script to

/etc/bash_completion.d/

in Debian-based distros. So this might be nice to add to the binary packages.

@cperciva did warn of fallout from #170, but I didn't imagine it would be like this...

freebsd 10.3-RELEASE-p4

$ make
...
In file included from ../lib/util/getfstype.c:7:
In file included from /usr/include/sys/mount.h:36:
/usr/include/sys/ucred.h:83:2: error: unknown type name 'u_int'
        u_int   cr_version;             /* structure layout version */

This is because u_int is only defined in sys/types.h if __BSD_VISIBLE is defined, and _POSIX_C_SOURCE stops __BSD_VISIBLE from being defined.

There's an existing PR and discussion from 2011:
https://lists.freebsd.org/pipermail/freebsd-bugs/2011-April/044044.html
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=156637

wollman wrote:

I'd say it's the application code that is in error. It should not be defining _XOPEN_SOURCE and then including (implementation private) header files which are not defined in the relevant standard.

Do we seriously need to start writing our headers like:

include <sys/cdefs.h>

ifndef __BSD_VISIBLE

error "This is a non-standard header, but you have specified strict standard compliance."

endif

(in our case, we're definined _POSIX_C_SOURCE rather than _X_OPEN_SOURCE, but it's the same issue)

The chunkification cache allows tarsnap to avoid re-reading and re-chunkifying files if they haven't changed since the last time they were archived. This provides a huge speedup and is a very good thing.

It could be used for another purpose too: If we identify that a file shouldn't have changed, we could read it anyway, and make sure that it hasn't changed. This might allow us to warn users about failing disks before they would otherwise notice.

Obviously for performance reasons we wouldn't want to do this all the time. Maybe we can do this for a random subset of files though, and get the "scrubbing" benefit without adding very much extra I/O and CPU cost?

The siginfo_printinfo() function in tar/siginfo.c should obey the --humanize-numbers command.

We should probably change the storage_write_start API from "If ${dryrun} is nonzero, perform a dry run (which ignores ${lastseq} and sets ${seqnum} to 32 0s)." to "If ${dryrun} is nonzero, ignore ${lastseq} and ${seqnum}". Or possibly push dryrun resolution higher in the stack even -- don't create a storage cookie at all, but instead just have the multitape and chunks layers be aware of dryruns and not issue calls to the storage layer?

But storage_write_start is doing a silly amount of pointless initialization given that a dry-run storage cookie should never do anything. There's got to be a better way to organize this code.

tarsnap built from deb package, newly registered machine, first run of tarsnap:

$ tarsnap -cf foo ~
tarsnap: Removing leading '/' from member names
....
^C

(Set nodump flag on really big dir that I didn't want to back up)

$ tarsnap -cf foo ~
tarsnap: An archive already exists with the name "foo.part"
tarsnap: Error creating new archive
tarsnap: Error exit delayed from previous errors.

$ tarsnap --list-archives
foo.part

$ tarsnap -df foo.part
					Total size  Compressed size
All archives				18 EB            18 EB
  (unique data)				18 EB            18 EB
This archive				1.0 GB           997 MB
Deleted data				1.0 GB           997 MB

$ tarsnap -cf bar ~
tarsnap: Removing leading '/' from member names
tarsnap: Not adding cache directory to archive: /home/user/tarsnap_cache
					Total size  Compressed size
All archives				124 MB           100 MB
  (unique data)				123 MB           100 MB
This archive				124 MB           100 MB
New data					123 MB           100 MB

$ tarsnap --print-stats
					Total size  Compressed size
All archives				124 MB           100 MB
  (unique data)				123 MB           100 MB

Is 18EB just a placeholder value for when we don't know just how much data we have, or am I about to get a really big bill? 😃

Like all good UNIX utilities, tarsnap runs quietly unless it has a reason to be noisy (e.g., an error occurred, or the user asked for verbosity). Unfortunately this is provoking a FAQ: "Tarsnap doesn't seem to be doing anything, did it hang?"

I think the solution to this might simply be better documentation, but I'm creating the issue here in case anyone wants to chime in with suggestions.

Nitpicky build failure on freebsd 10.3-RELEASE-p4. A normal make works just fine; doing clang -Weverything -Werror -Wno-a-bunch-of-specific-warnings produces this:

../libcperciva/util/monoclock.c:41:6: error: implicit declaration of function
      'gettimeofday' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        if (gettimeofday(tv, NULL)) {

gettimeofday() is hidden inside #if __XSI_VISIBLE in /usr/include/sys/time.h. This isn't necessarily a huge problem since:

Applications should use the clock_gettime() function instead of the obsolescent gettimeofday() function.
http://pubs.opengroup.org/onlinepubs/9699919799/functions/gettimeofday.html

and monoclock.c tries to use clock_gettime() first. However, in order to work around OSX failing:

All implementations shall support a clock_id of CLOCK_REALTIME as defined in <time.h>
http://pubs.opengroup.org/onlinepubs/9699919799/functions/clock_gettime.html

We defined -DPOSIXFAIL_CLOCK_REALTIME in Makefile.am in 00bf976.

There's obviously a few solutions we could use here:

• enable XSI stuff in the Tarsnap build.
• use more intelligent checking to see whether we need -DPOSIXFAIL_CLOCK_REALTIME or not.
• fancier code in monoclock.c. For example, scrypt selects clocktouse by going through CLOCK_VIRTUAL, CLOCK_MONOTONIC, CLOCK_REALTIME, and if all those fail, it bails. If that works on OSX, maybe we could do a similar thing in monoclock.c?
• stop testing with -Weverything -Werror. :)

Suggestion from a Tarsnap user:

I’m a rather new but very happy user of tarsnap, and I have started pushing backups all over the place. To do this I’m using a git tree that holds configuration both of tarsnap proper, and that of feather that I use to schedule backups. In this context I would like to be able to run tarsnap with the single purpose of verifying the validity of the configuration file, so I can have tools prevent broken (well, syntactically broken) configuration files from being pushed to clients. Something as simple as: tarsnap --verify-config

We won't want this to load tarsnap keys, since we might want to check a configuration file on a system without keys installed. So the handling of the --keyfile option will need to change from "load the keys" to "store the path and load the keys later". But that should be fairly straightforward, and I don't think there are any other similar issues.

Then we should be able to check this option and bail out after the configuration file(s) are processed.

We need a way to tell libarchive that we've added padding to a block ourselves. Right now libarchive wants to add padding itself, so we can't make use of cached chunks which contain the padding.

It would be nice if, in addition to storing archives like tar, tarsnap could store arbitrary streams of data. This would allow things like " | tarsnap --raw-write" to work.

Since the stream would not have parts tagged as "tar headers" vs "tar data" and there would be no backing filesystem metadata to permit use of the chunkification cache, running "tar | tarsnap --raw-write" would be slower and less efficient than directly running tarsnap, so this would be inherently an added feature, not in any sense a replacement for existing functionality.

Some notes:

• I'm not sure what to call this. -w would have been nice, to match the -r option which outputs data, but that's used to request confirmation for each action. (Theoretically we could distinguish between tarsnap -w and tarsnap -[cx] -w, but that seems prone to user error.) So maybe --raw-write? Or possibly -c --raw?

• Archives need to be somehow flagged as being raw data, so that -[xt] are disabled. The obvious place to do this would be in archive metadata, but I can't see any way to do that without resulting in the archives being rejected as corrupt by earlier versions of tarsnap, and I'm not sure we want to do that. An alternative would be to put a header onto the multitape-layer data; then have -r look for and strip that header and make sure that the tar code which -[xt] uses will throw an error ("not a tar archive" or something like that).

In relation to 4640c06:

s/warnp/warn0/ for NETWORK_STATUS_CONNERR errors.

Originally we could rely on errno being meaningful at this
point, but with the switch to using libcperciva's network_connect
there's a possibility of getting a delayed failure callback since
network_connect sometimes queues an immediate callback rather than
making a direct call, in order to avoid problems with unbounded
recursion.

At some point in the future this might change back;
libcperciva would need to save and restore errno appropriately
first.  (Junior hacker task!)

Use case: Running tarsnap-keymgmt to create a super-secure encrypted key file because you're going to be printing it out and storing it offline.

Suggested by: Hugo Landau

From http://mail.tarsnap.com/tarsnap-users/msg00501.html:

I should probably add a --resume-extract option which only skips files if their
metadata matches the archive.

We can't use --keep-newer-files to "resume" an extract because it will skip a file which was half-extracted. If we check the modification time and the file size we'll catch this.

wget https://www.tarsnap.com/download/tarsnap-autoconf-1.0.37.tgz
tar xzvf  tarsnap-autoconf-1.0.*.tgz
cd tarsnap*
ln -s pkg/debian/
dpkg-buildpackage -rfakeroot -d -us -uc -sa -i -S
# produces: dpkg-source: error: can't build with source format '3.0 (quilt)':
#       no upstream tarball found at ../tarsnap_1.0.37.orig.tar.{bz2,gz,lzma,xz}

That's because you are not providing a diff & original source, but rather a Debian native package.

Workaround:

rm pkg/debian/source/format

Googling finds similar problems in other packages, e.g.:
SoftEtherVPN/SoftEtherVPN#90

Problem introduced by commit 09e7704

i reported this to admin 8 days ago but no reply

email address: [email protected]

Hello TarSnap Security Team

My name Is Shivbihari Pandey

Bug:
1.Poor Password Policy
2.Authentication Problem
3. Poor Encryption in the Web Browser

POC Video Attached
Link Here:

https://drive.google.com/file/d/0B8AftaXwHcwdZkRUa1d6OUhYbWs/view?usp=sharing

1.Poor Password Policy
Steps:

1. create an account
2. goto email and verify the account and set the password as " "(space)

3.now come to login section..now login with space as password

Boom you just logged In to account

actually if user choose this type of password then attacker can easily bruteforce it..and get the whole account

2.Authentication Problem

steps:

1.Login to 2 Browser With Same Account
2.Now from one Browser Just Change the password
3. Come to another account and just refresh the page ..you will see the
account still remain logged in

i test it ..token not expire even the password changed..

if someone get hacked or he left his account in the other PC
then even he change the password..but the account remain logged in to attacker PC

hope you get my point :P

1. Poor Encryption in the Web Browser

https://www.tarsnap.com/manage.cgi?address=[User Email]&token=b1e6b876a00e152ec52407e00a365d507cf004d0a2f25268272a95a6b9add237&action=payment

in the logged in account the account token and email not encrypted

Regard

ShivBihari Pandey

(Independent Security Researcher)

Building tarsnap 1.0.36 with OS X 10.10.5 and Xcode tools 6.4, the build encounters an error.

The file libcperciva/util/monoclock.c appears to be calling clock_gettime which may not exist on OS X:

make output

$ make all install clean
/Applications/Xcode.app/Contents/Developer/usr/bin/make  all-am
gcc -DHAVE_CONFIG_H -I.    -I./tar -I./libarchive -I./lib/crypto -I./lib/datastruct -I./lib/keyfile -I./lib/netpacket -I./lib/netproto -I./lib/network -I./lib/scryptenc -I./lib/util -I./libcperciva/alg -I./libcperciva/cpusupport -I./libcperciva/crypto -I./libcperciva/datastruct -I./libcperciva/events -I./libcperciva/network -I./libcperciva/util -I. -D CPUSUPPORT_CONFIG_FILE=\"cpusupport-config.h\" -DPOSIXFAIL_MSG_NOSIGNAL -g -O2 -MT libcperciva/util/lib_libtarsnap_a-monoclock.o -MD -MP -MF libcperciva/util/.deps/lib_libtarsnap_a-monoclock.Tpo -c -o libcperciva/util/lib_libtarsnap_a-monoclock.o `test -f 'libcperciva/util/monoclock.c' || echo './'`libcperciva/util/monoclock.c
libcperciva/util/monoclock.c:32:6: warning: implicit declaration of function 'clock_gettime' is
      invalid in C99 [-Wimplicit-function-declaration]
        if (clock_gettime(CLOCK_REALTIME, &tp) == 0) {
            ^
libcperciva/util/monoclock.c:32:20: error: use of undeclared identifier 'CLOCK_REALTIME'
        if (clock_gettime(CLOCK_REALTIME, &tp) == 0) {
                          ^
1 warning and 1 error generated.
make[1]: *** [libcperciva/util/lib_libtarsnap_a-monoclock.o] Error 1
make: *** [all] Error 2

compiler

$ gcc -v
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 6.1.0 (clang-602.0.53) (based on LLVM 3.6.0svn)
Target: x86_64-apple-darwin14.5.0
Thread model: posix

configure output

$ ./configure --prefix='/Volumes/Documents/tarsnap'
checking build system type... x86_64-apple-darwin14.5.0
checking host system type... x86_64-apple-darwin14.5.0
checking target system type... x86_64-apple-darwin14.5.0
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... config.aux/install-sh -c -d
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking for style of include used by make... GNU
checking dependency style of gcc... gcc3
checking for ranlib... ranlib
checking for strip... strip
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking minix/config.h usability... no
checking minix/config.h presence... no
checking for minix/config.h... no
checking whether it is safe to define __EXTENSIONS__... yes
checking whether to enable maintainer-specific portions of Makefiles... no
checking for ANSI C header files... (cached) yes
checking for dirent.h that defines DIR... yes
checking for library containing opendir... none required
checking for sys/wait.h that is POSIX.1 compatible... yes
checking acl/libacl.h usability... no
checking acl/libacl.h presence... no
checking for acl/libacl.h... no
checking attr/xattr.h usability... no
checking attr/xattr.h presence... no
checking for attr/xattr.h... no
checking ctype.h usability... yes
checking ctype.h presence... yes
checking for ctype.h... yes
checking err.h usability... yes
checking err.h presence... yes
checking for err.h... yes
checking errno.h usability... yes
checking errno.h presence... yes
checking for errno.h... yes
checking fcntl.h usability... yes
checking fcntl.h presence... yes
checking for fcntl.h... yes
checking grp.h usability... yes
checking grp.h presence... yes
checking for grp.h... yes
checking for inttypes.h... (cached) yes
checking langinfo.h usability... yes
checking langinfo.h presence... yes
checking for langinfo.h... yes
checking limits.h usability... yes
checking limits.h presence... yes
checking for limits.h... yes
checking linux/fs.h usability... no
checking linux/fs.h presence... no
checking for linux/fs.h... no
checking locale.h usability... yes
checking locale.h presence... yes
checking for locale.h... yes
checking paths.h usability... yes
checking paths.h presence... yes
checking for paths.h... yes
checking poll.h usability... yes
checking poll.h presence... yes
checking for poll.h... yes
checking pwd.h usability... yes
checking pwd.h presence... yes
checking for pwd.h... yes
checking regex.h usability... yes
checking regex.h presence... yes
checking for regex.h... yes
checking signal.h usability... yes
checking signal.h presence... yes
checking for signal.h... yes
checking stdarg.h usability... yes
checking stdarg.h presence... yes
checking for stdarg.h... yes
checking for stdint.h... (cached) yes
checking for stdlib.h... (cached) yes
checking for string.h... (cached) yes
checking sys/acl.h usability... yes
checking sys/acl.h presence... yes
checking for sys/acl.h... yes
checking sys/extattr.h usability... no
checking sys/extattr.h presence... no
checking for sys/extattr.h... no
checking sys/ioctl.h usability... yes
checking sys/ioctl.h presence... yes
checking for sys/ioctl.h... yes
checking sys/mkdev.h usability... no
checking sys/mkdev.h presence... no
checking for sys/mkdev.h... no
checking sys/param.h usability... yes
checking sys/param.h presence... yes
checking for sys/param.h... yes
checking sys/poll.h usability... yes
checking sys/poll.h presence... yes
checking for sys/poll.h... yes
checking sys/select.h usability... yes
checking sys/select.h presence... yes
checking for sys/select.h... yes
checking sys/time.h usability... yes
checking sys/time.h presence... yes
checking for sys/time.h... yes
checking sys/utime.h usability... no
checking sys/utime.h presence... no
checking for sys/utime.h... no
checking time.h usability... yes
checking time.h presence... yes
checking for time.h... yes
checking for unistd.h... (cached) yes
checking utime.h usability... yes
checking utime.h presence... yes
checking for utime.h... yes
checking wchar.h usability... yes
checking wchar.h presence... yes
checking for wchar.h... yes
checking wctype.h usability... yes
checking wctype.h presence... yes
checking for wctype.h... yes
checking windows.h usability... no
checking windows.h presence... no
checking for windows.h... no
checking for clock_gettime in -lrt... no
checking for clock_gettime... no
checking sys/sysinfo.h usability... no
checking sys/sysinfo.h presence... no
checking for sys/sysinfo.h... no
checking for sysinfo... no
checking for struct sysinfo... no
checking for struct sysinfo.totalram... no
checking for struct sysinfo.mem_unit... no
checking for sys/mount.h... yes
checking for sys/statfs.h... no
checking for sys/statvfs.h... yes
checking for sys/vfs.h... no
checking sys/sysctl.h usability... yes
checking sys/sysctl.h presence... yes
checking for sys/sysctl.h... yes
checking for struct statfs.f_fstypename... yes
checking for struct statfs.f_type... yes
checking for struct statvfs.f_fstypename... no
checking for struct statvfs.f_basetype... no
checking linux/magic.h usability... no
checking linux/magic.h presence... no
checking for linux/magic.h... no
checking zlib.h usability... yes
checking zlib.h presence... yes
checking for zlib.h... yes
checking for inflate in -lz... yes
checking openssl/rsa.h usability... yes
checking openssl/rsa.h presence... yes
checking for openssl/rsa.h... yes
checking for RSA_private_encrypt in -lcrypto... yes
checking bzlib.h usability... yes
checking bzlib.h presence... yes
checking for bzlib.h... yes
checking for BZ2_bzDecompressInit in -lbz2... yes
checking lzmadec.h usability... no
checking lzmadec.h presence... no
checking for lzmadec.h... no
checking for lzmadec_decode in -llzmadec... no
checking lzma.h usability... yes
checking lzma.h presence... yes
checking for lzma.h... yes
checking for lzma_stream_decoder in -llzma... yes
checking openssl/md5.h usability... yes
checking openssl/md5.h presence... yes
checking for openssl/md5.h... yes
checking openssl/ripemd.h usability... yes
checking openssl/ripemd.h presence... yes
checking for openssl/ripemd.h... yes
checking openssl/sha.h usability... yes
checking openssl/sha.h presence... yes
checking for openssl/sha.h... yes
checking md5.h usability... no
checking md5.h presence... no
checking for md5.h... no
checking ripemd.h usability... no
checking ripemd.h presence... no
checking for ripemd.h... no
checking rmd160.h usability... no
checking rmd160.h presence... no
checking for rmd160.h... no
checking sha.h usability... no
checking sha.h presence... no
checking for sha.h... no
checking sha1.h usability... no
checking sha1.h presence... no
checking for sha1.h... no
checking sha2.h usability... no
checking sha2.h presence... no
checking for sha2.h... no
checking sha256.h usability... no
checking sha256.h presence... no
checking for sha256.h... no
checking for library containing SHA1_Init... none required
checking for MD5_Init... yes
checking for RIPEMD160_Init... yes
checking for SHA1_Init... yes
checking for SHA256_Init... yes
checking for SHA384_Init... yes
checking for SHA512_Init... yes
checking for an ANSI C-conforming const... yes
checking for inline... inline
checking for uid_t in sys/types.h... yes
checking for mode_t... yes
checking for off_t... yes
checking for size_t... yes
checking for id_t... yes
checking for uintptr_t... yes
checking for struct stat.st_birthtime... yes
checking for struct stat.st_birthtimespec.tv_nsec... yes
checking for struct stat.st_mtimespec.tv_nsec... yes
checking for struct stat.st_mtim.tv_nsec... no
checking for struct stat.st_mtime_n... no
checking for struct stat.st_umtime... no
checking for struct stat.st_mtime_usec... no
checking for struct stat.st_blksize... yes
checking for struct stat.st_flags... yes
checking for uintmax_t... yes
checking for unsigned long long... yes
checking for unsigned long long int... yes
checking for long long int... yes
checking for intmax_t... yes
checking for int64_t... yes
checking for uintmax_t... (cached) yes
checking for uint64_t... yes
checking whether SIZE_MAX is declared... yes
checking whether SSIZE_MAX is declared... no
checking whether INT64_MAX is declared... yes
checking whether INT64_MIN is declared... yes
checking whether UINT64_MAX is declared... yes
checking whether UINT32_MAX is declared... yes
checking whether EFTYPE is declared... yes
checking whether EILSEQ is declared... yes
checking for wchar_t... yes
checking size of wchar_t... 4
checking whether time.h and sys/time.h may both be included... yes
checking whether gcc needs -traditional... no
checking whether sys/types.h defines makedev... yes
checking for _LARGEFILE_SOURCE value needed for large files... no
checking for working memcmp... yes
checking whether lstat correctly handles trailing slash... no
checking whether lstat accepts an empty string... no
checking whether stat accepts an empty string... no
checking whether strerror_r is declared... yes
checking for strerror_r... yes
checking whether strerror_r returns char *... no
checking for strftime... yes
checking for vprintf... yes
checking for _doprnt... no
checking for chflags... yes
checking for chown... yes
checking for chroot... yes
checking for fchdir... yes
checking for fchflags... yes
checking for fchmod... yes
checking for fchown... yes
checking for fcntl... yes
checking for fork... yes
checking for fstat... yes
checking for ftruncate... yes
checking for futimes... yes
checking for geteuid... yes
checking for getpid... yes
checking for lchflags... yes
checking for lchmod... yes
checking for lchown... yes
checking for lutimes... yes
checking for memmove... yes
checking for memset... yes
checking for mkdir... yes
checking for mkfifo... yes
checking for mknod... yes
checking for nl_langinfo... yes
checking for pipe... yes
checking for poll... yes
checking for readlink... yes
checking for select... yes
checking for setenv... yes
checking for setlocale... yes
checking for strchr... yes
checking for strdup... yes
checking for strerror... yes
checking for strrchr... yes
checking for timegm... yes
checking for tzset... yes
checking for unsetenv... yes
checking for utime... yes
checking for utimes... yes
checking for vfork... yes
checking for wcrtomb... yes
checking for wcscpy... yes
checking for wcslen... yes
checking for wctomb... yes
checking for wmemcmp... yes
checking for wmemcpy... yes
checking for lockf... yes
checking for posix_memalign... yes
checking for qsort_r... yes
checking for mmap... yes
checking whether D_MD_ORDER is declared... yes
checking for working POSIX fnmatch... yes
checking whether FNM_LEADING_DIR is declared... yes
checking for struct dirent.d_namlen... yes
checking for attr/xattr.h... (cached) no
checking sys/xattr.h usability... yes
checking sys/xattr.h presence... yes
checking for sys/xattr.h... yes
checking for setxattr in -lattr... no
checking for extattr_get_file... no
checking for extattr_list_file... no
checking for extattr_set_fd... no
checking for extattr_set_file... no
checking for fsetxattr... yes
checking for getxattr... yes
checking for lgetxattr... no
checking for listxattr... yes
checking for llistxattr... no
checking for lsetxattr... no
checking for sys/acl.h... (cached) yes
checking for acl_get_file in -lacl... no
checking for acl_create_entry... yes
checking for acl_init... yes
checking for acl_set_fd... yes
checking for acl_set_fd_np... yes
checking for acl_set_file... yes
checking for acl_permset_t... yes
checking for acl_get_perm_np... yes
checking for acl_get_perm... no
checking whether ACL_USER is declared... no
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking whether optarg is declared... yes
checking whether optind is declared... yes
checking whether to install bash completion... no
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating config.h
config.status: config.h is unchanged
config.status: executing depfiles commands

At the moment, tarsnap leaks a bit of memory when it exits. This is not such a big problem by itself, but warnings about those issues might obscure an more substantial message from code analysis tools such as valgrind if there are any errors during development. (in particular, when I'm writing code)

The https://github.com/Tarsnap/tarsnap/tree/WIP-valgrind branch is a work in progress towards addressing this. This is still very experimental at the moment; even the basic framework is in flux.

So this error started happening.

Example:

a Path/Project/.git/objects/e1/411758c15f602e6521843f1a4ac6106013c525
a Path/Project/.git/objects/e1/4cbe24166294c4a78918718eacd4c1b1d9f4ea
a Path/Project/.git/objects/e1/b194523aefef40ebcaee43294fd269ddca31b4
a Path/Project/.git/objects/e0/02fa2ae53eec2a26379859f8102b6278614ad0
a Path/Project/.git/objects/e0/4d9dd1811d729196ededdcf7d10c4d99685161
a Path/Project/.git/objects/e0/e682ab9aecb251d53faf652c5d371f85bae133
tarsnap: Path/Project/.git/objects/df: Unable to continue traversing directory tree

It always stops on this folder. I haven't worked on this project/git repo for a long time and the last-changed time of the folder dates back to 2015, it has been tarsnapped many times since then.

This option would circumvent the "will this require too much memory / time" mechanism in scryptenc and just assume that the user knows what they're doing. Probably worth adding the same option to scrypt too...

Occasionally I build tarsnap from the github but I had this failure this morning:

keygen/tarsnap_keygen-keygen.o test -f 'keygen/keygen.c' || echo './'keygen/keygen.c
mv -f keygen/.deps/tarsnap_keygen-keygen.Tpo keygen/.deps/tarsnap_keygen-keygen.Po
cc -I./libarchive -I./tar -I./lib/crypto -I./lib/datastruct -I./lib/keyfile -I./lib/netpacket -I./lib/netproto -I./lib/network -I./lib/util -I./libcperciva/crypto -I./libcperciva/util -DUSERAGENT="tarsnap-keygen-1.0.36-head" -g -O2 -o tarsnap-keygen keygen/tarsnap_keygen-keygen.o -lcrypto lib/libtarsnap.a lib/libtarsnap_aesni.a lib/libtarsnap_sse2.a -llzma -lbz2 -lcrypto -lz -lrt
keygen/tarsnap_keygen-keygen.o: In function main': /usr/home/jungle/bin/tarsnap/keygen/keygen.c:174: undefined reference tokeygen_network_register'
cc: error: linker command failed with exit code 1 (use -v to see invocation)
*** Error code 1

Stop.
make[1]: stopped in /usr/home/jungle/bin/tarsnap
*** Error code 1

Stop.
make: stopped in /usr/home/jungle/bin/tarsnap

This is after a simple ./configure and even a make clean.

Is there something I'm doing wrong?

Thanks!

First mentioned in http://mail.tarsnap.com/tarsnap-users/msg00294.html

Test case:

$ ./tarsnap --keyfile ~/wrong-keys.txt -c -f wrong-keys ~/backup
tarsnap: Sequence number mismatch: Run --fsck
tarsnap: Error creating new archive
tarsnap: Error exit delayed from previous errors.

We should have a more useful error -- in this case, running --fsck isn't actually the best fix, since that'll wreck the cache directory for the real key. Instead, we should be warning that it's the wrong keyfile.

When one refers to a filename or directory, the case sensitivity depends on the filesystem. So on OSX (with the default filesystem),

keyfile /users/me/tarsnap.key

works just fine. However, the include and exclude commands are always case sensitive, since it uses fnmatch() in the C layer, not the filesystem layer.

I'll update the docs, but I'm also leaning towards printing a warning if keyfile finds a file due to case-insensitivity when the string doesn't match.

For obvious reasons. SIGINFO support would be extremely nice.

Read a key file and print the machine ID #. Will be undocumented; this is a private interface for the GUI to use.

As per http://mail.tarsnap.com/tarsnap-users/msg01310.html

I don't see the point in the for loop.
And if strdup fails, it leaks mem pointed to by xp.

https://www.tarsnap.com/images/
https://www.tarsnap.com/download/

by this vulnerability grab any files directly
and should not be accessible by an unauthorized party who happens to know the URL. Nevertheless, directory listings can aid an attacker by enabling them to quickly identify the resources at a given path, and proceed directly to analyzing and attacking them.
Configure your web server to prevent directory listings for all paths beneath the web root;
Place into each directory a default file (such as index.htm) which the web server will display instead of returning a directory listing.

We should have a NEWS file, with the same level of granularity as the release notes on the website. This will make it easier for people who are updating packages in OS repositories to see what has changed; and if we keep it updated it could help to make sure we don't miss anything which belongs in the release notes.

I'd like Tarsnap to be distributed by Debian/Ubuntu for a few reasons:

• it's useful for other packages to depend on Tarsnap (e.g. an application's own backup scripts, which are packaged)
• compiling the package is more faff than apt-get install tarsnap
• I trust packages distributed by the Debian project moreso than those in unofficial apt repositories (of which there are numerous for distributing Tarsnap)

I was about to raise a Debian Request For Package for Tarsnap, but I noticed that Tarsnap's license includes:

Redistribution and use in source and binary forms, without modification,
is permitted for the sole purpose of using the "tarsnap" backup service
provided by Tarsnap Backup Inc.

As such, it is not DFSG (nor FSF's definition of "free"), and cannot be distributed with Debian.

Please could this be reconsidered, and ideally released under a DFSG license, e.g. a BSD license.

This is a feature (if you lose your cache directory and want to nuke a machine, you shouldn't have to run --fsck first), but it has two downsides:

1. Not using the cache directory means not locking the cache directory, so tarsnap --nuke will be interrupted if a cron job kicks in to create an archive.
2. Not using the cache directory means not updating the cache directory, so people need to run --fsck after running --nuke if they want to keep using the same keys.

I think the answer here might be to make the cache directory optional for --nuke -- use it if present, but retain the current behaviour if not. A bit of nuance is needed in case someone wants to nuke a machine while running on a different machine which has a valid cache directory; we don't want to mangle (or even lock) the wrong cache directory. The right answer might be to try a "normal" delete transaction first, and revert to the no-cache-directory nuke pathway if that fails with an out-of-sync error.

I need to think about this some more.

We should offer official prebuilt binaries for download&install.

We already have some support for this in pkg/archlinux/ and pkg/debian/. However,

• those files need to be tested on current OSes, and possibly updated for a newer packaging format (I remember seeing some warnings when playing with Debian tarsnap packages).
• any official packages must absolutely be signed
• any official packages must be reproducible (e.g., 2015-01 summary of Debian's reproducible builds https://lwn.net/Articles/630074/)
• all packages need to be tested locally (by me) before they go to the tarsnap-alphatest list.

In the future, we will strip following whitespace (issue #22), but before that we should allow quotes inside config file lines, just in case somebody actually wants to exclude "* ". And obviously \" within double-quotes should be parsed as a " character.

There's a glitch in the handling of the "all archives" statistics relating to recovered checkpoints. Specifically, if an archive is recovered from a checkpoint the "extra" statistics for the metadata and metaindex blocks aren't included; but they're deducted off when that recovered archive is deleted. This shows up when users delete all of their archives, and then get told that they have 18.4 exabytes of data stored (aka. a small negative number interpreted as an unsigned 64-bit value).

Some code in the multitape layer needs to be fixed -- I think in creating checkpoints we'll need to increment the "extra" statistics before writing out the directory file and then adjust back.

The bwlimit-rate options are not compatible with aggressive-networking. (If you have several TCP connections open, it becomes much harder to limit bandwidth usage since you need to figure out how to share it between connections.) Right now this is handled by refusing to accept those options simultaneously.

A better way of handling this would be to simply print a warning and turn aggressive-networking off if bandwidth limiting is enabled. This would solve the usability problem of having aggressive-networking set in a configuration file while trying to specify bandwidth limits at the command line; and it seems that since bandwidth limits are "what to do" and aggression is "how to do it", we should keep the bandwidth limiting but sacrifice the aggression. If the user wants the opposite, they can always pass the --no-maxbw-rate-* options on the command line along with --aggressive-networking.

This is not particularly useful:

tarsnap -cf foo foo
tarsnap: foo/bar/baz: Cannot stat: No such file or directory
tarsnap: Error exit delayed from previous errors.

If we're trying to archive a file because we found it in a directory we recursed through, but find that it doesn't exist, then it just means that we lost a race condition. That's fine; just like "file has grown while being archived" it's a normal occurrence when we're creating backups from a running system, and not something we should treat as an error.

If we were running tarsnap -cf foo foo/bar/baz and found that foo/bar/baz didn't exist, then it should be treated as an error, of course.

It would be nice to let people do

tarsnap --dry-run --print-stats --humanize-numbers -c ~/backup/

without having a keyfile or registering their machine (or even having a tarsnap.com account).

When we're checking to make sure we have all the keys we need, we're assuming that archive creation only requires write keys. But if there's an @@archive directive, we will need read keys as well, which means we could end up at "Programmer error: key not available".

I'm not sure what the best solution is for this; scanning the command line for @@archive directives? We need to be careful to not trigger on "-C @@foo" in case someone has a peculiarly named directory though. We could postpone the check until we actually hit the @@archive directive, but it would be much better if we could inform the user that they're using the wrong key file promptly...