asimihsan / arqinator Goto Github PK

View Code? Open in Web Editor NEW

194.0 16.0 10.0 82 KB

Cross-platform restoration of Arq backups

License: Apache License 2.0

Makefile 1.70% Go 97.18% Shell 1.12%

arqinator's Introduction

arqinator

Cross-platform restoration of Arq backups. Download arqinator binaries from the releases page.

Please explicitly test listing directories and recovering files before assuming it will work for you.

Features

Cross-platform support for Windows, Mac OS X, and Linux.
- Tested on Windows 7 32-bit, Mac OS X Yosemite 10.10.5 64-bit, and Ubuntu 14.04 LTS 64-bit
Deployable as a single executable file, no external dependencies required.
List all backup sets in a particular backup type using list-backup-sets
List the contents of directories or information about a file using list-directory-contents
Recover single files, sub-folders and their contents, or entire backup sets, using recover

Limitations

Currently only supports the following backup types:
- S3
- Google Cloud Storage
- SFTP (only unencrypted SSH private keys)
arqinator has been tested on backups created by Arq 4.14.5 only. I do not know if arqinator works on previous versions of Arq. I'm doubtful that arqinator will work on previous major versions of Arq (i.e. 3 or 2).
Files are downloaded in serial, might take a while to recover a lot of data.

TODO

soft links aren't supported
do you want to 'chown' files and folders to the UID/GID backed up?
- maybe offer as an option?
support multiple encryption passwords for multiple accounts
- maybe have a text-file based configuration?
support all backup types possible with Arq, start with Dropbox.
explicitly check SHA1 hashes of blobs to confirm no corruption.
Files are downloaded in serial, should do it in parallel.

Testing done so far

I've successfully listed backup sets, listed directory contents, and downloaded specific files and subfolders on:

Backups created by Mac OS X 10.10.5 onto S3, Google Cloud Storage, and SFTP, and then retrieved back onto the same Mac OS X host.
Backups created by Windows 7 onto S3, and then retrieved back onto the same Windows 7 host and a different Ubuntu 14.04 host.

Getting started

0. Prerequisites

How to set environment variables on:

Download arqinator binaries from the releases page.

1. Configure Credentials

For all backup methods, set the following environment variable to the password you're using for encrypting your Arq backups:

ARQ_ENCRYPTION_PASSWORD=mysecretpassword

Only Arq backup sets encrypted by this password will be visible to you when you run list-backup-sets. If the password is incorrect they will appear to be invisible. Run with --verbose if you want more information.

S3

Set the following environment variables depending on which backup method you are using:

AWS_ACCESS_KEY_ID=AKID1234567890
AWS_SECRET_ACCESS_KEY=MY-SECRET-KEY

Google Cloud Storage

Go to the Google Developers Console: https://console.developers.google.com//project/_/apiui/credential
Download a JSON private key file using: https://goo.gl/SK5Rb7
To determine your project ID, click on the gear in the top right, and click "Project information"
To determine your bucket name, on the left navigation pane go to Storage - Cloud Storage - Browser, then find your bucket.

SFTP

The preferred way of using SFTP is to use password-less SSH login by putting your SSH public key into the SFTP server's authorized_keys. When you do so currently arqinator only supports unencrypted SSH private keys. However if you want to log into the SFTP server using a plaintext password you can set the following environment variable:

ARQ_SFTP_PASSWORD=my-sftp-password

2. List backup sets

Note that there will be a difference between how paths appear on Windows and Linux/Mac:

S3, Mac

$ arqinator \
    --backup-type s3 \
    --s3-region us-west-2 \
    --s3-bucket-name arq-akiajmthnhpkz2ixzrxq-us-west-2 \
    list-backup-sets

ArqBackupSet
    UUID 98DB38F8-B9C6-4296-9385-3C1BF858ED5D
    ComputerName Mill
    UserName ai
    Folders
        LocalPath /Users/ai
        UUID 8D4FAD2A-9E08-46F7-829D-E9601A65455D

S3, Windows

arqinator ^
    --backup-type s3 ^
    --s3-region us-west-2 ^
    --s3-bucket-name arq-akiajmthnhpkz2ixzrxq-us-west-2 ^
    list-backup-sets

ArqBackupSet
    UUID E7CFDEED-AB08-4970-A377-78F8313AC39C
    ComputerName THE_RAIN
    UserName SYSTEM
    Folders
        LocalPath /C/Users/username/Downloads/apsw-3.7.15.1-r1
        UUID FE8BE3EE-B63B-4D1F-A7E9-6707297823B5

Google Cloud Storage, Mac

$ arqinator \
    --backup-type googlecloudstorage \
    --gcs-json-private-key-filepath /Users/ai/keys/gcs.json \
    --gcs-project-id midyear-courage-109219 \
    --gcs-bucket-name arq-560729839528 \
    list-backup-sets

ArqBackupSet
    UUID 7FE8D069-B218-4E17-8E58-0C7FAF8CFAFC
    ComputerName Mill
    UserName ai
    Folders
        LocalPath /Users/ai/temp/apsw-3.7.15.1-r1
        UUID E6F4BC5E-B21F-4828-ADCC-8521F9DBC4C9

SFTP, Mac, verbose mode

$ arqinator \
      --backup-type sftp \
      --sftp-host asims-mac-mini.local \
      --sftp-port 22 \
      --sftp-remote-path /Users/aihsan/arq_backup \
      --sftp-username aihsan \
      --sftp-private-key-filepath /Users/ai/.ssh/id_rsa \
      --verbose \
      list-backup-sets

ArqBackupSet
    UUID 76A4E004-FCB9-47D7-B080-16A236439F5C
    ComputerName Mill
    UserName ai
    Folders
        LocalPath /Users/ai/temp/apsw-3.7.15.1-r1
        UUID 1BFC0BD6-9877-4562-9692-05EB3A5EF20C

3. List directory contents of backups

Again note that paths for Windows will look a little unusual.

S3, Mac

$ arqinator \
    --backup-type s3 \
    --s3-region us-west-2 \
    --s3-bucket-name arq-akiajmthnhpkz2ixzrxq-us-west-2 \
    list-directory-contents \
    --backup-set-uuid 98DB38F8-B9C6-4296-9385-3C1BF858ED5D \
    --folder-uuid 8D4FAD2A-9E08-46F7-829D-E9601A65455D \
    --path /Users/ai/.ssh

-rwx------	2013-06-29 05:43:06 -0700 PDT	1.7kB	ai_keypair_3.pem
-rw-r--r--	2013-06-28 13:44:47 -0700 PDT	0B	    config
-rw-------	2014-12-31 19:41:47 -0800 PST	1.7kB	digitalocean
-rw-r--r--	2014-12-31 19:41:47 -0800 PST	402B	digitalocean.pub
-rw-------	2012-04-11 12:49:10 -0700 PDT	1.7kB	id_rsa
-rw-r--r--	2012-04-14 13:16:50 -0700 PDT	396B	id_rsa.pub
-rw-------	2014-05-17 05:04:13 -0700 PDT	1.7kB	interview-ec2.pem
-rw-r--r--	2015-09-08 14:09:39 -0700 PDT	18kB	known_hosts

S3, Windows

arqinator ^
    --backup-type s3 ^
    --s3-region us-west-2 ^
    --s3-bucket-name arq-akiajmthnhpkz2ixzrxq-us-west-2 ^
    list-directory-contents ^
    --backup-set-uuid E7CFDEED-AB08-4970-A377-78F8313AC39C ^
    --folder-uuid FE8BE3EE-B63B-4D1F-A7E9-6707297823B5 ^
    --path /C/Users/username/Downloads/apsw-3.7.15.1-r1

----------      2015-10-08 20:36:21 +0100 BST   6.1kB   .DS_Store
d---------      1970-01-01 00:00:00 +0000 GMT   9.3MB   build
----------      2012-12-22 09:05:48 +0000 GMT   7.1kB   checksums
d---------      1970-01-01 00:00:00 +0000 GMT   1.6MB   doc
----------      2010-01-05 22:53:28 +0000 GMT   699B    MANIFEST.in
----------      2009-09-13 05:50:06 +0100 BST   4.3kB   mingwsetup.bat
----------      2012-12-22 10:57:02 +0000 GMT   1.0kB   PKG-INFO
----------      2012-12-26 18:01:38 +0000 GMT   33kB    setup.py
d---------      1970-01-01 00:00:00 +0000 GMT   7.6MB   sqlite3
d---------      1970-01-01 00:00:00 +0000 GMT   554kB   src
----------      2012-12-26 18:05:00 +0000 GMT   29kB    testdbx
----------      2012-12-26 18:05:00 +0000 GMT   12kB    testdbx-journal
----------      2012-12-22 09:36:24 +0000 GMT   340kB   tests.py
----------      2012-12-26 18:03:31 +0000 GMT   285kB   tests.pyc
d---------      1970-01-01 00:00:00 +0000 GMT   168kB   tools

Google Cloud Storage, Mac

$ arqinator \
    --backup-type googlecloudstorage \
    --gcs-json-private-key-filepath /Users/ai/keys/gcs.json \
    --gcs-project-id midyear-courage-109219 \
    --gcs-bucket-name arq-560729839528 \
    list-directory-contents \
    --backup-set-uuid 7FE8D069-B218-4E17-8E58-0C7FAF8CFAFC \
    --folder-uuid E6F4BC5E-B21F-4828-ADCC-8521F9DBC4C9 \
    --path /Users/ai/temp/apsw-3.7.15.1-r1

-rw-r--r--	2015-10-08 12:36:21 -0700 PDT	6.1kB	.DS_Store
-rw-r--r--	2010-01-05 14:53:28 -0800 PST	699B	MANIFEST.in
-rw-rw-r--	2012-12-22 02:57:02 -0800 PST	1.0kB	PKG-INFO
drwxr-xr-x	2015-10-08 12:36:21 -0700 PDT	9.3MB	build
-rw-r--r--	2012-12-22 01:05:48 -0800 PST	7.1kB	checksums
drwxr-xr-x	2012-12-26 09:56:54 -0800 PST	1.6MB	doc
-rw-r--r--	2009-09-12 21:50:06 -0700 PDT	4.3kB	mingwsetup.bat
-rw-rw-r--	2012-12-26 10:01:38 -0800 PST	33kB	setup.py
drwxr-xr-x	2012-12-26 10:02:47 -0800 PST	7.6MB	sqlite3
drwxr-xr-x	2012-12-26 10:02:47 -0800 PST	554kB	src
-rw-r--r--	2012-12-26 10:05:00 -0800 PST	29kB	testdbx
-rw-r--r--	2012-12-26 10:05:00 -0800 PST	12kB	testdbx-journal
-rw-r--r--	2012-12-22 01:36:24 -0800 PST	340kB	tests.py
-rw-r--r--	2012-12-26 10:03:31 -0800 PST	285kB	tests.pyc
drwxr-xr-x	2012-12-26 09:56:54 -0800 PST	168kB	tools

SFTP, Mac

arqinator \
    --backup-type sftp \
    --sftp-host asims-mac-mini.local \
    --sftp-port 22 \
    --sftp-remote-path /Users/aihsan/arq_backup \
    --sftp-username aihsan \
    --sftp-private-key-filepath /Users/ai/.ssh/id_rsa \
    --verbose
    list-directory-contents \
    --backup-set-uuid 76A4E004-FCB9-47D7-B080-16A236439F5C \
    --folder-uuid 1BFC0BD6-9877-4562-9692-05EB3A5EF20C \
    --path /Users/ai/temp/apsw-3.7.15.1-r1/build

-rw-r--r--	2015-10-09 09:50:34 -0700 PDT	10kB	.DS_Store
drwxr-xr-x	2012-12-26 10:03:31 -0800 PST	1.5MB	lib.macosx-10.4-x86_64-2.7
drwxr-xr-x	2015-10-09 09:50:34 -0700 PDT	7.8MB	temp.macosx-10.4-x86_64-2.7

4. Restore

You can restore either individual files or entire folders. Note that you need to use a Linux-like directory path for Windows backups:

S3, Mac, recovering a single file

$ arqinator \
    --backup-type s3 \
    --s3-region us-west-2 \
    --s3-bucket-name arq-akiajmthnhpkz2ixzrxq-us-west-2 \
    recover \
    --backup-set-uuid 98DB38F8-B9C6-4296-9385-3C1BF858ED5D \
    --folder-uuid 8D4FAD2A-9E08-46F7-829D-E9601A65455D \
    --source-path /Users/ai/output.txt \
    --destination-path /Users/ai/temp/output.txt

S3, Windows, recovering an entire folder in verbose mode

arqinator ^
    --backup-type s3 ^
    --s3-region us-west-2 ^
    --s3-bucket-name arq-akiajmthnhpkz2ixzrxq-us-west-2 ^
    --verbose ^
    recover ^
    --backup-set-uuid E7CFDEED-AB08-4970-A377-78F8313AC39C ^
    --folder-uuid FE8BE3EE-B63B-4D1F-A7E9-6707297823B5 ^
    --source-path /C/Users/username/Downloads/apsw-3.7.15.1-r1/tools ^
    --destination-path /C/temp/tools

Google Cloud Storage, Mac, recovering a single file

$ arqinator \
    --backup-type googlecloudstorage \
    --gcs-json-private-key-filepath /Users/ai/keys/gcs.json \
    --gcs-project-id midyear-courage-109219 \
    --gcs-bucket-name arq-560729839528 \
    recover \
    --backup-set-uuid 7FE8D069-B218-4E17-8E58-0C7FAF8CFAFC \
    --folder-uuid E6F4BC5E-B21F-4828-ADCC-8521F9DBC4C9 \
    --source-path /Users/ai/temp/apsw-3.7.15.1-r1/PKG-INFO \
    --destination-path /Users/ai/temp/PKG-INFO

SFTP, Mac, recovering a folder

$ arqinator \
    --backup-type sftp \
    --sftp-host asims-mac-mini.local \
    --sftp-port 22 \
    --sftp-remote-path /Users/aihsan/arq_backup \
    --sftp-username aihsan \
    --sftp-private-key-filepath /Users/ai/.ssh/id_rsa \
    recover \
    --backup-set-uuid 76A4E004-FCB9-47D7-B080-16A236439F5C \
    --folder-uuid 1BFC0BD6-9877-4562-9692-05EB3A5EF20C \
    --source-path /Users/ai/temp/apsw-3.7.15.1-r1/build \
    --destination-path /Users/ai/temp/foobar

arqinator's People

Contributors

Stargazers

Watchers

Forkers

deanashton booi zph dwin ajinkyakulkarni natinux ukazap nick-forks yashodhank r3dey3

arqinator's Issues

GCE code is out of date, arqinator no longer cleanly builds

It appears that the google.golang.org/cloud/storage api has changed since this was written.

As a result, it no longer builds for a fresh checkout:

$ make build-mac-amd64
go get github.com/asimihsan/arqinator/arq github.com/asimihsan/arqinator/arq/types github.com/asimihsan/arqinator/crypto github.com/asimihsan/arqinator/connector 
# github.com/asimihsan/arqinator/connector
../../.go/src/github.com/asimihsan/arqinator/connector/googlecloudstorage.go:114: undefined: "google.golang.org/cloud/storage".ListObjects
../../.go/src/github.com/asimihsan/arqinator/connector/googlecloudstorage.go:200: undefined: "google.golang.org/cloud/storage".NewReader
../../.go/src/github.com/asimihsan/arqinator/connector/s3.go:48: undefined: s3manager.DownloadOptions
make: *** [internal-deps] Error 2

ListObjects doesn't seem to be around anymore. https://godoc.org/google.golang.org/cloud/storage

Close files after opening them!

Identify program code that opens files without closing them.
Identify third-party code that causes file handle leakage.

Figured out that mmap'ing files meant that after closing the file handle the file handle remained open.

Anyway to CREATE arq backups using Ubuntu Linux?

Would love to know. Thank you!

Unable to list directory contents.

I seem to be getting a form of timeout (assuming?) when attempting to contact s3. It fails consistently in ~14 seconds. Errors below. I've tried various directories including / /Users/justinlilly and ./. No luck.

Any thoughts or help?

Command I'm running:

./arqinator --backup-type s3 --s3-region us-west-2 --s3-bucket-name arq-akiaim7mmnq73dheykuq-us-west-2 list-directory-contents --backup-set-uuid=6D925883-DA12-4CAE-A7F1-687BE4BAE83E --folder-uuid=9DDA6D30-D2CF-4BE9-8707-5F34279C3A52 --path=/Users/justinlilly/.ssh

Error

./arqinator --backup-type s3 --s3-region us-west-2 --s3-bucket-
name arq-akiaim7mmnq73dheykuq-us-west-2 list-directory-contents --backup-set-uuid=6D925883-DA12-4CAE-A7F1-687BE4
BAE83E --folder-uuid=9DDA6D30-D2CF-4BE9-8707-5F34279C3A52 --path=/Users/justinlilly/.ssh
INFO[0001] Caching tree pack sets. If this is your first run, will take a few minutes... 
INFO[0014] Cached tree pack sets.                       
PANI[0014] Failed to mmap index file:  invalid argument 
panic: (*logrus.Entry) (0x85b76e0,0x1910bc20)

goroutine 1 [running]:
github.com/Sirupsen/logrus.Entry.log(0x18a10600, 0x1930db80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /Users/ai/Programming/golang/src/github.com/Sirupsen/logrus/entry.go:113 +0x4cd
github.com/Sirupsen/logrus.(*Entry).Panic(0x18a3d280, 0x18a3d244, 0x1, 0x1)
        /Users/ai/Programming/golang/src/github.com/Sirupsen/logrus/entry.go:158 +0x7b
github.com/Sirupsen/logrus.(*Entry).Panicln(0x18a3d280, 0x18a3d3b0, 0x2, 0x2)
        /Users/ai/Programming/golang/src/github.com/Sirupsen/logrus/entry.go:253 +0xe9
github.com/Sirupsen/logrus.(*Logger).Panicln(0x18a10600, 0x18a3d3b0, 0x2, 0x2)
        /Users/ai/Programming/golang/src/github.com/Sirupsen/logrus/logger.go:204 +0xa2
github.com/Sirupsen/logrus.Panicln(0x18a3d3b0, 0x2, 0x2)
        /Users/ai/Programming/golang/src/github.com/Sirupsen/logrus/exported.go:187 +0x3b
github.com/asimihsan/arqinator/arq.(*ArqPackSetIndex).GetTreePackFile.func1(0x18a3d440, 0x18a3d500, 0x18a3d404, 
0x18a3d438)
        /Users/ai/Programming/golang/src/github.com/asimihsan/arqinator/arq/pack_set_index.go:277 +0x389
github.com/asimihsan/arqinator/arq.(*ArqPackSetIndex).GetTreePackFile(0x18a3d740, 0x18b1cbd0, 0x18b1d260, 0xfcaa
4b8f, 0x8d7cb312, 0x443f2f92, 0xa1e79192, 0x9d641462, 0x0, 0x0, ...)
        /Users/ai/Programming/golang/src/github.com/asimihsan/arqinator/arq/pack_set_index.go:298 +0x223
github.com/asimihsan/arqinator/arq.(*ArqPackSetIndex).GetPackFileAsCommit(0x18a3d740, 0x18b1cbd0, 0x18b1d260, 0x
fcaa4b8f, 0x8d7cb312, 0x443f2f92, 0xa1e79192, 0x9d641462, 0xf76470f8, 0x0, ...)
        /Users/ai/Programming/golang/src/github.com/asimihsan/arqinator/arq/pack_set_index.go:130 +0x61
github.com/asimihsan/arqinator/arq.FindNode(0x18b59ce0, 0x1f, 0x18b1cbd0, 0x18b1d260, 0x18b59cc0, 0x17, 0x24, 0x
18b1d260, 0x0, 0x0)
        /Users/ai/Programming/golang/src/github.com/asimihsan/arqinator/arq/pack_set_index.go:73 +0x134
main.listDirectoryContents(0x18a96500, 0xf7647ba0, 0x18b59ca0, 0x0, 0x0)
        /Users/ai/Programming/golang/src/github.com/asimihsan/arqinator/main.go:234 +0x79d
main.main.func2(0x18a96500)
        /Users/ai/Programming/golang/src/github.com/asimihsan/arqinator/main.go:429 +0x208
github.com/codegangsta/cli.Command.Run(0x86233a0, 0x17, 0x0, 0x0, 0x0, 0x0, 0x0, 0x8655380, 0x25, 0x0, ...)
        /Users/ai/Programming/golang/src/github.com/codegangsta/cli/command.go:127 +0xdeb
github.com/codegangsta/cli.(*App).Run(0x18a6bd40, 0x18a0a060, 0xb, 0xc, 0x0, 0x0)
        /Users/ai/Programming/golang/src/github.com/codegangsta/cli/app.go:159 +0xa6c
main.main()
        /Users/ai/Programming/golang/src/github.com/asimihsan/arqinator/main.go:474 +0x143a

goroutine 12 [IO wait]:
net.runtime_pollWait(0xf7648c50, 0x72, 0x18a16110)
        /usr/local/Cellar/go/1.5.1/libexec/src/runtime/netpoll.go:157 +0x55
net.(*pollDesc).Wait(0x18b5bc38, 0x72, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_poll_runtime.go:73 +0x35
net.(*pollDesc).WaitRead(0x18b5bc38, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_poll_runtime.go:78 +0x33
net.(*netFD).Read(0x18b5bc00, 0x18e34000, 0x8000, 0x8000, 0x0, 0xf7643030, 0x18a16110)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_unix.go:232 +0x19a
net.(*conn).Read(0x18a2d080, 0x18e34000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/net.go:172 +0xb9
crypto/tls.(*block).readFromUntil(0x18b86b40, 0xf7648e68, 0x18a2d080, 0x5, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/crypto/tls/conn.go:455 +0xa4
crypto/tls.(*Conn).readRecord(0x18a8e340, 0x86dd917, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/crypto/tls/conn.go:540 +0x269
crypto/tls.(*Conn).Read(0x18a8e340, 0x18db0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/crypto/tls/conn.go:901 +0x130
net/http.noteEOFReader.Read(0xf764d338, 0x18a8e340, 0x18b8a02c, 0x18db0000, 0x1000, 0x1000, 0x18a000e0, 0x0, 0x0
)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/http/transport.go:1370 +0x55
net/http.(*noteEOFReader).Read(0x18d990b0, 0x18db0000, 0x1000, 0x1000, 0x1, 0x0, 0x0)
        <autogenerated>:126 +0xae
bufio.(*Reader).fill(0x18d94c00)
        /usr/local/Cellar/go/1.5.1/libexec/src/bufio/bufio.go:97 +0x172
bufio.(*Reader).Peek(0x18d94c00, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/bufio/bufio.go:132 +0xad
net/http.(*persistConn).readLoop(0x18b8a000)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/http/transport.go:876 +0xe3
created by net/http.(*Transport).dialConn
        /usr/local/Cellar/go/1.5.1/libexec/src/net/http/transport.go:685 +0xabc

goroutine 13 [select]:
net/http.(*persistConn).writeLoop(0x18b8a000)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/http/transport.go:1009 +0x303
created by net/http.(*Transport).dialConn
        /usr/local/Cellar/go/1.5.1/libexec/src/net/http/transport.go:686 +0xadc

Directory Contents

./arqinator --backup-type s3 --s3-region us-west-2 --s3-bucket-
name arq-akiaim7mmnq73dheykuq-us-west-2 list-backup-sets
ArqBackupSet
    UUID 6D925883-DA12-4CAE-A7F1-687BE4BAE83E
    ComputerName ramataz
    UserName justinlilly
    Folders
        LocalPath /Users/justinlilly
        UUID 9DDA6D30-D2CF-4BE9-8707-5F34279C3A52

Automatically delete cache if zero-byte or corrupted files found

If the cache contains zero-byte or corrupted files we can no longer unpack tree pack sets, so we'll be unable to list directories or recover files. How to reproduce:

Delete the cache directory
Run:

arqinator \
    --backup-type s3 \
    --s3-region us-west-2 \
    --s3-bucket-name arq-akiajmthnhpkz2ixzrxq-us-west-2 \
    --verbose list-directory-contents \
    --backup-set-uuid 98DB38F8-B9C6-4296-9385-3C1BF858ED5D \
    --folder-uuid 8D4FAD2A-9E08-46F7-829D-E9601A65455D \
    --path /Users/ai/.ssh

While it is caching tree pack sets CTRL-C it.
Run the command again. You get the following output:

INFO[0006] Cached tree pack sets.
PANI[0006] Failed to mmap index file:  invalid argument
panic: (*logrus.Entry) (0x71c880,0xc8200112c0)

goroutine 1 [running]:
github.com/Sirupsen/logrus.Entry.log(0xc820072180, 0xc8203044b0, 0x0, 0x0, 0x0, 0xc820423b00, 0x0, 0x0, 0x0, 0xc820304510, ...)
    /Users/ai/Programming/golang/src/github.com/Sirupsen/logrus/entry.go:113 +0x618
github.com/Sirupsen/logrus.(*Entry).Panic(0xc820722550, 0xc8207224e0, 0x1, 0x1)
    /Users/ai/Programming/golang/src/github.com/Sirupsen/logrus/entry.go:158 +0x99
github.com/Sirupsen/logrus.(*Entry).Panicln(0xc820722550, 0xc820722750, 0x2, 0x2)
    /Users/ai/Programming/golang/src/github.com/Sirupsen/logrus/entry.go:253 +0x120
github.com/Sirupsen/logrus.(*Logger).Panicln(0xc820072180, 0xc820722750, 0x2, 0x2)
    /Users/ai/Programming/golang/src/github.com/Sirupsen/logrus/logger.go:204 +0xe7
github.com/Sirupsen/logrus.Panicln(0xc820722750, 0x2, 0x2)
    /Users/ai/Programming/golang/src/github.com/Sirupsen/logrus/exported.go:187 +0x41
github.com/asimihsan/arqinator/arq.(*ArqPackSetIndex).GetTreePackFile.func1(0xc820722870, 0xc8207229f0, 0xc8207227f8, 0xc820722860)
    /Users/ai/Programming/golang/src/github.com/asimihsan/arqinator/arq/pack_set_index.go:277 +0x3da

I've confirmed that there are zero-byte files in the cache directory.

I think we should be doing the following:

Scan the cache for zero-byte files. If any exist delete the cache directory and retry.
Scan the cache for corrupted files (how?). If any exist delete the cache directory and retry.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.