demizer / gds Goto Github PK

Need a proper argument parser and arguments!

Need to clear the line before sending a new one. Also need to fix carriage return in go-elog.

If enter is accidentally pressed before a device switch is made, it would be catastrophic. gds should verify the switch has been made before continuing. If not, ask the user again to switch out the device. Checks can be made against the PARTUUID. This should be done from the start.

Just like copying, but measure bytes being hashed.

The function is pages of code long, break it up! Split out the io.Copy() calls to separate functions so that they can be mocked in testing.

Some parts of the code base use fmt.Errorf() to add context to the error output. gds should really be using https://godoc.org/gopkg.in/inconshreveable/log15.v2#hdr-Context_loggers

If the destination drive is nearly full, and the current file is larger than the available space, the next files that can fit into the available space should be copied and the larger files saved for the next mounted drive.

Use external program to verify results of saving file checking. This may also speed the process.

sync() copy throws an error, and goroutines are racing.

time="2015-07-29T23:47:23-07:00" level=error msg="Error copying file!" deviceSize=5005516800 deviceUsage=4965451465 filePath="/home/demizer/documents/development/HTML5/jQuery eBooks Collection/Packt Publishing jQuery 1.4 Plugin Development, Beginner's Guide (2010).pdf" fileSize=10319491

Actual device size in this case:

demizer@helium$ \df -B1 /mnt/backup1 /mnt/backup2
Filesystem      1B-blocks       Used  Available Use% Mounted on
/dev/loop0     5284429824 5015994368          0 100% /mnt/backup1
/dev/loop1     5284429824   10477568 5005516800   1% /mnt/backup2

I modified the preSync() function to dump the file size + file name of all the files that will be saved to the device. Once I got the output, I added up the sizes of the files:

demizer@helium $ cat 2015-07-30T01:01:44-0700_output | cut -d"," -f 1  | awk '{total = total + $0}END{print total}'
5008878454

Something in NewCatalog() is off.

Basic sync includes getting the files on formatted devices, this is largely done. Update sync is syncing backup paths to devices that already contain files.

• A file is added
• A file is added to a directory, and the directory has previously been synced to a device that is now full.
• A file is removed from the source
• A file is removed at the destination
• A file is changed at the source
• A file exists that is read-only (0444 perms)
• Sync changed files with rsync algorithm

After the user has mounted a drive, the permissions of the mounted drive can be root, which would cause the write to fail if todrives has been run as a regular user.

It would be nice to be able to sync files to multiple drives at once to speed up the sync process and saturate IO.

It takes 30 seconds to run about 20 tests.

Support gzip, bzip2, zlib, and lzma (http://godoc.org/code.google.com/p/lzma)

The scrolling log output is hard to read, especially when being piped to a file due to logrus's logfmt support. It would be nice to have a terminal GUI for showing progress bars and such.

https://github.com/jroimartin/gocui

• Total done percent as progress bar
• Current device progress stats and ETA
• Previous device stats, time to complete, number of files, and total data size
• Scrolling log panel with bps, time it took to copy, and file path
• Show progress in the scrolling log panel when scanning files for metadata in NewFileList()https://en.wikipedia.org/wiki/Bash_(Unix_shell)
• Space for multiple device streams
• Issue #4 - Show progress output when hashing destination paths
• Issues #24 and #8 - Don't let the user proceed without a proper mount

The hashes for the files are saved once backup is complete. There should be an option to use these hashes to check the origin files for corruption.

func TestSha1Sum(t *testing.T) {
    _, err := sha1sum("/root")
    if err == nil {
        t.Error("EXPECT: error permission denied GOT: No errors")
    }
}

Root user can read "/root", so this test will fail if being run by root user. This test should not fail if being run by a privileged user.

If the values passed to the --log, --config``, and--contextarguments contain the suffix.gz`, the the output will be contained in a compressed archive.

Add a command argument to send log output to file.

Need to fully test the command argument parsing.

When running gds for the first time on my huge dataset, a panic occurred when it encountered my root only easyrsa files with perms 2700/drwx--S---.

For now the work around is to run as root. gds should gracefully notify the user that files cannot be read and give them the option to continue or cancel.

Need to dump the catalog to a file. Not really sure if this will be useful when update syncing is implemented, but for now we just need to serialize this data.

• Command options for dumping the sync context to file.
• Dump context to XDG_HOME_CONFIG/gds/$(date --iso-8601=seconds)_context.json
• Save context as last file of last device
• Test last device full before saving context
• Command flag to disable saving sync context data to last device

It would be nice to have a command option for copying the gds static binary (for multiple OSes) and catalog copied to backup device for ease of use. This would need to be duplicated across all devices so that the user can plug in any device and get to the files they need quickly without first having to install gds.

My test config.yml had "drives" instead of "devices" and gds continued anyways. This caused a nil pointer dereference panic in NewCatalog() when it attempted to load device info that did not exist.

https://github.com/HouzuoGuo/tiedot

The first major backup of 6.5TB had only a single apparent failure:

time="2015-07-31T19:32:44-07:00" level=fatal msg="Sync failed: createFile: setMetaData chown: chown /mnt/backup/data/home/demizer/code/code/c/pacman/lib/libalpm/util-common.c: no such file or directory"

util-common.c is a symlink, so the problem could be that chmod is following the symlink but the file it is pointing at doesn't exist.

• Document stuff
• Get code test coverage above 90%

After a drive has become full, save the metadata log in its current state to the drive before unmounting. This is currently done manually.

The issue is complete when tests have been written for the following scenarios:

• A file is too big for all drives.
• A file has gid and uid that are not installed on the host.
• Rsync-like source directory behavior. I.e., backup vs. backup/
• File splitting
  • A file is partly split across two devices
  • A file takes up 1.5 devices.
    Make checks of hardcoded device usage expectations.
  • A file is too big for any one drive, but can fit on all backup drives if split.
  • Not enough devices for sync.