Wrapper for rsync, parallelize backup. Used to backup 200 servers in production on usb disk 1To.
This script is written in bash.
It use a simple bash config file (eg: /etc/backup_servers.conf), here are the default values:
- BACKUP_DIR=""
- SERVERS=""
- DIRS=""
- PARALLEL=1
- RSYNC="rsync -au --force --numeric-ids --relative --stats"
- RSYNC_OPT=""
Or arguments on the command line:
backup_servers --backup-dir /home/backup --dirs /etc --servers my.server.com
The variable DIRS is used by runing ls -1d DIRS
on the server via ssh.
Only exiting directories will be listed wich permit you to have one global directory policy for all servers.
You can use as well any $(shell command)
to get the list of directories passed to ls
.
# /etc/backup_servers.conf:
BACKUP_DIR=/home/backup
PARALLEL=4
SERVERS=$(mysql sys -NBe "SELECT host FROM server")
# If you want specifics directories list per server, you can get it from a file on the server
DIRS='$(if [ -e /etc/backup_servers.dirs ]; then cat /etc/backup_servers.dirs; else echo "/etc /home"; fi)'
RSYNC_OPT="-z --rsync-path='nice +19 rsync' --bwlimit 1024"
You can then test your config with arguments to change values of the config file:
backup_servers --conf /etc/backup_servers.conf --servers myserver.mydomain.loc --dirs /etc --debug
Or run the backup in rsync dry mode:
backup_servers --conf /etc/backup_servers.conf -n