Quite often big files (like database dumps) are backed up on a regular basis. Such backups become a disk space hogs and when time comes to restoring data, it is cumbersome to find the right file for the job. This script checks files of a certain pattern and sorts them by date into daily, monthly, and yearly directories, progressively deleting more and more of the aged files.

Traverses a directory with a huge number of disorganized backup files recursively, deletes unnecessary ones, and packs the rest into a predictable directory structure.

• Works with multiple backup directories, each of them independent from another.
• Takes files generated by backups and sorts them either by using yyyy-mm-dd pattern in the files name or by files' modification date. Then picks some files for archiving and deletes other files.
• It keeps all files from the last 2 days. 1 file per day for the last month, and one file per month afterwards.
• Files are recognized by a regular expression that matches their name. Each directory can have several regular expressions, and each of them (assuming they do not overlap) is handled separately.

Download the latest release for your operating system, unzip or untar it, copy backme file to /usr/local/bin or any other directory registered with your $PATH.

Create .backme.yml file in your $HOME directory, use a config example to set it up.

The configuration file would look something like this:

---
outputDir: archive
inputDirs:
  - path: /some/dir/to/backup1
    fileRegexPatterns:
      - "^backup"
      - "^local"
  - path: /another/dir/to/backup2
    keepAllFiles: true
    fileRegexPatterns:
      - "\\.dump$"

outputDir : An optional setting (the default is archive). It is the name of the backme's archive directories. They will be placed inside of input directories.

inputDirs : A list of directories where you keep backup files.

inputDirs.path : The path to a backup directory

inputDirs.fileRegexPatterns : A list of regular expression patterns that allow to distinguish one or more series of backup files from other files. All files of such patterns will be used recursively inside of an input directory.

inputDirs.keepAllFiles : An optional boolean parameter (default is false). If true files that would be marked for deletion, will be placed to a delete-me directory.

There are two ways to learn when backup is done. First is by modification timestamp. However moving backup to a new directory will reset this timestamp, so it is not a bullet-proof solution.

To get a better experience with partitioning files by time make sure your backup files contain the following pattern in their name that corresponds to the date they were created:

yyyy-mm-dd

For example mybackup-2018-01-02.sql.tar.gz. Otherwise files modification dates will be used instead.

To check version

backme -v

To organize backups

backme

After backme run the excess of backup files will be deleted, and the rest will spread over an archive directory inside of your backup directory. The structure of the directory is following:

archive
  recent
  last-month
  2015
  2016
  2017
  2018

recent : files from the last 2 days. All existing files from this period are preserved.

last-month : files from the last 31 day, only one file per day is preserved.

yyyy (2015 for example) : files from a particular year. Only the latest file from each month is preserved

If everything works correctly, make a daily cronjob file that would run backme regularly updating the archive directories.