Would be nice to have this. Thanks for putting some thought into it.

I'm wondering whether there's any merit in a two-level directory schema such as snapname.YYYYMMDD/HH:MM.

Probably not in that it wouldn't be backward compatible and possibly a nuisance to those who don't care about hourly snapshots (though I can see value in knowing at a glance when the snapshot was taken). But it may cut away some ugly special cases.

from btrbk.

My suggestion is to use ISO 8601 which allows time to be appended to the end with a 'T' prefix. The format YYYYMMDDTHHMM is acceptable (e.g. 20150920T2200).

To handle the daily vs hourly cleanup you can create two snapshots when detecting that this is the first snapshot taken for the day. One has only the date, the other with both date and time. All snapshots after that first one until the next day with have both date and time. This way there should be no backwards compatibility issues.

A schedule of every hour should result in these snapshots for example:

/mnt/backups/home.20150919
/mnt/backups/home.20150919T0000
/mnt/backups/home.20150919T0100
/mnt/backups/home.20150919T0200
...
/mnt/backups/home.20150919T2300
/mnt/backups/home.20150920
/mnt/backups/home.20150920T0000

from btrbk.

Adding support for the proposed timestamp postfix YYYYMMDDTHHMM should be not much of a problem to implement (there will still be a _N postfix if run more than once a minute, but this will probably not hurt anyone). A new configuration option timestamp_format short/long should be enough to fix this.

Creating two snapshots should not be necessary, as btrbk internally operates with UUIDs rather than filenames (with one exception being the scheduler, which looks at the file postfix).

The problem of different management of the preserve logic remains (hourly instead of daily, see point 1 of my original post), meaning that all the snapshots and backups will be preserved for the configured {snapshot,target}_preserve_daily period. I still don't see any easy and non-confusing solution for this.

Anyways, these two things can be seen as different issues, and I think a lot of people will be happy to be able to set the timestamp postfix to a format including HHMM. It's on my TODO list, and will hopefully find it's way to btrbk-0.21.0. (yet so many other things to finish and so little time...)

from btrbk.

Implemented timestamp_format short|long as part of raw_target (see #40): e177ae1

Note that you can enable the "long" format without the need of touching any existing snapshots/backups.

from btrbk.

I like the idea of a preserve_hourly option, as I usually take hourly backup to minimize the amount of data loss in case of major issue. However, I usually don't need that many backups for more than 24 hours.

Also, it would be great to have a flexible day of week option. For instance, if I don't use my computer on the preserve_day_of_week day but I use it another day of the week, it would be great to keep the next best day for backup. Another solution would be to keep the next backup which is at least 7 days younger than the previous weekly backup if the user sets preserve_day_of_week any.

from btrbk.

I like the idea of a preserve_hourly option, as I usually take hourly backup to minimize the amount of data loss in case of major issue. However, I usually don't need that many backups for more than 24 hours.

This is still on my TODO list (as all of the "enhancements"), but as it's not that simple to implement without breaking other things, I don't give it very high priority. Furthermore, it's more a "cosmetic" issue, as your btrfs filesystem can easily handle 50+ snapshots without major slowdown. From my experience, the additional metadata usage is also pretty low. For 2000+ snapshots, you'll start to loose some performance.

Also, it would be great to have a flexible day of week option [...]
btrbk already keeps the next possible day of backup of "preserve_day_of_week". From the man-page:

target_preserve_weekly  all|<number>
       Defines for how many weeks back weekly backups should be preserved.
       Every backup created at preserve_day_of_week (or the next backup in
       this week if none was made on the exact day) is considered as a weekly
       backup. Defaults to “0”.

If you run btrbk -v run|dryrun, you'll get output from the scheduler like this: "preserved weekly: 7 weeks ago, +1 days after sunday" (having preserve_day_of_week=sunday)

from btrbk.

If you run btrbk -v run|dryrun, you'll get output from the scheduler like this: "preserved weekly: 7 weeks ago, +1 days after sunday" (having preserve_day_of_week=sunday)

My bad, I didn't read the manual, only the example script when writing my configuration.

Furthermore, it's more a "cosmetic" issue, as your btrfs filesystem can easily handle 50+ snapshots without major slowdown.

My SDD sometimes freezes since I switched to BTRFS, but I don't know if it is because of the number of snapshots or BTRFS itself.

from btrbk.

My SDD sometimes freezes since I switched to BTRFS, but I don't know if it is because of the number of snapshots or BTRFS itself.

Off-Topic:

Do you have "discard" (trim) turned on? I had similar issues on a SSD,
and disabling trim helped.

grep discard /etc/fstab

dmesg | grep discard

• Axel

from btrbk.

Do you have "discard" (trim) turned on? I had similar issues on a SSD, and disabling trim helped.

Good point thank you. It is even set on my HDD. I'll try without it as I don't think it is working very well: fstrim -v / reports that all my free space have been trimmed, even after 2-3 runs while it was reporting a few MB when I had ext4.

from btrbk.

Here is a request I got via email:

Currently, I notice the code only allows specifying snapshot and backup preservation for days, weeks, and months. I was wondering if it would be possible to extend this range so that the user could specify years or even decades.

This can be easily implemented, and I will do it when I finally get around to close this "hourly schedule" issue.

from btrbk.

currently, setting {snapshot,target}_preserve_daily 14 means "keep ALL
snapshots/backups for 14 days", no matter how many there are (there might be several snapshots created per day)

And:

Furthermore, it's more a "cosmetic" issue, as your btrfs filesystem can easily handle 50+ snapshots without major slowdown

Frequent backups are useful for avoiding data loss. Example: CrashPlan defaults to making new backups every 15 minutes, and retaining those 15 minute backups for 7 days. Your first statement sounds like for users that want frequent backups, btrbk can already do this. Your second statement suggests the consequences are minimal. (Where this ticket is useful would be replicating the CrashPlan default of storing daily backups for 90 days.)

Maybe in the meantime, the documentation can be updated? For example,

target_preserve_daily all|
How many days of backups should be preserved. Defaults to “all”.

implied to me that only one backup per day is retained, which apparently is not the case.

from btrbk.

Implemented in a6cb0c6, on the preserve_hourly branch.

I'm not yet quite satisfied with this solution, as setting {snapshot,target}_preserve_hourly to a number will change the semantics of preserve_daily to "preserve only last present daily for <preserve_daily> days".

Excerpt from btrbk.conf.5:

target_preserve_hourly  all|<number>
    Preserve all backups for the given amount of hours, regardless of
    how many there are. Defaults to "all".

    Note that if you use this feature, make sure to also set
    timestamp_format to "long", or the scheduler will interpret the
    time as "00:00".

target_preserve_daily  all|<number>
    If target_preserve_hourly is set to "all" (the default), preserve
    all backups for the given amount of days, regardless of how many
    there are. If target_preserve_hourly is set to a number, preserve
    the last backup of a day for the given amount of days. Defaults to
    "all".

This looks quite confusing at first sight, but it's the only way (I can think of) of implementing this without breaking compatibility. If I would change the semantics of preserve_daily to strict "preserve only last present daily for <preserve_daily> days", the first btrbk run might delete some backups for users who created multiple backups per day and want to keep them ALL for N days.

Another solution I have in mind is to introduce a preserve_all <NN>d, with semantics: "keep ALL backups for NN hours|days|weeks...". This would make everything much easier to understand.

from btrbk.

That said, I've implemented a more invasive variant (0a0775f in the preserve_all branch), which now allows configuration (as discussed in #70) like:

snapshot_preserve_all   24h
snapshot_preserve       48h  14d  *m

with slightly new semantics for the "daily" part. Excerpt from btrbk.conf.5:

OPTIONS

[...]

       snapshot_preserve  <retention_policy>
           Set retention policy for snapshots (see RETENTION POLICY below).

       snapshot_preserve_all  forever|<number>{h,d,w,m,y}
           Preserve all snapshots for the given amount of hours (h), days (d),
           weeks (w), months (m) or years (y), regardless of how many there
           are. Defaults to “forever”.

       target_preserve  <retention_policy>
           Set retention policy for backups (see RETENTION POLICY below).

       target_preserve_all  forever|<number>{h,d,w,m,y}
           Preserve all backups for the given amount of hours (h), days (d),
           weeks (w), months (m) or years (y), regardless of how many there
           are. Defaults to “forever”.

[...]

RETENTION POLICY
       A <retention_policy> is defined as:

           [<hourly>h] [<daily>d] [<weekly>w] [<monthly>m] [<yearly>y]

       With the following semantics:

       hourly
           Preserve last backup of the hour for the given amount of hours.
           Note that if you use <hourly> scheduling, make sure to also set
           timestamp_format to ?long?, or the scheduler will interpret the
           time as "00:00".

       daily
           Preserve the last backup of a day for the given amount of days.

       weekly
           Defines for how many weeks back weekly backups should be preserved.
           Every backup created at preserve_day_of_week (or the next backup in
           this week if none was made on the exact day) is considered as a
           weekly backup.

       monthly
           Defines for how many months back monthly backups should be
           preserved. Every last weekly backup in a month is considered a
           monthly backup.

       yearly
           Defines for how many years back yearly backups should be preserved.
           Every last monthly backup in a year is considered a yearly backup.

       Use an asterisk for ?all? (e.g. "target_preserve 60d *m" states:
       "preserve daily backups for 60 days back, and all monthly backups").

Regarding compatibility: The old snapshot_preserve_{daily,weekly,monthly} configuration options are marked as "deprecated", and if found in the config, snapshot_preserve_daily <NN> automatically sets snapshot_preserve_all <NN>d and some warnings are printed.

This way I can force users to adapt their config files, while nothing blows up if they run the new version of btrbk with an old config.

from btrbk.

Implemented in btrbk-v0.23.0-rc1 (a27f99a)

Please read the upgrade guide before updating to v0.23.0.

from btrbk.