Comments (14)
Would be nice to have this. Thanks for putting some thought into it.
I'm wondering whether there's any merit in a two-level directory schema such as snapname.YYYYMMDD/HH:MM
.
Probably not in that it wouldn't be backward compatible and possibly a nuisance to those who don't care about hourly snapshots (though I can see value in knowing at a glance when the snapshot was taken). But it may cut away some ugly special cases.
from btrbk.
My suggestion is to use ISO 8601 which allows time to be appended to the end with a 'T' prefix. The format YYYYMMDDTHHMM is acceptable (e.g. 20150920T2200).
To handle the daily vs hourly cleanup you can create two snapshots when detecting that this is the first snapshot taken for the day. One has only the date, the other with both date and time. All snapshots after that first one until the next day with have both date and time. This way there should be no backwards compatibility issues.
A schedule of every hour should result in these snapshots for example:
/mnt/backups/home.20150919
/mnt/backups/home.20150919T0000
/mnt/backups/home.20150919T0100
/mnt/backups/home.20150919T0200
...
/mnt/backups/home.20150919T2300
/mnt/backups/home.20150920
/mnt/backups/home.20150920T0000
from btrbk.
Adding support for the proposed timestamp postfix YYYYMMDDTHHMM
should be not much of a problem to implement (there will still be a _N
postfix if run more than once a minute, but this will probably not hurt anyone). A new configuration option timestamp_format short/long
should be enough to fix this.
Creating two snapshots should not be necessary, as btrbk internally operates with UUIDs rather than filenames (with one exception being the scheduler, which looks at the file postfix).
The problem of different management of the preserve logic remains (hourly instead of daily, see point 1 of my original post), meaning that all the snapshots and backups will be preserved for the configured {snapshot,target}_preserve_daily
period. I still don't see any easy and non-confusing solution for this.
Anyways, these two things can be seen as different issues, and I think a lot of people will be happy to be able to set the timestamp postfix to a format including HHMM. It's on my TODO list, and will hopefully find it's way to btrbk-0.21.0. (yet so many other things to finish and so little time...)
from btrbk.
Implemented timestamp_format short|long
as part of raw_target (see #40): e177ae1
Note that you can enable the "long" format without the need of touching any existing snapshots/backups.
from btrbk.
I like the idea of a preserve_hourly
option, as I usually take hourly backup to minimize the amount of data loss in case of major issue. However, I usually don't need that many backups for more than 24 hours.
Also, it would be great to have a flexible day of week option. For instance, if I don't use my computer on the preserve_day_of_week
day but I use it another day of the week, it would be great to keep the next best day for backup. Another solution would be to keep the next backup which is at least 7 days younger than the previous weekly backup if the user sets preserve_day_of_week any
.
from btrbk.
I like the idea of a preserve_hourly option, as I usually take hourly backup to minimize the amount of data loss in case of major issue. However, I usually don't need that many backups for more than 24 hours.
This is still on my TODO list (as all of the "enhancements"), but as it's not that simple to implement without breaking other things, I don't give it very high priority. Furthermore, it's more a "cosmetic" issue, as your btrfs filesystem can easily handle 50+ snapshots without major slowdown. From my experience, the additional metadata usage is also pretty low. For 2000+ snapshots, you'll start to loose some performance.
Also, it would be great to have a flexible day of week option [...]
btrbk already keeps the next possible day of backup of "preserve_day_of_week". From the man-page:
target_preserve_weekly all|<number>
Defines for how many weeks back weekly backups should be preserved.
Every backup created at preserve_day_of_week (or the next backup in
this week if none was made on the exact day) is considered as a weekly
backup. Defaults to “0”.
If you run btrbk -v run|dryrun
, you'll get output from the scheduler like this: "preserved weekly: 7 weeks ago, +1 days after sunday" (having preserve_day_of_week=sunday)
from btrbk.
If you run btrbk -v run|dryrun, you'll get output from the scheduler like this: "preserved weekly: 7 weeks ago, +1 days after sunday" (having preserve_day_of_week=sunday)
My bad, I didn't read the manual, only the example script when writing my configuration.
Furthermore, it's more a "cosmetic" issue, as your btrfs filesystem can easily handle 50+ snapshots without major slowdown.
My SDD sometimes freezes since I switched to BTRFS, but I don't know if it is because of the number of snapshots or BTRFS itself.
from btrbk.
My SDD sometimes freezes since I switched to BTRFS, but I don't know if it is because of the number of snapshots or BTRFS itself.
Off-Topic:
Do you have "discard" (trim) turned on? I had similar issues on a SSD,
and disabling trim helped.
grep discard /etc/fstab
dmesg | grep discard
- Axel
from btrbk.
Do you have "discard" (trim) turned on? I had similar issues on a SSD, and disabling trim helped.
Good point thank you. It is even set on my HDD. I'll try without it as I don't think it is working very well: fstrim -v /
reports that all my free space have been trimmed, even after 2-3 runs while it was reporting a few MB when I had ext4.
from btrbk.
Here is a request I got via email:
Currently, I notice the code only allows specifying snapshot and backup preservation for days, weeks, and months. I was wondering if it would be possible to extend this range so that the user could specify years or even decades.
This can be easily implemented, and I will do it when I finally get around to close this "hourly schedule" issue.
from btrbk.
currently, setting {snapshot,target}_preserve_daily 14 means "keep ALL
snapshots/backups for 14 days", no matter how many there are (there might be several snapshots created per day)
And:
Furthermore, it's more a "cosmetic" issue, as your btrfs filesystem can easily handle 50+ snapshots without major slowdown
Frequent backups are useful for avoiding data loss. Example: CrashPlan defaults to making new backups every 15 minutes, and retaining those 15 minute backups for 7 days. Your first statement sounds like for users that want frequent backups, btrbk can already do this. Your second statement suggests the consequences are minimal. (Where this ticket is useful would be replicating the CrashPlan default of storing daily backups for 90 days.)
Maybe in the meantime, the documentation can be updated? For example,
target_preserve_daily all|
How many days of backups should be preserved. Defaults to “all”.
implied to me that only one backup per day is retained, which apparently is not the case.
from btrbk.
Implemented in a6cb0c6, on the preserve_hourly branch.
I'm not yet quite satisfied with this solution, as setting {snapshot,target}_preserve_hourly
to a number will change the semantics of preserve_daily to "preserve only last present daily for <preserve_daily> days".
Excerpt from btrbk.conf.5:
target_preserve_hourly all|<number>
Preserve all backups for the given amount of hours, regardless of
how many there are. Defaults to "all".
Note that if you use this feature, make sure to also set
timestamp_format to "long", or the scheduler will interpret the
time as "00:00".
target_preserve_daily all|<number>
If target_preserve_hourly is set to "all" (the default), preserve
all backups for the given amount of days, regardless of how many
there are. If target_preserve_hourly is set to a number, preserve
the last backup of a day for the given amount of days. Defaults to
"all".
This looks quite confusing at first sight, but it's the only way (I can think of) of implementing this without breaking compatibility. If I would change the semantics of preserve_daily
to strict "preserve only last present daily for <preserve_daily> days", the first btrbk run might delete some backups for users who created multiple backups per day and want to keep them ALL for N days.
Another solution I have in mind is to introduce a preserve_all <NN>d
, with semantics: "keep ALL backups for NN hours|days|weeks...". This would make everything much easier to understand.
from btrbk.
That said, I've implemented a more invasive variant (0a0775f in the preserve_all branch), which now allows configuration (as discussed in #70) like:
snapshot_preserve_all 24h
snapshot_preserve 48h 14d *m
with slightly new semantics for the "daily" part. Excerpt from btrbk.conf.5:
OPTIONS
[...]
snapshot_preserve <retention_policy>
Set retention policy for snapshots (see RETENTION POLICY below).
snapshot_preserve_all forever|<number>{h,d,w,m,y}
Preserve all snapshots for the given amount of hours (h), days (d),
weeks (w), months (m) or years (y), regardless of how many there
are. Defaults to “forever”.
target_preserve <retention_policy>
Set retention policy for backups (see RETENTION POLICY below).
target_preserve_all forever|<number>{h,d,w,m,y}
Preserve all backups for the given amount of hours (h), days (d),
weeks (w), months (m) or years (y), regardless of how many there
are. Defaults to “forever”.
[...]
RETENTION POLICY
A <retention_policy> is defined as:
[<hourly>h] [<daily>d] [<weekly>w] [<monthly>m] [<yearly>y]
With the following semantics:
hourly
Preserve last backup of the hour for the given amount of hours.
Note that if you use <hourly> scheduling, make sure to also set
timestamp_format to ?long?, or the scheduler will interpret the
time as "00:00".
daily
Preserve the last backup of a day for the given amount of days.
weekly
Defines for how many weeks back weekly backups should be preserved.
Every backup created at preserve_day_of_week (or the next backup in
this week if none was made on the exact day) is considered as a
weekly backup.
monthly
Defines for how many months back monthly backups should be
preserved. Every last weekly backup in a month is considered a
monthly backup.
yearly
Defines for how many years back yearly backups should be preserved.
Every last monthly backup in a year is considered a yearly backup.
Use an asterisk for ?all? (e.g. "target_preserve 60d *m" states:
"preserve daily backups for 60 days back, and all monthly backups").
Regarding compatibility: The old snapshot_preserve_{daily,weekly,monthly}
configuration options are marked as "deprecated", and if found in the config, snapshot_preserve_daily <NN>
automatically sets snapshot_preserve_all <NN>d
and some warnings are printed.
This way I can force users to adapt their config files, while nothing blows up if they run the new version of btrbk with an old config.
from btrbk.
Implemented in btrbk-v0.23.0-rc1 (a27f99a)
Please read the upgrade guide before updating to v0.23.0.
from btrbk.
Related Issues (20)
- Failed to fetch subvolume detail for snapshot_dir HOT 1
- Double daily backups sometimes... HOT 1
- understanding target_preserve_min
- Is 650 GB per day between hard disks connected by 1 GBit ethernet ok? HOT 1
- Feature request : BTRBK + grub-btrfs - Restore the snapshot currently booted - like Timeshift
- Why is btrbk only creating snapshots and not transfering?
- dry-run ignores lock
- whatever default shell is used despite it is not a posix compatible shell
- Not optimal behavior for low quality network connections
- [Question] How to specify clone sources from snapshots of other unrelated subvolumes?
- ERROR: ... unable to destroy quota group: Device or resource busy
- ERROR: clone: did not find source subvol HOT 4
- Question: duplicated instance when /etc/crontab set to run hourly at 00 minute HOT 1
- [Question] Quarterly retention
- [Feature Request] configurable timestamps / names for snapshots
- snapshot_dir required even when `snapshot_create` set to `no`
- btrbk doesn't append a command with "sudo -n" if ssh is used and backend_local_user=btrfs-progs-sudo
- [Bug Report] "btrbk diff" without "sudo" throws error (backend_local_user=btrfs-progs-sudo)
- Backup of backups HOT 1
- send to a raw target which does not support output redirection
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from btrbk.