Giter Club home page Giter Club logo

ansible-role-borgbackup's Introduction

Ansible Role: BorgBackup Client

Test Ansible Galaxy

Set up encrypted, compressed and deduplicated backups using BorgBackup and Borgmatic. Currently supports Debian/Ubuntu, CentOS/Red Hat/Fedora, Archlinux and Manjaro.

Works great with BorgBase.com - Simple and Secure Hosting for your Borg Repositories. To manage BorgBase repos via Ansible, also see Andy Hawkins' BorgBase Collection.

Main features

  • Install Borg and Borgmatic from PyPi or distro packages
  • Set up Borgmatic config
  • Schedule regular backups using Cron or Systemd timer

Breaking changes

  • Older versions of this role set up a separate Cron job for creating and checking backups. With recent Borgmatic version, this feature is now managed in Borgmatic. As a result the extra Cron job will be removed by this role.
  • Older versions of this role only supported Cron for scheduling. If you use Systemd timers, be sure to remove the Cron job in /etc/cron.d/borgmatic first. The role will also alert you when trying to use both timers.

Example playbook with root as backup user and Cron timer

- hosts: all
  roles:
  - role: borgbase.ansible_role_borgbackup
    borg_encryption_passphrase: CHANGEME
    borg_repository:
      - ssh://[email protected]/./repo
    borg_source_directories:
      - /var/www
    borgmatic_hooks:
      before_backup:
      - echo "`date` - Starting backup."
      postgresql_databases:
      - name: users
        hostname: database1.example.org
        port: 5433

Example playbook with service user and Systemd timer

- hosts: all
  roles:
  - role: borgbase.ansible_role_borgbackup
    borg_encryption_passphrase: CHANGEME
    borg_repository: ssh://[email protected]/./repo
    borgmatic_timer: systemd
    borg_user: "backupuser"
    borg_group: "backupuser"
    borg_source_directories:
      - /var/www
    borg_retention_policy:
      keep_hourly: 3
      keep_daily: 7
      keep_weekly: 4
      keep_monthly: 6

Installation

Download from Ansible Galaxy

$ ansible-galaxy install borgbase.ansible_role_borgbackup

Clone latest version from Github

$ git clone https://github.com/borgbase/ansible-role-borgbackup.git roles/ansible_role_borgbackup

Role Variables

Required Variables

  • borg_repository: Full path to repository. Your own server or BorgBase.com repo. Can be a list if you want to backup to multiple repositories.

Optional Variables

  • borg_dep_packages: Dependency Packages to install borg(backup) and borgmatic.

  • borg_distro_packages: contains the names of distributions packages for borg(backup) and borgmatic, only used if borg_install_method is set to package.

  • borg_encryption_passcommand: The standard output of this command is used to unlock the encryption key.

  • borg_encryption_passphrase: Password to use for repokey or keyfile. Empty if repo is unencrypted.

  • borg_exclude_from: Read exclude patterns from one or more separate named files, one pattern per line.

  • borg_exclude_patterns: Paths or patterns to exclude from backup. See official documentation for more.

  • borg_install_method: By default pip is used to install borgmatic. To install via your distributions package manager set this to package and (if needed) overwrite the borg_distro_packages variable to contain your distributions package names required to install borgmatic. Note that many distributions ship outdated versions of borgbackup and borgmatic; use at your own risk.

  • borg_require_epel: When using borg_install_method: package on RHEL-based distributions, the EPEL repo is required. To disable the check (e.g. when using a custom mirror instead of the epel-release package), set this to false. Defaults to {{ ansible_os_family == 'RedHat' and ansible_distribution != 'Fedora' }} (i.e. true on Enterprise Linux-based distros).

  • borg_lock_wait_time: Config maximum seconds to wait for acquiring a repository/cache lock. Defaults to 5 seconds.

  • borg_one_file_system: Don't cross file-system boundaries. Defaults to true

  • borg_pip_packages: Dependancy Packages (pip) to install borg(backup) and borgmatic.

  • borg_remote_path: Path to the borg executable on the remote. It will default to borg.

  • borg_remote_rate_limit: Remote network upload rate limit in kiBytes/second.

  • borg_retention_policy: Retention policy for how many backups to keep in each category (daily, weekly, monthly, etc).

  • borg_source_directories: List of local folders to back up. Default is /etc/hostname to prevent an empty backup.

  • borg_ssh_key_name: Name of the SSH public and pivate key. Default id_ed25519

  • borg_ssh_key_file_path: SSH-key to be used. Default ~/.ssh/{{ borg_ssh_key_name }}

  • borg_ssh_key_type: The algorithm used to generate the SSH private key. Choose: rsa, dsa, rsa1, ecdsa, ed25519. Default: ed25519

  • borg_ssh_command: Command to use instead of just "ssh". This can be used to specify SSH options.

  • borg_version: Force a specific borg version to be installed

  • borg_venv_path: Path to store the venv for borg(backup) and borgmatic

  • borgmatic_check_last: Number of archives to check. Defaults to 3

  • borgmatic_checks: List of consistency checks. Defaults to monthly checks. See docs for all options.

  • borgmatic_config_name: Name to use for the Borgmatic config file. Defaults to config.yaml

  • borgmatic_timer_hour: Hour when regular create and prune cron/systemd-timer job will run. Defaults to {{ 6 | random }}

  • borgmatic_timer_minute: Minute when regular create and prune cron/systemd-timer job will run. Defaults to {{ 59 | random }}

  • borgmatic_hooks: Hooks to monitor your backups e.g. with Healthchecks. See official documentation for more.

  • borgmatic_timer: If the variable is set, a timer is installed. A choice must be made between cron and systemd.

  • borgmatic_relocated_repo_access_is_ok: Bypass Borg error about a repository that has been moved. Defaults to false

  • borgmatic_store_atime: Store atime into archive. Defaults to true

  • borgmatic_store_ctime: Store ctime into archive. Defaults to true

  • borgmatic_version: Force a specific borgmatic version to be installed

  • borg_user: Name of the User to create Backups (service account)

  • borg_group: Name of the Group to create Backups (service account)

Contributing

Pull requests (PR) are welcome, as long as they add features that are relevant for a meaningful number of users. All PRs are tested for style and functionality. To run tests locally (needs Docker):

$ pip install -r requirements-dev.txt
$ molecule test

License

MIT/BSD

Author

© 2018-2023 Manuel Riel and contributors.

ansible-role-borgbackup's People

Contributors

conloos avatar dnmvisser avatar doesnotcompete avatar ekamil avatar goneri avatar jobroe avatar jonas-hagen avatar jorti avatar karouf avatar kenayagi avatar kgizdov avatar kogelvis avatar lukasleitsch avatar m3nu avatar madhermit avatar marienfressinaud avatar michalroxorpl avatar moregeek avatar p-rintz avatar ramblurr avatar samh avatar samyak-jain avatar savoiringfaire avatar sebclem avatar stroobl avatar strugee avatar tabic avatar varac avatar wolfskaempf avatar woomymy avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ansible-role-borgbackup's Issues

Review performance-related options

Best performance and upstream default:

    atime: false
    bsd_flags: false
    ctime: true
  • Conservative new default: atime=false, flags=true, ctime=true
  • Review options for flags (bsd, acl)

borg v2?

Hi,
Just wondering if there is an option for compatibility with borg v2?
I didnt see anything related to this in the repo documentation...
Thanks

AnsibleUndefinedVariable: 'ansible.parsing.yaml.objects.AnsibleUnicode object' has no attribute 'h'

Playbook config:

- hosts: all
  become: true
  debugger: on_failed

  roles:
  - role: m3nu.ansible_role_borgbackup
    borg_encryption_passphrase: 70E348BE-B802-4ED3-AB3E-710CA8925398
    borg_repository: ssh://[email protected]/./backup/vultr/instance-1
    borgmatic_hooks: https://hc-ping.com/a-b-c-d
    borg_exclude_patterns:
      - '*.pyc'
      - '~/*/.cache'
      - '.DS_Store'
      - '/var/lib/lxcfs/'
      - '/var/cache/*'
      - '/var/tmp/*'
      - '/home/*/.cache/*'
      - '/root/.cache/*'
      - '/var/lib/yum/yumdb'
    borg_remote_path: borg1
    borg_source_directories:
      - '/etc'
      - '/home'
      - '/root'
      - '/var'
      - '/srv'
    borg_retention_policy:
      keep_daily: 3
      keep_weekly: 2
      keep_monthly: 2
      keep_yearly: 2

Partial logs:

TASK [m3nu.ansible_role_borgbackup : Set Repository Fact] **********************
skipping: [instance-1]

TASK [m3nu.ansible_role_borgbackup : Ensures /etc/borgmatic exists] ************
ok: [instance-1]

TASK [m3nu.ansible_role_borgbackup : Add Borgmatic Configuration] **************
fatal: [instance-1]: FAILED! => {"changed": false, "msg": "AnsibleUndefinedVariable: 'ansible.parsing.yaml.objects.AnsibleUnicode object' has no attribute 'h'"}

Running on CentOS 7 x64.

$ ansible-playbook --version
ansible-playbook 2.10.3
  config file = /Users/birkhofflee/.ansible.cfg
  configured module search path = ['/Users/birkhofflee/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/Cellar/ansible/2.10.4/libexec/lib/python3.9/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.9.1 (default, Feb 13 2021, 13:16:42) [Clang 12.0.0 (clang-1200.0.32.29)]

I have tried both the version on Ansible Galaxy and here master branch of this repo. Please shed some light on this issue. Thanks

llfuse is not installed (used for borg mount)

Hi,

The llfuse python module is missing during the Borg installation. We need it to use borgmatic mount

In debian you need to add these apt package:

  • fuse
  • libfuse-dev

and the llfuse Pypi package in the Borgmatic venv.

I don't know for the others distribution witch package is needed.

MySQL/PostgreSQL backups

Hi, first of all, thanks for the role it's really helpful.

I saw that you implemented before/after backups commands, but in borgmatic databases dump hooks are native. Is it possible to use them with your role ? It would a shame to use this role and have to deploy a jinja template of the config file just for those commands.

Thanks.

No module named "pip" Ubuntu 22.04 LTS

I'm using borg on both my linux server with Ubuntu 22.04 LTS installed. I updated to the version today. After running my playbook with this role included I got the following error.

image

Some parameters are broken with latest borgmatic releases

Hello,

I've been using your role a lot lately for our servers and I appreciate your work. Thank you.

This morning I found out a problem with the latest Borgmatic version which breaks the cron task.
Basically, this role always installs the latest Borgmatic pip package (variable borg_python_packages). However, from Borgmatic version Borgmatic 1.7.0 onwards, Borgmatic parameters such as --create or --prune (which are the parameters called by the cron task) have been deprecated in favor of create or -c for example for creating an archive.

I see two solutions for this quite major issue, since my latest installation of Borg using this role wouldn't work (the cron task failed) because of this change of parameters:

  • Create a new variable which gives the user of this role the ability to set a specific version of the Borgmatic pip package (such as it is done here
  • Or, update the various Borgmatic parameters in the different tasks to make them compatible with the latest Borgmatic update.

Personally, as a workaround, I have written a post-task which rollback Borgmatic pip package to 1.6.5 to be compatible with your parameters. However, it can't be a permanent solution.

Thank you!

ModuleNotFoundError: No module named …

Hello,
I am facing the same problem than #98 since I upgrade from Debian 10 to Debian 11.

  • python3-pip is installed
  • pip is also installed by geerlingguy.pip (2.2.0) ansible role

I notice that this role uses /opt/borgmatic/bin/pip3 and /opt/borgmatic/bin/pip. Removing them and creating a symlink (ln -s /usr/local/bin/pip3 /opt/borgmatic/bin/pip3 and ln -s /usr/local/bin/pip /opt/borgmatic/bin/pip) solve this problem.

Now I have a new problem when using borgmatic command, probably linked to previous one: "ModuleNotFoundError: No module named 'borgmatic'".

But

root:~# pip3 list | grep borg
borgbackup          1.2.3
borgmatic           1.7.8
root:~# which borgmatic
/usr/local/bin/borgmatic

root:~# whereis borgmatic
borgmatic: /etc/borgmatic /usr/local/bin/borgmatic /opt/borgmatic/bin/borgmatic

root:~# borgmatic --version
Traceback (most recent call last):
  File "/opt/borgmatic/bin/borgmatic", line 5, in <module>
    from borgmatic.commands.borgmatic import main
ModuleNotFoundError: No module named 'borgmatic'

root:~# /usr/local/bin/borgmatic --version
Traceback (most recent call last):
  File "/opt/borgmatic/bin/borgmatic", line 5, in <module>
    from borgmatic.commands.borgmatic import main
ModuleNotFoundError: No module named 'borgmatic'

root:~# /opt/borgmatic/bin/borgmatic --version
Traceback (most recent call last):
  File "/opt/borgmatic/bin/borgmatic", line 5, in <module>
    from borgmatic.commands.borgmatic import main
ModuleNotFoundError: No module named 'borgmatic'

I don't understand what is the problem now.

Create mode removed

It looks like the create mode has been removed in #100. There is trace in the documentation on how it should be replaced, not even a reference to https://galaxy.ansible.com/adhawkins/borgbase and nothing in the release notes. This is not really friendly for the users that have been using the role with borgbase. Suddenly a feature is gone, you can no longer create new repos and you can start digging around to find out how to replace it...

Improve role idempotence and avoid duplicate object creation at Borgbase.com

I'm currently testing to replace our existing Borg backup role with this one and the repos hosted at Borgbase and some issues I ran into as a new user:

  • idempotence for cron - fix proposed in PR #54
  • if you rerun the creation of a repo and it already exists, it should be made consistent with the requested state IMHO (and it that case, you can probably drop the create mode). This appears to be included in PR #53.
  • Same for SSH keys: seen the fact that there's an API, it should probably be possible to check if an SSH key is already defined on borgbase.com and to avoid to recreate it?

What's your idea about these proposed changes?

location.exclude_patterns': None is not of type 'array'

It looks like something has changed in recent borgmatic versions. If I reinstall borgmatic after a dist-upgrade I now get:

/etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found

summary:
/etc/borgmatic/config.yaml: Error parsing configuration file
An error occurred while parsing a configuration file at /etc/borgmatic/config.yaml:
At 'location.exclude_patterns': None is not of type 'array'
At 'location.exclude_from': None is not of type 'array'

I did a quick test and setting both parameters to an empty array fixes the issue. I'll try to send a fix PR later.

Remove outdated include statement(s)

TASK [m3nu.ansible_role_borgbackup : Run OS-specific tasks] ********************
  [DEPRECATION WARNING]: "include" is deprecated, use
  include_tasks/import_tasks/import_playbook instead. This feature will be
  removed in version 2.16. Deprecation warnings can be disabled by setting
  deprecation_warnings=False in ansible.cfg.

Allow multiple config files

First of all, thx for this nice role!

Please allow multiple config files so hosts can get configured to backup to 2 locations i.e.

Support to copy config to management server for pruning?

With my previous Borg backup role (without Borgmatic - it didn't exist at the time we started using Attic), I had a secured management host where I did the pruning for all hosts configured in append only mode.
Would you be interested to add a feature to this role that makes it possible to copy the config to such a management host?

I already did some tests and something like this should work:

- name: Copy config to management host (for pruning)
  template:
    src: config.yaml.j2
    dest: "{{ borgmatic_mgmt_host_path }}/{{ inventory_hostname }}.yaml"
    mode: 0600
  remote_user: "{{ borgmatic_mgmt_user }}"
  become: false
  delegate_to: "{{ borgmatic_mgmt_host }}"
  when: borgmatic_mgmt_host is defined

Cython requirement

Seems the role should check for Cython being installed. Had to install it on a debian 10 system for borgbackup to install, if not, i got the following error:

raise ImportError('The GIT version of Borg needs Cython. Install Cython or use a released version.')\n

Can we create local backups?

From my understanding, this seems to only work for remote borg repos? Can I use this to automate backing up to a local repo? My use case is to use rclone in the post backup hook and move the backup to either s3 or b2.

pip depreciation notice

[DEPRECATION WARNING]: Invoking "pip" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply
multiple items and specifying name: "{{ item }}", please use name: '{{ borg_python_packages }}' and remove the loop. This feature will
be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

ansible 2.7.7

Missing c flag in cronjob

If i set borgmatic_config_name and borgmatic_large_repo to false the cronjob is missing the -c flag for the specific config file.

- name: Add cron-job for borgmatic (normal-sized repo)
  block:
    - name: Add cron job for create, check and prune
      cron:
        name: "{{ borgmatic_cron_name }}"
        hour: "{{ borgmatic_cron_hour }}"
        minute: "{{ borgmatic_cron_minute }}"
        user: "root"
        cron_file: "{{ borgmatic_cron_name }}"
        job: "/usr/local/bin/borgmatic"
    - name: Ensure separate check cron job is absent
      cron:
        name: "{{ borgmatic_cron_name }}-check"
        state: absent
  when: not borgmatic_large_repo

Auto-create of BorgBase repo not working because of missing borgbase_api_client

The role cant create a borgbase repo, because it can't find the borgbase_api_client.

FAILED! => {"msg": "Could not find imported module support code for ansible.modules.borgbase. Looked for (['ansible.module_utils.borgbase_api_client.client.GraphQLClient', 'ansible.module_utils.borgbase_api_client.client'])"}

If I follow the README in module_utils, and pit the borgbase_api_client from the repo (not the repo itself), after which it works. (Edited because it did not work at first, because I did not properly set my API-Token

I would expect that the dependencies get installed automatically, or are at least documented how to get them up and running.

Am I missing something? Help would be apprecieated, thanks

My requirements.yml:

- src: m3nu.ansible_role_borgbackup

And I run the role like this:

- hosts: example.com
  become: true
  roles:
  - role: m3nu.ansible_role_borgbackup
    borg_encryption_passphrase: "{{ lookup('env', 'BORG_ENCRYPTION_PASSPHRASE') }}"
    create_repo: True
    bb_token: "{{ lookup('env', 'BORGBASE_TOKEN') }}"
    bb_region: eu
    bb_new_sshkey: True
    borg_source_directories:
      - /example/dir

/usr/local/bin/borgmatic calls itself recursively on Fedora

Problem

/usr/local/bin/borgmatic calls itself recursively on Fedora systems

Explanation

This commit has introduced a bug on Fedora systems (and possibly Fedora derivatives like CentOs)

b99de01

For Fedora systems borg_python_packages is not defined, so the pip install never happens.

This results in the first borgmatic in PATH being /usr/local/bin/borgmatic, so when the script calls borgmatic it calls itself.

It seems that Fedora was forgotten when the switch from distro packages to pip occurred, as on Fedora the distro package is still installed.

Proposed Solutions

One solution is to simply define the proper borg_python_packages so that the pip install occurs.

However I want to push back against the "always install from pip" behavior.

Some of use prefer using distro packages when available. Eventually I'd like to see that as an option, but for now I'll create a PR that correctly installs on fedora systems.

Multiple entries created for SSH-key on borgbase.com web gui

We found out after some testing with default settings, keys are imported multiple times in the GUI. If you run the script multiple times, you'll see that multiple entries are added to the Borgbase.com SSH-keys page. Check screenshot below.

These are our settings:

---
# Full path to repository. Your own server or BorgBase.com repo. Not required when using auto creation of repositories. Can be a list if you want to backup to multiple repositories.
borg_encryption_passphrase: "testing"
borg_source_directories:
  - /etc/netplan
borg_exclude_patterns:
  - /srv/www/old-sites
borg_retention_policy:
  keep_hourly: 1
  keep_daily: 1
  keep_weekly: 1
  keep_monthly: 1

# Variables
create_repo: True
bb_token: "aay..................Q"
bb_region: "eu"

Screenshot

Allow extracting files from a backup archive

Dear contributors,

I'd like to kindly request adding the necessary code to be able to extract files from an archive in a remote repository. This, in turn, could be use in combination with other tasks to, say, restore a database, etc.

It would be the equivalent to executing the following commands in the console:

BORG_PASSPHRASE="CHANGEME" \
[email protected]:repo \
borg extract --strip-components 3 ::<archive> var/backups/postgresql/database.bak

The command would be executed in the host, so the file would be extracted in the host where Ansible is connected to. The target directory (e.g. /root/database.bak) should be configurable.

Thanks in advance.

Feature Request: provide method for running config changes only

Would be nice if there was a way so that if we need to changes configs such as retention policies or directory exclusions, etc, we can run only the needed tasks for that instead of the whole install process. It might already be possible, but my limited experience with ansible is hindering my ability to see it. Right now I am using the following in my ansible hosts file to set a groups vars for the role:

vars:
  borg_source_directories: ['/home/virtual']
  borg_exclude_patterns: ['/home/virtual/FILESYSTEMTEMPLATE','/home/virtual/site*/fst','/home/virtual/site*/shadow/var/lib/mysql','/home/virtual/site*/shadow/var/lib/pgsql']
  borg_retention_policy:
    keep_hourly: 3
    keep_daily: 7
    keep_weekly: 4
    keep_monthly: 6

Then in my borg_config.yaml playbook, i just have something as simple as:

- hosts: apnscp

  roles:
  - role: borgbackup
    borg_repository: "{{ ansible_local['borgbase_repo'] }}"

Now I just have to figure out if its possible to only run the config/cron related tasks.

properly merge borgmatic_hooks and extra variables

at least 3 extra variables failure_command, before_backup and after_backup are introduced while the rest of the hooks configuration has to be added manually in the hooks variable.

hooks:
# List of one or more shell commands or scripts to execute before creating a backup.
before_backup:
{% for cmd in borgmatic_before_backup_command %}
- {{ cmd }}
{% endfor %}
# List of one or more shell commands or scripts to execute after creating a backup.
after_backup:
{% for cmd in borgmatic_after_backup_command %}
- {{ cmd }}
{% endfor %}
# List of one or more shell commands or scripts to execute in case an exception has occurred.
on_error:
{% for cmd in borgmatic_failure_command %}
- {{ cmd }}
{% endfor %}
{% for hook in borgmatic_hooks %}
{{ hook }}: {{ borgmatic_hooks[hook] }}
{% endfor %}

this can cause errors like the following if one configures the role using e.g. on_error instead of failure_command:

Message: Validation_error('/etc/borgmatic/config.yaml', ('while constructing a mapping\n  in "/etc/borgmatic/config.yaml", line 123, column 5\nfound duplicate key "on_error" with value "[]" (original value: "[]")\n  in "/etc/borgmatic/config.yaml", line 135, column 5\n\nTo suppress this check see:\n    http://yaml.readthedocs.io/en/latest/api.html#duplicate-keys\n',))

I suggest this should be documented more clearly and further avoided altogether by merging these variables appropriately.

Allow for supressing of cron email notifications

I'm using this role to configure backup for hosts in a Proxmox cluster. Proxmox is debian based and includes postfix out of the box to send cron notifications.

I want to leave cron notifications enabled generally, but for the borgmatic job I want to disable them (as I have healthchecks alerting me to failed runs). The proper way to do this is to either append >/dev/null 2>&1 to the command so no output is generated, or pass --verbosity=-1 to borgmatic, or to define MAILTO="" in the crontab file.

The first and second is more portable, while the third might depend on the cron implementation.

Passing the --verbosity=-1 flag will supress output in the happy-path, but if there is an error it still produces output.

Personally, I'd rather no emails be sent and instead rely on healthchecks to notify me.

I propose borgmatic_cron_supress_output boolean variable to conditionally append the /dev/null redirect to the command.

Error on borgmartic configuration on a Debian 10 and 11 server

I have this error on a Debian 10 / 11 server:

TASK [m3nu.ansible_role_borgbackup : Add Borgmatic Configuration] **************
fatal: [fuster.calbasi.net]: FAILED! => {"changed": false, "msg": "AnsibleError: Unexpected templating type error occurred on (# Full config: https://torsion.org/borgmatic/docs/reference/config.yaml\nlocation:\n source_directories:\n{% for dir in borg_source_directories %}\n - {{ dir }}\n{% endfor %}\n\n # Stay in same file system (do not cross mount points).\n one_file_system: {{ borg_one_file_system }}\n repositories:\n{% if borg_repository is iterable and (borg_repository is not string and borg_repository is not mapping) %}\n {% for repo in borg_repository %}\n - {{ repo }}\n {% endfor %}\n{% elif borg_repository is defined and borg_repository is string %}\n - {{ borg_repository }}\n{% endif %}\n\n # Store atime into archive.\n atime: {{ borgmatic_store_atime }}\n\n # Store ctime into archive.\n ctime: {{ borgmatic_store_ctime }}\n\n{% if borg_exclude_patterns|length %}\n # Any paths matching these patterns are excluded from backups. Globs and tildes\n # are expanded. See the output of "borg help patterns" for more details.\n exclude_patterns:\n{% for dir in borg_exclude_patterns %}\n - '{{ dir }}'\n{% endfor %}\n{% endif %}\n\n{% if borg_exclude_from|length %}\n # Read exclude patterns from one or more separate named files, one pattern per\n # line. See the output of "borg help patterns" for more details.\n exclude_from:\n{% for dir in borg_exclude_from %}\n - {{ dir }}\n{% endfor %}\n{% endif %}\n\n # Exclude directories that contain a CACHEDIR.TAG file. See\n # http://www.brynosaurus.com/cachedir/spec.html for details.\n exclude_caches: true\n\n # Exclude directories that contain a file with the given filename.\n exclude_if_present: .nobackup\n\n # Alternate Borg remote executable. Defaults to "borg".\n # remote_path: borg1\n{% if borg_remote_path %}\n remote_path: {{ borg_remote_path }}\n{% endif %}\n\n# Repository storage options. See\n# https://borgbackup.readthedocs.io/en/stable/usage.html#borg-create and\n# https://borgbackup.readthedocs.io/en/stable/usage/general.html#environment-variables for\n# details.\nstorage:\n encryption_passphrase: {{ borg_encryption_passphrase }}\n\n # The standard output of this command is used to unlock the encryption key. Only\n # use on repositories that were initialized with passcommand/repokey encryption.\n # Note that if both encryption_passcommand and encryption_passphrase are set,\n # then encryption_passphrase takes precedence.\n # encryption_passcommand: secret-tool lookup borg-repository repo-name\n{% if borg_encryption_passcommand %}\n encryption_passcommand: {{ borg_encryption_passcommand }}\n{% endif %}\n\n # Type of compression to use when creating archives. See\n # https://borgbackup.readthedocs.org/en/stable/usage.html#borg-create for details.\n # Defaults to no compression.\n compression: auto,zstd\n\n # Remote network upload rate limit in kiBytes/second.\n{% if borg_remote_rate_limit %}\n remote_rate_limit: {{ borg_remote_rate_limit }}\n{% endif %}\n\n # Command to use instead of just "ssh". This can be used to specify ssh options.\n # ssh_command: ssh -i ~/.ssh/id_ed25519\n{% if borg_ssh_command %}\n ssh_command: {{ borg_ssh_command }}\n{% endif %}\n\n # Umask to be used for borg create.\n umask: 0077\n\n # Maximum seconds to wait for acquiring a repository/cache lock.\n lock_wait: {{ borg_lock_wait_time }}\n\n # Name of the archive. Borg placeholders can be used. See the output of\n # "borg help placeholders" for details. Default is\n # "{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}". If you specify this option, you must\n # also specify a prefix in the retention section to avoid accidental pruning of\n # archives with a different archive name format. And you should also specify a\n # prefix in the consistency section as well.\n archive_name_format: '{hostname}-{now:%Y-%m-%d-%H%M%S}'\n\n # Bypass Borg error about a repository that has been moved.\n relocated_repo_access_is_ok: {{ borgmatic_relocated_repo_access_is_ok }}\n\n# Retention policy for how many backups to keep in each category. See\n# https://borgbackup.readthedocs.org/en/stable/usage.html#borg-prune for details.\n# At least one of the "keep" options is required for pruning to work.\nretention:\n{% if borg_retention_policy.keep_secondly is defined %}\n # Number of secondly archives to keep.\n keep_secondly: {{ borg_retention_policy.keep_secondly }}\n{% endif %}\n\n{% if borg_retention_policy.keep_minutely is defined %}\n # Number of minutely archives to keep.\n keep_minutely: {{ borg_retention_policy.keep_minutely }}\n{% endif %}\n\n{% if borg_retention_policy.keep_hourly is defined %}\n # Number of hourly archives to keep.\n keep_hourly: {{ borg_retention_policy.keep_hourly }}\n{% endif %}\n\n{% if borg_retention_policy.keep_daily is defined %}\n # Number of daily archives to keep.\n keep_daily: {{ borg_retention_policy.keep_daily }}\n{% endif %}\n\n{% if borg_retention_policy.keep_weekly is defined %}\n # Number of weekly archives to keep.\n keep_weekly: {{ borg_retention_policy.keep_weekly }}\n{% endif %}\n\n{% if borg_retention_policy.keep_monthly is defined %}\n # Number of monthly archives to keep.\n keep_monthly: {{ borg_retention_policy.keep_monthly }}\n{% endif %}\n\n{% if borg_retention_policy.keep_yearly is defined %}\n # Number of yearly archives to keep.\n keep_yearly: {{ borg_retention_policy.keep_yearly }}\n{% endif %}\n\n # When pruning, only consider archive names starting with this prefix.\n # Borg placeholders can be used. See the output of "borg help placeholders" for\n # details. Default is "{hostname}-".\n prefix: '{hostname}-'\n\n# Consistency checks to run after backups. See\n# https://borgbackup.readthedocs.org/en/stable/usage.html#borg-check and\n# https://borgbackup.readthedocs.org/en/stable/usage.html#borg-extract for details.\nconsistency:\n # List of one or more consistency checks to run: "repository",\n # "archives", "data", and/or "extract". Defaults to\n # "repository" and "archives". Set to "disabled" to disable\n # all consistency checks. "repository" checks the consistency\n # of the repository, "archives" checks all of the archives,\n # "data" verifies the integrity of the data within the\n # archives, and "extract" does an extraction dry-run of the\n # most recent archive. Note that "data" implies "archives".\n checks:\n {% for checks in borgmatic_checks %}\n - {{ checks }}\n {% endfor %}\n\n # Restrict the number of checked archives to the last n. Applies only to the "archives" check.\n check_last: {{ borgmatic_check_last }}\n\n # When performing the "archives" check, only consider archive names starting with\n # this prefix. Borg placeholders can be used. See the output of\n # "borg help placeholders" for details. Default is "{hostname}-".\n prefix: '{hostname}-'\n\n# Shell commands or scripts to execute before and after a backup or if an error has occurred.\n# IMPORTANT: All provided commands and scripts are executed with user permissions of borgmatic.\n# Do not forget to set secure permissions on this file as well as on any script listed (chmod 0700) to\n# prevent potential shell injection or privilege escalation.\nhooks:\n # List of one or more shell commands or scripts to execute before creating a backup.\n before_backup:\n{% for cmd in borgmatic_before_backup_command %}\n - {{ cmd }}\n{% endfor %}\n\n # List of one or more shell commands or scripts to execute after creating a backup.\n after_backup:\n{% for cmd in borgmatic_after_backup_command %}\n - {{ cmd }}\n{% endfor %}\n\n # List of one or more shell commands or scripts to execute in case an exception has occurred.\n on_error:\n{% for cmd in borgmatic_failure_command %}\n - {{ cmd }}\n{% endfor %}\n\n{% for hook in borgmatic_hooks %}\n {{ hook }}: {{ borgmatic_hooks[hook] }}\n{% endfor %}\n): object of type 'NoneType' has no len()"}

PLAY RECAP *********************************************************************
MYSERVER : ok=9 changed=5 unreachable=0 failed=1 skipped=2 rescued=0 ignored=0

Python packages renamed in Fedora 32

The python packages have been renamed in Fedora 32 (python3-* -> python-*). The variables for Fedora 32 onwards should be set to:

---
borg_packages:
  - libacl-devel
  - libacl
  - gcc
  - gcc-c++
  - openssl-devel
  - python-pip
  - python-wheel
  - python-devel
  - python-setuptools
  - openssh-clients
  - cronie

python_bin: python
pip_bin: pip

borg_python_packages:
  - cython
  - borgbackup
  - borgmatic

Because ansible-os-family is used to select the variables, RedHat and Fedora are basically treated the same. Maybe this should be changed to ansible-distribution? This would be ansible_distribution == 'Red Hat Enterprise Linux' for RedHat.

I would have time to prepare a PR, but would like to get some input on handling os_family vs. distribution beforehand.

Explicitly declare dependencies

As a follow-up to the issue #101 can you set the versions of the dependencies?
First of all borgbackup and borgmatic.

This is to prevent unexpected changes in cli.
This is a good practice promoted by the 12 factors.

I think you can do this here whit this code :

borg_python_packages:
  - borgbackup==1.2.2
  - borgmatic==1.7.1

Release versions for the role

It's great that the role is now available on ansible galaxy. But it would be even better if there were tagged/released versions so we can identify the need for an upgrade (and then actually perform an upgrade).

This could also probably accompany a simple CHANGELOG file to assist to identifying problematic changes.

adjust to new borgmatic checks

Hey I've been (mis)using the borgmatic_checks variable to set up the new timestamp based checks added in borgmatic 1.6.2. I did this in my vars:

borgmatic_checks:
  - name: repository
    frequency: 1 week
  - name: archives
    frequency: 1 month
borgmatic_large_repo: false

It comes out ugly but totally valid YAML and works fine:

    checks:
        - {'name': 'repository', 'frequency': '1 week'}
        - {'name': 'archives', 'frequency': '1 month'}

I can make a patch to correct the formatting here, but I was wondering if we should do a bit more because IMO we don't need the large repo stuff now that borgmatic can do checks with different frequencies. Am not sure what you think the right approach is, should we just deprecate it and get rid of the differentiated cronjobs/timers, and set the defaults to something like the above, or should we have setting it change the frequency settings to the borgmatic_checks.

Add setting for reduced NICE

I had a few cases where a VPS overloaded during a backup. Could be useful to run it with lower priority by adding this to CRON or setting the nice setting in the Systemd timer.

/usr/bin/nice -n 19 /usr/local/bin/borgmatic ...

Including this role makes it impossible to use --tags to select certain tasks/roles

So this is a strange one.

I have a playbook for managing a server. The playbook includes baseline host setup (ntp, sshd config, etc), backup setup (via this role), and finally workload setup that manages the services running on the host.

When I want to target a specific set of roles/tasks using the --tags option, I receive the following error from this role. Notably this only happens when I've specified a tag that doesn't include the borgbackup role.

$ ansible-playbook services.yml --tags workload-setup

... snip...

TASK [m3nu.ansible_role_borgbackup : Run OS-specific tasks] ****************************************************
Tuesday 24 August 2021  21:54:14 +0200 (0:00:21.284)       0:00:21.284 ******** 
fatal: [redactedhostname]: FAILED! => 
  msg: No file was found when using first_found. Use errors='ignore' to allow this task to be skipped if no files are found

Here is a simple playbook to repro this:

---
- name: backup config
  hosts: example
  become: true
  tags: [backup]
  roles:
    - role: m3nu.ansible_role_borgbackup

- name: workload
 hosts: example
  become: true
  tags: [workload]
  tasks:
    - debug:
        msg: setup app services

Running this with ansible-playbook test.yml will work fine, but running it with ansible-playbook test.yml --tags workload results in the error above.

cron tasks will skipped

the cron tasks will not added in crontab by playbook.

`
TASK [ansible-role-borgbackup : cron] **********************************************************************************************
skipping: [127.0.0.1]

TASK [ansible-role-borgbackup : cron] **********************************************************************************************
skipping: [127.0.0.1]
`

the configuration is the standard one

Don't provide a default password

in defaults/main.yml the borg_encryption_passphrase default to an empty string.
I propose to remove this default var, so users don't configure their backup unencrypted when forgetting to set the pw.
If they forget, ansible should break and remind them to explicitly set it.

Create a new release of the role?

Hello,

We can no longer use your latest release v.0.9.3 safely because:

  • Contrary to what is in your master branch, we can't fix the borgmatic version being used and therefore it installs the latest one (1.7.5 currently) but it is not compatible with your "old" cron tasks (for example borgmatic -c /etc/borgmatic/borgmatic_config--create --prune).

You fixed these issues here #107 and there #102 but if we prefer to use your latest release we are stuck.

Thank you

unusual YAML syntax for hooks

I have a healthchecks hook set up:

borgmatic_hooks:
  healthchecks: https://hc-ping.com/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

The generated YAML file is as follows:

hooks:
    healthchecks:
        https://hc-ping.com/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
        ...

The ... appears to imply that to_nice_yaml has truncated something, but nothing has been truncated. Perhaps it's harmless but it does look odd. I was digging around to try and find why this was happening, and it does seem that to_nice_yaml has some odd side effects of truncating/splitting tokens longer than 80 characters, which seems extremely undesirable. However, this token is less than 80 characters so adding width= didn't change anything. I can't work out why the ... is being added at all.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.