Giter Club home page Giter Club logo

enacrestic's Introduction

ENACrestic

License: MIT release to PyPI Python Version

Last commit GitHub commit activity Github Stars Github Forks Github Watchers GitHub contributors

ENACrestic

A simple Qt GUI to automate backups with restic

  1. Automate your restic backups at a choosen frequency
  2. Run restic forget in a regular basis (and transparently) to keep your backup light and useful
  3. Let you see when:
  • pre_backup_in_progress pre_backup script is running
  • backup_in_progress restic backup is running
  • forget_in_progress restic forget is running
  • unlock_in_progress restic unlock is running
  • backup_success backup is completed
  • error last operation failed
  • no_network last backup failed because of a network timeout (maybe the VPN is not running?)

Installation

This has been tested and validated on

  • Ubuntu 18.04 LTS
  • Ubuntu 20.04 LTS
  • Ubuntu 22.04 LTS
sudo apt install restic python3-pip qt5dxcb-plugin python3-pyqt5
pip3 install --user --upgrade pip
pip3 install --user enacrestic

Upgrade

To upgrade ENACrestic to latest release, just run the following command:

pip3 install --user --upgrade enacrestic

Config ENACrestic

Note: For this documentation, we have chosen to use the vi text editor. Adapt the commands below by replacing it with the editor of your choice. (nano, gedit, ...)

mkdir ~/.enacrestic

Write environment setup file

Choose the right section according to your destination storage

vi ~/.enacrestic/env.sh
# 1. recommended destination: S3 Bucket
export RESTIC_REPOSITORY=s3:s3.epfl.ch/bucket_name/restic_MyComputerName
export AWS_ACCESS_KEY_ID=TheBucketRWAccessKey
export AWS_SECRET_ACCESS_KEY=TheBucketRWSecretKey

# 2. alternative destination: SSH / SFTP
export RESTIC_REPOSITORY=sftp:my-server.epfl.ch:/home/username/path

Note, although Restic is able to manage several computers being backed up on a same respository, it's not recommended with ENACrestic. Keep a dedicated RESTIC_REPOSITORY per machine.

Write password file (mandatory)

Add a one line password in it. This is used to encrypt your backups.

vi ~/.enacrestic/.pw

Be careful ! If you loose this password ... you loose your backups.

Define what to backup (mandatory)

Add one line per folder / file that has to be backed up.

vi ~/.enacrestic/bkp_include
# 1. recommended scenario: backup all your home directory
/home/username/

# 2. alternative scenario: backup only choosen folders
/home/username/.enacrestic/
/home/username/Documents/
/home/username/Teaching/
/home/username/Pictures/
/home/username/Projects/
/home/username/Learn/
/home/username/.gitconfig
/home/username/.mozilla/
/home/username/.ssh/
# heavy !
/home/username/Videos/

note: Lines starting with a # are ignored.

Define what to exclude from the backup (optional but recommended)

Add one line per folder / file / expression that has to be excluded.

Before running your first backup, you might want to exclude heavy and unnecessary folders (Like the Downloads or the Trash). You can use the baobab utility to find those.

Here is an example of some typical things you might want to exclude from backup:

vi ~/.enacrestic/bkp_exclude
*.iso
*.sav
*.bak
*.bak2
*.log
*.ost
*.part
*.temp
*.tmp
*.vhd
*.vhdx
*.vmdk
*.vdi
/home/username/Downloads/
/home/username/ENACdrives/
/home/username/.local/share/Trash/
/home/username/VirtualBox VMs/
/home/username/snap/
/home/username/.cache/
/home/username/**/nobackup*
/home/username/.local/share/virtualenvs/
/home/username/.arduino15/
/home/username/.atom/
/home/username/.npm/
/home/username/.nvm/

Exact syntax is described here

Make it available to your shell (mandatory)

Add the following 2 lines to have:

  • enacrestic in your $PATH
  • enacrestic's env variables available.
vi ~/.bashrc # or ~/.zshrc or whatever is your shell rc file
export "PATH=$PATH:$HOME/.local/bin"
. $HOME/.enacrestic/env.sh

Now close + open a new terminal to get it all into your environment ... or simply reload your rc file:

. ~/.bashrc # or ~/.zshrc or whatever is your shell rc file

All done !

๐ŸŽ‰ Setup is now complete! You're now ready to send your 1st backup. ๐ŸŽ‰

Run ENACrestic on your computer

  • from Ubuntu's Application launcher
  • or from command line with the single command enacrestic

You'll see a new icon in the system tray (upper-right corner of your screen) with following icon.

just_launched

This is the indicator that ENACrestic is running in the background and it'll change over time, reflecting current state.

By clicking on it, you can view detailed status and opt-in for the auto-start feature (start ENACrestic when Ubuntu user session is started).

From now on, ENACrestic is running in the background and doing the backups on a regular basis.

You can check it's activity by reading the ~/.enacrestic/last_backups.log file.

Note: First backup can take a long time! Please consider having enough time for the 1st backup to complete. It'll be the longest backup ever, since everything has to be copied. All future backups will then be only incremental.

Run ENACrestic on a server

Add a dedicated Systemd service file:

vi /etc/systemd/system/enacrestic.service
[Unit]
Description=ENACrestic

[Install]
WantedBy=multi-user.target

[Service]
Type=simple
User=root
Group=root
WorkingDirectory=/root
ExecStart=/root/.local/bin/enacrestic --no-gui
KillSignal=SIGTERM
Restart=on-failure
RestartSec=30

Enable and start it:

systemctl daemon-reload
systemctl enable enacrestic.service
systemctl start enacrestic.service

That will ensure enacrestic service is started when the server boot.

You can know its status with the following commands:

systemctl status enacrestic.service
tail -n 50 -f /root/.enacrestic/last_backups.log

Note on old backups retention policy

By default, every 10 backups, a restic forget will clean repository from backups that don't need to be kept, according the following retention policy:

  • keep the last 3 backups
  • keep the last 24 hourly backups
  • keep the last 7 daily backups
  • keep the last 4 weekly backups
  • keep the last 12 monthly backups
  • keep the last 5 yearly backups

What ENACrestic doesn't do

ENACrestic is here to help you, running backups on a regular basis. If you want to browse backups, restore files/folders, you'll have to use restic itself. Here are basic commands:

List the snapshots (backups)

restic snapshots -c --password-file  ~/.enacrestic/.pw

Mount the backups ...

... and be able to

  • browse the different snapshots
  • restore any file / folder
mkdir -p ~/mnt/my_backups
restic mount ~/mnt/my_backups --password-file  ~/.enacrestic/.pw

Now you can browse ~/mnt/my_backups folder and copy from it anything you want to restore. When done, you can simply Ctrl-c in the terminal where you had issued the restic mount ... command.

enacrestic's People

Contributors

github-actions[bot] avatar sbancal avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar

enacrestic's Issues

[bug] PIDFile on /run/user/<uid>/enacrestic.pid doesn't work on server

/run/user/0/enacrestic.pid path is created on user login.
Because no user logins on a server, the backup won't start.

We need a fix for server scenario. 3 possibilities:

  • choose an other location, which doesn't depend on user session, but that the current user has a write access
  • make it different for user root than the other users and expect that servers are backed up by root user
  • put it in ~/.enacrestic/enacrestic.pid

feat - Auto log rotation

Log is going to accumulate over years ... โ–ถ๏ธ too heavy !

Need to implement an automatic log rotation.

No network detection

All following strings from stderr should result to Status.NO_NETWORK:

dial tcp 128.178.32.7:443: i/o timeout

dial tcp: lookup s3.epfl.ch: Temporary failure in name resolution

feat - Add easy way to set interval between automatic backups

As requested in #1
Give the end-user an easy way to adjust the interval between automatic backups.

As said, this can be set by editing ~/.enacrestic/state.json file while ENACrestic is off and changing the backup_every_n_minutes value.

But it could be even an easy select from a drop-down values : 30min, 1h, 2h for instance.

feat - Wizzard for 1st time setup

When launching ENACrestic for the 1st time, would be great that everything could be configured with default values + wizzard for S3 setup.

Steps that would/could be automated:

  • mkdir ~/.enacrestic
  • vi ~/.enacrestic/env.sh
  • vi ~/.enacrestic/.pw
    note: not sure this is safe to be automated ... user has to write it down somewhere else to be sure to be able to recover his files if he loses his laptop!
  • vi ~/.enacrestic/bkp_include
  • vi ~/.enacrestic/bkp_exclude
  • vi ~/.bashrc ~/.zshrc
  • restic init --password-file ~/.enacrestic/.pw #96

fix release-please setup

release-please config is missing to update into src/ python scripts.

  • src/enacrestic/__init__.py
  • pyproject.toml
  • .release-please-manifest.json line 3

restic forget fails when repo is locked

Sometime we get the following error :

Running restic forget!
! unable to create lock in backend: repository is already locked by PID 300029 on dell-2020 by sbancal (UID 1000, GID 1000)
! lock was created at 2021-08-26 11:16:14 (22h47m52.477653081s ago)
! storage ID 732f566b
! the unlock command can be used to remove stale locks
!

That means that previous restic forget didn't complete and left the repo locked. This is unexpected for the end-user ... Strategy is to be found !

refactor State, Conf, App, ... to be more coherent

Things are note done at the right place. This is annoying for other issues to be fixed -> refactor !

  • State need to be split into State + Conf
  • current_state to be split into Operation + Status
  • we need a queue for the commands to run

Last successful backup within 24h

In case of an error in last backup, don't display a red icon if the last successful backup is within a certain period (let's say 24h).

Maybe a gray version of the backup_success would be great.

Note: we need to think about recurring failed cleanup !

bug: AttributeError: 'ResticBackup' object has no attribute 'state'

Testing ENACrestic raise the following error

Nov 30 17:17:56 enacvm0086 poetry[5002]: Traceback (most recent call last):
Nov 30 17:17:56 enacvm0086 poetry[5002]: File "/root/ENACrestic/src/enacrestic/restic_backup.py", line 179, in _handle_stderr
Nov 30 17:17:56 enacvm0086 poetry[5002]: if self.state.current_operation in (
Nov 30 17:17:56 enacvm0086 poetry[5002]: AttributeError: 'ResticBackup' object has no attribute 'state'

bug: timeout and stale locks aren't recognized by ENACrestic

ENACrestic doesn't recognize repo locked or network timeout anymore:


2022-12-02 11:21:49.377499
Running restic backup!
! unable to create lock in backend: repository is already locked exclusively by PID 273246 on
dell2020 by sbancal (UID 1000, GID 1000)
! lock was created at 2022-12-01 11:30:48 (23h51m1.929575232s ago)
! storage ID ace9387e
! the unlock command can be used to remove stale locks
!
Process finished (1) in 0.61 seconds with status: 'last_operation_failed'


2022-12-04 21:12:35.214405
Running restic backup!
! Fatal: unable to open config file: Stat: Get "https://s3.epfl.ch/something/?location=": dial tcp 128.178.32.7:443: i/o timeout
! Is there a repository at the following location?
! s3:s3.epfl.ch/10208-7796cdead8c3d3f2115c040c7a7b1589/restic_dell-2020
!
Process finished (1) in 90.08 seconds with status: 'last_operation_failed'

Both of them finish in a 'last_operation_failed' which is not OK !

Save protected files

I would like to backup important configuration files present in the folder /etc, but restic complains that

! error: Open: open /etc/filename: permission denied

Any ways of achieving this with ENACrestic?

feat - Add pause feature

First of all, that's really helpful to have such an automated backup tool.
Installation is straightforward for someone who knows how to use linux.

Now some points of possible improvements:

  1. Include the exclude_files list or document it if it is already included
  2. Add some more functionality to the button:
    • Frequency of backups
    • Pause restic (pause backups when not on EPFL network)

Exclude folders

Thanks for the great work! Seems to work pretty seamlessly! One question: is it possible to exclude some folders by editing a file like ~/.enacrestic/bkp_exclude for instance? As an example, one may want to include /home/user but exclude /home/user/.local/Trash. Is this possible?

Last forget date is wrong !

Apparently last forget was on "2022-05-20 14:17:02". This is confirmed by state.json.
But log says :


2022-09-05 14:40:38.078373
Running restic forget!
Applying Policy: keep 3 latest, 24 hourly, 7 daily, 4 weekly, 12 monthly, 5 yearly snapshots
[...]
done

Process finished in 35.106 seconds.

What is going wrong ?

Does it impact last backup also in state.json ?

restic backup fails because of a lock

Running restic backup!

! unable to create lock in backend: repository is already locked exclusively by PID 40541 on dell-2020 by sbancal (UID 1000, GID 1000)
! lock was created at 2022-02-01 15:02:55 (18h7m38.888008415s ago)
! storage ID 2c0cfe22
! the unlock command can be used to remove stale locks

potential solution :

  • check if a process is still running
  • unlock with :
restic unlock --password-file ~/.enacrestic/.pw

[BUG] segmentation fault on server with no GUI

Launching enacrestic on a server with no GUI displays this error:

enacrestic
--------------------------------------------------
2022-11-07 07:56:25.784803
Started ENACrestic 0.1.8

--------------------------------------------------
2022-11-07 07:56:25.785084
Checking for latest release
ok
qt.qpa.xcb: could not connect to display
Segmentation fault (core dumped)

Tested on Ubuntu srv 22.04

Documentation should be adapted for servers with no GUI?

enacrestic.pid lock is still here after reboot

On a VM, enacrestic.pid lock is still here after reboot. This prevents enacrestic to start !

We can see it in /root/.enacrestic/last_backups.log:

--------------------------------------------------
2023-03-24 10:18:23.798925
Started ENACrestic (noGUI) 0.7.1

Already running -> quit

Temporary errors less visible

It might happen that a file disappear during the backup. This gives:

! error: lstat /home/username/.thunderbird/bla.default-release/global-messages-db.sqlite-journal: no such file or directory
!

Files: 82 new, 117 changed, 1345252 unmodified
Dirs: 20 new, 135 changed, 181152 unmodified
Added to the repo: 50.099 MiB

processed 1345451 files, 75.801 GiB in 1:16

snapshot 12345678 saved

! Warning: at least one source file could not be read
!
Process finished (3) in 76.79 seconds with status: 'last_operation_failed'

Would it be possible to have something less alarming than the red cross icon for such scenario !?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.