garethgeorge / backrest Goto Github PK

Backrest downloads and installs restic automatically. An https connection is used to be sure that we are connecting to the real github.com but for added security backrest should verify the pgp signature of the downloaded artifact.

Implement protections matching the process used by restic's self-update script: https://github.com/restic/restic/tree/master/internal/selfupdate

Hi there! Thanks for mentioning backrest on HN. I downloaded the version for macOS and first ran backrest via Right Click > Open. This ran the program in a foreground shell, so I realized I should run install.sh. I opened a terminal, ran it, and got the following output:

❯ ./install.sh
Installing on Darwin
Installing backrest to /usr/local/bin
Password:
Creating launchd plist at /Library/LaunchAgents/com.backrest.plist
Enabling launchd plist com.backrest.plist
Unload failed: 5: Input/output error
Try running `launchctl bootout` as root for richer errors.

That being said, it apperas to be running successfully. Anything I should be aware of?

Background

The UI is built against a specific version of restic. This ensures that changes to restic do not break the way the UI handles it's output (each UI release is validated with a single version of restic).

Proposal

Use https://github.com/marketplace/actions/create-pull-request and simple sed based automation to determine the latest version of restic and create a pull request if the hard coded version is out of date. Each time a PR is generated a new minor release should additionally be published that releases Restic UI with the latest minor version of restic.

Add support to check the repo.

The following checks are available :

• Repo Integrity
• Packfile Integrity
  • Full check --read-data
  • Specified subset of repo --read-data-subset=n/t
  • Random subset of repo
    • Random subset by percent --read-data-subset=x%
    • Random subset by size --read-data-subset=nS

The existing plans can be classified into backup plan and check plan. Check plan can be scheduled similar to backup plans.

This could be considered a high priority issue since repo can be corrupted due to updates or existing undiscovered bugs (eg : recent compression bug). If repo corruption is known soon, it can rectified.

For me, thr big problem with Restic is to manage, I mean, for each server it's installed, will need a different management location.
It's difficult to manage a big set of servers, so I would like to know if is there a plan for Backrest to manage all the Restic backups from a single WebUI.
It could be like Cockpit do with Linux Server, but it will be better if there is a dashbord where we can follow the latest backups results from all servers.

Improve UI performance with an on-disk caching layer for commonly fetched data -- particularly snapshots

Refresh policy

• On startup all repos should be checked (e.g. restic.Repo.Init)
• On startup all snapshot storage should be refreshed
• After a backup or prune operation the list of snapshots in the cache should be refreshed.

Caching should be done on a per-repository basis.

On Windows, the path selection is a bit strange as it displays both slash and backslash:

Hi Gareth,

I noticed that when the restic version is updated, the previous version is not deleted.

Here's the content of my data folder:

The API protos (e.g. v1.*) should not be leaked in the interfaces exposed by the restic wrapper package.

The repo view should show the most recent operations on the repository.

The sidebar doesn't update its status when one backup finishes and another starts:

As you can see, based on the sidebar, companies is the currently running backup. But it already finished and the currently active backup is apps, as correctly indicated by the header.

Hi Gareth,

for some reason, one of my backups (I have 2), is now scheduled to run twice at the same time:

My backup configuration in the json file is the following:

    {
      "id": "Backup",
      "repo": "Backup_Repo",
      "paths": [
        "/raid/2Backup"
      ],
      "excludes": [],
      "cron": "0 11 * * *",
      "retention": {
        "maxUnusedLimit": "",
        "keepLastN": 7,
        "keepHourly": 0,
        "keepDaily": 0,
        "keepWeekly": 0,
        "keepMonthly": 0,
        "keepYearly": 0,
        "keepWithinDuration": ""
      },
      "hooks": []
    },

Any idea why it happened?

Hi,

I tried to build a new docker image from the git code. Everything runs well, the image is created but when I create a container from that image it does not run, instead I get the error:
exec /backrest: no such file or directory

I'm using the following commands to build the image and container:

docker build . -t testbackrest:v1 --no-cache

docker container run -it --name=atest -p 9899:9898 -e BACKREST_DATA=/data -e BACKREST_PORT=9898 -e XDG_CACHE_HOME=/cache -v $(pwd)/.backrest/config:/.config/backrest -v $(pwd)/.backrest/data:/data -v $(pwd)/.backrest/cache:/cache  testbackrest:v1

If I run the built backrest executable, it runs well outside the container.

I then modified the Dockerfile to add a shell, and modified the entrypoint to launch the sheet instead of backrest. I looks as follows:

FROM busybox:1.35.0-uclibc as busybox

FROM golang:alpine as gobuild
RUN mkdir /tmp-orig
COPY backrest /backrest

FROM scratch
COPY --from=gobuild /tmp-orig /tmp
COPY --from=gobuild /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/

# Now copy the static shell into base image.
COPY --from=busybox /bin/sh /bin/sh
COPY --from=busybox /bin/mkdir /bin/mkdir
COPY --from=busybox /bin/cat /bin/cat
COPY --from=busybox /bin/ls /bin/ls

ENTRYPOINT ["/bin/sh"]
#ENTRYPOINT ["/backrest"]
COPY backrest /backrest

Building the container from the new image now launches the shell, from where I can see the backrest executable, but launching it still throws the same error.

/ # ls -lsa
total 19124
     4 drwxr-xr-x    1 0        0             4096 Feb  7 18:30 .
     4 drwxr-xr-x    1 0        0             4096 Feb  7 18:30 ..
     4 -rw-------    1 0        0                8 Feb  7 18:30 .ash_history
     4 drwxr-xr-x    3 0        0             4096 Feb  7 18:30 .config
     0 -rwxr-xr-x    1 0        0                0 Feb  7 18:30 .dockerenv
 19088 -rwxr-xr-x    1 0        0         19542680 Feb  7 16:50 backrest
     4 drwxr-xr-x    1 0        0             4096 Feb  7 17:18 bin
     4 drwxr-xr-x    2 0        0             4096 Feb  7 16:53 cache
     4 drwxr-xr-x    2 0        0             4096 Feb  7 16:53 data
     0 drwxr-xr-x    5 0        0              340 Feb  7 18:30 dev
     4 drwxr-xr-x    1 0        0             4096 Feb  7 18:30 etc
     0 dr-xr-xr-x  224 0        0                0 Feb  7 18:30 proc
     0 dr-xr-xr-x   12 0        0                0 Feb  7 18:30 sys
     4 drwxr-xr-x    2 0        0             4096 Feb  7 17:18 tmp
/ # ./backrest
/bin/sh: ./backrest: not found

Am I doing something wrong? Why is backrest not working inside the container, but works outside?

The only problem I had when creating the development environment was running this command:

go install github.com/GeertJohan/go.rice@latest

package github.com/GeertJohan/go.rice is not a main package

Tried to remove the @latest part from the command, but it does not seem to do anything, but there's no error as well.

I get the error package github.com/GeertJohan/go.rice is not a main package

I then installed rice from the package manager:

sudo apt install -y golang-rice

At least I learned how to add the shell command and external cli tools.

Hi,

I've added the hostname to my docker compose file, but at the top of backrest interface, the hostname shows some ID.

I've looked at the container details, and the hostname is correctly showing there:

Then I realized the hostname was being pulled from the config.json file:

This was probably the ID from the 1st time I've created the backrest container...

Is this the expected behavior? For me I would expect the hostname to be dynamic, either by reading the container hostname when opening the user interface, or updating the config file on every time the containter is started.

rclone is ostensibly supported by repo URI's such as rclone:name:path but users hit an error when backrest tries to fork the rclone executable (e.g. it can not be found on the path).

Let's try to figure out why the environment isn't properly being propagated such that the executable isn't found.

Right now, Backrest requires AZURE_ACCOUNT_KEY when configuring a Azure Blob Repo. However, according to the documentation, both AZURE_ACCOUNT_KEY as well as AZURE_ACCOUNT_SAS are interchangeably valid.

Hi! Coming from the forums.

I believe the best way to implement this would be to have a "password command" field next to the password field where one could input the command. Similar to restic's own RESTIC_PASSWORD_COMMAND env. Backrest will then run that command everytime it needs the password.

For the docker container, an env variable could be used.

This way multiple repositories could have different commands/env variables.

The state store will store quick-access information about operations performed by Restic UI. This is not for metadata that can be computed from the live repo (a caching abstraction should be built for that purpose).

Example data for state storage

• Backup operation log
• Health operation log
• Last known config (e.g. for offline edit detection)

Hi,

I would like to propose 3 UI improvements:

1. Add, somewhere in the frontend, a link to the Backrest Git repository
2. Increase the text size of the backup executed additional details (I don't know exactly what to call to this, so check the image below)
   
   I find this text very difficult to read, I often have to use the browser zoom to read it.
3. There is now a logout link on the top right corner (after the host), but it's written in black with a grey background color, so it's not readable
   

Cron monitoring services like healthchecks.io can be pinged using GET or POST requests. They can also store log information upto 100KB.

Reference : https://healthchecks.io/docs/http_api/

Considering adding support for get and post request using hooks.

The difference between this and existing hooks is cron monitoring services can detect if backrest is not running. They need to ping regularly. If they are not pinged then they determine backrest is down.

Current webhooks will not notify if backrest itself is not running.

Adding GET and POST requests will allow many other services to be used.

Scheduling wrt last execution time will prove important for very infrequent backups e.g. monthly or yearly which will not run unless resticui is online at exactly the time the backup is scheduled for.

https://connectrpc.com/

It should be possible to eliminate the global config state in favor of passing around a specific instance.

The cron selector UI is a bit unintuitive for the uninitiated. Improve the schedule selection UI / UX. One improvement might be to include human readable text explaining the currently configured policy e.g. similar to https://crontab.guru/

At start of install.sh i got the following error on a fresh Linux-Debian-Bookworm:
`++ uname -s

• OS=Linux
• '[' Linux = Darwin ']'
• '[' Linux = Linux ']'
• echo 'Installing on Linux'
  Installing on Linux
• install_unix
• echo 'Installing backrest to /usr/local/bin'
  Installing backrest to /usr/local/bin
• sudo mkdir -p /usr/local/bin
  ++ ls -1 backrest
  ++ head -n 1
  ls: Zugriff auf 'backrest' nicht möglich: Datei oder Verzeichnis nicht gefunden
• sudo cp /usr/local/bin
  cp: Fehlender Zieldatei‐Operand hinter '/usr/local/bin'
  „cp --help“ liefert weitere Informationen.
• create_systemd_service
• '[' '!' -d /etc/systemd/system ']'
• echo 'Creating systemd service at /etc/systemd/system/backrest.service'
  Creating systemd service at /etc/systemd/system/backrest.service
• sudo tee /etc/systemd/system/backrest.service
  ++ whoami
  ++ whoami
• echo 'Reloading systemd daemon'
  Reloading systemd daemon
• sudo systemctl daemon-reload
• echo 'Enabling systemd service backrest.service'
  Enabling systemd service backrest.service
• sudo systemctl enable backrest
• sudo systemctl start backrest
  `

On my laptop, I prefer to trigger backups on-demand rather than using a schedule. It would be great if it were possible to create a plan without a schedule - or offer on-demand backups directly from the repo. The only issue I see with the latter is that I do still want to prune after a backup, and retention polices are currently configured on the plan.

By the way, how do retention polices work with multiple plans? Does the restic forget command target the plan tag so it only applies to snapshots created by a given plan? Is there a way to see the actual command that was run (or params passed to a restic library) in the operation list?

install.sh is sensitive to the current working directory.

Add

cd $(dirname "$0")

to the install and uninstall scripts to correct the working directory when invoked from different folders.

Hello,

I have these 2 hooks:

If I delete Hook 0, Hook 1 will become Hook 0 and will loose it's configuration, becoming this:

It is then not possible to save the configuration due to missing configuration:

Removing all Hooks and adding a new one does not work, unless we save the configuration after removing the existing hooks.

It happens in the plans and in the repositories configuration.

Collect and delete old entries in the operations log. An entry can be deleted if it is pushed out of the last 100 operations for any repo or plan.

hello!
FYI - I am trying out backrest on linux, deployed via docker...

When i choose Add Repo to "Add Restic Repository", the first field Repo Name will not accept a dot in the name. As my normal repository nomenclature is myname.repo so Backrest complained: invalid symbol.

Restic allows me to put dots in the name, why not Backrest?

p.s. for now I continued with myname_repo so it is a minor item for me.

Basic idea

• Forget operation's retention policy should be configurable in the plan GUI
• If forget deletes anything append "delete" events for the removed snapshot IDs.

Open questions: how should garbage collection in the case of an offline prune or forget operation be handled?

• Proposal: schedule indexing snapshots as a task, when it runs a set of known snapshot IDs should be loaded and fully diffed with the state of the repo. This is a bit expensive but far from terrible (e.g. order of hundreds of milliseconds even when scaling to tens of thousands of operations in the log).

Right now, we can modify BACKREST_PORT, BACKREST_DATA, BACKREST_RESTIC_COMMAND, etc only through environment variables. Allow modifying them through commandline parameters or config file for easier install/usage.

Currently the only way is to use a script or terminal to modify these settings, atleast in windows.

To manage performance in the case of VERY large append only repositories --latest n flag should be applied to restic snapshots command.

A reasonable limit might be the most recent 2000 snapshots.

backrest.exe
2024-02-06T21:44:57.660+0100    INFO    Installing restic C:\Program Files (x86)/restic/restic-0.16.3.exe to 0.16.3...
2024-02-06T21:44:59.574+0100    INFO    Decompressing zip archive (size=9163895)...
2024-02-06T21:45:01.300+0100    INFO    Installed restic 0.16.3
2024-02-06T21:45:01.301+0100    INFO    Generating new auth secret
2024-02-06T21:45:01.301+0100    FATAL   Error writing secret to file: open C:\Users\asd\AppData\Roaming/backrest/data/jwt-secret: The system cannot find the path specified.

Change
/backrest/data/jwt-secret

to \backrest\data\jwt-secret

First I'd like to thank you for this great project -- very nice idea, can't wait to see further releases! Makes restic so much easier to use and/or setup.

A thing I really miss is the support for pre / post scripts which can run before / after (successful or failed) backups. In my specific case I need to wake up my NAS (via WOL, can be done with an external script) before backrest can start the actual backup.

Would be really great to have such a feature on a per-repo basis. Thanks!

Can a hook test button be implemented?
When we create new hooks, it would be nice to force a test hook trigger to test the configuration.

Hi Gareth,

I've got a warning in a job about at least one source file was not able to be added to the backup. This is the log message I have:

2024-02-11T11:00:00.193Z	INFO	running task	{"task": "backup for plan \"Backup_Job\""}
2024-02-11T11:26:11.911Z	DEBUG	hook output: ---- payload ----{"content":"Backrest Notification
Task: \"backup for plan \"Backup_Job\"\" at 2024-02-11T11:26:10Z
Error: failed to backup: command \"/data/restic-0.16.4 backup --json --exclude-caches -o sftp.args=-oBatchMode=yes /source/To_Backup --tag plan:Backup_Job --parent 17fa10ea92cb8342ff6bc620ff5da678c01ac778b003a68f801b8c649b3d062d\" failed: incomplete backup
Process STDOUT: 
{\"message_type\":\"status\",\"seconds_elapsed\":4,\"percent_done\":0}
{\"message_type\":\"status\",\"seconds_elapsed\":4,\"percent_done\":0}
{\"message_type\":\"status\",\"seconds_elapsed\":4,\"percent_done\":0}
{\"message_type\":\"status\",\"seconds_elapsed\":4,\"percent_done\":0}
{\"message_type\":\"status\",\"seconds_elapsed\":4,\"percent_done\":0}
{\"message_type\":\"status\",\"seconds_elapsed\":4,\"percent_done\":0,\"total_files\":1,\"total_bytes\":288}
{\"message_type\":\"status\",\"seconds_elapsed\":4,\"percent_done\":0,\"total_files\":1,\"total_byt...[17259073 bytes dropped]..._bytes\":46112872238,\"bytes_done\":46112853380,\"error_count\":2}
{\"message_type\":\"summary\",\"files_new\":402,\"files_changed\":818,\"files_unmodified\":134051,\"dirs_new\":6,\"dirs_changed\":807,\"dirs_unmodified\":219415,\"data_blobs\":11842,\"tree_blobs\":579,\"data_added\":6488076064,\"total_files_processed\":135271,\"total_bytes_processed\":46112853380,\"total_duration\":1563.640250311,\"snapshot_id\":\"6fa17b6516676fb8ca86697af0cd2c3de767c66b0fed418068b097dc1b15ba6a\"}
Warning: at least one source file could not be read

"}
2024-02-11T11:26:15.474Z	INFO	task finished	{"task": "collect garbage", "duration": "208.288397ms"}
2024-02-11T11:26:11.911Z	INFO	Backup complete	{"plan": "Backup_Job", "duration": "26m11.121661696s", "summary": {"message_type":"summary","files_new":402,"files_changed":818,"files_unmodified":134051,"dirs_new":6,"dirs_changed":807,"dirs_unmodified":219415,"data_blobs":11842,"tree_blobs":579,"data_added":6488076064,"total_files_processed":135271,"total_bytes_processed":46112853380,"total_duration":1563.640250311,"snapshot_id":"6fa17b6516676fb8ca86697af0cd2c3de767c66b0fed418068b097dc1b15ba6a","percent_done":0,"total_files":0,"files_done":0,"total_bytes":0,"bytes_done":0,"current_files":null}}

How can I know what is the failed file?
Is there any other log available than the one from the docker container?

Hi,

Sorry for the dumb question, but completely new to restic / backrest. HI have an S3 bucket repo setup, and a plan. But how do you configure a client to use backrest to push backups? Sorry if this is a completely stupid question.

I added a hook to one of my backups with the following configuration:

but it seems this hook is generating an error when executed:

This is the log I see in the frontend:

------- script -------
#! sh
echo 'Task: "backup for plan "Series"" at 2024-02-06T08:02:08Z
Event: snapshot end
Repo: Series 
Plan: Series 
Snapshot: 0e734343be9e288cafeaec9de515d473f709fb59babb6de0b30fdd6ead50c066
Overview:
- Data added: 1.001 GB
- Total files processed: 6610
- Total bytes processed: 1.001 TB

Backup Statistics:
- Files new: 4
- Files changed: 6
- Files unmodified: 6600
- Dirs new: 0
- Dirs changed: 9
- Dirs unmodified: 227
- Data blobs: 2026
- Tree blobs: 10
- Total duration: 115.67551034s

'
------- output -------
Error: exec: "sh": executable file not found in $PATH

I'm using the latest version 0.11.0, running on Docker.
I also can't open a shell connection to the docker container, so the shell binary is probably missing.

Recoil for state is unnecessary, React's builtin providers are sufficient for this project.

During my tests, I created 2 plans to backup to the same repository. I then deleted one of the plans, but the repository still shows the old backup.
Is there a way to remove that data without deleting the repository? I see no option for this in the interface...

Investigate good options for generating a documentation site to elaborate in more detailes on the capability of backrest.

Areas to expand in particular are:

• Repo configuration
• Scheduling options and health tasks
• Hooks and hook templates + working examples (hooks are Golang templates, this is worth an article)

Note: windows support is experimental

Right now windows fails to automatically download and install restic from a zip archive.

This can be worked around by manually installing restic to the path prompted at first run.

Hi, I just noticed the plans retention policy is not showing properly in the UI.
I've set the plan to keep the last 7 backups, but the UI is showing none:

Selecting retention by count defaults to 30:

The config.json file shows the correct configuration:

I also tested setting the retention by time, and the same behavior happens, but the config file is correctly saved.