I'm not sure if this is possible, but I wonder whether there should be an option to not compress the backups?
For some people, I suspect this is not an issue, but my backup is going to take 6 hours to compress and is beating up the CPU pretty badly. I suspect the compression ratio won't be huge either as the majority of the files are already compressed anyway. Obviously, the database backups will be compressed massively, but they only make up a tiny proportion of my overall backup.
Thoughts?

Since many people use Cloudflare as their dns service, it would be nice to have this integrated into Raptor. With using the Cloudflare DNS no manual changes have to be done when restoring, because dns changes can be done with the Cloudflare api. Could you please take a look at this? I would like to help, but have no experience with BASH scripting.

I had to manually kill the backup script after it said it was exiting after failing due to a shortage of space.

:~# bash openspace42/raptor/tools/raptor-backup

Starting raptor-backup script on 2018-04-11_12-55-56...

You have 1 local backups occupying 53017 MB of space.

Estimated backup size [148188 MB] LARGER than available disk space [138618 MB].

Now purging old backups while respecting your specified number of minimum backups to keep...

root@shost7:~# The specified number of minimum backups to keep [1] is too high to free enough disk space for the next backup.

Lower this figure by re-running the install and lowering this figure during configuration or make more space available on this machine.

Exiting...

^C

your installer used on debian 8 ends with: Unable to locate package cronic

Whilst the install script stated that it had created the cronjob, crontab -l only lists the default ee letsencrypt cron job

"Enoguh" should be "Enough"

"Estimated backup size [148308 MB] LARGER than available disk space [142705 MB].

Now purging old backups while respecting your specified number of minimum backups to keep...

Enoguh free space [191634 MB] now available for new backup."

Hello,

I tried adding my digitalocean space (a s3 compatible object-storage service) to the raptor backup tool, but I got the following error: S3 connection details do NOT appear to be valid. Could you please help fix this? If necessary, I can provide you the keys for testing.

I get a message:
Now [re-]installing and updating EasyEngine...

Executing apt-get update, please wait...
[ Tue Apr 3 12:49:23 UTC 2018 ] You already have EasyEngine 3.7.5, exit status = 1
Downloading update script Unable to download file, /tmp/eeupdate20180403-124924

And then the installer stops.

When I try to restore from a local backup I get to following error:

/root/openspace42/bash-functions/functions/backup_restore: line 905: restore_type: unbound variable

Perhaps a check early in the script to test whether composer is installed, rather than crashing the script after installing the EE stack? Or install composer as part of the install?

Inititally the script will crash during S3 config due to missing Pwgen. Installing pwgen and restarting the script results in encryption phrase being blank. I assume that although pwgen is missing the script has already created the config file for the passphrase and then leaves it blank.

It would be useful to be able to reconfigure without having to run the whole setup routine again. I know you can edit the files directly, but that is not user friendly and skips any validation. When you rerun setup most of the time is wasted simply watching it redownload etc.

After following the commands like stated in the readme, i get the following error when i want to execute the command: bash raptor/setup

Initiating setup...
/root/raptor/functions: line 181: /root/openspace42/dna/functions/*: No such file or directory

please guide me on this error.

I wiped the installation by removing the openspace directory and then reinstalled using the -be flags.

I did also change the local backup location with the new install. Now backups won't run at all. The process starts, but then it seems to jump directly to testing the restore. I'm not sure if maybe I missed something whilst removing the previous install that is now interfering.

bash openspace42/raptor/tools/raptor-backup               

Starting raptor-backup script on 2018-04-15_10-13-26...

You have 0 local backups occupying 1 MB of space.

Now testing restore from S3 to /var/backups/raptor-backups/restores/s3/ to ensure existing backups integrity and passphrase match...

This is going to take some time. Please let the process run, it won't print any output until its done.

Testing restore from S3 failed.

This is likely due to an incorrect passphrase [check | /var/backups/raptor-backups/temp/duplicity-log | for messages like: | gpg: decryption failed: bad key |].

If this is the case, you've likely changed your backups encryption passphrase on this machine between your last backup and this one.

Aborting backup as proceeding would cause your S3 backups to be encrypted with two or more encryption keys at the same time.

This would render them unusable in the future should you ever need them.

Edit the file | /root/openspace42/raptor/conf/backups/s3/backups_passphrase | and set once again your previous encryption passphrase, then re-run this script.

If you have lost your previous encryption passphrase, delete all your S3 backups and re-run this script to make an entirely new backup set encrypted solely with the currently specified passphrase.

Exiting...

When I try to install raptor using the install instructions my installation terminates and gives the following error:

Now [re-]installing and updating EasyEngine...
Executing apt-get update, please wait...
[ Sat Apr 7 12:56:25 CEST 2018 ] You already have EasyEngine 3.7.5, exit status = 1
Downloading update script Unable to download file, /tmp/eeupdate20180407-125625

Could you please help me fix this?

root@cloud:~# bash raptor/setup
Initiating setup...

/root/raptor/functions: line 184: /root/openspace42/dna/functions/*: No such file or directory

Using Ubuntu 18.04

I can't see how I would set a different local storage location.
Several of the sites to backup are very large, meaning the backup size is much larger than the available disk space.

Ideally, I wouldn't store any backups locally, but I assume the script requires local storage to create the backups before pushing off to S3.

I would, therefore, like to set an alternative local storage location so that I can complete backups. Whilst I imagine I could do it by hacking the script, I think it would make sense to be able to set this during setup?

Once the installer had configured the cron job is going on to modify the ssh config. The installer crashed after trying to restart ssh:

Apr 04 09:50:19 shost7 systemd[1]: Starting OpenBSD Secure Shell server...
Apr 04 09:50:19 shost7 sshd[22013]: /etc/ssh/sshd_config line 88: garbage at end of line; "Match".
Apr 04 09:50:19 shost7 systemd[1]: ssh.service: Main process exited, code=exited, status=255/n/a
Apr 04 09:50:19 shost7 systemd[1]: Failed to start OpenBSD Secure Shell server.
Apr 04 09:50:19 shost7 systemd[1]: ssh.service: Unit entered failed state.
Apr 04 09:50:19 shost7 systemd[1]: ssh.service: Failed with result 'exit-code'.

Adding a line break on line 88 fixed the problem.

Original:
UsePAM yes Match Group sftp_access
ChrootDirectory /home/%u
ForceCommand internal-sftp
PasswordAuthentication yes
PermitTunnel no
AllowAgentForwarding no
AllowTcpForwarding no
X11Forwarding no

After adding line break:
UsePAM yes
Match Group sftp_access
ChrootDirectory /home/%u
ForceCommand internal-sftp
PasswordAuthentication yes
PermitTunnel no
AllowAgentForwarding no
AllowTcpForwarding no
X11Forwarding no

After it finally finished compressing my backup, the next section failed.
I wonder if it may be to do with the double slashes? Also, "/var/backups//temp/duplicity-log" didn't exist, with or without the double slash.
I did not change my encryption password.

"
Finished compressing backup and stored in | /var/backups//archive/ |

Finished local backup.

Now testing restore from S3 to /var/backups//restores/s3/ to ensure existing backups integrity and passphrase match...

This is going to take some time. Please let the process run, it won't print any output until its done.

Testing restore from S3 failed.

This is likely due to an incorrect passphrase [check | /var/backups//temp/duplicity-log | for messages like: | gpg: decryption failed: bad key |].

If this is the case, you've likely changed your backups encryption passphrase on this machine between your last backup and this one.
"

First attempt to install the software, I get the error "/root/openspace42/bash-functions/functions/install_dependencies: line 35: packages: unbound variable"

This happened the first time I tried to install (following the issue with EasyEngine failing to update being resolved), but I had answered No to the question to install the full stack of EE. I assumed that the error was due to something being required from the EE stack, so I retried and answered Y to install the stack and everything was fine. Therefore I assumed I must answer Y to avoid the error in future.
However, on this fresh install, I answered Y to install the stack and still got this error. SImply restarting the installation seems to fix the problem and the install continues.

Slightly larger dump of the script output.
Now installing dependencies...

Hit:1 http://ppa.launchpad.net/chris-lea/redis-server/ubuntu xenial InRelease
Hit:2 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:3 http://ppa.launchpad.net/ondrej/php/ubuntu xenial InRelease
Ign:7 http://apt.newrelic.com/debian newrelic InRelease
Hit:8 http://apt.newrelic.com/debian newrelic Release
Ign:9 http://download.opensuse.org/repositories/home:/rtCamp:/EasyEngine/xUbuntu_16.04 InRelease
Hit:10 http://download.opensuse.org/repositories/home:/rtCamp:/EasyEngine/xUbuntu_16.04 Release
Hit:4 http://lon1.mirrors.digitalocean.com/ubuntu xenial InRelease
Hit:5 http://lon1.mirrors.digitalocean.com/ubuntu xenial-updates InRelease
Hit:6 http://lon1.mirrors.digitalocean.com/ubuntu xenial-backports InRelease
Hit:11 http://sfo1.mirrors.digitalocean.com/mariadb/repo/10.1/ubuntu xenial InRelease
Hit:12 https://repos.sonar.digitalocean.com/apt main InRelease
Hit:14 https://packages.amplify.nginx.com/ubuntu xenial InRelease
Reading package lists... Done
/root/openspace42/bash-functions/functions/install_dependencies: line 35: packages: unbound variable

I have this error on Debian 8, when all is almost finished.

Setting up fuse (2.9.3-15+deb8u2) ...
Creating fuse device...
/run/udev or .udevdb or .udev presence implies active udev. Aborting MAKEDEV invocation.
chmod: cannot access '/dev/fuse': No such file or directory
dpkg: error processing package fuse (--configure):
subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of bindfs:
bindfs depends on fuse; however:
Package fuse is not configured yet.

dpkg: error processing package bindfs (--configure):
dependency problems - leaving unconfigured
Processing triggers for libc-bin (2.19-18+deb8u10) ...
Errors were encountered while processing:
fuse
bindfs
E: Sub-process /usr/bin/dpkg returned an error code (1)

An error occured performing APT operations.

Exiting...

By running command bash raptor/setup get this:
/root/raptor/functions: line 33: dna-define_vars: command not fount

Cronic detected failure or error output for the command:
bash /root/openspace42/raptor/tools/raptor-backup

RESULT CODE: 0

ERROR OUTPUT:
/root/openspace42/dna/functions/backup_restore: line 115: [: Avail: integer expression expected

If I have a subdomain listed on exclusion list than script is skipping main domain too. And saying that this site is on exclusion list

I'm not sure how accurate the estimated space required is. Perhaps there is a requirement for a large amount of space whilst completing the backups, but this is going to cause issues for me potentially.

Looking at the messages (pasted below) my last backup took up 53GB, but the estimate for a backup is 148GB! I assume that it is calculated by looking at /var/www and then using a 3x multiple. This is likely a little high? The way I see it, we take a copy of that location, and we then create an archive of the location, so worst case it is going to 2x /var/www + whatever the DBs take up. Perhaps for smaller sites 3x /var/www works, but for larger sites, it doesn't make sense.

"You have 1 local backups occupying 53017 MB of space.

Estimated backup size [148188 MB] LARGER than available disk space [138618 MB]."

Enter hostname [fqdn]:hula.host
Adding repository for MySQL, please wait...
Adding repository for NGINX, please wait...
Adding repository for PHP, please wait...
Updating apt-cache, please wait...
Installing packages, please wait...
Oops Something went wrong!!
Check logs for reason `tail /var/log/ee/ee.log` & Try Again!!!

/root/openspace42/dna/functions/install_easyengine: line 38: cd: /var/www/22222/htdocs/db/pma: No such file or directory
root@ubuntu-s-2vcpu-2gb-sfo2-01:~#

Distributor ID: Ubuntu
Description: Ubuntu 18.04.1 LTS
Release: 18.04
Codename: bionic

Fails on a brand new exclusive VPS and initial install with 18.04 LTS here Nick.