Giter Club home page Giter Club logo

mongodb-gcs-backup's Introduction

mongodb-gcs-backup

This project aims to provide a simple way to perform a MongoDB server/db backup using mongo-tools and to upload it to Google Cloud Storage. It was greatly inspired from takemetour/docker-mongodb-gcs-backup.

We provide a kubernetes support thanks to the helm chart located in the chart folder of this repository.

Docker image

You can pull the public image from Docker Hub:

docker pull zoov/mongodb-gcs-backup:latest

Configuration

The following table lists the configurable parameters you can set up.

Environment Variable Required Default Description
BACKUP_DIR No /tmp The path where the mongodump result will be temporarily stored.
BOTO_CONFIG_PATH No /root/.boto The path where gsutil will search for the boto configuration file.
GCS_BUCKET Yes The bucket you want to upload the backup archive to.
GCS_KEY_FILE_PATH Yes The location where the GCS serviceaccount key file will be mounted.
MONGODB_HOST No localhost The MongoDB server host.
MONGODB_PORT No 27017 The MongoDB port.
MONGODB_DB No The database to backup. By default, a backup of all the databases will be performed.
MONGODB_USER No The MongoDB user if any.
MONGODB_PASSWORD No The MongoDB password if any.
MONGODB_OPLOG No true if you want to perform a mongodump with the --oplog flag, false otherwise.
SLACK_ALERTS No true if you want to send Slack alerts in case of failure.
SLACK_WEBHOOK_URL No The Incoming WebHook URL to use to send the alerts.
SLACK_CHANNEL No The channel to send Slack messages to.
SLACK_USERNAME No The user to send Slack messages as.
SLACK_ICON No The Slack icon to associate to the user/message.

You can set all of these variables within your values.yaml file under the env dict key.

Usage

Run locally

You can run the script locally:

cd /path/to/mongodb-gcs-backup
chmod +x backup.sh
GCS_BUCKET=<gs://bucket_name> \
./backup.sh

Please note that you can set any environment variable described in the previous section! As an example, to enable the Slack alerts on failure:

SLACK_ALERTS=true \
SLACK_WEBHOOK_URL=<webhook_url> \
SLACK_CHANNEL=<slack_channel> \
SLACK_USERNAME=<slack_username> \
SLACK_ICON=<slack_icon> \
GCS_BUCKET=<gs://bucket_name> \
./backup.sh

Run within Kubernetes

Installing the Chart

To install the chart with the release name my-release within you Kubernetes cluster:

$ helm install --name my-release chart/mongodb-gcs-backup

The command deploys the chart on the Kubernetes cluster in the default namespace. The configuration section lists the parameters that can be configured during installation.

Uninstalling the Chart

To uninstall/delete the my-release deployment:

$ helm delete my-release

The command removes all the Kubernetes components associated with the chart and deletes the release.

Authenticate with GCS

Using the gcloud CLI

If you are running the script locally, the easiest solution is to sign in to the google account associated with your Google Cloud Storage data:

gcloud init --console-only

More information on how to setup gsutil locally here.

Using a service account within Kubernetes

You can create a service account with the required roles to write to GCS attached.

To use the resulting JSON key file within Kubernetes you can create a secret from it by running the following command:

  kubectl create secret generic mongodb-gcs-backup \
  --from-file=credentials.json=/path/to/your/key.json

Then you will need to specify this secret name via the --set secretName=<your_secret_name> argument to the helm install command or by specifying it directly in your values.yaml file (by default, the secret name is set to mongodb-gcs-backup). The key file will be mounted by default under /secrets/gcp/credentials.json and the GCS_KEY_FILE_PATH variable should point to it.

mongodb-gcs-backup's People

Contributors

cmorent avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

mongodb-gcs-backup's Issues

Misc tweaks

I started using this a while ago at work, and wondered if you would be interested in any of this commit. I haven't done a PR, because I made other local changes at the time and there is a caveat about logging.

  • I added a MONGODB_QUIET option, on seeing how much log output the progress reporting generates when running under GKE, but then didn't actually use it.
  • Changing the .tar.gz extension of the output archives, as it's not a tarball, it's actually a binary format specific to mongodump.
    • In fact even the version of the mongo tools affects it- eg dumps taken using the mongo 4.2 tools inside the 0.2.0 docker image, in my testing were not understood by a 4.0 mongorestore even though the backup was taken from a 4.0 server.
  • Keep the log output as it goes (using tee), and in the event of an error, emit the last line of that on Slack. I had noticed that the errors outputted to Slack otherwise amounted to "something went wrong", but gave me little idea what went wrong.
    • The caveat of what I did here, is that the 2>&1 where I pipe the output to tee, also means the logs aren't differentiated between stdout/stderr when output to the console, and therefore aren't differentiated between info/error in my GKE logs.
  • I needed to put quotes on something in the helm yaml to get it to deploy.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.