Giter Club home page Giter Club logo

mgob's Introduction

mgob

This project is no longer maintained. I lost interest in MongoDB some years ago and I can't drive an OSS project forward in my spare time without being passioned about it.

But MGOB lives on thanks to @maxisam. The new home of MGOB is https://github.com/maxisam/mgob


Build Status Docker Pulls

MGOB is a MongoDB backup automation tool built with Go.

Features

  • schedule backups
  • local backups retention
  • upload to S3 Object Storage (Minio, AWS, Google Cloud, Azure)
  • upload to gcloud storage
  • upload to SFTP
  • upload to any Rclone supported storage
  • notifications (Email, Slack)
  • instrumentation with Prometheus
  • http file server for local backups and logs
  • distributed as an Alpine Docker image

Install

MGOB is available on Docker Hub at stefanprodan/mgob.

Supported tags:

  • stefanprodan/mgob:latest latest stable release
  • stefanprodan/mgob:edge master branch latest successful build

Compatibility matrix:

MGOB MongoDB
stefanprodan/mgob:0.9 3.4
stefanprodan/mgob:0.10 3.6
stefanprodan/mgob:1.0 4.0
stefanprodan/mgob:1.1 4.2

Docker:

docker run -dp 8090:8090 --name mgob \
    -v "/mgob/config:/config" \
    -v "/mgob/storage:/storage" \
    -v "/mgob/tmp:/tmp" \
    -v "/mgob/data:/data" \
    stefanprodan/mgob \
    -LogLevel=info

Kubernetes:

A step by step guide on running MGOB as a StatefulSet with PersistentVolumeClaims can be found here.

Configure

Define a backup plan (yaml format) for each database you want to backup inside the config dir. The yaml file name is being used as the backup plan ID, no white spaces or special characters are allowed.

Backup plan

scheduler:
  # run every day at 6:00 and 18:00 UTC
  cron: "0 6,18 */1 * *"
  # number of backups to keep locally
  retention: 14
  # backup operation timeout in minutes
  timeout: 60
target:
  # mongod IP or host name
  host: "172.18.7.21"
  # mongodb port
  port: 27017
  # mongodb database name, leave blank to backup all databases
  database: "test"
  # leave blank if auth is not enabled
  username: "admin"
  password: "secret"
  # add custom params to mongodump (eg. Auth or SSL support), leave blank if not needed
  params: "--ssl --authenticationDatabase admin"
# Encryption (optional)
encryption:
  # At the time being, only gpg asymmetric encryption is supported
  # Public key file or at least one recipient is mandatory
  gpg:
    # optional path to a public key file, only the first key is used.
    keyFile: /secret/mgob-key/key.pub
    # optional key server, defaults to hkps://keys.openpgp.org
    keyServer: hkps://keys.openpgp.org
    # optional list of recipients, they will be looked up on key server
    recipients:
      - [email protected]
# S3 upload (optional)
s3:
  url: "https://play.minio.io:9000"
  bucket: "backup"
  # accessKey and secretKey are optional for AWS, if your Docker image has awscli
  accessKey: "Q3AM3UQ867SPQQA43P2F"
  secretKey: "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG"
  # Optional, only used for AWS (when awscli is present)
  # The customer-managed AWS Key Management  Service (KMS) key ID that should be used to
  # server-side encrypt the backup in S3
  #kmsKeyId:
  # Optional, only used for AWS (when awscli is present)
  # Valid choices are: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA  |  ONE-
  #     ZONE_IA  |  INTELLIGENT_TIERING  |  GLACIER | DEEP_ARCHIVE.
  # Defaults to 'STANDARD'
  #storageClass: STANDARD
  # For Minio and AWS use S3v4 for GCP use S3v2
  api: "S3v4"
# GCloud upload (optional)
gcloud:
  bucket: "backup"
  keyFilePath: /path/to/service-account.json
# Azure blob storage upload (optional)
azure:
  containerName: "backup"
  connectionString: "DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net"
# Rclone upload (optional)
rclone:
  bucket: "my-backup-bucket"
  # See https://rclone.org/docs/ for details on how to configure rclone
  configFilePath: /etc/rclone.conf
  configSection: "myrclonesection"
# SFTP upload (optional)
sftp:
  host: sftp.company.com
  port: 2022
  username: user
  password: secret
  # you can also specify path to a private key and a passphrase
  private_key: /etc/ssh/ssh_host_rsa_key
  passphrase: secretpassphrase
  # dir must exist on the SFTP server
  dir: backup
# Email notifications (optional)
smtp:
  server: smtp.company.com
  port: 465
  username: user
  password: secret
  from: [email protected]
  to:
    - [email protected]
    - [email protected]
# Slack notifications (optional)
slack:
  url: https://hooks.slack.com/services/xxxx/xxx/xx
  channel: devops-alerts
  username: mgob
  # 'true' to notify only on failures
  warnOnly: false

ReplicaSet example:

target:
  host: "mongo-0.mongo.db,mongo-1.mongo.db,mongo-2.mongo.db"
  port: 27017
  database: "test"

Sharded cluster with authentication and SSL example:

target:
  host: "mongos-0.db,mongos-1.db"
  port: 27017
  database: "test"
  username: "admin"
  password: "secret"
  params: "--ssl --authenticationDatabase admin"

Web API

  • mgob-host:8090/storage file server
  • mgob-host:8090/status backup jobs status
  • mgob-host:8090/metrics Prometheus endpoint
  • mgob-host:8090/version mgob version and runtime info
  • mgob-host:8090/debug pprof endpoint

On demand backup:

  • HTTP POST mgob-host:8090/backup/:planID
curl -X POST http://mgob-host:8090/backup/mongo-debug
{
  "plan": "mongo-debug",
  "file": "mongo-debug-1494256295.gz",
  "duration": "3.635186255s",
  "size": "455 kB",
  "timestamp": "2017-05-08T15:11:35.940141701Z"
}

Scheduler status:

  • HTTP GET mgob-host:8090/status
  • HTTP GET mgob-host:8090/status/:planID
curl -X GET http://mgob-host:8090/status/mongo-debug
{
  "plan": "mongo-debug",
  "next_run": "2017-05-13T14:32:00+03:00",
  "last_run": "2017-05-13T11:31:00.000622589Z",
  "last_run_status": "200",
  "last_run_log": "Backup finished in 2.339055539s archive mongo-debug-1494675060.gz size 527 kB"
}

Logs

View scheduler logs with docker logs mgob:

time="2017-05-05T16:50:55+03:00" level=info msg="Next run at 2017-05-05 16:51:00 +0300 EEST" plan=mongo-dev
time="2017-05-05T16:50:55+03:00" level=info msg="Next run at 2017-05-05 16:52:00 +0300 EEST" plan=mongo-test
time="2017-05-05T16:51:00+03:00" level=info msg="Backup started" plan=mongo-dev
time="2017-05-05T16:51:02+03:00" level=info msg="Backup finished in 2.359901432s archive size 448 kB" plan=mongo-dev
time="2017-05-05T16:52:00+03:00" level=info msg="Backup started" plan=mongo-test
time="2017-05-05T16:52:02+03:00" level=info msg="S3 upload finished `/storage/mongo-test/mongo-test-1493992320.gz` -> `bktest/mongo-test-1493992320.gz` Total: 1.17 KB, Transferred: 1.17 KB, Speed: 2.96 KB/s " plan=mongo-test
time="2017-05-05T16:52:02+03:00" level=info msg="Backup finished in 2.855078717s archive size 1.2 kB" plan=mongo-test

The success/fail logs will be sent via SMTP and/or Slack if notifications are enabled.

The mongodump log is stored along with the backup data (gzip archive) in the storage dir:

aleph-mbp:test aleph$ ls -lh storage/mongo-dev
total 4160
-rw-r--r--  1 aleph  staff   410K May  3 17:46 mongo-dev-1493822760.gz
-rw-r--r--  1 aleph  staff   1.9K May  3 17:46 mongo-dev-1493822760.log
-rw-r--r--  1 aleph  staff   410K May  3 17:47 mongo-dev-1493822820.gz
-rw-r--r--  1 aleph  staff   1.5K May  3 17:47 mongo-dev-1493822820.log

Metrics

Successful backups counter

mgob_scheduler_backup_total{plan="mongo-dev",status="200"} 8

Successful backups duration

mgob_scheduler_backup_latency{plan="mongo-dev",status="200",quantile="0.5"} 2.149668417
mgob_scheduler_backup_latency{plan="mongo-dev",status="200",quantile="0.9"} 2.39848413
mgob_scheduler_backup_latency{plan="mongo-dev",status="200",quantile="0.99"} 2.39848413
mgob_scheduler_backup_latency_sum{plan="mongo-dev",status="200"} 17.580484907
mgob_scheduler_backup_latency_count{plan="mongo-dev",status="200"} 8

Failed jobs count and duration (status 500)

mgob_scheduler_backup_latency{plan="mongo-test",status="500",quantile="0.5"} 2.4180213
mgob_scheduler_backup_latency{plan="mongo-test",status="500",quantile="0.9"} 2.438254775
mgob_scheduler_backup_latency{plan="mongo-test",status="500",quantile="0.99"} 2.438254775
mgob_scheduler_backup_latency_sum{plan="mongo-test",status="500"} 9.679809477
mgob_scheduler_backup_latency_count{plan="mongo-test",status="500"} 4

Restore

In order to restore from a local backup you have two options:

Browse mgob-host:8090/storage to identify the backup you want to restore. Login to your MongoDB server and download the archive using curl and restore the backup with mongorestore command line.

curl -o /tmp/mongo-test-1494056760.gz http://mgob-host:8090/storage/mongo-test/mongo-test-1494056760.gz
mongorestore --gzip --archive=/tmp/mongo-test-1494056760.gz --drop

You can also restore a backup from within mgob container. Exec into mgob, identify the backup you want to restore and use mongorestore to connect to your MongoDB server.

docker exec -it mgob sh
ls /storage/mongo-test
mongorestore --gzip --archive=/storage/mongo-test/mongo-test-1494056760.gz --host mongohost:27017 --drop

mgob's People

Contributors

azhelev avatar chris-scentregroup avatar ehlertjd avatar flaccid avatar floweb avatar freyert avatar h0x91b avatar harsh-2711 avatar jaben avatar jussikuosa avatar kevinsandow avatar log2 avatar mathewpeterson avatar maulal avatar paic avatar priyesh2609 avatar rebrendov avatar seanson avatar skbly7 avatar solidnerd avatar stefanprodan avatar wonko avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mgob's Issues

Remote renention?

Do you plan to add remote retention? Or it is not so needed in your view?

mongodump 3.5.x incompatible with mongodb 3.6.x

Running mgob against current stable/mongodb chart results in immediate falure after connection to remote database.

Tracked it down the admin.system.keys collection which cannot be dumped. I believe that this collection holds keys for the replication set. It cannot be accessed even from root user within the local mongo client.

See: https://jira.mongodb.org/browse/TOOLS-1797

bash-4.3# mongodump --version
mongodump version: v3.5.0-4965-g6c9498c36b

bash-4.3# mongodump -h mongo-mongodb-primary-0.mongo-mongodb-headless.default.svc.cluster.local -u root  --authenticationDatabase=admin -d admin
Enter password:

2018-07-26T23:52:01.769+0000	Failed: error counting admin.system.keys: not authorized on admin to execute command { count: "system.keys", query: {}, $readPreference: { mode: "secondaryPreferred" }, $db: "admin" }
bash-4.3# mongodump -h mongo-mongodb-primary-0.mongo-mongodb-headless.default.svc.cluster.local -u root  --authenticationDatabase=admin -d local
Enter password:

2018-07-26T23:52:29.369+0000	writing local.oplog.rs to 
2018-07-26T23:52:29.369+0000	writing local.startup_log to 
2018-07-26T23:52:29.369+0000	writing local.me to 
2018-07-26T23:52:29.369+0000	writing local.replset.minvalid to 
2018-07-26T23:52:29.386+0000	done dumping local.startup_log (3 documents)
2018-07-26T23:52:29.386+0000	writing local.replset.election to 
2018-07-26T23:52:29.388+0000	done dumping local.replset.election (1 document)
2018-07-26T23:52:29.389+0000	writing local.replset.oplogTruncateAfterPoint to 
2018-07-26T23:52:29.389+0000	done dumping local.me (1 document)
2018-07-26T23:52:29.390+0000	done dumping local.replset.oplogTruncateAfterPoint (1 document)
2018-07-26T23:52:29.393+0000	done dumping local.replset.minvalid (1 document)
2018-07-26T23:52:29.538+0000	done dumping local.oplog.rs (27764 documents)

When we dump directly in the mongodb container, works fine:

root@mongo-mongodb-primary-0:/# mongodump --version
mongodump version: r3.6.6
...
root@mongo-mongodb-primary-0:/# cd /tmp
root@mongo-mongodb-primary-0:/tmp# mongodump -u root -d admin
Enter password:

2018-07-27T00:01:25.996+0000	writing admin.system.users to 
2018-07-27T00:01:25.998+0000	done dumping admin.system.users (3 documents)
2018-07-27T00:01:25.998+0000	writing admin.system.version to 
2018-07-27T00:01:26.000+0000	done dumping admin.system.version (2 documents)
2018-07-27T00:01:26.000+0000	writing admin.movie to 
2018-07-27T00:01:26.002+0000	done dumping admin.movie (2 documents)

Suggest that mgob be updated to play with stable/mongodb chart or branched to support different releases.

Proposal: Pros Cons Documentation

Most of the teams I have worked on have opted for disk snapshots to have point in time backups, but I think I also see the value of using mongodump if you have a large number of databases. However, I'm also worried that using mongodump for large databases is unreasonable.

Would love to hear some of folks' experience using MGOB as a backup/restore solution as compared to disk snap shots.

Stats like: Backup finished in 2.359901432s archive size 448 kB make me queezy considering this option for databases that can be several GB.

Backup does not work at all due to moving log file error

When you run the mongodump command with --gzip and --archive options there is no output to stdout.
In logToFile function you're creating the file only if data length > 0.
Later, you're trying to move this log file (which not created) and it fails.

err = sh.Command("mv", log, planDir).Run()
if err != nil {
return res, errors.Wrapf(err, "moving file from %v to %v failed", log, planDir)
}

the parameter "authSource" is invalid

Hi Stefan,

I tried pkdone/gke-mongodb-shards-demo, and the mongo url is the following:

mongodb://main_admin:[email protected]:27017/test?authSource=admin

I added "params" to my mgob-cfg.yaml, like this:

...
    target:
      host: "mongos-router-service.default.svc.cluster.local"
      port: 27017
      database: "test"
      username: "main_admin"
      password: "abc123"
      params: "--authSource=admin"
...

But an error occurred:

error parsing command line options: unknown option "authSource"

What should I do?

Best,
Owen

S3 Backup fails for non us-east-1 region

When setting the plan up as follows:

s3:
  url: "https://s3.amazonaws.com/"
  bucket: "bucket-in-ap-southeast-1"
  accessKey: "accessKey"
  secretKey: "secretKey"
  api: "S3v4"

It results in this error during backup:

{"error":"S3 uploading /storage/db/db-1508351867.gz to db/bucket-in-ap-southeast-1 failed `/storage/db/db-1508351867.gz` -\u003e `db/bucket-in-ap-southeast-1/db-1508351867.gz` mc: \u003cERROR\u003e Failed to copy `/storage/db/db-1508351867.gz`. The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'ap-southeast-1' mc: \u003cERROR\u003e Session safely terminated. To resume session `mc session resume bityhPIa` : exit status 1"}

I have also tried setting the url to https://s3-ap-southeast-1.amazonaws.com/, but this too failed for another reason.

{"error":"S3 uploading /storage/db/db-1508350708.gz to db/bucket-in-ap-southeast-1 failed `/storage/db/db-1508350708.gz` -\u003e `db/bucket-in-ap-southeast-1/db-1508350708.gz` mc: \u003cERROR\u003e Failed to copy `/storage/db/db-1508350708.gz`. Amazon S3 endpoint should be 's3.amazonaws.com'. mc: \u003cERROR\u003e Session safely terminated. To resume session `mc session resume zDCASxDp` : exit status 1"}

I see you are using minio to handle the s3 cp. I've tried looking at their existing tickets to see if there's a known issue or a configuration option missing in either your source or theres for specifying a region.

This closed ticket would suggest that it should just work. minio/minio-java#494

Any thoughts you could provide would be most appreciated.

Rely on kubernetes scheduler

When you are doing backups from a database with considerable sizes we need to tune the limits of the deployment, using a statefulset means we need to have this running permanently and requesting a lot of resources from the cluster.

Ideally if we could rely on the kubernetes cronjobs and have successfull exit code we would prevent long running executions of this process and triggered them on scheduled time only.

Any considerations on this? Is this doable with start flags?

add minikube configuration and chart

Would be nice to have a configuration guide to create a fully functioning minikube test configuration for mgob. Suggest separate directory to document helm chart and storage provisioning required for minikube, or perhaps add commented section for minikube in values.yaml:

Changes to values.yaml for helm chart to provision minikube storage:

storage:
  longTerm:
    accessMode: "ReadWriteOnce"
    storageClass: "manual"
    name: "mgob-storage"
    size: 5Gi
  tmp:
    accessMode: "ReadWriteOnce"
    storageClass: "manual"
    name: "mgob-tmp"
    size: 1Gi

Config to create storage:

kubectl apply -f - << EOF
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv0001
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: manual
  capacity:
    storage: 5Gi
  hostPath:
    path: /data/pv0001/
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv0002
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: manual
  capacity:
    storage: 1Gi
  hostPath:
    path: /data/pv0002/
EOF

Create storage directories on minikube:

minikube ssh sudo mkdir /data/pv0001
minikube ssh sudo mkdir /data/pv0002

Install mongodb from helm chart:

helm install stable/mongodb

All that remains is to add mgob config to backup database "local" and "admin" of default mongodb to provide a fully functional example.

Clean way to backup a target using SSL?

Hello,

To begin with, thank you very much for this awesome work!

The situation here is that I need to backup a mongod that uses SSL encryption (eg. MongoDB Atlas).

I think I've found my way out using the target: password: field in my .yml plan file, since I can't find any mention of --ssl parameter in your code, but it feels a bit too hacky for me.

Maybe I missed something... and you can enlighten me on that, thank you!

Startup script fails on minikube.

Just applied the files from the documentation but the startup sript pod is erroring.

Environment: Minikube

[bweston@dell k8s]$ kubectl get --namespace=db pods
NAME                   READY     STATUS    RESTARTS   AGE
mongo-0                0/2       Pending   0          1m
startup-script-ftk2l   0/1       Error     3          1m
[bweston@dell k8s]$ kubectl -n db logs startup-script-ftk2l
bash: line 5: /sys/kernel/mm/transparent_hugepage/enabled: No such file or directory
!!! startup-script failed! exit code '1'

Failed: error connecting to db server: no reachable servers

Hi, error ocurred when i try to backup db, here is my plan

scheduler:
  # run every day at 19:00 UTC (Beijing 3:00 AM)
  # cron: "0 19 */1 * *"
  cron: "*/1 * * * *"
  # number of backups to keep locally
  retention: 14
  # backup operation timeout in minutes
  timeout: 60
target:
  # mongod IP or host name
  host: "127.0.0.1"
  # mongodb port
  port: 27017
  # mongodb database name, leave blank to backup all databases
  database: "studyland"
  # leave blank if auth is not enabled
  username: ""
  password: ""
  # username: "admin"
  # password: "secret"
  # # add custom params to mongodump (eg. Auth or SSL support), leave blank if not needed
  # params: "--ssl --authenticationDatabase admin"

Failed to create backup - mongoDB X509 authentication

Hi,

I have setup a mongoDB replica set with X509 client authentication. My MGOB service fails to create backups. I have already added a role backup and restore for the MGOB service.

If you have any idea please do share.

Thanks


time="2018-04-10T02:21:00Z" level=error msg="Backup failed mongodump log 2018-04-10T02:21:00.035+0000 Failed: error getting collections for database my-db: error running listCollections. Database: my-db Err: not authorized on my-db to execute command { listCollections: 1, cursor: {} } : exit status 1" plan="primary_database"

My config file:
target:
host: "mongod-0.mongodb-service,mongod-1.mongodb-service,mongod-2.mongodb-service"
port: 27017
database: "my-db"
params: "--ssl --sslCAFile /keys/root/root-ca.pem --sslPEMKeyFile /keys/self/mongod.pem --authenticationMechanism MONGODB-X509"
scheduler:
cron: "* * * * *"
retention: 5
timeout: 60

Pass database credentials as secret

Hello!

in kubernetes manifests, I describe a mgob configuration file, mounted as configMap. In this file i have to provide user database credentials and it's not a good practice. Is there some way to configure db credentials creating a secret and mounting it along mogb configuration?

Thanks in advance and congratulations for the project

mongodump: positional arguments not allowed

I'm encountering

curl -X POST http://localhost:8090/backup/production 
{"error":"mongodump log 2018-11-07T06:10:28.476+0000\tpositional arguments not allowed: [false] 2018-11-07T06:10:28.476+0000\ttry 'mongodump --help' for more information : exit status 3"}

Could this be due to the version of mongodump on the server? Or issue #53?

Proposal: Give the possibility to schedule also the oplog

Hi,
First, Great job !!! I'm really interested to use your tool ;-) And i'm also interested to help you if you need.

Today we can consider than mgob can only perform full backup, it's the first step for backuping databases, but depending on the scheduling, it can be a pain to loss data between 2 backups... What could be very usefull, is to schedule also a periodic backup of the oplog (operations log), or even better stream the oplog to the s3/gcloud/sftp storage.
Backuping the oplog can give the possibility to recover to a dedicated time, for example:

  • Just before a crash or a data corruption
  • Just before a drop collection (malicious order)

With full backup and oplog backup, Point In Time Recovery (PITR) become possible.

In fact, it's already possible with your tool to do an oplog backup, but we need to launch 2 processes, one for the full backup and another one for the oplog, and there are no link between them (in case of error with the full backup, no need to backup the oplog...).

As i said before, what could be even better, is to stream the oplog, it's the best way to optimise the size and the time for the oplog backup. overhead < for the backup consumption

Regards,

tmp Storage cleanup

Hi Stephan,
I see backups staying on the container in /storage after they are transferred to an s3 store. Are those suppose to be cleaned up after they are transferred? Or are they cleaned up with the retention policy set? Will they get cleaned up on s3 and the container?

auth for /storage

It would probably be a cool feature to have the possibility to protect the content in /storage - path only for authorized access.

In the actual situation, everybody with access to the webserver can get all the dumped data. Is there any solution for this?

Thanks
Martin

Proposal: Azure blob storage

Hi,

This is cool solution. I want to use this but I use Azure which is not supported by mgob.
Can I add some code to support Azure blob storage?

Support for dumping all databases

Hello,

would it be possible to add support for dumping all databases by omitting the --db parameter on mongodump command line? It's quite tedious to create the config when there are multiple databases present on the server.

mc: \u003cERROR\u003e Failed to copy

mc command failed to run copy to S3 storage.

Below error is observed.
Object name cannot be empty mc: \u003cERROR\u003e Session safely terminated. To resume session mc session resume bWEhwoRn : exit status 1"

Looks like "mc" command is not appending target with object name to copy.

mc command should be,
mc cp /storage/project/db.gz s3/projectname/db.gz

Existing command is.
mc cp /storage/project/db.gz s3/bucketname/

Due to missing target object name the command is failing to copy object into S3 bucket.

Feature request: helm chart

Kubernetes has an official package manager, Helm, for which creating charts is really simple. If we could implement this, installation of this package on Kubernetes would be as simple as $ helm install stable/mgob.

Setting up Helm is done through $ helm create --client-only.

More information can be found here.

Mongo 4 tools release date

Hello there,

I'm trying the tool and so far it looks awesome. We are using mongo 4 and setup the mgob edge image and it looks fine. I would like to avoid living the "edge" version so the questions are:
When do you plan to make the next release?
Are you planning to include the mongo 4 tools?
Can we help in any way to make that happen?

backup using s3 with k8s

Hey there,
I'm wondering if s3 will work with k8s here. I have existing dbs outside of k8s and want to leverage the backup function here if that's possible.

Some routes are not working

The routes with URLparams are not working currently.

@master branch:

make run
Successfully built f83769b68199
>>> Starting mgob container
f75cb6474318021421d1b2ef7003a07095e5a935dd9bdf42426fe665dff15663
➜  mgob git:(master) docker ps
CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS              PORTS                    NAMES
f75cb6474318        stefanprodan/mgob:0.5   "./mgob -ConfigPat..."   7 seconds ago       Up 5 seconds        0.0.0.0:8090->8090/tcp   mgob-0.5

curl -v http://localhost:8090/status
*   Trying ::1...
* Connected to localhost (::1) port 8090 (#0)
> GET /status HTTP/1.1
> Host: localhost:8090
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Thu, 17 Aug 2017 13:43:35 GMT
< Content-Length: 113
< 
[{"plan":"mongo-dev","next_run":"2017-08-17T13:45:00Z"},{"plan":"mongo-test","next_run":"2017-08-17T13:44:00Z"}]
* Connection #0 to host localhost left intact

curl -v -X POST http://localhost:8090/backup/mongo-dev
*   Trying ::1...
* Connected to localhost (::1) port 8090 (#0)
> POST /backup/mongo-dev HTTP/1.1
> Host: localhost:8090
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Thu, 17 Aug 2017 13:44:00 GMT
< Content-Length: 19
< 
404 page not found
* Connection #0 to host localhost left intact

Expected, at least a 500 error response.

I have changed:

r.Route("/backup", func(r chi.Router) {
		r.Use(configCtx(*s.Config))
		r.Post("/:planID", postBackup)
	})

to

r.Route("/backup", func(r chi.Router) {
		r.Use(configCtx(*s.Config))
		r.Post("/{planID}", postBackup)
	})

which result is the expected outcome:

curl -v -X POST http://localhost:8090/backup/mongo-dev
*   Trying ::1...
* Connected to localhost (::1) port 8090 (#0)
> POST /backup/mongo-dev HTTP/1.1
> Host: localhost:8090
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 500 Internal Server Error
< Content-Type: application/json
< Date: Thu, 17 Aug 2017 13:51:18 GMT
< Content-Length: 132
< 
{"error":"mongodump log 2017-08-17T13:51:18.515+0000\tFailed: error connecting to db server: no reachable servers : exit status 1"}
* Connection #0 to host localhost left intact

Releases after 0.8 breaking S3 integration

Hi, after a few weeks of pain I decided to find out what was happening with my backup routine. I was using the edge image at my cluster and all my backups was breaking with the error:

mc config host for plan google failed: exit status 1

I tried a few times to solve the problem but the only way was to downgrade to the version 0.8. After that, everything is working great again.

Store backup plans in a remote storage

I want to be able to run this application in a Swarm environment and configure using a remote storage API.
Something like Etcd or just from grabbing the plans from a S3 bucket.

Permission denied : exit status 126 while deploying on OpenShift

I am trying to deploy mgob on openshift however it is having CrashLoopBackOff

Here are the logs.


  | time="2018-11-14T08:07:34Z" level=info msg="Starting with config: &{LogLevel:debug JSONLog:false Port:8090 ConfigPath:/config StoragePath:/storage TmpPath:/tmp DataPath:/data}"
  | time="2018-11-14T08:07:34Z" level=info msg="mongodump version: v3.8.0-3220-g866b921892 git version: 866b9218921c22d317340e4e1e580a7cd5f084b8 Go version: go1.11.1    os: linux    arch: amd64    compiler: gc OpenSSL version: OpenSSL 1.1.1  11 Sep 2018 "
  | time="2018-11-14T08:07:34Z" level=fatal msg="mc failed /bin/sh: mc: Permission denied : exit status 126"

oc get pods -n db

NAME      READY     STATUS             RESTARTS   AGE
mgob-0    0/1       CrashLoopBackOff   3          1m
mongo-0   2/2       Running            0          16h

Here are the relevant events Events:

Type     Reason     Age                From                         Message
 ----     ------     ----               ----                         -------
 Normal   Scheduled  17s                default-scheduler            Successfully assigned db/mgob-0 to dehamku1332xxx.yz
 Normal   Pulled     13s (x2 over 14s)  kubelet, denotsl1332.int.kn  Container image "stefanprodan/mgob:0.10" already present on machine
 Normal   Created    13s (x2 over 14s)  kubelet, denotsl1332.int.kn  Created container
 Normal   Started    13s (x2 over 14s)  kubelet, denotsl1332.int.kn  Started container
 Warning  BackOff    11s (x2 over 12s)  kubelet, denotsl1332.int.kn  Back-off restarting failed container

Error while backup routine

Hi guys, I already deployed mgob at my Kubernetes Cluster and it was working fine.

But now, I'm getting this error and I don't know how to handle this:

{"error":"mongodump log 2018-06-29T21:53:31.016+0000\twriting googlecrawler.reviews to archive '/tmp/googlecrawler-1530309210.gz' 2018-06-29T21:53:31.017+0000\twriting googlecrawler.processes to archive '/tmp/googlecrawler-1530309210.gz' 2018-06-29T21:53:31.017+0000\twriting googlecrawler.credentials to archive '/tmp/googlecrawler-1530309210.gz' 2018-06-29T21:53:31.017+0000\twriting googlecrawler._properties to archive '/tmp/googlecrawler-1530309210.gz' 2018-06-29T21:53:31.023+0000\tFailed: archive writer: error writing data for collection `googlecrawler.credentials` to disk: error reading collection: Failed to parse: { find: \"credentials\", skip: 0, snapshot: true, $readPreference: { mode: \"secondaryPreferred\" }, $db: \"googlecrawler\" }. Unrecognized field 'snapshot'. / Mux ending but selectCases still open 3 : exit status 1"}

Does anyone already went through this?

MongoDB 4.0 build

Hi!
Do you have any plans on making a build for mongo 4.0?

I would also like to take the opportunity to thank you for all the work you put on this image, It has saved me from a lot of headaches. 🙃

/André

Adding Authentication crash loops the pod

I've tried adding

    containers:
  - image: mongo
    name: mongo
    command: ["mongo", "--auth"]

    containers:
  - image: mongo
    name: mongo
    args: ["--auth"]

but both the methods didn't seem to work out as it crash loops the pod.

 Normal   Created                14s (x2 over 17s)  kubelet, aks-nodepool1-17011719-0  Created container
 Normal   Started                13s (x2 over 17s)  kubelet, aks-nodepool1-17011719-0  Started container
 Warning  BackOff                12s                kubelet, aks-nodepool1-17011719-0  Back-off restarting failed container

Am i missing something ?

Sftp without password auth

Im trying to connect my sftp backup with a remote server which has password auth disabled.
Is it possible connect only with the ssh key?
Im getting this error:
sshd[26884]: Connection closed by my ip port 38328 [preauth]
ssh: unable to authenticate, attempted methods [none], no supported methods remain"

Backup failed S3

"Backup failed S3 uploading /storage/plan/plan-1522584360.gz to x/x failed /storage/x/x-1522584360.gz -> x/x/x-1522584360.gz mc: Failed to copy /storage/x/x-1522584360.gz. Put http://x.s3.eu-west-3.amazonaws.com/x-1522584360.gz: http: ContentLength=767 with Body length 0 mc: Session safely terminated. To resume session mc session resume RItPCzjD : exit status 1" plan=x

Im trying with sftp failed, gcloud failed and aws failed, please any idea what Im doing wrong?

Logs from s3
5485e3f48a1f48a29fad584 x [01/Apr/2018:12:47:01 +0000] my.ip - 2382DD5AA85C6897 REST.PUT.OBJECT x-1522586820.gz
 "PUT /x-1522586820.gz HTTP/1.1" 307 TemporaryRedirect 447 - 2 - "-" "Minio (linux; amd64) minio-go/4.0.0 mc/2017-12-12T01:08:02Z" -

backup executed two times simultaneously

Hello,

i have configured a single backup plan which is being executed twice at the same time. The backup is failing since the two backup processes try to write the same file.

My config file:

bash-4.3# cat prod-mongodb.yml 
scheduler:
  # run every day at 4:00 UTC
  cron: "10 4 * * *"
  # number of backups to keep locally
  retention: 10
  # backup operation timeout in minutes
  timeout: 10
target:
  # mongodb IP or host name
  host: "mongo-0.mongo.default.svc.cluster.local"
  # mongodb port
  port: 27017
  # mongodb database name
  database: ""
  # leave blank if auth is not enabled
  username: ""
  password: ""

My log file:

time="2017-12-21T09:45:23Z" level=info msg="Starting with config: &{LogLevel:debug Port:8090 ConfigPath:/config StoragePath:/storage TmpPath:/tmp DataPath:/data}" 
time="2017-12-21T09:45:23Z" level=info msg="mongodump version: v3.5.0-4965-g6c9498c36b git version: 6c9498c36b40b091bde3b79c88ddb837e94fc9c5 Go version: go1.8.1    os: linux    arch: amd64    compiler: gc OpenSSL version: LibreSSL 2.4.5 " 
time="2017-12-21T09:45:23Z" level=info 
time="2017-12-21T09:45:24Z" level=info msg="Google Cloud SDK 181.0.0 bq 2.0.27 core 2017.11.28 gsutil 4.28 " 
time="2017-12-21T09:45:24Z" level=info msg="Next tmp cleanup run at 2017-12-21 10:00:00 +0000 UTC" 
time="2017-12-21T09:45:24Z" level=info msg="Next run at 2017-12-21 09:50:00 +0000 UTC" plan=prod-mongodb 
time="2017-12-21T09:45:24Z" level=info msg="Next run at 2017-12-21 09:50:00 +0000 UTC" plan=prod-mongodb 
time="2017-12-21T09:45:24Z" level=info msg="Starting HTTP server on port 8090" 
time="2017-12-21T09:50:00Z" level=info msg="Backup started" plan=prod-mongodb 
time="2017-12-21T09:50:00Z" level=info msg="Backup started" plan=prod-mongodb 
time="2017-12-21T09:50:18Z" level=error msg="Backup failed mongodump log 2017-12-21T09:50:00.122+0000   writing admin.system.version to archive '/tmp/prod-mongodb-1513849800.gz' 2017-12-21T09:50:00.125+0000  done dumping admin.system.version (1 document) 2017-12-21T09:50:00.125+0000  writing elastic_srv.tutoringlogs to archive '/tmp/prod-mongodb-1513849800.gz' 2017-12-21T09:50:00.127+0000      writing elastic_srv.flashcardsstats to archive '/tmp/prod-mongodb-1513849800.gz' 2017-12-21T09:50:00.135+0000writing elastic_srv.users to archive '/tmp/prod-mongodb-1513849800.gz' 2017-12-21T09:50:00.227+0000     writing elastic_srv.practicequestions to archive '/tmp/prod-mongodb-1513849800.gz' 2017-12-21T09:50:03.122+0000 [........................]       elastic_srv.tutoringlogs   101/23125   (0.4%) 2017-12-21T09:50:03.122+0000  [........................]    elastic_srv.flashcardsstats   101/20926   (0.5%) 2017-12-21T09:50:03.122+0000     [######..................]              elastic_srv.users  4397/16254  (27.1%) 2017-12-21T09:50:03.122+0000  [........................]  elastic_srv.practicequestions   101/20921   (0.5%) 2017-12-21T09:50:03.122+0000      2017-12-21T09:50:03.233+0000   [########################]  elastic_srv.users  16254/16254  (100.0%) 2017-12-21T09:50:06.032+0000    [........................]       elastic_srv.tutoringlogs  101/23125  (0.4%) 2017-12-21T09:50:06.032+0000       [........................]    elastic_srv.flashcardsstats  101/20926  (0.5%) 2017-12-21T09:50:06.032+0000   [........................]  elastic_srv.practicequestions  101/20921  (0.5%) 2017-12-21T09:50:06.032+0000         2017-12-21T09:50:09.033+0000   [####....................]       elastic_srv.tutoringlogs  4395/23125  (19.0%) 2017-12-21T09:50:09.033+0000 [........................]    elastic_srv.flashcardsstats   101/20926   (0.5%) 2017-12-21T09:50:09.033+0000      [........................]  elastic_srv.practicequestions   101/20921   (0.5%) 2017-12-21T09:50:09.033+0000      2017-12-21T09:50:12.122+0000[####....................]       elastic_srv.tutoringlogs  4395/23125  (19.0%) 2017-12-21T09:50:12.123+0000     [........................]    elastic_srv.flashcardsstats   101/20926   (0.5%) 2017-12-21T09:50:12.123+0000     [........................]  elastic_srv.practicequestions   101/20921   (0.5%) 2017-12-21T09:50:12.123+0000   2017-12-21T09:50:13.525+0000   done dumping elastic_srv.users (16254 documents) 2017-12-21T09:50:13.525+0000   writing tutoring_srv.requests to archive '/tmp/prod-mongodb-1513849800.gz' 2017-12-21T09:50:15.123+0000      [########................]       elastic_srv.tutoringlogs  8590/23125  (37.1%) 2017-12-21T09:50:15.123+0000     [........................]    elastic_srv.flashcardsstats   101/20926   (0.5%) 2017-12-21T09:50:15.123+0000  [........................]  elastic_srv.practicequestions   101/20921   (0.5%) 2017-12-21T09:50:15.123+0000      2017-12-21T09:50:17.548+0000   done dumping tutoring_srv.requests (128 documents) 2017-12-21T09:50:17.548+0000 writing noondb.matches to archive '/tmp/prod-mongodb-1513849800.gz' Killed : exit status 137" plan=prod-mongodb 
time="2017-12-21T09:50:18Z" level=info msg="Next run at 2017-12-21 10:00:00 +0000 UTC" plan=prod-mongodb 
time="2017-12-21T09:50:18Z" level=error msg="Backup failed mongodump log 2017-12-21T09:50:00.053+0000   writing admin.system.version to archive '/tmp/prod-mongodb-1513849800.gz' 2017-12-21T09:50:00.059+0000  done dumping admin.system.version (1 document) 2017-12-21T09:50:00.059+0000  writing elastic_srv.tutoringlogs to archive '/tmp/prod-mongodb-1513849800.gz' 2017-12-21T09:50:00.059+0000      writing elastic_srv.flashcardsstats to archive '/tmp/prod-mongodb-1513849800.gz' 2017-12-21T09:50:00.059+0000writing elastic_srv.users to archive '/tmp/prod-mongodb-1513849800.gz' 2017-12-21T09:50:00.060+0000     writing elastic_srv.practicequestions to archive '/tmp/prod-mongodb-1513849800.gz' 2017-12-21T09:50:03.126+0000 [####################....]              elastic_srv.users  13615/16254  (83.8%) 2017-12-21T09:50:03.126+0000 [........................]  elastic_srv.practicequestions    101/20921   (0.5%) 2017-12-21T09:50:03.126+0000    [........................]       elastic_srv.tutoringlogs    101/23125   (0.4%) 2017-12-21T09:50:03.126+0000 [........................]    elastic_srv.flashcardsstats    101/20926   (0.5%) 2017-12-21T09:50:03.126+0000     2017-12-21T09:50:03.328+0000   [########################]  elastic_srv.users  16254/16254  (100.0%) 2017-12-21T09:50:06.032+0000    [........................]  elastic_srv.practicequestions   101/20921   (0.5%) 2017-12-21T09:50:06.032+0000     [#####...................]       elastic_srv.tutoringlogs  5595/23125  (24.2%) 2017-12-21T09:50:06.032+0000 [........................]    elastic_srv.flashcardsstats   101/20926   (0.5%) 2017-12-21T09:50:06.032+0000       2017-12-21T09:50:09.126+0000   [........................]  elastic_srv.practicequestions   101/20921   (0.5%) 2017-12-21T09:50:09.126+0000 [########................]       elastic_srv.tutoringlogs  8513/23125  (36.8%) 2017-12-21T09:50:09.126+0000      [........................]    elastic_srv.flashcardsstats   101/20926   (0.5%) 2017-12-21T09:50:09.126+0000      2017-12-21T09:50:12.032+0000[........................]  elastic_srv.practicequestions   101/20921   (0.5%) 2017-12-21T09:50:12.032+0000     [########................]       elastic_srv.tutoringlogs  8513/23125  (36.8%) 2017-12-21T09:50:12.032+0000     [........................]    elastic_srv.flashcardsstats   101/20926   (0.5%) 2017-12-21T09:50:12.032+0000   2017-12-21T09:50:15.123+0000   [........................]  elastic_srv.practicequestions   101/20921   (0.5%) 2017-12-21T09:50:15.123+0000     [########................]       elastic_srv.tutoringlogs  8590/23125  (37.1%) 2017-12-21T09:50:15.123+0000  [........................]    elastic_srv.flashcardsstats   101/20926   (0.5%) 2017-12-21T09:50:15.123+0000      2017-12-21T09:50:17.548+0000   done dumping elastic_srv.users (16254 documents) 2017-12-21T09:50:17.549+0000        writing tutoring_srv.requests to archive '/tmp/prod-mongodb-1513849800.gz' Killed : exit status 137" plan=prod-mongodb 
time="2017-12-21T09:50:18Z" level=info msg="Next run at 2017-12-21 10:00:00 +0000 UTC" plan=prod-mongodb

I am running mgob:edge container from today in a kubernetes cluster

Environment variables

Is there a way to set a password through an environment variable (from the Kubernetes secret) to avoid storing in the repository? Thanks for this great tool!

Question: Mongo Sharded Cluster Support

This looks killer and full featured. Taking a quick glance at the readme I saw the yaml config file only supports on database.

Everything I read about doing backups of a sharded cluster talks about turning off the balancer, fsyncLocking read replica in each shard and doing a dump of config, and each shard. Then reversing those steps once done.

Is your application intended to backup a mongo sharded cluster via mongos, have support for the suggested steps for by MongoDB, or have no support for sharded clusters?

In either case great work!

slack alerts

Hi,
I have slack alerts setup to warnOnly: true, but I'm getting alerts for a regular backup starting.
I'm redacting some lines but why does it say failed yet its starting the backup and running?

"Backup failed mongodump log 2018-06-26T23:30:00.123+0000\twriting admin.system.users to archive '/tmp/mongo162-main-s02-r03-1530055800.gz' 2018-06-26T23:30:00.129+0000\tdone dumping admin.system.users (1 document) 2018-06-26T23:30:00.129+0000\twriting admin.system.version to archive '/tmp/mongo162-main-s02-r03-1530055800.gz' 2018-06-26T23:30:00.136+0000\tdone dumping admin.system.version (3 documents) 2018-06-26T23:30:00.136+0000\twriting DBNAME.TemplateRevs to archive '/tmp/mongo162-main-s02-r03-1530055800.gz' 2018-06-26T23:30:00.166+0000\twriting DBNAME.TemplateDataBindings to archive '/tmp/mongo162-main-s02-r03-1530055800.gz' 2018-06-26T23:30:00.167+0000\twriting DBNAME.ContentCollections to archive '/tmp/mongo162-main-s02-r03-1530055800.gz' 2018-06-26T23:30:00.167+0000\twriting DBNAME.ContentItems to archive '/tmp/mongo162-main-s02-r03-1530055800.gz' 2018-06-26T23:30:03.023+0000\t[........................] DBNAME.TemplateRevs 118151/313932049 (0.0%) 2018-06-26T23:30:03.023+0000\t[........................] DBNAME.TemplateDataBindings 56793/17733566 (0.3%) 2018-06-26T23:30:03.023+0000\t[........................] DBNAME.ContentCollections 76530/16294369 (0.5%) 2018-06-26T23:30:03.023+0000\t[........................] DBNAME.ContentItems 47176/253699225 (0.0%) 2018-06-26T23:30:03.023+0000\t 2018-06-26T23:30:06.023+0000\t[........................] DBNAME.TemplateRevs 236300/313932049 (0.1%)

GCP Retention Policy

Hi,

When using a retention policy (using GCP bucket):

scheduler:
cron: "*/1 * * * *"
retention: 1
timeout: 60

It doesn't actually only keep 1 file.

I am expecting it to remove the old file and upload the new file when retention is 1.

Thanks

Use proxy to connect to MongoDB

My container is running on a VM in a corporate environment, where we have to use a proxy server to connect to the internet.

I added the HTTP_PROXY and HTTPS_PROXY environment variables to the run command.

docker run -dp 8090:8090 --name mgob \
    --env HTTP_PROXY="http://myproxy:port" \
    --env HTTPS_PROXY="https://myproxy:port" \
    -v "/mogb/config:/config" \
    -v "/mogb/storage:/storage" \
    -v "/mgob/tmp:/tmp" \
    -v "/mgob/data:/data" \
    stefanprodan/mgob \
    -LogLevel=info

However, mongodump is unable to connect to my external MongoDB instance. I am aware that these are HTTP proxies and mongodump is opening a direct TCP connection.

Do you have any suggestions on how to use a proxy server within your (very nice!) backup solution?

restore backup plan

Hi,

first, thanks about amazing project.. it is very useful

Have you already think of implementing a restore plan to validate if a backup worked?

mongodump version

I see the latest repo image pull is coming up with mongodump version: v3.7.0-4757-gc31a4d0fd5
The release page says mongodb-tools to 3.6.4-r0
Any idea why they differ?

The access key ID you provided does not exist in our records

I'm trying to use the credentials of a IAM user I created on the AWS console, but when the backup is tried to be done, it throws the "The access key ID you provided does not exist in our records" error, even though my user from which I'm taking the credentials has full access to S3. Anyone happened to have the same issue?

Backup failures do not clean up

I've noticed that if a backup exits after an error the partial backup file is left in /tmp which eats up space. Is there a way this could be cleaned if an exit occurs?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.