Giter Club home page Giter Club logo

py-mongo-sync's Introduction

py-mongo-sync

An oplog-based realtime sync tool for MongoDB written in Python.

Features

  • full sync including data and indexes
  • oplog-based incremental sync
  • sync the specified databases
  • sync the specified collections
  • support MongoDB(v2.4 or later) and TokuMX

Notes

  • source must be a replica set
  • Ignore the following databases:
    • admin
    • local
  • ignore the following collections
    • system.users
    • system.profile
  • create users for destination manually if necessary
  • run as superuser through setting '--src-username' and '--src-password' if source authentication is enabled
  • you could sync a sharded-cluster by running a process for each shard, first of all, guarantee that balancer is off

Not Supported

  • geospatial index

Dependency

  • pymongo 3.0 or later

Usage

sync

usage: sync.py [-h] --from [FROM] [--src-authdb [SRC_AUTHDB]]
               [--src-username [SRC_USERNAME]] [--src-password [SRC_PASSWORD]]
               [--src-engine [SRC_ENGINE]] --to [TO]
               [--dst-authdb [DST_AUTHDB]] [--dst-username [DST_USERNAME]]
               [--dst-password [DST_PASSWORD]] [--dbs DBS [DBS ...]]
               [--colls COLLS [COLLS ...]] [--src-db [SRC_DB]]
               [--dst-db [DST_DB]] [--start-optime [START_OPTIME]]
               [--log [LOG]]

Sync data from a replica-set to another mongod/replica-set/sharded-cluster.

optional arguments:
  -h, --help            show this help message and exit
  --from [FROM]         the source must be a member of replica-set
  --src-authdb [SRC_AUTHDB]
                        authentication database, default is 'admin'
  --src-username [SRC_USERNAME]
                        src username
  --src-password [SRC_PASSWORD]
                        src password
  --src-engine [SRC_ENGINE]
                        src engine, the value could be mongodb or tokumx,
                        default is mongodb
  --to [TO]             the destination should be a mongos or mongod instance
  --dst-authdb [DST_AUTHDB]
                        authentication database, default is 'admin'
  --dst-username [DST_USERNAME]
                        dst username
  --dst-password [DST_PASSWORD]
                        dst password
  --dbs DBS [DBS ...]   databases to sync, conflict with --colls
  --colls COLLS [COLLS ...]
                        collections to sync, conflict with --dbs
  --src-db [SRC_DB]     src database name, work with '--dst-db', conflict with
                        '--dbs' and '--colls'
  --dst-db [DST_DB]     dst database name, work with '--src-db', conflict with
                        '--dbs' and '--colls'
  --start-optime [START_OPTIME]
                        start optime, a timestamp value in second for MongoDB
                        or a 'YYYYmmddHHMMSS' value for TokuMX
  --log [LOG]           log file path

check

usage: check.py [-h] --from [FROM] [--src-authdb [SRC_AUTHDB]]
                [--src-username [SRC_USERNAME]]
                [--src-password [SRC_PASSWORD]] --to [TO]
                [--dst-authdb [DST_AUTHDB]] [--dst-username [DST_USERNAME]]
                [--dst-password [DST_PASSWORD]] [--dbs DBS [DBS ...]]
                [--src-db [SRC_DB]] [--dst-db [DST_DB]]

Check data consistency including data and indexes.

optional arguments:
  -h, --help            show this help message and exit
  --from [FROM]         the source must be a mongod instance of replica-set
  --src-authdb [SRC_AUTHDB]
                        authentication database, default is 'admin'
  --src-username [SRC_USERNAME]
                        src username
  --src-password [SRC_PASSWORD]
                        src password
  --to [TO]             the destionation should be a mongos or mongod instance
  --dst-authdb [DST_AUTHDB]
                        authentication database, default is 'admin'
  --dst-username [DST_USERNAME]
                        dst username
  --dst-password [DST_PASSWORD]
                        dst password
  --dbs DBS [DBS ...]   databases to check
  --src-db [SRC_DB]     src database to check, work with '--dst-db', conflict
                        with '--dbs'
  --dst-db [DST_DB]     dst database to check, work with '--src-db', conflict
                        with '--dbs'

py-mongo-sync's People

Contributors

caosiyang avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.