Giter Club home page Giter Club logo

mongo-swarm's Introduction

mongo-swarm

Build Status Docker Image

Mongo-swarm is a POC project that automates the bootstrapping process of a MongoDB cluster for production use. With a single command you can deploy the Mongos, Config and Data replica sets onto Docker Swarm, forming a high-available MongoDB cluster capable of surviving multiple nodes failure without service interruption. The Docker stack is composed of two MongoDB replica sets, two Mongos instances and the mongo-bootstrap service. Mongo-bootstrap is written in Go and handles the replication, sharding and routing configuration.

Overview

Prerequisites

In oder to deploy the MongoDB stack you should have a Docker Swarm cluster made out of eleven nodes:

  • 3 Swarm manager nodes (prod-manager-1, prod-manager-2, prod-manager-3)
  • 3 Mongo data nodes (prod-mongodata-1, prod-mongodata-2, prod-mongodata-3)
  • 3 Mongo config nodes (prod-mongocfg-1, prod-mongocfg-2, prod-mongocfg-3)
  • 2 Mongo router nodes (prod-mongos-1, prod-mongos-2)

You can name your Swarm nodes however you want, the bootstrap process uses placement restrictions based on the mongo.role label. For the bootstrapping to take place you need to apply the following labels:

Mongo data nodes

docker node update --label-add mongo.role=data1 prod-mongodata-1
docker node update --label-add mongo.role=data2 prod-mongodata-2
docker node update --label-add mongo.role=data3 prod-mongodata-3

Mongo config nodes

docker node update --label-add mongo.role=cfg1 prod-mongocfg-1
docker node update --label-add mongo.role=cfg2 prod-mongocfg-2
docker node update --label-add mongo.role=cfg3 prod-mongocfg-3

Mongos nodes

docker node update --label-add mongo.role=mongos1 prod-mongos-1
docker node update --label-add mongo.role=mongos2 prod-mongos-2

Deploy

Clone this repository and run the bootstrap script on a Docker Swarm manager node:

$ git clone https://github.com/stefanprodan/mongo-swarm
$ cd mongo-swarm

$ ./bootstrap.sh

The bootstrap.sh script creates two overlay networks and deploys the mongo stack:

docker network create --attachable -d overlay mongo
docker network create --attachable -d overlay mongos

docker stack deploy -c swarm-compose.yml mongo

Networking

The config and data replica sets are isolated from the rest of the swarm in the mongo overlay network. The routers, Mongos1 and Mongos2 are connected to the mongo network and to the mongos network. You should attach application containers to the mongos network in order to communicate with the MongoDB Cluster.

Persistent storage

At first run, each data and config node will be provisioned with a named Docker volume. This ensures the MongoDB databases will not be purged if you restart or update the MongoDB cluster. Even if you remove the whole stack the volumes will remain on the disk. If you want to delete the MongoDB data and config you have to run docker volume purge on each Swarm node.

Bootstrapping

After the stack has been deploy the mongo-bootstrap container will do the following:

  • waits for the data nodes to be online
  • joins the data nodes into a replica set (datars)
  • waits for the config nodes to be online
  • joins the config nodes into a replica set (cfgrs)
  • waits for the mongos nodes to be online
  • adds the data replica set shard to the mongos instances

You can monitor the bootstrap process by watching the mongo-bootstrap service logs:

$ docker service logs -f mongo_bootstrap

msg="Bootstrap started for data cluster datars members [data1:27017 data2:27017 data3:27017]"
msg="datars member data1:27017 is online"
msg="datars member data2:27017 is online"
msg="datars member data3:27017 is online"
msg="datars replica set initialized successfully"
msg="datars member data1:27017 state PRIMARY"
msg="datars member data2:27017 state SECONDARY"
msg="datars member data3:27017 state SECONDARY"
msg="Bootstrap started for config cluster cfgrs members [cfg1:27017 cfg2:27017 cfg3:27017]"
msg="cfgrs member cfg1:27017 is online"
msg="cfgrs member cfg2:27017 is online"
msg="cfgrs member cfg3:27017 is online"
msg="cfgrs replica set initialized successfully"
msg="cfgrs member cfg1:27017 state PRIMARY"
msg="cfgrs member cfg2:27017 state SECONDARY"
msg="cfgrs member cfg3:27017 state SECONDARY"
msg="Bootstrap started for mongos [mongos1:27017 mongos2:27017]"
msg="mongos1:27017 is online"
msg="mongos1:27017 shard added"
msg="mongos2:27017 is online"
msg="mongos2:27017 shard added"

High availability

A MongoDB cluster provisioned with mongo-swarm can survive node failures and will start an automatic failover if:

  • the primary data node goes down
  • the primary config node goes down
  • one of the mongos nodes goes down

When the primary data or config node goes down, the Mongos instances will detect the new primary node and will reroute all the traffic to it. If a Mongos node goes down and your applications are configured to use both Mongos nodes, the Mongo driver will switch to the online Mongos instance. When you recover a failed data or config node, this node will rejoin the replica set and resync if the oplog size allows it.

If you want the cluster to outstand more than one node failure per replica set, you can horizontally scale up the data and config sets by modifying the swarm-compose.yml file. Always have an odd number of nodes per replica set to avoid split brain situations.

You can test the automatic failover by killing or removing the primary data and config nodes:

root@prod-data1-1:~# docker kill mongo_data1.1....
root@prod-cfg1-1:~# docker rm -f mongo_cfg1.1....

When you bring down the two instances Docker Swarm will start new containers to replace the killed ones. The data and config replica sets will choose a new leader and the newly started instances will join the cluster as followers.

You can check the cluster state by doing an HTTP GET on mongo-bootstrap port 9090.

docker run --rm --network mongo tutum/curl:alpine curl bootstrap:9090

Client connectivity

To test the Mongos connectivity you can run an interactive mongo container attached to the mongos network:

$ docker run --network mongos -it mongo:3.4 mongo mongos1:27017 

mongos> use test
switched to db test

mongos> db.demo.insert({text: "demo"})
WriteResult({ "nInserted" : 1 })

mongos> db.demo.find()
{ "_id" : ObjectId("59a6fa01e33a5cec9872664f"), "text" : "demo" }

The Mongo clients should connect to all Mongos nodes that are running on the mongos overlay network. Here is an example with the mgo golang MongoDB driver:

session, err := mgo.Dial("mongodb://mongos1:27017,mongos2:27017/")

Load testing

You can run load tests for the MongoDB cluster using the loadtest app.

Start 3 loadtest instances on the mongos network:

docker stack deploy -c swarm-loadtest.yml lt

The loadtest app is a Go web service that connects to the two Mongos nodes and does an insert and select:

http.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
    session := s.Repository.Session.Copy()
    defer session.Close()

    log := &AccessLog{
        Timestamp: time.Now().UTC(),
        UserAgent: string(req.Header.Get("User-Agent")),
    }

    c := session.DB("test").C("log")

    err := c.Insert(log)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    logs := []AccessLog{}

    err = c.Find(nil).Sort("-timestamp").Limit(10).All(&logs)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    b, err := json.MarshalIndent(logs, "", "  ")
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    w.Header().Set("Content-Type", "application/json")
    w.Write(b)
})

The loadtest service is exposed on the internet on port 9999. You can run the load test using rakyll/hey or Apache bench.

#install hey
go get -u github.com/rakyll/hey

#do 10K requests 
hey -n 10000 -c 100 -m GET http://<SWARM-PUBLIC-IP>:9999/

While running the load test you could kill a Mongos, Data and Config node and see what's the failover impact.

Running the load test with a single loadtest instance:

Summary:
  Total:	58.3945 secs
  Slowest:	2.5077 secs
  Fastest:	0.0588 secs
  Average:	0.5608 secs
  Requests/sec:	171.2490
  Total data:	8508290 bytes
  Size/request:	850 bytes

Response time histogram:
  0.304 [1835]	|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.549 [3781]	|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.793 [2568]	|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  1.038 [1153]	|∎∎∎∎∎∎∎∎∎∎∎∎
  1.283 [400]	|∎∎∎∎

Running the load test with 3 loadtest instances:

Summary:
  Total:	35.5129 secs
  Slowest:	1.9471 secs
  Fastest:	0.0494 secs
  Average:	0.3223 secs
  Requests/sec:	281.5877
  Total data:	8508392 bytes
  Size/request:	850 bytes

Response time histogram:
  0.239 [5040]	|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.429 [2358]	|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.619 [1235]	|∎∎∎∎∎∎∎∎∎∎
  0.808 [741]	|∎∎∎∎∎∎
  0.998 [396]	|∎∎∎

Scaling up the application from one instance to three instances made the load test 23 seconds faster and the requests per second rate went from 171 to 281.

Monitoring with Weave Scope

Monitoring the load test with Weave Cloud shows how the traffic is being routed by the Docker Swarm load balancer and by the Mongos instances:

Traffic

Weave Scope is a great tool for visualising network traffic between containers and/or Docker Swarm nodes. Besides traffic you can also monitor system load, CPU and memory usage. Recording multiple load test sessions with Scope you can determine what's the maximum load your infrastructure can take without a performance degradation.

Monitoring a Docker Swarm cluster with Weave Cloud is as simple as deploying a Scope container on each Swarm node. More info on installing Weave Scope with Docker can be found here.

Local deployment

If you want to run the MongoDB cluster on a single Docker machine without Docker Swarm mode you can use the local compose file. I use it for debugging on Docker for Mac.

$ docker-compose -f local-compose.yml up -d

This will start all the MongoDB services and mongo-bootstrap on the bridge network without persistent storage.

mongo-swarm's People

Contributors

stefanprodan avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mongo-swarm's Issues

mongorestore issue

Hello

I have launched a client instance
docker run --network mongos -it mongo:3.4 mongo mongos1:27017

and from there I am trying to restore one of my database but mongorestore is not working and giving me this error

2018-02-07T16:07:53.369+0000 Failed: error connecting to db server: no reachable servers

would you please tell me how I can restore database in mongo swarm

fail to build stefanprodan/mongo-bootstrap:latest image

hi, I modify the code of bootstrap, then build a new image base on Dockfile, but when I do it, it happens to occur these error, can you help me ?
2017-12-21_201255

I review your code in bootstarp, It happen to support only one shard, ie, datars shard replSet, can you modify the go code to support multiple shard, for example, I can write shard config as
2017-12-21_201208

last, how to add Authenticate MongoDB shard cluster in Docker swarm, can you tell me in details, looking forward to your reply?

Bootstrap not working for latest mongo image

Hi,

I know this repo hasn't been updated in 2 years so this is a bit of a reach. The mongo bootstrap image does not work for any mongo image above version 3.4. I tried deploying the stack with version 3.6 and also 4.0 but it's unable to find the 2 mongo servers (mongs1 and mongos2 after initializing the config and data services.

mongo_bootstrap.1.kjqj8sfndjtz@ip-10-50-1-39 | time="2019-06-21T12:24:21Z" level=warning msg="mongos1:27017 is offline retrying in 5 seconds" mongo_bootstrap.1.msxdw6cg7v5r@ip-10-50-1-39 | time="2019-06-21T12:07:14Z" level=warning msg="mongos1:27017 is offline retrying in 5 seconds" mongo_bootstrap.1.ibwky8in8ucy@ip-10-50-1-39 | time="2019-06-21T12:40:24Z" level=warning msg="mongos1:27017 is offline retrying in 5 seconds" mongo_bootstrap.1.kjqj8sfndjtz@ip-10-50-1-39 | time="2019-06-21T12:24:32Z" level=warning msg="mongos1:27017 is offline retrying in 5 seconds" mongo_bootstrap.1.msxdw6cg7v5r@ip-10-50-1-39 | time="2019-06-21T12:07:19Z" level=fatal msg="%!m(string=mongos1:27017)ongos connection failed" mongo_bootstrap.1.ibwky8in8ucy@ip-10-50-1-39 | time="2019-06-21T12:40:35Z" level=warning msg="mongos1:27017 is offline retrying in 5 seconds" mongo_bootstrap.1.ibwky8in8ucy@ip-10-50-1-39 | time="2019-06-21T12:40:45Z" level=warning msg="mongos1:27017 is offline retrying in 5 seconds" mongo_bootstrap.1.kjqj8sfndjtz@ip-10-50-1-39 | time="2019-06-21T12:24:42Z" level=warning msg="mongos1:27017 is offline retrying in 5 seconds" mongo_bootstrap.1.kjqj8sfndjtz@ip-10-50-1-39 | time="2019-06-21T12:24:53Z" level=warning msg="mongos1:27017 is offline retrying in 5 seconds" mongo_bootstrap.1.ibwky8in8ucy@ip-10-50-1-39 | time="2019-06-21T12:40:56Z" level=warning msg="mongos1:27017 is offline retrying in 5 seconds" mongo_bootstrap.1.ibwky8in8ucy@ip-10-50-1-39 | time="2019-06-21T12:41:06Z" level=warning msg="mongos1:27017 is offline retrying in 5 seconds" mongo_bootstrap.1.kjqj8sfndjtz@ip-10-50-1-39 | time="2019-06-21T12:24:58Z" level=fatal msg="%!m(string=mongos1:27017)ongos connection failed" mongo_bootstrap.1.ibwky8in8ucy@ip-10-50-1-39 | time="2019-06-21T12:41:17Z" level=warning msg="mongos1:27017 is offline retrying in 5 seconds" mongo_bootstrap.1.ibwky8in8ucy@ip-10-50-1-39 | time="2019-06-21T12:41:27Z" level=warning msg="mongos1:27017 is offline retrying in 5 seconds" mongo_bootstrap.1.ibwky8in8ucy@ip-10-50-1-39 | time="2019-06-21T12:41:38Z" level=warning msg="mongos1:27017 is offline retrying in 5 seconds" mongo_bootstrap.1.ibwky8in8ucy@ip-10-50-1-39 | time="2019-06-21T12:41:48Z" level=warning msg="mongos1:27017 is offline retrying in 5 seconds" mongo_bootstrap.1.ibwky8in8ucy@ip-10-50-1-39 | time="2019-06-21T12:41:59Z" level=warning msg="mongos1:27017 is offline retrying in 5 seconds" mongo_bootstrap.1.ibwky8in8ucy@ip-10-50-1-39 | time="2019-06-21T12:42:09Z" level=warning msg="mongos1:27017 is offline retrying in 5 seconds" mongo_bootstrap.1.ibwky8in8ucy@ip-10-50-1-39 | time="2019-06-21T12:42:20Z" level=warning msg="mongos1:27017 is offline retrying in 5 seconds" mongo_bootstrap.1.ibwky8in8ucy@ip-10-50-1-39 | time="2019-06-21T12:42:30Z" level=warning msg="mongos1:27017 is offline retrying in 5 seconds" mongo_bootstrap.1.ibwky8in8ucy@ip-10-50-1-39 | time="2019-06-21T12:42:35Z" level=fatal msg="%!m(string=mongos1:27017)ongos connection failed"

It'll be nice to get a pointer as to how to fix this (or maintain the repo) if you don't have enough time to get around to fixing it yourself

Thanks

Shard Configuration

I was looking thorugh the docs and I didn't see how you can specify different shards on the data configuration?

Dynamic Services?

Can it be possible to make it dynamic?
For example :

.....
  data:
    image: mongo:3.4
    networks:
      - mongo
    volumes:
      - data:/data/db
    command: mongod --shardsvr --replSet datars --smallfiles --port 27017
    deploy:
      mode: replicated
      replicas: 3
      placement:
        constraints:
          - node.labels.mongo.role == data

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.