Giter Club home page Giter Club logo

pioardi / ring-election Goto Github PK

View Code? Open in Web Editor NEW
113.0 12.0 6.0 3.67 MB

A node js library with a distributed leader/follower algorithm ready to be used

Home Page: https://medium.com/@alessandroardizio94/how-to-build-high-available-fault-tolerant-services-in-node-js-25a3785b2e46

License: MIT License

Dockerfile 1.01% JavaScript 95.95% Shell 3.03%
partitioning distributed-systems leader-election nodejs concurrency scalability leader partition leader-node ring

ring-election's Introduction

Ring election ✨ 💫

Is your dream to build a service like cassandra,kafka,zipkin,jaeger,redis,etc...? You are in the right place , join ring-election project !!!

Coverage Status Actions Status Actions Status Codacy Badge npm version Gitter chat
JavaScript Style Guide

Contents

What the ring-election driver offers you ?

  • A default partitioner that for an object returns the partition to which it is assigned.
  • Mechanism of leader election
  • Failure detection between nodes.
  • Assignment and rebalancing of partitions between nodes
  • Automatic re-election of the leader
  • Listen for new assigned/revoked partitions

What problems can you solve with this driver ?

  • Scalability
  • High availability
  • Concurrency between nodes in a cluster
  • Automatic failover

Getting started

Install with npm !
  npm i ring-election --save

Example You do not need to choose a node as leader , just indicate all your nodes and start everyone as follower.
The first node to start will be the leader , the leader do not have assigned partitions so try to start 2 instances after your integration

How to integrate

const ring = require('ring-election')
let follower = ring.follower
const {
  BECOME_LEADER,
  PARTITIONS_ASSIGNED,
  PARTITIONS_REVOKED
} = ring.constants;
follower.createClient()
// if you want REST API as monitoring , invoke startMonitoring
follower.startMonitoring()
// to get ring info
ring.follower.ring()
// to get assigned partitions
let assignedPartitions = ring.follower.partitions()
// now let me assume that a follower will create some data
// and you want to partition this data
let partition = ring.follower.defaultPartitioner('KEY')
// save your data including the partition on a storage
// you will be the only one in the cluster working on the partitions assigned to you.

// If you want to handle partitions assigned
// ( use other constants to listen other events ) you can do in this way.
ring.follower.eventListener.on(PARTITIONS_ASSIGNED , (newAssignedPartitions) => {
   // DO STUFF
})

Start your development cluster

You will find some helpful files into the dev folder , please see the following video

Watch the video

Check assigned partitions to local:9000/status or change the port to 9001/9002

Try to stop and restart processes and observe the behaviour.

Overview and rationale

In modern systems it is often needed to distribute the application load to make the system scalable so that every data is processed by a single instance.
Ring-election is a driver that implements a distributed algorithm that assigns to each node the partitions to work on . In a simple use case each node can obtain data that are part of the partitions of which it is owner and work on them.
The algorithm will assign to each node one or more partitions to work with.
A node will be removed if it does not send an heart beat for a while , this process is called heart check.
Each node in the ring will have an ID and a priority , if the leader node will die the node with lower priority will be elect as leader.
If a node is added or removed from the cluster, the allocated partitions will be rebalanced.

Use cases

This section introduce you on what you can build on top of ring-election using it as driver/library.

Distributed Scheduler
Each Scheduler instance will work on the assigned partitions .
A real implementation of this use case is available here https://github.com/pioardi/hurricane-scheduler
Dynamic diagram

Distributed lock
Distributed cache
Distributed computing

Try it out !

   docker image build -t ring-election .
   docker-compose up

Configuration

PORT : The leader will start to listen on this port , default is 3000
TIME_TO_RECONNECT: The time to wait for a follower when he has to connect to a new leader in ms , default is 3000ms
HEART_BEAT_FREQUENCY: The frequency with which a heart beat is performed by a follower , default is 1000ms
HEART_BEAT_CHECK_FREQUENCY: The frequency with which an heart check is performed by a leader , default is 3000ms
LOG_LEVEL: Follow this https://www.npmjs.com/package/winston#logging-levels , default is info.
NUM_PARTITIONS: Number of partitions to distribute across the cluster , default is 10.
SEED_NODES : hostnames and ports of leader node comma separated, Ex . hostname1:port,hostname2:port
MONITORING_PORT : port to expose rest service for monitoring , default is 9000

Monitoring API

To monitor your cluster contact any node on the path /status (HTTP verb : GET) or contact a follower node on /partitions (HTTP verb : GET).

High Level Diagram

See wiki page.

How to contribute

See contributing guidelines CONTRIBUTING

Versioning

We use (http://semver.org/) for versioning.

License

This project is licensed under the MIT License - see the [LICENSE](./LICENSE) file for details

ring-election's People

Contributors

dependabot-preview[bot] avatar dependabot-support avatar pioardi avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ring-election's Issues

Feedbacks

Give feedbacks on the documentation and on the project in general

Build a distributed data structure

Implement into a separated repo a distributed data structure ( i.e. maps or array ) to show everyone what is possible to do with ring-election

Mesh

Hi,

I was thinking about putting with external facing servers instead of being local servers and then I can focus on firewall rules to make sure only those servers are ok for talking between each other.

I was wondering what do you think about this approach and if you think it's possible?

Great initiative.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.