kikobeats / farm-cli Goto Github PK

View Code? Open in Web Editor NEW

49.0 4.0 4.0 130 KB

Resilient multi-process architecture from your CLI

Home Page: https://nicedoc.io/Kikobeats/farm-cli

License: MIT License

JavaScript 100.00%

farm-cli's Introduction

farm-cli

farm allows allows you to create a resilient multi-process architecture from your CLI. It's based on worker-farm.

Note: See examples for common user case.

Installation

First, install the library globally:

$ npm install farm-cli --global

Additionally, you can invoke it using npx:

$ npx farm-cli

Getting Started

Basic Usage

You need to specify a file and export a main method as entry point.

const createLog = n => (...args) => console.log(`[#${n}] ${args}`)

module.exports = function (opts, exit) {
  const { isMaster, maxWorkers, worker } = opts
  const log = createLog(worker)
  log(`I'm worker ${worker + 1} of ${maxWorkers} ${isMaster ? '(master)' : ''}`)
}

Finally, invoke the file using farm:

$ farm examples/basic

[#0] Hello I'm worker 1 of 4 (master)
[#1] Hello I'm worker 2 of 4
[#2] Hello I'm worker 3 of 4
[#3] Hello I'm worker 4 of 4

Creating Your Own Farm

By default, the library will create as many processes as number of CPUs in the machine and a thread per process.

Let's use the same example but this time specifying multiple threads per process

$ farm examples/basic --threads=2

[#0] Hello I'm worker 1 of 8 (master)
[#1] Hello I'm worker 2 of 8
[#2] Hello I'm worker 3 of 8
[#4] Hello I'm worker 4 of 8
[#5] Hello I'm worker 5 of 8
[#6] Hello I'm worker 6 of 8
[#7] Hello I'm worker 7 of 8
[#8] Hello I'm worker 8 of 8

This time the farm has 2 threads per process (2 threads * 4 cores = 8 workers).

Type farm --help to know more.

Passing File Arguments

The parameters passed after the filename will used as file arguments:

$ farm examples/fiboniacci --memoize
[#0] Enable memoize mode!

Load Configuration File

The same things passed to farm from a .farmrc file created in the same directory.

$ ls -al examples/load-config

-rw-r--r--@ 1  50 Jan 21  2017 .farmrc
-rw-r--r--@ 1 105 Nov 26  2017 README.md
-rw-r--r--@ 1  50 Jan 21  2017 index.js

Resilient Errors

If a process unexpectedly dies for any reason, it automagically re-enter.

This can be adjusted using --retries flag.

Type farm --help to know more.

Finishing the Execution

When you want to finish, call the second argument:

module.exports = function (opts, exit) {
  const { worker } = opts

  setTimeout(() => {
    console.log(`[#${worker}] bye bye!`)
    exit()
  })
}

It will finish gracefully when all the threads exit.

Debug Mode

If you need to debug, enable it passing DEBUG=farm as environment variable

DEBUG=farm examples/fiboniacci --memoize

License

farm-cli © Kiko Beats, released under the MIT License.
Authored and maintained by Kiko Beats with help from contributors.

kikobeats.com · GitHub Kiko Beats · Twitter @Kikobeats

farm-cli's People

Contributors

Stargazers

Watchers

Forkers

evergreen-travis evergreen-travis-bot

farm-cli's Issues

Use Node Native API

https://github.com/watson/worker-threads-pool

Goal

The idea is add/remove workers per core based in a CPU load medition.

Currently, only based on cpu load [1, 5, 15], using 15 mins value for be more predictable.

API

--autoscale
--autoscale-umbral=0.70
--autoscale-check=5min
--autoescale-quantity=2

Design

We need to meause the efficient every X times.

Efficient is num of CPU's / load average for 15min

Under this medition, we need to determinate the next action:

case 1

If the result is similar to the last medition, do nothing. We can do this for example if the new value is in a +-25% umbral related with the last medition.

case 2

Otherwise, we need to determinate the next action, add or remove based if the new value is higher or lower compared with the last medition.

Inspiration

The "Need to Look into it" Rule of Thumb: 0.70 If your load average is staying above > 0.70, it's time to investigate before things get worse.

https://github.com/remy/nodemon#controlling-shutdown-of-your-script

Action required: Greenkeeper could not be activated 🚨

🚨 You need to enable Continuous Integration on all branches of this repository. 🚨

To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because we are using your CI build statuses to figure out when to notify you about breaking changes.

Since we did not receive a CI status on the greenkeeper/initial branch, we assume that you still need to configure it.

If you have already set up a CI for this repository, you might need to check your configuration. Make sure it will run on all new branches. If you don’t want it to run on every branch, you can whitelist branches starting with greenkeeper/.

We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.

Once you have installed CI on this repository, you’ll need to re-trigger Greenkeeper’s initial Pull Request. To do this, please delete the greenkeeper/initial branch in this repository, and then remove and re-add this repository to the Greenkeeper integration’s white list on Github. You'll find this list on your repo or organiszation’s settings page, under Installed GitHub Apps.

Add web GUI

Inspired in github timeline, I want to create a status page that ping every X times to endpoints URL (following basic way using a http client or custom way, maybe using phantomjs or something similar) to detect fails system.