Giter Club home page Giter Club logo

memored's Introduction

Memored

Memored implements an in-memory shared cache to use in nodejs applications which uses cluster module.

Let's say you want your application to take advantage of multi-core CPUs using nodejs cluster module; you will be able to run several isolated processes which shared nothing but a communication channel with parent process. If you need a fast volatile cache, common solutions would create an in-memory map for every process you run, so you end up with the same data stored several times in your machine RAM.

Memored uses communication channel between master process and its workers to use a unique in-memory storage, reducing the amount of memory your application would use.

Getting Started

Install this module with npm:

npm install memored

Store and read values is straightforward:

var cluster = require('cluster'),
	memored = require('memored');

if (cluster.isMaster) {
	cluster.fork();
} else {
	var han = {
			firstname: 'Han',
			lastname: 'Solo'
		},
		luke = {
			firstname: 'Luke',
			lastname: 'Skywalker'
		};

	// Store and read
	memored.store('character1', han, function() {
		console.log('Value stored!');

		memored.read('character1', function(err, value) {
			console.log('Read value:', value);
		});
	});

	// You can also set a ttl (milliseconds)
	memored.store('character2', luke, 1000, function(err, expirationTime) {
		console.log('Value stored until:', new Date(expirationTime));

		setTimeout(function() {
			memored.read('character2', function(err, value) {
				console.log('Value is gone?', value === undefined);

				process.exit();
			});
		}, 1050);
	});
}

Invalidation management

By default, memored will evict cache entries (stored with ttl) passively. This is, when you read an expired entry, you will get no value on return and memored will delete the value from its internal cache.

You can also configure memored to actively evict expired entries every N milliseconds. For this to work, you need to pass the attribute purgeInterval to the setup function. This will trigger an internal function which looks for expired entries and deletes them from its internal cache.

Example:

var cluster = require('cluster'),
	async = require('async'),
	memored = require('memored');

if (cluster.isMaster) {

	cluster.fork();
	memored.setup({ purgeInterval: 500});

} else {

	async.series({
		storeValue: function(next) {
			memored.store('key1', 'My simple string value', 100, next);
		},
		readCacheSize: function(next) {
			memored.size(function(err, size) {
				console.log('Current size is 1?', size === 1);
				next();
			});
		},
		wait: function(next) {
			setTimeout(next, 600);
		},
		readCacheSizeAgain: function(next) {
			memored.size(function(err, size) {
				console.log('Current size is 0?', size === 0);
				next();
			});
		}
	}, process.exit);
}

API

Documentation for every module function:

setup(options)

This function is used to configure memored.

Arguments:

  • purgeInterval {Number} (optional): Configures and triggers memored expired entries auto deletion. Value expressed in milliseconds. It's only used when called this method from the master process of your application.
  • logger {Object} (optional): In you want memored to log something, you must provide an object implementing log and warn functions.

Example:

memored.setup({
	purgeInterval: 15000,
	logger: console
});

store(key, value, [ttl], [callback])

This function stores a value in the cache. It is intended to be called from a worker process.

Arguments:

  • key {String} (required): Key used to lookup the entry
  • value {Mixed} (required): Whatever you want to store
  • ttl {Number} (optional): Time to live for this value in the cache (milliseconds)
  • callback {Function} (optional): Function to be call on store completion. Callback arguments:
    • err {Error}: Optional error
    • expirationTime {Number}: The timestamp of the moment when this entry will expire. If ttl is not used, this value will be undefined.

Examples:

memored.store('key1', {firstname: 'Han', lastname: 'Solo'}, function() {
	console.log('Value stored!');
});

memored.store('key2', ['a', 'b', 'c'], 15000, function(err, expirationTime) {
	console.log('This value will expire on:', new Date(expirationTime));
});

multiStore(map, [ttl], [callback])

This function stores several values in the cache. It is intended to be called from a worker process.

Arguments:

  • map {Object} (required): Map where the keys represents the keys for the entry in the cache and the values represent the data to be stored.
  • ttl {Number} (optional): Time to live for this value in the cache (milliseconds). All the entries will have the same ttl. As all entries will be stored in the same tick, its expiration time will be practically the same.
  • callback {Function} (optional): Function to be call on store completion. Callback arguments:
    • err {Error}: Optional error
    • expirationTime {Number}: The timestamp of the moment when the first of the entries will expire. If ttl is not used, this value will be undefined.

Examples:

var users = {
    'user1': { name: 'Han Solo' },
    'user2': { name: 'Princess Leia' },
    'user3': { name: 'Luke Skywalker' }
};
memored.multiStore(users, function() {
    console.log('Users saved');
});

memored.multiStore(users, 15000, function(err, expirationTime) {
    console.log('First value will expire on:', new Date(expirationTime));
});

read(key, callback)

This function reads a value from the cache. It is intended to be called from a worker process.

Arguments:

  • key {String} (required): Key used to lookup the entry
  • callback {Function} (required): Function to be called on read completion. Callback arguments:
    • err {Error}: Optional error
    • value {Mixed}: Contents of the cached entry. If the value is not found or is expired, it will be undefined.

Example:

memored.read('key1', function(err, value) {
	console.log('Key1 value:', value);
});

memored.read('key1', function(err, value, expirationTime) {
	console.log('Key1 value:', value);
	console.log('Key1 expiration time:', new Date(expirationTime));
});

memored.read('unknownKey', function(err, value) {
	console.log('No data read?', value === undefined);
});

multiRead(keys, callback)

This function reads several values from the cache. It is intended to be called from a worker process.

Arguments:

  • keys {Array(string)} (required): List of keys to lookup entries in the cache
  • callback {Function} (required): Function to be called on read completion. Callback arguments:
    • err {Error}: Optional error
    • values {Object}: An object where its keys will be the keys used in the keys array and their values will be objects representing cached entries with the attributes value and expirationTime. If a cache entry is not found for a given key, that key will not be included in the values. Only found entries will exist in the result.

Example;

memored.multiRead(['key1', 'key2', 'unknownKey'], function(err, values) {
    console.log('Key1 value:', values.key1.value);
    console.log('Key1 expiration time:', values.key1.expirationTime);
    
    console.log(Object.keys(values)); // ['key1', 'key2']
    
    console.log('unknownKey:', values.unknownKey); // undefined
});

remove(key, callback)

This function removes an entry from the cache. It is intended to be called from a worker process.

Arguments:

  • key {String} (required): Key for the entry to be removed.
  • callback {Function} (optional): Function to be called on removal completion.

Example:

memored.remove('key1', function() {
	console.log('Key removed from the cache.');
});

multiRemove(keys, callback)

This function removes several entries from the cache. It is intended to be called from a worker process.

Arguments:

  • keys {Array(string)} (required): Keys for the entries to be removed. If any key is not found in the cache, it's just ignored.
  • callback {Function} (optional): Function to be called on removal completion.

Example:

memored.multiRemove(['key1', 'key2', 'unknownKey'], function() {
    console.log('Entries foundn in the cache has been removed.')
});

clean(callback)

This function removes all the entries from the cache. It is intended to be called from a worker process.

Arguments:

  • callback {Function} (optional): Function to be called on read completion.

Example:

memored.clean(function() {
	console.log('All cache entries have been deleted.');
});

size(callback)

This function returns the number of entries in the cache.

Arguments:

  • callback {Function} (required): Function to be called on size calculation is complete. Callback arguments:
    • err {Error}: Optional error
    • size {Number}: The number of entries in the cache.

Example:

memored.size(function(err, size) {
	console.log('Cache size:', size);
});

keys(callback)

This function returns an array of the keys for objects in the cache.

Arguments:

  • callback {Function} (required): Function to be called when keys calculation is complete. Callback arguments:
    • err {Error}: Optional error
    • keys {Array}: An array of strings for the keys of the entries in the cache.

Example:

memored.keys(function(err, keys) {
	console.log('Cache keys:', keys);
});

### version

This is an attribute which provides module's version number

Final note

All the callbacks first parameter is an optional error object. Actually, this param will never be an error because there is no expected error in the internal code. There's no function call that can possible throw an expected error that this module would deal with. The existence of this param is to follow the convention about libraries callbacks in nodejs. As everybody expects this first callback parameter to be an optional one, I decided to include it.

License

Copyright (c) 2014 PaquitoSoft
Licensed under the MIT license.

memored's People

Contributors

paquitosoft avatar uzquiano avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

memored's Issues

Typescript-file

I created a Typescript-Definition file .. so maybe you can add it in your lib (https://stackoverflow.com/questions/37210059/how-to-add-a-typescript-definition-file-to-a-npm-package).

declare module "memored" {

    // https://github.com/PaquitoSoft/memored#setupoptions
    export interface IMemoredLogger {
        log: (message: string) => any
        warn: (message: string) => any
    }
    function setup<T extends IMemoredLogger>(options?: { purgeInterval?: number, logger?: T }): void

    // https://github.com/PaquitoSoft/memored#storekey-value-ttl-callback
    function store(key: string, value: any, ttl: number, cb?: (err: Error, expirationTime: number) => any): void
    function store(key: string, value: any, cb?: (err: Error | null, expirationTime: number) => any): void

    // https://github.com/PaquitoSoft/memored#multistoremap-ttl-callback
    interface IMultiKeyValue {
        [index: string]: any
    }
    function multiStore(IMultiKeyValue: IMultiKeyValue, ttl: number, cb?: (err: Error
                                                                               | null, expirationTime: number) => any): void
    function multiStore(IMultiKeyValue: IMultiKeyValue, cb?: (err: Error | null, expirationTime: number) => any): void

    // https://github.com/PaquitoSoft/memored#readkey-callback
    function read(key: string, cb: (err: Error | null, value: any) => any): void

    // https://github.com/PaquitoSoft/memored#multireadkeys-callback
    function multiRead(keys: Array<string>, cb: (err: Error | null, values: IMultiKeyValue) => any): void

    // https://github.com/PaquitoSoft/memored#removekey-callback
    function remove(key: string, cb?: () => any): void

    // https://github.com/PaquitoSoft/memored#multiremovekeys-callback
    function multiRemove(keys: Array<string>, cb?: () => any): void

    // https://github.com/PaquitoSoft/memored#cleancallback
    function clean(cb?: () => any): void

    // https://github.com/PaquitoSoft/memored#sizecallback
    function size(cb: (err: Error | null, size: number) => any): void

    // https://github.com/PaquitoSoft/memored#keyscallback
    function keys(cb: (err: Error | null, keys: Array<string>) => any): void
}

questions

Hi , I liked this project a lot. couple of questions:

  1. where do you save the data? on the main process memory or on the disk?
  2. is it possible to work with it without cluster? i.e only with main process?

Thanks,
Guy

Just a comment

I am writing this comment to commend the writer of this module.

It is really good for clusters management in Node.js. It scales so well to several applications.

  1. I have been using it with mongoDB and it gives excellent in-memory session management.
  2. I have also used it to handle data manipulation in a real-time "scientific/numeric computation" application.

It is very small and compact.

I have been using it for over 4 months now and I can't believe the excellent performance I am getting from it, especially with well structured IIFE codes (https://en.wikipedia.org/wiki/Immediately-invoked_function_expression).

I hope the author can write a small book on this module to let people know of its functionalities.

Question about master

Is there any reason the master cant have access to the cache data (or even the cache itself)? It looks like the cache will return keys and can be cleared via reset(). I was looking to show the cache contents (from an express server running in the master) just for status/debugging, but the best i can do is show the keys. Thanks!

Other data structures

Hey there,

I was searching for a way to share data between cluster workers and found your project.

I don't want to increase the complexity of my architecture by adding Redis as a dependency, but I need support to more advanced data structures though, like lists maps, etc.

Do you plan to support those in your library? Cheers.

Storing keys from master

Is it possible to store keys in the master and have child processes access the data? I have a long running process I only want run as one instance, and in a multi-cpu server I would have that process running multiple times unnecessarily consuming resources.

High Load Server

Hello everyon. Could you tell me, how memored work with high load servers?

The load of my server equal 1 billion request per day. How I understand first I have to add require('memoried') to master script, then I have to add it to workers lib.

I think Server Master could drop under pressure of traffic.

Several domains

Is there a way, to add several domains of cache?

let memored = require('memored')

let cache1 = new memored();
cache1.setup({ domain: "ipV6", purgeInterval: 3 * 60 * 60 * 1000});

let cache2 = new memored();
cache2.setup({ domain: "ipV4", purgeInterval: 24 * 60 * 60 * 1000}});

why i cant add below object?

{

    DefaultFlow: require('./shared/defaultFlow/defaultFlow'),
    DefaultFlowChild: require('./shared/defaultFlow/defaultFlowChild'),
    User: require('./auth/user'),
    Task: require('./shared/Task/task'),
    Process: require('./shared/Task/process'),
    Workflow: require('./shared/Task/workflow'),
    Message: require('./shared/Message/message'),
    Mail: require('./official/mail/mail'),
    Fat: require('./technical/panel/fat'),

}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.