Giter Club home page Giter Club logo

node-tiny's Introduction

node-tiny

node-tiny is an in-process document/object store for node.js.

It is largely inspired by nStore, however, its goal was to implement real querying which goes easy on the memory.

It supports mongo-style querying, or alternatively a mapreduce-like interface similar to CouchDB's views (this isn't needless fluff by the way, the function is used internally).


It's called node-tiny because I challenged myself to keep it to a single JS file with no dependencies. I also wanted it to be simple: There are no schemas and nothing you need to know to get started, just store your data: a portable database that you can drag around in a single file.

Install

$ npm install tiny

How Tiny works...

Tiny takes advantage of the fact that, normally, when you query for records in a database, you're only comparing small properties (<128b) in the query itself. For example, when you query for articles on a weblog, you'll usually only be comparing the timestamp of the article, the title, the author, the category, the tags, etc. - pretty much everything except the content of the article itself.

Tiny stores each document/object's property individually in the DB file and caches all the small properties into memory when the DB loads, leaving anything above 128b behind. When a query is performed, Tiny only lets you compare the properties stored in memory, which is what you were going to do anyway. Once the query is complete, Tiny will perform lookups on the FD to grab the large properties and put them in their respective objects before results are returned to you.

This my attempt at combining what I think the best aspects of nStore and node-dirty are. node-dirty is incredibly fast and simple (everything is in-memory), and nStore is very memory effecient, (but this only lasts until you perform a query). node-tiny allows for queries that perform lookups on the db file, and it selectively caches properties as well, so its fast and easy on memory.

The benefits you receive from using node-tiny depend on the kind of data you're working with. With the blog example mentioned above, if you consider that the metadata for a blog post may be as little as 200 bytes, a half-million articles would use less than 100mb of memory.

Example Querying

var Tiny = require('./tiny');
Tiny('articles.tiny', function(err, db) {
  var time = Date.now()
    , low = time - (60*60*1000)
    , high = time - (30*60*1000);

  // mongo-style query
  db.find({$or: [ 
    { timestamp: { $lte: low } }, 
    { timestamp: { $gte: high } }  
  ]})
  .desc('timestamp')
  .limit(3)(function(err, results) {
    console.log('Results:', results);
  });

  // is equivalent to...
  db.fetch({
    desc: 'timestamp',
    limit: 3
  }, function(doc, key) {
    if (doc.timestamp <= low 
        || doc.timestamp >= high) {
      console.log('Found:', key); 
      return true;
    }
  }, function(err, results) {
    console.log('Results:', results);
  });
});

The mongo-style querying should be fairly self-explanatory. The second query is supposed to be similar to a mapreduce interface, but it's the rough equivalent of a .filter function.

Note: there is a shallow parameter for .fetch, .find, and .get, wherein it will only lookup properties that are under 128b in size. This is to go easy on the memory. .each and .all are shallow by default, but they do have a deep parameter, (which I don't recommend using).

You can configure the limit at which properties are no longer cached by calling Tiny.limit, which accepts a number of bytes. e.g. Tiny.limit(1024);

Other Usage

// save a document
db.set('myDocument', {
  title: 'a document',
  content: 'hello world'
}, function(err) {
  console.log('set!');
});

// .each will iterate through
// every object in the database
// it is shallow by default
db.each(function(doc) { 
  console.log(doc.title);
});

// returns every object in the DB
// in an array, this is shallow
// by default
db.all(function(err, docs) {
  console.log(docs.length);
});

// remove a doc
db.remove('myDocument', function(err) {
  console.log('deleted');
}); 

// retrieve an object from the database
db.get('someOtherThing', function(err, data) {
  // data._key is a property which 
  // holds the key of every object
  console.log('found:', data._key); 
});

// updates the object 
// without overwriting its other properties
db.update('article_1', { 
  title: 'new title'
}, function(err) {
  console.log('done');
});

// close the file descriptor
db.close(function(err) {
  console.log('db closed');
});

// clean up the mess
db.compact(function(err) {
  console.log('done');
});

// dump the entire database to a JSON file
// in the same directory as the DB file
// (with an optional pretty-print parameter)
db.dump(true, function(err) {
  console.log('dump complete');
});

Making data more efficient

Because of the way Tiny works, there are ways to alter your data to make it more memory efficient. For example, if you have several properties on your objects that aren't necessary to for queries, its best to nest them in an object.

user: {
  name: 'joe',
  prop1: 'data',
  prop2: 'data',
  prop3: 'data'
}

user: {
  name: 'joe',
  data: {
    prop1: 'data',
    prop2: 'data',
    prop3: 'data'
  }
}

That way, the data will not be cached if it exceeds 128b collectively. Eventually there may be an ignore method or an index method, which will be explicitly inclusive or exclusive to which properties are cached and which properties are able to be referenced within a query.

Documentation

Database

Querying

Database

### Tiny(name, callback) Creates and returns a database with the given name.

Arguments

  • name - filename to store and load the Tiny database
  • callback(err, db) - Called after the database file is opened and loaded

Example

var tinydb;
Tiny('articles.tiny', function(err, db) {
    tinydb = db;
});

### dump(pretty, func) or dump(func) Dumps the a database to a JSON file with the name as name.json. Pretty specifies whether to indent each line with two spaces or not. Alternatively, dump(func) can be called.

Arguments

  • pretty - if true, the JSON file will be indented with two spaces
  • func(err) - called after the dump is complete.

Example

db.dump(true, function(err) {
  console.log('dump complete');
});

### close(func) Closes the Tiny database file handle. A new Tiny object must be made to reopen the file.

Arguments

  • func() - callback function after the database has been closed

Example

db.close(function(err) {
  console.log('db closed');
});

### kill(func) Closes the Tiny database file, deletes the file and all the data in the database, and then creates a new database with the same name and file.

Arguments

  • func() - callback function after the database has been reloaded

Example

db.kill(function(err) {
  console.log('db has been destroyed and a new db has been loaded');
});
### set(docKey, doc, func) Saves a object `doc` to database under the key `docKey`. Ideally, docKey should be 128b or smaller.

Arguments

  • docKey - a key to search the database for
  • doc - an object to save to the database under the given key
  • func - callback function after the doc object has been saved to the database

Example

db.set('myDocument', {
  title: 'a document',
  content: 'hello world'
}, function(err) {
  console.log('set!');
});

### each(func, deep) or each(func) Iterates through every object in the database.

Arguments

  • func(doc) - Callback function that is called with every iterated object doc from the database
  • deep - true if every object should be returned, false or unset if only cacheable objects should be returned (ones smaller than 128b)

Example

db.each(function(doc) {
  console.log(doc.title);
});

### all(func, deep) or all(func) Returns an array containing all the objects in the database. By default, it will only return cacheable objects (ones smaller than 128b).

Arguments

  • func(err, docs) - Callback function called with docs, the entire array of objects from the database
  • deep - true if every object should be returned, false or unset if only cacheable objects should be returned (ones smaller than 128b)

Example

db.all(function(err, docs) {
  console.log(docs.length);
});

See LICENSE (MIT).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.