use node-webkit.

var db = new loki();

will throw error

Uncaught node.js Error 

ReferenceError: document is not defined
    at Loki.getENV (D:\workspace\radium\canvas\node_modules\lokijs\src\lokijs.js:228:15)
    at LokiEventEmitter.Loki (D:\workspace\radium\canvas\node_modules\lokijs\src\lokijs.js:237:18)

change source code in file lokijs.js:224 like this will be ok

   if (typeof exports === 'object' || typeof window === 'undefined') {

When using this on client side then a worker would the perfect place for it?

Agree?

This way any query etc would happen in its own thread, and not really block any other stuff

Does this lib persist records in browser or just in node.js?

Right now, the behavior for getCollection is to throw an error if the collection requested isn't present. I think it might be more convenient to return null if the collection isn't found. Then a user can test the value and initialize the collection if necessary. e.g.

var col = db.getCollection('test');
if(!col)
  col = db.addCollection('test')

Right now, there's no great way to know if a collection already exists before trying to get it.

If one accidentally tries to get using string id it does not throw but fails to find the document. This is due to isNaN returning false for string id so it is not cast to number, but later strict equality === causes id not to be found.

I found this particular strange bug. If you're using an index, when you update an item that already exists in the collection it appends a new item to the collection. This bug only occurs though if you're fetching on the index itself.

Here's a very basic minimal example:

λ node
> var loki = require('lokijs')
undefined
> var db = new loki('db.json')
undefined
>
undefined
> test = db.addCollection('test', ['index'])
{ name: 'test',
  data: [],
  idIndex: [],
  binaryIndices: { index: { name: 'index', dirty: false, values: [] } },
  objType: 'test',
  transactional: false,
  cachedIndex: null,
  cachedBinaryIndex: null,
  cachedData: null,
  maxId: 0,
  DynamicViews: [],
  events:
    { insert: [ [Function] ],
      update: [ [Function] ],
      close: [],
      flushbuffer: [],
      error: [],
      delete: [] } }
> item = test.insert({ index: 'key', a: 1 })
{ index: 'key',
  a: 1,
  objType: 'test',
  meta: { version: 0 },
  id: 1 }
 > test.find({ index: 'key' }).length
1
> item.a = 2
2
> test.update(item)
undefined
> test.find({ index: 'key' }).length
2

I'm expecting the length to be 1. If you inspect the collection returned it'll contain an item where a = 1 and another item where a = 2

I'm using the following code in node-webkit/angular:

app.service('Database', [
    function() {
        return function(dbName) {

            var self = this;

            this.initialized = false;
            this.db = new loki(dbName + '.json', {
                env: 'NODEJS',
                autoload: true
            });
            this.models = {};

            this.init = function(collections, callback) {
                if (fs.existsSync(dbName + '.json')) {
                    this.db.loadDatabase({}, function() {
                        initDb();
                    });
                } else {
                    initDb();
                }

                function initDb() {
                    collections.forEach(function(collection) {
                        self.db.addCollection(collection);
                        self.models[collection] = self.db.getCollection(collection);
                    });
                    self.initialized = true;
                    callback();
                }
            };

            this.save = function(callback) {
                self.db.saveDatabase(callback);
            };

            return this;
        };
    }
]);

This worked in 1.0, bar the API changes I've made for 1.1. It would load an empty database which would then allow it to save if it hadn't been created, now it just throws an error because it's trying to parse JSON that doesn't exist.

SyntaxError: Unexpected end of input
    at Object.parse (native)
    at LokiEventEmitter.Loki.loadJSON (C:\njs\arch\node_modules\lokijs\src\lokijs.js:396:22)
    at readFileCallback (C:\njs\arch\node_modules\lokijs\src\lokijs.js:677:16)
    at fs.js:289:14
    at Object.oncomplete (fs.js:97:15)

Edit: Your desktop_app demo throws the same error. Also if there's no file present, it throws an ENOENT error. Instead, it should probably create the file and initialize a database.

Whenever I use a for-loop to remove objects from a collection, loki skips every other reference and ends up deleting only half of the objects in the result array.

Although when I reverse the loop, it works as expected.
Using Node-webkit and lokijs 1.1.14

        var loki = require("lokijs")
        var db = new loki("test3");

        var folders = db.addCollection("folders");

        folders.insert({name : "data", type: "system"});
        folders.insert({name : "images", type: "user"});
        folders.insert({name : "datacache", type: "system"});
        folders.insert({name : "imagecache", type: "system"});
        folders.insert({name : "sample", type: "user"});
        folders.insert({name : "documents", type: "user"});
        folders.insert({name : "feeds", type: "user"});
        folders.insert({name : "site", type: "user"});

        db.saveDatabase();
        console.log(folders);


        // Load the database

        db2 = new loki("test3");
        db2.loadDatabase({}, function(){

            var folders = db2.getCollection("folders");
            console.log(folders);
            var list = folders.find();

            for(var i = 0; i < list.length; i++) {
                console.log(i);
                console.log(list[i]);

                folders.remove(list[i]);
            }


            // Reverse loop works fine though

            /*for(var z = list.length -1; z >= 0; z--) {
                console.log(z);
                console.log(list[z]);

                folders.remove(list[z]);
            }*/


        });

Gordian, Istanbul and Mocha are already listed in devDependencies, so they could be removed from ordinary dependencies? Now those dependencies bring in 10MB extra to ordinary install...

bulk inserts should work.

In a node env, load should create a file if it doesn't exist, so as to avoid err 34, especially when autoload is true.

Already developed in experimental branch, remains to be decided how key-value stores will relate to the db as opposed to collections.

This code from Collection.add:

if (isNaN(this.maxId)) {
    this.maxId = (this.data[this.data.length - 1].id + 1);
}

Does not work if the collection is empty and leads to this error: Cannot read property 'id' of undefined

How to reproduce:

• Load a database with a non-empty collection. The collections maxId is initialized with undefined: copyColl.maxId = (coll.data.length === 0) ? 0 : coll.data.maxId; (from loadJSON)
• Remove all entries from the collection
• Add an entry to the collection

I'm guessing the error is in the loadJSON method line above, since coll.data.maxId is never set. Plus the code in Collection.add basically assumes that some other code did set the maxId to 0 if the collection is empty.

It would be great if DynamicViews would also emit change events, but only if changes to the underlying collection are actually affecting the DynamicView.

Suppose I have a DynamicView whose data I'm feeding to an HTML template. It would be good if I could re-render my HTML template only if the DynamicView actually changes, instead of listening to every change event on the underlying collection. This would help avoid unnecessary template rendering.

Btw great work with LokiJS! I like that it's so small and easy to grasp compared to the other alternatives out there.

Docs say that collection asyncListeners defaults to false, but in code it defaults to true (even though comment there says otherwise). I was wondering why inserting a new doc results in its meta.created being 0 if inspected in the same control flow right after insert, this explains why.

Also collection option disableChangesApi defaults to true in code, which is a good default I guess, but is said to default to false in docs.

I found some new commit also use gordian instead of mocha, we should use just on test framework, and add more base unit test.

mocha is more popular, isn't it ?

Hello All!
First of all sorry for the presumptuous title. I do not know much about the internals of lokiJS to state how easy it is to introduce the simple changes I'm thinking about. LokiJS seems very promising and prompted me to think of stuff that would improve it -in my mind-. I very well might have had things wrong, but the easiest and best for everyone is to write down my thoughts and await your feedback.

Please find the time to go through this post to see what I'm getting at.

1- localStorage that's being used as storage engine is already an 'in memory database'. Hence its synchronous nature. Users do not have access to when the actual flush to disk' happens. One option is to avoid loading all data into memory -it's already in memory- and saving it back at the end. Rather one would work on localStorage entities directly (actually through a wrapper/interface, more on that later).This means that if your 'MyApp' database has two tables/entities Todo_Items and Activity_Type the localStorage would have two key-value entries: Todo_Items and Activity_Type instead of a single entry/file: MyApp. This would also mean some extra work to allow porting data around (it's no longer a single JSOn-file that maps to a single key-value entry)

2- One could instead of using localStorage by default feed the storage engine to lokiJS (there'a a simple interface to implement). Better yet, one could have different storage engines for different entities. Since every entity has its own key-value entry. Some entities can live in sessionStorage, others in localStorage or in memory (real in memory storage that would be re-read from the server on page refresh).

3- Going further, one can use something like localForage to also have the option of using indexedDB/WebSQL as storage engine. Many valid use cases would require more than 5 Mega of storage and also need to use different storage engines at once -for instance using sessionStorage to save sensitive data such as authentication tokens etc...-. Note that introducing localForage would also require us to change the interface to become asynchronous. For localStorage's implementation that would simply mean the promise/callback will be immediately invoked/resolved.

4- On the server side Loki should be very easy to port as one could use levelDB as storage engine. One should simply need to implement the 'interface' with levelDB. One important advantage of levelDB is that it'll be used by the same process as the nodejs instance (I assume you want to use nodejs). This eases scalling and deployment ...etc

I hope that made sense.
Cheers~.

Hello,

The project README mentions: "In a nodejs/node-webkit environment LokiJS also persists to disk whenever an insert, update or remove is performed.".

Is this accurate? I tried to look for this functionality in the source and it doesn't seem to be there - nor could I reproduce it with code. It looks like we have to explicitly call db.save() when we want an on-disk checkpoint created.

Am I missing something? If not, might be worth correcting.

Because fs.unlink is asynchronous, sometimes the filesystem will attempt to write to a file currently being deleted (just how it looks like to me, I'm no expert) and that results in no file at all (win 8).

fs.writeFile should be called either from fs.unlink's callback, or use fs.unlinkSync.

Ideally the exposed Collection class should also have the ability to be serialized and loaded.

This should be either at Collection level, or an optional parameter in the insert method. Most likely the former option.

Given the interest in lokijs' MongoDB compatibility it seems odd that there is no mention in the documentation as to what exactly is supported in terms of queries/methods.

Is there a one place where I can find all the currently supported queries?

To verify that this was an issue, I made a simple index.html file w/ the following code (taken from the docs)

    var db = new loki('test', { env: 'BROWSER'}),
            db2 = new loki('test', { env: 'BROWSER'});

    var users = db.addCollection('users');
    users.insert({
        name: 'joe'
    });
    users.insert({
        name: 'john'
    });
    users.insert({
        name: 'jack'
    });
    console.log(users.data);
    db.saveDatabase();

    db2.loadDatabase(function () {
        var users2 = db2.getCollection('users')
        console.log(users2.data);
    });

loadDatabase breaks in the following code snippet (which exists in two areas):

        if (localStorageAvailable()) {
          self.loadJSON(localStorage.getItem(this.filename));
          cFun(null, data);
        } else {
          cFun(new Error('localStorage is not available'));
        }

Data is undefined and an error is thrown. I've verified that removing data from the callback resolves the issue.

I find some design problem, it will make people confused.

After I insert a object into collection, the origin object is modified, just like:

var john = {name: 'John'};
users.insert(john);
console.log(john);  //{ name: 'John', objType: 'user', meta: { version: 0 }, id: 1 }

1. The insert() should clone a new object to insert.
2. An array should also be inserted, instead of inserting every item.
3. pass id to remove() instead of passing the whole obj
4. I think using some api like mongoose is better for update and remove :

var john = users.find({name:'John'});
john.age = 20;
console.log(john);  //{ name: 'John', age: 20, objType: 'user', meta: { version: 0 }, id: 1 }
john.save();
john.remove();

1. the obj save in db should distinguish the origin properties from Loki properties, the data may be better looks like:

{ data: {
    name: 'John',
    age: 20
  }, 
  objType: 'user', 
  meta: { version: 0 }, 
  id: 1 
} 

How about these 5 points ? I find it's complex for fix old unit test, so if modify these problems, I will make some new tests.

null would be consistent with MongoDB and would make testing whether the item is found easier.
This applies to v1. I have not tested v1.1

Could you tell me more about this, will it be a public function?
Want much about collections in docs

Is it like a set with id's only?
That you could filter without looking the original set?

In wakanda(wakanda.org, nosql) every set gets a unique entityset when u do a query.
This way you could just pass the unique ID to a method from client side to server side if you wanted to do work with on server side, this way you relaly dont need to pass a entire collection

Hi,

Do you plan to support mongo-like logical operators ?

For reference: http://docs.mongodb.org/manual/reference/operator/query-logical

remove and update, do they also support array like insert?

Use Case

I work on a project involving the MEAN stack (without A). This project is divided into RESTful API server(s) and Web UI Server(s). We needed a way to mock data access and instead of using SinonJS to mock $.ajax requests we have deemed simpler to mock data access with a second set of observable models getting data and setting data to an in-memory database.

Reasons for choosing LokiJS

We have quickly evaluated PouchDB, TaffyDB, Lawnchair, DB.js, jQuery.IndexedDB.js, minimongo, etc. Just from reading the documentation we felt that having an in-memory database that would match mongoDB apis would make things simpler and since minimongo requires Browserify, we tunred to LockiJS.

On the long run, we need a database to store JSON documents in a Cordova app that we want to synchronize with our RESTful server backed by MongoDB, and this QA/test mockup exercise seemed like a good way to evaluate LokiJS.

Disappointments with LokiJS

Basically, everything that is good and appealing in LokiJS pertains to its resemblance with MongoDB. The bad comes from the differences.

The main issue with LokiJS is ids. IMHO, renaming an id of type string (actually a MongoDB ObjectId converted to a string) into originalId to have an id of type number is a terrible option that leads to all sorts of headaches either in our mockup scenario and most certainly in synchronization scenarios. In other words, do not change ids. As we have documents that reference documents in other collections, we could only use find({originalId: 'xxxxxxxxxxxxxxxxxxxxxxxx'}), and since we have never been able to use get, I doubt there is any performance benefit (unless LokiJS is used in scenarios that do not involve MongoDB, but then why all the fuss about matching its API?).

The second important issue is synchronicity. We heavily use callbacks in our server code and promises in our browser/cordova code. Using a deferred and setTimeout to wrap all synchronous code is tedious. In the long run, LokiJS should think about a plugin architecture for persistence to WebSQL, IndexedDB, localStorage and use callbacks/promises for all data access functions.

Opportunity

There is a real opportunity for LokiJS. IMHO, this is to become the PouchDB of MongoDB, not another light database that resembles more or less MongoDB and that requires all sorts of workarounds to make it store and find MongoDB documents.

Although there might be an opportunity for LokiJS to be competitive as a nodeJS database that I do not grasp, I recommend considering focusing on browser and cordova scenarios (including synchronization) where there is a definitive gap/need. At least, we have this need.

This is just one user's feedback that might trigger some thoughts. I hope this helps.

Database, Collection, resultset and DynamicView should be in separate files.

I noticed that after clearing a collection I wasn't able to insert new objects without error. It seems that the clear collection code sets the idIndex to an object instead of an array which prevents insertion. The constructor does the same thing, but then also calls ensureIndex, which sets idIndex to an array.

Is this behavior intended for some reason?

For some reason the showCollections method throws an error on collections.length

Optimize removal of multiple records with the above methods.

Does it support relations?
one to one
one to many
many to one

If a collection has a binary index set, and the collection is empty, querying it will result in an error. The error happens in the calculateRange function on the following lines:

var minVal = rcd[index[min]][prop];
var maxVal = rcd[index[max]][prop];

If the collection is empty, then rcd[index[min/max]] will be undefined. We may want to set default min and max values here, but also, maybe shortcut the find() function to return an empty result set if the data is empty to save some work.

remove(id) should work.

Timestamp logic for distributed architecture.

Do you plan to add a changes event feature like the CouchDB change API ?

Implement re-hydration of serialized data that enables to instantiate a particular prototype per object so methods are made available subsequently to a loadDatabase() operation.

• Collection.insert()
  • Documentation and example is talking about the document(obj) function, which seems to have been renamed to insert(obj). At first I didn't realise that and I was wondering why I couldn't call the document function so I looked at the source and the comment here is rather confusing, especially the last line :-)
• Loki.loadDatabase()
  • Documentation is missing the inflate option (btw "inflate" is a rather confusing word in this context, "loader" like in the source seems clearer imho)

Implement some form of transaction that caches the collection's state and rolls back to it if something fails during the operation.

If i insert a float value with 4.51 and read the dataset after that from loki i get a value of 2.41 for this field. I have no idea why .... If i store it as string everything is fine...

Users should explicitly set listeners to be async, otherwise they are to be executed in a synchronous fashion.

Currently I have a problem in the area of loading/saving database. After object update and try to save db the following error occurs:

stream.js:94
      throw er; // Unhandled stream error in pipe.
            ^
TypeError: Converting circular structure to JSON
    at Object.stringify (native)
    at LokiEventEmitter.Loki.serialize (/home/teef/leonardo/WebApplications/Leonardo-CommHub/node_modules/lokijs/src/lokijs.js:385:19)
    at LokiEventEmitter.Loki.saveDatabase (/home/teef/leonardo/WebApplications/Leonardo-CommHub/node_modules/lokijs/src/lokijs.js:744:47)

I am using one collection as well as one dynamic view.

Seems like there are two dynamic views which should not happen.

DynamicViews: 
   [ { collection: [Circular],
       name: 'aciveDevices',
       persistent: false,
       resultset: [Object],
       resultdata: [],
       resultsdirty: true,
       cachedresultset: null,
       filterPipeline: [],
       sortFunction: null,
       sortCriteria: null,
       sortDirty: false,
       events: [Object] },
     { collection: [Circular],
       name: 'aciveDevices',
       persistent: false,
       resultset: [Object],
       resultdata: [],
       resultsdirty: false,
       cachedresultset: null,
       filterPipeline: [Object],
       sortFunction: null,
       sortCriteria: null,
       sortDirty: false,
       events: [Object] } ],

Database: https://gist.github.com/teef/c3f8f2139a1e13188e09

Either rename our id something like lokiID or make the id field configurable through options object in the Collection constructor.

if I do this:

var loki = require('storage/lokijs')
var db = new loki('Example');
var users = db.addCollection('users','User', undefined, undefined, undefined);
users.insert(x);
var xt = [];
for(var i = 0; i < 100; i++){
    xt.push({ name : 'odin', email: '[email protected]', age: i });
}
users.insert(xt);
users.findOne({age:1});

I get this back... ( I would have expected the one with age 1

name:
"odin"
email:
"[email protected]"
age:
99
id:
500100
objType:
"User"

Have you done any benchmark on it compared to others?

Thx, will give this a try on wakanda server (wakanda.org)

Using browserify I'm getting this error message:

 Uncaught TypeError: undefined is not a functionlokijs.js:738 

It's not properly detecting ENV when commonjs modules are used in the browser. Easy fix would be to change line 226:

81:     var fs = (typeof exports === 'object') ? require('fs') : false;

226:      if (fs) {
227:        return 'NODEJS';
228:      }

226:      if (typeof window === 'undefined') {
227:        return 'NODEJS';
228:      }

Also need to configure browserify to ignoring the 'fs' module. I can submit a pull request for line 226.

I'd like to be able to subclass Collection into a class that I'm building without getting any of the dynamicview, loadDatabase, or other loki things. The reason for this is because I want:

• Loki's indexing.
• Something small (code-wise).
• Mongodb like query methods.

Pseudo:

class Foo extends LokiCollection {
  constructor() {
    this.name = 'foo'
    this.indices = ['bar']
  }

  getSomeBars(n = 4) {
    return this.where({ bar: n })
  }
}

var foo = new Foo()
foo.insert({ bar: 4, baz: 'awesome' })
foo.insert({ bar: 3 })

foo.getSomeBars() == [{ bar: 4, baz: 'awesome' }]

Does it support multisort?
Like "country asc, town desc" on a view you are about to get?