Emit events on DynamicViews on changes about lokijs HOT 23 CLOSED

Thanks for the kind words! I'm not sure i understand 100% so forgive me if I misinterpret anything you said. The point of DynamicView is to take away the worry of having to recompute the data contained in it, as the view itself listens to changes to the underlying collection, and up-to-date data is always available through the data() method of dynamicview. For example, say you have an angular controller and in the controller you bind a scope property to dynamicview.data(), then changes applied to the underlying collection that satisfy the dynamicview criteria will be reflected in the dynamicview. In angular's case, dirty checking will reveal if dynamicview.data() has changed and if - and only if - it has, will then the template be re-rendered. I'm not too familiar with other frameworks at the moment (i'm really trying to stick to vanilla JS even for complex stuff) but please feel free to give more details so i understand where DynamicView could be improved to satisfy you use case.

from lokijs.

Unfortunately I'm not familiar with Angular and most other frameworks as well, but from my understanding it shouldn't work as you describe it. Because how does Angular get notified that the underlying dynamicview.data(), which is just a plain JS array, has changed?

But I'm familiar with Knockout so I can give you an example here: the data has to be in a ko.observableArray for automatic dirty checking and re-rendering templates, if required. But this will only work if we manipulate the array through the functions ko.observableArray exposes, like push, splice, etc. If we modify the underlying array directly then Knockout will have no idea that somethinghas changed.
Here is a quick and dirty jsfiddle to show the problem: http://jsfiddle.net/d0oftu4t/
Only clicking on "Add to ko array" will automatically update the template.

Another problem with Knockout is that a ko.observableArrayonly tracks which items are in the array, but not if an object itself has been modified. So it would be really awesome if Loki would emit events when an item has been added, removed, it's position has changed OR if the position stays the same, but the object itself has been changed. Then I could listen for those events and tell Knockout to re-render my templates.

I hope I'm not missing something and I could make my use case clear. If not then just let me know :-)

from lokijs.

DynamicView does lazy rebuild of data... as documents get added/updated/removed I don't know for sure if the views will change until they execute the filters, so I mark them as 'dirty'. Only when you call data() will the array be rebuilt (if needed).

Couple of options :
I can raise an event when collection is flagged as dirty... but if you rebuild view every time this event is fired, you are setting up triggers which drastically slow the system.

I could use our timer (currently used for autosave) to periodically rebuild dirty views (if dirty).

We could and probably should offer ability to wrap view result data into an observable object (where supported by es6). Since I rebuild the whole results array this would probably raise one big 'change' everytime a dirty view is recomputed. Probably best to do this with a timer approach so user doesn't need to do anything after set up.

I can see adding this to next release if @techfort is in agreement. Maybe you could help us test such functionality?

from lokijs.

The whole issue is very interesting. Where it becomes problematic is consistency across the board, which is not just made of plain JS + (all the other frameworks), but also has to take into account that Loki runs in node and cordova. A Reactive Programming approach to developing a UI using Loki's current functionality would definitely yield all the necessary information to correctly handle changes. Scheduling is definitely a good idea, but to avoid crippling an app there would have to be some kind of algorithm which estimates the size of a view and increases rebuild interval based on it (the bigger the view, the longer the interval between rebuilds), which means the app may not be quick enough to react to changes.

In conclusions there are a number of factors to be considered:

1. as @obeliskos said, we should offer some kind of facility to make a view observable.
2. we should probably look at the feasibility of doing a smart update on views that inserts an item in the right position / moves an item to the right position / removes an item without rebuilding the entire view.

So a short term solution could be to emit an event on dirty sort | results, but we should leave this discussion open (and going) to determine the best approach on a long-term.

from lokijs.

I wasn't aware that DynamicViews are lazy. The documentation states that DynamicViews are 'live', which I interpreted as that no additional function calls are necessary and the returned array by calling data() the first time will mutate accordingly, since it's a reference. This is also how I interpreted @techfort s answer above.

Also I'm not sure if I fully understand the options you have mentioned. Did you mean that you could raise an event when the DynamicView, instead of collection, is flagged as dirty? Because I wouldn't want to rebuild my view if the change doesn't affect my view. Other than that, I'd be fine with rebuilding my view everytime a change affects my view, either by receiving an event and calling data() myself or automatically done internally by Loki. Maybe such an "auto rebuild view on change" feature could be made optional by passing an option to the DynamicView constructor? Actually I'd prefer to do the rebuilding myself by receiving an event, because then I could rebuild the view only if I actually have to display the data somewhere, otherwise the lazy approach is fine.

I'm also not sure why you would use a timer for rebuilding dirty views. Maybe there are use cases for this, but in my case I'd need the new view data immediately, because I need to display it as soon as the data changes. Maybe you could elaborate a bit more on this approach, I think I'm missing something here.

And having the option to return view data wrapped into an observable object would be great! But I'm not quite sure how that could work. In ES6 there is Object.observe() and in ES7 there will be Array.observe(), and those functions don't return a new, "observed" object but instead attach a callback to existing objects that get fired if something changes. And one big problem (for web development) with those native ES observe functions would be that Object.observe() won't be available in IE any time soon and Array.observe() is proposed for ES7 and is currently only supported by Chrome 36+. And I don't think a polyfill will be possible at all for those functions.

But maybe it could be possible to make DynamicViews extensible in a way that would allow to return the data wrapped in any framework specific observable, like a Knockout observable, or Angular observable, or whatever. I have no idea yet how this could be implemented, but I will think about it and let you know if I have any ideas.

And yes, I'd be more than willing to help test such functionality. I'll also dig deeper into Lokis internals and see if there is anything I could contribute (once I have figured out how Github and Git in general works, I just signed up for Github today).

from lokijs.

Actually - much as it's a bit of a fragmented solution, the framework specific adapters (supplied as a separate lib to loki) is a nice approach, a. because they don't fiddle with internals, and b. because they would arise as the community needs them. In Angular, for example, if you're ok with a rebuild per digest cycle you simply need to bind a controller's property to the view data() method and off you go. I've done this and it works, I swear :) Glad to see you're available for contribution, Loki is growing quite quickly and we need more than 4 hands on those keyboards.

from lokijs.

@techfort What?? What kind of black magic is Angular using there? Don't tell me Angular is doing dirty checking in an interval!

Okay, since I don't know anything about Angular, I searched and found a great explanation about $apply and $digest here
If I understand it correctly then an automatic $digest cycle is only triggered through Angulars one-way binding by manipulating a model through an Angular view (by entering something into a textbox or clicking a button that changes a model). But if you change a model outside of Angulars context then you'd have to call $apply manually to trigger a $digest cycle. So I suppose in your case you were calling $apply manually, after manipulating a collection?

from lokijs.

You can make an instance of loki or a particular collection available as an angular service and it will be included in the normal $digest cycles.
To clarify: you can do something like this:

angular.module('myApp').factory('Loki', function () {
  var db = new loki();
  return db;
}).controller('MyController', ['$scope', 'Loki', function ($scope, Loki) {
  // .. some code to initialize a collection or something like that
  var dv = mycollection.addDynamicView('myview');
  $scope.view = dv.data();
});

I seem to remember getting this to work perfectly. Should this not work then you can try a $scope.$watch making sure to specify the third argument to be true, which means an object is deeply observable.

Getting back to the main topic, I think framework specific adapters are actually the way to go. There would be only 5 or 6 major framework anyway (Angular, Ember, Knockout, Backbone and very recently Aurelia).

from lokijs.

Thanks alot for the clarification! I guess I'll try Angular and see how they do it. (by the way I just noticed that there is no discussion functionality here on Github! What a shame :(

And I absolutely agree that framework specific adapters are the way to go. Looking forward to see how this will develop!

Sorry, a little offtopic again, but since you mentioned reactive programming earlier: I stumbled accross RiotJS a couple of days ago. It's a micro library that does things the reactive way. The reactive pattern is definitively something I want to look into. And I think a riot with lokis help could be really fun :-)

from lokijs.

Hmm, actually your right the resultdata array is kept up to date. I guess I am thinking nonpersistent mode which keeps live indices and generates array on data(); Been awhile since I was in that code :)
If you have a sort applied it lazy sorts though so that might be relevant to a lot of dynamic views.

Loki is designed to handle hundreds of thousands of operations per second. So assuming we used eventing instead of obsersable we may want to limit to no more than 1 event every 2 seconds (user could set this) or so. In that case a timer might be good to buffer and balance load to event.

If you would like to help out that sounds great :) Both of you sound like you have more familiarity with those frameworks, but now would be good time to look into those frameworks so i'll try to keep up as needed.

Hmm yea I guess status.modern.ie is showing 'under consideration' for object.observe. Odd since they positioning their forked ie 'spartan' to be the most 'edge'-e browser for win10. Not to mention most applications would need compatible solutions for older browsers.

Riot.js does look fun to try using against loki. I may try some hello worlds with a couple of these frameworks myself over the weekend.

Sure dig into source code, we should have fairly good comments but feel free to ask if anything is confusing.

from lokijs.

Actually, before it goes out of my mind... We may include a time
measurement within the data() method so we can gauge exactly how often a
view can be rebuilt. A bit like the dynamic fps calculation in 3d apps or
games in general (2d or 3d). The challenge would be to find a lib as
precise as process.hrtime in node.js
On 5 Feb 2015 23:50, "obeliskos" [email protected] wrote:

Hmm, actually your right the resultdata array is kept up to date. I guess
I am thinking nonpersistent mode which keeps live indices and generates
array on data(); Been awhile since I was in that code :)
If you have a sort applied it lazy sorts though so that might be relevant
to a lot of dynamic views.

Loki is designed to handle hundreds of thousands of operations per second.
So assuming we used eventing instead of obsersable we may want to limit to
no more than 1 event every 2 seconds (user could set this) or so. In that
case a timer might be good to buffer and balance load to event.

If you would like to help out that sounds great :) Both of you sound like
you have more familiarity with those frameworks, but now would be good time
to look into those frameworks so i'll try to keep up as needed.

Hmm yea I guess status.modern.ie is showing 'under consideration' for
object.observe. Odd since they positioning their forked ie 'spartan' to be
the most 'edge'-e browser for win10. Not to mention most applications would
need compatible solutions for older browsers.

Riot.js does look fun to try using against loki. I may try some hello
worlds with a couple of these frameworks myself over the weekend.

Sure dig into source code, we should have fairly good comments but feel
free to ask if anything is confusing.

—
Reply to this email directly or view it on GitHub
#71 (comment).

from lokijs.

I regards to the angularJS implementation, I am currently working on a project experiment with nodewebkit + Angular + lokiJS. So far I like how even the normal collection.find() result reference keeps itself updated.

In order to keep things updated on the view though, I have to put a $watch for the variable on my controllers. Apparently, $apply and $digest didn't work too well for me.

Here's a snippet of what I use in a controller:

var folders = collection.find();

$scope.list = folders;

$scope.$watch( function(){
   return folders;
}, function(newVal) {
   updateView(newVal);
}

function updateView(value) {
  $scope.list = value;
}

And yes, whenever I modify something behind the scenes it updates the view, and when I modify something using angular's view directives (e.g. adding records using ng-click), it updates the view.

Not sure if this is the best way to do this just yet though and the performance implications, as I am still starting out with the prototype. I'll probably start with dynamic views soon too and his idea looks interesting.

from lokijs.

ok, i created a plunker for you at http://plnkr.co/edit/2B3CA6CRa2P8Y1yYtvBI?p=preview
LokiJS is included in the angular lifecycle by wrapping it into a service. There's a dynamic view that only displays user names longer than 3 chars. If you try and add names with only 3 chars they won't be displayed, if the new user has 4 chars or more it will, and no need for $watch. Let me know if this works for you.
I utilized this approach in a node-webkit app and tt was flying :)

from lokijs.

Wow thanks! I apologize as I'm not exactly a js wizard but that looked so neat! Looks like that will work well for me too.

P.S
I think we need more examples like this in the docs section, especially when utilizing lokijs within frameworks.

from lokijs.

I'm sure your JS is great.

I will definitely include a wiki page to show a couple of examples of LokiJS integration with other frameworks, at the very least AngularJS, with which I'm familiar, thanks for the tip!

from lokijs.

I tried to add a removeWhere from the plunker you gave me but it seems to error out.

http://plnkr.co/edit/aA1nllguAAD4pi3jsNJH?p=preview

from lokijs.

I see - weird! I'll create a separate issue to this one, if we're all agreed on framework-specific adapters then we should probably close this.
Opened bug on removeWhere here

from lokijs.

Hi guys,

just wanted to give a quick update regarding this issue/the framework specific adapters and the resulting discussion we had here.

I was setting up a playground with a simple grid, backed by Loki and Knockout, to be able to experiment and see how such an adapter could work/look like and make everything fit my specific use-case. In the end I ended up developing a prototype of my own "in-memory DB" implementation (it's actually not as DB-like as Loki, so I wouldn't call my solution an in-memory DB).

I was greatly inspired by Loki (specifically the dynamic view feature and the binary indexes), as well as some other libraries, like Lazy.js for fast querying of non-indexed data and linq.js for the lambda-like string syntax.

At the moment I'm also working on framework specific adapters to make things observable. I'll post what I've come up with once it's somewhat ready. A similar approach then could maybe be used for Loki as well.

from lokijs.

Thanks for the update and keep us posted. Don't know if it comes in handy for you, but I also wrote an extremely mnimalist and mainly functional programming version of Loki called fundb, based on a more classic FP approach (like Ramda.js or in general how you would do things in Haskell/Clojure)

from lokijs.

Thanks for mentioning fundb! I'll have a look and maybe I can gracefully steal some ideas get inspired 😆

I'll definitively keep you posted but it may take at least a week or two until I get things done.

from lokijs.

Just wanted to follow up if anything was ever implemented here? or if for Angular it is still best off evaluating the DynamicView.data() function each cycle?

from lokijs.

I can't speak too much for angular, but I guess it depends on how many results are in your view and whether sorts and/or transforms are involved. What kind of rough estimates do you foresee you will encounter?

I would start with simple calls to data() and if that proves inefficient you can try out some of the dynamic view constructor options :

• the internal resultset is always kept up-to-date (not including sorts)
• we have 'sortPriority' option if your results to always be sorted (probably better for node/server than browser)
• we have 'persistent' option on dynamic views to keep internal results in an internal array (resultdata)
• we do emit a 'rebuild' event whenever the view changes if you want to listen for it

from lokijs.

Thanks for the great answer.

I'm not sure on any performance barriers I may hit. I'm expecting it will be fine. I was just more curious as to the best approach.

I think calling data() will be fine. Angular will compare the new array to the old array object by object and then only update the UI where needed (and reshuffle where needed). And it sounds like the call to data() is very cheap if no changes have been made.

I'd more foresee performance issues in the DynamicView being rebuilt after the data has changed. But I figure long queries would only be solved by running it async in a webworker. I'd imagine there would need to be 100,000s of records before that became an issue though.

But it's really good to know there are options if performance issues do arise!

from lokijs.