My approach to plugins about community HOT 25 OPEN

Why all this complex machinery when you can call rangeDel(db, ...). Is there a way to write a C++ extension that plays nicely without leveldown with global pollution?

from community.

Its good to see a discussion like this to allow plugins architecture to be performant.

Is it possible that the advantages and disadvantages of the JS versus c++ approach be listed to provide more clarity ?

For my own usage i favour a pure JS solution to allow easy of developing plugins.
I also favour that the actual CRUD's do not need to have any knowledge of the plugins.
to do this i would have thought that a serious of base handlers be available that get called by levelup, which they runs the plugins automagically on the query.

from community.

@Raynos I'm trying to come up with an approach that makes a more consistent plugin ecosystem so that users don't have to do something new & different for each thing they want to bolt on. I know you're comfortable plugging many tiny bits together to form a complete, customised whole but I'd like to make this as easy as possible for the average dev who doesn't have time to dig into the tiny details of each element they need in order to come up with a solution that fits their problem. I'm only using .rangeDel() as a single example and sure, there's plenty of other ways to do this without the "machinery" inside LevelUP but that machinery should also allow many varied different plugins that modify behaviour or bolt on new functionality. For example: transparent namespacing of keys. Or: transparently inserting additional entries to form indexes for primary entries and adding a .getBy() method that can fetch by those indexed properties. Or: bypassing a proper .get() in favour of .batch(). These can all be achieved by other means but each would take a different approach and would require a different procedure to make it work and it'd be difficult to know if they could work together. My "machinery" should make these bits work together nicely and would provide a consistent interface to load and insert plugins to the core.

@Raynos can you clarify your question about C++ extensions and global pollution?

@gedw99 pure JS implementations are fine for most things that could conceivably extend LevelUP but there are some cases where that's not possible (e.g. comparators that adjust internal sorting in LevelDB) and other cases where the efficiency gain is significant, such as .rangeDel()--but if this is not going to be a common operation and your data set isn't large then doing it all in JS-land is fine. It's crossing the V8->native boundary that's costly, plus with .rangeDel() you have to go back and forth on that boundary and between threads for each entry whereas when you do it in C++ you do it in a single thread, a single async call and you don't actually need to pass anything across that boundary to get the job done.

from community.

If anyone's interested in the speed impact of doing a native .rangeDel() vs doing it in JS-land, see simple benchmark for it here https://gist.github.com/rvagg/5116514 - results here:

Native .rangeDel() benchmark
Filling database
Starting .rangeDel()...
Native .rangeDel() on 100000 entries took 262 ms
JS rangeDel() benchmark
Filling database
Starting rangeDel()...
JS rangeDel() on 100000 entries took 1636 ms
Done

from community.

massive difference.

i guess then we have to have some plugins at c++ level. so for the c+
newbies then we need to be able to toggle it on and off easily ?

G

On 8 March 2013 14:45, Rod Vagg [email protected] wrote:

If anyone's interested in the speed impact of doing a native .rangeDel()vs doing it in JS-land, see simple benchmark for it here
https://gist.github.com/rvagg/5116514 - results here:

Native .rangeDel() benchmark
Filling database
Starting .rangeDel()...
Native .rangeDel() on 100000 entries took 262 ms
JS rangeDel() benchmark
Filling database
Starting rangeDel()...
JS rangeDel() on 100000 entries took 1636 ms
Done

—
Reply to this email directly or view it on GitHubhttps://github.com/rvagg/node-levelup/issues/92#issuecomment-14619731
.

Contact details:
+49 1573 693 8595 (germany)
+46 73 364 67 96 (sweden)
skype: gedw99

from community.

Well @gedw99, they wouldn't be enabled or even available by default. So far nobody is suggesting they actually get bundled with the core. So you'd have to explicitly install them with npm and either invoke them with their own custom magic (@Raynos' approach) or inject them with special, consistent LevelUP magic (my approach).

I keep on thinking up awesome native plugins now that it's (almost) possible. As I was doing those benchmarks, I included a countEntries() function that had to use a KeyStream to do the counting which is not cool. Why not have a plugin that does a simple .count() using iterators at the native level?

Of course I must not get too carried away here with C++, LevelJS awaits!

from community.

ok well NPM install is easy. I thought you were going to embed it wit the
core.

god design

I was looking at the NodeJS of rails last iht. Its called Sails and acts as
an ORM.
It is very useful for the developer that just wants to get things done.

they have adapters for many differnt DB's and its a very smooth and easy
architecture.

i reckon a plugin for them at the CORE level woudl be amazing

g

On 8 March 2013 15:06, Rod Vagg [email protected] wrote:

Well @gedw99 https://github.com/gedw99, they wouldn't be enabled or
even available by default. So far nobody is suggesting they actually get
bundled with the core. So you'd have to explicitly install them with npm
and either invoke them with their own custom magic (@Raynoshttps://github.com/Raynos'
approach) or inject them with special, consistent LevelUP magic (my
approach).

I keep on thinking up awesome native plugins now that it's (almost)
possible. As I was doing those benchmarks, I included a countEntries()function that had to use a KeyStream to do the counting which is not cool.
Why not have a plugin that does a simple .count() using iterators at the
native level?

Of course I must not get too carried away here with C++, LevelJS awaits!

—
Reply to this email directly or view it on GitHubhttps://github.com/rvagg/node-levelup/issues/92#issuecomment-14621282
.

Contact details:
+49 1573 693 8595 (germany)
+46 73 364 67 96 (sweden)
skype: gedw99

from community.

also plugins for GIS spatial stuff will be able to b put into the leveldown
code when they shoudl be.

It will be really interesting to see how we can get master master
replication going with a map reduce style of querying

g

On 8 March 2013 15:26, Ged Wed [email protected] wrote:

ok well NPM install is easy. I thought you were going to embed it wit the
core.

god design

I was looking at the NodeJS of rails last iht. Its called Sails and acts
as an ORM.
It is very useful for the developer that just wants to get things done.

they have adapters for many differnt DB's and its a very smooth and easy
architecture.

i reckon a plugin for them at the CORE level woudl be amazing

g

On 8 March 2013 15:06, Rod Vagg [email protected] wrote:

Well @gedw99 https://github.com/gedw99, they wouldn't be enabled or
even available by default. So far nobody is suggesting they actually get
bundled with the core. So you'd have to explicitly install them with npm
and either invoke them with their own custom magic (@Raynoshttps://github.com/Raynos'
approach) or inject them with special, consistent LevelUP magic (my
approach).

I keep on thinking up awesome native plugins now that it's (almost)
possible. As I was doing those benchmarks, I included a countEntries()function that had to use a KeyStream to do the counting which is not cool.
Why not have a plugin that does a simple .count() using iterators at the
native level?

Of course I must not get too carried away here with C++, LevelJS awaits!

—
Reply to this email directly or view it on GitHubhttps://github.com/rvagg/node-levelup/issues/92#issuecomment-14621282
.

Contact details:
+49 1573 693 8595 (germany)
+46 73 364 67 96 (sweden)
skype: gedw99

Contact details:
+49 1573 693 8595 (germany)
+46 73 364 67 96 (sweden)
skype: gedw99

from community.

@rvagg I have zero objection to the underlying machinery needed to standardize the authoring and creating of plugins from a module author point of view. I think all plugin authors should create them in a similar fashion.

I have an objection to the user facing API for people that use plugins

instead of

require('upper-rangedel').use()

db.delRange(...)

I'd be far nicer to have

var delRange = require('upper-rangedel')

delRange(db, ...)

The reasoning for this is that it's explicit and less magical plugins that mutate global db even if AT THE IMPLEMENTATION LEVEL plugins mutate the global db

from community.

Right, I see what you mean by 'global' then. Btw, my implementation doesn't force mutating the global, you can do it only for specific instances too: var db = levelup('foo', { use: require('upper-rangedel') }) or, with an existing db: db.use(require('upper-rangedel')). The require('upper-rangedel').use() version is if you want it done on all instances automatically (which I suspect will be the preferred use-case for most people for a plugin like this).

And, re the LevelDOWN version, yes I think it might be possible to do your preferred approach there too rather than injecting directly into the instance. require('downer-rangedel')(this._db). I think it comes down to similar questions as in JS-land re which properties and methods you expose publicly. One nice feature of the externr approach is that you can expose any (or all) of that private junk we have in the closure that we use to create an hold new instances. So plugins can have special access to private properties that aren't available on the public API.

But I guess ultimately it comes down to preferences and I'd like to hear more voices on this topic before anything happens. I maintain that my approach provides more consistency and simplicity for end-users who just want to build stuff and not be too concerned with how LevelUP and it's ecosystem works. "Just give me a database with these features, I have awesome stuff to build".

from community.

This is interesting. When we originally discussed rangeDel I had assumed that it would have to be added to leveldown. Very interesting if they could be loosly coupled to the db.

In the case of something like rangeDel which just iterates over a range, doing deletes, couldn't you just pass the C object to the range delete plugin?

rangeDel = RangeDel(db) ?

passing leveldown to the plugin could be simpler here.

However, this issue is discussing two quite different things - C++ leveldown plugins, and particular extension points.

I think @Raynos' concern is valid, adding extensions willy nilly is gonna end in pain.
I'm not sure that encouraging users to extend the base db object with their own stuff is such a good idea,
I know from building those things that they are quite hard to get right. So intrinsically encouraging that stuff is asking for trouble (bugs)

I described the plugin approach I was using here, https://github.com/rvagg/node-levelup/wiki/Plugin-Pattern-Discussion,
in brief, this worked my monkey-patching certain things, and then applying special rules on keys inserted into particular ranges.

It was possible to create all sorts of interesting extra features with that approach, but I ended up duplicating code for managing ranges and prefixes in every plugin, they where messy, and hard to perfect.

I have recently been working on a new approach, based on level-sublevel

As this issue is already covering several topics, I'll start a new issue.

from community.

my revised plugin approach: https://github.com/rvagg/node-levelup/issues/97

from community.

one thing I've needed to do is hook into a put or del call, decide to add something, and turn it into a batch.
is it possible to do that with extenr?

from community.

On your last question, this is from the externs_example.js, a simple, sneak plugin that cancels a put() if it doesn't like the look of your key.

   var sneakyextern = {
        put: function (key, options, valueEnc, callback, next) {
          if (key == 'foo2') {
            // bypass the real get() operation, jump straight to the user callback
            return callback()
          }
          // internal next() callback for the extern chain
          next(key, options, valueEnc, callback)
        }
    }

You'd just extend this a little and get this:

  var batchplugin = {
        put: function (key, options, valueEnc, callback, next) {
          this.batch([ { type: 'put', 
          if (key == 'foo2') {
            // bypass the real get() operation, jump straight to the user callback
            return callback()
          }
          // internal next() callback for the extern chain
          next(key, options, valueEnc, callback)
        }
    }

But I thought that a more interesting example would be in order so I created an indexing plugin using the externr approach. See https://github.com/rvagg/node-levelup/blob/pluggability/externs_example.js

The example stores this data (top npm packages):

        {  {
      name    : 'underscore'
    , author  : 'jashkenas'
    , version : '1.4.4'
    , url     : 'https://github.com/documentcloud/underscore'
  }
, {
      name    : 'async'
    , author  : 'caolan'
    , version : '0.2.6'
    , url     : 'https://github.com/caolan/async'
  }
, {
      name    : 'request'
    , author  : 'mikeal'
    , version : '2.14.0'
    , url     : 'https://github.com/mikeal/request'
  }
/* duplicate indexed properties are left as an exercise for the reader!
, {
      name    : 'coffee-script'
    , author  : 'jashkenas'
    , version : '1.6.1'
    , url     : 'https://github.com/jashkenas/coffee-script'
  }
*/
, {
      name    : 'express'
    , author  : 'tjholowaychuk'
    , version : '3.1.0'
    , url     : 'https://github.com/visionmedia/express'
  }
, {
      name    : 'optimist'
    , author  : 'substack'
    , version : '0.3.5'
    , url     : 'https://github.com/substack/node-optimist'
  }

and then it enables the database to be operated on like this, note the new .getBy() method:

// a standard get() on a primary entry
db.get('underscore', function (err, value) {
  if (err) throw err
  console.log('db.get("underscore") =', JSON.stringify(value))
})

// some gets by indexed properties
db.getBy('author', 'jashkenas', function (err, value) {
  if (err) throw err
  console.log('db.getBy("author", "jashkenas") =', JSON.stringify(value))
})

db.getBy('author', 'mikeal', function (err, value) {
  if (err) throw err
  console.log('db.getBy("author", "mikeal") =', JSON.stringify(value))
})

db.getBy('version', '0.2.6', function (err, value) {
  if (err) throw err
  console.log('db.getBy("version", "0.2.6") =', JSON.stringify(value))
})

producing this output:

db.get("underscore") = {"name":"underscore","author":"jashkenas","version":"1.4.4","url":"https://github.com/documentcloud/underscore"}
db.getBy("author", "jashkenas") = {"name":"underscore","author":"jashkenas","version":"1.4.4","url":"https://github.com/documentcloud/underscore"}
db.getBy("author", "mikeal") = {"name":"request","author":"mikeal","version":"2.14.0","url":"https://github.com/mikeal/request"}
db.getBy("version", "0.2.6") = {"name":"async","author":"caolan","version":"0.2.6","url":"https://github.com/caolan/async"}

tell me that's not awesome!

from community.

and in that example's case, the plugin is all of 28 lines without comments; of course it's incomplete but it works nicely as an example of what's possible.

from community.

Sorry. That example is here: https://github.com/rvagg/node-levelup/blob/pluggability/externs_example2.js

The db init code is simply this:

  , db = levelup('/tmp/indexer.db', {
        // inject the plugin
        use             : indexer
        // our plugin reads this option
      , indexProperties : [ 'author', 'version' ]
      , keyEncoding     : 'utf8'
      , valueEncoding   : 'json'
    })

Note how it is able to take options of its own.

from community.

Why can we not have simple things like

var getBy = indexer(db, ["author", "version"]

// no magic. Just functions
getBy("author", "jashkenas", function () { ... })

I think we should definitely have low level hooks & extensions points for leveldb MODULE authors. But those things are implementation details.

But we shouldn't encourage a magical use and just drop all the properties in one big massive config hash approach.

from community.

@rvagg Very nice! But I think @Raynos has a point. Lets say I'm completely new to LevelUP and want to use this new cool indexing module that I heard so much about. It's hard to see the connection between the values in the options hash and the actual plugin, and if there are more plugins used, then this config is going to be massive. What if some other plugin also uses the 'indexProperties' property?

from community.

@Raynos I have a feeling we're talking about slightly different things here. This is only about module authors, being able to make modules that extend levelup in a consistent way that provide module users with a simple and consistent approach to using plugins together.

There's no massive config hash, I don't know where you're seeing that. Plugins are objects that expose the extension points they care about, they don't get munged in together into some big hash. Plugin extension points extend from each other so when you add a plugin with the externr mechanism they form a chain with one executing after the other for the same extension point. So, when you have an db instance that has an "index" plugin, it can also have a totally different plugin working at the same time and the "indexer" doesn't have to care about functionality beyond its own stuff--in your approach, how do you extend on top of your "indexer" without reimplementing large chunks of levelup or saying that it can't be extended itself so you can only have indexer and nothing else significant on it as well?

I'd appreciate it if you had a look at externr and try to understand what it's doing rather than dismissing this approach so easily.

from community.

@ralphtheninja the indexProperties isn't important, it could just as easily have worked like this:

var db = levelup('indexed.db', {
    use           : indexer([ 'author', 'version' ])
  , keyEncoding   : 'utf8'
  , valueEncoding : 'json'
})

It's the use here that's important, it can be an array too:

var db = levelup('indexed.db', {
    use           : [ geospatial, indexer([ 'author', 'version' ]), rangeDel ]
})

Or, if that's not to your taste:

var db = levelup('indexed.db')
db.use([ geospatial, indexer([ 'author', 'version' ]), rangeDel ])

or

var db = levelup('indexed.db')
db.use(require('levelup-geospatial'))
db.use(require('levelup-indexer')([ 'author', 'version' ]))
db.use(require('upper-rangedel'))

The point being, these all have the same mechanism for working inside of levelup and can play happily together and the user doesn't have to get lost in this kind of cruft:

var db = levelup('indexed.db')
db = require('levelup-geospatial')(db)
var indexedDb = require('levelup-indexer')(db, [ 'author', 'version' ]) // ooops, can I extend the geospatial version of the db?
var rangeDel = require('upper-rangedel')
rangeDel(db) // which instance of `db` am I working on here? can it handle this?

from community.

I like this version the most:

var db = levelup('indexed.db')
db.use(require('levelup-geospatial'))
db.use(require('levelup-indexer')([ 'author', 'version' ]))
db.use(require('upper-rangedel'))

How about not having to use require?

var db = levelup('indexed.db')
db.use('levelup-geospatial')
db.use('levelup-indexer', [ 'author', 'version' ])
db.use('upper-rangedel')

from community.

@rvagg

var db = levelup('indexed.db')
var getBy = require('levelup-indexer')(db, [ 'author', 'version' ]) 
getBy("author", "raynos", cb); /* will just work */
var rangeDel = require('upper-rangedel')
rangeDel(db, { /* some range */ }) /* this will just work */

I simply think this api is nicer then the .use() api. That's all. Heck for all I care all those functions can do db._use under the hood.

I prefer the attitude of "we have a module system and a bunch of leveldb compatible modules" versus "hey guys we have a plugin system. you can write plugins for our platform"

The former says we write commonJS modules that work with leveldb and follow the node style, if you know how to write modules, you know how to make a modular database.

The latter says come use our special plugin system that's different from every other plugin system for every other framework. It's kind of like node based commonJS modules, but not!

from community.

the standard commonJS way of doing it is also less limitting i feel.

plugin systems MUST get the API right for all possible intended and
unintended use cases. This is hard to get right.

On 9 March 2013 23:27, Raynos [email protected] wrote:

@rvagg https://github.com/rvagg

var db = levelup('indexed.db')var getBy = require('levelup-indexer')(db, [ 'author', 'version' ]) getBy("author", "raynos", cb); /* will just work /var rangeDel = require('upper-rangedel')rangeDel(db, { / some range / }) / this will just work */

I simply think this api is nicer then the .use() api. That's all. Heck
for all I care all those functions can do db._use under the hood.

I prefer the attitude of "we have a module system and a bunch of leveldb
compatible modules" versus "hey guys we have a plugin system. you can write
plugins for our platform"

The former says we write commonJS modules that work with leveldb and
follow the node style, if you know how to write modules, you know how to
make a modular database.

The latter says come use our special plugin system that's different from
every other plugin system for every other framework. It's kind of like node
based commonJS modules, but not!

—
Reply to this email directly or view it on GitHubhttps://github.com/rvagg/node-levelup/issues/92#issuecomment-14672039
.

Contact details:
+49 1573 693 8595 (germany)
+46 73 364 67 96 (sweden)
skype: gedw99

from community.

Also if the plugin API is only used by module AUTHORS. And not by leveldb USERS then we can make more changes on it in an aggressive fashion.

Because honestly most of the initial module AUTHORS are you used to this idea being experimental. The module authors can then change the internals of their modules to match the new plugin / extension API and not break back compat on their surface api.

of course there are disadvantages to this. The module(db, { /* do stuff */ }) approach hides the fact that the module extends the database and intercepts puts. Which may be suprising.

from community.

yes. Private API and a Public API. Its a good way to give room to grow.

g

On 9 March 2013 23:45, Raynos [email protected] wrote:

Also if the plugin API is only used by module AUTHORS. And not by leveldb
USERS then we can make more changes on it in an aggressive fashion.

Because honestly most of the initial module AUTHORS are you used to this
idea being experimental. The module authors can then change the internals
of their modules to match the new plugin / extension API and not break back
compat on their surface api.

of course there are disadvantages to this. The module(db, { /* do stuff
*/ }) approach hides the fact that the module extends the database and
intercepts puts. Which may be suprising.

—
Reply to this email directly or view it on GitHubhttps://github.com/rvagg/node-levelup/issues/92#issuecomment-14672314
.

Contact details:
+49 1573 693 8595 (germany)
+46 73 364 67 96 (sweden)
skype: gedw99

from community.