Feature request: defining custom Operators. about mingo HOT 13 CLOSED

I have considered this feature in the past but there hasn't been a strong demand for it just yet and neither have I found personal use. My initial thoughts were to add a method addOperator which accepts the new operator name and a function receiving the following:

1. raw field expression
2. resolved field value
3. resolved argument values.

This is for simple test operators only. As an example consider.

Mingo.addOperator('$between', function (expr, value, args) {
  // field: raw field expression passed to function
  // value: resolved value of the field expression on the current object
  // args: argument to operator as an Array (even for a single argument)
  // return type: {Boolean | Mingo.Query}
  return value >= args[0] && value <= args[1];

  // OR
  // var query = {};
  // query[expr] = {'$gte': args[0], '$lte': args[1]}
  // return new Mingo.Query(query);
});

// example usage
var result = Mingo.find(collection, {'age': {$between: [18, 30]}});

Any thoughts. Would this approach meet your needs?

from mingo.

Hmm, this approach would only partially meet my needs.

What I'd like to add would be all kinds of operators, such as pipeline operators that are used to restructure data more thoroughly than is currently allowed (e.g. $pluck, which essentially performs Underscore's pluck operation on a collection), mathematical operators to compute various values (mostly analytical), operators that convert between types, and so on. The ability to add any kind of operator that's currently defined by Mingo itself should be enough (e.g. operators that work on entire collections, operators that compare N values for sorting, ternary operators that do some check and then return some value, etc.).

My use case here would be to write an analytics / data exploration tool where users can define their own set of widgets on a dashboard. These widgets would be populated by the data that's coming out of aggregation expressions being applied to a static dataset. The lack of some mathematical/conversion functions as well as the lack of a "$pluck" makes this slightly annoying to deal with.

from mingo.

Your use case seems pretty advance. Quite interesting how far you wanna go with this though 👍
Although it is possible to support your requirements, we must take care to avoid any compromises that may affect other users or the future development of this library (compatibility with the original MongoDB query language). My biggest concern is exposing some internal functions even if marked as private by the _ prefix convention. The good news is that the functions for the pipeline operators and aggregate operators have a consistent interface so we should be able to come up with a workable solution.

Will be happy to brainstorm some ideas.

from mingo.

I agree; the core featureset and ease of use of the library shouldn't become worse as a direct result of this feature request.

I've looked through the code a bit for some of the internals that may need to be exposed. I don't have a full grasp yet regarding how things work, but based on some code readthroughs I think these are the most important things to expose:

• settings object, or alternatively the value of settings.key. This also serves another purpose for people who don't want to make use of this use case: $group and $project use the value of settings.key as part of their core functionality, so the ability to retrieve what this setting was at runtime could be useful e.g. to generate queries at runtime after having set things up.
• resolve function. This is actually a very neat function to expose to users as well, for the utility it provides.
• computeValue function. I must admit that I don't know if this function has any usage for people who don't fall in my use case scenario.
• accumulate function?

Once these functions are exposed, it should be possible to add the following functions to allow defining new (or overriding existing) operators via the following methods:

• addSimpleOperators({...})
• addCompoundOperators({...})
• addAggregateOperators({...})
• addGroupOperators({...})
• addPipelineOperators({...})
• addProjectionOperators({...})

All of these functions should essentially perform a _.extend on the relevant object and then refresh the keys in the Ops object. Alternatively, a single function (addOperators({compoundOperators: {...}, ...}) can be implemented which does all of this in one go and keeps the Mingo object free of a large amount of exposed functions.

from mingo.

Nice idea. I prefer the single interface addOperator, passing in type of operator and the needed Mingo internals via an object of functions. This would include key(), resolve(), and computeValue(). accumulate is just an internal helper that need not be exposed.

I feel overriding existing operators should be disallowed to ensure their integrity and consistency with that of the original spec. Attempting to do so should raise an exception and do same for already registered operators.

Below is an example of the interface I am proposing.

Mingo.addOperators('query', function(m) {
  return {
    '$between': function (expr, value, args) {
      return value >= args[0] && value <= args[1];
    }
  };
});

m contains the following functions:

• key() - returns the id key field set to be used
• resolve(obj, field) - resolve the value of the field (dot separated) on the given object
• compute(obj, expr, field) - compute the actual value of the expression using the given object as context

Here is another for a pipeline operator

Mingo.addOperators('pipeline', function (m) {
  return {
    '$pluck': function (collection, expr) {
       var newCollection = [];
       // code goes here
       return newCollection;
    }
  }
});

The interface of the various operators types would be documented for clarity. Any thoughts?

from mingo.

I like your proposal of passing in the Mingo internals via a function argument. That should keep things neat and tidy.

One point of critique I have with regards to your proposed addOperator function: the function you pass in to it should return an object containing a single key as per your example, but it's not really made clear what happens if an object with more than 1 key is returned instead. I'd propose either renaming the function to addOperators, allowing for the registering of multiple operators in one go, or having the inner function (the one where m is passed into) return something like {name: '$between', type: 'pipeline', func: function (...) {...}} instead, thus forcing and making it clear that only a single operator can be added with each call of addOperator.

Regarding overriding existing operators: maybe it's indeed better if existing operators can't be overridden.

from mingo.

Oops, I did not pay much attention to the name. I do agree with you that the function should be addOperators since it allows returning more than one operator. I have implemented the feature and it is available in the unminified version at the moment. Will quickly update the method name.

For type query operators which are meant to test whether an object passes a given query criteria, I expose a simplified interface than what is used internally in order to reduce boilerplate. This interface is simpler but could pose a challenge for users wanting to write more complicated query operators. Nonetheless it can be easily fixed so we will leave it at that for now. :)

from mingo.

@jorisvddonk Could you please confirm whether the you are able to achieve what you wanted and if possible provide some custom operators for testing. I have added a wiki for reference. Version not updated yet so use the unminified version.

from mingo.

I've looked into it and a quick examination seemed to suggest that it should be OK, but I haven't replied yet as I haven't actually tested it out in my codebase to verify, which I'll do as soon as I get home from work.

from mingo.

Looks good to me!

I've been able to successfully implement the custom operators I was most eagerly anticipating myself ($pluck and $stddev). See the gistfile.

from mingo.

Nice use of the new features.

It will be useful to drop the use of $avg though. A new update I am yet to push only allows running custom operator functions within their own object context to prevent them from tampering with the predefined ones. This happens implicitly for OP_QUERY and OP_PROJECTION operators since their interfaces are wrapped.

I believe this is a reasonable trade off, also given that users are at liberty to execute queries within their implementations. Any thoughts?

from mingo.

That makes sense, yeah. I originally had my own implementation of $avg but figured I'd simplify and use the one that was already predefined. Running queries within a query makes sense and is a lot nicer than calling query functions directly.

from mingo.

👍

from mingo.