For a non-programmer, % is immediately recognizable as a per cent sign, not as modulo. The mathematical symbol for a modulo is the “mod” keyword. Filtrex v3 will introduce a mod keyword and deprecate the % operator.

In a future version, % could be potentially reintroduced as percentage.

The ideal filtrex expression should be reliable and return the same result every time it is evaluated. Letting the user call Math.random() by default goes against this principle and might lead to pathological applications states – imagine filtering a paginated list based on a random criterion.

Therefore, the random() function will be removed in v3. Anyone who needs it can pass it as a custom function.

Right now, unknown symbols return undefined, rather than returning a descriptive error:

compileExpression('a', {b: 4})
// undefined
compileExpression('foo of bar')({ bar: 4 })
// undefined

This way, expressions are prone to bugs:

compileExpression('widht > 500', {width: 600})
// false (0)

There should be an API for errors. Errors should be machine-readable and easily localisable by the developer. Errors should be thrown returned much more often.

Don't let the user throw an error. Instead, catch it and return it.

Most errors are now i18n-friendly (see #35), however errors produced by Jison are not. Investigate the possibility to modify Jison, so that it produces better errors.

Hey,

I would be more than happy if its possible to add an async extra functions support.
I've done it in my fork.
simply alter the following files:
at filtrex.mjs file.

    return async function (data) {
        try {
            return await func(createCall(functions), operators, std, prop, data)
        } catch (e) {
            return e
        }
    };

The use-case that we're having is filtering data utilizing some db queries and it requires async functions :#

Additionally, I've added a test file to test it out:

const options = {
    extraFunctions: {
        asyncFunc: async (a, b) => {
            return new Promise((resolve, reject) => {
                resolve(a + b);
            })
        }
    }
}
const eval = (str, obj) => compileExpression(str, options)(obj);

describe('Arithmetics', () => {
    it('can do simple numeric expressions', async () => {
        expect( eval('1 + 2 * 3') ).equals(7);
    });

    it('can do simple async expressions', async () => {
        expect( await eval('asyncFunc(1,2)') ).equals(3);
    });

});

Right now, the compileExpression function has the following signature:

function compileExpression(expr: string, customFunctions?: { [k: any]: Function }, customProp?: Function);

It should be changed to:

function compileExpression(expr: string, options: object)

This would allow us to introduce new API more easily and without breaking changes.

Todo list

• Change the code
• Update README
• Add to changelog

Version 3 brings a lot of changes, including breaking ones. Either a readme or a wiki page should include a complete changelog. Furthermore, make sure that each new feature is properly documented in the readme.

BREAKING DESIGN CHANGES:

• #22 (Returning runtime errors instead of undefined.)
• #27 (Not converting booleans to numbers anymore.)
• #36 (Modulo can no more return a negative result.)
• #47

BREAKING BUG FIXES:

• #41
• #42 (Changed not in to have the same precedence as in.)
• #43 (Changed exponentiation to be right-associative.)

DEPRECATIONS:

• #34 (Ternary ? : operator deprecated in favor of if then else.)
• #48 (% deprecated in favor of mod.)

NEW FEATURES:

• #30
• #35
• #37
• #38
• #44 (Optional chaining and dot accessor syntax.)
• customProp now has additional argument type: 'unescaped' | 'single-quoted'

Make this:

 a^2 + 2 a b + b^2

parse as this:

  a**2 + 2*a*b + b**2

The implicit multiplication operator can be overloaded (see #30) using options.operators.juxtapos. If you want to disable multiplication by juxtaposition, redefine the operator to throw an error.

Let people willing to contribute know some basics facts about this repo

Hi!
Is there a way to get a list of used variables in the expression after compiling it.
E.g.
Expr: transactions <= 5 and abs(profit) > 20.5
Variables: ["transactions","profit"]

Thanks!

Hi,

my JSON object contains the dash symbol (-) within the, e.g. { 'seller-sku': 'AB-CDEF-GH' } and my filter expression var expression = 'seller-sku == "AB-CDEF-GH"'; does not work with that.

So I would need to pre-parse the object and change every object key to camelCase or PascalCase.

Would it be possible to include all possible JSON keys within the expression?

For example:

<script src="filtrex.2.2.0.min.js"></script>
<pre>
<script>
var fn1 = filtrex.compileExpression("x==1");
document.write('{x:1}:',fn1({x:1}),"\n"); //1
document.write('{x:2}:',fn1({x:2}),"\n"); //0
document.write('{y:1}:',fn1({y:1}),"\n"); //0

var fn2 = filtrex.compileExpression("x!=1");
document.write('{x:1}:',fn2({x:1}),"\n"); //0
document.write('{x:2}:',fn2({x:2}),"\n"); //1
document.write('{y:1}:',fn2({y:1}),"\n"); //1 (this should be 0)

var fn3 = filtrex.compileExpression("x!=1 and x!=null");
document.write('{x:1}:',fn3({x:1}),"\n"); //0
document.write('{x:2}:',fn3({x:2}),"\n"); //1
document.write('{y:1}:',fn3({y:1}),"\n"); //0 (this is ok)
</script>
</pre>

I can resolve this by using x!=1 and x!=null, but a little troublesome.

Could you provide a way to make that default false? Maybe add an option or something.

Sometimes, it is necessary to provide various constants to the user. Right now, the only way to achieve this is to modify the data, which is rather unfortunate – it would be much better to specify the constants only once at compile time. Example usage:

const options = {
  constants: { pi: Math.PI }
}
const fn = compileExpression(`2 * pi * radius`, options)

fn({ radius: 1/2 })
// → Math.PI

It should be possible to access data fields shadowed by constants using single-quoted symbols:

const options = { constants: { a: "a_const " } }
const data = { a: "a_data ", b: "b_data " }

// single-quotes give access to data
const expr = `'a' + a + 'b' + b`

compileExpression(expr, options)(data)
// → "a_data a_const b_data b_data"

In order to keep the API simple, constants will bypass a customProp function – if a developer wants total control over the identifiers, they shouldn't define any constants.

Getting this message with v2.1.0 (works fine with v2.0.0) using Yarn PnP.

As rightly pointed out by @​rici in this SO post, the expression 4 + 3 not in (7, 8) groups as 4 + (3 not in (7,8)), which produces a nonsense answer 4 + true == 5. This will be fixed in v3.

In theory, it should be possible to detect the return type of expressions. Investigate whether it would be viable and useful.

Need to use it in a browser. Thank you for any advice.

The options object should allow for methods like customAdd and customGreaterThan that would override the default action of these operators.

All errors, both thrown and compile time and returned at runtime, should have an identifier which would make translation possible. It is not acceptable to show English errors in a non-English application.

Make the default modulo always return a positive number, inspired by CoffeeScript's %%. This will be configurable via #30.

If a NUMBER is followed by a SYMBOL, we should treat it as a suffixed number and transformed to something like options.suffixedNumber( num, sfx ). If there is no suffixedNumber method, we should throw an error that makes sense to the user as well as the developer.

This would allow us to write expressions like:

width of screen < 800px

I'm trying to implement a way to filter out objects based on the user supplied string. I wanted to string match using a fuzzy match algorithm, but when defining a custom function:

fuzzy(inputString, matchString) {
   // return boolean
}

and setting the filter expression to this (example data object: { name: 'John K', age: 18 }):
fuzzy(name, 'John')

doesn't pass 'John' to the custom function. Is there a way to do this?

It looks that the Makefile refers to some "src/jison.js" file which is not in the repository
so when you call const Jison = require("jison").Jison; it will reach the jison package in node_modules which depends on some nodejs stuff like the 'fs' module:

https://github.com/zaach/jison/blob/master/package.json#L37 (enbf-parser)
-> https://github.com/zaach/ebnf-parser/blob/master/ebnf-parser.js#L2 (ebnf-transform)
-> https://github.com/zaach/ebnf-parser/blob/master/ebnf-transform.js#L2 (transform-parser)
-> https://github.com/zaach/ebnf-parser/blob/master/transform-parser.js#L623 (fs)

that's why import * as filtrex from "filtrex" fails in browser runtime:

This dependency was not found:
⠀
* fs in ./node_modules/ebnf-parser/parser.js, ./node_modules/ebnf-parser/transform-parser.js and 2 others

To install it, you can run: npm install --save fs

is there any way to use the npm package in the browser ?

This issue is similar to #11, but applies to properties escaped using single quotes instead. Because there is no support for escaping single quotes, it is not possible to refer to any properties in the input that have single quotes in them.

When creating an array of 1 value i.e. (42) the return value is just 42 instead of an array of a single element 42

This test passes

it('array support', () => {
        const arr = eval('(42, "fifty", pi)', {pi: Math.PI});

        expect(arr).is.array();
        expect(arr).to.be.equalTo([42, "fifty", Math.PI]);
    });

This test fails

it('array single element', () => {
        const arr = eval('(42)');

        expect(arr).is.array();
        expect(arr).to.be.equalTo([42]);
    });

The current implementation only allows identifier in arrayLiteral syntax. Variants arrayLiteral in identifier and identifier in identifier should also be allowed for completeness.

Dear,

Thanks for sharing this very useful library!

I'm a bit confused about nested object properties.
I pass an object like this:

Which we try to validate using following expression:

msg.payload.open == true

From the readme page I understood this should be possible:

But I might have misinterpreted this??
And indeed while debugging, I see that this library checks whether the object has its own property "msg.payload.open" which is of course not the case:

Which means the nested property cannot be found :-(

However when I change the code to this:

    function prop(name, obj) {
        return name.split('.').reduce((obj, prop) => {
            return obj && obj[prop] ? obj[prop] : undefined;
        }, obj)
    }

    function safeGetter(obj) {
        return function get(name) {
            return name.split('.').reduce((obj, prop) => {
                return obj && obj[prop] ? obj[prop] : undefined;
            }, obj)
        }
    }

Then the nested property can be found without problems:

Thanks !!!
Bart

Add comparisons like

200 > cholesterol > 60
a == b != c == d

Inspired by CoffeeScript.

Hi! I just want to say that I love this package and it's super useful for things like offloading dashboard search and filtering logic to config files so thank you!

My only hangup is that you have to do y of x of config instead of config.x.y which to me feels a bit confusing and counter-intuitive. And it makes it harder to teach others about how to use the expressions because it goes against how you access data in most other languages.

I wonder if it'd be possible to add a filter compile option that allows you to enable dot attribute access? That way we could keep the default functionality, but allow for a more familiar config.x.y to access attributes (instead of key literals) if a user prefers.

A headstart on a solution?

I'm not very familiar with language grammar definitions so I'm sure this isn't all right, but I'm hoping it's in the right direction.

Add ['\\.', 'return ".";'], after this line?

Replace this regex with: '[a-zA-Z$_][a-zA-Z0-9$_]*'

Add ['left', '.'], after

Add ['e . SYMBOL', code(['prop(', 1, ',', 3, ')'])], (I think?)

Dear,

I wanted to test whether an object property is true.
Don't see booleans values mentioned on the readme page, but found an Add booleans pull request.
So I assume this is supported...

However I get the impression that it only works for a boolean as string ("true" or "false")...

Because when I use this expression:

msg.payload.open == true

Then it tries to get a property with the name "true" from my object:

I was wondering if you could do something like this in your code: when the property name is "true" or "false", then it is no property but a boolean. So you simply leave it untouched in the expression, without wrapping it with prop(...)?

P.S. I don't think that "true" or "false" are allowed as property names in JavaScript, when looking at this Stackoverflow discussion. When you click on the Mozilla link in that this discussion you find this:

Additionally, the literals null, true, and false cannot be used as identifiers in ECMAScript.

A JavaScript identifier is a sequence of characters in the code, that identifies a variable, function, or property...

Thanks!!!
Bart

The Jison parser generator makes it hard to write advanced language features (like implicit multiplication, see #39). Maybe, migrating to a modern PEG parser would be worth it. Peggy looks quite good.

While using ones and zeros is fun, returning an actual boolean would make things more robust. It would, for example, allow the developer to detect more potential mistakes on the user's side and warn them.

Strings (surrounding with double quotes) in filtrex expressions cannot contain any double quote characters themselves.

I'll open a PR to address this.

The examples section hasn't changed much since v0. Make it fresh and cool.

Right now, jison builds filtrex parser in the runtime which takes a significant amount of time (approx. 800ms). It is possible to build the parser and save it as source code. This would make the first run of filtrex much faster.

Hi folks,

We are happily using filtrex for quite some time but we see some of our clients complain that the filtrex code is unsecure becuse it uses new Function.

See https://stackoverflow.com/questions/52573756/what-are-the-eval-related-functions-to-be-avoided-when-csp-is-enabled for instance.

Is that something that can be fixed? I guess more and more people will run into this with filtrex eventually as more organisations start to use more strict CSP rules.

Cheers,
Danny

Hi,

I have recently upgraded the version of the filtrex package to 2.0.0 but our previously working code is now giving a TypeError as soon as the app loads.

The error seems to be come from the jison.js file in line 2143:

"push shift unshift forEach some every join sort".split(' ').forEach(function (e,i) {
    setMixin[e] = function () { return Array.prototype[e].apply(this._items, arguments); };
    setMixin[e].name = e;
});

The error stack is as follows:

Uncaught TypeError: Cannot assign to read only property ‘name’ of function ‘function () { return Array.prototype[e].apply(this._items, arguments); }’
   at index-0e1ff3fc.js:5286
   at Array.forEach (<anonymous>)
   at Object.factory [as jison/util/set] (index-0e1ff3fc.js:5284)
   at r (index-0e1ff3fc.js:3150)
   at Object.factory [as jison] (index-0e1ff3fc.js:3166)
   at r (index-0e1ff3fc.js:3150)
   at Module.../../../build/@shared/stencil/esm/index-0e1ff3fc.js (index-0e1ff3fc.js:6477)
   at __webpack_require__ (bootstrap:84)
   at Module.../../../build/@shared/stencil/esm/index-563ba7d7.js (index-563ba7d7.js:1)
   at __webpack_require__ (bootstrap:84)

Any help with this would be appreciated. If you need any further info, feel free to contact me and I will help with what I can.

Is there scope in this project to allow expressions that calculate a return value?

var returnExpression = 'text in response'';
let myReturn = compileReturnExpression(returnExpression);
return myReturn({response: {code: 200, text: 'hi'}}); // returns hi

The core engine should be able to compare numbers, alphabetical strings (with natural sorting) and arrays (element by element, lexicographically, useful for multisort), so that filtrex can be used for custom user-defined sorting.

Using '(' ')' to represent arrays of length 1 does not work and returns the item inside the array instead.

Example usecase:
equals(account_numbers, ("1"))
the function receives 1.
When adding another value to the list, for example ("1", "2")
The function receives [1, 2] as expected.

(equals is a custom function)

Many languages, including Python and CoffeeScript, use keywords if and else instead of the ternary operator ?:. This approach is better aligned with the goals of filtrex. In particular, the CoffeeScript version (if a then b else c) is very close to natural language which makes it layman-friendly.

This change is scheduled as a breaking change in v3.0.0. Any discussion is welcome.

As pointed out here, filtrex has left-associative exponentiation x^y^z == (x^y)^z, while everyone else uses right-associative exponentiation x^y^z == x^(y^z). The current behavior is undocumented and there is no reason to do it the non-standard way. This will be fixed in v3.

There are some old and outdated links in the README, specifically the examples and unit tests

Add an async flag to the options of compileExpression, which will automatically await all promises inside the expression. See #26 on the old repo for discussion.

In filtrex v2, the syntax -x^2 is interpreted as (-x)^2, not as -(x^2). This is contrary to the common order of operation and will be fixed in version 3. Thanks to @​rici for pointing this out.