proposal: refactor conditions to support operator configuration about connect HOT 5 CLOSED

Hey @cludden, I'm open to it but would be worried about the rate at which the config spec might grow as we add more operators. As an alternative suggestion, what if we changed the type of arg to be an interface{}?

Also for the example of adding a json_schema operator to jmespath I think it might be more generally useful to have something similar to the process_field processor for conditions, maybe called something like check_field, where the config might look something like this:

condition:
  type: check_field
  check_field:
    path: foo
    conditions:
    - type: json_schema
      json_schema:
        type: object
        ... etc ...

That way we could recycle it for doing things like text conditions on a JSON field.

from connect.

that seems reasonable. I think my only concern with the interface{} approach is the lack of built in documentation for operator configuration and the addition of a lot of type coercion / reflection in each of the operator factories which may clutter the code base unnecessarily. are you imaging something like the json processor's value field?

and the json_schema operator was a made up example. at the moment I really only need enum support for both metadata and json payloads, and was hoping to avoid building gigantic configurations by combining an or condition with a long list of subconditions. I do however really like the idea of a check_field condition. Also, what are your thoughts on adding operator support to the jmespath condition? currently it acts as a bool operator (foo == 'bar'), but I think it could be useful to allow some other common operators (regexp_partial, regexp_exact, less_than, greater_than, enum)

from connect.

Sorry if I posted this already elsewhere, I thought I did but can't seem to find it. Anyways, I started looking at this and it's unclear to me how a check_field condition would pass the extracted json values to child conditions. Currently conditions expect to receive a types.Message as opposed to something like []byte. The only way I can currently think to implement this without a significant refactor would be to wrap the extracted bytes as a new message and pass this to child conditions, which seems kind of hacky. I could be missing something obvious, but It seems more straight forward to me to either a) expand the jmespath condition or b) add a new json_field condition.

Either of these could be built/refactored to support a list of operators (and in the case of the jmespath condition, refactor the existing implementation to act as a bool operator). These could then implement many of the common operators shared by the text and metadata conditions but could expand to include additional json specific operators like json_schema.

from connect.

I imagine it could work similar to process_field: https://github.com/Jeffail/benthos/blob/master/lib/processor/process_field.go#L149. I can do it when I get a chance, made a new issue for it: #88

from connect.

Closing this, lets continue any further conversation in #88

from connect.