Improved Comparator and Modifier Handling about sigma-go HOT 3 OPEN

Hey @calebstewart! In short, yes, external collaborators would definitely be welcomed :)

but noticed that the comparison logic doesn't really comply with the Sigma specification

Yeah the spec is so large (and sometimes vaguely defined) that I just tried to implement the smallest subset and expand it as we ran into missing features we needed.

The comparator implementation specifically is quite sketchy: I've currently written it as this chaining of functions system but it's not exactly clear to me if that's always valid (e.g. currently you can try to stack contains and re which won't make any sense)

There's also the new gt and lt operators which would be very useful to implement but will require some sort of type coercion

Are you able to share your use cases for custom comparators?

from sigma-go.

Sorry for the delay on getting back to you here!

I think the biggest custom comparator use-case I had was the case where an event field contains an array of data. I don't believe this is covered in the Sigma spec at all, but the logical solution that we use is that a field comparison with an array matches if any of the field elements match the given value (e.g. it's treated as an OR against all field elements). As an example, if we had an event that looked like:

{
  "field": [
    "first",
    "second"
    "third"
  ]
}

And a selector that looked like:

selector1:
  field: "second"

Then it would still match. I'm not sure if that's the same behavior everyone would expect, but that is how we have treated the situation. I imagine there are other corner-cases to the comparison that may be different depending on who is using the library, so that's why I suggested a custom comparison implementation.

What we have ended up doing is implementing a pseudo "compilation" layer on top of this package, which pre-compiles the expression(s) and search(es) to golang coroutines, and then simply executes the coroutines when evaluation is requested for an event. This has a couple benefits:

1. The compiler is flexible. It allows the user to override the core comparison logic prior to compilation as well as add or remove specific modifiers.
2. Patterns in field comparisons can be pre-compiled (e.g. regular expressions) to simplify evaluation.
3. It's faster since each search or expression results in a relatively small and simple coroutine with the static arguments/values/patterns already bound to the coroutine.
4. Allowed us to implement the other comparison features we needed (e.g. keyword matching, faster glob matching, and caseless default comparison).

In the end, we ended up not using the evaluator that is built-in to this library. I would love to eventually discuss merging the compiler upstream, but as it's part of an internal project currently, I'm not sure about the possibility or process there internally. I'll get back to you on that if it's still relevant once our project matures a bit and we can revisit that possibility.

In summary, I don't think it makes sense for me to attempt to implement what I discussed above due to the direction we ended up heading with our project, but I think giving a library user the flexibility to customize the evaluation of events against rules is still valuable. The Sigma specification is geared to the format and intent of the rule itself, and largely ignores direct evaluation since it focuses on converting the rules to a backend which has it's own concrete process for evaluation that differ between backends. With that in mind, the sigma-go evaluator has essentially become a custom "backend" from Sigma's point of view and can define it's own evaluation process separate from the specification (although in this case ideally close to it, obviously).

from sigma-go.

Ah that array use case is absolutely something we're using this library for at @monzo

To support it, sigma-go takes some liberties with Sigma's fieldmapping support and let's you use jsonpath expressions. So for your example you could have a config like so:

fieldmappings:
    field: $.field[*]

Which would then make your example rule work as expected. Internally this is essentially as if you flattened the array and had a config:

fieldmappings:
    field:
        - field_0
        - field_1
        - field_2
        - ....

In the end, we ended up not using the evaluator that is built-in to this library

Ah so you've got something built on top of the AST that this library outputs? Neat! I did wonder about an approach like that: evaluators optimised for speed in specific cases vs the current generic evaluator

Appreciate the challenges of open sourcing what you've implemented but, if you're able, I'd love to chat privately about what you've built! Sounds very cool. I'm [email protected] if you're up for it 🙂

from sigma-go.