Add new metric processor about connect HOT 13 CLOSED

Added: 18ba0ab
Released: https://github.com/Jeffail/benthos/releases/tag/v0.26.2

Thanks @bigkraig!

from connect.

That sounds interesting, we could add something like a metric processor that allows you to mutate either a counter, gauge or timing metric, and the value you specify in configuration could use function interpolation (https://github.com/Jeffail/benthos/blob/master/docs/config_interpolation.md#functions) so that either you specify a static value, or extract an integer value either from metadata, a json path, etc.

If I add that as a processor you could then use a https://github.com/Jeffail/benthos/tree/master/docs/processors#conditional processor to make it conditional based on the type/contents of the message.

It could look something like this:

type: conditional
conditional:
  condition:
    type: jmespath
    jmespath:
      query: 'doctor.error == "people died"'
  processors:
  - type: metric
    metric:
      type: counter
      path: people.that.have.died
      operator: increment
      value: ${!json_field:doctor.people.killed}

Or, the shorthand version might just be:

type: metric
metric:
  type: counter
  path: people.that.have.died
  operator: increment
  value: ${!json_field:doctor.people.killed}

In which case if doctor.people.killed doesn't exist, isn't a number, or returns 0 it's a no-op.

from connect.

Cool, you could use the file output and set the path to /dev/null, we use that a lot for testing pipelines.

from connect.

Hey @bigkraig, would the goal here be to extract the value of a field from a document and change a metric by that amount? So, for example, for a document {"foo":{"bar":4}} we might increment the counter stats.foo_bars by 4?

from connect.

I guess it could go a few ways.

I may want to count the amount of time I see a field (like a 4xx error) or I may want to expose the value of them, like a transaction total. If I think even further if someone is reading in their logs with response times, they may want to do a histogram.

Obviously it would make sense to just instrument the application, but we have quite a lot of legacy products where that would be difficult to do.

from connect.

@Jeffail I think the labels should get included at this point, its going to be kind of limiting without them as I try to apply this to some use cases.

It doesn't look difficult to modify but I'm not sure what should be done with the statsd and other metric outputs. I'd have to change the metrics.Type interface which takes a ...string parameter. From what I can see its always joining with ., so it may not be needed, otherwise I can put the labels as a parameter before ...path

from connect.

@bigkraig, let's crack some eggs. The original intention of path ...string was to eventually migrate all my components away from dot separated paths, but I never actually got around to doing it as it worked fine just replacing dot separators with underscores for prometheus. I'm going to create an experimental branch to play around with different interfaces that include labels map[string]string somewhere. If we're happy we can develop your processor on top of that branch and then merge once it's tested.

from connect.

@Jeffail lets go 🙌

from connect.

https://github.com/Jeffail/benthos/blob/feature/add-metrics-labels/lib/metrics/type.go#L80

I'm mostly copying the Prometheus API, where you create a metric with a path and a list of label names, then you create an aggregator with label values, then you interact with that aggregator.

You'll be able to do:

mFooCounter := stats.GetCounterVec("path.here", []string{"name","something_else"})
mFooCounter.With("label1_value", "label2_value").Incr(1)

from connect.

d883c91 :)

from connect.

Dope, I'm gonna sleep, can you give it a go and make sure it behaves as you'd expect? Also I just squashed the commits but I slipped a cheeky variadic into the With call so it's now With(labelValues ...string).

from connect.

@Jeffail its looking really promising. have a good night!

from connect.

Maybe i'm missing it, but I am trying to find a way to drop the messages after I have processed them. Since I am only consuming to generate prometheus metrics, I don't need to do anything with them afterwards. It looks like there isn't a /dev/null option?

Otherwise I think the current iteration of the patch may be good enough. I am at the point where I am deploying it to Kubernetes to let it run and see what may need changing.

from connect.