in_dummy: increasing rate stops working after 55-60 messages/sec about fluent-bit HOT 6 CLOSED

I don't believe the dummy plugin was ever intended to be used for perf testing but I think there are some extra config options you can use to improve things.

@pwhelan may know but you can do a -i dummy --help to see the config options available.

from fluent-bit.

It is much better to use the copies parameter to attempt to test performance since that will create multiple records in a single buffer instead of attempting to submit each record individually. Both parameters can be used together as well to try and create more load.

from fluent-bit.

@cazorla19 We can use counter and flowcounter output plugins to count up records.
https://docs.fluentbit.io/manual/pipeline/outputs/flowcounter

Following configuration is an example to count up records per second.
Some filter/output plugin may affect throughput. Could you try to check in_dummy is bottleneck or not ?

[INPUT]
    name dummy
    Tag  local.system
    Rate  200 # gradually increase from 1 to 50 then to 100 and further
    Dummy  {"message":"hello"}

[OUTPUT]
    name flowcounter
    Match  *
    unit second

from fluent-bit.

@patrick-stephens @pwhelan @nokute78

Thanks everyone for helping me out. With your advices, I was able to resolve the issue and use the dummy plugin for performance testing.

Indeed, the Copies parameter is much more useful than the Rate in my case, I was able to observe the correlation between the copies count increase and the general load increase. In the end, I removed the Rate from my config, because I don't really want to attempt to submit each record individually.

I was also able to see some logs from the flowcounter plugin, which also gave me the proof of the load increase. The only thing I've noticed is that it doesn't work as expected on Windows. While the Linux node worked just fine, the Windows node was launched with the flowcounter, and the fluent-bit even managed to run the tasks, as I saw in the shutdown logs, it just doesn't log the counter.

Before closing this issue as it was resolved, I just wanted to ask if this is the expected behavior from the maintainer's perspective.

from fluent-bit.

I don't know is my answer, @cosmo0920 may know more on the Windows side but I would suggest raising as a separate issue with the easy reproducer you have to make it simpler to find/follow.

from fluent-bit.

ok will do

from fluent-bit.