how to inprove gelf-forwarder performance about gelf-forwarder HOT 4 CLOSED

7、gelf-forwarder tcp ouput testing result
source--> gelf-forwarder -->graylog tcp gelf

qps is 17898.86

[17:24:54][root@38d15 ~]# ./vegeta -cpus 16 attack -targets tatget.txt.bak -body pay.json -timeout=20s -rate 18000 -duration=60s | tee results.bin | ./vegeta report
[17:25:54]Requests [total, rate, throughput] 1079999, 17999.88, 17898.86
[17:25:54]Duration [total, attack, wait] 1m0s, 1m0s, 2.871ms
[17:25:54]Latencies [min, mean, 50, 90, 95, 99, max] 198.944µs, 7.645ms, 931.316µs, 18.815ms, 29.835ms, 105.36ms, 1.464s
[17:25:54]Bytes In [total, mean] 0, 0.00
[17:25:54]Bytes Out [total, mean] 1511714400, 1399.74
[17:25:54]Success [ratio] 99.44%
[17:25:54]Status Codes [code:count] 0:203 200:1073989 429:5807

from gelf-forwarder.

Did you modify channel-buffer-size option (CHANNEL_BUFFER_SIZE env var or via flag)?

Default is 100 which might cause a lot of HTTP 429 errors. You might want to increase it at the cost of higher memory usage during stress.

I'll also consider adding an option to disable backpressure which might remove HTTP 429 errors at the cost of higher latencies during stress.

from gelf-forwarder.

You can also now try out version v0.4.2 which introduces option to disable backpressure, for example:

./gelf-forwarder --backpressure=0

BACKPRESSURE=0 ./gelf-forwarder

If your source doesn't support backpressure then it's probably the best option.

from gelf-forwarder.

Add this option channel-buffer-size=20000 and BACKPRESSURE=0, it get the little bit more result : qps is 18303 and the ES DISK IO is 60%. It time to upgrade the single ES to cluster ES.

thank you !

from gelf-forwarder.