http server timing out about connect HOT 3 CLOSED

Hey @sathiya-mit, there's a general guide for configuring pipelines for maximum throughput here: https://github.com/Jeffail/benthos/blob/master/docs/concepts.md#maximising-io-throughput

Kafka specifically benefits a lot from batching.

from connect.

thanks @Jeffail It worked. I did both batching and increased number of parallel output sinks. I tested up to 200 parallel http connection.

BTW is that make any sense to increase http input source like mentioned below

input:
type: broker
broker:
copies: 4
inputs:
- type: http_server
http_server:
address: ""
cert_file: ""
key_file: ""
path: /post
timeout_ms: 20000
ws_path: /post/ws
processors:
- type: combine
combine:
parts: 10
buffer:
type: none
none: {}

from connect.

@sathiya-mit, great! Just use a single http_server, it's a special case as each request handler will be running on its own goroutine, so you wouldn't benefit from having multiple instances, they would also overwrite each others' endpoints.

from connect.