Big memory consumption about clickhouse_sinker HOT 3 CLOSED

We are java shop, however we're brainstorming some ideas... Could it be, that kafka sends too many records to consumer in one consumer request (that would explain -why problem only occurs when we have a big backlog of messages to process in kafka)

In java could limit Kafka consumer with
properties.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG,MAXKAFKAFETCH);

Now checking if something similar could be done in go.

https://godoc.org/github.com/Shopify/sarama#Config
// The maximum number of message bytes to fetch from the broker in a
// single request. Messages larger than this will return
// ErrMessageTooLarge and will not be consumable, so you must be sure
// this is at least as large as your largest message. Defaults to 0
// (no limit). Similar to the JVM's fetch.message.max.bytes. The
// global sarama.MaxResponseSize still applies.
Max int32

from clickhouse_sinker.

I think the memory consumption problem may be here. https://github.com/housepower/clickhouse_sinker/blob/master/input/kafka.go#L72.

There is a big channel. Yet we can reduce this channel size, we make it be 1024 in production.
clickhouse_sinker still takes lots of memory because it must buffer the records to batch insert into ClickHouse. There are some ways to reduce the memory usage, such as use row binary format, but it throws the load to clickhouse server.

In our production env, 256GB machine * 36 instances of clickhouse_sinker and other services. It takes about 10% memory to consume 100 tasks.

I don't think it's a memory leak, the memory will not grow endless, it grows up to a cap limit.

from clickhouse_sinker.

Sorry.. No, it's was not a memory leak (continous memory grow). It was just excessive memory usage... However the fix that you commited helped us to cut down 2GB memory limit with OOM to 700MB.

That means this fix works and is great!

from clickhouse_sinker.