Out-of-Memory errors when sinking large topics about s3-connector-for-apache-kafka HOT 4 OPEN

After some investigation we managed to confirm that the out-of-memory errors are due to the connector consuming from the Kafka topics faster than it writes to S3. This happens especially in the cases when the topics already contain a lot of data.

We found that using the offset_flush_interval_ms configuration property we can control how often the connector writes to S3. By default, this is done once every 60 seconds. Setting this to a much lower value (5 seconds or even 1 second) triggers the writing to S3 much quickly. Therefore, memory is being freed up on time.

Nonetheless, the main problem remains: If the connector needs to copy large amounts of data and for some reason writing to S3 takes up more time (maybe also due to other reasons unrelated to the configured flush interval), the connector will consume the entire topic in memory, thus, overloading the Kafka Connect cluster.

We would like to request a new feature that allows us to configure a maximum size of the input records buffer. When this size is reached, the Kafka consumer should pause and wait until memory has been freed up.
In this way, we can easily configure the maximum memory consumption of each connector and avoid overloading the cluster.

from s3-connector-for-apache-kafka.

Out-of-memory logs:
[kafka-development-connect-6]2024-01-16T13:41:47.138064[kafka-connect]Terminating due to java.lang.OutOfMemoryError: Java heap space

from s3-connector-for-apache-kafka.

@vpapanchev thanks for reporting this issue!
Agree with your assessment. This is something we were aware of and the workaround described is the current alternative to deal with this issue. Nonetheless, your feature request is valid as there should be a better way to avoid OOM on this connector.

Let us know if you are planning to work on this, otherwise we will add it to our backlog

from s3-connector-for-apache-kafka.

Thank you @jeqo for the response.

I am not currently planning on working on this, so please add it to your backlog. I would appreciate it if you give an update here once you start working on it.

If it's a known issue, then you might be able to help me with my current struggles :)
So far, we were only able to configure the offset_flush_interval_ms property on the Kafka Connect cluster itself. Do you know if it's possible to configure it per connector, i.e., whether a connector is able to override the value configured in the cluster?

We enabled the parameter https://docs.aiven.io/docs/products/kafka/kafka-connect/reference/advanced-params#connector-client-config-override-policy for the Kafka Connect cluster by setting its value to “All”. I then tried to configure a particular offset_flush_interval_ms for an S3 Connector using various properties, such as:

• offset.flush.interval.ms
• admin.override.offset.flush.interval.ms
• consumer.override.offset.flush.interval.ms
• kafka_connect.offset.flush.interval.ms
• override.kafka_connect.offset.flush.interval.ms
• override.offset.flush.interval.ms
• ...
  None of it seemed to work..

If you have any suggestions, that would be great!

Kind regards,
Vasil

from s3-connector-for-apache-kafka.