Kinesis streaming from large feeds - out of memory errors about aws-sdk-ruby HOT 10 CLOSED

Sorry you're running into this! A minimal example would be great, but in the meantime, can you provide some details on how the producers/consumers are structured - eg, are these multiple threads in a single application?

from aws-sdk-ruby.

Yeah I'm sorry about not getting a minimal example up, we have found it difficult to produce one so far...

We currently have 8 active shards in the kinesis feed (4 were split) this is being processed by 4 ruby workers (individual processes running on different kubernetes pods) each consuming 2 shards. These workers are run on kubernetes with 2GB of RAM. This works fine until a burst of data comes through or the consumers slow down (even a little bit) at which point memory usage spikes and the pods get reaped.

from aws-sdk-ruby.

Yeah - creating examples for these is hard. I'm also working on trying to reproduce this...

How are you creating async clients on the workers? Are you sharing a single async client per worker?

Each async client should have a single H2::Connection, which should have only a single socket thread which is reading from the socket - and that thread is what runs any event handlers. So if your event handlers slow down, it should effectively slow down the reads from that socket.

How are you processing events? Can you provide any details?

from aws-sdk-ruby.

We spawn a thread for each shard (2 per worker). Within each of these threads we create an async client and subscribe that thread scoped AsyncClient to consume from 1 shard. We are processing each event as it arrives by passing a block to AsyncClient#subscribe_to_shard.

each thread blocks using async_resp.wait. When that wait step unblocks we allow the thread to die and create a new one.

We have played around with creating a single AsyncClient and passing it to each thread. That change did not stop our kubernetes pods being reaped with memory exceeding 2GB

from aws-sdk-ruby.

Thanks for description - that sounds pretty reasonable. I've been working on creating some similar small test case and was able to do a bit of memory profiling. I'm not 100% confident that this will fix your issue, but it does significantly reduce the number of large string allocations that are being done: #2834 and overall reduces the memory used (as long as you are not setting http_wire_log to true, which I assume you aren't for a production application....).

from aws-sdk-ruby.

@jpaulgs Are you seeing this issue again with the latest version of kinesis and aws-sdk-core 3.170.1?

from aws-sdk-ruby.

@mullermp we'll bump to using that version tomorrow but we are currently keeping up with records in prod so it'll take a little while to confirm.

from aws-sdk-ruby.

Thanks, just let us know. Re-open the ticket if it auto closes and it's still not resolved.

from aws-sdk-ruby.

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.

from aws-sdk-ruby.

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

from aws-sdk-ruby.