Consider batching incoming partition messages before flushing them together to partition segment about tank HOT 2 OPEN

Imagine having 1,000 different clients all consistently producing 1 message (say 500 some messages/second) to the same Tank broker -- that would mean the Tank broker would need to execute over 500k write() calls (excluding any possible writes for updating indices) in order to write those messages to the current segment.

If instead we would collect all writes for each distinct segment received in the same poll events loop and then combine and apply them outside the loop and then notify the clients, that would perhaps work great, and shouldn't have been that hard to implement.

from tank.

We should eventually support buffering published messages on the client, and flush them all as a single bundle when the buffered messages count exceeds a threshold, or a long time has passed since the first message was buffered.

This should be configurable on a per partition basis. It obviously means that if the application doesn't get to flush the data in-time, the data will be lost. It will still be useful if you can tolerate that possibility and/or the thresholds are kept low.

Alternatively, you could e.g buffer that into a file and periodically flush that file to Tank.

from tank.