Workload skewed across workers about vmware-go-kcl HOT 3 CLOSED

It looks like this is mitigated somewhat with the .WithLeaseStealing(true) chained function when setting up your config.NewKinesisClientLibConfig(). It doesn't get to the point of "perfect" parity, but it does help somewhat. I was trying to avoid setting .WithMaxLeasesForWorker(maxLeasesPerWorker) because if the kinesis stream scales in-flight, that would have to be re-evaluated...

The current suggestion in another issue is to have a polling loop checking for the shard count, and possibly just exiting when it changes. That way the process will come back up in a container orchestration scenario with the new values, and work out anyway. I would like to see a solution that is a bit more graceful...

from vmware-go-kcl.

In your case, set WithMaxLeasesForWorker(8) to put a cap on the workload for workers.
https://github.com/vmware/vmware-go-kcl/blob/master/test/worker_test.go#L76

from vmware-go-kcl.

Yes, that is what I ended up doing. I understand that the behavior of this is meant to coincide with AWS's java version of the KCL, and it would act the same way and require me to use the same setting.

My original request probably would need to go to AWS to have them create a balanced mode that does this load balancing automatically without me needing to change settings and restart clients to apply those settings when some scaling happens.

I can live with restarting my clients whenever my kinesis stream OR my clients scale horizontally to redistribute the load. I just think it could be done more gracefully in runtime without as much interruption and restarting needed.

from vmware-go-kcl.