Stuck in waitOnParentShard after resharding about vmware-go-kcl HOT 4 CLOSED

A possible solution is to change the code in https://github.com/vmware/vmware-go-kcl/blob/master/clientlibrary/worker/shard-consumer.go#L256 adding input.Checkpointer.Checkpoint(nil):

if getResp.NextShardIterator == nil {
	log.Infof("Shard %s closed", shard.ID)
	if err := input.Checkpointer.Checkpoint(nil); err != nil {
		return err
	}
	shutdownInput := &kcl.ShutdownInput{ShutdownReason: kcl.TERMINATE, Checkpointer: recordCheckpointer}
	sc.recordProcessor.Shutdown(shutdownInput)
	return nil
}

The problem with this solution is that if ProcessRecords fails somehow, the checkpoint is set to SHARD_END regardless of that failure. I think ProcessRecords would be the right place to set the SHARD_END checkpoint (but it's missing the knowledge to do it) or its return value must be checked (returning an error) before checking whether the shard is closed, because if SHARD_END is put in the checkpointer but ProcessRecords failed, we're probably going to lose data

from vmware-go-kcl.

The library has been tested well for resharding scenario. It does not require restart.
Also, make sure to do checkpointing during shutting down. See example:
https://github.com/vmware/vmware-go-kcl/blob/master/test/worker_test.go#L300

from vmware-go-kcl.

Thanks for the hint on the shutdown, that's exactly what I was looking for! 👍

As per the resharding scenario, we decided to shutdown instances after a resharding not for the resharding itself, but to balance the shards between kcl instances.
Some details: we have long living kcl instances reading from the kinesis streams. Those instances live on AWS EC2 spot servers and, hence, those servers can be killed sometimes due to their spot nature. Without shard stealing we found out that instances with longer lives will read from more shards over time, leading to unbalanced works between servers.
So we decided to use MaxShards based on (n.shards / n. instances) + 1 so that all kcl instances will always read from the ~same number of shards over time.
Since the number of shards changes during a resharding operation, we quit all instances when a resharding happens and then, during the new startup, the correct MaxShards value gets calculated and passed to kcl.

from vmware-go-kcl.

I often found automatic resharding is very annoying and using MaxShards is much easier. See
#4 for more discussion.

Glad to know the issue has been resolved.

from vmware-go-kcl.