Recording a lot of ProvisionedThroughputExceededException about vmware-go-kcl HOT 1 CLOSED

Each KCL app only handle one stream with multiple shards. If there are 5 streams, you need to create 5 KCL app and give different app name which will create multiple DynamoDB tables.

KCL does not have a way to handle individual shard. If you need to directly read from any specific shard, you need to use AWS go-sdk directly instead.

For ProvisionedThroughputExceededException, it is because the KCL reads Kinesis stream too fast which mean there isn't any data in to stream. So, you do need to configure IdleTimeBetweenReadsInMillis to put delay between empty read.

If you read record each time, you must have very faster processor. Then, you need to make WithMaxRecords a larger number to get more data in a batch.

If all of them do not work, you need to split shard. Kinesis has fix capacity for one shard. Need to split shard whenever exceeding its capacity.

from vmware-go-kcl.