Consumer from given sequence num about vmware-go-kcl HOT 4 CLOSED

I don't think KCL support configuring the initial position at all.

Here is AWS KCL config:
https://github.com/awslabs/amazon-kinesis-client/blob/master/amazon-kinesis-client/src/main/java/software/amazon/kinesis/common/InitialPositionInStream.java

vmware-go-kcl follows AWS's KCL:
https://github.com/vmware/vmware-go-kcl/blob/master/clientlibrary/config/config.go#L46

To make your case work, you can set starting position to TRIM_HORIZON and skip all previous record and start processing until reach the sequence num.
https://github.com/vmware/vmware-go-kcl/blob/master/test/worker_test.go#L69

from vmware-go-kcl.

Thank you for your reply!
I think aws kcl do have the ability to consume from initial point.
https://docs.aws.amazon.com/zh_cn/streams/latest/dev/kcl2-standard-consumer-python-example.html

 def checkpoint(self, checkpointer, sequence_number=None, sub_sequence_number=None):
        """
        Checkpoints with retries on retryable exceptions.

        :param amazon_kclpy.kcl.Checkpointer checkpointer: the checkpointer provided to either process_records
            or shutdown
        :param str or None sequence_number: the sequence number to checkpoint at.
        :param int or None sub_sequence_number: the sub sequence number to checkpoint at.
        """
        for n in range(0, self._CHECKPOINT_RETRIES):
            try:
                checkpointer.checkpoint(sequence_number, sub_sequence_number)
                return
            except kcl.CheckpointError as e:
                if 'ShutdownException' == e.value:
                    #
                    # A ShutdownException indicates that this record processor should be shutdown. This is due to
                    # some failover event, e.g. another MultiLangDaemon has taken the lease for this shard.
                    #
                    print('Encountered shutdown exception, skipping checkpoint')
                    return

from vmware-go-kcl.

I don't think KCL support configuring the initial position at all.

Here is AWS KCL config:
https://github.com/awslabs/amazon-kinesis-client/blob/master/amazon-kinesis-client/src/main/java/software/amazon/kinesis/common/InitialPositionInStream.java

vmware-go-kcl follows AWS's KCL:
https://github.com/vmware/vmware-go-kcl/blob/master/clientlibrary/config/config.go#L46

To make your case work, you can set starting position to TRIM_HORIZON and skip all previous record and start processing until reach the sequence num.
https://github.com/vmware/vmware-go-kcl/blob/master/test/worker_test.go#L69

Another question, From my view of scene, TRIM_HORIZON means the oldest msg which I never read before in given consumer group. When I want to debug, I may want to read from some checkpoint serveral time. Does it mean I should change my consumer group name and dynamodb table name?

from vmware-go-kcl.

For your first question, it is an interesting solution but I don't know which shard you want to work on. The normal use case for KCL is for multi-shard stream.

If you want to debug, you can manually overwrite the value of checkpoint inside dynamodb table via AWS console UI. Or, manually call the function to checkpoint back to old location.

from vmware-go-kcl.