This connector is for reading data from Amazon Kinesis and writing the data to Apache Kafka.
name=connector1
tasks.max=1
connector.class=com.github.jcustenborder.kafka.connect.kinesis.KinesisSourceConnector
# Set these required values
aws.secret.key.id=
aws.access.key.id=
kafka.topic=
kinesis.stream=| Name | Description | Type | Default | Valid Values | Importance |
|---|---|---|---|---|---|
| aws.access.key.id | aws.access.key.id | string | high | ||
| aws.secret.key.id | aws.secret.key.id | password | high | ||
| kafka.topic | The kafka topic to write the data to. | string | high | ||
| kinesis.stream | The Kinesis stream to read from. | string | high | ||
| kinesis.shard.id | The shard of the Kinesis stream to read from. This is a regex which can be used to read all of the shards in the stream. | string | .* | high | |
| kinesis.empty.records.backoff.ms | The number of milliseconds to backoff when the stream is empty. | long | 5000 | [500,...,2147483647] | medium |
| kinesis.position | The position in the stream to reset to if no offsets are stored. | string | TRIM_HORIZON | ValidEnum{enum=ShardIteratorType, allowed=[AT_SEQUENCE_NUMBER, AFTER_SEQUENCE_NUMBER, TRIM_HORIZON, LATEST, AT_TIMESTAMP]} | medium |
| kinesis.record.limit | The number of records to read in each poll of the Kinesis shard. | int | 500 | [1,...,10000] | medium |
| kinesis.region | The AWS region for the Kinesis stream. | string | US_EAST_1 | ValidEnum{enum=Regions, allowed=[GovCloud, US_EAST_1, US_EAST_2, US_WEST_1, US_WEST_2, EU_WEST_1, EU_WEST_2, EU_CENTRAL_1, AP_SOUTH_1, AP_SOUTHEAST_1, AP_SOUTHEAST_2, AP_NORTHEAST_1, AP_NORTHEAST_2, SA_EAST_1, CN_NORTH_1, CA_CENTRAL_1]} | medium |
| kinesis.throughput.exceeded.backoff.ms | The number of milliseconds to backoff when a throughput exceeded exception is thrown. | long | 10000 | [500,...,2147483647] | medium |
A partition key is used to group data by shard within a stream.
| Name | Optional | Schema | Default Value | Documentation |
|---|---|---|---|---|
| partitionKey | true | String | A partition key is used to group data by shard within a stream. The Streams service segregates the data records belonging to a stream into multiple shards, using the partition key associated with each data record to determine which shard a given data record belongs to. Partition keys are Unicode strings with a maximum length limit of 256 bytes. An MD5 hash function is used to map partition keys to 128-bit integer values and to map associated data records to shards. A partition key is specified by the applications putting the data into a stream. Identifies which shard in the stream the data record is assigned to. See Record.getPartitionKey() |
The unit of data of the Amazon Kinesis stream, which is composed of a sequence number, a partition key, and a data blob. See Record
| Name | Optional | Schema | Default Value | Documentation |
|---|---|---|---|---|
| sequenceNumber | true | String | The unique identifier of the record in the stream. See Record.getSequenceNumber() | |
| approximateArrivalTimestamp | true | Timestamp | The approximate time that the record was inserted into the stream. See Record.getApproximateArrivalTimestamp() | |
| data | true | Bytes | The data blob. See Record.getData() | |
| partitionKey | true | String | A partition key is used to group data by shard within a stream. The Streams service segregates the data records belonging to a stream into multiple shards, using the partition key associated with each data record to determine which shard a given data record belongs to. Partition keys are Unicode strings with a maximum length limit of 256 bytes. An MD5 hash function is used to map partition keys to 128-bit integer values and to map associated data records to shards. A partition key is specified by the applications putting the data into a stream. Identifies which shard in the stream the data record is assigned to. See Record.getPartitionKey() | |
| shardId | true | String | A shard is a uniquely identified group of data records in a stream. A stream is composed of one or more shards, each of which provides a fixed unit of capacity. Each shard can support up to 5 transactions per second for reads, up to a maximum total data read rate of 2 MB per second and up to 1,000 records per second for writes, up to a maximum total data write rate of 1 MB per second (including partition keys). The data capacity of your stream is a function of the number of shards that you specify for the stream. The total capacity of the stream is the sum of the capacities of its shards. | |
| streamName | true | String | The name of the Kinesis stream. |
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent