Hi ,

I am reading records from Kinesis Stream and writing it into S3 Bucket with below configuration in the properties file.

Name of the control table in DynamoDB

appName = KCLAPPLICATION

Source

kinesisInputStream = dev-test-kinesis-s3_AUGUST2017

Target

s3Bucket = dev-test-kinesis-s3-17aug-aug

Your AWS region

regionName = ap-southeast-2
retryLimit = 3

Flush buffer to S3 when any of these conditions is met:

bufferByteSizeLimit = 2097152

bufferMillisecondsLimit = 720000

bufferRecordCountLimit = 500

Though , i have given the above configuration for bufferByteSizeLimit ,bufferMillisecondsLimit, bufferRecordCountLimit ,but still i can see only single file in S3 bucket.

For Eg. if i am reading 2000 from kinesis which is around size of 2MB and which takes 2 minutes to process , so accordingly from the above configuration , i should have received 4 files in amazon S3 Bucket , each will contain 500 records.

But in S3 i am only getting single file with 2000 records only. i.e. these parameters are not working as intended . Same is for bufferMillisecondsLimit and bufferByteSizeLimit as well.

Could you please suggest , if still there is another configuration needs to be done in configuration file to make it split.

Great job on this project @codeflows

I've just set this up and have it running in production. I am thinking about contributing to this project. Is this something you'd be open to?

Specifically because I am running this in production I was thinking about scalability and error handling.

• Scalability
  • What happens when 1 process cannot manage the load?
  • Can we run more than 1 process a time?
• Error handling
  • What happens when there's an error?
  • Can anyone be notified of it?
  • Will data be lost?