SEVERE error when creating cloud watch metrics about amazon-kinesis-aggregators HOT 15 CLOSED

Hello,

Sorry to hear you are having this problem. This date convention was scrapped in a really old version of the aggregators package, so I'm concerned that someone you've got an out of date repo. Can you confirm that the jar you are using is .9.2.6.9?

Thx,

Ian

from amazon-kinesis-aggregators.

Actually I am using the command line arguments that are on the main page on github, which actually invoke the AmazonKinesisAggregators.jar-complete.jar JAR and not the amazon-kinesis-aggregators-.9.2.6.9.jar. However, I can confirm that when I cloned the repo, I DO have the 9.2.6.9 JAR...

This is my command I am running:
java -cp AmazonKinesisAggregators.jar-complete.jar -Dconfig-file-path=json-aggregator.json -Dstream-name=TestStream -Dapplication-name=MyApp -Dregion=ap-southeast-2 com.amazonaws.services.kinesis.aggregators.consumer.AggregatorConsumer

from amazon-kinesis-aggregators.

OK - Can you paste an example of the input event?

from amazon-kinesis-aggregators.

are you talking about the input even from the "producer" application (i.e. the payload send to Kinesis) or the record in the DynamoDB table?

If you want to replicate this bug, just run the command with what I pasted earlier, the run the KCL producers sample console app, and put some messages into Kinesis.

from amazon-kinesis-aggregators.

Hi,

Yeah can repro now - couldnt before. Will have a fix to you shortly.

Ian

from amazon-kinesis-aggregators.

Woot, thanks Ian!!

from amazon-kinesis-aggregators.

Fixed in .9.2.7 - now available in the dist folder.

from amazon-kinesis-aggregators.

Also fixed an issue whereby the DevNullDataStore was not used. Now you should NOT get a DynamoDB table created with the configuration you've cited above.

from amazon-kinesis-aggregators.

Thanks Ian, I will be re-trying this very soon

from amazon-kinesis-aggregators.

Hi Ian, doesnt look like this worked... sorry, I pulled the latest and got the new distro's but now I get an error:

Sep 17, 2015 11:05:14 PM com.amazonaws.services.kinesis.aggregators.metrics.CloudWatchMetricsEmitter emit
SEVERE: Unable to Parse Date Value -

The command I am running is this:
java -cp AmazonKinesisAggregators.jar-complete.jar -Dconfig-file-path=json-aggregator.json -Dstream-name=MyStreamName -Dapplication-name=MyApp -Dregion=ap-southeast-2 com.amazonaws.services.kinesis.aggregators.consumer.AggregatorConsumer

It also doesnt look like its creating the tableName that I specified in the config

There is a table called MyApp (from the parameters above), and it has 2 rows, one for each shard in Kinesis (i assume).

Here is a bigger dump of the log when I add events to the Kinesis stream:

INFO: Aggregating 13 records for Kinesis Shard shardId-000000000001
Sep 17, 2015 11:11:54 PM com.amazonaws.services.kinesis.aggregators.StreamAggregator logInfo
INFO: [shardId-000000000001] Aggregation Complete - 13 Records and 13 Elements in 15 ms
Sep 17, 2015 11:11:54 PM com.amazonaws.services.kinesis.aggregators.cache.AggregateCache logInfo
INFO: [shardId-000000000001] Cache Flushed 3 modifications in 0ms
Sep 17, 2015 11:11:54 PM com.amazonaws.services.kinesis.aggregators.metrics.CloudWatchMetricsEmitter emit
SEVERE: Unable to Parse Date Value -

from amazon-kinesis-aggregators.

Ah yes I see - the FOREVER metric was not formatted properly. This should be fixed in .9.2.7.1.

As for the table, you will have a table called MyApp, which is created by the KCL for lease tracking, but when you use the DevNullDataStore you will not have any other tables in DynamoDB.

from amazon-kinesis-aggregators.

OK we have some progress :)

Looks like there is a new cloud watch metric called TestTable, which is good, but it says that the Calculation is SUM when I specified COUNT in the config file?

from amazon-kinesis-aggregators.

Hi,

Yes, that's correct. We have to SUM the eventCount, as this is the only way to add up all the eventCount entries for a period of time in CloudWatch.

Does that make sense?

Ian

from amazon-kinesis-aggregators.

Yeah, it makes sense, I just need to make sure I setup my reporting correctly with the config file to capture the volumes of traffic/events that I want to test under heavy load.

from amazon-kinesis-aggregators.

Closing now

from amazon-kinesis-aggregators.