jeremydowens / aws-glue-streaming-example Goto Github PK

I need to investigate what happens when we change the behavior of a running job. Does it need to restart? If it does restart, do we risk losing processing on a portion of the stream?

This is much better suited to a "launch stack" button. I'll get that going today.

I have already incorporated some of the items from https://github.com/Gamesight/aws-glue-local-scala to ensure that we can locally compile this, and verify compilation before updated the s3 resources in the action, but there are some additional handling pieces that we need to cover. I want to be able to run this locally.

• Research mocking Kinesis streams. I think there is some solid Java tooling already out there for this, but I need to look it up.
• Set test files
• I could potentially set up a test bucket for output and read the existing stream. This is a reasonable option if mocking proves to be difficult. The project already assumes that the IoT device is sending the MQTT packet
• Figure out how to test without starting a never-ending job. The biggest thing to test is whether or not it correctly reads 100 seconds worth of the stream and joins properly. We don't want this to continually wait for more data. I may need to add some additional boilerplate to the job that short-circuits after a single "batch"

Right now, the s3 bucket in the job is handled as a hard coded string in the job script. This should be provided as an argument and handled in the cloudformation with a reference to the bucket we are creating.

A lot of the listed policies in the Cloudformation are excessively permissive. I need to lock that down.