axiomhq / axiom-cloudfront-lambda Goto Github PK

As we parse a TSV, everything is a string. Figure out what other fields should be non-strings and parse them accordingly.

taken from #13

@a-khaledf :

Default lambda Memory allocation is 128MB, I am afraid if we are batching 10000 events that might sometimes lead to memory exceeding which would crash the lambda function. if we are doing that I would say let's make the batching configurable and add note that memory allocation should be increased relevant to the batching configuration

@seiflotfy:

I tried it already and did some backfilling. memory did not exceed and I was hitting the 10k. But you might be right...
I am making it configurable...
However if we are doing multiple request back to axiom because of the batch sizes I don't think the default 300ms TTL of lambda would suffice!
WDYT?

You can configure the subdirectory for CloudFront logs, so we should make it configurable in the stack(s) too

In the base template when providing your own bucket I believe the template is not properly adding a put object trigger on the s3 bucket. The thing is when I try to add it manually it says the trigger is over lapping.

How many events can be in one file? Maybe batch-ingest every 1k or so.

This way, if people have an existing bucket, they can use our stack to set up the lambda function (including the role and all).