Does not support large amounts of streams? about awslogs HOT 4 CLOSED

When you call describe_log_streams from boto you get some useful information back:

{u'firstEventTimestamp': 1428133161913, 
 u'lastEventTimestamp': 1428219579747, 
 u'creationTime': 1428133168061,
 u'uploadSequenceToken': u'XXX', 
 u'logStreamName': u'NAME',
 u'lastIngestionTime': 1428219581611,
 u'arn': u'XXX', u'storedBytes': 2382365}

We could easily make AWSLogs.get_streams don't return as "posible candidate streams" streams which don't fall in the start, end window using firstEventTimestamp and lastIngestionTime.

Regardless of how many streams you have I don't thing listing them should be a problem. As far as we remove (and don't try to retrieve logs) from the ones we know we don't have interesting information, it should be "fine".

from awslogs.

Give it a look to: https://github.com/jorgebastida/awslogs/compare/feature/epic-streams It did the trick to me with a group with around 300 stream (I get 300 are not thousands!) (not all of them had logs in the date rage I was querying) so the number of streams awslogs needs to query for logs was smaller.

Completely useless benchmark:

• master: 320 streams -> 33s
• patch: 320 streams -> 41 streams -> 2s to get all logs

I get this is not a solution per se, but it will help to not waste time querying streams with no useful information.

OT question: It is the case the all of those streams have useful information in the date range?

from awslogs.

Hi,

thank you for your response. I'm sorry for the delay on my behalf.

I tried the upgraded version via pip install. Unfortunately the results are the same. The process gets stuck in trying to retrieve all the streams.

All of the updated streams have useful information in the date range. There are at least thousands of streams that don't have updates. This would make the OrderBy parameter very handy.

Inspired by your work I wrote a proof-of-concept in Node.js by using OrderBy. It seems to work. Let's see if I have time for polishing. However this week Amazon made it possible to subscribe logs through Kinesis so that might be the best solution.

-mfonsen

from awslogs.

I've just pushed a new version (0.1.0) to pypi which should fix this issue. It basically use a new api available in boto3 which merge streams in their end.

https://pypi.python.org/pypi/awslogs/0.1.0

from awslogs.