Comments (4)
When you call describe_log_streams
from boto you get some useful information back:
{u'firstEventTimestamp': 1428133161913,
u'lastEventTimestamp': 1428219579747,
u'creationTime': 1428133168061,
u'uploadSequenceToken': u'XXX',
u'logStreamName': u'NAME',
u'lastIngestionTime': 1428219581611,
u'arn': u'XXX', u'storedBytes': 2382365}
We could easily make AWSLogs.get_streams
don't return as "posible candidate streams" streams which don't fall in the start
, end
window using firstEventTimestamp
and lastIngestionTime
.
Regardless of how many streams you have I don't thing listing them should be a problem. As far as we remove (and don't try to retrieve logs) from the ones we know we don't have interesting information, it should be "fine".
from awslogs.
Give it a look to: https://github.com/jorgebastida/awslogs/compare/feature/epic-streams It did the trick to me with a group with around 300 stream (I get 300 are not thousands!) (not all of them had logs in the date rage I was querying) so the number of streams awslogs
needs to query for logs was smaller.
Completely useless benchmark:
- master: 320 streams -> 33s
- patch: 320 streams -> 41 streams -> 2s to get all logs
I get this is not a solution per se, but it will help to not waste time querying streams with no useful information.
OT question: It is the case the all of those streams have useful information in the date range?
from awslogs.
Hi,
thank you for your response. I'm sorry for the delay on my behalf.
I tried the upgraded version via pip install. Unfortunately the results are the same. The process gets stuck in trying to retrieve all the streams.
All of the updated streams have useful information in the date range. There are at least thousands of streams that don't have updates. This would make the OrderBy parameter very handy.
Inspired by your work I wrote a proof-of-concept in Node.js by using OrderBy. It seems to work. Let's see if I have time for polishing. However this week Amazon made it possible to subscribe logs through Kinesis so that might be the best solution.
-mfonsen
from awslogs.
I've just pushed a new version (0.1.0) to pypi which should fix this issue. It basically use a new api available in boto3 which merge streams in their end.
https://pypi.python.org/pypi/awslogs/0.1.0
from awslogs.
Related Issues (20)
- Traceback bug :( HOT 1
- allow sorting groups by date created and/or filter on groups created after some time
- Changing the format of the logs HOT 2
- How do I use hyphenated characters in filter pattern properties? HOT 3
- Is this project still maintained or has it been abandoned? HOT 3
- running awslogs under working sso profile HOT 8
- Found bug
- You’ve found a bug!
- Bug when running while-loop
- Trace when getting a log and grepping for ERROR when AWS token expired
- awslogs starting issue
- awslogs streams finds no stream HOT 1
- Log streams with square brackets cannot be used
- Allow awslogs to be directly executed with python3 -m
- awslogs get /aws/batch/job *d7a784ac4ac440d98d0dcef4f63378c3
- after brew install: ModuleNotFoundError: No module named 'six' HOT 5
- awslogs command doesn't work HOT 2
- You've found a bug HOT 2
- NoRegionError when default region defined in config file
- Traceback error logs dump when SSO token is expired
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from awslogs.