etsy / logster Goto Github PK
View Code? Open in Web Editor NEWParse log files, generate metrics for Graphite and Ganglia
License: Other
Parse log files, generate metrics for Graphite and Ganglia
License: Other
The error message "WARNING Failed to get lock. Is another instance of logster running?" is rather vague.
It would be helpful if the lock file that could not be acquired is outputted so it is easier to manually delete the lock file if necessary/
I forked logster and added new feature for amazon cloudwatch.
And I used this feature from last year on my production service.
Until now, I used this feature alone but I'd like to share code.
Plz review this idea.
Hi
After sys.path.append("/usr/share/logster") has been removed from logster and logster/parsers has been moved
I can't just call
sudo make install
I have to run
python setup.py install instead
Unfortunately, logster doesn't work on Windows due to the dependency on fcntl
. You do intend to support Windows, don't you? (Judging by what I've read about the use of pygtail
.)
Would it be OK to use portalocker
instead of fcntl
? Or use it optionally, e.g. logster --tailer pygtail --locker portalocker
?
(logster)travis@travis-mint ~/new_hk/scaleHarness/agent $ logster -d --state-dir=.logster.state --output=graphite --graphite-host=dc1-graphite01:2003 --metric-prefix=hk.travis-mint --parser-options=logs/agent-0.log x.GrinderLogster logs/agent-0-data.log
Traceback (most recent call last):
File "/home/travis/venv/logster/bin/logster", line 5, in
pkg_resources.run_script('logster==0.0.1', 'logster')
File "/home/travis/venv/logster/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 499, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/home/travis/venv/logster/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 1235, in run_script
execfile(script_filename, namespace, namespace)
File "/home/travis/venv/logster/lib/python2.7/site-packages/logster-0.0.1-py2.7.egg/EGG-INFO/scripts/logster", line 381, in
main()
File "/home/travis/venv/logster/lib/python2.7/site-packages/logster-0.0.1-py2.7.egg/EGG-INFO/scripts/logster", line 297, in main
parser = getattr(module, parser_name)(option_string=options.parser_options)
TypeError: 'module' object is not callable
I can import x.GrinderLogster
(logster)travis@travis-mint ~/new_hk/scaleHarness/agent $ python
Python 2.7.3 (default, Sep 26 2012, 21:51:14)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import x.GrinderLogster
Hi!
I just wanted to note somewhere that I am maintaining a fork of logster for the Wikimedia Foundation over at https://github.com/wikimedia/operations-debs-logster. I've made a few changes and fixes as you can see here: https://github.com/wikimedia/operations-debs-logster/commits/master. One of the reasons I haven't submitted pull requests, is this: https://github.com/wikimedia/operations-debs-logster/commit/8023713369dadf206459ef74dbad18b60181de32. It was a slight restructuring to allow use of the core logster stuff as a module, rather than having the core in the executable script. I wasn't sure how this would be received.
If Etsy would like me to submit our changes as one big pull request for consideration, I'd be happy to do so, just let me know :)
If i just want to publish metrics then i have to log:
$logger->info(" METRIC_COUNT metric=foo value=1 ");
Why is this necessary? couldn't the regexp allow no spaces as well? This is a source of bugs in our code where metrics don't get published because people forget to add spaces around them.
Right now I am parsing when the server has been restarted. From the logfile I have extracted the timestamp. I would like to use this timestamp in stead of the timestamp when MetricObject is created.
Thx in advance
Is it planned to support influxdb ? As currently done for graphite
Hi there,
Not a bug but a feature request. :)
Would be convenient to have logger object available in parser classes for debugging purposes.
Apologies if this is not the right forum.
Inspired by the Etsy talk at Velocity, I'm analysing a few Apache files with logster and the total run (admittedly on a VM on my desktop) takes around 3 minutes sometimes, depending on how smoothly our servers flush their log volume. As a result I was getting warnings when running logster every minute. I've switched to every 5, but now I end up with gaps in my data in Graphite. I've included my invocation of logster below (I've renamed SampleLogster to ApacheLogster). Am I missing something?
/usr/sbin/logster --output=graphite --graphite-host=graphite:2003 -p apache.dc1.liveapp125 ApacheLogster /home/vmplanet/dws/dc1/liveapp125/apache/access.log
Thanks in advance, Mark
Trying to write a new FooLogster class, I got no output at all when testing the simplest log files.
Then, I realized these messages in the log:
WARNING /usr/sbin/logtail2 -f foo.log -o run/logtail-logster.parsers.[...].foo.state returned bad exit code None
In https://github.com/etsy/logster/blob/master/bin/logster#L388 the check of retval is wrong; according to the python docs, the return code of popen().close() is None if the command exited with status code zero.
Edit: I must admit that my analysis of the problem was incomplete. I realized that logster makes no output at all on the first run. So this warning is just spurious.
I am running Log4jLogster on my log file. If I write code in init function then the code changes are reflecting in output but when I try to change Parse_Line function then it does not have any effect. It seems as if it is not able to read the log file being passed in command line. Please help me out so that I can keep using Logster with my large sized logs.
I've got a Logging setup going that pumps data to Logstash -> Elasticsearch -> Kibana.
Instead of having to connect Logster to the raw log sources, it would imho make sense/be great (?) to be able to tail Logstash instead.
Has someone attempted this, or am I completely missing the point?
In logster itself, end_locking(), the fcntl.LOCK_NB is unwelcome in the following call on Solaris:
fcntl.flock(lockfile_fd, fcntl.LOCK_UN | fcntl.LOCK_NB)
Using the following lets the code run at least:
fcntl.flock(lockfile_fd, fcntl.LOCK_UN)
The flock call in start_locking() works fine with LOCK_NB, of course:
fcntl.flock(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
How would you like me to handle a patch for this?
isn't working ๐
copying build/lib/logster/parsers/__init__.py -> build/bdist.linux-x86_64/egg/logster/parsers
copying build/lib/logster/parsers/ErrorLogLogster.py -> build/bdist.linux-x86_64/egg/logster/parsers
copying build/lib/logster/parsers/stats_helper.py -> build/bdist.linux-x86_64/egg/logster/parsers
copying build/lib/logster/parsers/Log4jLogster.py -> build/bdist.linux-x86_64/egg/logster/parsers
copying build/lib/logster/parsers/SquidLogster.py -> build/bdist.linux-x86_64/egg/logster/parsers
copying build/lib/logster/parsers/PostfixLogster.py -> build/bdist.linux-x86_64/egg/logster/parsers
copying build/lib/logster/parsers/MetricLogster.py -> build/bdist.linux-x86_64/egg/logster/parsers
copying build/lib/logster/parsers/SampleLogster.py -> build/bdist.linux-x86_64/egg/logster/parsers
byte-compiling build/bdist.linux-x86_64/egg/logster/__init__.py to __init__.cpython-32.pyc
byte-compiling build/bdist.linux-x86_64/egg/logster/logster_helper.py to logster_helper.cpython-32.pyc
byte-compiling build/bdist.linux-x86_64/egg/logster/parsers/__init__.py to __init__.cpython-32.pyc
byte-compiling build/bdist.linux-x86_64/egg/logster/parsers/ErrorLogLogster.py to ErrorLogLogster.cpython-32.pyc
File "build/bdist.linux-x86_64/egg/logster/parsers/ErrorLogLogster.py", line 55
raise LogsterParsingException, "regmatch failed to match"
^
SyntaxError: invalid syntax
byte-compiling build/bdist.linux-x86_64/egg/logster/parsers/stats_helper.py to stats_helper.cpython-32.pyc
byte-compiling build/bdist.linux-x86_64/egg/logster/parsers/Log4jLogster.py to Log4jLogster.cpython-32.pyc
File "build/bdist.linux-x86_64/egg/logster/parsers/Log4jLogster.py", line 80
raise LogsterParsingException, "regmatch failed to match"
^
SyntaxError: invalid syntax
byte-compiling build/bdist.linux-x86_64/egg/logster/parsers/SquidLogster.py to SquidLogster.cpython-32.pyc
File "build/bdist.linux-x86_64/egg/logster/parsers/SquidLogster.py", line 89
raise LogsterParsingException, "regmatch failed to match"
^
SyntaxError: invalid syntax
byte-compiling build/bdist.linux-x86_64/egg/logster/parsers/PostfixLogster.py to PostfixLogster.cpython-32.pyc
File "build/bdist.linux-x86_64/egg/logster/parsers/PostfixLogster.py", line 61
except Exception, e:
^
SyntaxError: invalid syntax
byte-compiling build/bdist.linux-x86_64/egg/logster/parsers/MetricLogster.py to MetricLogster.cpython-32.pyc
byte-compiling build/bdist.linux-x86_64/egg/logster/parsers/SampleLogster.py to SampleLogster.cpython-32.pyc
File "build/bdist.linux-x86_64/egg/logster/parsers/SampleLogster.py", line 72
raise LogsterParsingException, "regmatch failed to match"
^
SyntaxError: invalid syntax
creating build/bdist.linux-x86_64/egg/EGG-INFO
installing scripts to build/bdist.linux-x86_64/egg/EGG-INFO/scripts
running install_scripts
running build_scripts
creating build/bdist.linux-x86_64/egg/EGG-INFO/scripts
copying build/scripts-3.2/logster -> build/bdist.linux-x86_64/egg/EGG-INFO/scripts
changing mode of build/bdist.linux-x86_64/egg/EGG-INFO/scripts/logster to 755
copying logster.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying logster.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying logster.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying logster.egg-info/not-zip-safe -> build/bdist.linux-x86_64/egg/EGG-INFO
copying logster.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
creating 'dist/logster-0.0.1-py3.2.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing logster-0.0.1-py3.2.egg
removing '/opt/logster/vev/lib/python3.2/site-packages/logster-0.0.1-py3.2.egg' (and everything under it)
creating /opt/logster/vev/lib/python3.2/site-packages/logster-0.0.1-py3.2.egg
Extracting logster-0.0.1-py3.2.egg to /opt/logster/vev/lib/python3.2/site-packages
File "/opt/logster/vev/lib/python3.2/site-packages/logster-0.0.1-py3.2.egg/logster/parsers/ErrorLogLogster.py", line 55
raise LogsterParsingException, "regmatch failed to match"
^
SyntaxError: invalid syntax
File "/opt/logster/vev/lib/python3.2/site-packages/logster-0.0.1-py3.2.egg/logster/parsers/Log4jLogster.py", line 80
raise LogsterParsingException, "regmatch failed to match"
^
SyntaxError: invalid syntax
File "/opt/logster/vev/lib/python3.2/site-packages/logster-0.0.1-py3.2.egg/logster/parsers/SquidLogster.py", line 89
raise LogsterParsingException, "regmatch failed to match"
^
SyntaxError: invalid syntax
File "/opt/logster/vev/lib/python3.2/site-packages/logster-0.0.1-py3.2.egg/logster/parsers/PostfixLogster.py", line 61
except Exception, e:
^
SyntaxError: invalid syntax
File "/opt/logster/vev/lib/python3.2/site-packages/logster-0.0.1-py3.2.egg/logster/parsers/SampleLogster.py", line 72
raise LogsterParsingException, "regmatch failed to match"
^
SyntaxError: invalid syntax
logster 0.0.1 is already the active version in easy-install.pth
Installing logster script to /opt/logster/vev/bin
Installed /opt/logster/vev/lib/python3.2/site-packages/logster-0.0.1-py3.2.egg
Processing dependencies for logster==0.0.1
Finished processing dependencies for logster==0.0.1
(vev)[soporte@v455uprod logster]$ logster
Traceback (most recent call last):
File "/opt/logster/vev/bin/logster", line 5, in <module>
pkg_resources.run_script('logster==0.0.1', 'logster')
File "/opt/logster/vev/lib/python3.2/site-packages/pkg_resources.py", line 534, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/opt/logster/vev/lib/python3.2/site-packages/pkg_resources.py", line 1434, in run_script
execfile(script_filename, namespace, namespace)
File "/opt/logster/vev/lib/python3.2/site-packages/pkg_resources.py", line 55, in execfile
exec(compile(open(fn).read(), fn, 'exec'), globs, locs)
File "/opt/logster/vev/lib/python3.2/site-packages/logster-0.0.1-py3.2.egg/EGG-INFO/scripts/logster", line 194
print "%s %s %s" %(metric.timestamp, metric_name, metric.value)
^
SyntaxError: invalid syntax
With logtail not being available everywhere and often a perl implementation, I suggest implementing an embedded python solution which should also simplify process handling etc. Below is a snippet what could be a useful starting point.
with open(filename) as log_fd:
offset_filename = os.path.join(OFFSET_ROOT_DIR,filename)
if not os.path.exists(offset_filename):
os.makedirs(os.path.dirname(offset_filename))
with open(offset_filename, 'w') as offset_fd:
offset_fd.write(str(0))
with open(offset_filename, 'r+') as offset_fd:
log_fd.seek(int(offset_fd.readline()) or 0)
new_logrows_handler(log_fd.readlines())
offset_fd.seek(0)
offset_fd.write(str(log_fd.tell()))
Looks like there is a typo in a recent commit which leads to the following syntax error:
$ logster --dry-run --output=graphite --graphite-host=localhost:2003 -p test SampleLogster /opt/log/something.txt
Traceback (most recent call last):
File "/usr/local/bin/logster", line 4, in <module>
__import__('pkg_resources').run_script('logster==0.0.1', 'logster')
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 735, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 1651, in run_script
code = compile(source, script_filename, 'exec')
File "/usr/local/lib/python2.7/dist-packages/logster-0.0.1-py2.7.egg/EGG-INFO/scripts/logster", line 170
if (options.log_conf)
^
SyntaxError: invalid syntax
hi, this commit b172276 breaks the compatibility for python2
if (not self.dry_run):
s.sendall(bytes("%s\n" % metric_string, 'ascii'))
else:
print("%s %s" % (self.graphite_host, metric_string))
>>> bytes('foo', 'ascii')
Traceback (most recent call last):
File "<input>", line 1, in <module>
TypeError: str() takes at most 1 argument (2 given)
Logster does not understand --statsd-host
even when using --output=statsd
.
me@my-macbook ~/Source $ virtualenv foo
me@my-macbook ~/Source $ source foo/bin/activate
(foo) me@my-macbook ~/Source $ git clone https://github.com/etsy/logster && cd logster
(foo) me@my-macbook ~/Source/logster $ logster -d --statsd-host=foo:8125 --output=statsd SampleLogster /path/to/file.log
Usage: logster [options] parser logfile
logster: error: no such option: --statsd-host
(foo) me@my-macbook ~/Source/logster $ logster -d --output=statsd SampleLogster /path/to/file.log
Usage: logster [options] parser logfile
...
StatsdOutput options:
--statsd-host=STATSD_HOST
Hostname and port for statsd collector, e.g.
statsd.example.com:8125
Usage: logster [options] parser logfile
logster: error: You must supply --statsd-host when using 'statsd' as an output type.
This appears to have regressed sometime between the introduction of the statsd (68e26f3) and the recent commit.
This issue does not occur if I use 68e26f3.
Hi everybody!
The Wikimedia foundation is a happy user of logster, we use it to push Varnishkafka json logs to graphite and visualize useful metrics. While reviewing our traffic via tcpdump for an unrelated issue with noticed a ton of traffic towards our DNS resolvers for the statsd host passed to logster. From what I can see in https://github.com/etsy/logster/blob/master/logster/outputs/statsd.py#L23-L36 it seems that if you don't pass a raw ip to --statsd-host
then the udp_sock.sendto
call will trigger a DNS query to solve the domain.
We are using the 0.0.1 version but the issue seems still present in master. Would it be possible to add an command line option to cache the resolution of the IP address for X seconds?
Luca
Hi everyone,
I'm using logster and graphite in order to visualize the code response of HTTP. But there is a problem : size of file log grows very fast, so I want remove old lines . I tried by a script shell, it keep only 5000 lines and remove all old lines . But it didn't work. Is there a solution for this one ?
Thanks and sorry my poor english ^^
Hello,
I'm issuing logster with the following command
logster --dry-run --debug --output=stdout --log-dir=log --state-dir=run myparser.CustomParser log.txt
The content of myparser.py is
from logster.logster_helper import MetricObject, LogsterParser
from logster.logster_helper import LogsterParsingException
import re
import logging
class CustomParser(LogsterParser):
def __init__(self, option_string=None):
self.lines = [MetricObject('ip', '0.0.0.0')]
self.reg = re.compile('(?P<ipaddress>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) - - \[(?P<dateandtime>\d{2}\/[a-z]{3}\/\d{4}:\d{2}:\d{2}:\d{2} (\+|\-)\d{4})\] ((\"(GET|POST) )(?P<url>.+)(http\/1\.1")) (?P<statuscode>\d{3}) (?P<bytessent>\d+) (["](?P<refferer>(\-)|(.+))["]) (["](?P<useragent>.+)["])')
def parse_line(self, line):
self.lines.push(MetricObject('ip', '127.0.0.1'))
try:
regMatch = self.reg.match(line)
if regMatch:
logging.debug(regMatch.groupdict())
else:
raise LogsterParsingException("regmatch failed to match")
except Exception as e:
raise LogsterParsingException("regmatch or contents failed with %s" % e)
def get_state(self, duration):
return self.lines
However for some reason, parse_line does not get called.
$ wc -l log.txt
10 log.txt
$ logster --dry-run --debug --output=stdout --log-dir=log --state-dir=run myparser.CustomParser log.txt
1430448601 ip 0.0.0.0
I hope someone can help.
Thank you.
When using MetricLogster parser, results for counts are different than I'd expect. Here's a simple example:
$ rm ~/logs/sample.log
$ echo " METRIC_COUNT metric=some.metric.count value=4 " >> ~/logs/sample.log
$ echo " METRIC_COUNT metric=some.metric.count value=4 " >> ~/logs/sample.log
$ ~/.virtualenvs/logster/bin/logster --log-dir=/home/asdf/logster --debug --state-dir=/home/asdf/logster --output=stdout MetricLogster /home/asdf/logs/sample.log
1427841578 some.metric.count 0.421052631579 # expect value to be 8
$ rm ~/logs/sample.log
$ echo " METRIC_COUNT metric=some.metric.count value=4 " >> ~/logs/sample.log
$ echo " METRIC_COUNT metric=some.metric.count value=6 " >> ~/logs/sample.log
$ echo " METRIC_COUNT metric=some.metric.count value=6 " >> ~/logs/sample.log
$ ~/.virtualenvs/logster/bin/logster --log-dir=/home/asdf/logster --debug --state-dir=/home/asdf/logster --output=stdout MetricLogster /home/asdf/logs/sample.log
1427841596 some.metric.count 0.75 # expect value to be 16
Is something going wrong with the call to get_state in MetricLogster? I don't totally understand that logic yet. Thanks!
I noticed that passing multiple --output arguments is supported, however, there seems to be a bug when used with a prefix. In my test, I attempted to send to stdout, statsd, and graphite. Below is the shell output & logster.log data for one test without a prefix, and one with.
I'm not strong on python, but I would guess that the bug is somehow related to the multiple calls to submit_XXX(metrics, ...) actually modifying the contents of the metrics data when adding the prefix.
Thank you for looking into this.
[cosmo:/tmp]$ logster --output=stdout --output=statsd --output=graphite --graphite-host=127.0.0.1:2003 --statsd-host=127.0.0.1:8125 --state-dir=/tmp/logster --debug SampleLogster /tmp/access.log 1377652162 http_1xx 0.0 1377652162 http_2xx 0.0 1377652162 http_3xx 0.0 1377652162 http_4xx 0.0 1377652162 http_5xx 0.0 [cosmo:/tmp]$ logster --output=stdout --output=statsd --output=graphite --graphite-host=127.0.0.1:2003 --statsd-host=127.0.0.1:8125 --state-dir=/tmp/logster -p BUG --debug SampleLogster /tmp/access.log 1377652194 BUG_BUG.http_1xx 0.0 1377652194 BUG_BUG.http_2xx 0.0 1377652194 BUG_BUG.http_3xx 0.0 1377652194 BUG_BUG.http_4xx 0.0 1377652194 BUG_BUG.http_5xx 0.0 [cosmo:/tmp]$
and the corresponding /var/log/logster/logster.log
2013-08-27 20:09:22,492 INFO Executing parser logster.parsers.SampleLogster.SampleLogster on logfile /tmp/access.log 2013-08-27 20:09:22,492 DEBUG Using state file /tmp/logster/logtail-logster.parsers.SampleLogster.SampleLogster-tmp-access.log.state 2013-08-27 20:09:22,493 DEBUG Locking successful 2013-08-27 20:09:22,493 DEBUG Setting duration to 437.0 seconds. 2013-08-27 20:09:22,512 DEBUG Submitting Graphite metric: http_1xx 0.0 1377652162 2013-08-27 20:09:22,512 DEBUG Submitting Graphite metric: http_2xx 0.0 1377652162 2013-08-27 20:09:22,513 DEBUG Submitting Graphite metric: http_3xx 0.0 1377652162 2013-08-27 20:09:22,513 DEBUG Submitting Graphite metric: http_4xx 0.0 1377652162 2013-08-27 20:09:22,513 DEBUG Submitting Graphite metric: http_5xx 0.0 1377652162 2013-08-27 20:09:22,514 DEBUG Submitting statsd metric: http_1xx:0.0|g 2013-08-27 20:09:22,514 DEBUG Submitting statsd metric: http_2xx:0.0|g 2013-08-27 20:09:22,514 DEBUG Submitting statsd metric: http_3xx:0.0|g 2013-08-27 20:09:22,515 DEBUG Submitting statsd metric: http_4xx:0.0|g 2013-08-27 20:09:22,515 DEBUG Submitting statsd metric: http_5xx:0.0|g 2013-08-27 20:09:22,515 INFO Total execution time: 0.0 seconds. 2013-08-27 20:09:22,515 DEBUG Unlocking successful 2013-08-27 20:09:54,159 INFO Executing parser logster.parsers.SampleLogster.SampleLogster on logfile /tmp/access.log 2013-08-27 20:09:54,159 DEBUG Using state file /tmp/logster/logtail-logster.parsers.SampleLogster.SampleLogster-tmp-access.log.state 2013-08-27 20:09:54,159 DEBUG Locking successful 2013-08-27 20:09:54,160 DEBUG Setting duration to 32.0 seconds. 2013-08-27 20:09:54,180 DEBUG Submitting Graphite metric: BUG.http_1xx 0.0 1377652194 2013-08-27 20:09:54,180 DEBUG Submitting Graphite metric: BUG.http_2xx 0.0 1377652194 2013-08-27 20:09:54,180 DEBUG Submitting Graphite metric: BUG.http_3xx 0.0 1377652194 2013-08-27 20:09:54,180 DEBUG Submitting Graphite metric: BUG.http_4xx 0.0 1377652194 2013-08-27 20:09:54,180 DEBUG Submitting Graphite metric: BUG.http_5xx 0.0 1377652194 2013-08-27 20:09:54,181 DEBUG Submitting statsd metric: BUG.BUG_BUG.http_1xx:0.0|g 2013-08-27 20:09:54,181 DEBUG Submitting statsd metric: BUG.BUG_BUG.http_2xx:0.0|g 2013-08-27 20:09:54,181 DEBUG Submitting statsd metric: BUG.BUG_BUG.http_3xx:0.0|g 2013-08-27 20:09:54,181 DEBUG Submitting statsd metric: BUG.BUG_BUG.http_4xx:0.0|g 2013-08-27 20:09:54,181 DEBUG Submitting statsd metric: BUG.BUG_BUG.http_5xx:0.0|g 2013-08-27 20:09:54,181 INFO Total execution time: 0.0 seconds. 2013-08-27 20:09:54,181 DEBUG Unlocking successful
I create a file, call it "logfile" with just 3 lines in it. "ParserLogster.py" contains nothing but a print(line) in parse_line().
Running
logster --output=stdout ParserLogster logfile
Does not produce any output the first time. Running it again prints out all the lines.
I've created empty file for test and run logster like this:
logster --dry-run -o stdout MetricLogster /tmp/diff_log
Then I've put data into test log file as in example:
METRIC_COUNT metric=some.metric.count value=1
METRIC_COUNT metric=some.metric.count value=2.2
and run logster again and it gave me this output:
some.metric.count 0.2
It's written that COUNT metrics are totalled which should give 3.2.
Why it's not working as intended ?
version:
ii logster 0.0.1-1 all Generate metrics from logfiles for Graphite and G
Hi,
I am trying to test out the latest version of logster currently. I have copied the bin file and and running the same command instead on the file which is called logsternew.
It keeps bringing back the below error that its unable to find the parser modules but they do exist as when running it with the old file it runs ok.
Is there anywhere the specify the location of the parsers?
Details:
File "/usr/local/graphite/sources/logster/logsternew", line 358, in main
module = import(module_name, globals(), locals(), [parser_name])
ImportError: No module named logster.parsers.LsofcountLogster
Can logster analyze iptables? Count of certain IP, port, build total tables?
The README states:
Logster depends on the "logtail" utility that can be obtained from the logcheck package, either from a Debian package manager or from source:
But this actually uses the logtail2 executable by default, which is provided by the 'logtail' package (at least in Ubuntu Precise).
As mentioned in #92, I've asked Enthought to include logster (and pygtail and portalocker) in a Canopy repo. They've agreed to do so but would appreciate if you could tag a version. For obvious reasons they don't want to pick an arbitrary point (would you bless one instead?) and the constant version number of 0.0.1 isn't too convenient for them either.
Thanks,
Sam
I was not able to locate any kind of logster mailing list, so I'm entering this as a ticket. Apologies if this is the wrong place for this question.
I'm not a python expert, but looking at the logster code, it seems like the first time logster reads through a large log file, all the data in the log will be sent as a single data point to graphite. Successive log file reads (presumably via cron) will then continue to generate new graphite data points more or less in realtime. Is that correct? What I want to do is read in older log files that might be days (or more) old, without having them collapse down into a single data point.
What is the take on this? Am I'm correct that this is not currently supported? If so, is there interest in accepting an implementation of this feature as a patch?
Thanks!
-Travis
I have recently tested most of the log parsers and none of them seem to work. The output just returns 0. If any one is actively working on this project do let me know.
File "/usr/sbin/logster", line 58, in
from logster.logster_helper import LogsterParsingException, LockingError
ImportError: No module named logster.logster_helper
Please help
Thanks
Ankit Agarwal
The default format of the error log has changed from something like
[Thu May 12 08:28:57.652118 2011] [error] ...
to
[Thu May 12 08:28:57.652118 2011] [core:error] ...
Note the new module info. The ErrorLogLogster parser should be extended to understand this (i.e. probably ignore the module and successfully extract the severity).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.