keeprocking / pygelf Goto Github PK

Hi there

I'm wanting to push GELF over the Internet and so want to use graylog-server's TLS support on it's GELF port - but I can't find a client that supports TLS (besides their java graylog-collector)

Would it be possible to add TLS support to pygelf?

Hello.
The error appear in this place: https://github.com/keeprocking/pygelf/blob/master/pygelf/gelf.py#L40

Traceback (most recent call last):
  File "/usr/lib/python3.5/logging/handlers.py", line 621, in emit
    s = self.makePickle(record)
  File "/usr/local/lib/python3.5/dist-packages/pygelf/handlers.py", line 86, in makePickle
    return self.convert_record_to_gelf(record)
  File "/usr/local/lib/python3.5/dist-packages/pygelf/handlers.py", line 34, in convert_record_to_gelf
    gelf.make(record, self.domain, self.debug, self.version, self.additional_fields, self.include_extra_fields),
  File "/usr/local/lib/python3.5/dist-packages/pygelf/gelf.py", line 41, in make
    gelf['full_message'] = '\n'.join(traceback.format_exception(*record.exc_info))
TypeError: format_exception() argument after * must be an iterable, not bool

In documentation:

If exc_info does not evaluate as false, it causes exception information to be added to the logging message. If an exception tuple (in the format returned by sys.exc_info()) or an exception instance is provided, it is used; otherwise, sys.exc_info() is called to get the exception information.

May be add check?

Currently GelfHttpHandler and GelfHttpsHandler are not utilising handleError which violates python documentation about writing own handlers.

Solution:

def emit(record):
    try:
         ... # logic here
    except:
        self.handleError(record)

Hi there

On the graylog website there's a link to pygelf - but it's not this pygelf!

https://marketplace.graylog.org/addons/77b60e7c-f041-4b2d-86d0-378884f387ec

It looks like someone else has a similar app - although it only supports HTTP

https://github.com/tgalyean/pygelf

Pistols at dawn perhaps? :-)

If Python3's QueueHandler/QueueListener are used then record.exc_info is
None, but record.exc_text contains the traceback. Reading the source of
the logging module it would appear that exc_text can safely be assumed
to contain the exception text if it is not None, so can be used to populate
graylog's full_message.

#18 contains an attempt to fix this

Feature introduced in PR#47 #47 seems to fail.

Python version 3.8
OS: Windows 10
PYGELF Version: 0.4.1

Example of log handler used:

    gray_handler = GelfTcpHandler(
        host='1.2.3.4',
        port='12201',
        include_extra_fields=True,
        debug=True,
        static_fields={'api': 'my-api', 'env': 'test'},
        additional_env_fields={'env': 'FLASK_ENV'}
    )

Error message:

--- Logging error ---
Traceback (most recent call last):
  File "C:\bin\Python\Python38\lib\logging\handlers.py", line 631, in emit
    s = self.makePickle(record)
  File "...\lib\site-packages\pygelf\handlers.py", line 62, in makePickle
    return self.convert_record_to_gelf(record) + b'\x00'
  File "...\lib\site-packages\pygelf\handlers.py", line 42, in convert_record_to_gelf
    gelf.make(record, self.domain, self.debug, self.version, self.additional_fields, self.additional_env_fields, self.include_extra_fields),
  File "...\lib\site-packages\pygelf\gelf.py", line 60, in make
    for name, env in additional_env_fields:
ValueError: too many values to unpack (expected 2)
Call stack:

Problematic code:

for name, env in additional_env_fields.items():

Hi there

I guess pygelf is written for python-2.7. Under python-2.6, those classes aren't liked - they generate the following error

Traceback (most recent call last):
File "/usr/local/bin/filelogs-to-gelf2", line 76, in
handle = GelfTcpHandler(gelf_server, 12201,0,0)
File "/usr/lib/python2.6/site-packages/pygelf/tcp.py", line 19, in init
super(GelfTcpHandler, self).init(host, port)
TypeError: super() argument 1 must be type, not classobj

The fix is simple: it just needs to be defined as a "object". ie

class GelfTcpHandler(SocketHandler,object):

Apparently (this is what google told me) under python-2.7 "object" is the default, whereas under 2.6 it has to be declared?

Anyway, making that change makes pygelf work on CentOS-6 (and any other python-2.6 system I guess)

When using GelfUdpHandler with a hostname (rather than an IP address) as a destination, it gets resolved by DNS on every log message, which is a big performance concern. This is behaviour inherited from Python's DatagramHandler (see https://bugs.python.org/issue47149), but presumably pygelf could work around it by resolving the hostname itself on startup.

I am using pygelf to send logs to graylog

this is my code

self.gelfhandler = GelfTcpHandler(host=str(host), port=int(port),include_extra_fields=True)

and then i am sending the log entry ( already converted to gelf)

self.gelfhandler.send(json.dumps(message))

but I am not getting any error if graylog port is not open or suddenly is closed inbetween.

Currently, there is no way to specify which fields from the log record I want to skip and which ones I want to keep.
This would be a nice improvement.

Would you be open to supporting a timestamp format as specified by RFC 5424 (section 6.2.3) for readability? I know that this is not specified by GELF 1.1 so I would suggest to make it configurable using a timestamp_format='rfc-3339' parameter.

Related standards: ISO 8601, RFC 3339.

Ref: severb/graypy#99

Hi, I just started to use this package recently, and I found there's a possible change to make this package more flexible on configuration.

Currently I configured gelf handler in django settings.py like this:

handlers =  {
            'level': 'INFO',
            'class': 'pygelf.GelfUdpHandler',
            'host': GRAYLOG_ADDR,
            'port': GRAYLOG_PORT,
            '_extra': 'myExtra', 
            '_one_more': 'myOneMore'
        }

But I don't want my handlers to be a giant dictionary with a bunch of messed-up fields. So I want to pull these things out to an independent extra_fields_dict and pass it in handlers like blew:

extra_fields_dict = {
    '_extra': 'myExtra',
    '_one_more': 'myOneMore'
}

handlers =  {
            'level': 'INFO',
            'class': 'pygelf.GelfUdpHandler',
            'host': GRAYLOG_ADDR,
            'port': GRAYLOG_PORT,
            'additional_fields': extra_fields_dict
        }

I assume this will not be a difficult change, so I just wonder if you can add this flexibility to configuration.

as a test, im not including any certs and have validate to false without any luck. Switching to a TCP handler does work though. Correct me if im wrong, but wouldnt a TLS handler act the same as a TCP handler if validate is set to false?

maybe describe on the README how to set full_message? if it's possible.
or make it possible if it's not?

Hi guys,

what I needed is the feature that logs created in my pyhton code (e.g. "logger.info("whatever")") are converted into the GELF format and then sent via a Kafka bus to graylog (Gelf Kafka Input). I found this module here and it provided nearly exactly what I needed - except the "GelfKafkaHandler".

I was thinking of a additional Handler and were glad to see that such an additional handler was very easy to implement. I wrote some additional lines of code and tested it on a local setup - it works !

So now - please let me know what you are thinking! Is this a sensible feature?

Looking forward to get your feedback.

Hi there

I was debugging an unrelated issue and detected tonnes of DNS lookups for the hostname I was running pygelf on (permanent process, converting squid logs into GELF in realtime). It looks to me like pygelf is doing a DNS lookup for $HOSTNAME for every GELF record it generates? That's a nasty overhead if you're generating 100+/sec streams of GELF data

I don't know for sure, but could it be the "'source': socket.getfqdn()" in gelf.py that's doing that?

Shouldn't that be replaced by defining that as a variable at startup, and then just reusing that variable afterwards? I know the hostname could be put into /etc/hosts, but actually strace showed the DNS lookups occurring before /etc/hosts was opened - so I don't even think that would help - and even that's an overhead that could be skipped if it only did it once?

http://docs.graylog.org/en/2.2/pages/gelf.html#gelf-payload-specification

This is causing my log files to be filled with thousands of warning messages.

Hi, GELF says nothing about logger name but IMHO field you call "source" should be named "host" per "http://docs.graylog.org/en/2.1/pages/gelf.html" and "source" should be logger name.

Anyway i think logger name should be added as additional field per default like other python handler handle it.

EDIT: btw i found this one "https://github.com/severb/graypy" and they write client host in "source" too =)

As pointed out in #1 (comment), HTTPS support should be added

After some debugging time, I think I just came to the following conclusion: compress=True cannot work with TCP and TLS handlers, due to the GELF message structure.

In particular, I was experiencing lots of message lost in simple (non-load) test scenarios using TCP and TLS handlers. Instead, delivery was fine when switching to UDP or when disabling compression. I think that this is due to GELF using \x00 as a message separator, and the zlib compression introducing such bytes as artifacts of the deflation.

To prevent such subtle losses in the future, I think the GelfTcpHandler and GelfTlsHandler should be changed to ignore the compress parameter and enforce it to False for all streams.

I noticed the log level is logged as a number. but the README seems to promise the level name.

What do you think about an option to send the name instead of the level number? Its less ambiguous IMO.

It seems that socket.getfqdn() (handlers.py) falls back to assigning the first FQDN it finds in /etc/hosts. So when a message comes into graylog, the source is localhost.localdomain (for people that haven't edited /etc/hosts manually). Making a call to socket.gethostname() returns the system's hostname as you'd expect to see it, so you know where the messages are actually coming from.

I'm happy to create a PR for the change.

If it'd be a breaking change (I don't see how it would be), maybe a hint in the document would save some time and Googling.

Very cool tool, by the way! Saved me a ton of headaches:)

In the last line of that sample:

logging.info('hello gelf')

You should use your defined logger to log info, not the logging module. So I request to fix this typo to:

logger.info('hello gelf')

just in case if some else went into the wrong place.

Sorry newbie here. I get the following which seems like

--- Logging error --- Traceback (most recent call last): File "/usr/lib/python3.6/logging/handlers.py", line 633, in emit s = self.makePickle(record) File "/usr/local/lib/python3.6/dist-packages/pygelf/handlers.py", line 58, in makePickle return self.convert_record_to_gelf(record) + b'\x00' File "/usr/local/lib/python3.6/dist-packages/pygelf/handlers.py", line 38, in convert_record_to_gelf gelf.make(record, self.domain, self.debug, self.version, self.additional_fields, self.include_extra_fields), File "/usr/local/lib/python3.6/dist-packages/pygelf/gelf.py", line 37, in make 'level': LEVELS[record.levelno], KeyError: 5

Which seems like this...

"enumeration" type dict doesn't handle the event that a Level 5 is thrown. I don't know the specifics here but is this possibly a bug?

I would clone the repo and propose a PR but i'm still new to GH and would prefer to avoid the cognitive burden while I'm working on this task.

I'm trying to use gelf to log to seq as shown in https://docs.datalust.co/docs/using-python
But I couldn't find how to pass a api key in any documentation

I'd like to validate the certificate but I'm using a legit cert issued by Let's Encrypt so I don't need to specify a custom CA certs bundle.

Is there no way to just use the operating system's CA files?