joaodrp / gelf-formatter Goto Github PK

Problem:
Despite the fact that exception should be captured, when
doing so in a django request handler, gelf-formatter produces
a logging error caused by

TypeError: Object of type WSGIRequest is not JSON serializable

Solution:
This can be fixed by converting value to string for non json serialzable objects:
log_record[_prefix(key)] = str(value)

To test against the newest python version, it should be added to tox.ini and .travis.yml

Hey! My team is looking for exactly something with the idea of just have the gelf formatter to use with fluentd on k8s, which seems to be exactly the purpose of this library, but I wonder if the project is still maintained because of the long open PRs and issues

In #1 a new functionality was added to make it possible to ignore specific attributes at the root level. Example:

attrs = ["secret", "password"]

formatter = GelfFormatter(ignored_attrs=attrs)
# or
formatter.ignored_attrs = attrs

logging.debug("app config", extra={"connection": "local", "secret": "verySecret!", "mysql": {"user": "test", "password": "will_be_logged"}})

{"version": "1.1", "short_message": "app config", "timestamp": 1557346554.989846, "level": 6, "host": "my-server", "_connection": "local", "_mysql": {"user": "test", "password": "will_be_logged"}}

This issue is to propose the ability of ignoring fields at any depth. In the example above, this would allow us to ignore the mysql/password attribute as well.

One way to achieve this is to accept a special syntax on the ignored_attrs list, where the depth level is represented by a dot ., e.g.:

attrs = ["secret", "mysql.password"]

The format function would then need to inspect the extra values taking into account nested fields.

Hi!

Firsly gelf-formatter is awesome, I don't understand how other projects even justify blocking entire program execution to send logs over network...

WIth that said, tailing logs on server is bit more complicated with gelf format and I wanted to add a human readable timestamp. Fortunately this awesome library supports adding allowed_reserved_attrs, however asctime does not seem to work, whereas other attributes work.

This is the configuration:
gelf_formatter = GelfFormatter(allowed_reserved_attrs=["processName", "asctime", "filename", "created"]

Log line:
{"version": "1.1", "short_message": "beat: Waking up in 5.00 minutes.", "timestamp": 1576665042.5591564, "level": 7, "host": "playsignage-int-2019", "_filename": "beat.py", "_created": 1576665042.5591564, "_processName": "MainProcess"}

The log line contains extra attrs, such as _processName, _filename and _created, however asctime is missing...any ideas?

Hello!

Seems that there is no way to change output of this formatter because it simply calls json.dumps(...). Personally I was needed to add indent=2 and ensure_ascii=False keyword arguments into dumps call to be able to format log output in proper readable way. I think that solution can be very simple: jsut add something like dumps_args={} to constructor, for example:

def __init__(self, allowed_reserved_attrs=[], ignored_attrs=[], dumps_args={}):
        """Initializes a GelfFormatter."""
        super(GelfFormatter, self).__init__()
        ...
        self.dumps_args = dumps_args
...
        return json.dumps(log_record, **self.dumps_args)

It is quite simple and works well.

I would like to be able to get at the intermediate log_record used in the GelfFormatter.

My use case is advanced adding/removing of fields from the logs. I could then subclass GelfFormatter to access the data without having to json.loads(gelf_message).

A simple idea of this could be to move the implementation logic from format to a new method def convert(self, record) -> dict in the GelfFormatter. The format method would only consist of return json.dumps(self.convert(record)) but easily used in subclasses to add additional functionality.

This would keep the current API and I'm happy to discuss the name convert 😃. What do y'all think?