All events from Docker Imported as Error Level in Seq about seq-input-gelf HOT 6 CLOSED

I am having a similar issue with containers that write to stderr. eg postgres

{
	"version":"1.1",
	"host":"docker-desktop",
	"short_message":"2020-08-24 07:54:10.273 UTC [1] LOG:  database system is ready to accept connections",
	"timestamp":1598255650.273,
	"level":3,
	"_command":"docker-entrypoint.sh postgres",
	"_container_id":"8328b64fc67d229c87ff6cddca94e57cd23436e3f961cfd91f95e1221748ef27",
	"_container_name":"a9-automation-tool-server_postgres_1",
	"_created":"2020-08-24T07:45:12.2754228Z",
	"_image_id":"sha256:62473370e7eeea08c9ad60273081a32452c5ba461314490b893c11021e01aefd",
	"_image_name":"postgres",
	"_tag":"postgres"
}

from seq-input-gelf.

Hey @andymac4182!

The culprit will be that "level":3, which corresponds to syslog's Error level. That's interesting. So your postgres container is logging to stdout and Docker's own log driver is forwarding those as GELF?

from seq-input-gelf.

Hi Isaiah, thanks for dropping by! We'll dig into this and let you know what we find 👍

from seq-input-gelf.

Hi @isaiahlg 👋

The server uses the GELF messages level property to determine what level to forward the event as and doesn't consider any content in the message itself. The level property should follow the standard syslog integer values, with 1 being alert, 3 being error, and 6 being info. I have noticed some logging frameworks give different meanings to these numbers though (I think it was fluentd I was looking at, which is Ruby).

I've merged #75, which is published on Docker Hub under the image datalust/sqelf-ci:2.0.290-dev. Is it possible to update your seq-input-gelf compose to something like:

 seq-input-gelf:
    image: datalust/sqelf-ci:2.0.290-dev
    ports:
      - "12201:12201/udp"
    network_mode: host
    environment:
      SEQ_ADDRESS: "http://localhost:5341"
      GELF_ENABLE_DIAGNOSTICS="True"
      restart: unless-stopped
    depends_on:
      - seq

Then, when events are logged in Seq you should see an extra raw_payload property attached that should give a clue of what level the message is being sent with.

from seq-input-gelf.

That was from the start up logs from postgres:latest I hadn't done anything other to set any options around logging. I am seeing the same behavior with a node library that is logging to stderr not stdout. In this case the offending library is https://www.npmjs.com/package/debug . I don't control the library that is using debug so I am looking into options to set config from the outside.

By default debug will log to stderr

from seq-input-gelf.

I wouldn't go as far as to say that integer-based levels are bad but... integer-based levels are bad 🙂

This is something we could try and fix in the input here itself with configuration, but I'm a little concerned about accumulating too many ad-hoc pipeline-esc features (we've already got a few). I think the best way forward would be to use something like fluentbit to massage your logs on their way through and then treat the GELF input as a sink. It's unfortunately more machinery, but is shared infrastructure and already suitable for transforming logs from any shape into any other.

from seq-input-gelf.