"GELF processing failed irrecoverably" stops the receiver about seq-input-gelf HOT 7 CLOSED

Thanks for the report @MattMofDoom!

I’ll take a closer look but I suspect this could be to fail hard in cases that could have torn down the accept listener so the server can be restarted instead of leaving a zombie around.

That particular error should absolutely be recoverable though and it shouldn’t ever be possible for a misbehaved client to bring the server down like that.

from seq-input-gelf.

Hey @MattMofDoom 👋 I've made some changes to how we handle errors on client connections that hopefully will fix this. They're published as 2.0.318-dev. If you get a chance to give it a try I'd be very keen to hear how you go!

from seq-input-gelf.

Thanks for the report, Matt!

For anyone following along, the line of code in question is:

https://github.com/datalust/sqelf/blob/dev/sqelf/src/server/mod.rs#L228

I suspect that this is a bug - but I'll need to check in with Ashley, who's out for a few days. Just in case it turns out to be relevant - is the input listening on TCP or on UDP?

Thanks again 👍

from seq-input-gelf.

@nblumhardt Good point - it's listening on TCP.

from seq-input-gelf.

Thanks for the extra info @MattMofDoom

from seq-input-gelf.

@KodrAus Interestingly, I haven't seen it recur since reverting to v2.0.298. That may be entirely coincidental, but I have an OpsGenie alert in place that would tell me if no logs have been seen for 15 mins. Further steps that I took included updating Seq to the current 2021.2 build, and restarting the server, but those are reasonably standard for periodic maintenance that I perform ... server is restarted at least monthly and I try to keep reasonably up to date on Seq releases.

from seq-input-gelf.

Thanks @KodrAus - yeah I saw, thanks very much mate and will do 😁

from seq-input-gelf.