Greetings!

I'm implementing a Filebeats reader with the lumberjack protocol in Elixir.

What I've noticed so far is that upon connection Filebeat sends me a Window frame together with a Compressed one. These ones I manage to parse, inflate and read several JSON events. But after that Filebeats sends me a bunch of weird frames not mentioned anywhere in the protocol docs. Are those frames the continuation of the first compressed one filebeats sent me? I thoght I would receive only "whole frames" to parse thanks to TCP. Am I getting something wrong here?

Thanks a lot for you attention!

For example, here:

func (s *server) run() {
	defer s.wg.Done()
	for {
		client, err := s.netListener.Accept()
		if err != nil {
			break
		}

		s.handle(client)
	}
}

netListener is commonly a TLS listener, which can return an error from Accept for many reasons. Breaking out of the loop stops the server from accepting new connections. And, as far as I can tell, there is no way to know from the outside (since run gets called in a goroutine) that this has happened.

I am searching the protocol documentation (frame, encoding etc...) in order to implement it with Java (or maybe a Java impl already exist?).

Hand casting each key of a complex map[string]interface{} is very painful.

Hi, I am using go-lumber to write a server that can process some beats logs. I found that my server was only processing one batch per lj connection at a time. I was able to get concurrent batches by making DefaultHandler's ack-signaling channel buffered, so we don't block here when there is an outstanding un-ACKed batch. Is there a chance we can get something like that? I'd be happy to work on a PR if it helps the cause!

The current klauspost/compress version is at v1.11.2 (updated 2 years ago), while the latest compress package version is v1.15.9. The newer versions have included quite some optimizations on flate package, which powers the zlib package used by go-lumber.

Furthermore, we might want to just setup a dependabot to update the package version if it's an option for this repository. Some example of setting this up:
https://bl.ocks.org/magnetikonline/6f215db058e327905bce66c37f92426c

Reference: urso/go-lumber-old#4

There is a protocol description for the old Lumberjack protocol at https://github.com/logstash-plugins/logstash-input-beats/blob/v2.0.0/PROTOCOL.md but I'm missing like this for the Beats protocol.

While having a working library for the Lumberjack/Beats protocol is nice and a specification could be "reverse engineered" by looking at the Go code, it would be great if a textual protocol spec of the Lumberjack (v1) and Beats (v2) protocols existed.

I have implemented a change to support CRLs into go-lumber, but the implementation is lacking. I'm wondering if anyone has any suggestions on how I can improve it. You can see my current change here: master...jsmaupin:crls.

Some of the issues I see are:

1. CRLs are now supported in Go since 1.15, so this version is required as I used some of the new functions
2. The Go standard packages do not directly support checking client certificates against CRLs, so that needed to be implemented. It only has implementations for loading CRLs.
3. Because of (2), I CRLs is a first class option next to the TLS option.

I'm thinking that maybe a callback mechanism would make more sense? Any thoughts?

In ReceiveAck we first set a deadline on the read at

And then do the read a few lines later at:

If the conn is ever replaced in between those calls, the Read can run forever, this means Receive Ack could wait forever waiting for Acks that will never arrive, because the conn they would come in on has been closed, and no Acks will arrive on the new conn.

This can happen if another go routine can replace the conn.