svetob / logstash-json Goto Github PK

Should use Travis CI to build and test, and also upload releases to hex.

We need to add a field to our logs that is not available at compile-time.

While we can add metadata dynamically we'd like the value to be a separate field rather than a metadata sub-field.

Is there any way we can dynamically update the logger configuration to add a new field?

If not, would you be open to a pull request to add this? What would be the best way of doing this?

Thanks in advance!

Hey there!

I just add the lib to my project {:logstash, github: "svetob/logstash-json"} and when I try to run tests or the app Im getting:

Unchecked dependencies for environment test:
* logstash (https://github.com/svetob/logstash-json.git)
  could not find an app file at "_build/test/lib/logstash/ebin/logstash.app". This may happen if the dependency was not yet compiled, or you specified the wrong application name in your deps, or the dependency indeed has no app file (then you can pass app: false as option)
** (Mix) Can't continue due to errors on dependencies

Any clue? Thanks!

Hi, I'm trying to use logstash-json with logstash 2.3 (in a docker container). Here is how I launch logstash :

docker run --rm -ti -p 5001:5001 logstash:2.3 logstash --debug -e 'input { tcp {codec => json port => "5001"} } output { stdout { } }'

I can see with wireshark that the packets arrive to the server, but nothing in logstash :-(, If I remove the codec => json it works but I have a warning telling me that the charset is not good (not utf-8). If I change the charset to ASCII for example, it doesn't work. So do you have a config that works with logstash ?

I'm finding I need to mess around with some of the hard-coded attributes in the log message. For instance, our logstash expects 'level' to be in integer -- if we send strings, we have trouble. Same with 'pid'.

Might it be possible to have a configurable formatter that can dynamically modify the log event? So we could do something like:

config :logger, :logstash
   formatter: {MyApp, :formatter}

with module MyApp having

def formatter(event) do
   event
   |> Map.put(:thread_id, event[:pid])
   |> Map.put(:level, convert_level(event[:level])
   |> Map.delete(:pid)
end

Does this make sense? I did a quick and dirty you can look at in my fork. I'll add tests and docs if this is of interest to you. Also -- am pretty new to Elixir and am open to criticism and guidance.

Thanks!

Hi!

Are you planing on releasing a new version of the hex package with the fix #30? I would like to update my version :)

The :console message formatter does a good job of formatting the message, let's use it to ensure the message is always a nicely formatted string.

IO.puts is Bad(TM) so let's add an :error_log parameter that defaults to :console.

The problem with timex is that it's a large dependency, plus it also comes with tzdata, which requires starting the :timex application. This also starts tzdata which performs HTTP requests to an external database to download the latest timezone updates. The dependencies between the logger backend and the :timex application at startup also causes crashes if not carefully handled.

I want to do this much simpler by removing this dependency and logging @timestamp in UTC time, ignoring local timezone. Perhaps later we'll discover a simpler way to get the local timezone, but for now this is good enough.

I came across this issue when I was using this in production. We had an increase in traffic and because of some detailed logging the application became unresponsive as logging appeared to be blocking on this line: https://github.com/svetob/logstash-json/blob/master/lib/logstash_json_tcp.ex#L49

When I looked in observer there were several hundred thousand messages waiting on this line of code.

Phoenix and may others now don't use JSON encoding libraries implicitly, instead allowing for configuration via config.exs. This is great, because it allows to use on package across the whole application instead of use multiple variations of the same.

I think this library would benefit from switching to a similar approach, to drop explicit dependency on Poison in favor of config option.

Thanks a lot for a great library!

Must close socket and attempt to reconnect if a connection issue is encountered.

This also means either buffering or dropping messages until connection is restored.