mhart / gelf-stream Goto Github PK

Bunyan's warn level doesn't map to the Gelf level.

I think function mapGelfLevel, uses gelfling.WARN instead of gelfling.WARNING.

Thanks for the library - It works great.

Getting this in stdout

<--- Last few GCs --->

  101344 ms: Mark-sweep 1376.9 (1457.0) -> 1375.7 (1457.0) MB, 146.9 / 0 ms [allocation failure] [GC in old space requested].
  101523 ms: Mark-sweep 1376.9 (1457.0) -> 1376.9 (1457.0) MB, 169.9 / 0 ms [allocation failure] [GC in old space requested].
  101686 ms: Mark-sweep 1376.9 (1457.0) -> 1376.9 (1457.0) MB, 163.3 / 0 ms [last resort gc].
  101869 ms: Mark-sweep 1376.9 (1457.0) -> 1376.9 (1457.0) MB, 182.2 / 0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 000001385A6B4639 <JS Object>
    1: flatten(aka flatten) [C:\...\node_modules\gelf-stream\gelf-stream.js:73] [pc=000000A25884537E] (this=000001385A6041B9 <undefined>,obj=000003B7688B5629 <an Uint8Array with map 000001B5E0804629>,int
o=000003B7688B5611 <an Object with map 000002F4C0BD9E01>,prefix=00000373A1C06F71 <Very long string[5940]>,sep=000001385A635221 <St...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

This scares me only because logging should never throw and it brings down the whole server.

This works as expected:

log.info("message");
log.info({object: object}, "message");

But those get marked in graylog as "incomplete", while still being written normally to other streams:

log.info({"test":"test"});
log.info(config);
log.info("");

I'm using this for bunyan logging stream, via #forBunyan.
Is there a way to filter out only certain messages to be sent to graylog?

I tried it with options.filter, like this:

gelfStream.forBunyan(config.graylogHost, config.graylogPort, {
  filter: function filter(chunk) {
    console.log('**** in gelf stream #filter ****');
    return false; // or some arbitrary decision, based on `chunk`
  }
});

But, the first time the filter function returns false, no other message/chunk is ever sent to graylog. The idea is based on https://github.com/mhart/gelf-stream/blob/master/gelf-stream.js#L29, but I am probably missing something here.

Anyway, any pointer to to how to filter out / send only desired messages would be great. Thank you!

Hello.

How would I configure this to use a non-localhost? Just replace the localhost with the domain?

I'm setting up with the following implementation:

const stream = gelfStream.forBunyan('http://graylog-server/gelf', 12201);

const logger = bunyan.createLogger({
    name: serviceName,
    streams: [
      {
        level: 'info',
        stream: stream,
        formatter: 'pretty',
        period: '1d',
        count: 7
      },
      {
        level: 'error',
        stream: stream,
        formatter: 'pretty',
        period: '1d',
        count: 7
      }
    ]
  });

and upon lifting my application I get:

Error: getaddrinfo ENOTFOUND http://graylog-server/gelf
          at errnoException (dns.js:28:10)
          at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:76:26)

I also tried with plain http://graylog-server (without the /gelf part).

I am sure the connection is working properly, because when I do a curl request - it works fine:

curl -X POST -H 'Content-Type: application/json' -d '{ "message": "UAHSDUHAUSDHUASHUS" }' 'http://graylog-server:12201/gelf'

GelfStream already have method forBunyan and this is grate.
But we using Bole logger and it will be very useful to have similar method for creating a gelf-stream with specific mappings.

I've created a pull request for this issue, please give me your feedback.

The README contains sample code that calls stream.end() directly (because bunyan doesn't do that when the program finishes).

(1) Unfortunately, the sample code is not representative of a typical node.js server, which never "finishes" deliberately. In other words, a typical server is designed to run forever, and will never exits unless it crashes. So, typically, there may be no good place in the server code to call the gelf-stream's end() function. What bad things, if any, can happen if end() is never called during a crash? For example, can we lose log entries?

(2) Even if a server were to intentionally exit (or if the input script has no server at all), there can still be callbacks queued on the main thread that want to do logging. So, simply calling end() (on the gelf-stream) from a server's 'close' event handler is not always the right thing to do.

(3) Is calling the gelf-stream's end() function from a process 'end' event handler the solution to both (1) and (2), or does that also have drawbacks?

(4) Or, is it possible to remove the end() function from the gelf-stream implementation, so that clients don't have to worry about it?

Hi!

When the 'flatten' function try to flatten a circular reference, it try to flatten until a stack overflow occurs.
I see 3 solutions to solve that problem:

1. Limit flattening to a defined level (pretty easy)
2. Look for circular references (a bit harder)
3. Ask people to not log circular references

Tell me if you prefer a solution. I can make a PR if you want.

It is possible to create an object that does not have hasOwnProperty

Example:

var data = Object.create(null);

Some other libraries will emit these kinds of objects, so we have to call hasOwnProperty through a call via Object.prototype to work with these objects.

PR incoming.

this line references cb which seems to be undefined:

cc: @milas

Adding flattened properties as additional fields in the GELF structure, is certainly a very nice feature which allows this properties to be "searchable" in the Graylog GUI. Basically all of this fields are indexed by the ElasticSearch engine.

However this has a very adverse effect when the actual full_message contains JSON payloads that contain req/res type properties, like those captured to log roundtrip http request.

For instance:

"res": {
    "body": {
      "status":400,
.....
}

adds this GELF extra field:

"res.body.status":400,

Everything is great until, we try to log another response with:

"res": {
    "body": {
      "status":"ACTIVE",
.....
}

"res.body.status":"ACTIVE",

Then graylog server rejects the message due to a "MapperParsingException", Basically because of the indexing criteria, assuming in this case that res.body.status should be numeric. Basically the first type that made it through the log first, wins.

The solution is not to automatically add fields after flattening the log. So maybe bunyanToGelf can take an extra argument to selectively pick this behavior.

bunyanToGelf(log, addFlattened) {
......

  if (addFlattened) {
    for (key in flattenedLog) {
      if (ignoreFields.indexOf(key) < 0 && gelfMsg[key] == null)
        gelfMsg[key] = flattenedLog[key]
    }
  }

....
}