mhart / gelf-stream Goto Github PK
View Code? Open in Web Editor NEWA node.js stream to send JS objects to a Graylog2 server (in GELF format)
License: Other
A node.js stream to send JS objects to a Graylog2 server (in GELF format)
License: Other
Bunyan's warn
level doesn't map to the Gelf level.
I think function mapGelfLevel
, uses gelfling.WARN
instead of gelfling.WARNING
.
Thanks for the library - It works great.
Getting this in stdout
<--- Last few GCs --->
101344 ms: Mark-sweep 1376.9 (1457.0) -> 1375.7 (1457.0) MB, 146.9 / 0 ms [allocation failure] [GC in old space requested].
101523 ms: Mark-sweep 1376.9 (1457.0) -> 1376.9 (1457.0) MB, 169.9 / 0 ms [allocation failure] [GC in old space requested].
101686 ms: Mark-sweep 1376.9 (1457.0) -> 1376.9 (1457.0) MB, 163.3 / 0 ms [last resort gc].
101869 ms: Mark-sweep 1376.9 (1457.0) -> 1376.9 (1457.0) MB, 182.2 / 0 ms [last resort gc].
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 000001385A6B4639 <JS Object>
1: flatten(aka flatten) [C:\...\node_modules\gelf-stream\gelf-stream.js:73] [pc=000000A25884537E] (this=000001385A6041B9 <undefined>,obj=000003B7688B5629 <an Uint8Array with map 000001B5E0804629>,int
o=000003B7688B5611 <an Object with map 000002F4C0BD9E01>,prefix=00000373A1C06F71 <Very long string[5940]>,sep=000001385A635221 <St...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
This scares me only because logging should never throw and it brings down the whole server.
This works as expected:
log.info("message");
log.info({object: object}, "message");
But those get marked in graylog as "incomplete", while still being written normally to other streams:
log.info({"test":"test"});
log.info(config);
log.info("");
I'm using this for bunyan logging stream, via #forBunyan
.
Is there a way to filter out only certain messages to be sent to graylog?
I tried it with options.filter
, like this:
gelfStream.forBunyan(config.graylogHost, config.graylogPort, {
filter: function filter(chunk) {
console.log('**** in gelf stream #filter ****');
return false; // or some arbitrary decision, based on `chunk`
}
});
But, the first time the filter
function returns false
, no other message/chunk is ever sent to graylog. The idea is based on https://github.com/mhart/gelf-stream/blob/master/gelf-stream.js#L29, but I am probably missing something here.
Anyway, any pointer to to how to filter out / send only desired messages would be great. Thank you!
Hello.
How would I configure this to use a non-localhost? Just replace the localhost with the domain?
I'm setting up with the following implementation:
const stream = gelfStream.forBunyan('http://graylog-server/gelf', 12201);
const logger = bunyan.createLogger({
name: serviceName,
streams: [
{
level: 'info',
stream: stream,
formatter: 'pretty',
period: '1d',
count: 7
},
{
level: 'error',
stream: stream,
formatter: 'pretty',
period: '1d',
count: 7
}
]
});
and upon lifting my application I get:
Error: getaddrinfo ENOTFOUND http://graylog-server/gelf
at errnoException (dns.js:28:10)
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:76:26)
I also tried with plain http://graylog-server
(without the /gelf
part).
I am sure the connection is working properly, because when I do a curl
request - it works fine:
curl -X POST -H 'Content-Type: application/json' -d '{ "message": "UAHSDUHAUSDHUASHUS" }' 'http://graylog-server:12201/gelf'
GelfStream already have method forBunyan and this is grate.
But we using Bole logger and it will be very useful to have similar method for creating a gelf-stream with specific mappings.
I've created a pull request for this issue, please give me your feedback.
The README contains sample code that calls stream.end() directly (because bunyan doesn't do that when the program finishes).
(1) Unfortunately, the sample code is not representative of a typical node.js server, which never "finishes" deliberately. In other words, a typical server is designed to run forever, and will never exits unless it crashes. So, typically, there may be no good place in the server code to call the gelf-stream's end() function. What bad things, if any, can happen if end() is never called during a crash? For example, can we lose log entries?
(2) Even if a server were to intentionally exit (or if the input script has no server at all), there can still be callbacks queued on the main thread that want to do logging. So, simply calling end() (on the gelf-stream) from a server's 'close' event handler is not always the right thing to do.
(3) Is calling the gelf-stream's end() function from a process 'end' event handler the solution to both (1) and (2), or does that also have drawbacks?
(4) Or, is it possible to remove the end() function from the gelf-stream implementation, so that clients don't have to worry about it?
Hi!
When the 'flatten' function try to flatten a circular reference, it try to flatten until a stack overflow occurs.
I see 3 solutions to solve that problem:
Tell me if you prefer a solution. I can make a PR if you want.
It is possible to create an object that does not have hasOwnProperty
Example:
var data = Object.create(null);
Some other libraries will emit these kinds of objects, so we have to call hasOwnProperty through a call
via Object.prototype to work with these objects.
PR incoming.
Adding flattened properties as additional fields in the GELF structure, is certainly a very nice feature which allows this properties to be "searchable" in the Graylog GUI. Basically all of this fields are indexed by the ElasticSearch engine.
However this has a very adverse effect when the actual full_message contains JSON payloads that contain req/res type properties, like those captured to log roundtrip http request.
For instance:
"res": {
"body": {
"status":400,
.....
}
adds this GELF extra field:
"res.body.status":400,
Everything is great until, we try to log another response with:
"res": {
"body": {
"status":"ACTIVE",
.....
}
"res.body.status":"ACTIVE",
Then graylog server rejects the message due to a "MapperParsingException", Basically because of the indexing criteria, assuming in this case that res.body.status should be numeric. Basically the first type that made it through the log first, wins.
The solution is not to automatically add fields after flattening the log. So maybe bunyanToGelf can take an extra argument to selectively pick this behavior.
bunyanToGelf(log, addFlattened) {
......
if (addFlattened) {
for (key in flattenedLog) {
if (ignoreFields.indexOf(key) < 0 && gelfMsg[key] == null)
gelfMsg[key] = flattenedLog[key]
}
}
....
}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.