sematext / winston-logsene Goto Github PK

View Code? Open in Web Editor NEW

14.0 22.0 6.0 1.31 MB

Winston Transport for Logsene

Home Page: http://sematext.com/logsene

License: Apache License 2.0

JavaScript 96.36% Handlebars 3.64%

logs logging logshipper log-management nodejs javascript devops devops-tools

winston-logsene's Introduction

This is a transport module for the winston logger winstonjs/winston for logging with Sematext Cloud.

Winston-Logsene combines the flexibility of using the Winston logging framework with Sematext Cloud (think Hosted Elasticsearch + Kibana). Create your free account and access token here.

Usage

const Logsene = require('winston-logsene')
const { createLogger, format, config } = require('winston')

const logger = createLogger({
  levels: config.npm.levels,
  transports: [new Logsene({
    token: process.env.LOGS_TOKEN,
    level: 'debug',
    type: 'test_logs',
    url: 'https://logsene-receiver.sematext.com/_bulk'
  })]
})

Options

token - Create your free account and access token here.
type - Type of your logs - please note you can define Elasticsearch mapping templates in Sematext Cloud
url - Sematext Cloud receiver URL (or URL for Sematext Enterprise / On Premises), defaults to Sematext Cloud (US) receiver https://logsene-receiver.sematext.com/_bulk. To ship logs to Sematext Cloud (EU) in Europe use https://logsene-receiver.eu.sematext.com/_bulk
handleExceptions - boolean 'true' logs 'uncaught exceptions'
handleErrors - boolean 'true' logs 'unhandled errors'
exitOnError - if set to 'false' process will not exit after logging the 'uncaught exceptions'
source - name of the logging source, by default name of the main node.js module
setSource - "true" adds "source" to the log event (modifies the object!), default false
rewriter - similar to rewriters in winston, rewriter allows modifying of log meta but only for the logsene transport. This is a simple function which takes level, msg, meta as parameter and returns the new meta array
flushOnExit - Handling SIGTERM, SIGQT, SIGINT and 'beforeExit' to flush logs and terminate. Default value: true.

Examples

var Logsene = require('winston-logsene')
const { createLogger, format, config } = require('winston')

// example for custom rewriter, e.g. add myServerIp field to all logs
var myServerIp = '10.0.0.12'

var logger = createLogger({
  levels: config.npm.levels,
  transports: [new Logsene({
    // set log level, defaut is 'info'
    level: 'debug',
    // optional set a format function
    format: format.splat(),
    token: process.env.LOGS_TOKEN,
    rewriter: function (level, msg, meta) {
      meta.ip = myServerIp
      return meta
    }
  })]
})

logger.info('debug', 'Info Message')
// use dynamic list of placeholders and parameters for format.splat(), and any Object as Metadata
logger.info('Info message no. %d logged to %s', 1, 'Sematext Cloud', {meta: 'test-log', count: 1, tags: ['test', 'info', 'winston']})
// message placeholders work the same way as in util.format()
// utilize tags (in the metadata object) as filter to be used in Sematext Cloud user interface
logger.info('Info Message', {tags: ['info', 'test']})
logger.error('Error message no. %d logged to %s', 1, 'Sematext Cloud', {meta: 'test-error', count: 1, tags: ['test', 'error', 'winston']})
logger.warn('Warning message no. %d logged to %s', 1, 'Sematext Cloud', {meta: 'test-warning', count: 1, tags: ['test', 'warning', 'winston']})
logger.debug('Debug message no. %d logged to %s', 1, 'Sematext Cloud', {meta: 'test-debug', count: 1})

Schema / Mapping definition for Meta-Data

It is possible to log any JSON Object as meta data, but please note Sematext Cloud stores data in Elasticsearch and therefore you should define an index template, matching your data structure. More about Elasticsearch mapping and templates for Sematext Cloud: https://sematext.com/blog/custom-elasticsearch-index-templates-in-logsene/

In addition you should use different types for different meta data structures to avoid type conflicts in Elasticsearch. Include a type name in the meta-data like {type: 'logType1', ...} - this overwrites the "type" property, specified in the contstructor.

logger.add (logsene, {token: process.env.LOGSENE_TOKEN, type: 'my_logs'})
// numeric id, log type from constructor
logger.info('hello', {id: 1})
// The next line will cause a type conflict in Elasticsearch/Sematext Cloud, because id was a number before
logger.info('hello', {id: 'ID-1'})
// using a different type, OK no type conflict for the field 'id' in Elasticsearch//Sematext Cloud
// because we use a different type in the Elasticsearch//Sematext Cloud index
logger.info('hello', {type: 'my_type_with_string_ids',{id: 'ID-1'}})

Security

HTTPS is enabled by default
Environment variables for Proxy servers:
- For HTTPS endpoints (default): HTTPS_PROXY / https_proxy

        export HTTPS_PROXY=https://my-ssl-proxy.example
        export HTTPS_PROXY=http://my-proxy.example

For HTTP endpoints (e.g. On-Premises): HTTP_PROXY / http_proxy

        export HTTP_PROXY=http://my-proxy.example
        export HTTP_PROXY=https://my-ssl-proxy.example

License

Apache 2, see LICENSE file

winston-logsene's People

Contributors

Stargazers

Watchers

Forkers

seti123 hver adrienneveu exzeo jukbot joonhocho

winston-logsene's Issues

Add winston-logsene to winston Transports

Add winston-logsene to winston Transports https://github.com/winstonjs/winston/blob/c01add63e639d43573aeb3df4bc06cc70313317f/docs/transports.md

Winston-logsene is modifying objects

We switched from loggly to logsene, because of some memory issues of loggly with big ammount of data and logsene is working fine.

But winston-logsene is modifying our objects that we put into logger. I think this is big issue, the logger should never ever do anything with code, or if it is because of speed (you do not have to create copy of the whole object), it should be possible to turn it on/off in options.

Right now, the winston-logsene is adding "source" field to our object, when we are logging it. For example this code :

var myObj = {hi: 'I am simple object'};
console.log(myObj);
winston.info(myObj);
console.log(myObj);

is having this output

{ hi: 'I am simple object' }
info:  hi=I am simple object
{ hi: 'I am simple object',
  source: 'c:\\prace\\nodejs\\node-template\\node_modules\\winston-logsene\\lib\\index.js' }

Update/remove vulnerable packages from npm

morgan-json is a well-known vulnerable package, and this agent uses it. I suggest removing it and replacing with custom winston transport, as right now, this vuln makes this soft unusable in production

Add support for handlesExceptions option

Feedback from a Logsene user:
The winston-logsene transport would benefit from the handlesExceptions option.

Does not send successfully unless I wait 30 seconds

I just started trying to use the example code with my test Logsene instance. Flush on exit does not appear to be working. So far the only way I've been able to get a log message to be sent in a barebones Node script is to add a 30-second timeout after the log (after reading through the source and seeing the periodic send call). I also read issue #10 and tried the workaround there with the dummy console transport, and have not been successful.

Based on my console.logs, send() of the underlying Logsene object is getting called on exit, but the callback from the http request never executes.

I have winston-logsene 1.2.0, winston 2.2.0, node 6.6.0. Thanks.

(To try to simplify the issue I went a layer down. The following works consistently.)

var Logsene = require('logsene-js');
var logger = new Logsene(process.env.LOGSENE_TOKEN);

logger.log('info', 'exit hook');

process.on('beforeExit', function () {
  if (logger.logCount) {
    logger.send();
  }
});

Memory leak and process.exit issue

There is a fairly large memory leak that is happening. I am unsure of the root cause, but you can see the memory leak in action here:
https://github.com/chrisblossom/memory-leak-example

There is also an issue when I have this in my development environment. It stops my node process from restarting and end up with this error:

27 Jul 10:56:48 - uncaughtException: c8841060f9114c6895e59c61d194b312
{ message: 'uncaughtException',
  error:
   { [Error: listen EADDRINUSE :::3002]
     code: 'EADDRINUSE',
     errno: 'EADDRINUSE',
     syscall: 'listen',
     address: '::',
     port: 3002 } }

Ideas?

Thank you!

Can't log exception instances

Using the winston console logger one can do log.error(new Error()) and the exception including stack will be printed to the console. However the same in winston-logsene results in an empty message and the error is lost.

Logs not shipped if the process is very short-lived

Reported by @clounie:

I'm having a problem where if the process quits too soon, none of the logs get sent. Here's the example:

https://gist.github.com/clounie/0d0b13477116deda3c000dbbd2d48952

My winston-logsene package is version 1.1.11, the latest I see on Github.

tags don't appear to be received properly

From your README, logger.info('Info Message', {tags: ['info', 'test']})

does not appear to be stored properly as a tag under the "TAGS" in the UX, did I miss something? (I also tried all caps TAGS for the key as well, no dice)

Add a Native UI or Kibana 4 screenshot to README

Add a Native UI or Kibana 4 screenshot to README, below Usage.

Update to work with Winston 3.x

The current version of winston-logsene works with Winston 2.x, but not 3.x. Thanks for pointing this out @samuelt1.
Winston 3.x was first released in June 2018.

Need to update the README, as mentioned in #14

Add instructions for EU instance

Hi,
I've been trying to get the logs writing to the EU instance, but couldn't get it working. I've resorted to create an account on the US instance and it's working fine. Could you include instructions for configuring the module for the EU please?
Regards,
Andy

Use
To use the Logsene transport in winston, you simply need to require it:

var logger = require('winston');
var logsene = require('winston-logsene');
Pass an instance of the Logsene transport to the new winston logger:

logger.add(logsene, {
  token: 'my-token',
});
Add logging code:

logger.debug('Hello, Logsene!');
logger.info('Hello, Logsene!');
logger.warn('Hello, Logsene!');
logger.error('Hello, Logsene!');

that gives errors unless you use [email protected]
the readme on this page gives more info but still does not work. without downgrading winston.