Giter Club home page Giter Club logo

pino-datadog-transport's Introduction

pino-datadog-transport

Note: The project has has moved to theogravity/datadog-transports.

NPM version CircleCI built with typescript JavaScript Style Guide

A pino v7+ transport for sending logs to Datadog.

It uses datadog-api-client-typescript to send logs using the client v2.LogsApi#submitLog method.

  • Performs batch sending of logs on a periodic basis, or when log capacity is reached in overall log batch size or count.
  • Will retry failed sends.
  • Can disable batch sending and always send for each log entry.

Table of Contents

Installation

yarn add pino-datadog-transport

import { LoggerOptions, pino } from 'pino';

const pinoConf: LoggerOptions = {
  level: 'trace',
}

const logger = pino(pinoConf, pino.transport({
  target: 'pino-datadog-transport',
  options: {
    ddClientConf: {
      authMethods: {
        apiKeyAuth: <your datadog API key>
      }
    },
    ddServerConf: {
      // Note: This must match the site you use for your DataDog login - See below for more info
      site: "datadoghq.eu"
    }
  }
}))

Configuration options

export interface DDTransportOptions {
  /**
   * DataDog client configuration parameters.
   * @see https://datadoghq.dev/datadog-api-client-typescript/interfaces/client.Configuration.html
   */
  ddClientConf: ConfigurationParameters
  /**
   * Datadog server config for the client. Use this to change the Datadog server region.
   * @see https://github.com/DataDog/datadog-api-client-typescript/blob/1e1097c68a437894b482701ecbe3d61522429319/packages/datadog-api-client-common/servers.ts#L90
   */
  ddServerConf?: {
    /**
     * The datadog server to use. Default is datadoghq.com.
     * Other values could be:
     * - us3.datadoghq.com
     * - us5.datadoghq.com
     * - datadoghq.eu
     * - ddog-gov.com
     */
    site?: string
    subdomain?: string
    protocol?: string
  }
  /**
   * The integration name associated with your log: the technology from which
   * the log originated. When it matches an integration name, Datadog
   * automatically installs the corresponding parsers and facets.
   * @see https://docs.datadoghq.com/logs/log_collection/?tab=host#reserved-attributes
   */
  ddsource?: string
  /**
   * Comma separated tags associated with your logs. Ex: "env:prod,org:finance"
   */
  ddtags?: string
  /**
   * The name of the application or service generating the log events.
   * It is used to switch from Logs to APM, so make sure you define the same
   * value when you use both products.
   * @see https://docs.datadoghq.com/logs/log_collection/?tab=host#reserved-attributes
   */
  service?: string
  /**
   * Called when the plugin is ready to process logs.
   */
  onInit?: () => void
  /**
   * Error handler for when the submitLog() call fails. See readme on how to
   * properly implement this callback.
   */
  onError?: (err: any, logs?: Array<Record<string, any>>) => void
  /**
   * Define this callback to get debug messages from this transport
   */
  onDebug?: (msg: string) => void
  /**
   * Number of times to retry sending the log before onError() is called.
   * Default is 5.
   */
  retries?: number
  /**
   * Interval in which logs are sent to Datadog.
   * Default is 3000 milliseconds.
   */
  sendIntervalMs?: number
  /**
   * Set to true to disable batch sending and send each log as it comes in. This disables
   * the send interval.
   */
  sendImmediate?: boolean
}

Implementing the onError() / onDebug() callback

You cannot specify the callbacks directly as they are not serializable.

Doing so will result in the following error:

DOMException [DataCloneError]: (e)=>{
} could not be cloned.

Instead, you need to create another file that implements it:

// Your logger file
const p = pino({}, pino.transport({
    target: join(__dirname, 'pino-datadog-logger.js'),
    options: {
      ddClientConf: {
        authMethods: {
          apiKeyAuth: <your datadog API key>
        }
      },
    }
  }))
/* eslint-disable */
// pino-datadog-logger.js
// https://github.com/pinojs/pino-pretty#handling-non-serializable-options
// Functions as options on the pino transport config are not serializable as they
// are workers, so we create this worker file which includes our callbacks

module.exports = (opts) => {
  return require('pino-datadog-logger')({
    ...opts,
    onError: (data, logItems) => {
      // Your error handling here
    }
  })
}

Note: Log entries can only be a maximum of 1MB in size. This is a Datadog imposed limit. This library will call onError() if a log entry is 0.95MB (to account for serialization and metadata).

Sending logs to Datadog with pino-socket instead

It is possible to send logs to Datadog using raw TCP instead of HTTPS. Datadog recommends HTTPS as the logs are compress-able, whereas TCP-sent logs are not. HTTPS also has content length limits, whereas TCP does not.

Datadog recommends sending logs over HTTPS instead of raw TCP, as the latest version of the Datadog agent uses HTTPS with a TCP fallback:

// https://docs.datadoghq.com/agent/logs/log_transport?tab=https
For Agent v6.19+/v7.19+, the default transport used for your logs is
compressed HTTPS instead of TCP for the previous versions.
When the Agent starts, if log collection is enabled, it runs a
HTTPS connectivity test. If successful, then the Agent uses
the compressed HTTPS transport, otherwise the Agent falls back to a TCP transport.

However, you can send logs using raw TCP + TLS using pino-socket.

See instructions here on how to do this.

pino-datadog-transport's People

Contributors

andreasmarkussen avatar danielhochman avatar ooga avatar theogravity avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

pino-datadog-transport's Issues

Doesnt seem to play nice with google cloud functions?

When using this package in google cloud functions, we get errors with the function not terminating.
In google cloud console the function is forcibly terminated after 15 minutes with the error "Error: Process exited with code 143 "

Even tho the process has completed, there is something still "hanging"

I tracked it down to exit_hook. Using patch package to remove it from process-logs.ts seemed to solve it for me....

Is exit_hook needed if sendImmediate is true?
Should this issue be opened in exit_hook instead?

Thanks!

onError deos not work

onError deos not work, code below:

    console.error('error', err, logs);
  },

results in:

error undefined undefined

Question: This package vs. logging to file and having Datadog agent tail the file

The Node.js Log Collection Documentation from Datadog says:

To send your logs to Datadog, log to a file and tail that file with your Datadog Agent

You can stream your logs from your application to Datadog without installing an Agent on your host. However, it is recommended that you use an Agent to forward your logs as it provides a native connection management.

I know that the documentation is written about Winston, but it seems like the guidance would still apply regardless of the logging library used.

What is the difference between using this transport vs. logging to a file and tailing that file with the Datadog Agent? Is this package strictly useful in an environment without the Datadog Agent, or are there benefits to this transport that make it worth using even when the Datadog Agent is available?

Thanks!

How to set the Datadog hostname and service name?

Screenshot 2022-09-27 at 9 58 35 AM

I have DD_SERVICE and DD_SERVICE_NAME environment variables defined but the Datadog logs are not showing the service name. I am using this transport with Pino NestJS. When I use the Winston Nestjs then I am able to see the correct service name and host. How can I set these hostname and service name fields?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.