Giter Club home page Giter Club logo

logrus-cloudwatchlogs's Introduction

Cloud Watch Logs hook for Logrus godoc reference

Use this hook to send your Logrus logs to Amazon's Cloud Watch Logs.

Options

The formatter has options available to it. Please check the godoc.

Example

Look in the examples directory for more examples.

package main

import (
	"io/ioutil"
	"log"
	"os"

	"github.com/sirupsen/logrus"
	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/credentials"
	"github.com/kdar/logrus-cloudwatchlogs"
)

func main() {
	group := os.Getenv("AWS_CLOUDWATCHLOGS_GROUP_NAME")
	stream := os.Getenv("AWS_CLOUDWATCHLOGS_STREAM_NAME")

	// logs.us-east-1.amazonaws.com
	// Define the session - using SharedConfigState which forces file or env creds
    	sess, err := session.NewSessionWithOptions(session.Options{
    		SharedConfigState: session.SharedConfigEnable,
    		Config:            aws.Config{Region: aws.String("us-east-1")},
    	})
    	if err != nil {
    		panic("Not going to be able to write to cloud watch if you cant create a session")
    	}
    
    	// Determine if we are authorized to access AWS with the credentials provided. This does not mean you have access to the
    	// services required however.
    	_, err = sts.New(sess).GetCallerIdentity(&sts.GetCallerIdentityInput{})
    	if err != nil {
    		panic("Couldn't Validate our aws credentials")
    	}

	hook, err := logrus_cloudwatchlogs.NewHook(group, stream, sess)
	if err != nil {
		log.Fatal(err)
	}

	l := logrus.New()
	l.Hooks.Add(hook)
	l.Out = ioutil.Discard
	l.Formatter = logrus_cloudwatchlogs.NewProdFormatter()

	l.WithFields(logrus.Fields{
		"event": "testevent",
		"topic": "testtopic",
		"key":   "testkey",
	}).Fatal("Some fatal event")
}

logrus-cloudwatchlogs's People

Contributors

jmunson avatar kdar avatar punya-asapp avatar sd-charris avatar stevehnh avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

logrus-cloudwatchlogs's Issues

NewBatchingHook doesn't seem to grab the sequence token

I am attempting to use batching to avoid CloudWatch rate limiting. During my automated testing, I am seeing the following:

=== RUN   TestCloudWatchBatching100
Failed to fire hook: InvalidSequenceTokenException: The given sequenceToken is invalid. The next expected sequenceToken is: 49570318328963403756104352477615904793123053365436134642
	status code: 400, request id: f15938de-5063-11e7-8263-adbc854f83ad
{"app":"middlewares.test","host":"45f28166fed1","level":"info","msg":"test-host-100 - username [ts] \"method uri proto\" %!d(string=status) %!d(string=size) \"referer\" \"agent\" fblh.reqid \"frontend\" \"backend\" elapsed\n","time":1497377582}Failed to fire hook: InvalidSequenceTokenException: The given sequenceToken is invalid. The next expected sequenceToken is: 49570318328963403756104352477615904793123053365436134642
	status code: 400, request id: f15e8fab-5063-11e7-bf73-6fbae9798f07
{"app":"middlewares.test","host":"45f28166fed1","level":"info","msg":"test-host-100 - username [ts] \"method uri proto\" %!d(string=status) %!d(string=size) \"referer\" \"agent\" fblh.reqid \"frontend\" \"backend\" elapsed\n","time":1497377582}Failed to fire hook: InvalidSequenceTokenException: The given sequenceToken is invalid. The next expected sequenceToken is: 49570318328963403756104352477615904793123053365436134642
	status code: 400, request id: f16482f0-5063-11e7-acda-e388958e2b61
{"app":"middlewares.test","host":"45f28166fed1","level":"info","msg":"test-host-100 - username [ts] \"method uri proto\" %!d(string=status) %!d(string=size) \"referer\" \"agent\" fblh.reqid \"frontend\" \"backend\" elapsed\n","time":1497377582}Failed to fire hook: InvalidSequenceTokenException: The given sequenceToken is invalid. The next expected sequenceToken is: 49570318328963403756104352477615904793123053365436134642
	status code: 400, request id: f168c988-5063-11e7-abdd-139e182e19f7
{"app":"middlewares.test","host":"45f28166fed1","level":"info","msg":"test-host-100 - username [ts] \"method uri proto\" %!d(string=status) %!d(string=size) \"referer\" \"agent\" fblh.reqid \"frontend\" \"backend\" elapsed\n","time":1497377582}Failed to fire hook: InvalidSequenceTokenException: The given sequenceToken is invalid. The next expected sequenceToken is: 49570318328963403756104352477615904793123053365436134642

/snip

If I can find a fix, I will submit a PR.

batchFrequency is hard code to zero?

just wonder why batchFrequency is hard code to zero? and Write method of Hook struct looks wired

	if lastErr, ok := <-h.err; ok {
			return 0, fmt.Errorf("%v", lastErr)
	}

if there is no err written to err chan, it will just hang and not return?

Interested in finding out how you found the magic number 26 to add to message size

Hi,

Thanks for writing this package, it's really cool! I'm currently writing something similar for zerolog.

When you work out the total size of the batch to send to AWS you are taking into account the 1MB limit mentioned here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/cloudwatch_limits_cwl.html
In the code you write (https://github.com/kdar/logrus-cloudwatchlogs/blob/master/hook.go#L131):

messageSize := len(*p.Message) + 26

I'm just wondering where the 26 comes from and how you calculated it.

The way I would calculate it is by assuming that they're using json encoding and then counting the extra characters. When I do that, I arrive at a different number, 36. I do that by considering the json encoding to be:

{"message":"<len(*p.Message)>","timestamp":<int64, i.e. 8 bytes>},

The final comma will be present for all the messages except for the last one, so it's reasonable to count it. That looks like the total number of bytes per message would therefore be len(*p.Message) + 36 as there are 28 characters in the json (other than the message) and 8 bytes for the the timestamp.

I look forward to hearing back from you!

Marc

format log messages don't show up in cloudwatch

I was able to get this hook set up with no issues, and it works like a charm. However when making format calls to an instance of logrus.FieldLogger - like Infof - these messages do not show up in cloudwatch. I only see messages if I make non-format calls like Info. Any idea why?

If log stream already exists, batchFrequency is ignored

Unless I'm missing something, I think the batchFrequency is not honored when the log stream already exists :

https://github.com/kdar/logrus-cloudwatchlogs/blob/master/hook.go#L46-L66

	// grab the next sequence token
	if len(resp.LogStreams) > 0 {
		h.nextSequenceToken = resp.LogStreams[0].UploadSequenceToken
		return h, nil
	}

	// create stream if it doesn't exist. the next sequence token will be null
	_, err = h.svc.CreateLogStream(&cloudwatchlogs.CreateLogStreamInput{
		LogGroupName:  aws.String(groupName),
		LogStreamName: aws.String(streamName),
	})
	if err != nil {
		return nil, err
	}

	if batchFrequency > 0 {
		h.ch = make(chan *cloudwatchlogs.InputLogEvent, 10000)
		go h.putBatches(time.Tick(batchFrequency))
	}

	return h, nil

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.