blendle / zapdriver Goto Github PK

There is still a problem with concurrency in the Write function. Take the following snippet:

	lbls.mutex.RLock()
	c.tempLabels.mutex.Lock()
	for k, v := range lbls.store {
		c.tempLabels.store[k] = v
	}
	c.tempLabels.mutex.Unlock()
	lbls.mutex.RUnlock()

	// ... 

	c.tempLabels = newLabels()

The c.tempLabels variable is re-initialized after it is unlocked, but if another go routine has already locked the mutex before this line executes then it will end up calling c.tempLabels.mutex.Unlock() on the new instance of c.tempLabels which will panic because that mutex is not locked since it is newly created.

I think a better solution is to add a reset function which re-creates the internal map (protected by a mutex) without destroying the actual mutex.

As the docs describe labels needs to be logged in a JSON log field called: "logging.googleapis.com/labels".

After deploying our application to CloudRun, we seeing that the labels are not part of the labels in the LogEntry, but still are part of jsonPayload.

jsonPayload: { caller: "logging.go:61" logging.googleapis.com/labels: { x-cloud-debug: "test" } logging.googleapis.com/trace_sampled: "true" message: "passing metadata [x-rid]=[1234354]" timestamp: "2019-11-07T13:42:33.524859762Z" } labels: { instanceId: "00bf4bf02d6b0da606a48c835435d7757866a1931b821b6f7bc6402039a3ebd98be580e0cff0273b976d46eddbdc9f5b8f740972efa2866598246ddf6b4c9d9682436f" }

I'm trying to achieve the following using Zapdriver with docker's --log-driver=gcplogs driver.

But I can only manage to get the severity information in jsonPayload.data:

How can I achieve this?

Hi,

2 days ago jsonPayload mapping for ZAP to StackDriver stopped working - we now get JSON as textPayload.

It looks like that coincides with an auto-upgrade of master that updated the fluentd-gcp DaemonSet to gcr.io/stackdriver-agents/stackdriver-logging-agent:0.7-1.6.1-1

Are you aware of changes that might break zapdriver ?

I have forked this repo and am going to be maintaining it moving forward. I just merged some of the useful PRs from this repository and made a new release. I'd really appreciate it if we can have new issues and PRs created on the new fork. Thanks!

https://github.com/asahasrabuddhe/zapdriver

Is there a reason why default Encoding mode for developing config is json instead of console, like zap core does?

Is there anyway to log trace and spanId as documented

Hi, I found this line will print out fatal error: concurrent map writes
Maybe it should be wrap with a lock?

Hi, @JeanMertz.

I can't find license file.
Could you tell me this library license? (and please do commit LICENSE file!)

Best regards.

When I log error using

config := zapdriver.NewProductionConfig()
logger, err := config.Build(zapdriver.WrapCore(
	zapdriver.ReportAllErrors(true),
	zapdriver.ServiceName("service"),
))
logger.Error("something happened", zap.Error(errors.New("my error"))

I can see error in "Error reporting', no stacktrace.
Stacktrace is in stacktrace field, but it is not recognized by GCP error reporting?

According to: https://cloud.google.com/error-reporting/reference/rest/v1beta1/projects.events/report#ReportedErrorEvent
it should be included in message.

Even though this library has support for defining log labels, this isn't actually properly parsed by Stackdriver at the moment. See this PR for more details.

If the outcome of that PR is that it won't be supported, I'll remove the code from this library, if it is, I'll change the key to the right one (probably logging.googleapis.com/labels).

Until then, I'll leave it as is.

I am trying to get (some) error level logs reported to Error Reporting service. It doesn't seem to be happening when I use this library. To get reported an error log entry must have a context object attached to it: https://cloud.google.com/error-reporting/docs/formatting-error-messages

Does this library support attaching such a context so that the error gets reported?

For example https://github.com/kubernetes-sigs/controller-runtime/tree/master/pkg/log/zap returns a zap Log with a custom encoder. I can call log.Core() on it, but WrapCore uses the internal core type rather than a zapcore.Core

What I'd like to do is have the controller-runtime log (which is part of kubebuilder among others) wrapped in zapdriver.

Maybe I'm going about this all wrong...

Thanks for the package!

The following demo program prints two matching log entries:

package main

import (
	"os"

	"github.com/blendle/zapdriver"
	"go.uber.org/zap"
	"go.uber.org/zap/zapcore"
)

func main() {
	consoleEncoder := zapcore.NewConsoleEncoder(zap.NewDevelopmentEncoderConfig())

	core := zapcore.NewTee(
		zapcore.NewCore(consoleEncoder, os.Stderr, zapcore.ErrorLevel),
		zapcore.NewCore(consoleEncoder, os.Stdout, zapcore.InfoLevel),
	)

	logger := zap.New(core, zap.AddCaller())

	logger = logger.WithOptions(zapdriver.WrapCore())
	defer logger.Sync()

	logger.Info("Starting...")
}

I am actually not sure this can be fixed or simply this is a limitation of Zap, but it seems to me that Check simply cannot work properly in this case without calling Check on the underlying core. Will investigate the issue further...

Hi, is there a way to include traceId for every log? I'm using gin with otel and trying to figure out how to approach this issue

logging.googleapis.com/labels gets included twice, once with the intended content and again as an empty object.

"logging.googleapis.com/labels":{"hello":"world","hi":"universe"},"logging.googleapis.com/labels":{},

$ cat go.mod 
module example.com/m

go 1.14

require github.com/blendle/zapdriver v1.3.1

$ cat main.go 
package main

import (
        "log"

        "github.com/blendle/zapdriver"
)

func main() {
        logger, err := zapdriver.NewProduction()
        if err != nil {
                log.Fatal(err)
        }
        logger.Info(
                "Did something.",
                zapdriver.Labels(
                        zapdriver.Label("hello", "world"),
                        zapdriver.Label("hi", "universe"),
                ),
        )
}

$ go run .
{"severity":"INFO","timestamp":"2020-05-12T15:15:43.21090722-06:00","caller":"z/main.go:14","message":"Did something.","logging.googleapis.com/labels":{"hello":"world","hi":"universe"},"logging.googleapis.com/labels":{},"logging.googleapis.com/sourceLocation":{"file":"/home/tv/z/main.go","line":"14","function":"main.main"}} 

Maybe I'm doing something wrong, this is my code:

logger, _ := zapdriver.NewProduction()
logger.Info("Test log message", zap.String("hello",  "world"))

And this logs to console:

{
  "severity": "INFO",
  "timestamp": "2020-05-18T17:04:30.725484-03:00",
  "caller": "redacted/main.go:32",
  "message": "Test log message",
  "hello": "world",
  "logging.googleapis.com/labels": {},
  "logging.googleapis.com/sourceLocation": {
    "file": "redacted/main.go",
    "line": "32",
    "function": "main.init.0"
  }
}

Shouldn't hello:world be on the labels section? Thanks

For the httpRequest log object: remoteIP should be remoteIp and serverIP should be serverIp (ref: https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#HttpRequest).

Hey, thanks for zapdriver!

I am struggling to fit it in my toy project, and feel that a minimal working example as quickstart guide could be of benefit here.

For example, going with the very last section of the current README file, this does not yield the promised everything just works ™:

//name:above.go

package main

import (
        "fmt"

        "go.uber.org/zap"
        "go.uber.org/zap/zapcore"
        "github.com/blendle/zapdriver"
)

func main() {
        var log *zap.Logger
        {
                config := zap.NewDevelopmentConfig()
                config.EncoderConfig.EncodeLevel = zapcore.CapitalColorLevelEncoder
                log, _ = config.Build(zapdriver.WrapCore()) // ←
                // log = log.Named("hello-world")
                defer log.Sync()
        }

        log.Info("stub start")
        fmt.Println("Hello world!")
        log.Info("stub end")
}

$ export GOOGLE_APPLICATION_CREDENTIALS="….json"
$ go run above.go

… nothing shows up in Stackdriver,

Stackdriver seems to be ignoring log entries with the time field encoded as an RF3339 string. I was able to get it to recognize the compound formats (separate timestampSeconds and timestampNanos fields, or a timestamp object with seconds and nanos fields) when appending to the log by hand, but I couldn't figure out how to get zap to encode a time as anything but a native type.

I did discover that zapcore.EpochTimeEncoder (floating point seconds since the epoch) works with the time field. I'll file a bug with Google to clarify (or correct?) the docs.

Tests are currently failing on master

panic: runtime error: index out of range [recovered]
	panic: runtime error: index out of range

goroutine 11 [running]:
testing.tRunner.func1(0xc4200f4690)
	/usr/local/go/src/testing/testing.go:711 +0x2d2
panic(0x6d5680, 0x8cee40)
	/usr/local/go/src/runtime/panic.go:491 +0x283
github.com/blendle/zapdriver.TestWriteConcurrent(0xc4200f4690)
	<go path>/src/github.com/blendle/zapdriver/core_test.go:138 +0xa40
testing.tRunner(0xc4200f4690, 0x740eb8)
	/usr/local/go/src/testing/testing.go:746 +0xd0
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:789 +0x2de
exit status 2
FAIL	github.com/blendle/zapdriver	0.008s

Is the purpose of this test running core.Write concurrently? The test passed when I added a wait group so that logs.All() does not return an empty array.