Giter Club home page Giter Club logo

serilog-formatting-compact's Introduction

Serilog Build status NuGet Version NuGet Downloads Stack Overflow

Serilog is a diagnostic logging library for .NET applications. It is easy to set up, has a clean API, and runs on all recent .NET platforms. While it's useful even in the simplest applications, Serilog's support for structured logging shines when instrumenting complex, distributed, and asynchronous applications and systems.

Serilog

Like many other libraries for .NET, Serilog provides diagnostic logging to files, the console, and many other outputs.

using var log = new LoggerConfiguration()
    .WriteTo.Console()
    .WriteTo.File("log.txt")
    .CreateLogger();

log.Information("Hello, Serilog!");

Unlike other logging libraries, Serilog is built from the ground up to record structured event data.

var position = new { Latitude = 25, Longitude = 134 };
var elapsedMs = 34;

log.Information("Processed {@Position} in {Elapsed} ms", position, elapsedMs);

Serilog uses message templates, a simple DSL that extends .NET format strings with named as well as positional parameters. Instead of formatting events immediately into text, Serilog captures the values associated with each named parameter.

The example above records two properties, Position and Elapsed, in the log event. The @ operator in front of Position tells Serilog to serialize the object passed in, rather than convert it using ToString(). Serilog's deep and rich support for structured event data opens up a huge range of diagnostic possibilities not available when using traditional loggers.

Rendered into JSON format for example, these properties appear alongside the timestamp, level, and message like:

{"Position": {"Latitude": 25, "Longitude": 134}, "Elapsed": 34}

Back-ends that are capable of recording structured event data make log searches and analysis possible without log parsing or regular expressions.

Supporting structured data doesn't mean giving up text: when Serilog writes events to files or the console, the template and properties are rendered into friendly human-readable text just like a traditional logging library would produce:

09:14:22 [INF] Processed {"Latitude": 25, "Longitude": 134} in 34 ms.

Upgrading from an earlier Serilog version? Find release notes here.

Features

  • Community-backed and actively developed
  • Format-based logging API with familiar levels like Debug, Information, Warning, Error, and so-on
  • Discoverable C# configuration syntax and optional XML or JSON configuration support
  • Efficient when enabled, extremely low overhead when a logging level is switched off
  • Best-in-class .NET Core support, including rich integration with ASP.NET Core
  • Support for a comprehensive range of sinks, including files, the console, on-premises and cloud-based log servers, databases, and message queues
  • Sophisticated enrichment of log events with contextual information, including scoped (LogContext) properties, thread and process identifiers, and domain-specific correlation ids such as HttpRequestId
  • Zero-shared-state Logger objects, with an optional global static Log class
  • Format-agnostic logging pipeline that can emit events in plain text, JSON, in-memory LogEvent objects (including Rx pipelines) and other formats

Getting started

Serilog is installed from NuGet. To view log events, one or more sinks need to be installed as well, here we'll use the pretty-printing console sink, and a rolling file set:

dotnet add package Serilog
dotnet add package Serilog.Sinks.Console
dotnet add package Serilog.Sinks.File

The simplest way to set up Serilog is using the static Log class. A LoggerConfiguration is used to create and assign the default logger, normally in Program.cs:

using Serilog;

Log.Logger = new LoggerConfiguration()
    .WriteTo.Console()
    .WriteTo.File("log.txt",
        rollingInterval: RollingInterval.Day,
        rollOnFileSizeLimit: true)
    .CreateLogger();

try
{
    // Your program here...
    const string name = "Serilog";
    Log.Information("Hello, {Name}!", name);
    throw new InvalidOperationException("Oops...");
}
catch (Exception ex)
{
    Log.Error(ex, "Unhandled exception");
}
finally
{
    await Log.CloseAndFlushAsync(); // ensure all logs written before app exits
}

Find more, including a runnable example application, under the Getting Started topic in the documentation.

Getting help

To learn more about Serilog, check out the documentation - you'll find information there on the most common scenarios. If Serilog isn't working the way you expect, you may find the troubleshooting guide useful.

Serilog has an active and helpful community who are happy to help point you in the right direction or work through any issues you might encounter. You can get in touch via:

We welcome reproducible bug reports and detailed feature requests through our GitHub issue tracker; note the other resource are much better for quick questions or seeking usage help.

Contributing

Would you like to help make Serilog even better? We keep a list of issues that are approachable for newcomers under the up-for-grabs label (accessible only when logged into GitHub). Before starting work on a pull request, we suggest commenting on, or raising, an issue on the issue tracker so that we can help and coordinate efforts. For more details check out our contributing guide.

When contributing please keep in mind our Code of Conduct.

Detailed build status

Branch AppVeyor
dev Build status
main Build status

Serilog is copyright © Serilog Contributors - Provided under the Apache License, Version 2.0. Needle and thread logo a derivative of work by Kenneth Appiah.

serilog-formatting-compact's People

Contributors

adamchester avatar bbrandt avatar bdovaz avatar joostas avatar liorbanai avatar nblumhardt avatar simoncropp avatar sungam3r avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

serilog-formatting-compact's Issues

CompactJsonFormatter and RenderedCompactJsonFormatter options

Hi, thanks for an awesome package. Request to make a PR / feature change.

We are logging with Serilog via a common adapter to loki.. With CompactJsonFormatter and RenderedCompactJsonFormatter as the formatter.

We would like veryt much that information level is always added to the message. and we can rename the default @fields to specific values. Yes, we can use the ExpressionTemplate formatted also. But this does not include the convenient logic included in the CompactJsonFormatter and RenderedCompactJsonFormatter.

Can I make a PR to add options to the ctor for CompactJsonFormatter and RenderedCompactJsonFormatter? Then ppl can control some behaviour, but keep the default as is?

How to change property names

I'm using the CompactJsonFormatter because I want my custom properties to be at the root of the JSON object (i.e., not nested under "Properties" and including other properties I haven't explicitly requested as part of the message template.
However, I really don't want to have "@" in my property names. It makes searching in some log sinks very difficult which defeats the purpose of semantic logging for me. How can I change @l and @t to other things?

Add Ability to log timestamps in local time with offset

Logging time in UTC by default is great and saves a few (5) bytes per log, but there may be cases (like mine) where I need to create the logs in local time so that timestamps line up with the machine time and don't require extra processing when viewed in a text editor or CLEF viewer.

Proposed change would be to add an optional Boolean parameter to the constructor with a default value of false that indicates logs should be written in local time with offset.

I have already created a modified version that accomplishes this, I will submit a pull request that enables this functionality if you want to merge it in.

Configuration flag for RenderedJsonFormatter

Is there a way to specify in the Serilog config that the desired format is RenderedJsonFormatter?
I know that in order to obtain it, we have to do a new JsonFormatter(renderMessage: true) when configuring the sink.

The value for CompactJsonFormatter is:
"Serilog.Formatting.Compact.CompactJsonFormatter, Serilog.Formatting.Compact".
Does the equivalent exist for RenderedJsonFormatter?

Allow overrides for @mt and @t

We have an entire custom formatter to rename @mt to MSG and @t to TS. It'd be great if we could just ditch that code altogether and have a mechanism for renaming the properties in the compact json formatter.

Configuration syntax for dotnet core appsettings.json

For an app targeting dotnet core 2.1 with appsettings.json config file, I'm trying the following config section but doesn't seem to work

"Serilog": { "WriteTo": [ { "Name": "Console", "Args": { "formatter": "Serilog.Formatting.Compact.CompactJsonFormatter, Serilog.Formatting.Compact" } } ] }

Is the formatter supposed to be specified in the Args sub-section?

Custom mapping for context properties

Hi,

What do you think about providing a way to configure compact keys for context properties? Consider

{
	"@t": "2020-10-19T09:25:53.5767574Z",
	"@mt": "{Protocol} {Method} {RequestPath} {StatusCode} in {RequestDuration} ms",
// Inspired by Serilog middleware
	"Protocol": "HTTP/1.1",
	"Method": "GET",
	"StatusCode": 200,
	"RequestDuration": 23.5447,
// End inspired ...
	"SourceContext": "Namespace.Diagnostics.Hosting",
// Start HostingLoggerExtensions.cs HLE
	"RequestId": "0HM3FHOBGSJDL:0000204C",
	"RequestPath": "/metrics",
	"SpanId": "297e422451229e48",
	"TraceId": "1492670e02e7a8459a72c677fc9ce3be",
	"ParentId": "0000000000000000",
// End HLE
	"ConnectionId": "0HM3FHOBGSJDL"
}

RequestPath added by hosting so it's reused to avoid additional properties.

I've implemented a diagnostics event listener to emit logs lines identical to those in datalust middleware.

Is there a built-in way to combine JSON formatted events into a single JSON object (file) using Serilog?

I'm working on a project that uses Serilog to create its log files in a JSON format. The idea is that (probably) once per day, the more human-readable aspects of the log files will be used as a source to generate an email to the system's user, giving them the detail of any problems that have arisen.

Currently, I'm using the RenderedCompactJsonFormatter to pass events out to a log file, but they are recorded as individual JSON 'blobs', separated by a carriage return.

Does Serilog have a means by which a single JSON object, containing an array of event objects, can be generated per file?

compact json formatter @t overriding with utc +3

hi,
l am using serilog compact json formatter in my project. And my logs always have utc datetime in @t info. l saw @t gives an İso 8601 timestamp. l should override it with utc+3 .
is it possible? l should find anything.

Note: l try this , l used enricher and l got utc +3. but when i used compact json formatter, it remove timestamp utc+3 and gives again @t as iso 8601 timestamp. how can i override it?
thanks.

Shorthand property name for SourceContext

Hi, I was looking at my Serilog message formatted by RenderedCompactJsonFormatter in my ELK instance and I noticed that SourceContext is not among the special short-named keys. 99.9% of my log messages will contain the SourceContext property and I would expect that to hold for most projects.

Would it make sense to add it to the list of compact keys as "@sc" or something?

Publish the CLEF format via GitHub Pages

It'd be nice to publish the CLEF format details section from the README separately from this repository, since there are multiple tools now producing and consuming CLEF without necessarily using this library.

I've created @clef-json as an organization and plan to hook up GitHub pages to publish the spec via clef-json.org when I have a spare sec.

The format details will/should also stay here in the README, and I don't think there should be any changes to the content, except to clarify what's already covered. Just posting this here as a heads-up in case anyone's interested to follow along :-)

How to add more properties with a `@` sigil?

Currently these are escaped during output:

if (name.Length > 0 && name[0] == '@')
{
// Escape first '@' by doubling
name = '@' + name;
}

We wanted to add some more properties to use with Seq but they are escaped.
Is there any way to get these working?
I know we could just fork the repo and make the change, but I was just wondering why this part is like this and if there are plans to add an option to turn off escaping.

Include Information level to the log

Although not rendering "l : Information" saves a bit of bandwidth it's much less convinient to work with. Consider this JSON is going to ELK or something. Now you can't search for say "Information OR Debug" filter because you just can't create it via UI. You will also have to explain it to every user and users would have to explain it in their own teams.

I suggest to make it at least configurable.

Question: Is it possible to camelcase the properties?

Hello!

Thank you for an awesome lib!

Is it possible somehow to camelcase the outputed json properties?

Current:

{"@t":"2016-06-07T03:44:57.8532799Z","@mt":"Hello, {User}","User":"nblumhardt"}

Wanted (note the small u on user):

{"@t":"2016-06-07T03:44:57.8532799Z","@mt":"Hello, {User}","user":"nblumhardt"}

Reader

I've put together a basic reader for the (non-rendered) compact JSON format here:

https://github.com/nblumhardt/serilog-formatting-compact-reader

There would be some benefits maintaining this alongside the formatters, but also more effort in making the implementation complete and supportable. (In any case it would remain a separate package, as there's a JSON.NET dependency in there.)

Raising this for discussion/tracking.

Removing default values

Is there any way to remove default values from being logged? It would reduce our payloads quite significantly, especially for nullable types/

Supply `IFormatProvider` to `CompactJsonFormatter`/`RenderedCompactJsonFormatter`

Message template rendering is culture-specific; the constructors for the included formatter types should accept an optional IFormatProvider (likely through an overload) so that consumers can opt into the locale of their choosing.

Note that v2 dev currently uses CultureInfo.InvariantCulture inline in the source code, via #54, which is a breaking change (the old default would use the current locale).

When this change is implemented, we should revert that breakage and again default to the current culture.

Force Log level to be written

I can see that @l is written only for non-info cases, judging from the README:

image

I have a few issues:

  • @l seems to be not written for non-info cases as well (e.g. Error)
  • I want to enforce @l to be written in every case. Is that possible?

Introduce a way to format Exceptions

In your current implementation exceptions are formatted as "ex.ToString()".
It would be great to format the Exception as JSON, spitting out the Message, Type, Method and StackTrace as JSON properties. The StackTrace could be formatted as a JSON array, where each line is one string in the array. You can see that concept in action in this PR.
This is extremelly helpful when aggregating exceptions by Type or Message in tools like Splunk.
I ended implementing this in my own ITextFormatter, but I think this should be part of the library.
If/when you decide to iterate on the JSON formatter, I would be more than happy to contribute with my solution.
Since Exceptions are a very important element of error logging, have you considered introducing an IExceptionFormatter, with a method FormatException, and sink authors could implement their own formatting (in this case JSON).

RenderedCompactJsonFormatter doesn't allow control of Exception format

Line 83 of RenderedCompactJsonFormatter does a .ToString() call on LogEvent.Exception if not null.

I have an issue whereby our Splunk back-end has a limit to the number of characters in an event message (which sadly also includes the extra fluff Windows Event Log adds - company standard dictates logs are forwarded from Event Log.)

When an Exception gets logged in our aspnetcore applications the StackTrace is invariably quite long, due to the many call stack frames added by layers of framework code plus our own middleware and Controller/Domain/Infrastructure code.

I would like to have control over the rendered output of the LogEvent.Exception property, but since the property is read-only I can't replace it via an enricher, so at the moment it seems near impossible without editing RenderedCompactJsonFormatter or adding my own implementation of ILogger.

Please advise - is this achievable via some other means such as Expressions?

Aside from this I quite like RenderedCompactJsonFormatter, it does a good job of saving space with the abbreviated prop names, but seems it lacks a bit of extensibility.

Level field (@l) is not included when it's set to Information

Hello,

As per the lines below, the log level is not included if it's set to Informational:

I know the package is named compact, but the mentioned functionality is really not intuitive. Spent an hour or so to realize why events pushed to Graylog are missing the Log Level when using RenderedCompactJsonFormatter, yet, working OK when using JsonFormatter.

Would it be possible to to remove that condition?

Add built-in field name for EventId

When using Serilog behind the Microsoft.Extensions.Logging facade, which I think is becoming the recommended best practice in .NET Core, one often includes an EventId in a log call:

_logger.LogInformation(new EventId(1001, "TransactionBegin"), "Beginning transaction {TrxnId}...", 12345);

When using this formatter with the File sink, one gets output like the following:

{"@t":"2020-10-29T16:03:30.6943793Z","@mt":"Beginning transaction {TrxnId}...","EventId":{"Id":1001,"Name":"TransactionBegin"},"SourceContext":"MyNamespace.MyType","TrxnId":12345}

Because EventIDs are common with Microsoft.Extensions.Logging, I think it would be nice to recognize EventId as a built-in field, and shorten it to @e or @ei. Because Id and Name are nested JSON, you could also shorten them to @i and @n, thought that might require some JSON parsing that could slow down the formatter, idk. "EventId" -> "@e" would be the big one.

@ character

just curious: why are the properties prefixed with an @ character?

Render JSON without rendering @mt

I have a feeling that perhaps I'm 'doing Serilog wrong', but what I'm trying to do is log a message along with a some JSON-formatted data, and my own correlation ID. For example, this is the output I'm looking for:

{"@t":"2017-05-08T21:32:05.7713278Z","@m":"Oh noes!","@l":"Warning","@i":"bc5016e7-198c-4345-9735-b0c4003b5178","Properties":{"NodeId":"a1882552-17eb-42d1-b7cd-e895178167fb","ObjectName":"C:\\Windows\\System32\\something.exe","Other":"Other stuff here"}}

No matter what combination of JsonFormatter/CompactJsonFormatter/RenderedCompactJsonFormatter and reified properties I use in my message template, I just can't get the desired output.

Is there some way to do what I'm looking for? That is, to exclude @mt from the output, outputting a @m message while also outputting JSON?

Won't compile when referencing Serilog.Formatting.Compact in Azure DevOps

Hi!

I have an application that is built in Azure DevOps. When just adding Serilog and a few sinks, everything was fine. However, I went to add Serilog.Formatting.Compact so that I could try out CompactJsonFormatter() and when I do it suddenly breaks the build. Worth noting everything builds fine on my dev machine; only breaks on Azure DevOps using vmImage: windows-2019

Error Log:
Program.cs(26,26): Error CS0234: The type or namespace name 'Compact' does not exist in the namespace 'Serilog.Formatting' (are you missing an assembly reference?)

Code:
line 26: using Serilog.Formatting.Compact;

Why level info is not present in output

It is stated in documentation that absence of log level means information.

I want to know the reason behind this decision and if there's a way to workaround it and still output the log level in all cases.

Possibility to include both @m and @mt in json

Currently we can decide only either having the rendered message or the message template. For us it would be useful, to include both of those in the json, since it would tremendously simplified filtering and searching of logs in elastic search/kibana. It would allow us to easily filter out non interesting events by the message template, when digging in logs, while still keeping the benefit of having rendered message (better readibility).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.