mattwcole / gelf-extensions-logging Goto Github PK

Hey,
The Gelf.Extensions.Logging.dll is built against version 4.6.3 of System.Threading.Tasks.Dataflow:

This doesn't exist in the public nuget feed so it can cause binding issues without a redirect.

I am using this extension in several projects here in my company, however our Graylog servers are protected by basic authentication in the "Http" input, where it is necessary to add "Authorization: Basic" in the requests to append logs, where it is not possible with this extension. I found a temporary solution, use a proxy to add this header. I made a local implementation, where I added a 'Headers' option in GelfLoggerOptions and in HttpGelfClient added all the headers, could raise those changes via pull request and publish a version updated extension in nuget?

GelfLoggerOptions

/// <summary> /// Http /// </summary> public Dictionary<string, string> Headers { get; set; } = new Dictionary<string, string>();

HttpGelfClient

if(options.Headers != null && options.Headers.Count > 0) { foreach (var header in options.Headers) { _httpClient.DefaultRequestHeaders.Add(header.Key, header.Value); } }

Trying to use this with a .Net Core 3 give an error when running the host. Any ideas?

System.AggregateException
HResult=0x80131500
Message=An error occurred while writing to logger(s). (Could not load type 'Microsoft.Extensions.Logging.Internal.FormattedLogValues' from assembly 'Microsoft.Extensions.Logging.Abstractions, Version=3.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'.)
Source=Microsoft.Extensions.Logging
StackTrace:
at Microsoft.Extensions.Logging.Logger.ThrowLoggingError(List1 exceptions) at Microsoft.Extensions.Logging.Logger.Log[TState](LogLevel logLevel, EventId eventId, TState state, Exception exception, Func3 formatter)
at Microsoft.Extensions.Logging.Logger1.Microsoft.Extensions.Logging.ILogger.Log[TState](LogLevel logLevel, EventId eventId, TState state, Exception exception, Func3 formatter)
at Microsoft.Extensions.Logging.LoggerExtensions.Log(ILogger logger, LogLevel logLevel, EventId eventId, Exception exception, String message, Object[] args)
at Microsoft.Extensions.Logging.LoggerExtensions.Log(ILogger logger, LogLevel logLevel, String message, Object[] args)
at Microsoft.Extensions.Logging.LoggerExtensions.LogInformation(ILogger logger, String message, Object[] args)
at Coravel.Scheduling.HostedService.SchedulerHost.Dispose()
at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.DisposeAsync()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ValueTaskAwaiter.GetResult()
at Microsoft.Extensions.Hosting.Internal.Host.d__12.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ValueTaskAwaiter.GetResult()
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.d__4.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
at CappImporter.Program.Main(String[] args) in D:\HG_PROJECTS\CappImporter\Program.cs:line 25

Inner Exception 1:
TypeLoadException: Could not load type 'Microsoft.Extensions.Logging.Internal.FormattedLogValues' from assembly 'Microsoft.Extensions.Logging.Abstractions, Version=3.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'.

Configuration is:
.ConfigureLogging((hostContext, services) =>
{
services.ClearProviders();
services.AddConsole();
services.AddGelf(options =>
{
options.Host = "SERVER-NAME";
options.LogSource = "LOG-Source";
});

Hi Mattwcole,

As Gelf providses the facility of Logger.BeginScope() to add additional field to structured log.

If I have to add the correlationId in every log then I have to do something like
using (_logger.BeginScope(("correlation_id", correlationId)))
{
// Field will be added to all logs within this scope (using any ILogger instance).
}

So for everylog I have to wrap with using statement.

Is there a way that I dont have to write that everytime but just done once.

A quick solution I think is like I care wrapper to loginfo/logerror/logdebug with using statement having correlation id as scop.

Do you suggest a better way?

Thanks,
Keyur

Hi,

I tried to set the key CorrelationId using _logger.BeginScope(("CorrelationId", "my-correlation-id")), but it doesn't work because ASP.NET Core sets a correlation id scope on the logger outside of my control. I believe it would be better that if I set a CorrelationId (or any other duplicate keyed scope) the innermost scope should be used instead of the outermost.

I created a failing test which shows what I mean:

[Fact]
public async Task When_duplicate_scope_keys_inner_scope_should_be_used()
{
    var messageText = _faker.Lorem.Sentence();

    var sut = _loggerFixture.CreateLogger<GelfLoggerTests>();
    using (sut.BeginScope(("foo", "outer")))
    {
        using (sut.BeginScope(("foo", "inner")))
        {
            sut.LogCritical(messageText);
        }
    }

    var message = await _graylogFixture.WaitForMessageAsync();

    Assert.Equal("inner", message.foo); // Fails. message.foo is "outer"
}

I'm seeing the follow error periodically in my logs....~ 0.5% of requests.

System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.AspNetCore.Http.DefaultHttpContext.get_Items()

I only get this error when the GelfLoggerProvider is added. From what I've read this could be due to accessing the DefaultHttpContext from multiple threads. I've tried disabling scopes to see if that helped, but I still see the error.

I would also like to point out I only notice this when running on AWS ECS.

The default logger factory can provide a scope implementation via the ISupportExternalScope and IExternalScopeProvider interfaces.

For reference, see:
Microsoft.Extensions.Logging/LoggerFactory.cs and Microsoft.Extensions.Logging/LoggerFactoryScopeProvider.cs

This gets you features like dotnet/runtime#34305 when using the default logger factory instance.

Can support be added for ISupportExternalScope and IExternalScopeProvider?

Hi,

we've encountered an issue with our logging format strings which looks like this:

logger.LogInformation("Some testing! {0} {1}", 123, 456);

because these numerical placeholders ended up in Graylog like this

Is it possible to make structured logging optional, or maybe ignore numbers in format strings?

May you please add support for IEnumerable<KeyValueParir<string, string>> for supported fields? I have multiple key value pairs that must be sent in one log message, but currently, there is no way to achieve this.

I want to do the following:

using (_logger.BeginScope(new List<KeyValuePair<string, string>>
{
    new KeyValuePair<string, string>("host", ControllerContext.HttpContext.Request.Host.ToString()),
    new KeyValuePair<string, string>("user_agent", ControllerContext.HttpContext.Request.Headers["user-agent"])
}))
{
    _logger.LogInformation("Some mesage");
}

AFAIK, the current allowed types will not allow me to achieve this. I want to include all key value pairs in a single log message,

using (_logger.BeginScope(new Dictionary<string, object>
{
    ["host"] = ControllerContext.HttpContext.Request.Host.ToString(),
    ["user_agent"] = ControllerContext.HttpContext.Request.Headers["user-agent"]
}))
{
    _logger.LogInformation("Some mesage");
}

The above does not work, the additional fields are ignored.

As the title says. There is only the numeric level which is not human friendly.

The GELF specification supports numeric additional fields, and graylog requires fields to be numeric if we want to calculate statistics on the fields. This affects fixed additional fields, fields from scope and fields from structured logs.

I have expanded the test Sends_message_with_additional_fields_from_structured_log() with a failing test case for what I mean here

[Fact]
public async Task Sends_message_with_additional_fields_from_structured_log()
{
    var sut = _loggerFixture.CreateLogger<GelfLoggerTests>();
    sut.LogDebug("Structured log line with {first_value}, {second_value} and {numeric_value}", "foo", "bar", 123);

    var message = await _graylogFixture.WaitForMessageAsync();

    Assert.Equal("Structured log line with foo, bar and 123", message.message);
    Assert.Equal("foo", message.first_value);
    Assert.Equal("bar", message.second_value);
    Assert.Equal(123, message.numeric_value); // This line fails
}

How to use from MAUI app?

This is our code but we received System.MethodAccessException. Could you help please?

var builder = MauiApp.CreateBuilder();
builder.Logging.AddGelf(opt =>
{
    opt.Host = "172.182.28.1";
    opt.Port = 12201;
    opt.Protocol = GelfProtocol.Udp;
    opt.LogSource = "My App";
});

Exception

System.MethodAccessException: Method `Microsoft.Extensions.Logging.ProviderAliasUtilities.GetAlias(System.Type)' is inaccessible from method `Microsoft.Extensions.Logging.Configuration.LoggerProviderConfigurationFactory.GetConfiguration(System.Type)'
  at at Microsoft.Extensions.Logging.Configuration.LoggerProviderConfigurationFactory.GetConfiguration(Type providerType)
  at at Microsoft.Extensions.Logging.Configuration.LoggerProviderConfiguration`1[[Gelf.Extensions.Logging.GelfLoggerProvider, Gelf.Extensions.Logging, Version=2.4.0.0, Culture=neutral, PublicKeyToken=null]]..ctor(ILoggerProviderConfigurationFactory providerConfigurationFactory)
  at at System.Reflection.RuntimeConstructorInfo.InternalInvoke(Object obj, Object[] parameters, Boolean wrapExceptions)
  at at System.Reflection.RuntimeConstructorInfo.DoInvoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
  at at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.RuntimeResolverContext, Microsoft.Extensions.DependencyInjection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60],[System.Object, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].VisitCallSiteMain(ServiceCallSite callSite, RuntimeResolverContext argument)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.RuntimeResolverContext, Microsoft.Extensions.DependencyInjection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60],[System.Object, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].VisitCallSite(ServiceCallSite callSite, RuntimeResolverContext argument)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.RuntimeResolverContext, Microsoft.Extensions.DependencyInjection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60],[System.Object, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].VisitCallSiteMain(ServiceCallSite callSite, RuntimeResolverContext argument)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.RuntimeResolverContext, Microsoft.Extensions.DependencyInjection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60],[System.Object, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].VisitCallSite(ServiceCallSite callSite, RuntimeResolverContext argument)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitIEnumerable(IEnumerableCallSite enumerableCallSite, RuntimeResolverContext context)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.RuntimeResolverContext, Microsoft.Extensions.DependencyInjection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60],[System.Object, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].VisitCallSiteMain(ServiceCallSite callSite, RuntimeResolverContext argument)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.RuntimeResolverContext, Microsoft.Extensions.DependencyInjection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60],[System.Object, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].VisitNoCache(ServiceCallSite callSite, RuntimeResolverContext argument)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.RuntimeResolverContext, Microsoft.Extensions.DependencyInjection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60],[System.Object, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].VisitCallSite(ServiceCallSite callSite, RuntimeResolverContext argument)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.RuntimeResolverContext, Microsoft.Extensions.DependencyInjection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60],[System.Object, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].VisitCallSiteMain(ServiceCallSite callSite, RuntimeResolverContext argument)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitDisposeCache(ServiceCallSite transientCallSite, RuntimeResolverContext context)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.RuntimeResolverContext, Microsoft.Extensions.DependencyInjection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60],[System.Object, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].VisitCallSite(ServiceCallSite callSite, RuntimeResolverContext argument)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.RuntimeResolverContext, Microsoft.Extensions.DependencyInjection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60],[System.Object, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].VisitCallSiteMain(ServiceCallSite callSite, RuntimeResolverContext argument)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.RuntimeResolverContext, Microsoft.Extensions.DependencyInjection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60],[System.Object, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].VisitCallSite(ServiceCallSite callSite, RuntimeResolverContext argument)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.RuntimeResolverContext, Microsoft.Extensions.DependencyInjection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60],[System.Object, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].VisitCallSiteMain(ServiceCallSite callSite, RuntimeResolverContext argument)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.RuntimeResolverContext, Microsoft.Extensions.DependencyInjection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60],[System.Object, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].VisitCallSite(ServiceCallSite callSite, RuntimeResolverContext argument)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitIEnumerable(IEnumerableCallSite enumerableCallSite, RuntimeResolverContext context)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.RuntimeResolverContext, Microsoft.Extensions.DependencyInjection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60],[System.Object, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].VisitCallSiteMain(ServiceCallSite callSite, RuntimeResolverContext argument)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.RuntimeResolverContext, Microsoft.Extensions.DependencyInjection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60],[System.Object, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].VisitCallSite(ServiceCallSite callSite, RuntimeResolverContext argument)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.RuntimeResolverContext, Microsoft.Extensions.DependencyInjection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60],[System.Object, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].VisitCallSiteMain(ServiceCallSite callSite, RuntimeResolverContext argument)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.RuntimeResolverContext, Microsoft.Extensions.DependencyInjection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60],[System.Object, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].VisitCallSite(ServiceCallSite callSite, RuntimeResolverContext argument)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.RuntimeResolverContext, Microsoft.Extensions.DependencyInjection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60],[System.Object, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].VisitCallSiteMain(ServiceCallSite callSite, RuntimeResolverContext argument)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.RuntimeResolverContext, Microsoft.Extensions.DependencyInjection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60],[System.Object, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].VisitCallSite(ServiceCallSite callSite, RuntimeResolverContext argument)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.RuntimeResolverContext, Microsoft.Extensions.DependencyInjection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60],[System.Object, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].VisitCallSiteMain(ServiceCallSite callSite, RuntimeResolverContext argument)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.RuntimeResolverContext, Microsoft.Extensions.DependencyInjection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60],[System.Object, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].VisitCallSite(ServiceCallSite callSite, RuntimeResolverContext argument)
  at at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
  at at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(Type serviceType)
  at at System.Collections.Concurrent.ConcurrentDictionary`2[[System.Type, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Func`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope, Microsoft.Extensions.DependencyInjection, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60],[System.Object, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetOrAdd(Type key, Func`2 valueFactory)
  at at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
  at at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType)
  at at Microsoft.Maui.MauiContext.WrappedServiceProvider.GetService(Type serviceType)
  at at Microsoft.Maui.MauiContext.WrappedServiceProvider.GetService(Type serviceType)
  at at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
  at at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[IApplication](IServiceProvider provider)
  at at Microsoft.Maui.MauiUIApplicationDelegate.FinishedLaunching(UIApplication application, NSDictionary launchOptions)
  at at UIKit.UIApplication.Main(String[] args, Type principalClass, Type delegateClass)
  at fire_trails.Program.Main(String[] args) in /Users/tiendinh/Projects/fire-trails/Platforms/iOS/Program.cs:13

Hi ,

Based on the sample example for .NeCore2

I have configure GelfLogger and then injecting the Ilogger
.
Then using the logger.LogInformation("Log in gray log").

But its not appears in the Graylog.

On application start I am logging exactly as the sample example and its logs in gay log but not from controller.

`public void ConfigureServices(IServiceCollection services)
{
services.Configure(Configuration.GetSection("Graylog"))
.AddLogging(loggingBuilder => loggingBuilder
.AddConfiguration(Configuration.GetSection("Logging"))
.AddGelf());

using (var serviceProvider = services.BuildServiceProvider())
{
UseLogger(serviceProvider.GetRequiredService<ILogger>());
}
}`

private static void UseLogger(ILogger logger)
{
const string framework = "netcoreapp-4.0";

		logger.LogInformation("Information log from {framework}", framework);

		using (logger.BeginScope(("scope_field_1", "foo")))
		{
			logger.LogDebug("Debug log from {framework}", framework);

			using (logger.BeginScope(new Dictionary<string, object>
			{
				["scope_field_2"] = "bar",
				["scope_field_3"] = "baz"
			}))
			{
				logger.LogTrace("Trace log from {framework}", framework);
			}

			logger.LogError(new EventId(), new Exception("Example exception!"),
				"Error log from {framework}", framework);
		}
	}

public ValueController( ILogger logger)
{
this.logger = llogger;

	}

public async Task GetValues()
{
try
{

				logger.Log(LogLevel.Information, "Get all years method is called{framework}", "2.0");
				logger.LogDebug("Get all years method is called{framework}", "2.0");
				logger.LogError("error occured");

		

			
		}
		catch (System.Exception ex)
		{
			// log exceptions
			return StatusCode((int)HttpStatusCode.InternalServerError);
		}
	}

using .net core 2.1 final previews 2

Appreciate your help.

Thanks,
Keyur

Every time GelfMessageProcessor processes a message, it calls GelfUdpClient.SendMessageAsync, which in turn calls UDPClient.SendAsync, which in turn does a DNS lookup on every call.

The impact of this becomes especially apparent when running in an environment without OS-level DNS caching, which is common when running in a container on Linux. In Kubernetes, for example, this implies a round-trip to a coredns pod (potentially not even running on the same node) for every log message, and as GelfMessageProcessor is single-threaded, this can cause logging to start lagging behind quite a bit under load. As the capacity of the message buffer is limited, this will at some point even start blocking application code.

Substituting the server's hostname with an IP address resolved these issues for us. I can wrap together an MR to solve this issue in the package itself, but I'm a bit torn on how much caching makes sense here.

1. I have ran Docker
   https://docs.graylog.org/en/4.0/pages/installation/docker.html?highlight=docker

2. Then I little bit corrected appjson file:

{
  "Logging": {
    "Console": {
      "LogLevel": {
        "Default": "Debug"
      }
    },
    "GELF": {
      "Host": "127.0.0.1",
      "Port": 12201,
      "LogSource": "console-app-1",
      "LogLevel": {
        "Default": "Debug",
        "Microsoft": "Debug",
        "Gelf.Extensions.Logging.Samples.NetCore2": "Debug",
        "Gelf.Extensions.Logging.Samples.NetCore2.Program": "Debug"
      },
      "LogLevel2": {
        "Microsoft.AspNetCore.Mvc.Razor": "Error",
        "Default": "Trace"
      },
      "AdditionalFields": {
        "project_name": "my-project"
      }
    }
  }
}

3. Run the application, it works but when I go to browser
   http://localhost:9000/search?q=&rangetype=relative&relative=1800
   I do not see any records.

What I do wrong?

Hi,

Could I suggest adding some additional functionality to provide some more extensibility to the logger?
I would like to see a new property in GelfLoggerOptions, with a name AdditionalFunctionFields of type Dictionary<string, Func<GelfMessage, object>>.
Purpose of this new functionality is to add more information in log that are derived from the values in the log message.
Some of the possible scenarios:

1. We would like to introduce loglevel field with custom string value based on Level (we do lose some from LogLevel, but can live with that; maybe this is a hidden additional suggestion to keep this field in GelfMessage so it could be used for similar purpose)
2. We would like to log Exception type to a custom field based on Exception value

If you agree, I would fork and try to create solution which should be the least intrusive and send it to you for a review.

What do you think?

Thanks!

The library does not currently work with ASP .NET Core 2.1 prereleases.

Logging setup has changed in .NET 2.0. Details described here. Some small additions are needed to support the new API.

Hi,

From what I can see this only supports UDP endpoints. It would be nice to have support for HTTP(S) endpoints as well.
If you are willing to merge it, I can create pull request for that.

It seems this library does not go well with Microsoft.Extensions.Logging.Abstractions v. 3.0.
When I combine them, I get the following exception:

System.AggregateException: 'Could not load type 'Microsoft.Extensions.Logging.Internal.FormattedLogValues' from assembly 'Microsoft.Extensions.Logging.Abstractions, Version=3.0.0.0'

Even when I try to load v. 2.2.0 of the abstractions library into my netcore 3.0 app, it still produces this exception. Seems like the new framework is somehow forcing the 3.0 version.

If I configure GELF like this, I will receive messages from Microsoft packages to GrayLog.
GELF node doesn't contain LogLevel node.

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Warning",
    },
    "GELF": {
      ...
    }
  }
}

From what I understand GELF should inherit settings top level regarding LogLevel, but it doesn't. Is this intended behavior or bug?

We have an issue where we want to have a global counter for each log statement so that we can ensure that the logs are displayed in the correct order (the millisecond resolution of graylog is not granular enough for us) and I wondered if it is possible to support a Func callback in the AdditionalField dictionary which is executed every time we log something.

I am thinking we could support something like this

int count = 0;
using(_logger.BeginScope(new Dictionary<string, object>
{
    { "count", () => ++count },
    { "bar" , "baz" },
}) {
    _logger.LogInformation("Something happened. New state is {state}", "my state");
    ...
    _logger.LogWarning("Oh no. {exception}", ex);
}

Where the gelf output would be: something like

{
  ...
  "message": "Something happened. New state is my state",
  "_state": "my state",
  "_count": 1,
  "_bar": "baz"
}
{
  ...
  "message": "Oh no. NullReferenceExceptionOrSomethingLikeThat",
  "_exception": "NullReferenceExceptionOrSomethingLikeThat",
   "_count": 2,
   "_bar": "baz"
}

Could this be in scope for this project? I can create a pull request if you think this should be supported

The default value of UdpClient.DontFragment is true. This means that any UDP GELF chunk greater than the MTU will be silently dropped. The observed behavior is that all small log messages work fine but any large log messages just don't show up.

IMO, the best solution is to set DontFragment to false here. This does mean that large log messages will be chunked and then fragmented, but that's better than dropping them completely.

A more complex solution is possible: detecting MTU (and MTU changes) by keeping DontFragment set and handling the ICMP Fragmentation Needed responses, somewhat similar to how QUIC detects MTU. But that's probably very complex and way out of scope for this library.

Using the client with Graylog hosted on Azure Kubernetes I found out that big log messages were missing on my stream.
Troubleshooting it I notice that messages longer than 1493 bytes were never being received in the UDP port. To be sure of it I dropped the Graylog Input listener and used netcat to start listening on the same UDP port and submitted messages of different sizes.
Running Graylog on a VirtualBox machine I could not reproduce this issue, only when its running on AKS.

I found this related issue on Graylog repository: Graylog2/graylog2-server#4317
Quoting joschi:

Most GELF libraries support configuring the chunk size, so you might want to reduce the size in your network if you currently have problems.

Expanding the code I notice that the max chunk size is defined in a constant, making it impossible to configure the client to my network issue.

My environment

Graylog version: 4.0.2
Hosting: Azure AKS
Client: .NET Core 3.1 + Windows

Hi,

I stumbled upon a bug where field.Value.ToString() was producing a null exception because field.Value was null
see here:
https://github.com/mattwcole/gelf-extensions-logging/blob/dev/src/Gelf.Extensions.Logging/UdpGelfClient.cs#L54
To reproduce, I'm using latest version of Swagger UI:
Swashbuckle.AspNetCore in ASP.Net Core 2.1 (but I believe this can also be reproduced in ASP.Net Core 2.0

Could you transform this line with

messageJson[$"_{field.Key}"] = field.Value as string ?? field.Value?.ToString(); 

that did fix the issue for me

This might be a bit difficult for you to reproduce, please, do tell me if you need a sample

Thank you

Thomas

We're looking at using this Library for logging in our Asp.Net Core 2.1 web api, but we're getting an error in GrayLog.

WARN [Messages] Failed to index message: index=<graylog_7> id=<0ac6b401-1395-11e9-b13e-062322a0902c> error=<{"type":"mapper_parsing_exception","reason":"failed to parse [level]","caused_by":{"type":"number_format_exception","reason":"For input string: \"Informational\""}}>

Our default JSON settings use the StringEnumConverter which is causing the SysLogSeverity to not be serialized to an integer.

I believe the fix for this is to write a custom JsonConverter for the SysLogSeverity enum, but wanted to get your input on this.

Would it be possible to include 'messageTemplate' in log messages?
I.e.:

logger.LogError("This is a TEST message. Result is {result}", result);

would produce:

message_template | This is a TEST message from. Result is {result}
message | This is a TEST message from. Result is bar
result | bar

It seems the Logstash Gelf input plugin only handles compressed Gelf which this library doesn't support (as far as I can tell). The readme, however, mentions that it can be used for sending logs to Logstash. Could the readme describe which input plugin to use with Logstash?

I've made a pull request #2 to begin fixing it, but more improvements are needed.

Is it possible to get logger to send errors inside public static void Main(string[] args)?

According to https://messagetemplates.org/

Prefixing a property name with the @ structure capturing operator is a hint that the the structure of the corresponding argument should be preserved, for example, by serialization

What do you think about adding support for this?

Hi,
I use this package like this:

// config graylog
public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
{
    var gelfLoggerOptions = new GelfLoggerOptions()
    {
        Host = ConfigurationManager.AppSettings["grayLogHost"],
        Port = Convert.ToInt32(ConfigurationManager.AppSettings["grayLogPort"]),
        LogSource = ConfigurationManager.AppSettings["grayLogSource"],
        Protocol = GelfProtocol.Udp
    };
    loggerFactory.AddGelf(gelfLoggerOptions);

    app.UseMvc();
}

// write request info into graylog
public class GrayLogFilter : IActionFilter
{
    private readonly ILogger _logger;
    private const string REQUEST_START_TIME = "REQUEST_START_TIME";
    public GrayLogFilter(ILoggerFactory loggerFactory)
    {
        _logger = loggerFactory.CreateLogger("grayLog");
    }

    public void OnActionExecuted(ActionExecutedContext context)
    {
        try
        {
            var request = context.HttpContext.Request;
            var method = request.Method;
            var url = request.GetEncodedUrl();
            var statusCode = context.HttpContext.Response.StatusCode;
            var requestData = context.ActionDescriptor.Parameters;

            var requestStartTime = (DateTime)context.HttpContext.Items[REQUEST_START_TIME];
            var elapsed = (DateTime.Now - requestStartTime).Milliseconds;

            _logger.LogInformation($"{method} {url} {statusCode}", new { Request = requestData, Elapsed = elapsed });
        }
        catch
        {
            // ignore
        }
    }

    public void OnActionExecuting(ActionExecutingContext context)
    {
        context.HttpContext.Items[REQUEST_START_TIME] = DateTime.Now;
    }

}

But, the info write into graylog like this:

Where's the new { Request = requestData, Elapsed = elapsed }?

I try solve this problem in this way:

// GrayLogEntity is my custom type
var grayLogEntity = new GrayLogEntity()
{
    RequestHeader = $"{method} {url} {statusCode}",
    RequestData = requestData,
    Elapsed = elapsed
};

_logger.Log(LogLevel.Information, eventId: default, state: grayLogEntity, exception: null, formatter:
                    (entity, ex) => { return entity.RequestHeader; });

But,it annot write grayLogEntity into Graylog.

Finally,i try this way:

var gelfLoggerOptions = new GelfLoggerOptions()
{
    Host = ConfigurationManager.AppSettings["grayLogHost"],
    Port = Convert.ToInt32(ConfigurationManager.AppSettings["grayLogPort"]),
    LogSource = ConfigurationManager.AppSettings["grayLogSource"],
    Protocol = GelfProtocol.Udp
};
var messageProcesser = new GelfMessageProcessor(new UdpGelfClient(gelfLoggerOptions));
var message = new GelfMessage()
{
    Level = SyslogSeverity.Informational,
    Host = request.Host.Value,
    Timestamp = Math.Round((double)DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() / 1000.0, 2),
    ShortMessage = $"{method} {url} {statusCode}",
    Logger = "graylog",
    AdditionalFields = new Dictionary<string, object>()
    {
        ["Elapsed"] = elapsed,
        ["RequestData"] = requestData
    }
};
messageProcesser.SendMessage(message);

But,nothing write into graylog,:joy:

Please help me,thanks!

I followed the documentation, but I couldn't make it work. It would be nice if this lib had some example project to build/customize from it.

Could you create a sample for Asp Net Core 6, please. I've managed to get the library working using values in Program.cs but I couldn't get GELF to read the settings in appsettings.json

In AdditionalFields, it would be nice to support Func so that we can calculate fields from other values:

Example:

"HumanLevel", (m) => switch(m.Level){
case 1:
return "Error";
/***/
}

Here is the code to change in GelfMessageExtensions to allow that:

                if (field.Value is Func<GelfMessage, object> specialFunc)
                {
                    messageJson[$"_{field.Key}"] = specialFunc(message)?.ToString();
                }
                else if (IsNumeric(field.Value))

Tell me what you think :)

Note that the reason why I'm asking that is because my previous GELF appender was creating a "Level" field in plain text (Info, Debug, ...) that was very nice to filter on in graylog.
By switching to .Net Core & MS.Extension.Logging, we lost that capability, which forced us to fork your project and customize it.

The last "customized bit" of our fork is this little field.

I suspect there is a way to do that directly in GrayLog, but I could not find it and I'm not sure I'll be able/allowed to change that on our GrayLog instance

I would like to use this library for my ASP.Net 4.5 project, is there any source/guide how to implement this for my project?

It would be nice to add an option to enable/disable Structured/Semantic Logging.

For example, so that the parameters are not created if I use _logger.LogInformation("Order {order_id} took {order_time} seconds to process", orderId, orderTime);.

It would be nice to allow reload of config while app is running. It allows quick adjustments on production while debugging.

Probably ConsoleLogger approach could be an inspiration:

• https://github.com/dotnet/runtime/blob/57bfe474518ab5b7cfe6bf7424a79ce3af9d6657/src/libraries/Microsoft.Extensions.Logging.Console/src/ConsoleLoggerProvider.cs
• https://github.com/dotnet/runtime/blob/57bfe474518ab5b7cfe6bf7424a79ce3af9d6657/src/libraries/Microsoft.Extensions.Logging.Console/src/ConsoleLoggerExtensions.cs#L28

On less than optimal networks, UDP isn't the best protocol, as it's a fire-and-forget setup.
Http(s) is very poorly supported by Graylog and is very slow in processing input.

TCP on the other hand has proven to be stable and more reliable. It comes with some small performance considerations, but in some operations, reliability would outweigh the performance.

Currently, the timestamp is created as follow:

var totalMiliseconds = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds();
var totalSeconds = totalMiliseconds / 1000d;
return Math.Round(totalSeconds, 2);

The precision is too low and should be increased otherwise messages will get the same timestamp, and in our case, kibana will display them in a random order
(our setup is .Net => Graylog => ElsticSearch => Kibana)

I tested with Math.Round(totalSeconds, 3); and it solved my issue (messages always get a different timestamp

Thank you

Maybe a dumb question, but I'm not super familiar with logging in asp.net core and I can't see where to add fields like current user, client ip etc that would be available on the HttpContext but not available as part of this call .ConfigureLogging((context, builder) => builder.AddGelf(...?

I've seen issue #22 but it's still unclear to me is your extension is able to "feed" logstash "by default" or if it requires a plugin over logstash (like this one https://www.elastic.co/guide/en/logstash/current/plugins-inputs-gelf.html)

I believe it's the second

Could you confirm?
Could anyone who used this scenario help ? @loxybjorn ?

Thank you!

https://docs.graylog.org/en/3.1/pages/gelf.html#gelf-payload-specification

In gelf, the short_message field is considered to be the message title and the (optional) full_message to contain the full message.

This is helpfull when viewing multiple messages in graylog since by default the short_message is used as title.
One approach would be to check for the existing of a short_message field in additionalFields and if found set that as the GelfMessage.ShortMessage

Microsoft is pushing the use of the new native System.Text.Json methods in 3.0 as a replacement to Newtonsoft.Json.

As users migrate to the internal methods, relying on the Newtonsoft.JSON dependency will be redundant.

Is there a plan to convert these methods to the native methods for the 3.0 builds in the future?

Currently there is no way to determine if all queued log entries have been sent, and as a consequence when a logs may potentially be lost.

This can present a problem in scenarios where an web app has entered an unrecoverable state and can only perform a controlled crash. Or in the case where I discovered this issue, in a short lived console app.

There are 2 ways I can see this limitation being overcome.

1. Add a new method to check if queue is empty/wait for queue to empty
2. Add support for dumping logs somewhere, such that they can be reprocessed when the app is running again

Main difference across both would be how quick to close when there is a backlog.

We're getting the following error when trying to use this library with a more recent version of Microsoft.Extensions.Logging

Could not load type 'Microsoft.Extensions.Logging.Abstractions.Internal.NullScope' from assembly 'Microsoft.Extensions.Logging.Abstractions, Version=3.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60

I believe the issue is to do with this code https://github.com/mattwcole/gelf-extensions-logging/blob/dev/src/Gelf.Extensions.Logging/GelfLogger.cs#L7 and the fact that the NullScope has been moved and/or made internal.

Hello
While scanning our solution, we found a security issue with the current Gelf.Extensions.Logging Package which still has a dependency on System.Text.Regularexpressions.4.3.0 which has a security issue: Please fix the dependency as soon as possible, thanks

Issue: The log messages from the gelf client has double quotes around the timestamp value, which deserializes into a string and causes the graylog server to send error messages that looks like the following: 2017-11-03T19:43:16.768Z WARN [GelfCodec] GELF message <3cb9eaf2-c0cf-11e7-bfec-42010af0000e> (received from <10.180.2.6:41948>) has invalid "timestamp": 1509738196.720 (type: STRING). Below is a sample gelf message from the gelf client (Gelf.Extensions.Logging v1.1.0) extracted from a UDP packet capture using wireshark:

{
  "version": "1.1",
  "host": "blueprint2-backend",
  "short_message": "Pepe1S3mFSiVyM1V",
  "timestamp": "1510345173.212",
  "level": 2,
  "_logger": "app.api.controllers.TestController",
  "_machine_name": "500d5cb21c8f"
}

Dev Notes:

The graylog server (v2.3.2) throws an IllegalArgumentException when isNumber asserts false. See code:

        final JsonNode timestampNode = jsonNode.path("timestamp");
        if (timestampNode.isValueNode() && !timestampNode.isNumber()) {
            throw new IllegalArgumentException(prefix + "has invalid \"timestamp\": " + timestampNode.asText());
        }

https://github.com/Graylog2/graylog2-server/blob/4ffda524aac8054712b811f8e78c232be8b24a0d/graylog2-server/src/main/java/org/graylog2/inputs/codecs/GelfCodec.java#L262

Recommendation: Consider changing the DTO's Timestamp property from string to double in order to JSON serialize the timestamp value as a numeric data in compliance to the gelf spec (http://docs.graylog.org/en/2.2/pages/gelf.html#gelf-payload-specification).

I have tested the Gelf.Extensions.Logging.Samples.NetCore2 application locally with my Graylog server

In appsettings.json, i edited the Host and LogSource values
"Graylog": {
"Host": "host.name.net",
"Port": 12201,
"LogSource": "Graylog_Test"
}

When I run the project, I do not see any POST to my Graylog server (in fiddler) and there is no evidence the application is sending the POST.

Using curl, this works (I have removed my actual host name)
curl -X POST
http://host.name.net:12201/gelf
-H 'Cache-Control: no-cache'
-H 'Content-Type: application/json'
-d '{"short_message":"Hello there", "host":"example.org", "facility":"test", "_foo":"bar"}'

Do I need to my values in appsettings.json to be different?
"Graylog": {
"Host": "host.name.net/gelf",
"Port": 12201,
"LogSource": "Graylog_Test"
}

Hi @mattwcole ,

On HttpGelfClient shouldn't we use a static httpClient? To avoid opening and closing connections, even if it's disposed takes a while to close the connection

See:

https://aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/

Thanks!