Giter Club home page Giter Club logo

nlog.targets.http's Introduction

NLog.Targets.HTTP

Nuget (with prereleases)

2024.05.10 - With the introduction of NLog 5.0 WebServiceTarget this project will be heading towards becoming an archive.

NLog.Targets.HTTP is an HTTP POST target for NLog. When combined with JSON formatter it can be used to send events to an instance of Splunk and other HTTP based collectors.

This target is inherently asynchronous, with performance on par or better than with the AsyncWrapper. Remember to Flush and to give it enough time to complete.

Note that the async="true" attribute applied to <targets > will discard by default.

Getting started

Add the library as an extension to nlog:

<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >
  <extensions>
    <add assembly="NLog.Targets.Http" />
  </extensions>
  <targets>
    ...

Available Configuration Parameters

Listed below are available configuration parameters with their default values (where applicable)

<target name='target name' 
        type='HTTP' 
        URL='protocol://server:port/path'
        Method='POST'
        Authorization='phrase token' 
        BatchSize='1'
        MaxQueueSize='2147483647'
        IgnoreSslErrors='true'
        FlushBeforeShutdown='true'
        ContentType='application/json'
        Accept='application/json'
        DefaultConnectionLimit='2'
        Expect100Continue='false'
        UseNagleAlgorithm='true'
        ConnectTimeout='30000' 
        InMemoryCompression='true'
        ProxyUrl=''
        ProxyUser=''
        ProxyPassword=''
    >
    <!-- optional additional HTTP Headers -->
    <header name='key' value='value'/>
    <!-- additional headers -->
    <!-- layout element -->
</target>

URL

The URL (Layout element) to send messages to (mandatory).

Method

HTTP method to use (GET,POST).

Authorization

The Authorization Header value to pass.

BatchSize

Number of messages to be sent together in one call.

BatchAsJsonArray

If set to true, messages will be packaged as JSON Array instead of being separated with Environment.NewLine character. Default is false.

MaxQueueSize

Maximum number of messages awaiting to be send. Please note, that if this value is set too low, the logger might be blocking.

IgnoreSslErrors

Some SSL certificates might be invalid or not-trusted.

FlushBeforeShutdown

Force all messages to be delivered before shutting down. Note that by design .Net apps are limited to about 2 seconds to shutdown.
Make sure you leverage LogManager.Flush(TimeSpan.FromHours(24)) in most extreme scenarios.

ContentType

HTTP ContentType Header value. Default is application/json.

Accept

HTTP Accept Header value. Default is application/json.

DefaultConnectionLimit

How many connections might be used at the same time. Changes ServicePointManager.DefaultConnectionLimit, which might affect other parts of your system. Performance improvement was noticeable even with 16 connections, but your application might require more for other functionality. Use with caution.

Expect100Continue

See this article.

UseNagleAlgorithm

The Nagle algorithm is used to buffer small packets of data and transmit them as a single packet. This process, referred to as "nagling," is widely used because it reduces the number of packets transmitted and lowers the overhead per packet. The Nagle algorithm is fully described in IETF RFC 896.

ConnectTimeout

How long should the client wait to connect (default is 30 seconds).

InMemoryCompression

Reduces the amount of memory consumed at the expense of increased CPU usage. Significant performance improvement can be achieved by using default of false.

ProxyUrl

Designates a proxy server to use. Must include protocol (http|https) and port. It is a Layout element so can be dynamic.

ProxyUser

If proxy authentication is needed, you can specify it with a domain prefix, i.e. DOMAIN\USER.

ProxyPassword

Password to use for proxy authentication.

Additional Headers

Additional HTTP Headers can be specified by adding multiple <header name='..' value='..'/> elements. Elements with a blank name or value will not be included.

UnixTimeLayoutRenderer

The "Unix Time" renderer supports universalTime option (boolean), just like the date renderer does.

<attribute name='unixutc' layout='${unixtime:universalTime=true}' />

Sample SPLUNK Configuration

<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >
  <targets>
    <target name='splunk' 
            type='HTTP' 
            URL='https://localhost:8088/services/collector/event'
            Authorization='Splunk d758f3fa-740f-4bb7-96be-3da4128708bc' 
            BatchSize='100'>
      <layout type='JsonLayout'>
        <attribute name='sourcetype' layout='_json' />
        <attribute name='host' layout='${machinename}' />
        <attribute name='event' encode='false'>
          <layout type='JsonLayout'>
            <attribute name='level' layout='${level:upperCase=true}' />
            <attribute name='source' layout='${logger}' />
            <attribute name='thread' layout='${threadid}' />
            <attribute name='message' layout='${message}' />
            <attribute name='utc' layout='${date:universalTime=true:format=yyyy-MM-dd HH\:mm\:ss.fff}' />
          </layout>
        </attribute>
      </layout>
    </target>
  </targets>
  <rules>
    <logger name="*" minlevel="Debug" writeTo="splunk" />
  </rules>
</nlog>

Sample stats

On a Lenovo Xeon E3-1505M v6 powered laptop with 64GB of RAM and 3GB/s NVMe storage, this HTTP target was able to consistenlty accept about 125,000 messages per second, and 62,500 per second received and processed by local Dockerized Splunk Enterpise 8.2.4. Please note, that these stats depend heavily on the message size, batch size, and amount of bytes that can be submitted in a single POST message.

Running local Splunk

docker pull splunk/splunk:latest
docker run -d -p 8000:8000 -p 8088:8088 -e "SPLUNK_START_ARGS=--accept-license" -e "SPLUNK_PASSWORD=Pass@W0rd" --name splunk splunk/splunk:latest

After a few moments, depeneding on your systems capacity, login to Splunk at http://localhost:8000/ with admin and Pass@W0rd. The HttpEventCollector (HEC) will listen on port 8088 once created from Settings - Data Inputs menu option.

nlog.targets.http's People

Contributors

304notmodified avatar darekdan avatar lsfera avatar michelz avatar snakefoot avatar trympet avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

nlog.targets.http's Issues

[bug] Unknown exception occured Exception when use1.0.5 in netcore3.1

use 1.0.5 in netcore3.1, I found a error:

Warn Unknown exception occured Exception: System.ArgumentException: Object must be the same type as the enum. The type passed in was 'System.Int32'; the enum type was 'System.Net.HttpStatusCode'.
at System.Enum.CompareTo(Object target)
at NLog.Targets.Http.HTTP.SendFast(String message)

and then my http log was repeat to called many times!

Consumes processor resources even when idle.

Thank you for providing this extension to NLOG, the features and performance of which has been very useful to us.

We noticed one relatively minor performance problem with it. When idle, the background task has an infinite loop that polls a queue. On Windows, with a relatively high-spec i7 laptop this is no problem. No more than 0.5% of the CPU is ever consumed. On a Linux virtual machine, with many Docker containers, each container reports at least 30% CPU usage. We can of course discuss resource provisioning in Docker, and I am no expert in it, but I was able to fix the usage problem directly in your code.

I will attach a pull request to this issue in case you should find it useful.

Thank you for taking a look.

Flushing and Batching Create Invalid JSON

When using this library along with a JSONLayout batching and flushing create invalid json that causes a 400 request from the endpoint, because the json does not get wrapped in an array.

Code sample:
<target name='test' type='HTTP' URL='' Method='POST' BatchSize='3' FlushBeforeShutdown='true'> <layout type='JsonLayout'> <attribute name='sourcetype' layout='_json' /> <attribute name='host' layout='${machinename}' /> <attribute name='event' encode='false'> <layout type='JsonLayout'> <attribute name='level' layout='${level:upperCase=true}' /> <attribute name='source' layout='${logger}' /> <attribute name='thread' layout='${threadid}' /> <attribute name='message' layout='${message}' /> <attribute name='utc' layout='${date:universalTime=true:format=yyyy-MM-dd HH\:mm\:ss.fff}' /> </layout> </target>

Post request output (causes 400 bad request):
{"sourcetype":"_json", "host":"","event":{"level":"INFO", "source":"","thread:"1","message":"test","utc":"2021-10-05 15:01:01.375"}}
{"sourcetype":"_json", "host":"","event":{"level":"INFO", "source":"","thread:"1","message":"test","utc":"2021-10-05 15:01:01.375"}}
{"sourcetype":"_json", "host":"","event":{"level":"INFO", "source":"","thread:"1","message":"test","utc":"2021-10-05 15:01:01.375"}}

No async await?

Great plugin, thanks for making it!

Quick question: what is the reason for not using async/await for http?

Any chance of a .NET Standard 2.0 build?

We'd like to use this package within a .NET Standard 2.0 library that's used across .NET Framework v4.7.2 and .NET Core based on the age of the projects.
We're seeing some warnings as the build is trying to use the .NET Framework version as there's no .NET Standard v2.0 version.

XXXXX.csproj: [NU1701] Package 'NLog.Targets.Http 1.0.7' was restored using '.NETFramework,Version=v4.6.1, .NETFramework,Version=v4.6.2, .NETFramework,Version=v4.7, .NETFramework,Version=v4.7.1, .NETFramework,Version=v4.7.2, .NETFramework,Version=v4.8' instead of the project target framework '.NETStandard,Version=v2.0'. This package may not be fully compatible with your project.

label:bug Unknown exception occured Exception when use1.0.5 in netcore3.1

use 1.0.5 in netcore3.1, I found a error:

Warn Unknown exception occured Exception: System.ArgumentException: Object must be the same type as the enum. The type passed in was 'System.Int32'; the enum type was 'System.Net.HttpStatusCode'.
at System.Enum.CompareTo(Object target)
at NLog.Targets.Http.HTTP.SendFast(String message)

and then my http log was repeat to called many times!

ContentType is null if not explicitly specified

The documentation says that application/json is the default value for ContentType:

HTTP ContentType Header value. Default is `application/json`.

But this does not work as expected. If i omit the ContentType setting in my nlog configuration, the HTTP requests are sent with ContentType = null.

In the code, the _contentType field is initialized to the default value, but not _contentTypeHeader:

private string _contentType = "application/json";
private MediaTypeHeaderValue _contentTypeHeader;

The latter is only set in the ContentType setter, which is never called if the setting is not specified in the configuration:

public string ContentType
{
get => _contentType;
set
{
_contentType = value;
_contentTypeHeader = new MediaTypeHeaderValue(string.IsNullOrWhiteSpace(value) ? "text/plain" : value)
{ CharSet = Encoding.UTF8.WebName };
}
}

For the actual request, _contentTypeHeader is used, which is null in the default case:

request.Content.Headers.ContentType = _contentTypeHeader;

Error Instantiating Extension Using nlog.config

I am trying to use this extension with the UiPath application to send log events to Splunk.

I've copied the extsension assembly to the UiPath Studio folder, where the NLog.dll resides, and updated the UiPath nlog.config file to reference the extension.

At runtime, I see the following error logged:

Exception: NLog.NLogConfigurationException: Failed to create Target of type: HTTP ---> System.ArgumentException: Target symbol-name is unknown: 'HTTP'
at NLog.Config.Factory2.CreateInstance(String itemName) at NLog.Config.LoggingConfigurationParser.FactoryCreateInstance[T](String classType, INamedItemFactory2 factory)

Any insights into what the problem might be would be greatly appreciated.

Here is my config file, with the Splunk token and URL removed:

<?xml version="1.0" encoding="utf-8" ?> <nlog autoReload="true" xmlns="http://www.nlog-project.org/schemas/NLog.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" throwExceptions="true" internalLogFile="C:\Temp\nlog-internal.log" internalLogLevel="Trace" internalLogToConsole="true"> <extensions> <add assembly="NLog.Targets.Http"/> </extensions> <variable name="WorkflowLoggingDirectory" value="${specialfolder:folder=LocalApplicationData}/UiPath/Logs" /> <targets> <target xsi:type="File" name="WorkflowLogFiles" fileName="${WorkflowLoggingDirectory}/${shortdate}_Execution.log" layout="${time} ${level} ${message}" keepFileOpen="true" openFileCacheTimeout="5" concurrentWrites="true" encoding="utf-8" writeBom="true" /> <target name='splunk' xsi:type='HTTP' Url='splunkurl' Method='POST' Authorization='Splunk token' IgnoreSslErrors='true' FlushBeforeShutdown='true' ContentType='application/json' Accept='application/json' ConnectTimeout='30000' BatchSize='1'> <layout type='JsonLayout'> <attribute name='sourcetype' layout='_json' /> <attribute name='host' layout='${machinename}' /> <attribute name='event' encode='false'> <layout type='JsonLayout'> <attribute name='level' layout='${level:upperCase=true}' /> <attribute name='source' layout='${logger}' /> <attribute name='thread' layout='${threadid}' /> <attribute name='message' layout='${message}' /> <attribute name='utc' layout='${date:universalTime=true:format=yyyy-MM-dd HH\:mm\:ss.fff}' /> </layout> </attribute> </layout> </target> </targets> <rules> <logger name="WorkflowLogging" writeTo="WorkflowLogFiles" final="true" /> <logger name="*" minlevel="Debug" writeTo="splunk" /> </rules> </nlog>

NLog WebServiceTarget will probaby not replace NLog.Targets.HTTP

This is probably not true, since WebServiceTarget is 20 years old (and uses legacy API and doesn't support batching):

2024.05.10 - With the introduction of NLog 5.0 [WebServiceTarget](https://nlog-project.org/documentation/v5.0.0/html/T_NLog_Targets_WebServiceTarget.htm)
this project will be heading towards becoming an archive.

But yes I have been thinking about creating a dedicated HttpClient-target, but also think this is a great project, so no need to compete with it.

Not sending anything since 1.0.11

We have installed later versions of the target, and it does not seem to send anything anymore. When we switch back to 1.0.10 things work fine.

everything from 11 to 14 does not work for us currently. There seem to be no exceptions/errors in the internal log

Double log entries when using HTTP target

Thank you for providing a simple way to send log entries in batches to an HTTP endpoint, it's very simple to setup and get started.

One thing I noticed was that I seem to be getting double log entries sent to my data store. Everything logs successfully but I get the same logging statements twice.

I have the NLog configuration setup as a JSON file to point to an HTTP REST service. The REST service takes in the batch of log entries and is supposed to drop them into Couchbase. I am generating tasks for each log entry and waiting for all of them to complete, then returning a 200 OK status code.

At the end of the program execution, I am calling LogManager.Flush(TimeSpan.FromSeconds(60)) in a finally block to flush all log messages.

Port Exhaustion on .NET 5

Using this target on .NET 5 appears to eat up ports. We are hitting the 128 port limit out on Azure App Service when we have this target turned on. It's not immediately obvious to me how we can mitigate this, so wanted to start the conversation.

Variables and GDC not supported?

I'm trying to set the URL and Authorization parameters dynamically but they always end up being set as the placeholder ${var:MyVariable} or ${gdc:item=MyGdcItem} instead of the real value. Are they not supported?

Header value doesn't update dynamically.

The header value doesn't update. Layout value gets updated.
This would be useful in case of supplying metadata via header. The http target has BatchAsJsonArray="true" wrapped in BufferingWrapper.
custom layout render.
LayoutRenderer.Register("unixutcms", (logEvent) => DateTimeOffset.UtcNow.ToUnixTimeMilliseconds());

<header name="ClientUtcNowMs" value="${unixutcms}"/>
<layout type="JsonLayout" >
<attribute name="ts" layout="${unixutcms}" />
...

Relatively high CPU usage when HTTP target is enabled

Hello,

Problem

I've just setup a simple configuration using HTTP target, and I've noticed a relatively high CPU usage. A constant CPU usage of about 3% - 5% was observed.

Current Configuration

Nlog.config

<?xml version="1.0" encoding="utf-8"?>
<nlog
    xmlns="https://www.nlog-project.org/schemas/NLog.xsd"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <extensions>
        <add assembly="NLog.Targets.Http" />
    </extensions>
    <target name='http_target'
            type='HTTP'
            URL='https://url'
            Authorization='secret'
            BatchSize="50">
        <layout type='JsonLayout'>
            <attribute name='host' layout='${machinename}' />
            <attribute name='level' layout='${level:upperCase=true}' />
            <attribute name='message' layout='${message}' />
            <attribute name='timestamp' layout='${longdate:universalTime=true}' />
            <attribute name='meta_data' encode='false'>
                <layout type='JsonLayout'>
                    <attribute name='thread' layout='${threadid}' />
                    <attribute name='callsite' layout='${callsite:className=true:fileName=true:includeSourcePath=false:methodName=true}' />
                </layout>
            </attribute>
        </layout>
    </target>
    <rules>
        <logger name="*" minlevel="Error" writeTo="http_target" />
    </rules>
</nlog>

Investigation

It seems that changing timeToSleepBetweenBatches attribute didn't have any effect, I've tried to set it to value 0 or 1 and the problem remains.

After reading the source code, I've noticed a potential cause of this problem. For this line Http.cs#L254, it will wait for 1ms each time in the loop.
Does this mean the logger will be fired at least every 1ms ?

Environment

  • dotnet 5.0.101
  • NLog 4.7.9
  • NLog.Targets.Http 1.0.5

Perhaps the problem happens because of a mis-configuration of the nlog.config, any help would be appreciated.

Thanks,
Charles Y

NLog.Targets.HTTP.dll does not have a strong name

I'm try to install the dll on GAC, because we have many applications that will use it, but get the error:
"Failure adding assembly to the cache: Attempt to install an assembly without a strong name"

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.