It doesn't seam to be possible, or am I wrong ?

I'd like to log richer error events when something goes wrong when we're emitting points. Currently it looks like the only way to get notified of an error is via setting CollectorLog.Out to your own textwriter and logging those lines. There is no real way to associate an error with a point or instance of influxdb. This isn't ideal in complex applications which may have multiple collectors instantiated, etc. Additionally it'd be nice to have the Exception object, so that when we log via Serilog we can grab any further details off of the exception.

Do you have any thoughts on how we might improve the error reporting facilities? It seems like adding an Error event onto the MetricsCollector might be an option, or altering Write to return a Task. Not sure on what might be the best approach while maintaining backwards compatibility etc.

I haven't given this too much thought yet, hoping @nblumhardt might have some good ideas.

I am trying to use this library with projects running with .Net 4.5.2. I have made some very basic applications (a console application and an MVC website) inspired from the readme but none has been able to send the data to my server neither with the collector, nor with the raw API.

From what I have been able to find, it looks like the call to PostAsync in LineProtocolClient never ends or that it is not awaited successfully. I don't have any timeout or exception.

If I run the sample provided in the repository directly, it works fine.
If I copy the code from LineProtocol in my test application, it works fine as well.

I'm considering to contribute filtering capability backed into the configuration flow.
The idea behind it is to enable better control over the metrics you sending over the wire at runtime.

You may want limited amount of metrics that will give general health indication,
but when you got indication that something may went wrong,
you may want to illuminate specific spot by open other metrics.

In terms of Api I'm considering something similar to Serilog's filtering.
My suggestion is:

Metrics.Collector = new CollectorConfiguration()
.Filter.Tags.Match(tags => tags.ContainsKey("host") && tags["host"] == "BnayaMac")
.Tags.Exists.All(() => new[] { "Dev", "Staging" })
.Tags.Exists.Any(() => new[] { "OperationA", "OperationB" })
.Fields.Match(fileds => fileds.ContainsKey("cpu") && fileds["cpu"] > 80)

What do you think about the Api?

This issue arises when re-creating the LineProtocolClient a lot.

ERROR [24-07-2019 15:10:21]: System.Net.Http.HttpRequestException: Address already in use ---> System.Net.Sockets.SocketException: Address already in use at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken) at System.Threading.Tasks.ValueTask1.get_Result()
at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask1.get_Result() at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask1 creationTask)
at System.Threading.Tasks.ValueTask1.get_Result() at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at InfluxDB.LineProtocol.Client.LineProtocolClient.SendAsync(String payload, Precision precision, CancellationToken cancellationToken)
at EnergyHost.Services.Services.InfluxService.Write(String db, String measurement, IReadOnlyDictionary2 data, IReadOnlyDictionary2 tags, Nullable1 utcTimeStamp) in /app/EnergyHost.Services/Services/InfluxService.cs:line 82 ERROR [24-07-2019 15:10:21]: System.Net.Http.HttpRequestException: Address already in use ---> System.Net.Sockets.SocketException: Address already in use

It appears related to this: https://github.com/dotnet/corefx/issues/37044.

A solution may be this: https://docs.microsoft.com/en-us/dotnet/standard/microservices-architecture/implement-resilient-applications/use-httpclientfactory-to-implement-resilient-http-requests - although at this time it does not appear to be usable in netstandard2.0 without the addition of dependency injection in to the project.

It's hard to share a single LineProtocolClient across a lot of requests as the constructor takes the database name. The developer could create a little factory to handle this, which I plan to do - but it would be nice if the underlying usage of HttpClient handled this.

I'm not the greatest expert in concurrent code but can we somehow get rid of the lock by using Interlocked?

If influxdb server only allows HTTPS with self signed certificate, an SSL connection exception is silently thrown (catch it in CollectorLog.RegisterErrorHandler).

An option to bypass certificate validation could be useful in these cases.

Hi!

I see that in the official c# project is implemnted the write command to influxdb, but not the query command like the other implementation like https://github.com/ziyasal/InfluxDB.Net
Is in your roadmap?

Regards

Franco

I'm trying to use the LineProtocolClient to read data from a queue and upload to an https enabled influxdb. The following lines describe how we use the LineProtocol interface.

_client = new LineProtocolClient(new Uri("https://10.25.192.6:8086"), database: "datatest", username: "user", password: "password");

var influxResult = await _client.WriteAsync(payload);

The second line here fails when the influxdb has https enabled, but works fine otherwise.

We are currently using a self-signed certificate. Is the self-signed certificate the issue, or am I missing something? Any help is much appreciated.

The http and udp clients cannot be disposed.

I have several metrics setup to increment, measure and time. A using statement has been used to dispose of a metric timer. On calling Metrics.Close, the following Unhandled Exception is logged:

Unhandled exception: System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at InfluxDB.Collector.Platform.PortableTimer.d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<>c.b__6_1(Object state)
at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

Calling the following:

Metrics.Specialize().CreateCollector()

Will result in an ArgumentException with message:
"Parent may not be specified here"

As far as I can the assertion on parent being null in PipelinedCollectorEmitConfiguration.CreateEmitter makes Specialize unusable as it would always be raised.

The current API requires allocating lots of memory to write points. Which is not great for performance if you are writing lots of points.
Each point needs allocate a class. These point classes store values and tags in dictionaries which means a lot more allocations as well as having to box and struct values as the values dictionary stores objects.

I propose we expose the functionality in LineProtocolSyntax e.g. EscapeName , FormatTimespan In a class that wrapped the TextWriter that is passed to the Format method of LineProtocolPoint.

This would allow clients the option store the point in a more efficient manor if they found the need to. But still fairly easily write correct line protocol.

Example

struct MyPoint
{
    public MyPoint(string host, float value, DateTime timestanp)
    {
        Host = host;
        Value = value;
        Timestanp = timestanp;
    }

    public string Host { get; }
    public float Value { get; }
    public DateTime Timestanp { get; }

    public void Format(LineProtocolWriter writer)
    {
        writer.Measument("my_point");
        writer.Tag("host", Host);
        writer.Value("value", Value);
        writer.Timestamp(Timestanp);
    }
}

Here we have a custom struct that we can use to store the data that can be written out to line protocol when needed.

I think this could be done in a way that is mostly backwards compatible with the current api.
If we define an interface that has a Format(LineProtocolWriter writer) method which LineProtocolPoint and LineProtocolPayload would implement. Change the client to accept the interface instead of the LineProtocolPayload.
This would also allow us to implement LineProtocolWriter to write UTF8 directly would avoid converting the whole payload to a string and then UTF8 encoding it.

LineProtocolWriter would keep track of what has been written so if you tried to write a value before a measurement name it would error. This would allow it to know when to insert line breaks.

The Client should also have a generic method for WriteAsync that would define the type of points being written which would be constrained to the point interface. This would allow passing in an array of point structs without boxing.

I realise that there are already many libraries out there that enable querying data out of influx. But none of them I have found enabled me to easily query data out in a type safe way.
I have done some thinking and some hacking and think I have something that might be worth perusing. It's just a proposal at the moment. I have scaffolded the api to ensure it compiles and gives me the correct type safety.

I would appreciate it if you could give me some feedback on if you think this would be useful addition to the library, if I get it running, or if it is something you would not likely accept a pull request for.

Clinet Api

Query a single series

public class WaterTemperature
{
    public string location {get; set;}
    public double degrees {get; set;}
}

var db = new InfluxDb("NOAA_water_database");

var results = await db.Query<WaterTemperature>("SELECT degrees,location FROM h2o_temperature");

foreach (var (values, time) in resuts)
{
    Console.WriteLine($"{values.location} {values.degrees} {time}");
}

Each point is returned as a value tuple of values and time (as a DateTime).
We are using tuple destucturing in the foreach to print out the values and time.

Values are matched with columns based on propery names. Attribues could be used to customise the name matching.

Query multible series (GROUP BY)

public class WaterQuality{
    public class Fields{
        public double index {get; set;}
    }

    public class Tags{
        public string location {get; set;}
        public string randtag {get; set;}
    }
}

var results = await db.Query<WaterTemperature.Fields, WaterTemperature.Tags>("SELECT index FROM h2o_quality GROUP BY location,randtag");

foreach (var series in resuts)
{
    foreach (var (values, time) in series.Points)
    {
        Console.WriteLine($"{series.Tags.location} {series.Tags.randtag} {values.index} {time}");
    }
}

We are passing in Value and Tag types so we know this returns multible series.
Tag values are returned once for each series rather than for each point which is more efficient on the wire.

We can also flatten the results to make it more consise.

foreach (var (tags, values, time) in resuts.Flatten())
{
    Console.WriteLine($"{tags.location} {tags.randtag} {values.index} {time}");
}

Design

The basic design of the query api is to use seperate types that define the values and tags a query has.

This seams overly verbose but has benifits when building queries using a fluent api and still enabling full intellisense for the results. As we can track projections made to the values and tags types.

C# does not allow union types but by using an anonymous type in the select clause we can conbine both fields and tags into a single values type.

// Measurement type defintion used to build query projections.
public class WaterDepth : InfluxMeasurement<WaterDepth.Tags, WaterDepth.Fields>
{
    public WaterDepth() : base("h2o_feet")
    {
    }

    public class Tags
    {
        public string location { get; }
    }

    public class Fields
    {
        [InfluxKeyName("level description")]
        public string level_description { get; set; }

        public double water_level { get; set; }
    }
}

var query = WaterDepth.Select((fields, tags) => new { fields.level_description, tags.location });

foreach (var (values, time) in resuts)
{
    // Complie error as water_level not selected. 
    Console.WriteLine($"{values.location} {values.level_description} {values.water_level} {time}");
} 

The Select function takes an Expression so that we can parse the expression tree and produce the select clause.
In this case we are importing the InfluxAggregations.COUNT function this is just a place holder that accepts field value types, in this case double, and returns int.

using static InfluxAggregations;

var query = WaterDepth.Select(fields => new { count = COUNT(fields.water_level) }).GroupBy(tags => new { tags.location });

foreach (var (tags, values, time) in resuts.Flatten())
{
    Console.WriteLine($"{tags.location} {values.count} {time}");
}

Group by time clause is kept sperate from the tag selection. I haven't dsesigned the Where clause yet.

var query = WaterDepth.Select(fields => new { count = COUNT(fields.water_level) }).GroupBy(TimeSpan.FromMinutes(12), tags => new { tags.location });

I tried to separate the client api which accepts strings and result types to deserialise separate from the query builder api. I would likely build the client api first and then get the more complicated query api working.

The latest version needs to be published to NuGet.
Current version in NuGet is 6 months old and doesn't support time resolution.
Thank you.

Unable to connect to the remote server --->
System.Net.Sockets.SocketException:
A connection attempt failed because the
connected party did not properly respond after a period of time,
or established connection failed because connected host has failed to
respond xxx.xxx.xxx.xxx:8086.

This error occurs sometimes, unable to find any reason.
While connecting to influx at 8086, we are using HttpClient class with timeout of 30 min. At the influx db configuration we have specified, timeout to be max.

Any help shall be appreciated.

Thanks,
Manoj

With the repository move we need to update the projectUrl in the included packages. Should update package icon/description in the process.

Seems like an oversight that SendAsync supports a given precision (through a LineProtocolWriter) but WriteAsync assumes a precision of nanoseconds.

Related: #51

Expected: I should be able to use batching and not lose points.
Actual: If I use batching, then data is lost when sending lots of points in quick succession.
Version: Using V1.0.0 of InfluxDB.Collector and V1.0.0 of InfluxDb.LineProtocol from Nuget

See attached file for recreation. It includes the following code:

static void Main(string[] args)
{

    WriteLotsOfMetricsFast("UnbatchedPoints", useBatching: false); // Results in all 1000 points being written to InfluxDb
    WriteLotsOfMetricsFast("BatchedPoints", useBatching: true); // Results in very few points being written (about 3 on my machine);

    Console.WriteLine("Waiting for 5s to flush the batched stuff");
    System.Threading.Thread.Sleep(5000);

}


private static void WriteLotsOfMetricsFast(string measureName, bool useBatching)
{
    CollectorConfiguration config;
    if (useBatching)
    {
        config = new CollectorConfiguration().Batch.AtInterval(TimeSpan.FromSeconds(1));
    }
    else
    {
        config = new CollectorConfiguration();
    }
    config = config.WriteTo.InfluxDB("http://localhost:8086", "MyData");
    Metrics.Collector = config.CreateCollector();


    for (int i = 0; i < 1000; i++)
    {
        var fields = new Dictionary<string, object>()
        {
                 { "Iteration", i },
                 { "MyValue", 10}
         };

        Metrics.Write(measureName, fields, null);
    }
}

I've been trying the netstandard13 version of this lib, and was confused why Grafana wasn't able to find the stats that I'd been collecting. Once I created the db I was logging to, it worked as expected.

I would have expected to get an exception when logging to a database that does not exist (and a "create database if it doesn't exist" option on the MetricsCollector would be awesome, as well). If an exception isn't possible, it might at least be worth logging as a troubleshooting tip somewhere. 😄

Hi,

I ran into an issue with the MetricsCollector and a short lived application. By default we use a batch interval of 5 seconds. My application runs for less than that, so I was noticing that measurements were lost.

I addressed this by calling Influx.Metrics.Close(); at the end of the application to flush measurements.

What's the correct way of handling this kind of scenario? I'd like to take advantage of batches but it might be nice to be able to explicitly flush measurements without closing the db connection. It appears there is not Flush() facility.

we should link to the new csharp client lib in the readme: https://github.com/influxdata/influxdb-client-csharp

I would like to request a simple feature:
It should be possible to transform a point into a string, which follows the line protocol '
e.g. "cpu,dev=cpu0 temp=4,var1="bla",var2="miep" 1540900454326481400"

Our situation: We are communicating with Telegraf and use currently the Line Protocol. We generate the string ourselves including the handling of special characters. I hoped, that with this library we could save the additional work, as you guys have to generate an Inline String somewhere in the process eventually anyway.

There should be an option for splitting too large request sizes, e.g. by count.

We use this library in a heavily multithreaded environment, and noticed that in heavy loads, this library introduces very serious lock contention issues in the IntervalBatcher class, which locks around the internal queue when enqueuing or emitting.

This eventually leads to 100% CPU usage on our servers, each of them waiting for this Influx client to release its locks.

Hey sir, I am excited to see some work on InfluxDB coming from you. Are there plans or a current method of integrating this with Serilog.Metrics as a sink for that information?

Thanks for your work!

I am trying to use the batch feature to track page views in an ASP application. I am emitting new mesurements in Application_EndRequest and I get this error:

An asynchronous operation cannot be started at this time. Asynchronous operations may only be started within an asynchronous handler or module or during certain events in the Page lifecycle. If this exception occurred while executing a Page, ensure that the Page is marked <%@ Page Async="true" %>. This exception may also indicate an attempt to call an "async void" method, which is generally unsupported within ASP.NET request processing. Instead, the asynchronous method should return a Task, and the caller should await it.

Am I trying to do something that is impossible/unsupported? Is there another way of doing it? Do we need to update PortableTimer?

I see that Influx supports bind parameter to prevent SQL injections. Also, the java client supports SQL injection prevention. But I don't see this supported in the c-sharp client.
Is there any capability inbuilt in the c-sharp client to support the same? If not, are there any plans on doing the same?

I'm currently using https://github.com/agileharbor/telegraf-client to send metrics to Telegraf from .NET and while it works fine, it's unmaintained and obviously not officially supported. It would seem to me that adding Telegraf as a new output target to this package should be rather trivial. The only change to the public API would be the possibility to do .WriteTo.Telegraf("udp://localhost:8084").

.NET DateTime is accurate only to the nearest 100 nanoseconds and the last two digits of the line protocol timestamp, as produced by the .NET client, will always be zero.

It would be good to be able to write data to nanosecond precision, even if it means using long instead of DateTime.

Sometimes it's good to be able to send a timestamp with the measurement if it's coming off an offline queue.

I have just started working with the library passing points to a local instance. If I switch to an influx instance that requires auth, I simply don't find in the README file how to pass those details to a collector instance. Here's a sample of my code:

        Metrics.Collector = new CollectorConfiguration()
            .Tag.With("host", Environment.MachineName)
            .Batch.AtInterval(TimeSpan.FromSeconds(5))
            .WriteTo.InfluxDB(
                Environment.GetEnvironmentVariable("InfluxDBServer"),
                Environment.GetEnvironmentVariable("InfluxDBDatabase")
            )
            .CreateCollector();

Any suggestions on how to pass auth bearer and password?.

Many thanks,

We're using https://github.com/benaadams/Ben.BlockingDetector to detect blocking calls inside our application. We get the following warning:

Blocking method has been invoked and blocked, this can lead to threadpool starvation. " at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification) at InfluxDB.Collector.Pipeline.Emit.HttpLineProtocolEmitter.Emit(PointData[] points)

Accessing _client.WriteAsync(payload).Result; is not a good practice. Is this an issue and will it be addressed in the near future?

Most of the time, points are collected one by one by using one of the public methods of MetricsCollector. For every collected point however, an array has to be allocated because all the points go through the IPointEmitter.Emit(PointData[] points) method.

I propose to add a method virtual Emit(PointData point) to MetricsCollector that can be overriden by subclasses. The default implementation can simply delegate to the array accepting method.

Any chance a new version can be published to NuGet?

The migration to .NET standard 2.0 and UDP support are still under the 1.1.1-dev-10109 NuGet package. Also, the max batch size parameter #59 was merged but has not been released -- this would be useful.

We're lacking a shared GitHub account for authenticating builds (and a shared AppVeyor account for running them). Running them under my personal account's not really viable; I'll sort this out ASAP. Via #59.

The current code include migrating to .NET standard 2.0 and support for UDP channel.
But the NuGet version is way behind.
Can someone issue a new version?

I had an error where I was submitting my 1,000,0001th tag value and Influx was returning a very descriptive error about why my data was failing.

But my CollectorLog.RegisterErrorHandler((message, exception) was returning "BadRequest", nothing else. Had to man in the middle the request to figure out why it was failing.

Can you include the influxdb error into the message?

Hi everyone,

I'm encountering a problem in saving timestamps with correct milliseconds.
My code is this:

var point = new LineProtocolPoint(
    "Metrics",
    new Dictionary<string, object>
    {
        {nameof(metric.ReaderId), metric.ReaderId},
        {nameof(metric.BeaconId), metric.BeaconId},
        {nameof(metric.ReaderInfo), metric.ReaderInfo},
        {nameof(metric.BeaconInfo), metric.BeaconInfo},
        {nameof(metric.BatteryVoltage), metric.BatteryVoltage}
    },
    null,
    metric.Timestamp.UtcDateTime);

var payload = new LineProtocolPayload();
payload.Add(point);

var influxResult = _db.WriteAsync(payload).Result;
if (!influxResult.Success)
    throw new Exception(influxResult.ErrorMessage);

where metric.Timestamp.UtcDateTime is a DateTime with Ticks = 636265483200020000

After executing this code, in the db i got time = 1490951520001999872 but I expected time = 1490951520002000000

I'm using .NET Core in Visual Studio 2015 with this project.json

{
  "version": "1.0.0",

  "dependencies": {
    "Microsoft.Extensions.Configuration": "1.1.0",
    "InfluxDB.Collector": "1.0.0-beta-10050",
    "InfluxDB.LineProtocol": "1.0.0-beta-10050"
  },

  "frameworks": {
    "netcoreapp1.1": {
      "dependencies": {
        "Microsoft.NETCore.App": {
          "type": "platform",
          "version": "1.1.0"
        }
      },
      "imports": "dnxcore50"
    }
  }
}

When I try to insert oround 1M of points in packages every 10 points, sometimes (rather pretty offten) I got error "Unable to connect to the remote server" while server is running.

I noticed that in the dev branch, the full .NET Framework is no longer targeted. Is there any particular reason?

Is there a way to specialize a collector to a different database without passing the server information? I'd like for a library to write to the same server in a different database. I see the current server is not exposed in any way.

I'm looking for something like:

var newCollector = Metrics.Specialize()
    .WithTargetDatabase("foo")
    .CreateCollector();

Is this possible in any way?

This repository is now influxdata/influxdb-csharp. Adding this ticket as a pointer back to the original discussion: https://github.com/influxdata/influxdb-csharp-obsolete/issues/3. Apart from the change of organisation it's business as usual.

Please update your bookmarks :-) // cc @michael-wolfenden @HakanL

is there a way to use this client with the influxdb 2.0 alphas? i thought the v1 api is still there but it looks like something broke.

The solution is building against the RC1 version of the .NET Standard libraries and .NET Core CLI tooling. We need to get onto RC2 ASAP to iron out any issues before the new bits hit RTM on June 27th.

I'm having trouble specifying the timestamp of a point/record. I tried adding 'timestamp' as a tag but it just appears as a plain tag. Could give an example?

Edit: Found out I needed to use the Write method of Collector directly with .ToUTC on DateTime.

Metrics.Collector.Write(metric.Measurement, metric.Fields, metric.Tags, metric.EvenDateTime.ToUniversalTime());

    20.02.2017 13:53:29 IntervalBatcher.OnTick(): _stateLock // firs lock IntervalBatcher._stateLock
        20.02.2017 13:53:29 PortableTimer.Start(..): 00:00:01
            20.02.2017 13:53:29 PortableTimer.Start(..): _stateLock
            20.02.2017 13:53:29 InfluxdbTraceListener.WriteLine(..): // new log event is coming
                20.02.2017 13:53:29 MetricsCollector.Write(..)
                    20.02.2017 13:53:29 PipelinedMetricsCollector.Emit(..)
                        20.02.2017 13:53:29 IntervalBatcher.Emit(..): // deadlock IntervalBatcher._stateLock

If during IntervalBatcher.OnTick() a new event is coming to the log, then IntervalBatcher.Emit(..) is called and a deadlock occured.

If multiple Count() calls increment the same measurement in the sampling interval, these should be aggregated into single values before sending.

Hi,

One of my projects uses the LineProtocolClient. I am using Moq as a mocking framework and I am currently missing an interface so I can mock the injection of the LineProtocolClient.

Can I just clone the repository, add one and create a pull request? Or is there a reason that it doesn't exist yet?