wintoncode / winton.extensions.configuration.consul Goto Github PK

We are using consul configuration on services on a kubernetes cluster. We have noticed that when we add a new key-value, the consul configuration extension does not pick up the changes. The configuration remains unaware of the new key.

This is how we use the consul configuration extension:

public static IWebHostBuilder CreateWebHostBuilder(string[] args, CancellationTokenSource cancellationTokenSource) =>
            Microsoft.AspNetCore.WebHost.CreateDefaultBuilder(args)
                .UseApplicationInsights()
                .ConfigureAppConfiguration((builderContext, config) =>
                {
                    var env = builderContext.HostingEnvironment;

                    config
                        .AddJsonFile("appsettings.json", false, true)
                        .AddJsonFile($"appsettings.{env.EnvironmentName}.json", true, true)
                        .AddEnvironmentVariables();

                    var consulConfig = config.Build().GetSection("consulConfig").Get<ConsulConfig>();

                    if (consulConfig.Enabled)
                    {
                        config.AddConsul($"{env.EnvironmentName}/demo", cancellationTokenSource.Token,
                            o =>
                            {
                                o.ConsulConfigurationOptions = a => { a.Address = new Uri(consulConfig.Address); };
                                o.Optional = env.IsDevelopment();
                                o.ReloadOnChange = true;
                            });
                    }
                })
                .UseStartup<Startup>();
    }

Then we inject the configuration using IOptionsMonitor<> and DI.
Is this by design? Any pointers on how we could solve this without having to restart our pods so that they pick up the new values?

Let me start by giving some context. I've been developing an API gateway using Ocelot which has a buit-in mechanism for reading its configuration from Consul. Since that project seems to be abandoned we are evaluating switching over to YARP which doesn't have Consul configuration built-in. But it does read config from ASP.NET Core's configuration system out-of-the-box so I found this library which plugs into that so that we can switch over to YARP and keep our configuration in Consul as we have currently with Ocelot.

I did some testing of that setup yesterday and I ran into issue microsoft/reverse-proxy#410. My initial thought was that it was a YARP issue, but after trying the same setup using just appsettings.json I don't get the same issue which leads me to believe the issue might be in this library.

I've setup Consul configuration using this piece of code in my Program.cs:

public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
            .ConfigureAppConfiguration((context, builder) =>
            {
                // Only load Consul configuration for development
                if (!context.HostingEnvironment.IsDevelopment())
                {
                    // Add Consul configuration
                    builder.AddConsul("my/apigateway", options => { options.ReloadOnChange = true; });
                }
            });

Then in Consul I created the my/apigateway folders and in there I created a ReverseProxy folder in which I put a Clusters key with the following content:

{
  "test": {
    "Destinations": {
      "test/destination1": {
        "Address": "https://localhost:5003/api/"
      }
    }
  }
}

This is consistent with the documented configuration examples for YARP. However, this leads to an exception being thrown by YARP when using a route that uses the test cluster stating there are is no configuration for the destination. As soon as I remove the slash in test/destination1 everything is working as expected.

I'm not here to shift blame to this library or anything, I guess I'm just wondering if there is a known issue with having a slash in the key which makes the configuration incompatible with this library.

Hi @Choc13
I have a very simple question where I am stuck so wanted to see if you can help me out.

I have a node (with data) in Consul like below:

Node: data/stage/process
Data: 
{
    "key1": true,
    "key2": "value2",
    "key3": {
        "car1": "Ford",
        "car2": "BMW",
        "car3": "Fiat"
    }
}

As you can see above, my key3 is a json object instead of a single string or boolean value.

When I use the below code:

    var builder = new ConfigurationBuilder()
        .SetBasePath(env.ContentRootPath)
        .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
        .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true)
        .AddConsul(
            $"data/stage/process",
            options =>
            {
                options.ConsulConfigurationOptions =
                    cco => { cco.Address = new Uri("some-url"); };
                options.Optional = true;
                options.PollWaitTime = TimeSpan.FromSeconds(5);
                options.ReloadOnChange = true;
                options.OnWatchException = ctx => TimeSpan.FromSeconds(ctx.ConsecutiveFailureCount);
            })
        .AddEnvironmentVariables();
    Configuration = builder.Build();

And I try to extract each of those keys value like this:

string key1 = configuration["key1"];
string key2 = configuration["key2"];
string key3 = configuration["key3"];

The value of key3 is always null but key1 and key2 works fine. Any idea why? And how can I get the value of key3 which is a json object?

Thanks for this good piece of code.

Please consider renaming the package to something more generic as this package is not dependent on AspNetCore.

Choc13.Extensions.Configuration.Consul would be more suitable i would say, what do you think?

Hello, we are using your library in our product to configure microservices. We'are using appsettings.json per microservice, but I want to move general settings in a common file (with settings inheritence as is in the .net core). So the question is can I do this thing by

.AddConsul(key1, .....)
.AddConsul(key2, .....).

I am trying to implement a version for K/V retrieval for NET Framework 4.7.1 using KeyValueConfigBuilder (Microsoft.Configuration.ConfigurationBuilders) in order to migrate legacy .NET Frameworks to Consul.

Why is ConsulConfigurationClient an internal class?

Hello,

I really like the library but the latest version is basically not usable for projects targeting .NET Framework v4.5.2. How can I adopt it to use v4.5.2? It's not an option for me to upgrade the target right now.

Thank you

consuldotnet is now archived and doesn't seem to have been updated recently. Given it is a wrapper around Consul's REST API it should be possible to remove our dependency on it and call the REST API directly. Also, we only need to query a single endpoint in this lib, so the implementation using something like Flurl shouldn't be too difficult.

Environment:AspNetCore3.1
Visual Studio Version:16.4.0

On AspNetCore3.0 or AspNetCore3.1 when debugging,I got an "Bad Request - Invalid Hostname" when i put Consul config in program.class.

Host.CreateDefaultBuilder(args)
           .ConfigureAppConfiguration(config =>
           {
               config.AddConsul("appsettings.json", options =>
               {
                   options.ConsulConfigurationOptions = _ =>
                   {
                       _.Address = new Uri("http://192.168.2.100:8500");
                       _.Datacenter = "dc1";
                   };
                   options.ReloadOnChange = true;
                   options.OnLoadException = ex => { throw ex.Exception; };
               });
           })
               .ConfigureWebHostDefaults(webBuilder =>
               {
                   webBuilder.UseStartup<Startup>();
               });

will this package cache kv in client side in case consul server is unavailable for a while?

I have a node (with data) in Consul like below:

Node: data/stage/process
Data: {"key1":value1,"key2":"value2","key3":"value3"}

I don't have any other children of "process" node in Consul. I just have json data for that node and that's all.

Now I keep a watch on that same node from below code. My idea is to keep a watch on the same node and get the value of "process" key in the code and whenever it is updated again from outside I want my watch to be triggered as well and then get the value of "process" key again on every reload whenever there is a change.

    var builder = new ConfigurationBuilder()
        .SetBasePath(env.ContentRootPath)
        .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
        .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true)
        .AddConsul(
            $"data/stage/process",
            options =>
            {
                options.ConsulConfigurationOptions =
                    cco => { cco.Address = new Uri("some-url"); };
                options.Optional = true;
                options.PollWaitTime = TimeSpan.FromSeconds(5);
                options.ReloadOnChange = true;
                options.Parser = new SimpleConfigurationParser();
                options.OnWatchException = ctx => TimeSpan.FromSeconds(ctx.ConsecutiveFailureCount);
            })
        .AddEnvironmentVariables();
    Configuration = builder.Build();

With the above code when I run it, I get an exception as below:

'The key must not be null or empty. Ensure that there is at least one key under the root of the process or that the data there contains more than just a single value.'

But if I just keep a watch on "data/stage" node then it works fine without any error and I can get the value of "process" key in the code so my question is - can I not just keep a watch on the same node for which I want to get the value also?

Hi,

It seems there is an issue with reading configuration hierarchy in package version 2.1.1. Whenever there is an overlap in names of rootKey and actual key in configuration the hierarchy will be ommitted. For example:

• AppName: ServiceBus
• RootKey: ServiceBus/Development/appsettings.json (the one passed to builder .AddConsul("ServiceBusUtilApi/Development/settings/"),
• ConsulKey: ServiceBus/Development/appsettings.json/Bus
• KeyValue: QueueName: "xxx"
• ExpectedResult: Bus:QueueName=xxx
• ActualResult: QueueName=xxx

Steps to reproduce:

1. Go to consul and add the configuration from above

curl -X PUT localhost:8500/v1/kv/ServiceBus/
curl -X PUT localhost:8500/v1/kv/ServiceBus/Development/
curl -X PUT localhost:8500/v1/kv/ServiceBus/Development/appsettings.json/
curl -X PUT localhost:8500/v1/kv/ServiceBus/Development/appsettings.json/Bus/
curl -X PUT -d @- localhost:8500/v1/kv/ServiceBus/Development/appsettings.json/Bus/QueueName <<< xxx

2. Create new project and add nuget Winton.Extensions.Configuration.Consul
3. Add consul

builder
    .AddConsul(
        "ServiceBus/Development/appsettings.json",
        cancellationToken,
        options =>
        {
            options.Parser = new SimpleConfigurationParser();
        });

4. Try to read expected config value

public Startup(IConfiguration config)
            : base(config, Constants.ApiTitle)
        {
            var value = config["Bus:QueueName"];
        }

5. They key wont be found
   5a) Alternatively you can you the sample app I attached to this issue ConsoleApp1.zip

Cause of the problem

The culprit for this lays in KVPairExtensions.cs line 26, specifically

kvPair.Key.TrimStart(rootKey.ToCharArray())

namely for example above this line will return an empty string, due to the way TrimStart works with char array. To verify this you can run the code below in console

internal class Program
    {
        private static void Main(string[] args)
        {
            TrimMe("ServiceBus/Development/appsettings.json", "ServiceBus/Development/appsettings.json/Bus");
            TrimMe("ServiceBus/Development/appsettings.json", "ServiceBus/Development/appsettings.json/BusUtil");
            TrimMe("ServiceBus/Development/appsettings.json", "ServiceBus/Development/appsettings.json/Service");
            TrimMe("ServiceBus/Development/appsettings.json", "ServiceBus/Development/appsettings.json/ServiceConfig");

            Console.ReadLine();
        }

        private static void TrimMe(string root, string key)
        {
            var trimResult = key.TrimStart(root.ToCharArray());
            Console.WriteLine($"Trim '{root}' from '{key}'");
            Console.WriteLine($"Trim result: '{trimResult}'");
        }
    }

Solution

To address this, I quickly whipped up a piece of code that does exact "TrimStart", and replaced TrimStart call. There are many other ways of achieving this, this was just quickest for me.

internal static string GetKey(this KVPair kvPair, string rootKey, string pairKey)
        {
            string key = string.Empty;
            string consulKey = kvPair.Key;
            if (consulKey.StartsWith(rootKey))
            {
                key = consulKey.Remove(0, rootKey.Length);
            }

            key = key.TrimStart('/').TrimEnd('/');
            return $"{key}:{pairKey}"
                .Replace('/', ':')
                .Trim(':');
        }

Approach

I have already fixed this locally and added some unit tests, so I can also do a pull request if you like?
If you need more information feel free to contact me.

Microsoft recomends to use ConfigureAwait(false) when it possible

Hi,

we are overriding application settings from appsettings.json with Consul. It works perfectly if we configure a json in Consul, exactly as it looks like in appsettings.

appsettings.json:

{
	"key1": "value1",
	"key2": "value2",
	"endpoints": {
		"service_x": "_",
		"service_y": "_"
	}
}

consul:
key: appsettings/service1
value:

{
	"key1": "value1",
	"key2": "value2",
	"endpoints": {
		"service_x": "_",
		"service_y": "_"
	}
}

In code we would get the value of service_x as follows: Configuration["endpoints:service_x"].

We want to share the endpoints configuration among services, so it needs to be configured separately in Consul with a value like this:

{
	"endpoints": {
		"service_x": "_",
		"service_y": "_"
	}
}

while I would expect to have a key like, for example, this:

'appsettings/endpoints'

As you can see, we are forced to repeat the endpoints segment, because the library trims the consul key from the resulting .NET key:

string key = $"{kvPair.Key.RemoveStart(rootKey).TrimEnd('/')}:{pair.Key}"
                                .Replace('/', ':')
                                .Trim(':');

Moreover, we want to configure the endpoints as simple key/value pairs and load them separately:

.AddConsul("appsettings/endpoints/service_x", ...)
.AddConsul("appsettings/endpoints/service_y" ...)

which results in an InvalidKeyPairException, since the library trims the whole key.

The suggestion is to configure what is the Consul's configuration root that needs to be trimmed:

Data = (result?.Response ?? new KVPair[0])
                    .Where(kvp => kvp.HasValue())
                    .SelectMany(kvp => kvp.ConvertToConfig(
                           _source.Root ?? _source.Key,
                           _source.Parser))
                    .ToDictionary(kvp => kvp.Key, kvp => kvp.Value, StringComparer.OrdinalIgnoreCase);

In the example above, the Root will be "appsettings"

Or is there a better solution?

https://github.com/dotnet/core/blob/master/release-notes/3.1/3.1.0/3.1.0.md

Getting keys from wrong service with similar name... but wasn't expecting this.

Example Consul Keys

v1/Company.Service.Name/Foo/Foo = bar1
v1/Company.Service.Name.Subname/Foo/Foo = bar2

Example Service Setup

        hostBuilder
                .ConfigureAppConfiguration((context, builder) =>
                {
                    builder.AddConsul("v1/Company.Service.Name", x =>
                    {
                        x.Parser = new SimpleConfigurationParser();
                        x.Optional = true;
                        x.ReloadOnChange = true;
                    });
                })

Expected .NET IConfiguration Result

"Foo:Foo" = bar1

Actual .NET IConfiguration Result

".Subname:Foo:Foo" = bar2
"Foo:Foo" = bar1

Add the following properties to the csproj to improve metadata of published NuGet package:

• Authors
• Company
• PackageIconUrl
• PackageLicenseUrl
• PackageProjectUrl

It would be convenient to have an options pattern style overload of AddConsul that made it straightforward to configure the consul settings from a local config file.

Hi,

How can I determine the decimal separator without overriding the global culture when deserializing JSON? As a matter of fact, I am asking to learn the most elegant approach to this question, thanks.

According to the implementation details in the Consul documentation on blocking queries, there are circumstances under which the index can be non-monotonically increasing. We should handle this in the ConsulConfigurationClient by resetting the _lastIndex if the new one we receive is less than current one.

There are some nice features in C# 8 which will make the code terser.

Hi,

How can I manually trigger ConsulConfigurationProvider => Dispose or CancellationToken from my project?

Our project sometimes throw error like below, I suspect that this error is logged when the application pool in IIS is recycled, due to using this provider on legacy .NET Framework (4.8).

Unable to watchChanges in Consul (consecutive failure count: 1) Shared folder due to A task was canceled. and stacktrace    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Consul.GetRequest`1.<Execute>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Winton.Extensions.Configuration.Consul.ConsulConfigurationProvider.<GetKvPairs>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Winton.Extensions.Configuration.Consul.ConsulConfigurationProvider.<PollingLoop>d__11.MoveNext()

Additionally, since this error appears in the logs multiple times, I suspect it's as if the PollingLoop Task is running unnecessarily in parallel. Could it be that the PollingLoop is somehow started concurrently during startup?

Hi,

We are using an application .NET CORE 2.2 that should also be implemented in 3.2.
Now the Parsers expects to manipulate KV, we would expect that also in this class KVPairExtensions.cs line of code 19
using Stream stream = new MemoryStream(kvPair.Value);
in the stream both key and value would be injected because in our case we need to remove the underscores in our keys, then both can be used to populate the IConfiguration.

Here's what we need in MyCustomConfigurationParser.cs:

public IDictionary<string, string> Parse(Stream stream)
{

        var parsedDictionary = _configurationParser.Parse(stream);
        return parsedDictionary.ToDictionary(kv => kv.Key.Replace("_", string.Empty), kv => kv.Value);   
 }

but in kv.Key is empty. Our configuration is as a tree.
I hope I’ve made myself clear.

Thank you.

The test Winton.Extensions.Configuration.Consul.ConsulConfigurationClientTests+Watch.ShouldCallReloadOnChangeTokenIfIndexForKeyHasUpdated intermittently hangs and therefore seems to have a race condition in it.

I'm wiring up consul configuration using:

...
var address = new UriBuilder(config.Scheme, config.Host, config.Port).Uri;
builder.AddConsul("mps-config-environment/yaml",
    options =>
    {
        options.ConsulConfigurationOptions = client => client.Address = address;
        options.Parser = new YamlConfigurationParser();
        options.Optional = true;
        options.ReloadOnChange = true;
    });

Then interactively running a dotnet core 3.1 container and shutting it down via ctrl+c:

docker run -it -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_REGION -e ASPNETCORE_ENVIRONMENT=Development -P devops/chaos-monkey
[22:47:08 INF] (Lifetime) Now listening on: http://[::]:80
[22:47:08 INF] (AWSSDK) Found credentials using the AWS SDK's default credential search
[22:47:10 INF] (Lifetime) Application started. Press Ctrl+C to shut down.
[22:47:10 INF] (Lifetime) Hosting environment: Development
[22:47:10 INF] (Lifetime) Content root path: /app
^C[22:47:33 INF] (Lifetime) Application is shutting down...
chaos-monkey exception
System.AggregateException: One or more errors occurred. (A task was canceled.)
 ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout)
   at Winton.Extensions.Configuration.Consul.ConsulConfigurationProvider.Dispose()
   at Microsoft.Extensions.Configuration.ConfigurationRoot.Dispose()
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.DisposeAsync()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Extensions.Hosting.Internal.Host.DisposeAsync()
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Pharos.DevOps.ChaosMonkey.Program.<>c.<Main>b__0_0(String[] args) in D:\Devel\Mps\devops-chaos-monkey\Service\Program.cs:line 19
   at Pharos.Extensions.Logging.HostRunner.Run(String appName, String[] args, Action`1 runner)

TaskCanceledException thrown from ConsulConfigurationProvider.Dispose.

Perhaps the cancellation token should be passed to Task.Wait as well as being wrapped in a try-catch to swallow any cancellation related exception?

Do you have plan, upgrade library to .NET 7 ?
Actualy it throw excetpion:
"Cannot access a disposed object. Object name: 'ConfigurationManager'."

My mistake, documentation need correct ;)

I recently started working with asp.net (like few days old) and working on integrating our application with Consul as my first project so I am confuse on few things. I am trying to integrate Winton Consul library in my project as I need to download all keys/values from KV store in Consul. Also keep a watch on all the keys and get notified once any key is updated something like where I can properly register a reload handler to my IConfiguration and access the updated configuration values in the handler.

I noticed we have this class in our project which starts the server. Figured it out after reading asp.net fundamentals from here

    public class KestrelWebApiServer : IWebApiServer
    {
        public void StartWebApiServer()
        {
            var host = new WebHostBuilder()
                .UseContentRoot(Directory.GetCurrentDirectory())
                .UseKestrel(o =>
                {
                    o.Limits.MaxResponseBufferSize = null;
                    o.Limits.MinResponseDataRate = null;
                    o.Limits.MaxConcurrentConnections = 1000;
                    o.Limits.MaxConcurrentUpgradedConnections = 1000;
                    o.AllowSynchronousIO = true;
                })
                .UseStartup<KestrelBooter>()
                .UseUrls("http://*:1335")
                .Build();
            host.Run();
        }
    }

This is what I see in my KestrelBooter class:

    public KestrelBooter(IHostingEnvironment env)
    {
        var builder = new ConfigurationBuilder()
            .SetBasePath(env.ContentRootPath)
            .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
            .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true)
            .AddEnvironmentVariables();
        Configuration = builder.Build();
    }

    public IConfigurationRoot Configuration { get; }

    public void ConfigureServices(IServiceCollection services)
    {
        // we doing some stuff here related to DI
    }

    public void Configure(IApplicationBuilder app, IHostApplicationLifetime lifetime)
    {
        // some stuff here
    }

Question:

I am able to make Winton website example work as it is in my main method just for testing purpose and it works fine but I am confuse on where I should be adding this below particular piece of code in my above classes and methods (in my real application code)?

    public static IHostBuilder CreateHostBuilder(string[] args)
    {
        return Host
            .CreateDefaultBuilder(args)
            .ConfigureWebHostDefaults(builder => builder.UseStartup<Startup>())
            .ConfigureAppConfiguration(
                builder =>
                {
                    builder
                        .AddConsul(
                            "consul-testing",
                            options =>
                            {
                                options.ConsulConfigurationOptions =
                                    cco => { cco.Address = new Uri("http://consul.host.orcld.com/"); };
                                options.Optional = true;
                                options.PollWaitTime = TimeSpan.FromSeconds(5);
                                options.ReloadOnChange = true;
                            })
                        .AddEnvironmentVariables();
                });
    }

And then how can I use IConfiguration to get all the keys and values? Also how can I enable on change notification where my handler gets invoked whenever any key is updated? Any help is greatly appreciated.

Hi,
first of all, thank you to all the contributors for this wonderful library.
Our vulnerability scans have flagged some vulnerabilities that can be traced to this package.
Is it planned to release a new version of this library? Otherwise we'll have to find alternatives.

Currently we only support JSON config. Would be good to support YAML. Need to implement Winton.Extensions.Configuration.Consul.Parsers.IConfiguration by adding an Yaml folder under Parsers and adding Winton.Extensions.Configuration.Consul.Parsers.Yaml.YamlConfigurationParser.

Instead of writing a JSON parser here, we can make use of the one provided by Microsoft in their JSON configuration library. This does mean taking an explicit dep on the JSON implementation of the config abstractions, but it also means we can get rid of our explicit dep on NewtonSoft.JSON and also remove some reasonably complex code.

When Winton reloads the configuration from Consul, is there any event that we can listen to? I am asking so since we would like to invalidate our local cache once configuration gets reloaded. Currently, we are not sure of how to achieve this. Any suggestion to achieve the above?

When Consul is not available OnWatchException is triggered constantly. Because there is no delay being applied in the catch.

while (!cancellationToken.IsCancellationRequested)
            {
                try
                {
                    if (await HasValueChanged(key, cancellationToken).ConfigureAwait(false))
                    {
                        ConfigurationReloadToken previousToken = Interlocked.Exchange(
                            ref _reloadToken,
                            new ConfigurationReloadToken());
                        previousToken.OnReload();
                        return;
                    }
                }
                catch (Exception exception)
                {
                    var exceptionContext = new ConsulWatchExceptionContext(cancellationToken, exception);
                    onException?.Invoke(exceptionContext);
                }
            }

This is the exception being thrown:

  System.Net.Http.HttpRequestException: Connection refused ---> System.Net.Sockets.SocketException: Connection refused
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Consul.GetRequest`1.Execute(CancellationToken ct)
   at Winton.Extensions.Configuration.Consul.ConsulConfigurationClient.GetKvPairs(String key, CancellationToken cancellationToken, QueryOptions queryOptions)
   at Winton.Extensions.Configuration.Consul.ConsulConfigurationClient.HasValueChanged(String key, CancellationToken cancellationToken)
   at Winton.Extensions.Configuration.Consul.ConsulConfigurationClient.PollForChanges(String key, Action`1 onException, CancellationToken cancellationToken)

I have a single key value that I want to read (consul) in alongside the appsettings.json file(not consul)
The value is a string representation of a boolean (true, false)
but when calling the injected _configuration["test-key"] from within the controller I am not getting any result.
Can I please have some guidance of what is potentially happening? I don't see any exeptions in the logs.

     public static IWebHostBuilder CreateWebHostBuilder(string[] args, CancellationTokenSource cancellationTokenSource) =>
          WebHost.CreateDefaultBuilder(args)
                 .ConfigureAppConfiguration(
                      (hostingContext, builder) =>
                      {
                          builder
                              .AddConsul("test-key",
                                  cancellationTokenSource.Token,
                                  options =>
                                  {
                                      options.ConsulConfigurationOptions =
                                          cco => { cco.Address = new Uri("http://127.0.0.1:8500"); };
                                      options.Parser = new SimpleConfigurationParser();
                                      options.Optional = false;
                                      options.ReloadOnChange = true;
                                      options.OnLoadException = exceptionContext =>
                                      {
                                          Console.WriteLine($"{exceptionContext.Exception}");
                                          exceptionContext.Ignore = true;

                                      };
                              })
                              .AddEnvironmentVariables();

                 })
                 .ConfigureKestrel(options => { options.Limits.MinResponseDataRate = null; })
                 .UseStartup<Startup>()
                 .UseNLog();

consul sturcture would be

test-key:test-value
features/feature1:true
features/feature2:false

How am I able to get just keys prefixed by features?

While parsing Json data we noticed that the PrimitiveVisitor does not respect the default value of JsonTextReader's CultureInfo (which, by the way, is CultureInfo.InvariantCulture by default).

This causes issues when trying to read invariant double values from json data and try to convert them to actual invariant double values within code (e.g. the value 0.1).

We traced this issue to line 36 of JsonPrimitiveVisitor.cs:

new KeyValuePair<string, string>(key, primitive.ToString())

will cause the token to be converted to its stringified counter type ignoring the culture set in the parser (e.g. "0.1" in JSON becomes "0,1" in the configuration data)

This can be easily solved by using the following method provided by Newtonsoft Json instead:

new KeyValuePair<string, string>(key, primitive.Value<string>())

This will convert the token to a stringified counter type respecting the set culture (e.g. "0.1" in JSON remains "0.1" in code).

I'm in crunch mode at the moment but I can supply a PR later on if you want (or you can simply apply the fix and write a small test to confirm the fix yourself).

When we set the ReloadOnChange property of ConsulConfigurationSource to true, what is the time interval after which the package poll Consul for changes? Is there way to specify the time interval for polling? I couldn't locate a property in ConsulConfigruationSource to set the interval. Basically, we don't want to keep polling Consul for changes. Rather, we would like to poll it for every set interval of time like 10 mins or so. Is there a way to do this?

Hi.
Not sure this is the right place to post so do not hesitate to tell me.
I would like to use a retry policy (for instance with Polly) whenever a request fails. But I'm not sure the current API allows me to do this. Or I don't know how to do it :).
So what is the best way to achieve this ?

I am able to do it in the OnLoadException callback by forcing context.Source.Build(configurationBuilder).Load() and finally add it the current IConfigurationBuilder but for me we should be able to apply the policy directly on the Load method of the ConsulConfigurationProvider class, shouldn't we ?

I can also create a ConfigurationBuilder and add ConsulConfigurationSource (s) only, call the Build method, retry as long as it fails, and finally add the created IConfiguration to the ConfigurationBuilder that will be used in my StartUp (bootstrap) class.

Thx a lot and have a good day

Hello,

Even though I set "PollWaitTime" to 5 minutes, the values are updating immediately. How is this possible?

Hello,

Are there any plans on implementing "Add support for expanded configuration where the configuration is a tree of KV pairs under the root key", mentioned on in the backlog section?

Hi guys,

I have forked this project to add an enhancement. I've added an OnConfigChanged Action in the ConsulConfigurationSource. This action is called whenever the ChangeToken fires it's OnChange method.

What's the use case? Well, in my case I need to restart some computation services to work with the updated configuration values. I can now do this by hooking up some logic to OnConfigChanged. Like this:

configBuilder
     .AddConsul(
         "MyService.Development"
         cancellationTokenSource.Token,
         options =>
         {
             // ....
             options.ReloadOnChange = true;
             // Ta da.... Here it is
             options.OnConfigChanged = _ => { Console.WriteLine("Oh my god, change is upon us! We need to do stuff.") }
         })
     .AddEnvironmentVariables()
     .Build();

Is this worth a pull request? Or, better even, are there perhaps existing ways to achieve this? Let me know.
NB: The full commit is visible here.

When using the library in an application that targets netframework the following error is raised:

System.MissingMethodException
  HResult=0x80131513
  Message=Method not found: 'Void Consul.ConsulClient..ctor(System.Action`1<Consul.ConsulClientConfiguration>, System.Action`1<System.Net.Http.HttpClient>, System.Action`1<System.Net.Http.HttpClientHandler>)'.
  Source=Winton.Extensions.Configuration.Consul
  StackTrace:
   at Winton.Extensions.Configuration.Consul.ConsulConfigurationProvider.Load()
   at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at TestApp.Startup..ctor(IHostingEnvironment env, ILoggerFactory loggerFactory)

This seems to be due to conditional compilation in ConsulClient so that it takes a parameter of type Action<WebRequestHandler> rather than Action<HttpClientHandler> when targeting netframework.

This was not a problem in version 1.0.0. Which would suggest that it was because in version 1.0.0 this library specified net45 as a targetframework and explicitly referenced System.Net.Http and System.Net.WebRequest. This behaviour should be brought back for 1.1.0 and 2.0.0 to fix the runtime error for applications compiled against netframework.

git2consul supports a mode where it expands a json document in git into multiple key/value pairs in consul: https://github.com/Cimpress-MCP/git2consul/blob/master/README.md#expand_keys

Support reading config that's formatted like that.

Currently we only support JSON config. Would be good to support XML. Need to implement Winton.Extensions.Configuration.Consul.Parsers.IConfiguration by adding an XML folder under Parsers and adding Winton.Extensions.Configuration.Consul.Parsers.Xml.XmlConfigurationParser.

I am chasing a problem in my .NET Core 3.1 app using version 3.0.1 of this Winton.Extensions.Configuration.Consul library where I can properly register a reload handler to my IConfiguration and access the updated configuration values in the handler when I build app configuration like this:

      Host.CreateDefaultBuilder()
        .ConfigureAppConfiguration((ctx, config) => {
          config.AddJsonFile("appSettings.json", optional: true);
          config.AddJsonFile("appSettings.Production.json", optional: true);
          config.AddConsul("cabernet", opt =>
          {
            opt.ConsulConfigurationOptions = cco =>
            {
              cco.Address = new Uri("http://localhost:8500");
            };
            opt.ReloadOnChange = true;
            opt.Parser = new SimpleConfigurationParser();
          });
        })

and subscribe to change notifications like this:

       ChangeToken.OnChange(configuration.GetReloadToken, _ => ConfigChanged(), new object());

where configuration is the IConfiguration instance provided to the ConfigureServces() method inside the IHostingContext.Configuration property.

When I add the ConsulConfiguration to my configuration builder and then add environment and command line args like this, however:

      Host.CreateDefaultBuilder()
        .ConfigureAppConfiguration((ctx, config) => {
          config.AddJsonFile("appSettings.json", optional: true);
          config.AddJsonFile("appSettings.Production.json", optional: true);
          config.AddConsul("cabernet", opt =>
          {
            opt.ConsulConfigurationOptions = cco =>
            {
              cco.Address = new Uri("http://localhost:8500");
            };
            opt.ReloadOnChange = true;
            opt.Parser = new SimpleConfigurationParser();
          });
          config.AddEnvironmentVariables();
          config.AddCommandLine(args);
        })

I still get the change notification but the updated config values are no longer available and access my config values through the IConfiguration provider only returns me the old values.

I tried reproducing this in some unit tests (see code here) but am a bit uncertain how the setup for these tests works so my tests are failing when building the ConfigurationRoot.

In the end I'm not sure whether this problem I'm chasing is in the Winton.Extensions.Configuration.Consul library, the Microsoft.Extensions.Configuration library, or (most likely) my own use of these two libraries together through misunderstanding a nuance of setup and change token handling.

Would it be a good idea to not poll back-to-back? Maybe connections to consul are expensive (take a long time) and there should be an option to check for Consul changes every X seconds

i have a configuration template that i like to write to consul on service startup so i need to write new keys back to consul but currently writing in consul through this library is not supported.

If the configuration key is deleted in Consul or even set to an empty string, the configuration is not reloaded, it still has the old value.
This behavior is explicitly specified in the code, and I can't figure out the reason for it.