Feature Request: Specify configuration root about winton.extensions.configuration.consul HOT 10 CLOSED

Hi @paschka76,

Just to check I have understood you correctly, you would like to factor out the endpoints config into its own section in Consul so that it can be loaded by many applications? Is that correct?

In that case you have a few options I think.

1. Place the endpoints config under its own key in Consul. For example appsettings/serviceEndpoints which would contain the following json data:

{
	"endpoints": {
		"service_x": "_",
		"service_y": "_"
	}
}

Then in the application you could do .AddConsul("appsettings/serviceEndpoints", ...).

2. If you wanted to name the key in Consul 'endpoints' then you could instead change the JSON to be:

"endpoints": {
	"service_x": "_",
	"service_y": "_"
}

and place it under a key like appsettings in Consul and then do the following to load it ("appsettings", ...). This would then allow you to still access it as Configuration["endpoints:service_x"] in your code.

3. If you would rather have each endpoint as a single key value pair in Consul then this is also an option. In Consul you could have the following config arrangement:

-appsettings
    -endpoints
        -service_x: 'an endpoint'
        -service_y: 'another endpoint'

And then do .AddConsul("appsettings", cancellationToken, options => options.Parser = new SimpleConfigurationParser()). This will then parse each KV pair in Consul under the 'appsettings' key as a scalar value.

As of version 2.1 when you call .AddConsul it will load all keys in Consul under the specified root key. The root key you specify does not have to point directly to a value in Consul, it can point to a key that contains child keys. By default it will parse the values as JSON, but as you can see it is possible to also use the alternative SimpleConfigurationParser. So I believe you should be able to refactor your config in Consul so that you can load the endpoints config in the way that you desire. Also, as you seem to have already figured out, you can call .AddConsul multiple times to load different sections of config from Consul, so you can therefore use the default parser for some sections and the SimpleConfigurationParser for others.

Let me know if I've misunderstood your problem.

from winton.extensions.configuration.consul.

Hi @Choc13 ,

thanks for the quick feedback!

Yes, you understood everything correct. I should have emphasized though that the end-goal is exactly not to load the entire Consul subtree. That is what I mean by loading the endpoints separately.

In the following case:

-appsettings
    -endpoints
        -service_x: 'an endpoint'
        -service_y: 'another endpoint'

under appsettings I will have configurations for many microservices which I don't want to load. Endpoints is a simple example, but there will also be complex configurations of which I want to load only a specific segment into current microservice.

from winton.extensions.configuration.consul.

OK. This seems to me like this is more of a question of how you should store your config in Consul than any limitation of this library. Would it not be possible for you to instead group your config values by microservice? Perhaps something like:

-appsettings
    -service_x
        -endpoint: 'https://service_x
        -foo: 'bar'
    -service_y:
        -endpoint: 'https://service_y'
        -foo: 'baz'

That way you could target config on a per microservice basis, rather than trying to load on specific endpoints and specific foos for a subset of microservices.

I would like to avoid making the options to this library overly complex and I still feel like at present there is enough flexibility for you to achieve what you need to without adding additional optional parameters.

from winton.extensions.configuration.consul.

Such a Consul structure is exactly what we want to improve.

For example, in the following case, we will have to go through each microservice config in order to change https://endpoint_x to https://endpoint_y (provided microservice_x, microservice_y and microservice_z are different keys in Consul):

-appsettings
    -microservice_x
        -endpoint: 'https://endpoint_x'
        -foo: 'bar'
    -microservice_y:
        -endpoint: 'https://endpoint_x'
        -foo: 'baz'
    -microservice_z:
        -endpoint: 'https://endpoint_x'
        -foo: 'bay'

I understand that you want to avoid making the options more complex, as long as it fits the needs of the majority.

But regardless of how we try to organize our Consul here, I still think there is a general limitation to this library, in which you can't load a single simple (non-json) value, like '.../path/foo':'bar', because the library doesn't know what is the key for .NET in this case.

Or would you say this is a rare use case?

from winton.extensions.configuration.consul.

So I've been giving this some thought, and you are correct that there is a limitation in that it is not possible to load a single simple value from directly under the root key. However, I did not imagine anyone needing such functionality and I still think it is a rare case that can always be avoided by adding an additional key into Consul under the root key. I would have thought that, for those people wanting to use expanded config (simple values), they would most likely not want the Consul root key as a part of the key in the .NET configuration. For instance we have our configuration stored like:

-environment (e.g UAT)
    -application-1 (e.g MyApp)
        -config-section-1 (e.g. logging configuration)
    -application-2
        -config-section-1

So in that case it wouldn't make sense to have 'MyApp' be a section in the .NET configuration for MyApp, even though it is the root key for all config related to that application in Consul.

Also, I should point out that this behaviour is tied to the fact that initially this library only supported JSON based values and in that instance it required the full path in consul to the value, but would not use the Consul keys in anway when creating the keys in .NET. As of v2.1 I have added the ability to load expanded simple config trees from Consul. This led to me also adding the ability to point to any key in Consul (not necessarily one with a value) and the library will load the entire tree under the root key, using all child Consul keys as a sections in the .NET config.

As for your specific example, I am still a little bit unclear on how your config is structured, and I think if I can understand this better it might help me understand whether this is truly a limitation of this library, or something that can be fixed by re-organising your Consul setup.

In particular, why do all of your microservices have the same endpoint? Surely they would be accessible at different URLs? Or is this the URL for another service that each of these services need to communicate with? I think if you could answer those questions I could provide a more direct answer.

from winton.extensions.configuration.consul.

Oh yes, it seems to be a misunderstanding. It is a URL to another service:

-appsettings
    -microservice_x
        -mapping_service: 'https:/mapping_service_url'
        -foo: 'x'
    -microservice_y:
        -mapping_service: 'https:/mapping_service_url'
        -orders_service: 'https:/orders_service_url'
        -bar: 'y'
    -microservice_z:
        -mapping_service: 'https:/mapping_service_url'
        -orders_service: 'https:/orders_service_url'
        -baz: 'z'

The reason we don't want to load entire JSONs into a microservice when it needs only a part of it (hence breaking down the JSON with all shared endpoint URLs into single key value pairs) is that when one configuration is changed, only the affected microservices have to be reloaded (as in the example above, if orders service url is changed, microservice_x doesn't have to be reloaded).

And another thing that I mentioned earlier, since the library strips the key, we usually have to repeat the name of the segment in the JSON. For example, we have the following key:
/path/servicebus
In order to access servicebus configuration in .NET we would make a call like this:
Configuration["servicebus:host"]

For this, we need to repeat the keyword servicebus in the JSON value:

{
     servicebus: {
          host: https://...
          ...
     }
} 

Surely, one can leave with it, but this redundancy can be avoided if we find a solution for letting the library know which part of the path (if any) should go in the .NET configuration key.

Please let me know if I can provide additional clarifications.

Thank you!

from winton.extensions.configuration.consul.

Thanks for the clarifications and apologies for the slow reply, with it being Christmas I have been busy with other things.

I think your servicebus example is compelling and highlights a reasonable need for the feature you are requesting. I will attempt an implementation and see how it fits in and then decide whether or not this is something that can be accommodated.

On a side note though it sounds like in your case you would like to break up the configs for a couple of reasons:

1. It makes it easier for you to manage the config as you only need to update service addresses in one place, and
2. You can then only reload the relevant service configuration when they are modified, rather than reloading all upstream service configuration.

I would just like to point out a couple of things that I think you might have overlooked that might make you reconsider how to achieve those 2 above goals and whether you need this feature. These are only recommendations and suggestions so please feel free to ignore them if you are already aware.

On point 1), Consul provides service discovery as a feature in its own right. You might want to consider using this rather than having service addresses in the KV store.

As for point 2), be aware that when you specify that this library should watch Consul for changes, e.g. by calling AddConsul("/path/servicebus/", cancellationToken, options => options.ReloadOnChange = true), it will set up long polling on a background thread. So splitting up the config into lots of small pieces and calling .AddConsul many times might have more of a negative performance impact than reloading the entire config tree each time it changes. Obviously the actual performance characteristics will depend on the overall size of your config and the number of partitions you are going to create, but I thought it would be worth pointing out. Also, given that config changes are relatively infrequent I personally wouldn't worry too much about the overhead of reloading it. Again, that's for you to decide on given your circumstances and you would need to profile things to actually figure out the performance characteristics.

from winton.extensions.configuration.consul.

@paschka76 I've just opened a PR for this. If you have time take a look and let me know if this addresses your needs.

from winton.extensions.configuration.consul.

Thank you very much for your reply and for such a quick implementation of a solution!
Good point regarding service discovery. It will improve the way we configure microservice endpoints. But we will still have the same situation for another configurations like this.
Also great point regarding background threads. We will have to be extra careful with this.

So, having your 2 points in mind, we will still profit a lot from the solution you implemented. I took a look at the change - exactly what we need!

Thank you very much!

from winton.extensions.configuration.consul.

No problem. Version 2.2.0 has just been built and should be available on NuGet once it has been indexed.

from winton.extensions.configuration.consul.