heetch / confita Goto Github PK
View Code? Open in Web Editor NEWLoad configuration in cascade from multiple backends into a struct
License: MIT License
Load configuration in cascade from multiple backends into a struct
License: MIT License
Let f
be the number of fields in a struct, and b
be the number of backends, the worst case scenario makes Confita call the backends f
* b
times per structure.
If the backends are all remote ones (etcd, vault, etc.) that would mean f
* b
network calls.
We can improve that by taking advantage of the batch capabilities of some of the backends and do the cascading resolution in memory.
In order to avoid some useless processing we can specify for a field in which backend we can find its value.
e.g.
type Config struct {
Foo string `config:"foo",bcknd:"etcd"`
Bar string `config:"bar",bcknd:"consul"`
}
Currently, the TOML support that was added supports simple Key/Value configurations but doesn't handle some of TOML's more complex structures.
We should:
The spec for TOML can be found here: https://github.com/toml-lang/toml
jwt-go
has a security vulnerability (CVE-2020-26160). There is no patch available and users of jwt-go
are advised to migrate to golang-jwt
at version 3.2.1. ๐
Add go1.11.x
on the .travis.yml
file.
I have a struct with a field which is a []string
but when I call the help flag I have a wrong type and a wrong default value:
type Etcd struct {
Endpoints []string `config:"etcd-endpoints"`
}
var etcd Etcd
etcd.Endpoints = []string{"127.0.0.1:2379"}
The help output:
$> bin -h
Usage of bin:
-etcd-endpoints value
(default etcd-endpoints)
We don't need to support earlier Go versions, and it would be nice to simplify the travis configuration, so let's drop explicit support for versions before 1.12.
When using the env backend, then the only keys considered will be those with config:
field tags. However, if you're using a file backend, then the entire struct is unmarshaled from the source, and all fields will be considered, even those without a config tag.
So it's possible there are some fields which are configurable only with the file backend and not with the env backend.
I think that probably that any keys that don't specify the config:
tag should be ignored so backends are more consistent.
I noticed a regression in behaviour. Here is code for reproducing the issue.
package confitajson
import (
"context"
"log"
"testing"
"github.com/heetch/confita"
"github.com/heetch/confita/backend/file"
)
func TestLoad(t *testing.T) {
b := file.NewBackend("config.json")
var c struct {
Field string `json:"field"`
}
if err := confita.NewLoader(b).Load(context.Background(), &c); err != nil {
log.Fatalf("failed to load config: %v", err)
}
if c.Field != "value" {
log.Fatal("field value is not updated")
}
}
Content of config.json
file is following:
{
"field": "value"
}
When running this test on the latest confita release, struct field is not updated. But on version v0.7.0 code works fine.
prime@bee ~/C/confitajson> go get github.com/heetch/[email protected]
go: downloading github.com/heetch/confita v0.7.0
prime@bee ~/C/confitajson> go test .
ok github.com/slon/confitajson 0.002s
prime@bee ~/C/confitajson> go get github.com/heetch/confita
go: github.com/heetch/confita upgrade => v0.9.1
prime@bee ~/C/confitajson> go test .
2020/05/27 20:02:34 field value is not updated
FAIL github.com/slon/confitajson 0.002s
FAIL
I also noticed, that removing these 3 lines, seems to fix this issue for me. https://github.com/heetch/confita/blob/master/config.go#L183-L185
Is that change in behaviour intentional, or is it indeed a regression?
The new feature that allows flags to have descriptions precludes them having a comma in.
We should document that restriction.
I have the exact same issue as #30 when upgrading from v0.8.0 to v0.9.0.
A simple map, could be used by library wrappers to prefill some configuration variables during runtime for example.
Hello! Sometimes I need to do something like this: ./my_app --config configs/stable.yaml --port 7777
. It's quite the same as a file backend but with a dynamic path to the file. In this case, the port will be overwritten with the provided value.
I haven't find any simple or nice solution for this.
I think that it will be nice to have a sort of DynamicFile backend with options like key, short, required and so on.
Help usage generation would be improved if struct members supported a "description" tag (or a sub-tag of "config"), e.g.
type Configuration struct {
Port int `config:"port" description:"listening port number"`
}
or
type Configuration struct {
Port int `config:"port,description=listening port number"`
}
This was previously discussed in #7, but my tests are giving different results, If I define a variable in a file as an env var, the last backend on the list wins
Examples:
l := confita.NewLoader(file.NewBackend(path),env.NewBackend())
in this case the env backend overwrites the values in the file files
n := confita.NewLoader(env.NewBackend(),file.NewBackend(path))
same issue here, the file will overwrite any value previously set as env var
The docs say: "Confita scans a struct for config tags and calls all the backends one after another until the key is found" from my understanding once found it stops and continue looking for the next config key
There is a working example of the issue:
https://play.golang.org/p/c8l9NbRnJOT
Thanks to @rogpeppe for your go playground example on issue 61
Looking for backend currently implemented I was asking myself about the utility of a new one able to handle strings given as parameter.
For exemple if the configuration is fetched from a remote URL (that returns it as a JSON or YAML etc..) - it could be nice to be able to include the fetched result in the config.
As you already are handling Unmarshaler
here : ( https://github.com/heetch/confita/blob/master/config.go#L161 )
it could, for exemple, take the following form :
loader := confita.NewLoader(
data.NewBackend(json.NewDecoder(content)),
)
or, if for any reason you want to keep the content :
loader := confita.NewLoader(
data.NewBackend(content, "json"),
)
( maybe data is not the most relevant name tbh ๐ )
for the rest it's pretty similar with what you already are doing in the file backend (just replacing the file operations part)
The XCU specification says that the environment variable names consist solely of upper-case letters, digits and the "_" (underscore).
Maybe, we should respect this spec and not handle the case if names contains "-" (dash).
Most command-line tools support a short and a long form for flags, e.g. "-p" or "--port".
Currently confita only supports a single flag per member.
Suggestion: add a "short" sub-tag and handle it as a synonym of the default name. Also update help usage generation to reflect this.
type Configuration struct {
Port int `config:"port,short=p"`
}
With confita, it's usual to use Load on a struct which doesn't contain all the possible configuration information. For example, we might require information on just the configuration required by a single component.
Unfortunately this does not work well with the flags backend, because it's not possible to parse command line flags correctly without knowing all the possible flags.
This code demonstrates the issue: https://play.golang.org/p/rxws8K5Noxb
Hi! I am using confita configs as API for internal libraries in my projects. I've tried to use config structs multiple times in single project setup. And I found a problem for my usage. Field tags can't repeat in one config:
type HTTPConfig struct {
Addr string `config:"addr"`
Port int `config:"port"`
}
type AppConfig struct {
MetricsHTTPServer HTTPConfig
LoggingHTTPServer HTTPConfig
}
There are not possibilities for defining different options for different servers. Also if we'll try to use flags
as backend, we'll get panic (below):
panic: ./cmd/example/example flag redefined: addr
goroutine 1 [running]:
flag.(*FlagSet).Var(0xc0001371a0, 0x551e910, 0xc000614440, 0x4f288db, 0xc, 0x0, 0x0)
/-S/contrib/go/_std/src/flag/flag.go:871 +0x485
flag.(*FlagSet).StringVar(...)
/-S/contrib/go/_std/src/flag/flag.go:760
flag.(*FlagSet).String(0xc0001371a0, 0x4f288db, 0xc, 0x0, 0x0, 0x0, 0x0, 0xc000614430)
/-S/contrib/go/_std/src/flag/flag.go:773 +0xa5
flag.String(...)
/-S/contrib/go/_std/src/flag/flag.go:780
github.com/heetch/confita/backend/flags.(*Backend).LoadStruct(0x5d76df8, 0x552dea8, 0xc0005d2340, 0xc00025f080, 0x0, 0x0)
/-S/vendor/github.com/heetch/confita/backend/flags/flags.go:73 +0x548
github.com/heetch/confita.(*Loader).resolve(0xc00025f050, 0x552dea8, 0xc0005d2340, 0xc00025f080, 0x5d30240, 0x199)
/-S/vendor/github.com/heetch/confita/config.go:171 +0x788
github.com/heetch/confita.(*Loader).Load(0xc00025f050, 0x552dea8, 0xc0005d2340, 0x4f63ac0, 0x5d30240, 0x49, 0xd)
/-S/vendor/github.com/heetch/confita/config.go:58 +0x20a
It may be solved using something like prefixes. Example usage is following:
type HTTPConfig struct {
Addr string `config:"addr"`
Port int `config:"port"`
}
type AppConfig struct {
MetricsHTTPServer HTTPConfig `config:"prefix=metrics"`
LoggingHTTPServer HTTPConfig `config:"prefix=logging"`
}
Or more idiomatic if it needs. I think this feature will be useful not only me))
.properties file are a common configuration file format.
Support for this format could be added.
I created a config struct like this
type rabbitMQ struct {
USER string `config:"user,required"`
PASSWORD string `config:"password,required"`
HOST string `config:"host"`
PORT string `config:"port"`
}
type dbConfig struct {
DIALECT string `config:"dialect,required"`
NAME string `config:"name,required"`
USER string `config:"user,required"`
PASSWORD string `config:"password,required"`
HOST string `config:"host"`
PORT string `config:"port"`
}
type Config struct {
DATABASES map[string]dbConfig `config:"databases,required"`
RABBITMQ rabbitMQ `config:"rabbitmq,required"`
ENVIRONMENT string `config:"environment"`
}
confita seems to be ignoring the required tag for all but the password field in the rabbitMQ config
The file I'm reading from is a yaml file
Example:
confita.NewLoader(
file.NewBackend("somefile.json"),
file.NewBackend("some-other-file.yaml"),
)
We could use the right parser based on the extension.
Hey !
Using the library for a project I am looking for a way to be able to get a single value from the config and potentially load a different file following the returned value.
I think it should be really interesting to have a way to get this value without loading the whole config on a struct or having to create a struct with just one field to get it.
After taking a quick look at how you implemented the backends I think it should not be that hard to add a method like :
func (l *Loader) Get(name string) interface{}
or maybe something typed with a method for each ? (String, int ...)
WDYT ? I should take a look and try to work on it if this is something you are interested by.
Some examples of the weird behavior when mixing flag and envvar:
โ bar default overwrite bar value from envvar
$ BAR=bar-envvar ./test-confita -foo=foo-flag
2019/09/29 10:24:37 cfg.Foo: 'foo-flag'
2019/09/29 10:24:37 cfg.Bar: 'bar-default'
โ foo value from envvar it's overwritten by flag as expected
$ FOO=foo-envvar ./test-confita -foo=foo-flag
2019/09/29 10:29:22 cfg.Foo: 'foo-flag'
2019/09/29 10:29:22 cfg.Bar: 'bar-default'
โ foo keeps the value from envvar ignoring the flag
$ BAR=bar-envvar FOO=foo-envvar ./test-confita -foo=foo-flag
2019/09/29 10:25:33 cfg.Foo: 'foo-envvar'
2019/09/29 10:25:33 cfg.Bar: 'bar-envvar'
go code:
package main
import (
"context"
"log"
"github.com/heetch/confita"
"github.com/heetch/confita/backend/env"
"github.com/heetch/confita/backend/flags"
)
type Config struct {
Bar string `config:"bar"`
Foo string `config:"foo"`
}
func main() {
cfg := Config{
Bar: "bar-default",
}
loader := confita.NewLoader(
env.NewBackend(),
flags.NewBackend(),
)
err := loader.Load(context.Background(), &cfg)
if err != nil {
panic(err)
}
log.Printf("cfg.Foo: '%s'\n", cfg.Foo)
log.Printf("cfg.Bar: '%s'\n", cfg.Bar)
}
Currently if you use a file backend and the file is not found, the loader fails with
failed to open file at path "conf/xxx.toml": open conf/xxx.toml: The system cannot find the file specified.
It should either not fail by default, or there should be a way to specify that a file is optional.
Hi , I was looking for a config loading tool and this is exactly the one.
After several apps, we found that config file or other source is not always reliable.
In the case failed to retrievce config files, we have to manually set a default value for crucial fields.
Is it possible to have a default value written in tag?
Very apppreciate for it.
IMO we should specify in the doc that when a key is found in a backend, we won't continue to look for it in the remaining backends.
So there is an importance on how the backends are sorted inside the slice.
What do you think ?
We could add support for TOML files in the file backend.
Example:
file.NewBackend("config.tml")
last commit/pr was brought in coming up on 4 months ago... is this still maintained?
Given this type:
type migrationSettings struct {
Source string `config:"source,required"`
Destination string `config:"destination,required"`
Partitions int `config:"partitions,required"`
}
The following yaml:
- source: topic1
destination: topic2
partitions: 3
- source: topic1
destination: topic2
partitions: 3
- source: topic1
destination: topic2
partitions: 3
The snippet:
var s []migrationSettings
l := confita.NewLoader(file.NewBackend("config.yaml"))
l.Load(context.Background(), &s)
fmt.Println(s)
The output:
[]
Does confita support structured configuration? It would be useful to make configurations more clean/organized/structure, and for configurations of growing/large projects.
I have read the README.md, but there's no example for structure config,... And, I didn't find any TOML/YAML/JSON examples of structured configurations.
I would like to populate the following structure:
type Config struct {
// notifications configuration
Notifications struct {
// turns on user notifications for changes/updates/creations
Enabled bool
// from email address of emails
From string
// BCC addresses of every notification sent
BCC []string
}
// SMTP server/mail-sending configuration
SMTP struct {
// format server:port
ServerAddr string
// plain auth credentials
Username, Password string
// default email address to sent from
DefaultFrom string
}
}
For example, with TOML:
notifications.enabled = true
notifications.from = "[email protected]"
notifications.bcc = "[email protected]"
smtp.serveraddr = "smtp.gmail.com:587"
smtp.username = "[email protected]"
smtp.password = "secret"
smtp.defaultfrom = "[email protected]"
Or, with YAML:
notifications:
- enabled: true
- from: "[email protected]"
- bcc: "[email protected]"
smtp:
- serveraddr: "smtp.gmail.com:587"
- username: "[email protected]"
- password: "secret"
- defaultfrom: "[email protected]"
hi, guys.
I met some trouble with use confita. Anybody can help me?Please.
I have a nest struct:
type Database struct {
Which string `config:"which"`
DbDir string `config:"db_dir"`
}
type Config struct {
Database Database `config:"database,required"`
AppName string `config:"app_name,required"`
}
and then,when i use goconvey to test, it failed.
here is my code:
func createTempFile(name, content string) (string, func()) {
dir, err := ioutil.TempDir("", "testDir")
So(err, ShouldBeNil)
path := filepath.Join(dir, name)
f, err := os.Create(path)
So(err, ShouldBeNil)
_, err = fmt.Fprint(f, content)
So(err, ShouldBeNil)
So(f.Close(), ShouldBeNil)
return path, func() {
So(os.RemoveAll(dir), ShouldBeNil)
}
}
func TestLoadWithTempFile(t *testing.T) {
Convey("test Load with temp file", t, func() {
var cfg Config
path, cleanUp := createTempFile("app.yaml", `
app_name: blog
database:
which: sqlite3
db_dir: /tmp/test.db
`)
defer cleanUp()
loader := confita.NewLoader(file.NewBackend(path))
err := loader.Load(context.Background(), &cfg)
So(err, ShouldBeNil)
So(cfg.Database, ShouldNotBeNil)
So(cfg.AppName, ShouldEqual, "blog")
So(cfg.Database.Which, ShouldEqual, "sqlite3")
So(cfg.Database.DbDir, ShouldEqual, "/tmp/test.db")
})
}
and the test result:
....x.
Failures:
* /home/lks/go/src/github.com/crazypandas/goBlog/config/config_test.go
Line 46:
Expected: nil
Actual: 'required key 'app_name' for field 'AppName' not found'
6 total assertions
Is there anything wrong?
This bug is introduced since the last release.
Here how to reproduce:
type Config struct {
Host string `config:"host,required"`
}
func main() {
var cfg Config
l := confita.NewLoader(env.NewBackend(), flags.NewBackend())
_ = l.Load(context.Background(), &cfg)
}
When trying to execute the code above I have:
$> HOST=localhost go run main.go
required key 'host' for field 'Host' not found
exit status 1
Make the struct tag config
customizable per loader
We should have a backend that uses the flag
package to load conf from the command line.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.