caarlos0 / env Goto Github PK

I'm not sure if this is by design or a bug but if you specify a field as required and set an envDefault value, it returns error when trying to parse the environment.

The issue is this check here.
It checks if the env var exists but not the actual value of the variable so a non existing env var will cause it to return an error even though a value has been assigned.

If this is a bug I would be happy to make a PR to fix it.

My configuration can be defined either through ENV vars or via a JSON config file. I already have json tags in my struct, if I also add the 'env' tag I'd also be adding redundant info.

struct {
big bool json:"BIGBOOL" env:"BIGBOOL"
blah string json:"BLAHSTRING" env:"BLAHSTRING"
}

I'd rather just leave it as it was (w/out the env tag) and tell env to use the 'json' tag instead of the 'env' tag.

Of course there are fields where env doesn't support a json type (I assume). Well, that's on me to rectify by adding an 'env' tag but it would still save cluttering up the code with redundant information for simple structures.

Currently, when you either omit the env tag or specify a blank tag value the library will still attempt to fill the variable using a blank key. As experienced in #151 on some of my systems where there was a rouge blank key set by another tool this cause havoc with my configuration files.

I think it's safe to assume that a blank key "" will never be intentionally used by someone as a variable. and only leads to confusion when a variable with a blank key is accidentally set.

Ideally, we need some sort of key or tag check at https://github.com/caarlos0/env/blob/master/env.go#L232 that will ignore the variable from the environment if the key is blank.

Hi,

Do you think it would make sense to add a min/max validation option?
Beside of checking if the parameter a required or not, we could also check if it is in the range of specified numbers.

Do you need such PR?

when I try to use int64 , it says "Type is not supported"

I have a global var Dev, but i want to use it in submodule only when submodule's Config.Dev is not set, so i must use *bool, can you support this?
Thanks!

package main

import (
	"github.com/caarlos0/env"
	"github.com/golang/glog"
)

type Config struct {
	Dev *bool `env:"DEV"`
}

func main() {
	config := new(Config)
	err := env.Parse(config)
	glog.Errorln(config.Dev, err)
}

output:

<nil> Type is not supported

Hi !

Most recently started using your project for simplifying one of my largest cofiguration structures. Unfortunately there is no support for nested structures, yet.

My first idea would recursively checking the included structs.

type Config struct {
  
  // [...] root config

  Mysql struct {  
     Host string `env:"MYSQL_HOST"`
  }

  Redis struct {  
     Host string `env:"REDIS_HOST"`
  }
}

Alternatively, someone could attempt a prefix-esque style to nesting the elements.

type Config struct {
  
  // [...] root config

  Mysql struct {  
     Host string `env:"HOST"` // Should result in "MYSQL_HOST"
  } `env:"MYSQL" `
}

I can provide a pull request if you like, but wanted to ask first if something like this would fit into your concept.

Kind regards,
Jörn

I would like the library will support overriding part of the environment variables and the rest will be parsed from the real environment variables.
For example:

package main

import (
	"fmt"
	"github.com/caarlos0/env"
)

type Args struct {
	A string `env:"A"`
	B string `env:"B"`
}

func main() {
	args := &Args{}

	if err := env.Parse(args, env.Options{Environment: map[string]string{"A": "foo"}}); err != nil {
		panic(err)
	}
	fmt.Printf("%#v\n",args)
}

Running with: B=bar go run main.go

Will output: &main.Args{A:"foo", B:""}
I would expect it will output: &main.Args{A:"foo", B:"bar"}

So, the problem with aliasing the os.Unsetenv is that it didn't exist until go 1.4. Which means people using earlier versions of go can't use your library as-is.

If it were me, I'd just remove the aliases, but at least you should mention in the readme the min version required of Go.

This issue was generated by todo based on a TODO comment in ed82bdc when #70 was merged. cc @caarlos0.

Hi, I saw this library on the gophers slack and the one thing I think it's missing for me is the ability to support types that implement the TextUnmarshaler interface out of the box, like https://github.com/kelseyhightower/envconfig does.

Like the title says, it would be awesome to add a way to register cutom deserializers, o you could for example overwrite existing type deserializers (based ons truct texts) to support e.g. json, or for new (complex) types.

I was wondering if you could change this line ? so it doesnt use old code in github when you make a PR ?
https://github.com/caarlos0/env/blob/master/env_test.go#L9

I tried to change to
import env

but travis is not happy with it ...any idea what is the best way to make the PR pass the test?

Here's the error I'm seeing:

panic: reflect: reflect.Value.SetString using value obtained using unexported field

goroutine 1 [running]:
reflect.flag.mustBeAssignable(0x1b8)
	/usr/local/Cellar/go/1.11/libexec/src/reflect/value.go:231 +0x1ee
reflect.Value.SetString(0x1236e80, 0xc00009e1c0, 0x1b8, 0x12375cd, 0x4)
	/usr/local/Cellar/go/1.11/libexec/src/reflect/value.go:1551 +0x2b
github.com/caarlos0/env.set(0x1236e80, 0xc00009e1c0, 0x1b8, 0x1237592, 0x12, 0x120e897, 0x4, 0x12d95e0, 0x1236e80, 0x12375a6, ...)
	/Users/nick/go/src/github.com/caarlos0/env/env.go:161 +0xb55
github.com/caarlos0/env.doParse(0x126fba0, 0xc00009e1c0, 0x199, 0xc0000d3be0, 0xc00009e1c0, 0x199)
	/Users/nick/go/src/github.com/caarlos0/env/env.go:92 +0x2ce
github.com/caarlos0/env.Parse(0x122ba20, 0xc00009e1c0, 0x3, 0xc00009c3f0)
	/Users/nick/go/src/github.com/caarlos0/env/env.go:53 +0x1a1
main.main()
	/Users/nick/projects/tautulli-exporter/main.go:70 +0x5f
exit status 2

Here's my config struct:

type config struct {
	tautulliHttpMethod string `env:"TAUTULLI_HTTP_METHOD" envDefault:"http"`
	tautulliHost       string `env:"TAUTULLI_ADDRESS" envDefault:"127.0.0.1:8181"`
	tautulliApiKey     string `env:"TAUTULLI_API_KEY"`
	servePort          string `env:"SERVE_PORT" envDefault:"8282"`
}

Line 70 in my main.go is just a copy paste from the example in the README, so not sure what I'm doing wrong here...

I've removed some extraneous comments and posted the whole file here: https://gist.github.com/nwalke/db4825f01049afd95793f0fc15fb666d

Line 70 from the stack above is line 25 in that gist.

Any help would be much appreciated!

I noticed that if don't provide any of the required env vars, I only get an error about the first one. This would be great if I got errors about everything that went wrong.

Repro Code:

package main

import (
	"fmt"
	. "github.com/logrusorgru/aurora"
	"github.com/caarlos0/env"
)

type config struct {
	clusterUrl	string	`env:"K8S_CLUSTER_URL,required"`
	clusterCa	string	`env:"K8S_CLUSTER_CA,required"`
	clientKey	string  `env:"K8S_CLIENT_KEY,required"`
	clientCert	string	`env:"K8S_CLIENT_CERT,required"`
	namespace	string	`env:"K8S_NAMESPACE,required"`
}

func main() {
	cfg := config{}
	err := env.Parse(&cfg)
	if err != nil {
		fmt.Printf("%+v\n", err)
	}
	fmt.Printf("%+v\n", cfg)

	fmt.Println(Bold(Green("Done!")))
}

It's mandatory to use exported fields on the struct, which we need to parse. We should able to use unexported fields as well.

Steps to reproduce

• Use https://github.com/caarlos0/env#example
• Rename Home field of config struct to home (Line 11)
• run PRODUCTION=true HOSTS="host1:host2:host3" DURATION=1s go run main.go

Expected

• output {Home:/your/home Port:3000 IsProduction:true Hosts:[host1 host2 host3] Duration:1s}

Actual

panic: reflect: reflect.Value.Set using value obtained using unexported field

goroutine 1 [running]:
reflect.flag.mustBeAssignable(0x1b8)
        /usr/local/go/src/reflect/value.go:231 +0x1bd

Change default to envDefault or something like that, to avoid conflicts with other stuff

is anyone interested in helping me maintain this repo?

I have lots of OSS repos and this one needs more love from my part, but at the same time I don't have much free time lately... maybe someone is interested in helping?

Hi,
we are using your library a lot and it's awesome!

The only thing which is missing for us is handling secrets from a file. Personally I like the syntax used by curl tool which allows reusing the same flag for providing raw body or for the reading body from the file if the value has @ prefix.

Does it make sense to have similar feature in this package?

Not sure actually if it is desired default behavior, but it could be specified via additional tag:

type config struct {
	DatabaseUsername      string        `env:"DB_USER"`
	DatabasePassword      string        `env:"DB_PASSWORD", envFilePrefix:"@"`
}

package main

import (
	"fmt"
	"io/ioutil"
	"os"

	"github.com/caarlos0/env/v6"
)

type config struct {
	DatabaseUsername string `env:"DB_USER"`
	DatabasePassword string `env:"DB_PASSWORD", envFilePrefix:"@"`
}

func main() {
	cfg := config{}
	os.Setenv("DB_USER", "john_doe")

	{
		os.Setenv("DB_PASSWORD", "qwerty123")
		_ = env.Parse(&cfg)
		fmt.Printf("Values without file expansion:\t %+v\n", cfg)
		// Output:
		// Values without file expansion:	 {DatabaseUsername:john_doe DatabasePassword:qwerty123}
	}
	{
		os.Setenv("DB_PASSWORD", "@db_secrets.txt")

		p := []byte("asdfg456")
		_ = ioutil.WriteFile("db_secrets.txt", p, 0644)

		cfg := config{}
		_ = env.Parse(&cfg)

		fmt.Printf("Values with file expansion:\t %+v\n", cfg)
		// Output:
		// Values with file expansion:	 {DatabaseUsername:john_doe DatabasePassword:asdfg456}
	}
}

hi,

i had this struct working before pr#36:

type Config struct {
	MongodbURL   string `env:"MONGODB_URL" envDefault:"localhost:27017"`
	MongodbDb    string `env:"MONGODB_DB" envDefault:"ms-td-dispatch"`
	MqttBroker   string `env:"MQTT_BROKER" envDefault:"tcp://192.168.101.2:9601"`
	HTTPAddr     string `env:"HTTP_ADDR" envDefault:""`
	HTTPPort     string `env:"HTTP_PORT" envDefault:"9500"`
	LogLevel     string `env:"LOG_LEVEL" envDefault:""`
	LogFormatter string `env:"LOG_FORMATTER" envDefault:""`
	logger       *logrus.Logger
}

cfg := &Config{}
err := env.Parse(cfg)

but now it is failing with:
panic: reflect.Value.Interface: cannot return value obtained from unexported field or method

goroutine 1 [running]:
reflect.valueInterface(0x820920, 0xc42007c670, 0x1b6, 0x1, 0x1, 0x0)
/usr/lib/go-1.8/src/reflect/value.go:936 +0x1a5
reflect.Value.Interface(0x820920, 0xc42007c670, 0x1b6, 0x7, 0x820920)
/usr/lib/go-1.8/src/reflect/value.go:925 +0x44
github.com/caarlos0/env.doParse(0x802160, 0xc42007c600, 0x199, 0x802160, 0xc42007c600)
/home/juan/importante/proyectos/monorepo/go/src/github.com/caarlos0/env/env.go:49 +0x5ed
github.com/caarlos0/env.Parse(0x7b6000, 0xc42007c600, 0x20, 0xc4200cd160)
/home/juan/importante/proyectos/monorepo/go/src/github.com/caarlos0/env/env.go:40 +0xbd

perform like that

func TestParseWithPrefixNested(t *testing.T) {
	type config struct {
		Foo string `env:"FOO"`
		Nested struct{
			Doo string `env:"NESTED_DOO"`
		}
	}
	os.Setenv("APP_FOO","testval")
	os.Setenv("APP_NESTED_DOO","testval_nested")
	var conf config
	assert.NoError(t, ParseWithPrefix(&conf,"app"))
	assert.Equal(t, conf.Foo, "testval")
	assert.Equal(t, conf.Nested.Doo, "testval_nested")
	os.Clearenv()
}

Please support delegation.

package main

import (
	"fmt"
	"github.com/caarlos0/env"
)

type Foo struct {
	Food string        `env:"FOOD"`
}

type Bar struct {
	Baz string        `env:"BAZ"`
	Foo
}

func main() {
	bar := Bar{}
	err := env.Parse(&bar)
	if err != nil {
		fmt.Printf("%+v\n", err)
	}
	fmt.Printf("%+v\n", bar)
}

BAZ=1234 FOOD=4321 go run main.go

Is {Baz:1234 Foo:{Food:}}
Feature request {Baz:1234 Foo:{Food:4321}}

it would be nice to have a feature that adds in an env prefix for reusing structs.
for example lets say i have the following Connection Config object in a common module

type ServiceConnectionConfig struct {
     host `env:"HOST" envDefault:"localhost"`
}

and now i have a service config object that wants to use this config object -

type ServiceConfig struct {
    ServiceAHost  *common.ServiceConnectionConfig `envPrefix:"SERVICEA_"`
    ServiceBHost  *common.ServiceConnectionConfig `envPrefix:"SERVICEB_"`
}

The service that uses ServiceConfig will have to supply these environment variables:

• SERVICEA_HOST

• SERVICEB_HOST

requires further testing

The link on README is broken

http://carlosbecker.com/posts/env-structs-golang/

Thanks for making a great tool!

The following test breaks for me: I can't parse int32s. But int64 works. I'm on v3.5.0.

package main

import (
	"os"
	"testing"

	"github.com/caarlos0/env"
)

type A struct {
	A int32 `env:"A,required"`
}

func Test__A(t *testing.T) {
	os.Setenv("A", "12")
	config := A{}
	if err := env.Parse(&config); err != nil {
		t.Errorf("Could not unmarshal config from environment variables: %s", err)
	}
}

I do see that there are tests using int32, though. Am I simply using it wrong?

Anything I can do to help?

It looks like this commit broke the go.mod file:

0b4f1fc

Latest working version that I can update to and go mod tidy goes through is 5.1.1, 5.1.2 fails with following message:

go: github.com/caarlos0/[email protected]+incompatible: go.mod has post-v5 module path "github.com/caarlos0/env/v5" at revision v5.1.2

caarlos0/env is listed as github.com/caarlos0/env v5.1.1+incompatible in my go.mod file (upgrading from v4).

Custom parsers passed to ParseWithFuncs are not used when parsing pointers in the struct.

Hi
I read Postgres connection url from file, sometimes I get this error:

postgresql://postgres:postgres@postgres/scms?sslmode=disable\\n: invalid control character in URL

Bu I don't have any return or space character after this text in my file

Hi @caarlos0.
I'd like to propose a PR for supporting Duration Fields using ParseDuration method.
Do you have interest in this work?

Support a variant of Parse which takes a prefix to prepend to the environment variable names. In other words, something like Parse(&cfg, "myapp") would look for environment variables named "MYAPP_VAR1", "MYAPP_VAR2", etc.

This issue was generated by todo based on a TODO comment in 3f95503 when #73 was merged. cc @caarlos0.

I'm experiencing a problematic issue on Windows where fields in a struct that do not have the env tag are being set to "::=::" instead of being left alone.

Cloning the repository and running the tests also fails with the same issue:

--- FAIL: TestParsesEnv (0.00s)
    env_test.go:384:
                Error Trace:    env_test.go:384
                Error:          Should be empty, but was ::=::\
                Test:           TestParsesEnv

Hi, first of all: Awesome project, thank you for it.

I was wondering whether I could also load a byte slice from a file with your package.

type config struct {
	Certificate  []byte   `env:"CERTIFICATE,file" envDefault:"${CERTIFICATE_FILE}" envExpand:"true"`
        PrivateKey ed25519.PrivateKey `env:"PRIVKEY,file" envDefault:"${PRIVKEY_FILE}" envExpand:"true"`
}

Certificate is a byte slice
PrivateKey is a type alias to a byte slice (I just included it to ask, if this is also possible, if a byte slice would already work)

maybe we could remove testify, so we have a zero-deps module 🤔

This issue was generated by todo based on a TODO comment in ed82bdc when #70 was merged. cc @caarlos0.

I'm not sure if this is by design or a bug but if you specify a field as required and set env with empty value no errors returned
I have expected an error returned if the value is empty.

func main() {

	c := &struct {
		LOG string `env:"LOG,required"`
	}{}

	err := env.Parse(c)
				
	if err != nil {
		log.Fatal(err)
	}

	log.Printf("Log Level changed to %s", c.LOG)
}

go run main.go
2020/07/21 00:45:43 env: required environment variable "LOG" is not set
exit status 1 

LOG='' go run main.go
2020/07/21 00:46:10 Log Level changed to

related to.
https://github.com/caarlos0/env/blob/master/env.go#L226

In addition to the default types, please support parsing time.Time values

Just playing around with this using some example code, and I noticed this output:

env: required environment variable ""K8S_CLUSTER_URL"" is not set

notice the 2 sets of double quotes on each side. This seems like a bug.

Repro Code:

package main

import (
	"fmt"
	. "github.com/logrusorgru/aurora"
	"github.com/caarlos0/env"
)

type config struct {
	clusterUrl	string	`env:"K8S_CLUSTER_URL,required"`
	clusterCa	string	`env:"K8S_CLUSTER_CA,required"`
	clientKey	string  `env:"K8S_CLIENT_KEY,required"`
	clientCert	string	`env:"K8S_CLIENT_CERT,required"`
	namespace	string	`env:"K8S_NAMESPACE,required"`
}

func main() {
	cfg := config{}
	err := env.Parse(&cfg)
	if err != nil {
		fmt.Printf("%+v\n", err)
	}
	fmt.Printf("%+v\n", cfg)

	fmt.Println(Bold(Green("Done!")))
}

I have a Metadata type that is defined by pulling in values from different environment variables. There's one type on the struct that is of a custom Tag type which consists of a key and value.

(root) ~ # echo $ENV_TAGS
[{"foo1":"bar1"},{"foo2":"bar2"}]

type Metadata struct {
	ID string `env:"ENV_ID" json:"id,omitempty"`
	Tags             []Tag        `env:"ENV_TAGS" json:"tags,omitempty"`
}

type Tag struct {
	Key   string `json:"key,omitempty"`
	Value string `json:"value,omitempty"`
}

func (i *Metadata) Parse() error {
	var v string
	metadataParser := env.ParserFunc(parseMetadata)
	metadataType := reflect.TypeOf((*Metadata)(nil))
	customParsers := env.CustomParsers{}
	customParsers[instanceMetadataType] = instanceMetadataParser
	if err := env.ParseWithFuncs(&i, customParsers); err != nil {
		return err
	}
	return nil
}

func parseMetadata(metadataJSON string) (Metadata, error) {
	return Metadata{}, nil
}

Where metadataParser is defined, I'm getting the following error message: cannot convert ParseMetadata (value of type func(metadataJSON string) (Metadata, error)) to env.ParserFunc. Any help would be greatly appreciated! I need to get past the error message, and I simply need to use the custom parsers to set the environment variables to the proper fields and convert the map[string]string(s) of key-value pairs into a slice of Tags with the Metadata Type.

If I have a config struct of ENV vars and the fields are dependant it does not seem to work. i.e.

type envConfig struct {
	Mixin           string `env:"MIXIN" envDefault:"bar"`
	Package      string `env:"PKG" envDefault:"github.com/donmstewart/foo-${MIXIN}" envExpand:"true"`

Whilst the Mixinfield is correct after parsing i.e. bar the Package field becomes github.com/donmstewart/foo- rather than github.com/donmstewart/foo-bar in the situation where defaulting occurs.

It'd be useful to tag fields in the struct passed to env.Parse to indicate that they should be processed recursively. For example,

type config struct {
	Home string `env:"HOME"`
	Logging loggerConfig `envRecursive:"true"`
	Monitoring monitoring.Config `envRecursive:"true"`
}
type loggerConfig struct {
	Level string `env:"LOG_LEVEL`
}

This would make it easier to split up config options into smaller types (e.g., so that parts of the config can be shared across multiple apps).

Currently, the package only supports this if the field is unnamed, so I'd propose extending this behaviour to fields tagged envRecursive:"true" as well.

I'd be happy to implement this myself (or something similar, if you'd prefer a different solution) if you'd consider a PR

as of 4.0.0+, using go mods will get an v0.0.0 version for some reason.

I was using this package to set some fields of a struct, while other fields of the same struct were not controlled using environment variables.

This worked fine, until recently - unfortunately not sure exactly when the change was introduced, but now the following example main.go

package main

import (
	"github.com/caarlos0/env"
	"log"
	"net/http"
)

type Session struct {
	Token      string `env:"TOKEN"`
	HttpClient *http.Client
}

func main() {
	sess := Session{}
	err := env.Parse(&sess)
	if err != nil {
		log.Fatal(err)
	}
	sess.HttpClient = &http.Client{}
}

produces the output

$ go run main.go
2017/11/21 10:57:04 Expected a pointer to a Struct
exit status 1

but used to work just fine.

Now maybe it was coincidence that it worked just fine - maybe the package doesn't support structs that have fields that are not supposed to be handled by env vars.

But if it is supposed to be a feature (and I think at least I would like it to be ^^), then the above is probably a bug.

A workaround, btw, is to move the assignment before the parsing, as in

	sess := Session{}
	sess.HttpClient = &http.Client{}
	err := env.Parse(&sess)

This won't throw an error.

Noticed the import github.com/caarlos0/env/parsers in env.go and env_test.go:
"github.com/caarlos0/env/v5/parsers"
should it be:
"github.com/caarlos0/env/parsers" instead?

How to reproduce:

docker run -it --rm -v ${PWD}:/app -w /app golang:1.14 make test

output:

...
=== RUN   TestParseInvalidURL
    TestParseInvalidURL: env_test.go:984:
                Error Trace:    env_test.go:984
                Error:          Error message not equal:
                                expected: "env: parse error on field \"ExampleURL\" of type \"url.URL\": unable to parse URL: parse nope://s s/: invalid character \" \" in host name"
                                actual  : "env: parse error on field \"ExampleURL\" of type \"url.URL\": unable to parse URL: parse \"nope://s s/\": invalid character \" \" in host name"
                Test:           TestParseInvalidURL
--- FAIL: TestParseInvalidURL (0.00s)
=== RUN   ExampleParse
--- PASS: ExampleParse (0.00s)
=== RUN   ExampleParseWithFuncs
--- PASS: ExampleParseWithFuncs (0.00s)
FAIL
coverage: 85.9% of statements in ./...
FAIL    github.com/taygius/env  0.073s
FAIL
make: *** [Makefile:16: test] Error 1

Decrypt env vars directly when parsing

Would be nice to be able to decrypt env vars using an custom decryptor directly.
Perhaps with call to ParseWithDecrypt instead of just Parse. So that existing API doesn't change / break.

Please see PR #126 for feature.

Added simple example in PR as well as an example using AWS KMS in README.md.

I don't want to have to add PREFIX_ manually to each env definition.

I want to just set env.EnvPrefix = "PREFIX_" once.