hairyhenderson / gomplate Goto Github PK

Prior to vault 0.6.2, vault standby nodes would redirect HTTP to the active vault node. Current versions of the go http package do not seamlessly handle this redirection properly.

Attempting to use gomplate with pre-0.6.2 vault in HA configuration results in a panic, e.g.

echo -n '{{(datasource "vault" "secret/my_secret").value}}'  | gomplate -d vault="vault:///"
panic: runtime error: invalid memory address or nil pointer dereference [recovered]

🤦‍♂️ I forgot to document conv.ParseInt, conv.ParseFloat, conv.ParseUint, and conv.Atoi.

Reusing an http.Transport in the http.Client would allow reusing connections.

https://golang.org/src/net/http/transport.go

This would be useful for some datasource doing multiple queries like vault.

Also to keep in mind:

The http Client and Transport guarantee that Body is always non-nil, even on responses without a body or responses with a zero-length body. It is the caller's responsibility to close Body. The default HTTP client's Transport does not attempt to reuse HTTP/1.0 or HTTP/1.1 TCP connections ("keep-alive") unless the Body is read to completion and is closed. (https://golang.org/pkg/net/http/#Response)

This is perhaps overkill, but it'd be useful to resolve AWS EC2 metadata:

{{ec2meta "instance-id"}}

Or even to go so far as:

{{ec2tag "Name"}}

One thing I can't figure out is how to best get the region... Maybe as simple as {{ec2region}}

If you install gomplate via:

$ go get github.com/hairyhenderson/gomplate

and then run:

$ gomplate --version

It will output:

$ gomplate --version
gomplate version 0.0.0

Saying the current version is 0.0.0. I believe the latest relieve is 1.10.0?

I'm working with the CMD : gomplate -d zookeeper.json < zookeeper.tpl. How to use a json flow as input instead of a json file?

foo:
  bar:
    {{ `{"f1": {"f2": 2}}` | json | toYAML }}

Expected Result:

foo:
  bar:
    f1:
      f2: 2

Actual Result:

foo:
  bar:
    f1:
  f2: 2

If I run a few ec2* functions on my laptop, usually it times out in ~5s and then subsequent calls are fast. But for some reason there are some environments where this doesn't happen, for example CircleCI, which does run in EC2, but they clearly restrict access to the EC2 APIs.

For example, running in a CircleCI debug session:

$  time (echo '{{ec2region}}{{ec2region}}{{ec2region}}' | ./gomplate)
unknownunknownunknown

real    0m15.011s
user    0m0.007s
sys 0m0.004s
$ time (echo '{{ec2tag "foo"}}' | ./gomplate)

real    0m10.279s
user    0m0.010s
sys 0m0.006s

I think the right approach is to do 2 things:

1. make an initial call to the EC2 Instance Metadata API, and track success - short-circuit further ec2 functions to use defaults after that (#60)
2. cache EC2 API calls to the same URLs so that we don't needlessly make repeated calls

This is ok:

$ echo '{{ec2region}}' | gomplate
unknown

But this would be more useful:

$ echo '{{ec2region "myregion"}}' | gomplate
myregion

Would be awesome to provide base64 encoding and decoding.
" var | base64d "

Example:

$ wget -q -O bin/gomplate https://github.com/hairyhenderson/gomplate/releases/download/v0.2.1/gomplate_darwin-amd64
$ ./bin/gomplate -v
0.1.0

While datasource is useful, it only supports structured data. There are some cases where I need to just include the text from some source without worrying about whether it's structured data.

At first I'd thought of just enhancing datasource so that it falls back to pulling plain text for content-types it doesn't recognize, but it's possible that I could want to include a JSON file (or some other structured data) unparsed into the template.

So what I'm thinking is maybe a new function include that would support the same kinds of URLs as datasource, but just reads them and doesn't worry about parsing structured data.

The questions I'm pondering are:

• am I overthinking this? does this just belong in datasource?
  • there's a compatibility risk here - if I read an XML file (for example) and get its raw text in one version of gomplate, but then add XML support later, datasource will stop returning a string and start returning a map - this would be bad!
• would it make sense to take the same approach as datasource in how datasources are referenced (i.e. on the commandline with -d), or would it make more sense to in-line URLs
  • adding all manner of commandline arguments gets messy, but on the other hand hard-coding URLs in a template is undesirable. Perhaps there's a middle ground...

🤔

When using the has function to check if a key exists on a nested map, gomplate will panic:

$ cat template.yaml
hello: {{- if has (datasource "default").data "word" -}}
{{- (datasource "default").data.word -}}
{{- else -}}
bob
{{- end -}}

$ cat config.yaml
data:
  word: alice

$ gomplate -f template.yaml --datasource default=config.yaml
hello:panic: template: template:1:30: executing "template" at <"default">: wrong type for value; expected map[string]interface {}; got map[interface {}]interface {}

goroutine 1 [running]:
main.(*Gomplate).RunTemplate(0xc42022a060, 0xc420090f80, 0x80, 0x1a7ba60, 0xc42000e018)
	/Users/dee/gocode/src/github.com/hairyhenderson/gomplate/gomplate.go:33 +0x271
main.renderTemplate(0xc42022a060, 0xc420090f80, 0x80, 0x16c5313, 0x1, 0x0, 0x0)
	/Users/dee/gocode/src/github.com/hairyhenderson/gomplate/gomplate.go:99 +0xc1
main.processInputFiles(0x0, 0x0, 0xc420213050, 0x1, 0x1, 0xc420212fe0, 0x1, 0x1, 0xc42022a060, 0x16c5c1e, ...)
	/Users/dee/gocode/src/github.com/hairyhenderson/gomplate/process.go:23 +0x12f
main.runTemplate(0x1aae800, 0x0, 0x0)
	/Users/dee/gocode/src/github.com/hairyhenderson/gomplate/gomplate.go:88 +0x18f
main.newGomplateCmd.func1(0xc420019200, 0xc4201b9100, 0x0, 0x4, 0x0, 0x0)
	/Users/dee/gocode/src/github.com/hairyhenderson/gomplate/main.go:68 +0x7e
github.com/hairyhenderson/gomplate/vendor/github.com/spf13/cobra.(*Command).execute(0xc420019200, 0xc4200101a0, 0x4, 0x4, 0xc420019200, 0xc4200101a0)
	/Users/dee/gocode/src/github.com/hairyhenderson/gomplate/vendor/github.com/spf13/cobra/command.go:620 +0x3f8
github.com/hairyhenderson/gomplate/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc420019200, 0x2, 0x16c54c5, 0x2)
	/Users/dee/gocode/src/github.com/hairyhenderson/gomplate/vendor/github.com/spf13/cobra/command.go:699 +0x339
github.com/hairyhenderson/gomplate/vendor/github.com/spf13/cobra.(*Command).Execute(0xc420019200, 0xc4200f7f78, 0x0)
	/Users/dee/gocode/src/github.com/hairyhenderson/gomplate/vendor/github.com/spf13/cobra/command.go:658 +0x2b
main.main()
	/Users/dee/gocode/src/github.com/hairyhenderson/gomplate/main.go:96 +0x46

Checking for key in top-level map works as expected:

$ cat template.yaml
hello: {{- if has (datasource "default") "word" -}}
{{- (datasource "default").word -}}
{{- else -}}
bob
{{- end -}}

$ cat config.yaml
word: alice

$ gomplate -f template.yaml --datasource default=config.yaml
hello:alice

With new Docker Swarm Secrets, and when running gomplate inside a container running in a Swarm service, it'd be more secure for gomplate to be able to read secrets from files instead of environment variables. See https://docs.docker.com/engine/swarm/secrets/#build-support-for-docker-secrets-into-your-images

I'm thinking of pretty much everything passed through an environment variable, but especially:

• VAULT_APP_ID
• VAULT_USER_ID
• VAULT_AUTH_GITHUB_TOKEN
• VAULT_AUTH_USERNAME
• VAULT_AUTH_PASSWORD
• VAULT_ROLE_ID
• VAULT_SECRET_ID
• VAULT_TOKEN - even though ~/.vault-token is already read by default, this one is still valuable since swarm secrets can only be exposed at /run/secrets (because they're in a tmpfs in-memory filesystem)

Basically each one of these would have an equivalent *_FILE version that would point to a file to be read.

It'd be useful to support YAML files on the datasource function

gomplate seems to be quite big. Installing it via apk it takes 17.7M space. Is there a way to reduce it's size?

I'm using gomplate to generate the configurations of docker containers in a project I keep named Daspanel. So far I've used the datasource coming from a file with success, but now I need to get data from an API using authentication through a token that has to be passed using an HTTP header. An example of calling the API using curl is below:

curl -X GET --header 'Accept: application/json' --header 'Authorization: myauthtoken' 'http://api.daspanel.site/1.0/sites/httpconf/mydomain.com'

I looked in the documentation but found no example of how to pass a header when using the HTTP/HTTPS datasource. Is this possible ? If not, could you consider adding this feature to gomplate?

With this feature gomplate can directly consume json data from APIs that use token or JWT authentication.

Thanks a lot for this neat tool, really like the abstraction between input, datasources and output.

It would be awesome if gomplate would support bulk operations directly. E.g. when an argument --input-dir is given, then process all files (recursively) and store it to an --output-dir (maintaining the directory structure). With a --datasource-dir one could provide an directory for the datasources, where each file in this directory is used a datasource, where the basename is taken as the name of the datasource.

A use case for this is processing a hierarchy of configuration files whithout the need of iterating over each file.

Here you can find an external solution which loops over the files via a shell script. would be awesome if this could be performed from within gomplate so that one could even use scratch as base image.

Using .Env within range produces the following error:

panic: template: template:19:25: executing "template" at <.Env>: map has no entry for key "Env"

Example:

{{ range (datasource "gomplate").foobar }}
{{ .Env.FOOBAR }}
{{ end }}

I would expect that non-flag arguments would raise an error in gomplate.

Expected:

$ gomplate -in flibbit
I don't know what to do with 'flibbit'
# non-zero exit code
$

Actual:

$ gomplate -in flibbit
n<no newline> 
# zero exit code
$

I got bit by this because I typed -in instead of --in and spent a while trying to figure out what I did wrong.

FYI: -in can be rewritten as --in 'n'

The AWS functions (ec2tag, ec2meta, etc...) add a lot of size to the binary (something like 13MBs, though less when UPX-compressed), can slow things down, and aren't always used...

A whole plugin architecture may be overkill for this project, but some way of lessening the impact would be useful.

Alternatively a much lighter-weight client for the EC2 API calls would help.

Hi,
Just wondering if it's possible to use this as a sidekick process to trigger a HUP signal for instance once a template has been generated?
Or is there another way to do it?

I suspect the AWS SDK, because it has a 5 second timeout. This happens even when none of the EC2 functions are being used.

$ echo 'hello world' | time ./gomplate
./gomplate  0.01s user 0.01s system 0% cpu 5.017 total

$ echo 'hello world' | time ./gomplate
./gomplate  0.01s user 0.01s system 91% cpu 0.014 total

The default text/template delimiters {{ and }} are OK, but sometimes you need to generate a template or something crazy like that... This just means exposing some environment variable or command-line opt (how easy do I want to make it?) to https://golang.org/pkg/text/template/#Template.Delims

gomplate version 1.2.2 with vault datasources will panic in VAULT_ADDR is unset or if other auth strategy failures occur.

Example:

unset VAULT_ADDR
echo -n '{{(datasource "vault" "secret/foo").value}}' | /usr/local/Cellar/gomplate/1.2.2/bin/gomplate -d vault="vault://"

Results in a panic:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
       	panic: runtime error: invalid memory address or nil pointer dereference [recovered]
       	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xf0f0a]

Would be pretty useful to support HTTP(S) URLs with the datasource function.

It'd be cool to use the https datasource to grab secrets from HashiCorp's Vault.

A bunch of auth backends exist, but I'd like at least the following to work:

• Username & Password - #104
• AppID - #74
• AppRole - #105
• GitHub - #80
• Token - #74

The user interface could look something like:

$ gomplate --vault-app-id=abcd1234 --vault-user-id=defg4567 \
    --datasource vault=https://my.vault.server:8200/v1/secret/supersecret \
    "my secret is: {{ (datasource "vault").data.value }}"
my secret is: supersecretvalue!

From #162

It'd be useful to be able to call indent like so:

$ gomplate -i '{{ indent 4 "foo" }}'
    foo

This obviously assumes that the indent string is " " (a single space), and it needs to be possible to specify other indent strings. For this, a 3-arg form would be handy:

$ gomplate -i '{{ indent 2 "-" "foo" }}'
--foo

This implies that the default indent string would be " ".

The README's already ridiculously long (~900 lines), and I'm not happy with the level of detail in the built-in function docs (for example, it doesn't list function parameters in any sort of consistent way).

I think at least splitting the docs out of the README is a prerequisite to being able to start adding large numbers of functions (such as is being discussed in #145). Probably for now, a simple static site would be the best option. Just moving the relevant Markdown from the README would be a good first step.

I think it'd be useful to be able to feed a CSV to the datasource function... Not exactly sure how it'd work though.

Just a wrapper for LookupIP would probably do...

The use case would be resolving hostnames to IP addresses. In the case of multiple addresses, we could just return the first one. One open question would be what to do with IPv4 vs IPv6 - maybe we make this a v4-only function, but have a v6 variant?

Alternatively, maybe go-sockaddr has something to offer for this? (see #145)

As suggested by @sean- originally in #145 (comment):

Have you thought about breaking gomplate into a sub-package like github.com/hairyhenderson/gomplate/cmd/gomplate so that the "gomplate" template library could be the meta-library used to aggregate a bunch of different APIs and helper functions? It's pretty easy for me to see that this would be useful to embed in standalone programs. "Dear gomplate, please render my string." Either gomplate(1) or the library github.com/hairyhenderson/gomplate.

🤔

Would support for substitutions based on values stored in the key/value store within Consul be considered (optionally with authentication via Vault)?

Good day!

I kindly ask you to think about feature, that will help many people use this excellent tool for big projects with huge count of separated configuration files: load in one alias all supported files in specified dir.

I mean will be a great idea to have something like this --datasource base=base.yaml conf=path/to/prod/dir/ where in path/to/prod/dir/ there are configuration files like servers.yaml, connections.toml and users.csv (Just for examples)

Running gomplate in a windows docker container in an Win2016 EC2 instance (gomplate_windows-amd64-slim.exe)

In such context, about one in 5 or 10 calls to EC2 service takes 1s or +. In such case, gomplate doesn't wait enough. See last output below.
PS C:> echo '{{aws.EC2Region "unknown" -}}' | ./gomplate
us-east-1
PS C:> echo '{{aws.EC2Region "unknown" -}}' | ./gomplate
us-east-1
PS C:> echo '{{aws.EC2Region "unknown" -}}' | ./gomplate
us-east-1
PS C:> echo '{{aws.EC2Region "unknown" -}}' | ./gomplate
unknown

Should it wait more in the line below?
https://github.com/hairyhenderson/gomplate/blob/master/aws/ec2info.go#L45

The equivalent call does work every time - but it does occasionally take 1s+
(Invoke-WebRequest 'http://169.254.169.254/latest/dynamic/instance-identity/document').Content

Why it sometimes takes 1s+ is probably due to running inside a container and having to add this rule (windows docker containers do not have access to Amazon service by default)
[string] $gatewayIpAddress = (Get-NetIPConfiguration -All).IPv4DefaultGateway.NextHop
route add 169.254.169.254 $gatewayIpAddress

This is vaguely related to #134.

toJSON behaves badly:

$ gomplate -i '{{ "foo:\n  bar: baz\n" | yaml | toJSON }}'
2017/05/11 21:46:30 Unable to marshal object map[foo:map[bar:baz]]: json: unsupported type: map[interface {}]interface {}

However, toYAML doesn't:

$ gomplate -i '{{ "foo:\n  bar: baz\n" | yaml | toYAML }}'
foo:
  bar: baz

I think the problem is documented in the docs for json.Marshal

Map values encode as JSON objects. The map's key type must either be a string, an integer type, or implement encoding.TextMarshaler.

Obviously gpkg.in/yaml.v2 doesn't have this problem...

🤔

So uh... this happens:

$ echo '{{ec2dynamic "instance-identity/document" "unknown"}}' | ./gomplate
2016/03/27 22:40:39 Failed to GET from http://169.254.169.254/latest/dynamic/instance-identity/document: Get http://169.254.169.254/latest/dynamic/instance-identity/document: dial tcp 169.254.169.254:80: i/o timeout

I would've expected this:

$ echo '{{ec2dynamic "instance-identity/document" "unknown"}}' | ./gomplate_darwin-amd64 
unknown

It would be awesome if Kubernetes ConfigMaps could be used as datasources. This would be a perfect fit when gomplate is used as an init-container which can be used to prepare configuration files for an application (like a standalone.xml for Wildfly app server).

Currently this can be done via ConfigMap backed volumes but a direct K8s support would be nicer, of course as it reduces the configuration overhead within Kubernetes.

This is an issues to discuss the acceptability and design for a large feature which I think would be useful in gomplate - the ability to execute go-sockaddr pipelines.

The go-sockaddr library was written primarily by @sean- for use in HashiCorp runtime tools to allow for evaluation of network interfaces against heuristics to avoid hard-coding interface names, IP addresses and so forth during configuration. Consul and Nomad (and I believe soon Vault) all support the templates in configuration fields such as bind.

Templates take the form of a Go function pipeline, and provide functionality such as finding an IP address that:

• is attached to a default route (GetDefaultInterfaces())
• is contained within a CIDR block (IfByNetwork())
• is an RFC1918 address (IfByRFC("1918"))
• is ordered (OrderedIfAddrBy(args) where args includes AscIfType, AscNetworkSize)
• excludes all IPv6 addresses (IfByType("^(IPv4)$"))
• is larger than a /32 (IfByMaskSize(32))
• is not on a down interface (ExcludeIfs("flags", "down"))
• preferences an IPv6 address over an IPv4 address (SortIfByType() + ReverseIfAddrs()); and
• excludes any IP in RFC6890 address (IfByRFC("6890"))

I believe this would be a fairly straightforward lift-and-shift to support all of the functions supported by go-sockaddr in gomplate, and it would make rendering templates with networking aspects more flexible. Two possible approaches seem to exist:

1. Add the go-sockaddr-supplied functions to gomplate directly
2. Add a function named something like sockaddr which takes a template to render as a parameter.

I'm happy to take a look at doing this work, but want to confirm that the feature set is of interest prior to doing so.

Let's start with JSON support (because encoding/json). I figure usage like this would be useful:

data.json

{
  "name": {
    "first": "Joe"
  }
}

in.txt

Hello {{.data.name.first}}!

(where data comes from the name of the file)

$ gomplate -f data.json < in.txt
Hello Joe!

It'd be useful to have a sort of sed capability inside templates... Probably some simple wrapping of regexp.ReplaceAllString.

The usage should be something like regexp.Replace "expression" "replacement" "input":

$ gomplate -i '{{ "1.2.3-59" | regexp.Replace `-([0-9]*)` `.$1` }}'
1.2.3.59

I'm not sure about the function name though. replaceAll is taken and doesn't handle regexes - maybe we could just modify replaceAll to support regexp instead, though that would certainly break some users. I'm not sure I like regex.Replace, but... 🤔

See #149 (comment) for the genesis of this conversation.

The ideal would be to be able to use some of the same Markdown for both the doc site (gomplate.hairyhenderson.ca) and the man page. A few tools exist for this, looks like ronn(1) is a good option.

Once the man page generation is figured out, it needs to be packaged. It'd need to be packaged with both the Alpine and Homebrew packages (and any new packages...).

This is a bit meta, but I have template files generating config files containing gomplate functions in them :)

Command:

gomplate < template.tmpl

Broken template:

VERSION=$(gomplate -d package.json -i '{{ (ds "package").version }}'

Working template (Workaround):

VERSION=$(gomplate -d package.json -i '{{"{{"}} (ds "package").version {{"}}"}}'

Propositions:

VERSION=$(gomplate -d package.json -i '{{ gomplate "(ds \"package\").version" }}'

or

VERSION=$(gomplate -d package.json -i '{{ gomplate `(ds "package").version` }}'

Edit 1: Adding gomplate command using the templates

Currently, when a datasource is specified in the template, but not on the CLI, gomplate crashes with a nil dereference error, which can easily be reproduced with:

echo '{{(datasource "test")}}' | ./gomplate

Tested with gomplate version 1.2.4 on Ubuntu 14.04.

In my opinion it shouldn't crash like that in the first place, but it would also be nice if there was a possibility to test within the template if specific datasources are specified/available before trying to access them.

Right now I solved this by wrapping gomplate and assigning a dummy file/source externally - but that's not really a clean solution in my opinion.

It'd be useful to set defaults when environment variables can't be resolved

This syntax might work easily:

{{getenv "FOO" "default_value"}}

The following will not pass YAML lint at the moment:

foo:
  bar: {{ `["foo", "bar"]` | json | toJSON }}

Expected Result:

foo:
  bar: ["foo", "bar"]

Actual Result:

foo:
  bar: ["foo","bar"]

For the Ec2Meta group of template functions, there's a 500ms timeout on the HTTP client: https://github.com/hairyhenderson/gomplate/blob/master/aws/ec2meta.go#L56, but for the Ec2Info functions (i.e. the ones that use aws-sdk-go), no such timeout is set, and I've been observing up to 20s timeouts.

For example:

$ docker run -it --rm --entrypoint '' hairyhenderson/gomplate bash
root@dde9d13c242b:/go/src/app# echo '{{ec2region}}' > r.json
root@dde9d13c242b:/go/src/app# echo '{{ec2tag "foo"}}' > t.json
root@dde9d13c242b:/go/src/app# time gomplate < r.json 
unknown

real	0m0.511s
user	0m0.000s
sys	0m0.010s
root@dde9d13c242b:/go/src/app# time gomplate < t.json 


real	0m20.901s
user	0m0.010s
sys	0m0.000s

Note that this doesn't always happen -- when I run on my mac, ec2tag times out quickly:

$ echo '{{ec2region}}' | time gomplate
unknown
gomplate  0.01s user 0.00s system 2% cpu 0.518 total
$ echo '{{ec2tag "foo"}}' | time gomplate

gomplate  0.01s user 0.01s system 1% cpu 0.897 total

Note that this is still longer than 500ms...

Something missing in the string functions is to be able to do a traditional join to make a string out of a list of items with a defined delimiter character.

nameservers:
  - "8.8.8.8"
  - "8.8.4.4"

Template:

{{ (datasource "config").nameservers | join " " }}

output => "8.8.8.8 8.8.4.4"

Example failure using gomplate inside a linux docker container:

nobody@5f067e23aa9a:/srv$ gomplate -v
gomplate version 1.2.1
nobody@5f067e23aa9a:/srv$ echo -n "{{(datasource \"vault\" \"secret\")}}" | gomplate -d vault="vault://"
2016/11/19 22:58:14 user: Current not implemented on linux/amd64

This looks to be due to os/user having a dependency on libc which is broken if CGO_ENABLED=0 when gomplate was built (see golang/go#11797)

vault/token_strategy.go:54:     u, err := user.Current()
vault/token_strategy_test.go:26:        u, err := user.Current()

Would be interesting to run https://github.com/dvyukov/go-fuzz on gomplate somehow... 🤔