Giter Club home page Giter Club logo

envplate's Introduction

Envplate

Release Build Status Documentation Go Report Card

Trivial templating for configuration files using environment keys. References to such keys are declared in arbitrary config files either as:

  1. ${key} or
  • ${key:-default value}

Envplate (ep) parses arbitrary configuration files (using glob patterns) and replaces all references with values from the environment or with default values (if given). These values replace the keys inline (= the files will be changed).

Failure to resolve the supplied glob pattern(s) to at least one file results in an error.

Optionally, ep can:

  • backup (-b flag): create backups of the files it changes, appending a .bak extension to backup copies
  • dry-run (-d flag): output to stdout instead of replacing values inline
  • strict (-s flag): refuse to fallback to default values
  • verbose (-v flag): be verbose about it's operations
  • charset (-c flag): specify a custom charset to replace variables. Example: /usr/local/bin/ep -c iso8859-1 *.conf

ep can also exec() another command by passing

  • -- after all arguments to ep
  • the path to the binary and it's arguments

Example: /usr/local/bin/ep -v *.conf -- /usr/sbin/nginx -c /etc/nginx.conf

This can be used to use ep to parse configs and execute the container process using Dockers CMD

Escaping

In case the file you want to modify already uses the pattern envplate is searching for ( e.g. for reading environment variables ) you can escape the sequence by adding a leading backslash \. It's also possible to escape a leading backslash by adding an additional backslash. Basically a sequence with an even number of leading backslashes will be parsed, is the number of leading backslashes odd the sequence will be escaped.

See https://github.com/kreuzwerker/envplate#full-example

Why?

For apps running Docker which rely (fully or partially) on configuration files instead of being purely configured through environment variables.

You can directly download envplate binaries into your Dockerfile using Github releases like this:

RUN wget -q https://github.com/kreuzwerker/envplate/releases/download/v1.0.2/envplate_1.0.2_$(uname -s)_$(uname -m).tar.gz -O - | tar xz && mv envplate /usr/local/bin/ep && chmod +x /usr/local/bin/ep

...

CMD [ "/usr/local/bin/ep", "-v", "/etc/nginx/nginx.conf", "--", "/usr/sbin/nginx", "-c", "/etc/nginx/nginx.conf" ]

Full example

$ cat /etc/foo.conf
Database=${FOO_DATABASE}
DatabaseSlave=${BAR_DATABASE:-db2.example.com}
Mode=fancy
Escaped1=\${FOO_DATABASE}
NotEscaped1=\\${FOO_DATABASE}
Escaped2=\\\${BAR_DATABASE:-db2.example.com}
NotEscaped2=\\\\${BAR_DATABASE:-db2.example.com}

$ export FOO_DATABASE=db.example.com

$ ep /etc/f*.conf

$ cat /etc/foo.conf
Database=db.example.com
DatabaseSlave=db2.example.com
Mode=fancy
Escaped1=${FOO_DATABASE}
NotEscaped1=\db.example.com
Escaped2=\${BAR_DATABASE:-db2.example.com}
NotEscaped2=\\db2.example.com

Sample docker file

FROM nginx:latest
MAINTAINER Albert van t Hart <[email protected]>

ADD https://github.com/kreuzwerker/envplate/releases/download/v0.0.7/ep-linux /bin/ep
RUN chmod +x /bin/ep

EXPOSE 80 443

CMD [ "/bin/ep", "-v", "/etc/nginx/*.conf", "--", "/usr/sbin/nginx", "-g", "daemon off;" ]

Source: https://github.com/avthart/docker-nginx-env

envplate's People

Contributors

butjar avatar enc avatar hasnat avatar mavogel avatar neuroid avatar testwill avatar vladimir-mencl-eresearch avatar yawn avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

envplate's Issues

Cleanup

  • Move config state to type, dropping test setup on the way
  • Return on LogError instead of Log(ERROR)

Recursive substitution?

I would like to put substitutions on the right hand side of the :, or a better solution would be great too.

Expected Behavior

tunnel_host = "${SERVER_TUNNEL_HOST:-${SERVER_URL:-http://$(hostname)}}"
would render in this precedence:

  1. SERVER_TUNNEL_HOST
  2. SERVER_URL
  3. http://hostname

Actual Behavior

tunnel_host = "${SERVER_URL:-http://$(hostname)}"

Steps to Reproduce (including precondition)

SERVER_TUNNEL_HOST is unset
SERVER_URL=http://my.server.com
tunnel_host = "${SERVER_TUNNEL_HOST:-${SERVER_URL:-http://$(hostname)}}"

renders
tunnel_host = "${SERVER_URL:-http://$(hostname)}"

Your Environment

  • OS: docker.io/debian:11
  • envplate version: 1.0.2

Another scenario

auth = "${SERVER_AUTH_USER:?${SERVER_AUTH_PASSWORD:?${SERVER_AUTH_USER}:${SERVER_AUTH_PASSWORD}}}"
This should verify that SERVER_AUTH_USER & SERVER_AUTH_PASSWORD are both set, then set the line to the value of:
auth = "${SERVER_AUTH_USER}:${SERVER_AUTH_PASSWORD"

Don't expand $var placeholders

The current implementation is based on os.Expand which replaces ${var} and $var strings. This makes it difficult to handle files like Bash scripts. Replacement should be limited to one pattern preferably ${var}.

ENV as default value

I didn't test this but I like to know if something like this is/would/should be supported:

${ENV_NAME:-$ENV_FALLBACK}

Would be nice to know with this great piece of software; thanks!

Add output file option?

Hello Kreuzwerker team,

First of all thanks for the nice and simple tool! :)

I want to ask you guys for your opinion on something:
I'm using envplate for some of my personal projects and for one of them I use one Docker image which I volume mount a different config in depending on what I want to use the container for. This config contains a couple of variables that I want to template using envplate. Each of the configs gets used multiple times but with different values.
Since I mount these into the container running envplate on them changes the actual file on the host :P

I'm wondering what you guys think about this and if it would a reasonable addition to envplate.
Obvious workaround is to volume mount to a different filename and do a cp first (or to create multiple images with the config built into them)

Cannot match release with uname -m with Arm based Docker image

Expected Behavior

Matching Envplate release with uname -m does not work in Dockerfile for M1 (arm), please see:

curl -LJ --silent https://github.com/kreuzwerker/envplate/releases/download/v1.0.3/envplate_1.0.3_Linux_$(uname -m).tar.gz -o envplate_aarch64.tar.gz

I use this to go around:

#
# Envplate
#
FROM alpine AS envplate-builder

ARG REPO=https://github.com/kreuzwerker/envplate/releases/download
ARG VERSION=1.0.3

WORKDIR /tmp
RUN apk add curl

RUN <<EOF
curl -LJ --silent ${REPO}/v${VERSION}/envplate_${VERSION}_Linux_arm64.tar.gz -o envplate_aarch64.tar.gz
curl -LJ --silent ${REPO}/v${VERSION}/envplate_${VERSION}_Linux_x86_64.tar.gz -o envplate_x86_64.tar.gz
tar -xf envplate_$(uname -m).tar.gz
chmod +x envplate
EOF

Actual Behavior

  • uname -m will give aarch64 instead and thus cannot match to arm64 in release filename.

Steps to Reproduce (including precondition)

Try building Docker image with this Dockerfile in M1/M2 Mac (Silicon/ARM processor):

FROM alpine

RUN apk add curl
RUN curl -LJ --silent https://github.com/kreuzwerker/envplate/releases/download/v1.0.3/envplate_1.0.3_Linux_$(uname -m).tar.gz

Your Environment

  • OS: Alpine 3.18
  • envplate version: 1.0.2, 1.0.3

Empty string not a valid value

Feature Request
Given:
export PREFIX=

config:
my_prefix="${PREFIX}"

or even:
my_prefix="${PREFIX:-}"

expect:
my_prefix=""

actual:
requires undeclared environment variable 'FUNCTION_PREFIX', no default is given
requires undeclared environment variable 'FUNCTION_PREFIX:-', no default is given

Request: When a empty default is specifically given allow it, so allow ${PREFIX:-}, or another way of having no value

doubled binary size

The latest RC binary release is about twice as large as the historical releases. Is this a debug build?

It's only a couple megabytes more, but I'm curious if you know why the size jumped so much.

Derivate a github action from this

We may derive a github action from this.
There is https://github.com/microsoft/variable-substitution which has a similar purpose, but requires to specify the keys, which blows up the workflow file.

Example how to do that:
https://full-stack.blend.com/how-we-write-github-actions-in-go.html

Expected Behavior

  • Setup a new repo from which the action can be served ( Github market place )
  • create static bianaries
  • Have a trigger that updates and releases the action when a new release is made here

This issue is mainly a remainder for myself.

Please support reading from stdin

Expected Behavior

Expected: This should print the current value of $HOME on stdout:

echo "\${HOME}" |  go run github.com/kreuzwerker/envplate/bin@latest

Expected: This should fail:

echo "\${var-does-not-exist}" |  go run github.com/kreuzwerker/envplate/bin@latest -s

Actual Behavior

Up to not the code does not read from stdin

Your Environment

  • OS: Ubuntu 22.04
  • envplate version: latest

Related tools

This tool reads from stdin, but has no strict mode: https://github.com/drone/envsubst

This tools supports reading from stdin and has a strict-mode via -u: https://github.com/icy/genvsub

License?

What license is this code under?

ep doesn't work without -v option

$  es dec --region ap-southeast-1 -- `which ep` *.properties || echo failed
2016/12/23 16:02:30 [ ERROR ] Zero files matched passed globs '[]'
failed
$  es dec --region ap-southeast-1 -- `which ep` -v *.properties || echo failed
$ 

Escaping issues in 0.6

Example:

                <Context path="" docBase="\${catalina.home}/atlassian-jira" reloadable="false" useHttpOnly="true">

Better repository topics

Hello together,

you should include "envsubst" as a topic next to "golang" and "hacktoberfest".
Then it will be easier to find this tool on GitHub.

Best regards
Felix

Allow empty default values

Hi,

This is a follow-up to #15. The fix for #15 allows specifying empty values in environment variables.

I think it would be neat if envPlate also allowed specifying empty default value, like:

${EMPTY_VARIABLE:-}

Right now, the parsing misbehaves, and envPlate reports:

2016/01/21 03:20:32 [ ERROR ] 'test.txt' requires undeclared environment variable 'EMPTY_VARIABLE:-', no default is given

I.e., the :- becomes part of the variable name.

The part of the regexp parsing the default value has: (.+?). I.e., it says "any character, at least once, all of that at least zero times". Turns out, go regexp decides empty string does match this pattern and the :- becomes part of the variable name.

I can get around this by removing the +, i.e.:

diff --git a/envplate.go b/envplate.go
index b2da37d..b81186d 100644
--- a/envplate.go
+++ b/envplate.go
@@ -15,7 +15,7 @@ const (
    NotAnEscapeSequence = ""
 )

-var exp = regexp.MustCompile(`(\\*)\$\{(.+?)(?:\:\-(.+?))?\}`)
+var exp = regexp.MustCompile(`(\\*)\$\{(.+?)(?:\:\-(.?))?\}`)

 func Apply(globs []string) {

However, that still makes envPlate fail with:

2016/01/21 03:35:07 [ ERROR ] 'test.txt' requires undeclared environment variable 'EMPTY_VARIABLE', no default is given

That is because the code interpreting the parsing results has no way to tell apart an empty default value specification from no default value specification.

This might be done by making the \:\- an explicit capture group and getting information whether the default value syntax was used.

Would you be happy to accept a pull-request based on this?

Cheers,
Vlad

crash when missing glob pattern

$ es dec --region ap-southeast-1 -- `which ep`  -- `which cat` *.env
panic: runtime error: index out of range

goroutine 1 [running]:
panic(0x57a020, 0xc420010250)
        /usr/lib/go/src/runtime/panic.go:500 +0x1a1
github.com/spf13/cobra.(*Command).ExecuteC(0xc4200ae240, 0x5a6859, 0xf, 0xc42007a46a)
        /home/me/go/src/github.com/spf13/cobra/command.go:671 +0x7b6
github.com/spf13/cobra.(*Command).Execute(0xc4200ae240, 0x5a500e, 0x7)
        /home/me/go/src/github.com/spf13/cobra/command.go:648 +0x2b
main.main()
        /home/me/go/src/github.com/kreuzwerker/envplate/bin/ep.go:71 +0x51d

Add an option to exec() after parsing

Reason: no need for trivial wrapper scripts calling ep + exec() the real process afterwards.

Example: ep /nginx.conf -- /usr/sbin/nginx -c /nginx.conf

Compiling for ARMv6

We're trying to run ep from a Pi Zero, but are getting:
Illegal instruction (core dumped)
would is possible make a binary for ARMv6?

Parameter substitution '+' fails

Sometimes a configuration setting should only be included if the corresponding argument is included. With normal parameter substitution this should be able to be accomplished with '+'. For example:

foo=100
echo ${foo+setting} ${foo:-}
$ setting 100

echo ${bar+setting} ${bar:-}
$

What actually happens is envplate complains:

Error while parsing 'configuration.conf': 'configuration.conf' requires undeclared environment variable 'foo', no default is given

Syntax error running ep on RPI

Base image: Hypriot/rpi-node
OS: Hypriot 0.6.1 - http://blog.hypriot.com/downloads/

Step 23 : RUN curl -sLo /usr/local/bin/ep https://github.com/kreuzwerker/envplate/releases/download/v0.0.8/ep-linux && chmod +x /usr/local/bin/ep
---> Running in 38da1704a20d
---> 6c2ede3e1c94
Removing intermediate container 38da1704a20d
Step 24 : ADD config/interfaces /etc/network/interfaces
---> e8751d0eafcc
Removing intermediate container 29b6fe0e8b58
Step 25 : RUN ep /etc/network/interfaces
---> Running in 44dce7cb7251
/usr/local/bin/ep: 1: /usr/local/bin/ep: Syntax error: end of file unexpected (expecting ")")
The command '/bin/sh -c ep /etc/network/interfaces' returned a non-zero code: 2

any idea what could be causing this?

Make more easier to install in multiple arch docker image

I find it's not easy to install envplate in multiple arch docker image, because I need check system arch by shell, then curl the corresponding envplate package.

Proposal 1

Build a multiple arch docker image, which contains ep binary in /usr/local/bin, and publish to dockerhub.

Then we can use docker multi-stage build to simply copy the ep binary to my image, docker will auto pick the correct arch. Eg:

FROM kreuzwerker/envplate as envplate
FROM nginx:stable-alpine
COPY --from envplate /usr/local/bin/ep /usr/local/bin/ep 

Proposal 2

Compile envplate to Javascript which can use in cross-platform, and I see njs already packaged in Nginx docker image, it can execute the Javascript file.

I tried use https://github.com/gopherjs/gopherjs to compile, but it seems not work because of syscall problem.

$ GOOS=linux GOARCH=amd64 ~/go/bin/gopherjs build bin/ep.go
$ node ep.js README.md
warning: system calls not available, see https://github.com/gopherjs/gopherjs/blob/master/doc/syscalls.md
2021/12/30 00:12:52 [ ERROR ] Zero files matched passed globs '[README.md]'
fatal error: all goroutines are asleep - deadlock!

I think Proposal 1 is more easier to implement, any thought?

Add "recover" option for backups

Add an option to recover backups (when present). This should restore files (overwrite them) for which backups are present just before they are processed. Backups should be deleted when restoring successfully (I think).

Correct summary @brotbert ?

Downloaded ep-osx from releases on my mac and getting errors while running it

I am new to go, can someone look into it.
I Downloaded ep-osx from github releases on my mac and getting errors while running it
`failed MSpanList_Insert 0x222000 0xaececbcb9c3 0x0
fatal error: MSpanList_Insert

runtime stack:
runtime.throw(0x1d34eb)
/usr/local/Cellar/go/1.4.2/libexec/src/runtime/panic.go:491 +0xad fp=0x7ffeefbff6e0 sp=0x7ffeefbff6b0
runtime.MSpanList_Insert(0x1f07a8, 0x222000)
/usr/local/Cellar/go/1.4.2/libexec/src/runtime/mheap.c:692 +0x8f fp=0x7ffeefbff708 sp=0x7ffeefbff6e0
MHeap_FreeSpanLocked(0x1ed3a0, 0x222000, 0x100)
/usr/local/Cellar/go/1.4.2/libexec/src/runtime/mheap.c:583 +0x163 fp=0x7ffeefbff748 sp=0x7ffeefbff708
MHeap_Grow(0x1ed3a0, 0x8, 0x0)
/usr/local/Cellar/go/1.4.2/libexec/src/runtime/mheap.c:420 +0x1a8 fp=0x7ffeefbff788 sp=0x7ffeefbff748
MHeap_AllocSpanLocked(0x1ed3a0, 0x1, 0x0)
/usr/local/Cellar/go/1.4.2/libexec/src/runtime/mheap.c:298 +0x365 fp=0x7ffeefbff7c8 sp=0x7ffeefbff788
mheap_alloc(0x1ed3a0, 0x1, 0x12, 0x0)
/usr/local/Cellar/go/1.4.2/libexec/src/runtime/mheap.c:190 +0x121 fp=0x7ffeefbff7f0 sp=0x7ffeefbff7c8
runtime.MHeap_Alloc(0x1ed3a0, 0x1, 0x10000000012, 0xc909)
/usr/local/Cellar/go/1.4.2/libexec/src/runtime/mheap.c:240 +0x66 fp=0x7ffeefbff828 sp=0x7ffeefbff7f0
MCentral_Grow(0x1f5118, 0x0)
/usr/local/Cellar/go/1.4.2/libexec/src/runtime/mcentral.c:197 +0x8b fp=0x7ffeefbff890 sp=0x7ffeefbff828
runtime.MCentral_CacheSpan(0x1f5118, 0x0)
/usr/local/Cellar/go/1.4.2/libexec/src/runtime/mcentral.c:85 +0x167 fp=0x7ffeefbff8c8 sp=0x7ffeefbff890
runtime.MCache_Refill(0x21e000, 0x12, 0x0)
/usr/local/Cellar/go/1.4.2/libexec/src/runtime/mcache.c:90 +0xa0 fp=0x7ffeefbff8f0 sp=0x7ffeefbff8c8
runtime.mcacheRefill_m()
/usr/local/Cellar/go/1.4.2/libexec/src/runtime/malloc.c:368 +0x57 fp=0x7ffeefbff910 sp=0x7ffeefbff8f0
`

How to build ?

I'm new to Go and would like to build this package so I can use it on Ubuntu.

Anyone some directions / examples ?

Thanks!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.