confd should have an option to force a config update about confd HOT 11 CLOSED

@halcyonCorsair could you please share the .toml configuration file? It seems like the check_cmd returning a non-zero exit code for some reason.

from confd.

@gabrtv I could be wrong, but my suspicion is that what happened is:

• nodes were accidentally removed from etcd
• confd was run
  • the check was run against the old (working) config
  • confd put new (broken) config in place with all servers removed from the nginx upstream
  • a broken nginx config is now in place
• nodes were re-added to etcd
• confd was run
  • the check was run against the old (broken) config and failed

At which point, if I'm reading the source right, confd bailed out and didn't put the new config in place.

config:

[template]
src = "site-nginx-staging.tmpl"
dest = "/etc/nginx/sites-enabled/site.conf"
owner = "nginx"
mode = "0644"
keys = [
  "/site/upstream/staging",
]
check_cmd = "/usr/sbin/nginx -t -c /etc/nginx/nginx.conf"
reload_cmd = "/usr/sbin/service nginx reload"

template:

upstream site {
{{range $server := .site_upstream_staging}}
  server {{$server.Value}};
{{end}}
}

server {
  server_name site.mydomain.com;
  location / {
    proxy_pass http://site;
    proxy_redirect off;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  }
}

from confd.

@gabrtv does that make sense?

from confd.

Ah, you need to fix the check command. You are checking against the live config. Instead of

check_cmd = "/usr/sbin/nginx -t -c /etc/nginx/nginx.conf"

Use

check_cmd  = "/usr/sbin/nginx -t -c {{ .src }}"

from confd.

@halcyonCorsair Does my last comment work for you?

from confd.

@kelseyhightower I don't think so, the template I'm putting in place is a partial, included by the main /etc/nginx/nginx.conf. The intention (at least at this stage) is that nginx will be hosting multiple sites.

from confd.

I agree with @kelseyhightower that the problem is check_cmd returning a non-zero exit code. See:
https://github.com/kelseyhightower/confd/blob/master/resource/template/resource.go#L126:L131

Maybe a partial template will always fail an Nginx config check?

from confd.

@gabrtv @kelseyhightower I feel like it may not be possible to test a new piece of configuration in isolation especially for apps which have config split into many files (eg. nginx, exim, etc).

How about an option to move the original config out of the way temporarily while the check_cmd get's run, and on failure, moved back into place?

from confd.

@halcyonCorsair my personal opinion is that the check_cmd and reload_cmd constructs should be sufficient as is. I would try writing a check_cmd script that assembles and tests a complete config in a temp directory, including all your existing partial configs plus the newly templated one.

from confd.

@halcyonCorsair I'm with @gabrtv on this one. Now that I think about it, nothing is stoping you from moving the temp file in place before running the check_cmd, and then moving it back if it fails.

I think that would be a nice use case to write up.

@halcyonCorsair Closing this one as I don't think that feature should be added to confd. But I do like the idea of a check_cmd that did that.

from confd.

@kelseyhightower I see the reasoning, it's just an unfortunate amount of extra work for my use case. :(

from confd.