Validating nginx configs to prevent sites from going down on reload of bad config about nginx HOT 5 CLOSED

As for as the pattern I've implemented in dhcp and syslog-ng I've had good results with it thus far, no more erroneous service restarts to a bad config. Using guards alone I still ran into situations where the service resource may still get notified depending on resource ordering, hence adding the explicit resource_delete! for the service to ensure it can't run.

My only additional thought around the use of ExecStartPre is how it handles restarts when the config test fails, the docs mention that the unit is marked as failed so without testing I'm not sure whether that'd end up killing the original running process or not? Also come to think of it I'm not sure how it'd work with a reload rather than a restart either actually thinking about it.

from nginx.

This pattern in dhcp might be useful here too. @bmhughes implemented it there and can maybe provide some feedback there.

Another option is to the go the route we went with in haproxy where we manage the systemd unit file and just add a ExecStartPre parameter which does the same thing.

from nginx.

copying in a suggestion from slack:

Tensibai  2 days ago
IMHO replicating what's done on apache2 cookbook is the way to go, guard the service by validating the config, this way it doesn't restart/reload a bad Config and chef can fix the config, the drawback is that you end up with a successful run and ninx running on its old Config without something warning you :/
An alternative could be an execute resource notified before the service restart, if the conf check fail that should break the run and avoid the restart

from nginx.

Unless we delay the service restart/reload to the end of the play, the issue is that if the config is broken Chef can't run to fix it.
If we can delay the restart/reload as a delayed notification and still have another notification running before to check if the config is valid (and cancel the reload if needed) then we'll be okay.

Otherwise, on the path of using ExecStartPre and ExecReload, looks like a systemd unit can have multiple ExecReload and they'll be executed sequentially.

https://www.freedesktop.org/software/systemd/man/systemd.service.html#ExecReload=

This argument takes multiple command lines, following the same scheme as described for ExecStart= above.

https://www.freedesktop.org/software/systemd/man/systemd.service.html#ExecStart=

If more than one command is specified, the commands are invoked sequentially in the order they appear in the unit file. If one of the commands fails (and is not prefixed with "-"), other lines are not executed, and the unit is considered failed.

But that would then join @bmhughes question of "I'm not sure whether that'd end up killing the original running process or not?"

from nginx.

I went with a simple solution to avoid raising errors. I guard the service actions to not run unless the config is valid (apache2 style) and log an error message if the config is invalid so that anyone catching these in log monitoring can see them and address hosts with invalid configs that didn't get loaded.

This should prevent the runs from failing when trying to do service actions and allow for chef to continue and be able to replace the invalid config with a hopefully valid one. It also follows a similar pattern to apache2 which i thought was good for consistency.

from nginx.