Allow runtime config to be optionally loaded with strict or non-strict unmarshaling about cortex HOT 3 CLOSED

I am not a fan of non-strict. Currently, if the config was updated wrongfully, a metric is generated. Which can be used for alerting. If no cortex process are restarted, they keep the old config, everything works. If we are using kubernetes, with enough pods crashed, the pdb can protect us from having too many restarts, while we attend the alert.

With non-strict, now we might have broken things for a user and there is no alert for that. It makes hard reasoning about the health of the config.

from cortex.

If I have a typo on active_series field for example, with non-strict mode Cortex will ignore this field, then it will use the default value for active_series? This sounds even worse than crashing Cortex pods.

from cortex.

To answer your question @yeya24, yes a typo on the active_series field will force Cortex to use the default value.
I think these are fair points, we can use the Kubernetes PDB to protect Cortex from loading a bad runtime configuration and create a mechanism on our service side to rollback a bad configuration version.

from cortex.