Configuration management about nameko HOT 13 OPEN

How about accepting a config-file param?
As the number of extensions grows the list of args would grow making it a bit painful to list them as command line args.
To manage different sets of settings one is probably more likely to keep them in files, too.

from nameko.

Commandeering this issue to capture the discussion about registering configuration options.

Thanks to #255 we now have the ability to load config from a file, but there are a few other issues:

• nameko run accepts a --broker option but none of the other configuration keys
• most config is used by Extensions, which can be separate projects, but there's no way to inspect which config values are required or what their defaults are
• importing FOO_CONFIG_KEY everywhere is ugly and you have to know where each Extension chooses to define them

If we registered configuration options and default values (via setuptools entrypoints?) we could:

a) access them from a known location (e.g. container.config)
b) inspect the required keys and any default values for a given set of extensions
c) automatically populate the argparse options for nameko run as @davidszotten suggests

from nameko.

@mattbennett what about reading environment variables?

export NAMEKO_AMQP_URI="amqp://guest:guest@localhost"

then in nameko.cli.main

import os
config = {k.partition('NAMEKO_')[-1], v for k, v in os.environ if k.startswith('NAMEKO_')}

from nameko.

@rochacbruno yep, I think env vars should be supported, but as part of a larger piece of work.

from nameko.

Maybe this example could be a good way to use environment variables on YAML config file:
http://stackoverflow.com/questions/26712003/pyyaml-parsing-of-the-environment-variable-in-the-yaml-configuration-file

from nameko.

nameko_cli

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import yaml
import os
import re

from nameko.cli.main import main as nameko_cli_main


ENVVAR_PATTERN = re.compile(r'\$\{([a-zA-Z][^}:\s]+)(:([^}]+))?\}')


def replace_envvar(match):
    env_var, _, default = match.groups()
    if default == 'null':
        default = None
    return os.environ.get(env_var, default)


def envvar_constructor(loader, node):
    value = loader.construct_scalar(node)
    return ENVVAR_PATTERN.sub(replace_envvar, value)


def setup_yaml_parser():
    yaml.add_implicit_resolver ("!envvar", ENVVAR_PATTERN)
    yaml.add_constructor('!envvar', envvar_constructor)


def main():
    setup_yaml_parser()
    nameko_cli_main()

if __name__ == '__main__':
    main()

config.yaml

AMQP_URI: ${AMQP_URI:amqp://guest:guest@localhost}:${AMQP_PORT:5672}
EMAIL_PASSWORD: ${EMAIL_PASSWORD}
SENTRY:
    DSN: ${SENTRY_DSN:null}

Pattern must be ${VAR_NAME<:DEFAULT_VALUE>} allow null.
<:DEFAULT_VALUE> is optional

./nameko_cli run <module>[:<ServiceClass>] --config=config.yaml

from nameko.

this looks really nice. i like the bash-style ${VAR_NAME<:DEFAULT_VALUE>}. @fabiocerqueira would you like to raise a pull-request that implements this?

from nameko.

@davidszotten @fabiocerqueira I'll be happy to pick this up if that's ok.

from nameko.

@kooba go ahead :D I think this would be great. It will be nice if supports different types, int values, float values, boolean values. Thank you.

from nameko.

It would be interesting to allow multiple config files.

In many applications, you have part of the configuration that is common to many/all services (like AMQP_URI, exchange, serializer, database url...) and some services need vry specific paramters.

Allowing --config to be a list (nargs='+') and updating config with these files would do the job

something like this:

config = {}
for conf_file in args.config):
    with open(conf_file) as fle:
            config.update(yaml.load(fle))

from nameko.

@rejoc I like this idea. We could bundle it into #520 (cc @iky) or make it as a standalone change first. Do you want to raise a PR?

from nameko.

is it common for tools to support multiple config files like this? (none of the ones i can think of do)

(i tend to handle this kind of sharing in my config management tooling)

from nameko.

the config management of other tools allow you to "include" config files in config files. But there is no such thing with yaml (or I did not find it). It would have been my prefered choice.

Personnaly, I don't like the idea of having an external tool to build the config file from a "meta" configuration file.

from nameko.