Automatic lookup for items in parent sections about lib.py.configfile HOT 15 CLOSED

Um... you might be expecting inherit_options to do something different from what it was designed for. This is what it should do (and does for me), take this file:

test1 = foo

[foo.bar.baz]
test2 = bar

Then execute this:

cf1 = configfile.ConfigFile('./test.conf')
cf2 = configfile.ConfigFile('./test.conf', inherit_options=True)

print(cf1('foo')('bar')('baz').get_options())
print(cf1('foo')('bar')('baz').get_options(inherit_options=True))
print(cf2('foo')('bar')('baz').get_options(inherit_options=False))
print(cf2('foo')('bar')('baz').get_options())

You should see:

OrderedDict([('test2', 'bar')])
OrderedDict([('test2', 'bar'), ('test1', 'foo')])
OrderedDict([('test2', 'bar')])
OrderedDict([('test2', 'bar'), ('test1', 'foo')])

I.e., for the get_options method, if inherit_options is True (either globally or for the specific call) also the options from the ancestor sections are returned.

However, the section from which you run the method must exist, even if empty. What you want is instead a way to retrieve all the options from a section regardless whether it exists or not? Can you please elaborate, and possibly provide a full example with the source configuration, and describe what the expected result would be?

(About interpolation, it would be something similar to configparser's feature, but with support for subsections, hence the different syntax; I'll try to add an example though)

from lib.py.configfile.

(By the way, the interpolation algorithm got broken at some point in time (#3 still pending...), anyhow now it's fixed and I've added the example in the docstring at the top, all in the develop branch)

from lib.py.configfile.

I'm working on configuration for wiki-scripts. The specification should be as follows:

There is a top-level option to specify the main config section to be read, e.g. site = ArchWiki selects the [ArchWiki] section, where all other options can be specified. To override these global options on a per-script basis, there would be optional [sitename.scriptname] subsections, which would inherit all options from the parent section. For example:

site = ArchWiki

[ArchWiki]
api-url = https://wiki.archlinux.org/api.php
index-url = https://wiki.archlinux.org/index.php
cookie-name = ArchWiki.cookie

[ArchWiki.update-pkg-templates]
cookie-name = ArchWiki.bot.cookie

That is one scenario I'm thinking of supporting, but sections should not be required, i.e. placing everything in top-level scope should also work. Also mixing the sectioned and non-sectioned scenarios should be possible and favourable for some options like tmp-dir, cache-dir etc:

site = ArchWiki
tmp-dir = /tmp/wiki-scripts

[ArchWiki]
api-url = https://wiki.archlinux.org/api.php
index-url = https://wiki.archlinux.org/index.php
cookie-name = ArchWiki.cookie

[ArchWiki.update-pkg-templates]
cookie-name = ArchWiki.bot.cookie

[ArchWiki.testing-script]
# override top-level option only for testing a specific script
tmp-dir = ./tmp

Regarding the actual implementation, you can see the progress in the config branch (notably config.py). Note that you will need ConfigArgParse with changes from my config_files_without_merging branch; see also bw2/ConfigArgParse#28. It's not exactly in a working state right now though...

As for this issue, I just don't like nesting the try ... except blocks, it would be "nicer" to do it recursively from the Section.__call__() method.

from lib.py.configfile.

Uh now I get it, it's a fantastic idea! I can't test your implementation now, but in the develop branch you should be able to use a new section_fallback boolean parameter when instantiating ConfigFile, and override it in __call__ if needed, to do what you want. Please let me know if that fixes this report: if you think everything works correctly I can even release a new version of this library :)

from lib.py.configfile.

Oops, I've just thought of a bad bug with the current implementation: if you have a config like this:

foo = bar

[Test]
foo = baz

...and you call cf('Section')('Test')['foo'], it's going to read baz instead of bar. In your specific case though, you would only be affected if you had a script name that corresponds to a site name, which I think is unlikely.

from lib.py.configfile.

...and, a config like this would also be safe from the bug above:

[Section]
foo = bar

[Test]
foo = baz

Calling cf('Section')('Test')['foo'] would correctly read bar here.

from lib.py.configfile.

Thanks, there is just a small bug since the initial commit: d[o] = s._options[o][:] should be d.setdefault(o, s._options[o][:]) so that options from parent sections don't override children. Except for this, I think it works for my use case, let me test it more deeply...

from lib.py.configfile.

True, committed in develop, cheers.
About the bug above, the "problem" is that, in general, sections are resolved from the "left", so there can't be an elegant way to fix the bug in __call__.
The only way I see to fix it is to add an alternative method to resolve sections, something like get_section(par), where par can be a string with the relative path to a subsection (using the normal character as separator, . by default), or a sequence of strings, each representing a subsection name.

Edit: perhaps the new behavior could be added directly to __call__; also, the section_fallback parameter should be removed from the global options.

from lib.py.configfile.

Alternatively, instead of returning self from __call__, it could create a new empty section with the requested name. The question is whether it should be added to the tree or not; they would probably have to be excluded from exports etc.

Interestingly the get_tree method creates an empty entry for missing middle-sections, e.g. for this config

foo = value1

[section1]
foo = value2

[section1.section2.section3]
foo = value3

the tree is (note the entry for section2)

>>> cf = configfile.ConfigFile("sample.conf", inherit_options=True, section_fallback=True)
>>> pprint(cf.get_tree())
(OrderedDict([('foo', 'value1')]),
 {'section1': (OrderedDict([('foo', 'value2')]),
               {'section2': (OrderedDict(),
                             {'section3': ({'foo': 'value3'},
                                           OrderedDict())})})})

Also, consider these calls for the above config:

>>> cf.get_options()
OrderedDict([('foo', 'value1')])
>>> cf("section1").get_options()
OrderedDict([('foo', 'value2')])
>>> cf("section1")("section2").get_options()
OrderedDict([('foo', 'value2')])
>>> cf("section1")("section2")("section3").get_options()
OrderedDict([('foo', 'value3')])

This is what one would expect, but because __call__ returns self, I have no idea why the last call for section3 actually gives expected result...

from lib.py.configfile.

Oh, missing middle-sections are probably created during parsing/import, so it does not return self in this case...

from lib.py.configfile.

Having a getter, such as __call__ in this case, modify the internal data structure is not what I'd call an "elegant" fix :P (i.e. it's sure to cause trouble in the future)
Yes, middle sections are created when importing the data. When exporting to a file, empty sections that aren't found in the original file are filtered by https://github.com/kynikos/lib.py.configfile/blob/develop/src/configfile/__init__.py#L1430.

from lib.py.configfile.

Then perhaps a new method should supplement __call__ as the dict's setdefault method supplements get. The make_subsection method is almost there, only the created section is not returned.
But on second thought, accessing the subsections via "foo.bar.baz" with just one call would be great (and elegant)!

from lib.py.configfile.

I don't think that using make_subsection like setdefault would make code very clear; if you want I can make it return the new subsection, but I wouldn't like it to encourage obscure coding.
Instead, I've improved __call__ so that now it can be given more positional arguments (strings), representing a relative path to a descendant subsection. A safe_calls parameter can be set to True when instantiating ConfigFile, so that section calls will always return the deepest existing section in the given path, even falling back on the caller section if the first subsection doesn't exist. safe_calls can be overridden per each call with the safe parameter.
Note that safe_calls replaces section_fallback, which is deprecated. In order to exploit this functionality, all serial section calls (cf('section1')('section2')) must be replaced by cf('section1', 'section2').
Considering your example above:

cf = ConfigFile('/path/to/conf', safe_calls=True)
cf('section1', 'section3')  # Returns section1
cf('section1', 'section3', safe=False)  # Raises KeyError
...

If you think this fixes it, please close the report :) (I think you can since you opened it)

from lib.py.configfile.

Just to make it clear, "serial" section calls (cf('section1')('section2')) still work, it's just that they can't be "protected" by safe_calls of course.

from lib.py.configfile.

Thanks, this is just about perfect!

from lib.py.configfile.