I was wondering whether we could access the path to the current config. Something likeK ${__file__}

I'm wanted to test hydra, but don't want to hardcode the base directory to my working directory as different collaborators might have a different one. I was hoping that I define the base directory in the main config.yaml by using ${__file__}.

Freeze is nice, but does not work for unintended reads.
It's possible that the real solution here would be to relay on node level validation.

>>> a =  OmegaConf.empty()
>>> a.foo
>>> a.freeze(True)
>>> a
{}
>>> a.foo
>>>
>>> a.foo = a
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/private/home/omry/dev/omegaconf.git/omegaconf/dictconfig.py", line 47, in __setattr__
    self.__setitem__(key, value)
  File "/private/home/omry/dev/omegaconf.git/omegaconf/dictconfig.py", line 26, in __setitem__
    raise FrozenConfigError(self.get_full_key(key))
omegaconf.errors.FrozenConfigError: foo

Similar to standing on a root directory and asking whether any of the sub-directories has files in it

when I use the cfg's property such as cfg.kernel_sizes, I what to compare cfg.kernel_sizes with list, tuple, or int

my code is that:

if isinstance(cfg.kernel_sizes, (list, tuple, int)):
    ...

but the type(cfg.kernel_sizes) is omegaconf.listconfig.ListConfig, not list, so I can't get the true judgement

I want to know how to make the types become list, dict and other standard types

family, name = next(iter(default.items()))

It breaks at # Access and manipulation Input yaml file:

I get an invalid syntax error

In struct mode, accessing an invalid attribute through getattr() should result in an AttributeError and not in a KeyError.

c.get('not_there', 'default_value') throws if c is in struct mode.

use case:
config.yaml

foo:
  list: [1,2,3]
  bar: ${foo.list}

Python 2.7 is reaching end of life in 1/1/2020

1. Provide an error when running OmegaConf 1.5 with Python 2.7, pointing users to pin 1.4 if they wish to use Python 2.7
2. Remove 2.7 support from master, update docs, unit tests (Additional advice here
3. Clean up 2.7 specific and supporting dependencies
4. Leverage Python 3.x features: Type annotations, Mypy/Pyre. clean up Python 2.7 support

Issue

When instantiating OmegaConf from from_dotlist, yaml loader tries to convert date-formatted strings into date type, therefore raises error when validating the type of values.

 OmegaConf.from_dotlist(["my_date=2019-11-11"])

>>>

.../site-packages/omegaconf/config.py in _prepare_value_to_add(self, key, value)
    448         if not Config.is_primitive_type(value):
    449             full_key = self.get_full_key(key)
--> 450             raise ValueError("key {}: {} is not a primitive type".format(full_key, type(value).__name__))
    451
    452         if self._get_flag('readonly'):

ValueError: key my_date: date is not a primitive type

This error also blocks the overriding of configuration values by using cli arguments in hydra.

Potential workaround

Could be solved by removing implicit timestamp resolvers in default yaml.SafeLoader.
(Related stackoverflow: https://stackoverflow.com/questions/34667108/ignore-dates-and-times-while-parsing-yaml)

1. make sure 100% coverage.
2. add coverage to tox.
3. add coverage badge.

test args:
optimizer.lr_steps=[1,2,3]

This seems like a useful extension:

flap: 99

baz:
  flaf: 100

foo:
  bar: 10
  a: ${foo.bar}     # 10, absolute
  b: ${.bar}        # 10, relative
  c: ${..flap}      # 99, relative, level up
  d: ${..baz.flaf}  # 100, level up and then and then dig into baz

the following valid config throws an exception in the debugger when trying to inspect it.

defaults:
  - dataset: imagenet
  - model: alexnet
  - optimizer: nesterov

File "/scratch/pwang01/anaconda3/envs/rl-env/lib/python3.7/site-packages/omegaconf/omegaconf.py", line 48, in create
    return DictConfig(obj, parent)
  File "/scratch/pwang01/anaconda3/envs/rl-env/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 15, in __init__
    self.__setitem__(k, v)
  File "/scratch/pwang01/anaconda3/envs/rl-env/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 25, in __setitem__
    value = self._prepare_value_to_add(key, value)
  File "/scratch/pwang01/anaconda3/envs/rl-env/lib/python3.7/site-packages/omegaconf/config.py", line 446, in _prepare_value_to_add
    value = OmegaConf.create(value, parent=self)
  File "/scratch/pwang01/anaconda3/envs/rl-env/lib/python3.7/site-packages/omegaconf/omegaconf.py", line 48, in create
    return DictConfig(obj, parent)
  File "/scratch/pwang01/anaconda3/envs/rl-env/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 15, in __init__
    self.__setitem__(k, v)
  File "/scratch/pwang01/anaconda3/envs/rl-env/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 25, in __setitem__
    value = self._prepare_value_to_add(key, value)
  File "/scratch/pwang01/anaconda3/envs/rl-env/lib/python3.7/site-packages/omegaconf/config.py", line 446, in _prepare_value_to_add
    value = OmegaConf.create(value, parent=self)
  File "/scratch/pwang01/anaconda3/envs/rl-env/lib/python3.7/site-packages/omegaconf/omegaconf.py", line 48, in create
    return DictConfig(obj, parent)
  File "/scratch/pwang01/anaconda3/envs/rl-env/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 10, in __init__
    super(DictConfig, self).__init__()
RecursionError: maximum recursion depth exceeded while calling a Python object

Minimal reproduction config.yaml:

a: &a
    b: *a

OmegaConf.create(dict(foo={}))

expanding foo results in a stack trace in the debugger.

Some yaml files are organizing as the following, could you please consider to support that ?
Like the followling from the official documentation https://pyyaml.org/wiki/PyYAMLDocumentation

Aliases
Note that PyYAML does not yet support recursive objects.

Using YAML you may represent objects of arbitrary graph-like structures. If you want to refer to the same object from different parts of a document, you need to use anchors and aliases.

Anchors are denoted by the & indicator while aliases are denoted by ``. For instance, the document

left hand: &A
name: The Bastard Sword of Eowyn
weight: 30
right hand: *A
expresses the idea of a hero holding a heavy sword in both hands.

PyYAML now fully supports recursive objects. For instance, the document

&A [ *A ]
will produce a list object containing a reference to itself.

a: 1
b: 2.0
d: ${a} - ${b}
e: ${a} + ${b}
d: ${a} * ${b}
e: ${a} / ${b}

desired behavior:

conf = OmegaConf.load("file.yaml")
assert conf.d == -1.0
assert conf.d == 3.0
assert conf.d == 2.0
assert conf.d == 0.5

This should work only if both operands are numbers (int or float). result type should be int if both are ints or float otherwise.

EDIT (2022-12-20):

✨✨✨ See the OmegaConf docs on How to Perform Arithmetic Using eval as a Resolver ✨✨✨

Hi, thank you for this great repo!

I want to instantiate omegaconf.Config with protocol like class to tell IDE what member the config have, that is:

class A:
    a: int = 1

cfg: A = omegaconf.OmegaConf.create(A)
print(cfg.a)

1

The following should not throw:

cfg = OmegaConf.create()
cfg.foo = (1,2)

when converting to yaml string or container, provide an option to resolve interpolations.

A dict is supposed to have a .copy() method (https://docs.python.org/3/library/stdtypes.html#dict.copy). For an OmegaConf object to be compatible with a dict's API, it should also support the copy() method -

>>> from omegaconf import OmegaConf
>>> conf = OmegaConf.create({"a": 2, "b": [1,2,3]})
>>> conf
{'a': 2, 'b': [1, 2, 3]}
>>> conf.copy()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'NoneType' object is not callable

Currently in OmegaConf, for the following config:

server:
  host: localhost
  port: 80

client:
  url: http://${server.host}:${server.port}/
  server_port: ${server.port}

We can get conf["server"] and conf["client"], but then we lose the parent component name (e.g., "server" or "client").

This feature is to be able to get complete subtrees, rooted at the specified head.

Example config:

server:
  host: localhost
  port: 80

client:
  url: http://${server.host}:${server.port}/
  server_port: ${server.port}

Usage:

print(conf.root_from("server"))

Outputs

server:
  host: localhost
  port: 80

dest[key] = src[key] is resolving variables which goes against the strategy of resolving lazily.

    @staticmethod
    def map_merge(dest, src):
        """merge src into dest and return a new copy, does not modified input"""
        assert isinstance(dest, DictConfig)
        assert isinstance(src, DictConfig)

        dest = dest.content
        dest = copy.deepcopy(dest)
        src = copy.deepcopy(src)
        for key, value in src.items():
            if key in dest and isinstance(dest[key], Config):
                if isinstance(value, Config):
                    dest[key].merge_with(value)
                else:
                    dest[key] = value
            else:
                dest[key] = src[key]
        return dest

for a nested config:

foo:
  bar:
    baz: 10

if foo is struct, and we do

with open_dict(cfg.foo.bar):
   bar.baz = 20

the struct flag of foo.bar would be changed to True.

Since Config is a dynamic data type, I want to minimize the functions that exists on it to the bare minimum.
A user can easily create a save field that would potentially conflict with the save() function.
A future version would remove config.save() completely.

defaults object here is actually of type list instead of Config.

if main_cfg.defaults is None:
    main_cfg.defaults = []

See facebookresearch/hydra#266 (Which is actually one bug in Hydra and one in OmegaConf).

Also check for other missing list methods.

OmegaConf.set_readonly(cfg.whitelisted_orgs, True)
cfg = OmegaConf.merge(cfg, cli)

This python tool.py whitelisted_orgs=['blah'] changes the readonly variable instead of throwing.

from omegaconf.nodes import FloatNode

x = FloatNode()
x == 1

raises

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-163-39f43fc4ed44> in <module>
      2 
      3 x = FloatNode()
----> 4 x == 1

~/work/omegaconf/omegaconf/nodes.py in __eq__(self, other)
    115             other_val = other
    116 
--> 117         return self.val == other_val or (math.isnan(self.val) and math.isnan(other_val))
    118 
    119 

TypeError: must be real number, not NoneType

due to the following line:

math.isnan accepts typing.SupportsFloat but not NoneType.

OmegaConf runs into issues when the config object is constructed by incrementally adding new OmegaConf objects.

To reproduce:

conf = OmegaConf.create()
conf.a = 1
conf.b = OmegaConf.create()
conf.b.c = "${a}"

With conf above, trying to access conf.b.c gets me the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/private/home/kchua13/.conda/envs/hierarchy/lib/python3.6/site-packages/omegaconf/dictconfig.py", line 72, in __getattr__
    return self.get(key=key, default_value=None)
  File "/private/home/kchua13/.conda/envs/hierarchy/lib/python3.6/site-packages/omegaconf/dictconfig.py", line 83, in get
    return self._resolve_with_default(key=key, value=self.get_node(key), default_value=default_value)
  File "/private/home/kchua13/.conda/envs/hierarchy/lib/python3.6/site-packages/omegaconf/config.py", line 131, in _resolve_with_default
    return self._resolve_single(value) if isinstance(value, str) else value
  File "/private/home/kchua13/.conda/envs/hierarchy/lib/python3.6/site-packages/omegaconf/config.py", line 426, in _resolve_single
    return Config._resolve_value(root, match.group(1), match.group(2))
  File "/private/home/kchua13/.conda/envs/hierarchy/lib/python3.6/site-packages/omegaconf/config.py", line 407, in _resolve_value
    raise KeyError("{} interpolation key '{}' not found".format(inter_type, inter_key))
KeyError: "str interpolation key 'a' not found"

The associated PR (#86 ) is adding support for initializing OmegaConf from class files and objects.
Details below

1. Config classes and objects (dataclass or attrs classes) can be converted to OmegaConf object.

2. The following value types are supported:

• with a default value assigned to it.
• Optional fields (can accept None)
• assigned omegaconf.MISSING, Equivalent for '???' in the yaml files (Mandatory missing).
  The semantics here is that this value is not yet set. it must be set prior to read access.
• Interpolation using omegaconf.II() function :
  II("foo") is equivalent to ${foo} in the YAML file.
  Examples:
  • foo=II("bar") : Foo inherits the type and value of bar at runtime.
  • foo=II("http://${host}:${port}/") : foo is always a string. actual value is determined by host and port
  • foo=II("env:USER") : foo is evaluated on access, the type and value is determined by the function (env in this case)

Example for bool types (The same is supported for int, str, float and enum):

@dataclass
class BoolConfig:
    with_default: bool = True
    null_default: Optional[bool] = None
    mandatory_missing: bool = MISSING
    interpolation: bool = II("with_default")

3. the typing information in them is available and acted on at runtime (composition and command line overrides are at runtime).

4. type annotations can be used to get static type checking.

Currently the following is supported for both dataclasses and attr classes.
Examples below are @dataclasses. attr classes are similar.

@dataclass
class Database:
    name: str = "demo_app_db"
    # A an untyped list. will accept any type that can be represented by OmegaConf
    tables: List = field(default_factory=lambda: ["table1", "table2"])
    port: int = 3306

    # A list of integers, will reject anything that cannot be converted to int at runtime
    admin_ports: List[int] = field(default_factory=lambda: [8080, 1080])

    # A dict of string -> int, will reject any value that cannot be converted to int at runtime
    error_levels: Dict[str, int] = field(
        default_factory=lambda: {"info": 0, "error": 10}
    )

This class, and objects of this class can be used to construct an OmegaConf object:

# Create from a class
conf1 = OmegaConf.create(Database)

# Create from an object
conf2 = OmegaConf.create(Database(port=3307))

Python type annotation can then be used on the config object to static type checking:

def foo(cfg : Database):
   cfg.port = 10 # passes static type check
   cfg.port = "nope" # fails static type check

5. Composition and overrides are often happening at runtime (from files, or command line flags).
   To support that, there is also a runtime validation and conversion.
   The following will succeed at runtime:

conf.merge_with_dotlist(['port=30']) # success
conf.merge_with_dotlist(['port=nope']) # runtime error

6. OmegaConf objects created from Structured configs will be set to struct mode, this means that any access to a field not in the struct will result in a runtime error:

conf.pork = 80 # fail

7. Typed containers are validated at runtime:

conf: Database = OmegaConf.create(Database)
conf.admin_ports[0] = 999  # okay
assert conf.admin_ports[0] == 999

conf.admin_ports[0] = "1000"  # also ok!
assert conf.admin_ports[0] == 1000

with pytest.raises(ValidationError):
    conf.admin_ports[0] = "fail"

with pytest.raises(ValidationError):
    conf.error_levels["info"] = "fail"

8. Nested Object/Classes are supported, as well as Enums.

class Protocol(Enum):
    HTTP = 0
    HTTPS = 1


@dataclass
class ServerConfig:
    # Nested configs are by value.
    # This will be expanded to default values for Database
    db1: Database = Database()

    # This will be expanded to the values in the created Database instance
    db2: Database = Database(tables=["table3", "table4"])

    host: str = "localhost"
    port: int = 80
    website: str = MISSING
    protocol: Protocol = MISSING
    debug: bool = False

9. Enums can be assigned by Enum, by name string or by value (the later should be used at runtime only).

    conf: ServerConfig = OmegaConf.create(ServerConfig(protocol=Protocol.HTTP))
    assert conf.protocol == Protocol.HTTP

    # The HTTPS string is converted to Protocol.HTTPS per the enum name
    conf.protocol = "HTTPS"
    assert conf.protocol == Protocol.HTTPS

    # The value 0 is converted to Protocol.HTTP per the enum value
    conf.protocol = 0
    assert conf.protocol == Protocol.HTTP

    # Enum names are care sensitive
    with pytest.raises(ValidationError):
        conf.protocol = "https"
 
    for value in (True, False, "1", "http", 1.0):
        with pytest.raises(ValidationError):
            conf.protocol = value

10. Frozen state is preserved (and is recursive):

    @dataclass(frozen=True)
    class FrozenClass:
        x: int = 10
        list: List = field(default_factory=lambda: [1, 2, 3])

    conf = OmegaConf.create(FrozenClass)
    with pytest.raises(ReadonlyConfigError):
        conf.x = 20

    # Read-only flag is recursive
    with pytest.raises(ReadonlyConfigError):
        conf.list[0] = 20

Master task to release 1.4.0.

Change semantics of key in cfg to be reasonable when key points to '???', or
is an interpolation that points to '???', or an interpolation that cannot be resolved.

desired behavior:

@pytest.mark.parametrize("conf,key,expected", [
    ({"a": 1, "b": {}}, "a", True),
    ({"a": 1, "b": {}}, "b", True),
    ({"a": 1, "b": {}}, "c", False),
    ({"a": 1, "b": "${a}"}, "b", True),
    ({"a": 1, "b": "???"}, "b", False),
    ({"a": 1, "b": "???", "c": "${b}"}, "c", False),
    ({"a": 1, "b": "${not_found}"}, "b", False),
])
def test_in_dict(conf, key, expected):
    conf = OmegaConf.create(conf)
    ret = key in conf
    assert ret == expected