Comments (5)
kaimo455
commented on June 10, 2024
Well, seems that even a simple interpolation field will crash, not customized resolver needed.
import omegaconf
cfg = omegaconf.OmegaConf.create({'a': '???', 'b': '${.a}'})
omegaconf.OmegaConf.missing_keys(cfg)from omegaconf.
Jasha10
commented on June 10, 2024
For reference, here's the traceback produced by the example from the OP:
$ python tmp.py
Traceback (most recent call last):
File "/home/homestar/dev/omegaconf/tmp.py", line 6, in <module>
omegaconf.OmegaConf.missing_keys(cfg)
File "/home/homestar/dev/omegaconf/omegaconf/omegaconf.py", line 831, in missing_keys
gather(cfg)
File "/home/homestar/dev/omegaconf/omegaconf/omegaconf.py", line 828, in gather
elif OmegaConf.is_config(_cfg[key]):
File "/home/homestar/dev/omegaconf/omegaconf/dictconfig.py", line 373, in __getitem__
self._format_and_raise(key=key, value=None, cause=e)
File "/home/homestar/dev/omegaconf/omegaconf/base.py", line 230, in _format_and_raise
format_and_raise(
File "/home/homestar/dev/omegaconf/omegaconf/_utils.py", line 901, in format_and_raise
_raise(ex, cause)
File "/home/homestar/dev/omegaconf/omegaconf/_utils.py", line 799, in _raise
raise ex.with_traceback(sys.exc_info()[2]) # set env var OC_CAUSE=1 for full trace
File "/home/homestar/dev/omegaconf/omegaconf/dictconfig.py", line 367, in __getitem__
return self._get_impl(key=key, default_value=_DEFAULT_MARKER_)
File "/home/homestar/dev/omegaconf/omegaconf/dictconfig.py", line 449, in _get_impl
return self._resolve_with_default(
File "/home/homestar/dev/omegaconf/omegaconf/basecontainer.py", line 99, in _resolve_with_default
resolved_node = self._maybe_resolve_interpolation(
File "/home/homestar/dev/omegaconf/omegaconf/base.py", line 718, in _maybe_resolve_interpolation
return self._resolve_interpolation_from_parse_tree(
File "/home/homestar/dev/omegaconf/omegaconf/base.py", line 583, in _resolve_interpolation_from_parse_tree
resolved = self.resolve_parse_tree(
File "/home/homestar/dev/omegaconf/omegaconf/base.py", line 763, in resolve_parse_tree
return visitor.visit(parse_tree)
File "/home/homestar/miniconda3/envs/omega310/lib/python3.10/site-packages/antlr4/tree/Tree.py", line 34, in visit
return tree.accept(self)
File "/home/homestar/dev/omegaconf/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 206, in accept
return visitor.visitConfigValue(self)
File "/home/homestar/dev/omegaconf/omegaconf/grammar_visitor.py", line 101, in visitConfigValue
return self.visit(ctx.getChild(0))
File "/home/homestar/miniconda3/envs/omega310/lib/python3.10/site-packages/antlr4/tree/Tree.py", line 34, in visit
return tree.accept(self)
File "/home/homestar/dev/omegaconf/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 342, in accept
return visitor.visitText(self)
File "/home/homestar/dev/omegaconf/omegaconf/grammar_visitor.py", line 297, in visitText
return self.visitInterpolation(c)
File "/home/homestar/dev/omegaconf/omegaconf/grammar_visitor.py", line 125, in visitInterpolation
return self.visit(ctx.getChild(0))
File "/home/homestar/miniconda3/envs/omega310/lib/python3.10/site-packages/antlr4/tree/Tree.py", line 34, in visit
return tree.accept(self)
File "/home/homestar/dev/omegaconf/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 1041, in accept
return visitor.visitInterpolationResolver(self)
File "/home/homestar/dev/omegaconf/omegaconf/grammar_visitor.py", line 174, in visitInterpolationResolver
for val, txt in self.visitSequence(maybe_seq):
File "/home/homestar/dev/omegaconf/omegaconf/grammar_visitor.py", line 266, in visitSequence
yield _get_value(self.visitElement(child)), child.getText()
File "/home/homestar/dev/omegaconf/omegaconf/grammar_visitor.py", line 119, in visitElement
return self.visit(ctx.getChild(0))
File "/home/homestar/miniconda3/envs/omega310/lib/python3.10/site-packages/antlr4/tree/Tree.py", line 34, in visit
return tree.accept(self)
File "/home/homestar/dev/omegaconf/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 1406, in accept
return visitor.visitPrimitive(self)
File "/home/homestar/dev/omegaconf/omegaconf/grammar_visitor.py", line 211, in visitPrimitive
return self._createPrimitive(ctx)
File "/home/homestar/dev/omegaconf/omegaconf/grammar_visitor.py", line 313, in _createPrimitive
return self.visitInterpolation(child)
File "/home/homestar/dev/omegaconf/omegaconf/grammar_visitor.py", line 125, in visitInterpolation
return self.visit(ctx.getChild(0))
File "/home/homestar/miniconda3/envs/omega310/lib/python3.10/site-packages/antlr4/tree/Tree.py", line 34, in visit
return tree.accept(self)
File "/home/homestar/dev/omegaconf/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 921, in accept
return visitor.visitInterpolationNode(self)
File "/home/homestar/dev/omegaconf/omegaconf/grammar_visitor.py", line 158, in visitInterpolationNode
return self.node_interpolation_callback(inter_key, self.memo)
File "/home/homestar/dev/omegaconf/omegaconf/base.py", line 744, in node_interpolation_callback
return self._resolve_node_interpolation(inter_key=inter_key, memo=memo)
File "/home/homestar/dev/omegaconf/omegaconf/base.py", line 670, in _resolve_node_interpolation
raise InterpolationToMissingValueError(
File "/home/homestar/dev/omegaconf/omegaconf/base.py", line 663, in _resolve_node_interpolation
parent, last_key, value = root_node._select_impl(
File "/home/homestar/dev/omegaconf/omegaconf/base.py", line 513, in _select_impl
value, _ = _select_one(
File "/home/homestar/dev/omegaconf/omegaconf/omegaconf.py", line 1168, in _select_one
raise MissingMandatoryValue(
omegaconf.errors.InterpolationToMissingValueError: MissingMandatoryValue while resolving interpolation: Missing mandatory value: a
full_key: c
object_type=dict
It looks like invoking OmegaConf.missing_keys causes 'c' to be dereferenced.
from omegaconf.
Jasha10
commented on June 10, 2024
Hi @kaimo455, thanks for taking the time to file a bug report and open a PR.
I see that your PR #1117 works around this crash by modifying the OmegaConf.missing_keys function to skip keys that are interpolations. This seems like a reasonable approach to me.
from omegaconf.
omry
commented on June 10, 2024
@odelalleau, thoughts?
I feel like interpolation should be evaluated and True should be returned they resolve to a missing key.
For custom resolvers, I think the behavior should be True if they are dereferencing a missing key (which I think triggers a MissingMandatoryValue somewhere).
from omegaconf.
odelalleau
commented on June 10, 2024
@odelalleau, thoughts? I feel like interpolation should be evaluated and True should be returned they resolve to a missing key. For custom resolvers, I think the behavior should be True if they are dereferencing a missing key (which I think triggers a MissingMandatoryValue somewhere).
#1117 looks good to me. Back then we settled on interpolations never being considered as missing (so for instance for the config given as example in this issue, OmegaConf.is_missing(cfg, c) returns False). And trying to access cfg.c raises a InterpolationToMissingValueError, not a MissingMandatoryValue.
I don't remember the details, but I believe that there were potential issues in "propagating" missing values through interpolations / resolvers.
from omegaconf.
Related Issues (20)
- Interpolation order problem HOT 1
- omegaconf 2.1.0 has a non-standard dependency specifier PyYAML>=5.1.*. HOT 9
- Pre-commit hooks are not working correctly
- Default Not Being Set HOT 3
- Work with dots in keynames HOT 6
- Unexpected behavior for multiline-strings HOT 16
- Escaped resolver modifying `_root_` doesn't work HOT 2
- Inconsistencies in recursive behavior in `set_readonly()` HOT 8
- Improve error message for relative interpolations HOT 1
- Make `select` and `oc.select` more robust HOT 2
- Question : why we need "???" and what's for ? HOT 2
- Add function to check if key exists HOT 4
- Interpolations that resolve to missing value `???` don't get passed to resolvers HOT 3
- `OmegaConf.resolve` should crash when a resolver input is missing
- [Feature Request] integration between omegaconf and AWS Sagemaker Estimator's hyperparameters
- Add support for dataclasses._MISSING_TYPE
- Resolve relative variables inside of a list HOT 2
- readthedocs: No such file or directory: 'java'
- Node interpolation not working HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from omegaconf.