Comments (15)
hasan7n
commented on September 16, 2024
1
sure!
name: mockprepsep
description: mockprepsep
authors:
- { name: "Hasan Kassem" }
platform:
accelerator_count: 0
docker:
# Image name.
image: hasan7/mock-prep-sep:0.0.0
# Docker build context relative to $MLCUBE_ROOT.
build_context: "../project"
# Docker file name within docker build context, default is `Dockerfile`.
build_file: "Dockerfile"
tasks:
prepare:
parameters:
inputs:
data_path: vids_files/
labels_path: labels_files/
parameters_file: parameters.yaml
outputs:
output_path: data/
output_labels_path: labels/
sanity_check:
parameters:
inputs:
data_path: data/
labels_path: labels/
parameters_file: parameters.yaml
statistics:
parameters:
inputs:
data_path: data/
labels_path: labels/
parameters_file: parameters.yaml
outputs:
output_path:
type: file
default: statistics.yaml
from mlcube.
hasan7n
commented on September 16, 2024
Trying singularity == 3.7.3 causes another error:
(medperf) hasan@hasan-HP-ZBook-15-G3:~/work/medperf_ws/tmp$ mlcube configure --platform singularity --mlcube mlcube.yaml
2023-08-02 07:33:44 hasan-HP-ZBook-15-G3 mlcube_singularity.singularity_run[89285] WARNING Config.merge singularity configuration not found in MLCube file (singularity={}).
Traceback (most recent call last):
File "/home/hasan/anaconda3/envs/medperf/bin/mlcube", line 8, in <module>
sys.exit(cli())
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/mlcube/__main__.py", line 242, in configure
runner_cls, mlcube_config = parse_cli_args(
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/mlcube/cli.py", line 72, in parse_cli_args
mlcube_config = MLCubeConfig.create_mlcube_config(
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/mlcube/config.py", line 154, in create_mlcube_config
runner_cls.CONFIG.merge(mlcube_config)
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/mlcube_singularity/singularity_run.py", line 85, in merge
if client.supports_fakeroot():
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/mlcube_singularity/singularity_client.py", line 107, in supports_fakeroot
and self.version >= semver.VersionInfo(major=3, minor=5)
TypeError: '>=' not supported between instances of 'Version' and 'Version'
from mlcube.
sergey-serebryakov
commented on September 16, 2024
@hasan7n Thanks for reporting! Correct parsing for singularity-ce version string is fixed in #326.
Edit: rephrased the above sentence stating that #326 PR fixes only the first bug.
from mlcube.
hasan7n
commented on September 16, 2024
@sergey-serebryakov Thanks for the quick fix!! Did you by chance see the additional comment about the version comparison issue?
from mlcube.
sergey-serebryakov
commented on September 16, 2024
@hasan7n This one should be fixed in #327.
from mlcube.
hasan7n
commented on September 16, 2024
Thanks! There is another issue now: the MLCube is requiring singularity.image to be defined even if my intend is to make MLCube convert the docker image to singularity. When I run mlcube run ... with singularity platform and an mlcube.yaml that contains only docker configuration, it throws the following error:
Traceback (most recent call last):
File "/home/hasan/anaconda3/envs/medperf/bin/mlcube", line 8, in <module>
sys.exit(cli())
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/mlcube/__main__.py", line 324, in run
runner_cls, mlcube_config = parse_cli_args(
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/mlcube/cli.py", line 72, in parse_cli_args
mlcube_config = MLCubeConfig.create_mlcube_config(
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/mlcube/config.py", line 158, in create_mlcube_config
runner_cls.CONFIG.validate(mlcube_config)
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/mlcube_singularity/singularity_run.py", line 118, in validate
validator.check_unknown_keys(Config.DEFAULT.keys()).check_values(
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/mlcube/validate.py", line 84, in check_values
self._validate_string_values(keys, **kwargs)
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/mlcube/validate.py", line 41, in _validate_string_values
value = self.config.get(key, None)
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 432, in get
return self._get_impl(key=key, default_value=default_value)
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 445, in _get_impl
return self._resolve_with_default(
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/omegaconf/basecontainer.py", line 63, in _resolve_with_default
resolved_node = self._maybe_resolve_interpolation(
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/omegaconf/base.py", line 616, in _maybe_resolve_interpolation
return self._resolve_interpolation_from_parse_tree(
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/omegaconf/base.py", line 482, in _resolve_interpolation_from_parse_tree
resolved = self.resolve_parse_tree(
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/omegaconf/base.py", line 662, in resolve_parse_tree
return visitor.visit(parse_tree)
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/antlr4/tree/Tree.py", line 34, in visit
return tree.accept(self)
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 205, in accept
return visitor.visitConfigValue(self)
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/omegaconf/grammar_visitor.py", line 101, in visitConfigValue
return self.visit(ctx.getChild(0))
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/antlr4/tree/Tree.py", line 34, in visit
return tree.accept(self)
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 339, in accept
return visitor.visitText(self)
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/omegaconf/grammar_visitor.py", line 298, in visitText
return self.visitInterpolation(c)
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/omegaconf/grammar_visitor.py", line 125, in visitInterpolation
return self.visit(ctx.getChild(0))
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/antlr4/tree/Tree.py", line 34, in visit
return tree.accept(self)
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 911, in accept
return visitor.visitInterpolationNode(self)
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/omegaconf/grammar_visitor.py", line 158, in visitInterpolationNode
return self.node_interpolation_callback(inter_key, self.memo)
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/omegaconf/base.py", line 643, in node_interpolation_callback
return self._resolve_node_interpolation(inter_key=inter_key, memo=memo)
File "/home/hasan/anaconda3/envs/medperf/lib/python3.9/site-packages/omegaconf/base.py", line 573, in _resolve_node_interpolation
raise InterpolationKeyError(f"Interpolation key '{inter_key}' not found")
omegaconf.errors.InterpolationKeyError: Interpolation key 'singularity.image' not found
from mlcube.
sergey-serebryakov
commented on September 16, 2024
@hasan7n Working on it. Could you please provide the configuration file itself? Will be helpful for testing.
from mlcube.
hasan7n
commented on September 16, 2024
@sergey-serebryakov It seems that the problem happens when I use network=none when invoking mlcube run. When I don't include the network param, everything works fine.
A hint: passing network is passing a dict singularity: network: none to the config parser, which is making the full runner args checks get executed.
from mlcube.
sergey-serebryakov
commented on September 16, 2024
@hasan7n Could you please rerun the mlcube run command with log level set to debug (mlcube --log-level=debug run ...) and paste here the entire output? Thanks.
from mlcube.
hasan7n
commented on September 16, 2024
@sergey-serebryakov I deleted the previous debug logs because they were from another mlcube version. Please find below the logs from the current master branch version:
2023-08-02 21:17:52 hasan-HP-ZBook-15-G3 root[44037] INFO cli setting log Level from CLI argument to 'DEBUG'.
2023-08-02 21:17:52 hasan-HP-ZBook-15-G3 mlcube.__main__[44037] DEBUG cli command=['/home/hasan/anaconda3/envs/testmlcube/bin/mlcube', '--log-level=debug', 'run', '--network', 'none', '--mlcube=/home/hasan/.medperf/localhost_8000/cubes/5/mlcube.yaml', '--task=prepare', '--platform=singularity', 'data_path=/home/hasan/.medperf/localhost_8000/demo/394580303e7f6348615f7c9feeed75dcbe6255f5/cases', 'labels_path=/home/hasan/.medperf/localhost_8000/demo/394580303e7f6348615f7c9feeed75dcbe6255f5/labels', 'output_path=/home/hasan/.medperf/localhost_8000/tmp31654/2820990230/data', 'output_labels_path=/home/hasan/.medperf/localhost_8000/tmp31654/2820990230/labels', 'Ptasks.prepare.parameters.input.data_path.opts=ro', 'Ptasks.prepare.parameters.input.labels_path.opts=ro']
2023-08-02 21:17:52 hasan-HP-ZBook-15-G3 mlcube.system_settings[44037] INFO SystemSettings.__init__ MLCube system settings file exists (/home/hasan/mlcube.yaml)
2023-08-02 21:17:52 hasan-HP-ZBook-15-G3 mlcube.platform[44037] INFO Platform.get_installed_runners found installed MLCube runner: platform=docker, pkg=mlcube_docker
2023-08-02 21:17:52 hasan-HP-ZBook-15-G3 mlcube.platform[44037] INFO Platform.get_installed_runners found installed MLCube runner: platform=singularity, pkg=mlcube_singularity
2023-08-02 21:17:52 hasan-HP-ZBook-15-G3 mlcube.__main__[44037] INFO run input_arg mlcube=/home/hasan/.medperf/localhost_8000/cubes/5/mlcube.yaml, platform=singularity, task=prepare, workspace=None, network=none, security=None, gpus=None, memory=None, cpu=None
2023-08-02 21:17:52 hasan-HP-ZBook-15-G3 mlcube.system_settings[44037] INFO SystemSettings.__init__ MLCube system settings file exists (/home/hasan/mlcube.yaml)
2023-08-02 21:17:52 hasan-HP-ZBook-15-G3 mlcube.config[44037] DEBUG MLCubeConfig.create_mlcube_config input_arg mlcube_config_file=/home/hasan/.medperf/localhost_8000/cubes/5/mlcube.yaml, mlcube_cli_args={'singularity': {'--network': 'none'}}, task_cli_args={'data_path': '/home/hasan/.medperf/localhost_8000/demo/394580303e7f6348615f7c9feeed75dcbe6255f5/cases', 'labels_path': '/home/hasan/.medperf/localhost_8000/demo/394580303e7f6348615f7c9feeed75dcbe6255f5/labels', 'output_path': '/home/hasan/.medperf/localhost_8000/tmp31654/2820990230/data', 'output_labels_path': '/home/hasan/.medperf/localhost_8000/tmp31654/2820990230/labels', 'Ptasks.prepare.parameters.input.data_path.opts': 'ro', 'Ptasks.prepare.parameters.input.labels_path.opts': 'ro'}, runner_config={'runner': 'singularity', 'image': '${singularity.image}', 'image_dir': '${runtime.workspace}/.image', 'singularity': 'singularity', 'build_args': '--fakeroot', 'run_args': '', 'build_file': 'Singularity.recipe', '--network': None, '--security': None, '--nv': None, '--vm-ram': None, '--vm-cpu': None}, workspace=None
Traceback (most recent call last):
File "/home/hasan/anaconda3/envs/testmlcube/bin/mlcube", line 33, in <module>
sys.exit(load_entry_point('mlcube', 'console_scripts', 'mlcube')())
File "/home/hasan/anaconda3/envs/testmlcube/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/home/hasan/anaconda3/envs/testmlcube/lib/python3.9/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/home/hasan/anaconda3/envs/testmlcube/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/hasan/anaconda3/envs/testmlcube/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/hasan/anaconda3/envs/testmlcube/lib/python3.9/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/home/hasan/anaconda3/envs/testmlcube/lib/python3.9/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/hasan/work/medperf_ws/mlcube/mlcube/mlcube/__main__.py", line 324, in run
runner_cls, mlcube_config = parse_cli_args(
File "/home/hasan/work/medperf_ws/mlcube/mlcube/mlcube/cli.py", line 72, in parse_cli_args
mlcube_config = MLCubeConfig.create_mlcube_config(
File "/home/hasan/work/medperf_ws/mlcube/mlcube/mlcube/config.py", line 158, in create_mlcube_config
runner_cls.CONFIG.validate(mlcube_config)
File "/home/hasan/work/medperf_ws/mlcube/runners/mlcube_singularity/mlcube_singularity/singularity_run.py", line 118, in validate
validator.check_unknown_keys(Config.DEFAULT.keys()).check_values(
File "/home/hasan/work/medperf_ws/mlcube/mlcube/mlcube/validate.py", line 84, in check_values
self._validate_string_values(keys, **kwargs)
File "/home/hasan/work/medperf_ws/mlcube/mlcube/mlcube/validate.py", line 41, in _validate_string_values
value = self.config.get(key, None)
File "/home/hasan/anaconda3/envs/testmlcube/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 432, in get
return self._get_impl(key=key, default_value=default_value)
File "/home/hasan/anaconda3/envs/testmlcube/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 445, in _get_impl
return self._resolve_with_default(
File "/home/hasan/anaconda3/envs/testmlcube/lib/python3.9/site-packages/omegaconf/basecontainer.py", line 63, in _resolve_with_default
resolved_node = self._maybe_resolve_interpolation(
File "/home/hasan/anaconda3/envs/testmlcube/lib/python3.9/site-packages/omegaconf/base.py", line 616, in _maybe_resolve_interpolation
return self._resolve_interpolation_from_parse_tree(
File "/home/hasan/anaconda3/envs/testmlcube/lib/python3.9/site-packages/omegaconf/base.py", line 482, in _resolve_interpolation_from_parse_tree
resolved = self.resolve_parse_tree(
File "/home/hasan/anaconda3/envs/testmlcube/lib/python3.9/site-packages/omegaconf/base.py", line 662, in resolve_parse_tree
return visitor.visit(parse_tree)
File "/home/hasan/anaconda3/envs/testmlcube/lib/python3.9/site-packages/antlr4/tree/Tree.py", line 34, in visit
return tree.accept(self)
File "/home/hasan/anaconda3/envs/testmlcube/lib/python3.9/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 205, in accept
return visitor.visitConfigValue(self)
File "/home/hasan/anaconda3/envs/testmlcube/lib/python3.9/site-packages/omegaconf/grammar_visitor.py", line 101, in visitConfigValue
return self.visit(ctx.getChild(0))
File "/home/hasan/anaconda3/envs/testmlcube/lib/python3.9/site-packages/antlr4/tree/Tree.py", line 34, in visit
return tree.accept(self)
File "/home/hasan/anaconda3/envs/testmlcube/lib/python3.9/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 339, in accept
return visitor.visitText(self)
File "/home/hasan/anaconda3/envs/testmlcube/lib/python3.9/site-packages/omegaconf/grammar_visitor.py", line 298, in visitText
return self.visitInterpolation(c)
File "/home/hasan/anaconda3/envs/testmlcube/lib/python3.9/site-packages/omegaconf/grammar_visitor.py", line 125, in visitInterpolation
return self.visit(ctx.getChild(0))
File "/home/hasan/anaconda3/envs/testmlcube/lib/python3.9/site-packages/antlr4/tree/Tree.py", line 34, in visit
return tree.accept(self)
File "/home/hasan/anaconda3/envs/testmlcube/lib/python3.9/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 911, in accept
return visitor.visitInterpolationNode(self)
File "/home/hasan/anaconda3/envs/testmlcube/lib/python3.9/site-packages/omegaconf/grammar_visitor.py", line 158, in visitInterpolationNode
return self.node_interpolation_callback(inter_key, self.memo)
File "/home/hasan/anaconda3/envs/testmlcube/lib/python3.9/site-packages/omegaconf/base.py", line 643, in node_interpolation_callback
return self._resolve_node_interpolation(inter_key=inter_key, memo=memo)
File "/home/hasan/anaconda3/envs/testmlcube/lib/python3.9/site-packages/omegaconf/base.py", line 573, in _resolve_node_interpolation
raise InterpolationKeyError(f"Interpolation key '{inter_key}' not found")
omegaconf.errors.InterpolationKeyError: Interpolation key 'singularity.image' not found
from mlcube.
sergey-serebryakov
commented on September 16, 2024
@hasan7n I suspect you might be using different versions of MLCube runtime and runners somehow. Could you please double check how you installed mlcube and mlcube singularity runner? You mlcube version seems to be 0.0.9.
from mlcube.
hasan7n
commented on September 16, 2024
@sergey-serebryakov That's right! Sorry for the inconvenience, I deleted the comment containing the logs from mlcube==0.0.9 and now the new comment contains the logs from the latest mlcube
from mlcube.
hasan7n
commented on September 16, 2024
I think the problem originates from here:
As I mentioned before, when passing --network=none to mlcube run, the error is happening. When not passing --network=none, the singularity-docker conversion works fine.
If you see in the lines of code above in singularity_run.py, the code inside the if statement if not s_cfg: is responsible for creating the modified mlcube config that will fill in singularity.image and make the subsequent mlcube code understand that it should convert the docker image to singularity.
When you pass --network none in the command line, the code here assumes that the user provides complete singularity config: s_cfg will be equal to {singularity: {network: none}}, and hence the code in the if statement is not executed.
What do you think? maybe changing the if statement to something like if singularity image not in s_cfg ? is that ok?
from mlcube.
hasan7n
commented on September 16, 2024
for reference, here are the most recent logs of the failing command (i.e. when passing --network none):
mlcube --log-level=debug run --network none --mlcube mlcube.yaml --platform=singularity --task=prepare data_path="/home/hasan/.medperf/localhost_8000/demo/394580303e7f6348615f7c9feeed75dcbe6255f5/cases" labels_path="/home/hasan/.medperf/localhost_8000/demo/394580303e7f6348615f7c9feeed75dcbe6255f5/labels" output_path="/home/hasan/.medperf/localhost_8000/tmp31654/2820990230/data" output_labels_path="/home/hasan/.medperf/localhost_8000/tmp31654/2820990230/labels" Ptasks.prepare.parameters.input.data_path.opts="ro" Ptasks.prepare.parameters.input.labels_path.opts="ro"
2023-08-03 05:26:32 hasan-HP-ZBook-15-G3 root[68179] INFO cli setting log Level from CLI argument to 'DEBUG'.
2023-08-03 05:26:32 hasan-HP-ZBook-15-G3 mlcube.__main__[68179] DEBUG cli command=['/home/hasan/anaconda3/envs/mlcube/bin/mlcube', '--log-level=debug', 'run', '--network', 'none', '--mlcube', 'mlcube.yaml', '--platform=singularity', '--task=prepare', 'data_path=/home/hasan/.medperf/localhost_8000/demo/394580303e7f6348615f7c9feeed75dcbe6255f5/cases', 'labels_path=/home/hasan/.medperf/localhost_8000/demo/394580303e7f6348615f7c9feeed75dcbe6255f5/labels', 'output_path=/home/hasan/.medperf/localhost_8000/tmp31654/2820990230/data', 'output_labels_path=/home/hasan/.medperf/localhost_8000/tmp31654/2820990230/labels', 'Ptasks.prepare.parameters.input.data_path.opts=ro', 'Ptasks.prepare.parameters.input.labels_path.opts=ro']
2023-08-03 05:26:32 hasan-HP-ZBook-15-G3 mlcube.system_settings[68179] INFO SystemSettings.__init__ MLCube system settings file exists (/home/hasan/mlcube.yaml)
2023-08-03 05:26:32 hasan-HP-ZBook-15-G3 mlcube.platform[68179] INFO Platform.get_installed_runners found installed MLCube runner (platform=docker, pkg=mlcube_docker, info={'name': 'mlcube_docker', 'location': '/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/mlcube_docker/__init__.py', 'version': '0.0.10rc0'})
2023-08-03 05:26:32 hasan-HP-ZBook-15-G3 mlcube.platform[68179] INFO Platform.get_installed_runners found installed MLCube runner (platform=singularity, pkg=mlcube_singularity, info={'name': 'mlcube_singularity', 'location': '/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/mlcube_singularity/__init__.py', 'version': '0.0.10rc0'})
2023-08-03 05:26:32 hasan-HP-ZBook-15-G3 mlcube.__main__[68179] INFO run input_arg mlcube=mlcube.yaml, platform=singularity, task=prepare, workspace=None, network=none, security=None, gpus=None, memory=None, cpu=None
2023-08-03 05:26:32 hasan-HP-ZBook-15-G3 mlcube.system_settings[68179] INFO SystemSettings.__init__ MLCube system settings file exists (/home/hasan/mlcube.yaml)
2023-08-03 05:26:32 hasan-HP-ZBook-15-G3 mlcube.config[68179] DEBUG MLCubeConfig.create_mlcube_config input_arg mlcube_config_file=/home/hasan/work/medperf_ws/tmp/mlcube.yaml, mlcube_cli_args={'singularity': {'--network': 'none'}}, task_cli_args={'data_path': '/home/hasan/.medperf/localhost_8000/demo/394580303e7f6348615f7c9feeed75dcbe6255f5/cases', 'labels_path': '/home/hasan/.medperf/localhost_8000/demo/394580303e7f6348615f7c9feeed75dcbe6255f5/labels', 'output_path': '/home/hasan/.medperf/localhost_8000/tmp31654/2820990230/data', 'output_labels_path': '/home/hasan/.medperf/localhost_8000/tmp31654/2820990230/labels', 'Ptasks.prepare.parameters.input.data_path.opts': 'ro', 'Ptasks.prepare.parameters.input.labels_path.opts': 'ro'}, runner_config={'runner': 'singularity', 'image': '${singularity.image}', 'image_dir': '${runtime.workspace}/.image', 'singularity': 'singularity', 'build_args': '--fakeroot', 'run_args': '', 'build_file': 'Singularity.recipe', '--network': None, '--security': None, '--nv': None, '--vm-ram': None, '--vm-cpu': None}, workspace=None
2023-08-03 05:26:32 hasan-HP-ZBook-15-G3 mlcube.config[68179] ERROR MLCubeConfig.create_mlcube_config failed to validate MLCube config ({'name': 'mockprepsep', 'description': 'mockprepsep', 'authors': [{'name': 'Hasan Kassem'}], 'platform': {'accelerator_count': 0}, 'docker': {'image': 'hasan7/mock-prep-sep:0.0.0', 'build_context': '../project', 'build_file': 'Dockerfile'}, 'tasks': {'prepare': {'parameters': {'inputs': {'data_path': 'vids_files/', 'labels_path': 'labels_files/', 'parameters_file': 'parameters.yaml'}, 'outputs': {'output_path': 'data/', 'output_labels_path': 'labels/'}}}, 'sanity_check': {'parameters': {'inputs': {'data_path': 'data/', 'labels_path': 'labels/', 'parameters_file': 'parameters.yaml'}}}, 'statistics': {'parameters': {'inputs': {'data_path': 'data/', 'labels_path': 'labels/', 'parameters_file': 'parameters.yaml'}, 'outputs': {'output_path': {'type': 'file', 'default': 'statistics.yaml'}}}}}, 'singularity': {'--network': 'none'}, 'runtime': {'root': '/home/hasan/work/medperf_ws/tmp', 'workspace': '${runtime.root}/workspace'}, 'runner': {'runner': 'singularity', 'image': '${singularity.image}', 'image_dir': '${runtime.workspace}/.image', 'singularity': 'singularity', 'build_args': '--fakeroot', 'run_args': '', 'build_file': 'Singularity.recipe', '--network': 'none', '--security': None, '--nv': None, '--vm-ram': None, '--vm-cpu': None}, 'workspace': '${runtime.root}/workspace'}): Interpolation key 'singularity.image' not found.
Traceback (most recent call last):
File "/home/hasan/anaconda3/envs/mlcube/bin/mlcube", line 8, in <module>
sys.exit(cli())
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/mlcube/__main__.py", line 324, in run
runner_cls, mlcube_config = parse_cli_args(
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/mlcube/cli.py", line 72, in parse_cli_args
mlcube_config = MLCubeConfig.create_mlcube_config(
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/mlcube/config.py", line 179, in create_mlcube_config
runner_cls.CONFIG.validate(mlcube_config)
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/mlcube_singularity/singularity_run.py", line 118, in validate
validator.check_unknown_keys(Config.DEFAULT.keys()).check_values(
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/mlcube/validate.py", line 84, in check_values
self._validate_string_values(keys, **kwargs)
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/mlcube/validate.py", line 41, in _validate_string_values
value = self.config.get(key, None)
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 432, in get
return self._get_impl(key=key, default_value=default_value)
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 445, in _get_impl
return self._resolve_with_default(
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/omegaconf/basecontainer.py", line 63, in _resolve_with_default
resolved_node = self._maybe_resolve_interpolation(
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/omegaconf/base.py", line 616, in _maybe_resolve_interpolation
return self._resolve_interpolation_from_parse_tree(
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/omegaconf/base.py", line 482, in _resolve_interpolation_from_parse_tree
resolved = self.resolve_parse_tree(
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/omegaconf/base.py", line 662, in resolve_parse_tree
return visitor.visit(parse_tree)
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/antlr4/tree/Tree.py", line 34, in visit
return tree.accept(self)
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 205, in accept
return visitor.visitConfigValue(self)
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/omegaconf/grammar_visitor.py", line 101, in visitConfigValue
return self.visit(ctx.getChild(0))
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/antlr4/tree/Tree.py", line 34, in visit
return tree.accept(self)
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 339, in accept
return visitor.visitText(self)
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/omegaconf/grammar_visitor.py", line 298, in visitText
return self.visitInterpolation(c)
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/omegaconf/grammar_visitor.py", line 125, in visitInterpolation
return self.visit(ctx.getChild(0))
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/antlr4/tree/Tree.py", line 34, in visit
return tree.accept(self)
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 911, in accept
return visitor.visitInterpolationNode(self)
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/omegaconf/grammar_visitor.py", line 158, in visitInterpolationNode
return self.node_interpolation_callback(inter_key, self.memo)
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/omegaconf/base.py", line 643, in node_interpolation_callback
return self._resolve_node_interpolation(inter_key=inter_key, memo=memo)
File "/home/hasan/anaconda3/envs/mlcube/lib/python3.9/site-packages/omegaconf/base.py", line 573, in _resolve_node_interpolation
raise InterpolationKeyError(f"Interpolation key '{inter_key}' not found")
omegaconf.errors.InterpolationKeyError: Interpolation key 'singularity.image' not found
@sergey-serebryakov note that I can see in the logs that the config contains a runner dict, and also contains a separate dict as follows: 'singularity': {'--network': 'none'}. Just want to point this out in case it is not expected (looks weird to me)
from mlcube.
hasan7n
commented on September 16, 2024
Moved to #343
from mlcube.
Related Issues (20)
- mlcube using singularity run --nv
- unclear scope for the project - training only, no serving component?
- Updated the dynabench doc to gather requirements and pick out a path forward HOT 1
- Singularity run arguments not working HOT 2
- Add Option to Adjust Hyperparams via Env Vars HOT 1
- Upgrade click to `>=8.0.0,<9.0.0` HOT 7
- SPython incompatible with windows. MLCube fails at import time HOT 1
- MLCube doesn't recognize parameter type on Windows HOT 1
- [Medperf] update cookiecutter for dependency resolution HOT 1
- Singualrity-compatible MLCube templates and examples
- MLCube needs to run docker images using host user credentials (name and group).
- GPUs parameter doesn't seem to be working HOT 3
- Add support for `mlcube --version` CLI command.
- mlcube configure with local docker image
- Run docker image with Singularity fails when providing runner cli args
- Assign CUDA_VISIBLE_DEVICES for Docker when passing GPUs
- change the name of `--mount` arg
- Runtime parameters of mlcube can't include the letter "h" HOT 3
- Update OmegaConf version
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mlcube.