isaac-sim / omniisaacgymenvs Goto Github PK
View Code? Open in Web Editor NEWReinforcement Learning Environments for Omniverse Isaac Gym
License: Other
Reinforcement Learning Environments for Omniverse Isaac Gym
License: Other
I've tried installing this on three Windows machines and same error each time
Hi, thank you for the amazing project. Can we create an underwater environment for the robot in the Omniverse Gym? Thank you.
Hi everyone,
I want to domain randomize the dynamic and static friction coefficients between the agent and the ground. I follow the instruction (modified the config file, added codes needed for DR), but it seams that the domain randomization does not work.
Here's how I check if it works or not:
So I want to know what is the correct way to randomize the material properties? Do I need to first assign a material to the meshes? Actually I tried to create and assign materials to the meshes, but when I started training, the friction parameters in the UI did not change at all (I enabled the interaction between the env and the UI);
And I noticed that in the config file, there's a parameter called "default_physics_material". I suppose if I did not create any material for my agent, this will be the material for every mesh. Is this correct?
Hello, I've been looking around the doc and examples for a while but I can't find an example. Is there a way to generate videos, either during training at some points, or at the end of training ?
I've seen that contrary to the original IsaacGym repo, the render "rgb" function has not been implemented here, but on the other hand there's a very recent commit called "d62b283 support headless render" which would be exactly what I want.
Even just outputing a single image would be fine, I could do the script later to make a video.
Thanks.
Hi,
I tried to train an agent performing a custom RL task, but I found that the performances and behaviors of the agent are different, depending on the training device (CPU or GPU).
In general, with all other parameters, as well as task codes the same, CPU trained agent has noticeable better performance that GPU trained agent (red: CPU trained, purple: GPU trained):
And I have defined a success condition, the success rate of CPU trained agent is far better (about 90%) than GPU (about 50%):
I have trained on CPU for many times, with only minor differences (changed some reward scales for example), and the success rates are always high (above 90%); But when I switched to GPU, I trained twice but got 50% success rates both.
Another thing I noticed is that the robots trained using GPU are shaking very obviously, which is not a desired behavior since the success condition requires it to have very low joint velocities, base linear&angular velocities. CPU trained agents have an extremely low velocities near success regions, but I checked the tensorboard, the average joint acceleration of CPU trained is actually larger than GPU (and I wonder why the joint acceleration is optimized more than other important reward terms):
So my questions are:
Thanks in advance!
Hi,
when I set enable_cameras=True
and try to run a training in headless
mode ($ isaacpython omniisaacgymenvs/scripts/rlgames_train.py task=AllegroHand num_envs=4 test=True headless=True
), I get this error
[Error] [carb.events.python] AttributeError: 'PlayButtonGroup' object has no attribute '_play_button'
At:
/home/johannes/.local/share/ov/pkg/isaac_sim-2022.2.1/kit/exts/omni.kit.widget.toolbar/omni/kit/widget/toolbar/builtin_tools/play_button_group.py(129): _on_timeline_event
/home/johannes/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.core/omni/isaac/core/simulation_context/simulation_context.py(549): play
/home/johannes/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.core/omni/isaac/core/simulation_context/simulation_context.py(387): initialize_physics
/home/johannes/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.core/omni/isaac/core/simulation_context/simulation_context.py(408): reset
/home/johannes/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.core/omni/isaac/core/world/world.py(281): reset
/home/johannes/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.gym/omni/isaac/gym/vec_env/vec_env_base.py(94): set_task
/home/johannes/work/isaac-sim-envs/omniisaacgymenvs/envs/vec_env_rlgames.py(51): set_task
/home/johannes/work/isaac-sim-envs/omniisaacgymenvs/utils/task_util.py(72): initialize_task
omniisaacgymenvs/scripts/rlgames_train.py(114): parse_hydra_configs
/home/johannes/.local/share/ov/pkg/isaac_sim-2022.2.1/kit/python/lib/python3.7/site-packages/hydra/core/utils.py(160): run_job
/home/johannes/.local/share/ov/pkg/isaac_sim-2022.2.1/kit/python/lib/python3.7/site-packages/hydra/_internal/hydra.py(106): run
/home/johannes/.local/share/ov/pkg/isaac_sim-2022.2.1/kit/python/lib/python3.7/site-packages/hydra/_internal/utils.py(381): <lambda>
/home/johannes/.local/share/ov/pkg/isaac_sim-2022.2.1/kit/python/lib/python3.7/site-packages/hydra/_internal/utils.py(211): run_and_report
/home/johannes/.local/share/ov/pkg/isaac_sim-2022.2.1/kit/python/lib/python3.7/site-packages/hydra/_internal/utils.py(378): _run_hydra
/home/johannes/.local/share/ov/pkg/isaac_sim-2022.2.1/kit/python/lib/python3.7/site-packages/hydra/main.py(52): decorated_main
omniisaacgymenvs/scripts/rlgames_train.py(143): <module>
The simulation continues to run and using something like this we can even get camera images, but I would still like to resolve the error to make sure we don't run into problem later on.
cam = rep.create.camera(
position=position, look_at=look_at, clipping_range=(0.01, 10)
)
RESOLUTION = self._cfg["video_resolution"]
self._render_product_path = rep.create.render_product(
cam, resolution=(RESOLUTION, RESOLUTION)
)
self.rgb = rep.AnnotatorRegistry.get_annotator("rgb")
self.rgb.attach([self._render_product_path])
self.rgb.get_data()
In in_hand_manipulation.py
, rand_delta
is calculated as follows:
rand_floats = torch_rand_float(-1.0, 1.0, (len(env_ids), self.num_hand_dofs * 2 + 5), device=self.device)
# ...
delta_max = self.hand_dof_upper_limits - self.hand_dof_default_pos
delta_min = self.hand_dof_lower_limits - self.hand_dof_default_pos
rand_delta = delta_min + (delta_max - delta_min) * rand_floats[:, 5:5+self.num_hand_dofs]
pos = self.hand_dof_default_pos + self.reset_dof_pos_noise * rand_delta
Let delta_min
and delta_max
. The line below maps rand_floats
from
rand_delta = delta_min + (delta_max - delta_min) * rand_floats[:, 5:5+self.num_hand_dofs]
However, rand_delta
should be in rand_floats
from
rand_delta = delta_min + (delta_max - delta_min) * 0.5 * (rand_floats[:, 5:5+self.num_hand_dofs] + 1.0)
or change the assignment of rand_floats
instead:
rand_floats = torch_rand_float(0.0, 1.0, (len(env_ids), self.num_hand_dofs * 2 + 5), device=self.device)
The issue also exists in the IsaacGymEnvs repo: shadow_hand.py and allegro_hand.py.
Hi,
I just wonder if there's good way to get the updated transformation info of Xforms in the viewport? I checked the RL examples (eg. humanoids) and found the transformation info seems to be frozen in the Property (the part within red box in the attached figure) when objects move. xform.GetLocalTransformation()
in code also return the same frozen transformation matrix even though it would work for Issac Sim.
If I use the go1.usd instead of Anymal in the Anymal task (I have carefully set up all the joints correctly) It falls right though the ground plane during training. If I use the a1.usd instead it doesn't fall though the ground plane during training but does fall though during testing. Ironically the go1.usd does the opposite. It falls though on training but works fine during testing. What's going on here? How to go about debugging the collision detection.
Only thing I can think of is there is the warning:
[Warning] [omni.physx.plugin] PhysicsUSD: Prim at path /World/envs/env_0/go1/RL_hip/collisions is using obsolete 'customGeometry' attribute. To toggle custom geometry for cylinders, use the physics settings option.
How do you set this setting?
How do I receive data in headless mode? It works when headless = False
Hi,
The standalone Isaac Gym provides a convenient domain randomization method by adding some parameters in config files. But for Omniverse Gym, I currently cannot find a method of doing so. Is there any workaround to achieve randomizations like adding noises to observations/states, and randomizing the friction/mass of an object?
Thanks!
Hi there,
I'm currently modifying the ShadowHand example you provide in order to control the hand using efforts rather than position control. To do that, I use the set_joint_efforts
method on the hand view (instance of ShadowHandView(ArticulationView)
) instead of setting position targets and using set_joint_position_targets
as done in the provided example.
This seems to work (the hand does move when using a random policy), but since the get_applied_joint_efforts
method isn't working (apparently a known bug that will be fixed soon, or is fixed and the problem is on my side) and returns nothing but zero efforts in the joints, I cannot really investigate and check that everything is working properly.
What I am curious about is the set_drive
call in the ShadowHand
class. This call sets the drive type to either linear or angular through pxr.UsdPhysics.DriveAPI
, as well as the target position or velocity to 0.0. Is this some kind of initialization step required with usd, or is this related to some behind-the-scene mechanism underlying the control of the hand, not depending on the type of control I choose using the Python API (e.g. by using switch_control_mode
), and preventing me from doing effort control on the hand?
I apologize if this is a silly question, and don't hesitate to tell me if it shoud be asked on the dev forum rather than here. Thanks!
Hi there,
1) Interpenetration
I don't know if this is a bug or not, but I faced an interpenetration issue while manipulating the Shadow Hand model. As you can see in the screenshot below, fingers sometimes interpenetrate each other, as well as the palm. I'd tend to say that this behavior is not wanted (maybe naively), since such interpenetration doesn't happen with a physical robot.
I looked in the config file of the task and the sim.physx.shadow_hand.enable_self_collisions
parameter was set to True
. Setting it to False
didn't fix the issue. Could you provide me with some info/workaround about that?
2) Missing visual
When inspecting the shadow_hand_instanceable.usd
asset, I noticed that the first finger knuckle mesh seems to be missing (as visible below). This is not critical, but if that's indeed the case, I thought it would be worth mentioning it here(?) . Note that I fixed the issue in my local usd file.
# I am using a conda environment which has been set up properly as per instructions
python -m torch.distributed.run --nnodes=1 --nproc_per_node=2 scripts/rlgames_train.py headless=True task=Ant multi_gpu=True
Any ideas? I would love to be able to use my other 4090!
stack trace:
python -m torch.distributed.run --nnodes=1 --nproc_per_node=2 scripts/rlgames_train.py headless=True task=Ant multi_gpu=True
WARNING:main:
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
/home/bizon/anaconda3/envs/isaac-sim/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'config': Defaults list is missing _self_
. See https://hydra.cc/docs/upgrades/1.0_to_1.1/default_composition_order for more information
warnings.warn(msg, UserWarning)
/home/bizon/anaconda3/envs/isaac-sim/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:412: UserWarning: In config: Invalid overriding of hydra/job_logging:
Default list overrides requires 'override' keyword.
See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/defaults_list_override for more information.
deprecation_warning(msg)
/home/bizon/anaconda3/envs/isaac-sim/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'config': Defaults list is missing _self_
. See https://hydra.cc/docs/upgrades/1.0_to_1.1/default_composition_order for more information
warnings.warn(msg, UserWarning)
/home/bizon/anaconda3/envs/isaac-sim/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:412: UserWarning: In config: Invalid overriding of hydra/job_logging:
Default list overrides requires 'override' keyword.
See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/defaults_list_override for more information.
deprecation_warning(msg)
[Warning] [omni.isaac.kit.simulation_app] Modules: ['omniisaacgymenvs', 'omniisaacgymenvs.utils', 'omniisaacgymenvs.utils.hydra_cfg', 'omniisaacgymenvs.utils.hydra_cfg.hydra_utils', 'omniisaacgymenvs.utils.hydra_cfg.reformat', 'omniisaacgymenvs.utils.rlgames', 'omniisaacgymenvs.utils.rlgames.rlgames_utils', 'omniisaacgymenvs.utils.task_util', 'omniisaacgymenvs.utils.config_utils', 'omniisaacgymenvs.utils.config_utils.path_utils', 'omniisaacgymenvs.envs', 'omniisaacgymenvs.envs.vec_env_rlgames', 'omni', 'omni.isaac.gym', 'omni.isaac.gym.vec_env', 'omni.isaac.gym.vec_env.vec_env_base', 'omni.kit.app._impl.telemetry_helpers', 'omni.isaac.gym.vec_env.vec_env_mt'] were loaded before SimulationApp was started and might not be loaded correctly.
[Warning] [omni.isaac.kit.simulation_app] Please check to make sure no extra omniverse or pxr modules are imported before the call to SimulationApp(...)
Starting kit application with the following args: ['/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py', '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/apps/omni.isaac.sim.python.gym.headless.kit', '--/app/tokens/exe-path=/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/kit', '--/persistent/app/viewport/displayOptions=3094', '--/rtx/materialDb/syncLoads=True', '--/rtx/hydra/materialSyncLoads=True--/omni.kit.plugin/syncUsdLoads=True', '--/app/renderer/resolution/width=1280', '--/app/renderer/resolution/height=720', '--/app/window/width=1440', '--/app/window/height=900', '--/renderer/multiGpu/enabled=True', '--/app/fastShutdown=True', '--ext-folder', '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts', '--ext-folder', '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/apps', '--/physics/cudaDevice=0', '--portable', '--no-window']
Passing the following args to the base kit application: ['headless=True', 'task=Ant', 'multi_gpu=True']
[Warning] [omni.isaac.kit.simulation_app] Modules: ['omniisaacgymenvs', 'omniisaacgymenvs.utils', 'omniisaacgymenvs.utils.hydra_cfg', 'omniisaacgymenvs.utils.hydra_cfg.hydra_utils', 'omniisaacgymenvs.utils.hydra_cfg.reformat', 'omniisaacgymenvs.utils.rlgames', 'omniisaacgymenvs.utils.rlgames.rlgames_utils', 'omniisaacgymenvs.utils.task_util', 'omniisaacgymenvs.utils.config_utils', 'omniisaacgymenvs.utils.config_utils.path_utils', 'omniisaacgymenvs.envs', 'omniisaacgymenvs.envs.vec_env_rlgames', 'omni', 'omni.isaac.gym', 'omni.isaac.gym.vec_env', 'omni.isaac.gym.vec_env.vec_env_base', 'omni.kit.app._impl.telemetry_helpers', 'omni.isaac.gym.vec_env.vec_env_mt'] were loaded before SimulationApp was started and might not be loaded correctly.
[Warning] [omni.isaac.kit.simulation_app] Please check to make sure no extra omniverse or pxr modules are imported before the call to SimulationApp(...)
Starting kit application with the following args: ['/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py', '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/apps/omni.isaac.sim.python.gym.headless.kit', '--/app/tokens/exe-path=/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/kit', '--/persistent/app/viewport/displayOptions=3094', '--/rtx/materialDb/syncLoads=True', '--/rtx/hydra/materialSyncLoads=True--/omni.kit.plugin/syncUsdLoads=True', '--/app/renderer/resolution/width=1280', '--/app/renderer/resolution/height=720', '--/app/window/width=1440', '--/app/window/height=900', '--/renderer/multiGpu/enabled=True', '--/app/fastShutdown=True', '--ext-folder', '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts', '--ext-folder', '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/apps', '--/physics/cudaDevice=0', '--portable', '--no-window']
Passing the following args to the base kit application: ['headless=True', 'task=Ant', 'multi_gpu=True']
[Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
[Info] [carb] Logging to file: /home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/logs/Kit/Isaac-Sim/2022.2/kit_20230504_201332.log
2023-05-05 06:13:32 [1ms] [Warning] [omni.ext.plugin] [ext: omni.drivesim.sensors.nv.radar] Extensions config 'extension.toml' doesn't exist '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.drivesim.sensors.nv.radar' or '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.drivesim.sensors.nv.radar/config'
2023-05-05 06:13:32 [1ms] [Warning] [omni.ext.plugin] [ext: omni.drivesim.sensors.nv.lidar] Extensions config 'extension.toml' doesn't exist '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.drivesim.sensors.nv.lidar' or '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.drivesim.sensors.nv.lidar/config'
[Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
[Info] [carb] Logging to file: /home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/logs/Kit/Isaac-Sim/2022.2/kit_20230504_201332.log
2023-05-05 06:13:32 [1ms] [Warning] [omni.ext.plugin] [ext: omni.drivesim.sensors.nv.lidar] Extensions config 'extension.toml' doesn't exist '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.drivesim.sensors.nv.lidar' or '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.drivesim.sensors.nv.lidar/config'
2023-05-05 06:13:32 [1ms] [Warning] [omni.ext.plugin] [ext: omni.drivesim.sensors.nv.radar] Extensions config 'extension.toml' doesn't exist '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.drivesim.sensors.nv.radar' or '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.drivesim.sensors.nv.radar/config'
[0.090s] [ext: omni.stats-0.0.0] startup
[0.090s] [ext: omni.stats-0.0.0] startup
[0.094s] [ext: omni.rtx.shadercache-1.0.0] startup
[0.094s] [ext: omni.rtx.shadercache-1.0.0] startup
[0.098s] [ext: omni.assets.plugins-0.0.0] startup
[0.099s] [ext: omni.gpu_foundation-0.0.0] startup
[0.097s] [ext: omni.assets.plugins-0.0.0] startup
[0.098s] [ext: omni.gpu_foundation-0.0.0] startup
2023-05-05 06:13:32 [96ms] [Warning] [carb] FrameworkImpl::setDefaultPlugin(client: omni.gpu_foundation_factory.plugin, desc : [carb::graphics::Graphics v2.11], plugin : carb.graphics-vulkan.plugin) failed. Plugin selection is locked, because the interface was previously acquired by:
2023-05-05 06:13:32 [95ms] [Warning] [carb] FrameworkImpl::setDefaultPlugin(client: omni.gpu_foundation_factory.plugin, desc : [carb::graphics::Graphics v2.11], plugin : carb.graphics-vulkan.plugin) failed. Plugin selection is locked, because the interface was previously acquired by:
[0.106s] [ext: carb.windowing.plugins-1.0.0] startup
[0.105s] [ext: carb.windowing.plugins-1.0.0] startup
2023-05-05 06:13:32 [105ms] [Warning] [carb.windowing-glfw.gamepad] Joystick with unknown remapping detected (will be ignored): ASRock LED Controller [03000000ce260000a201000010010000]
[0.113s] [ext: omni.kit.renderer.init-0.0.0] startup
2023-05-05 06:13:32 [105ms] [Warning] [carb.windowing-glfw.gamepad] Joystick with unknown remapping detected (will be ignored): ASRock LED Controller [03000000ce260000a201000010010000]
[0.112s] [ext: omni.kit.renderer.init-0.0.0] startup
|---------------------------------------------------------------------------------------------|
| Driver Version: 530.30.2 | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name | Active | LDA | GPU Memory | Vendor-ID | LUID |
| | | | | | Device-ID | UUID |
|---------------------------------------------------------------------------------------------|
| 0 | NVIDIA GeForce RTX 4090 | Yes: 1 | | 24810 MB | 10de | 0 |
| | | | | | 2684 | 527cadd8.. |
|---------------------------------------------------------------------------------------------|
| 1 | NVIDIA GeForce RTX 4090 | Yes: 0 | | 24810 MB | 10de | 0 |
| | | | | | 2684 | 7c9328c9.. |
|=============================================================================================|
| OS: Linux dl, Version: 5.15.0-69-generic
| XServer Vendor: The X.Org Foundation, XServer Version: 12013000 (1.20.13.0)
| Processor: AMD Ryzen Threadripper PRO 5975WX 32-Cores | Cores: Unknown | Logical: 64
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 257558 | Free Memory: 230801
| Total Page/Swap (MB): 2047 | Free Page/Swap: 2047
|---------------------------------------------------------------------------------------------|
|---------------------------------------------------------------------------------------------|
| Driver Version: 530.30.2 | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name | Active | LDA | GPU Memory | Vendor-ID | LUID |
| | | | | | Device-ID | UUID |
|---------------------------------------------------------------------------------------------|
| 0 | NVIDIA GeForce RTX 4090 | Yes: 1 | | 24810 MB | 10de | 0 |
| | | | | | 2684 | 527cadd8.. |
|---------------------------------------------------------------------------------------------|
| 1 | NVIDIA GeForce RTX 4090 | Yes: 0 | | 24810 MB | 10de | 0 |
| | | | | | 2684 | 7c9328c9.. |
|=============================================================================================|
| OS: Linux dl, Version: 5.15.0-69-generic
| XServer Vendor: The X.Org Foundation, XServer Version: 12013000 (1.20.13.0)
| Processor: AMD Ryzen Threadripper PRO 5975WX 32-Cores | Cores: Unknown | Logical: 64
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 257558 | Free Memory: 230801
| Total Page/Swap (MB): 2047 | Free Page/Swap: 2047
|---------------------------------------------------------------------------------------------|
2023-05-05 06:13:33 [873ms] [Warning] [gpu.foundation.plugin] IOMMU is enabled. Found 50 items in /sys/kernel/iommu_groups/.
2023-05-05 06:13:33 [875ms] [Warning] [gpu.foundation.plugin] IOMMU is enabled. Found 50 items in /sys/kernel/iommu_groups/.
2023-05-05 06:13:33 [873ms] [Warning] [gpu.foundation.plugin] On Linux only, CUDA and the display driver does not support IOMMU-enabled bare-metal PCIe peer to peer memory copy.
However, CUDA and the display driver does support IOMMU via VM pass through. As a consequence, users on Linux,
when running on a native bare metal system, should disable the IOMMU. The IOMMU should be enabled and the VFIO driver
be used as a PCIe pass through for virtual machines.
2023-05-05 06:13:33 [875ms] [Warning] [gpu.foundation.plugin] On Linux only, CUDA and the display driver does not support IOMMU-enabled bare-metal PCIe peer to peer memory copy.
However, CUDA and the display driver does support IOMMU via VM pass through. As a consequence, users on Linux,
when running on a native bare metal system, should disable the IOMMU. The IOMMU should be enabled and the VFIO driver
be used as a PCIe pass through for virtual machines.
2023-05-05 06:13:33 [875ms] [Warning] [gpu.foundation.plugin]
2023-05-05 06:13:33 [873ms] [Warning] [gpu.foundation.plugin]
2023-05-05 06:13:33 [875ms] [Warning] [gpu.foundation.plugin] -----------------------------------------------------------------------
2023-05-05 06:13:33 [873ms] [Warning] [gpu.foundation.plugin] -----------------------------------------------------------------------
2023-05-05 06:13:33 [875ms] [Warning] [gpu.foundation.plugin] !!!!! Local system validation failed! Incorrect configuration detected.
2023-05-05 06:13:33 [875ms] [Warning] [gpu.foundation.plugin] Summary below. Details above.
2023-05-05 06:13:33 [873ms] [Warning] [gpu.foundation.plugin] !!!!! Local system validation failed! Incorrect configuration detected.
2023-05-05 06:13:33 [875ms] [Warning] [gpu.foundation.plugin] -----------------------------------------------------------------------
2023-05-05 06:13:33 [873ms] [Warning] [gpu.foundation.plugin] Summary below. Details above.
2023-05-05 06:13:33 [875ms] [Warning] [gpu.foundation.plugin]
2023-05-05 06:13:33 [873ms] [Warning] [gpu.foundation.plugin] -----------------------------------------------------------------------
2023-05-05 06:13:33 [875ms] [Warning] [gpu.foundation.plugin] - ECC: OK
2023-05-05 06:13:33 [875ms] [Warning] [gpu.foundation.plugin] - IOMMU: FAILED
2023-05-05 06:13:33 [873ms] [Warning] [gpu.foundation.plugin]
2023-05-05 06:13:33 [875ms] [Warning] [gpu.foundation.plugin]
2023-05-05 06:13:33 [875ms] [Warning] [gpu.foundation.plugin] -----------------------------------------------------------------------
2023-05-05 06:13:33 [873ms] [Warning] [gpu.foundation.plugin] - ECC: OK
2023-05-05 06:13:33 [873ms] [Warning] [gpu.foundation.plugin] - IOMMU: FAILED
2023-05-05 06:13:33 [873ms] [Warning] [gpu.foundation.plugin]
2023-05-05 06:13:33 [873ms] [Warning] [gpu.foundation.plugin] -----------------------------------------------------------------------
[1.630s] [ext: omni.kit.pipapi-0.0.0] startup
[1.634s] [ext: omni.kit.pip_archive-0.0.0] startup
[1.641s] [ext: omni.kit.loop-isaac-1.0.0] startup
[1.641s] [ext: omni.kit.async_engine-0.0.0] startup
[1.643s] [ext: omni.kit.test-0.0.0] startup
[1.645s] [ext: omni.kit.pipapi-0.0.0] startup
[1.649s] [ext: omni.kit.pip_archive-0.0.0] startup
[1.656s] [ext: omni.kit.loop-isaac-1.0.0] startup
[1.656s] [ext: omni.kit.async_engine-0.0.0] startup
[1.658s] [ext: omni.kit.test-0.0.0] startup
[1.664s] [ext: omni.usd.config-1.0.0] startup
[1.669s] [ext: omni.usd.libs-1.0.0] startup
[1.680s] [ext: omni.usd.config-1.0.0] startup
[1.686s] [ext: omni.usd.libs-1.0.0] startup
[1.832s] [ext: omni.isaac.core_archive-2.0.1] startup
[1.840s] [ext: omni.pip.torch-1_13_0-0.1.4] startup
[1.842s] [ext: omni.isaac.ml_archive-1.1.0] startup
[1.843s] [ext: omni.client-0.1.1] startup
[1.843s] [ext: omni.isaac.core_archive-2.0.1] startup
[1.852s] [ext: omni.appwindow-1.0.1] startup
[1.850s] [ext: omni.pip.torch-1_13_0-0.1.4] startup
[1.854s] [ext: omni.kit.renderer.core-0.0.0] startup
[1.853s] [ext: omni.isaac.ml_archive-1.1.0] startup
[1.854s] [ext: omni.client-0.1.1] startup
[1.858s] [ext: omni.kit.renderer.capture-0.0.0] startup
[1.860s] [ext: omni.kit.renderer.imgui-0.0.0] startup
[1.862s] [ext: omni.appwindow-1.0.1] startup
[1.865s] [ext: omni.kit.renderer.core-0.0.0] startup
[1.874s] [ext: omni.kit.renderer.capture-0.0.0] startup
[1.877s] [ext: omni.kit.renderer.imgui-0.0.0] startup
[1.912s] [ext: carb.audio-0.1.0] startup
[1.929s] [ext: carb.audio-0.1.0] startup
[1.932s] [ext: omni.ui-2.12.23] startup
[1.939s] [ext: omni.ui-2.12.23] startup
[1.942s] [ext: omni.uiaudio-1.0.0] startup
[1.944s] [ext: omni.kit.mainwindow-1.0.0] startup
[1.946s] [ext: omni.kit.uiapp-0.0.0] startup
[1.946s] [ext: omni.usd.schema.physics-1.0.0] startup
[1.949s] [ext: omni.uiaudio-1.0.0] startup
[1.951s] [ext: omni.kit.mainwindow-1.0.0] startup
[1.953s] [ext: omni.kit.uiapp-0.0.0] startup
[1.953s] [ext: omni.usd.schema.physics-1.0.0] startup
[1.991s] [ext: omni.usd.schema.audio-0.0.0] startup
[1.999s] [ext: omni.usd.schema.audio-0.0.0] startup
[2.001s] [ext: omni.usd.schema.semantics-0.0.0] startup
[2.007s] [ext: omni.usd.schema.semantics-0.0.0] startup
[2.012s] [ext: omni.usd.schema.omniscripting-1.0.0] startup
[2.020s] [ext: omni.usd.schema.omniscripting-1.0.0] startup
[2.021s] [ext: omni.usd.schema.geospatial-0.0.0] startup
[2.029s] [ext: omni.usd.schema.geospatial-0.0.0] startup
[2.032s] [ext: omni.usd.schema.physx-0.0.0] startup
[2.042s] [ext: omni.usd.schema.physx-0.0.0] startup
[2.061s] [ext: omni.usd.schema.anim-0.0.0] startup
[2.074s] [ext: omni.usd.schema.anim-0.0.0] startup
[2.142s] [ext: omni.usd.schema.omnigraph-1.0.0] startup
[2.151s] [ext: omni.gpucompute.plugins-0.0.0] startup
[2.152s] [ext: omni.hydra.scene_delegate-0.3.0] startup
[2.155s] [ext: omni.usd.schema.omnigraph-1.0.0] startup
[2.159s] [ext: omni.hydra.usdrt_delegate-4.1.1] startup
[2.165s] [ext: omni.gpucompute.plugins-0.0.0] startup
[2.165s] [ext: omni.hydra.scene_delegate-0.3.0] startup
[2.172s] [ext: omni.hydra.usdrt_delegate-4.1.1] startup
[2.175s] [ext: omni.usdphysics-104.1.6-5.1] startup
[2.178s] [ext: omni.kit.window.popup_dialog-2.0.15] startup
[2.185s] [ext: omni.kit.actions.core-1.0.0] startup
[2.187s] [ext: omni.kit.audiodeviceenum-1.0.0] startup
[2.189s] [ext: omni.usdphysics-104.1.6-5.1] startup
[2.189s] [ext: omni.kit.widget.nucleus_connector-1.0.2] startup
[2.192s] [ext: omni.kit.window.popup_dialog-2.0.15] startup
[2.192s] [ext: omni.kit.commands-1.4.5] startup
[2.197s] [ext: omni.kvdb-0.0.0] startup
[2.199s] [ext: omni.kit.actions.core-1.0.0] startup
[2.202s] [ext: omni.kit.audiodeviceenum-1.0.0] startup
[2.200s] [ext: omni.convexdecomposition-104.1.6-5.1] startup
[2.204s] [ext: omni.kit.widget.nucleus_connector-1.0.2] startup
[2.204s] [ext: omni.usd.core-1.0.0] startup
[2.207s] [ext: omni.kit.commands-1.4.5] startup
[2.206s] [ext: omni.timeline-1.0.5] startup
[2.209s] [ext: omni.kit.numpy.common-0.1.0] startup
[2.212s] [ext: omni.kvdb-0.0.0] startup
[2.211s] [ext: omni.usd-1.6.23] startup
[2.215s] [ext: omni.convexdecomposition-104.1.6-5.1] startup
[2.219s] [ext: omni.usd.core-1.0.0] startup
[2.221s] [ext: omni.timeline-1.0.5] startup
[2.224s] [ext: omni.kit.numpy.common-0.1.0] startup
[2.226s] [ext: omni.usd-1.6.23] startup
[2.256s] [ext: omni.localcache-0.0.0] startup
[2.259s] [ext: omni.isaac.version-1.0.0] startup
[2.260s] [ext: omni.physx-104.1.6-5.1] startup
[2.271s] [ext: omni.localcache-0.0.0] startup
[2.274s] [ext: omni.isaac.version-1.0.0] startup
[2.275s] [ext: omni.physx-104.1.6-5.1] startup
2023-05-05 06:13:34 [2,282ms] [Warning] [omni.kvdb.plugin] wasn't able to load the meta database, trying to repair it ...
[2.295s] [ext: omni.kit.widget.prompt-1.0.4] startup
[2.296s] [ext: omni.isaac.dynamic_control-1.2.2] startup
[2.302s] [ext: omni.kit.menu.utils-1.4.7] startup
[2.373s] [ext: omni.kit.widget.path_field-2.0.4] startup
[2.374s] [ext: omni.kit.notification_manager-1.0.4] startup
[2.376s] [ext: omni.kit.search_core-1.0.1] startup
[2.377s] [ext: omni.kit.widget.browser_bar-2.0.4] startup
[2.378s] [ext: omni.kit.widget.versioning-1.3.8] startup
[2.379s] [ext: omni.kit.widget.filebrowser-2.3.8] startup
[2.394s] [ext: omni.kit.collaboration.channel_manager-1.0.9] startup
[2.397s] [ext: omni.iray.libs-0.0.0] startup
[2.400s] [ext: omni.kit.window.filepicker-2.7.8] startup
[2.450s] [ext: omni.kit.usd.layers-2.0.10] startup
[2.458s] [ext: omni.mdl.neuraylib-0.1.0] startup
[2.461s] [ext: omni.kit.window.file_importer-1.0.8] startup
[2.462s] [ext: omni.volume-0.1.0] startup
[2.465s] [ext: omni.mdl-0.1.0] startup
[2.482s] [ext: omni.kit.clipboard-1.0.0] startup
[2.483s] [ext: omni.hydra.rtx-0.1.0] startup
[2.489s] [ext: omni.kit.material.library-1.3.21] startup
[2.494s] [ext: omni.activity.core-1.0.1] startup
[2.500s] [ext: omni.kit.hydra_texture-1.0.11] startup
[2.506s] [ext: omni.kit.menu.create-1.0.8] startup
[2.507s] [ext: omni.kit.viewport.registry-104.0.2] startup
[2.510s] [ext: omni.kit.widget.viewport-104.1.15] startup
[2.511s] [ext: omni.kit.context_menu-1.5.10] startup
[2.515s] [ext: omni.kit.window.file_exporter-1.0.10] startup
[2.516s] [ext: omni.ui.scene-1.5.17] startup
[2.522s] [ext: omni.kit.window.property-1.8.2] startup
[2.523s] [ext: omni.kit.widget.stage-2.7.18] startup
[2.526s] [ext: omni.kit.viewport.window-104.1.18] startup
2023-05-05 06:13:35 [2,520ms] [Warning] [omni.kit.ui] Failed to set value for menu with path: Window/Viewport/Viewport Next 1. It doesn't exist.
[2.553s] [ext: omni.kit.property.usd-3.18.12] startup
[2.561s] [ext: omni.kit.viewport.utility-1.0.12] startup
[2.561s] [ext: omni.kit.widget.graph-1.5.3] startup
[2.592s] [ext: omni.inspect-1.0.1] startup
[2.595s] [ext: omni.kit.widget.text_editor-1.0.2] startup
[2.596s] [ext: omni.ui_query-1.1.1] startup
[2.599s] [ext: omni.kit.window.extensions-1.1.1] startup
[2.602s] [ext: omni.graph.core-2.64.7] startup
[2.605s] [ext: omni.kit.primitive.mesh-1.0.6] startup
[2.609s] [ext: omni.kit.ui_test-1.2.9] startup
[2.611s] [ext: omni.kit.widget.settings-1.0.1] startup
[2.612s] [ext: omni.graph.tools-1.17.0] startup
[2.626s] [ext: omni.kit.usd_undo-0.1.1] startup
[2.628s] [ext: omni.kit.stage_templates-1.1.11] startup
[2.630s] [ext: omni.kit.widget.searchfield-1.0.8] startup
[2.631s] [ext: omni.kit.widget.zoombar-1.0.4] startup
[2.631s] [ext: omni.kit.window.preferences-1.3.7] startup
[2.672s] [ext: omni.graph-1.50.1] startup
[2.785s] [ext: omni.debugdraw-0.1.1] startup
[2.791s] [ext: omni.kit.browser.core-2.2.1] startup
[2.796s] [ext: omni.graph.ui-1.24.1] startup
[2.825s] [ext: omni.kit.window.file-1.3.30] startup
[2.827s] [ext: omni.kit.browser.folder.core-1.7.1] startup
[2.829s] [ext: omni.graph.nodes-1.48.3] startup
[2.840s] [ext: omni.graph.action-1.31.0] startup
[2.847s] [ext: omni.kit.browser.sample-1.2.4] startup
[2.849s] [ext: omni.syntheticdata-0.2.4] startup
2023-05-05 06:13:35 [2,852ms] [Warning] [omni.syntheticdata.scripts.extension] SyntheticData extension needs at least a stageFrameHistoryCount of 3
[2.863s] [ext: omni.physics.tensors-0.1.0] startup
[2.870s] [ext: omni.warp-0.6.1] startup
Warp 0.6.1 initialized:
CUDA Toolkit: 11.5, Driver: 12.1
Devices:
"cpu" | x86_64
"cuda:0" | NVIDIA GeForce RTX 4090 (sm_89)
"cuda:1" | NVIDIA GeForce RTX 4090 (sm_89)
Kernel cache: /home/bizon/.cache/warp/0.6.1
2023-05-05 06:13:35 [2,874ms] [Warning] [omni.kvdb.plugin] repair failed
[2.881s] [ext: omni.kit.widget.prompt-1.0.4] startup
[2.882s] [ext: omni.isaac.dynamic_control-1.2.2] startup
[2.889s] [ext: omni.kit.menu.utils-1.4.7] startup
[2.960s] [ext: omni.kit.widget.path_field-2.0.4] startup
[2.961s] [ext: omni.kit.notification_manager-1.0.4] startup
[2.963s] [ext: omni.kit.search_core-1.0.1] startup
[2.965s] [ext: omni.kit.widget.browser_bar-2.0.4] startup
[2.965s] [ext: omni.kit.widget.versioning-1.3.8] startup
[2.967s] [ext: omni.kit.widget.filebrowser-2.3.8] startup
[2.981s] [ext: omni.kit.collaboration.channel_manager-1.0.9] startup
[2.984s] [ext: omni.iray.libs-0.0.0] startup
[2.986s] [ext: omni.kit.window.filepicker-2.7.8] startup
[3.036s] [ext: omni.kit.usd.layers-2.0.10] startup
[3.043s] [ext: omni.mdl.neuraylib-0.1.0] startup
[3.044s] [ext: omni.graph.scriptnode-0.9.3] startup
[3.047s] [ext: omni.kit.window.file_importer-1.0.8] startup
[3.047s] [ext: omni.volume-0.1.0] startup
[3.046s] [ext: omni.command.usd-1.0.1] startup
[3.050s] [ext: omni.mdl-0.1.0] startup
[3.049s] [ext: omni.physx.tensors-0.1.0] startup
[3.055s] [ext: omni.replicator.core-1.6.4] startup
[3.066s] [ext: omni.kit.clipboard-1.0.0] startup
[3.067s] [ext: omni.hydra.rtx-0.1.0] startup
[3.073s] [ext: omni.kit.material.library-1.3.21] startup
[3.078s] [ext: omni.activity.core-1.0.1] startup
2023-05-05 06:13:35 [3,069ms] [Warning] [omni.replicator.core.scripts.annotators] Annotator PostProcessDispatch is already registered, overwriting annotator template
[3.084s] [ext: omni.kit.hydra_texture-1.0.11] startup
[3.090s] [ext: omni.kit.menu.create-1.0.8] startup
[3.091s] [ext: omni.kit.viewport.registry-104.0.2] startup
[3.093s] [ext: omni.kit.widget.viewport-104.1.15] startup
[3.095s] [ext: omni.kit.context_menu-1.5.10] startup
[3.099s] [ext: omni.kit.window.file_exporter-1.0.10] startup
[3.100s] [ext: omni.ui.scene-1.5.17] startup
[3.105s] [ext: omni.kit.window.property-1.8.2] startup
[3.107s] [ext: omni.kit.widget.stage-2.7.18] startup
[3.110s] [ext: omni.kit.viewport.window-104.1.18] startup
2023-05-05 06:13:35 [3,104ms] [Warning] [omni.kit.ui] Failed to set value for menu with path: Window/Viewport/Viewport Next 1. It doesn't exist.
[3.136s] [ext: omni.kit.property.usd-3.18.12] startup
[3.141s] [ext: omni.isaac.core-1.42.0] startup
[3.144s] [ext: omni.kit.viewport.utility-1.0.12] startup
[3.145s] [ext: omni.kit.widget.graph-1.5.3] startup
[3.174s] [ext: omni.inspect-1.0.1] startup
[3.177s] [ext: omni.kit.widget.text_editor-1.0.2] startup
[3.178s] [ext: omni.ui_query-1.1.1] startup
[3.181s] [ext: omni.kit.window.extensions-1.1.1] startup
[3.184s] [ext: omni.graph.core-2.64.7] startup
[3.188s] [ext: omni.kit.primitive.mesh-1.0.6] startup
[3.191s] [ext: omni.kit.ui_test-1.2.9] startup
[3.193s] [ext: omni.kit.widget.settings-1.0.1] startup
[3.194s] [ext: omni.graph.tools-1.17.0] startup
[3.208s] [ext: omni.kit.usd_undo-0.1.1] startup
[3.209s] [ext: omni.kit.stage_templates-1.1.11] startup
[3.211s] [ext: omni.kit.widget.searchfield-1.0.8] startup
[3.212s] [ext: omni.kit.widget.zoombar-1.0.4] startup
[3.213s] [ext: omni.kit.window.preferences-1.3.7] startup
[3.214s] [ext: omni.graph.bundle.action-1.3.0] startup
[3.214s] [ext: omni.replicator.isaac-1.7.3] startup
[3.224s] [ext: omni.kit.window.title-1.1.2] startup
[3.225s] [ext: omni.isaac.kit-1.2.1] startup
[3.225s] [ext: omni.isaac.sim.python.gym.headless-2022.2.0] startup
[3.226s] Simulation App Starting
[3.253s] [ext: omni.graph-1.50.1] startup
[3.368s] [ext: omni.debugdraw-0.1.1] startup
[3.374s] [ext: omni.kit.browser.core-2.2.1] startup
[3.379s] [ext: omni.graph.ui-1.24.1] startup
[3.408s] [ext: omni.kit.window.file-1.3.30] startup
[3.410s] [ext: omni.kit.browser.folder.core-1.7.1] startup
[3.413s] [ext: omni.graph.nodes-1.48.3] startup
[3.423s] [ext: omni.graph.action-1.31.0] startup
[3.430s] [ext: omni.kit.browser.sample-1.2.4] startup
[3.432s] [ext: omni.syntheticdata-0.2.4] startup
2023-05-05 06:13:36 [3,435ms] [Warning] [omni.syntheticdata.scripts.extension] SyntheticData extension needs at least a stageFrameHistoryCount of 3
[3.446s] [ext: omni.physics.tensors-0.1.0] startup
[3.453s] [ext: omni.warp-0.6.1] startup
Warp 0.6.1 initialized:
CUDA Toolkit: 11.5, Driver: 12.1
Devices:
"cpu" | x86_64
"cuda:0" | NVIDIA GeForce RTX 4090 (sm_89)
"cuda:1" | NVIDIA GeForce RTX 4090 (sm_89)
Kernel cache: /home/bizon/.cache/warp/0.6.1
[3.623s] [ext: omni.graph.scriptnode-0.9.3] startup
[3.625s] [ext: omni.command.usd-1.0.1] startup
[3.628s] [ext: omni.physx.tensors-0.1.0] startup
[3.634s] [ext: omni.replicator.core-1.6.4] startup
2023-05-05 06:13:36 [3,648ms] [Warning] [omni.replicator.core.scripts.annotators] Annotator PostProcessDispatch is already registered, overwriting annotator template
[3.717s] [ext: omni.isaac.core-1.42.0] startup
[3.790s] [ext: omni.graph.bundle.action-1.3.0] startup
[3.790s] [ext: omni.replicator.isaac-1.7.3] startup
[3.800s] [ext: omni.kit.window.title-1.1.2] startup
[3.801s] [ext: omni.isaac.kit-1.2.1] startup
[3.801s] [ext: omni.isaac.sim.python.gym.headless-2022.2.0] startup
[3.801s] Simulation App Starting
[7.783s] app ready
[7.792s] app ready
[7.996s] Simulation App Startup Complete
task:
name: Ant
physics_engine: physx
env:
numEnvs: 4096
envSpacing: 5
episodeLength: 1000
enableDebugVis: False
clipActions: 1.0
powerScale: 0.5
controlFrequencyInv: 2
headingWeight: 0.5
upWeight: 0.1
actionsCost: 0.005
energyCost: 0.05
dofVelocityScale: 0.2
angularVelocityScale: 1.0
contactForceScale: 0.1
jointsAtLimitCost: 0.1
deathCost: -2.0
terminationHeight: 0.31
alive_reward_scale: 0.5
sim:
dt: 0.0083
use_gpu_pipeline: True
gravity: [0.0, 0.0, -9.81]
add_ground_plane: True
add_distant_light: True
use_flatcache: True
enable_scene_query_support: False
disable_contact_processing: False
enable_cameras: False
default_physics_material:
static_friction: 1.0
dynamic_friction: 1.0
restitution: 0.0
physx:
worker_thread_count: 4
solver_type: 1
use_gpu: True
solver_position_iteration_count: 4
solver_velocity_iteration_count: 0
contact_offset: 0.02
rest_offset: 0.0
bounce_threshold_velocity: 0.2
friction_offset_threshold: 0.04
friction_correlation_distance: 0.025
enable_sleeping: True
enable_stabilization: True
max_depenetration_velocity: 10.0
gpu_max_rigid_contact_count: 524288
gpu_max_rigid_patch_count: 81920
gpu_found_lost_pairs_capacity: 8192
gpu_found_lost_aggregate_pairs_capacity: 262144
gpu_total_aggregate_pairs_capacity: 8192
gpu_max_soft_body_contacts: 1048576
gpu_max_particle_contacts: 1048576
gpu_heap_capacity: 67108864
gpu_temp_buffer_capacity: 16777216
gpu_max_num_partitions: 8
Ant:
override_usd_defaults: False
enable_self_collisions: False
enable_gyroscopic_forces: True
solver_position_iteration_count: 4
solver_velocity_iteration_count: 0
sleep_threshold: 0.005
stabilization_threshold: 0.001
density: -1
max_depenetration_velocity: 10.0
train:
params:
seed: 42
algo:
name: a2c_continuous
model:
name: continuous_a2c_logstd
network:
name: actor_critic
separate: False
space:
continuous:
mu_activation: None
sigma_activation: None
mu_init:
name: default
sigma_init:
name: const_initializer
val: 0
fixed_sigma: True
mlp:
units: [256, 128, 64]
activation: elu
d2rl: False
initializer:
name: default
regularizer:
name: None
load_checkpoint: False
load_path:
config:
name: Ant
full_experiment_name: Ant
env_name: rlgpu
device: cuda:1
device_name: cuda:1
multi_gpu: True
ppo: True
mixed_precision: True
normalize_input: True
normalize_value: True
value_bootstrap: True
num_actors: 4096
reward_shaper:
scale_value: 0.01
normalize_advantage: True
gamma: 0.99
tau: 0.95
learning_rate: 0.0003
lr_schedule: adaptive
schedule_type: legacy
kl_threshold: 0.008
score_to_win: 20000
max_epochs: 500
save_best_after: 100
save_frequency: 50
grad_norm: 1.0
entropy_coef: 0.0
truncate_grads: True
e_clip: 0.2
horizon_length: 16
minibatch_size: 32768
mini_epochs: 4
critic_coef: 2
clip_value: True
seq_len: 4
bounds_loss_coef: 0.0001
task_name: Ant
experiment:
num_envs:
seed: 42
torch_deterministic: False
max_iterations:
physics_engine: physx
pipeline: gpu
sim_device: gpu
device_id: 1
rl_device: cuda:1
multi_gpu: True
num_threads: 4
solver_type: 1
test: False
checkpoint:
headless: True
enable_livestream: False
mt_timeout: 30
wandb_activate: False
wandb_group:
wandb_name: Ant
wandb_entity:
wandb_project: omniisaacgymenvs
Setting seed: 42
Sim params does not have attribute: physx
Sim params does not have attribute: Ant
Pipeline: GPU
Pipeline Device: cuda:1
Sim Device: GPU
[8.015s] Simulation App Startup Complete
task:
name: Ant
physics_engine: physx
env:
numEnvs: 4096
envSpacing: 5
episodeLength: 1000
enableDebugVis: False
clipActions: 1.0
powerScale: 0.5
controlFrequencyInv: 2
headingWeight: 0.5
upWeight: 0.1
actionsCost: 0.005
energyCost: 0.05
dofVelocityScale: 0.2
angularVelocityScale: 1.0
contactForceScale: 0.1
jointsAtLimitCost: 0.1
deathCost: -2.0
terminationHeight: 0.31
alive_reward_scale: 0.5
sim:
dt: 0.0083
use_gpu_pipeline: True
gravity: [0.0, 0.0, -9.81]
add_ground_plane: True
add_distant_light: True
use_flatcache: True
enable_scene_query_support: False
disable_contact_processing: False
enable_cameras: False
default_physics_material:
static_friction: 1.0
dynamic_friction: 1.0
restitution: 0.0
physx:
worker_thread_count: 4
solver_type: 1
use_gpu: True
solver_position_iteration_count: 4
solver_velocity_iteration_count: 0
contact_offset: 0.02
rest_offset: 0.0
bounce_threshold_velocity: 0.2
friction_offset_threshold: 0.04
friction_correlation_distance: 0.025
enable_sleeping: True
enable_stabilization: True
max_depenetration_velocity: 10.0
gpu_max_rigid_contact_count: 524288
gpu_max_rigid_patch_count: 81920
gpu_found_lost_pairs_capacity: 8192
gpu_found_lost_aggregate_pairs_capacity: 262144
gpu_total_aggregate_pairs_capacity: 8192
gpu_max_soft_body_contacts: 1048576
gpu_max_particle_contacts: 1048576
gpu_heap_capacity: 67108864
gpu_temp_buffer_capacity: 16777216
gpu_max_num_partitions: 8
Ant:
override_usd_defaults: False
enable_self_collisions: False
enable_gyroscopic_forces: True
solver_position_iteration_count: 4
solver_velocity_iteration_count: 0
sleep_threshold: 0.005
stabilization_threshold: 0.001
density: -1
max_depenetration_velocity: 10.0
train:
params:
seed: 42
algo:
name: a2c_continuous
model:
name: continuous_a2c_logstd
network:
name: actor_critic
separate: False
space:
continuous:
mu_activation: None
sigma_activation: None
mu_init:
name: default
sigma_init:
name: const_initializer
val: 0
fixed_sigma: True
mlp:
units: [256, 128, 64]
activation: elu
d2rl: False
initializer:
name: default
regularizer:
name: None
load_checkpoint: False
load_path:
config:
name: Ant
full_experiment_name: Ant
env_name: rlgpu
device: cuda:0
device_name: cuda:0
multi_gpu: True
ppo: True
mixed_precision: True
normalize_input: True
normalize_value: True
value_bootstrap: True
num_actors: 4096
reward_shaper:
scale_value: 0.01
normalize_advantage: True
gamma: 0.99
tau: 0.95
learning_rate: 0.0003
lr_schedule: adaptive
schedule_type: legacy
kl_threshold: 0.008
score_to_win: 20000
max_epochs: 500
save_best_after: 100
save_frequency: 50
grad_norm: 1.0
entropy_coef: 0.0
truncate_grads: True
e_clip: 0.2
horizon_length: 16
minibatch_size: 32768
mini_epochs: 4
critic_coef: 2
clip_value: True
seq_len: 4
bounds_loss_coef: 0.0001
task_name: Ant
experiment:
num_envs:
seed: 42
torch_deterministic: False
max_iterations:
physics_engine: physx
pipeline: gpu
sim_device: gpu
device_id: 0
rl_device: cuda:0
multi_gpu: True
num_threads: 4
solver_type: 1
test: False
checkpoint:
headless: True
enable_livestream: False
mt_timeout: 30
wandb_activate: False
wandb_group:
wandb_name: Ant
wandb_entity:
wandb_project: omniisaacgymenvs
Setting seed: 42
Sim params does not have attribute: physx
Sim params does not have attribute: Ant
Pipeline: GPU
Pipeline Device: cuda:0
Sim Device: GPU
Task Device: cuda:1
RL device: cuda:1
/home/bizon/anaconda3/envs/isaac-sim/lib/python3.7/site-packages/gym/spaces/box.py:84: UserWarning: WARN: Box bound precision lowered by casting to float32
logger.warn(f"Box bound precision lowered by casting to {self.dtype}")
Task Device: cuda:0
RL device: cuda:0
/home/bizon/anaconda3/envs/isaac-sim/lib/python3.7/site-packages/gym/spaces/box.py:84: UserWarning: WARN: Box bound precision lowered by casting to float32
logger.warn(f"Box bound precision lowered by casting to {self.dtype}")
[8.827s] [ext: omni.isaac.sim.python.gym.headless-2022.2.0] shutdown
[8.827s] [ext: omni.replicator.isaac-1.7.3] shutdown
[8.827s] [ext: omni.replicator.core-1.6.4] shutdown
2023-05-05 06:13:41 [8,820ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2023-05-05 06:13:41 [8,820ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2023-05-05 06:13:41 [8,820ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2023-05-05 06:13:41 [8,820ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2023-05-05 06:13:41 [8,820ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2023-05-05 06:13:41 [8,820ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2023-05-05 06:13:41 [8,820ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2023-05-05 06:13:41 [8,820ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2023-05-05 06:13:41 [8,821ms] [Warning] [carb] [Plugin: omni.replicator.core.plugin] Module /home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/extscache/omni.replicator.core-1.6.4+104.1.lx64.r.cp37/bin/libomni.replicator.core.plugin.so remained loaded after unload request
[8.833s] [ext: omni.syntheticdata-0.2.4] shutdown
[8.834s] [ext: omni.isaac.core-1.42.0] shutdown
[8.834s] [ext: omni.graph.scriptnode-0.9.3] shutdown
[8.834s] [ext: omni.warp-0.6.1] shutdown
[8.838s] [ext: omni.graph.bundle.action-1.3.0] shutdown
[8.838s] [ext: omni.graph.nodes-1.48.3] shutdown
[8.839s] [ext: omni.graph.action-1.31.0] shutdown
[8.839s] [ext: omni.graph.ui-1.24.1] shutdown
2023-05-05 06:13:41 [8,834ms] [Warning] [omni.kit.ui] Failed to remove menu with path: Window/Visual Scripting/Toolkit. It doesn't exist.
2023-05-05 06:13:41 [8,834ms] [Warning] [omni.kit.menu.utils.scripts.utils] omni.kit.menu.utils remove_menu_items "Create" failed list.remove(x): x not in list
2023-05-05 06:13:41 [8,834ms] [Warning] [omni.kit.ui] Failed to remove menu with path: Window/Visual Scripting/Toolkit. It doesn't exist.
2023-05-05 06:13:41 [8,834ms] [Warning] [omni.kit.ui] Failed to remove menu with path: Window/Visual Scripting/Node Description Editor. It doesn't exist.
[9.077s] [ext: omni.kit.viewport.utility-1.0.12] shutdown
[9.077s] [ext: omni.kit.browser.sample-1.2.4] shutdown
[9.078s] [ext: omni.kit.browser.folder.core-1.7.1] shutdown
[9.079s] [ext: omni.kit.browser.core-2.2.1] shutdown
[9.079s] [ext: omni.kit.viewport.window-104.1.18] shutdown
2023-05-05 06:13:41 [9,094ms] [Warning] [omni.ext._impl._internal] omni.kit.viewport.window-104.1.18 -> <class 'omni.kit.viewport.window.extension.ViewportWindowExtension'>: extension object is still alive, something holds a reference on it. References: ["[0]:type: <class 'frame'>, id: 334733824", "[1]:type: <class 'frame'>, id: 334582032", "[2]:type: <class 'frame'>, id: 111020256"]
[9.448s] [ext: omni.isaac.sim.python.gym.headless-2022.2.0] shutdown
[9.448s] [ext: omni.replicator.isaac-1.7.3] shutdown
[9.448s] [ext: omni.replicator.core-1.6.4] shutdown
2023-05-05 06:13:42 [9,441ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2023-05-05 06:13:42 [9,441ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2023-05-05 06:13:42 [9,441ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2023-05-05 06:13:42 [9,441ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2023-05-05 06:13:42 [9,441ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2023-05-05 06:13:42 [9,441ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2023-05-05 06:13:42 [9,441ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2023-05-05 06:13:42 [9,441ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
2023-05-05 06:13:42 [9,441ms] [Warning] [carb] [Plugin: omni.replicator.core.plugin] Module /home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/extscache/omni.replicator.core-1.6.4+104.1.lx64.r.cp37/bin/libomni.replicator.core.plugin.so remained loaded after unload request
[9.454s] [ext: omni.warp-0.6.1] shutdown
[9.458s] [ext: omni.kit.browser.sample-1.2.4] shutdown
[9.459s] [ext: omni.kit.browser.folder.core-1.7.1] shutdown
[9.459s] [ext: omni.kit.browser.core-2.2.1] shutdown
[9.459s] [ext: omni.syntheticdata-0.2.4] shutdown
[9.460s] [ext: omni.isaac.core-1.42.0] shutdown
[9.460s] [ext: omni.graph.scriptnode-0.9.3] shutdown
[9.460s] [ext: omni.graph.bundle.action-1.3.0] shutdown
[9.460s] [ext: omni.graph.nodes-1.48.3] shutdown
[9.461s] [ext: omni.graph.action-1.31.0] shutdown
[9.461s] [ext: omni.graph.ui-1.24.1] shutdown
2023-05-05 06:13:42 [9,456ms] [Warning] [omni.kit.ui] Failed to remove menu with path: Window/Visual Scripting/Toolkit. It doesn't exist.
2023-05-05 06:13:42 [9,456ms] [Warning] [omni.kit.menu.utils.scripts.utils] omni.kit.menu.utils remove_menu_items "Create" failed list.remove(x): x not in list
2023-05-05 06:13:42 [9,456ms] [Warning] [omni.kit.ui] Failed to remove menu with path: Window/Visual Scripting/Toolkit. It doesn't exist.
2023-05-05 06:13:42 [9,456ms] [Warning] [omni.kit.ui] Failed to remove menu with path: Window/Visual Scripting/Node Description Editor. It doesn't exist.
[9.697s] [ext: omni.kit.viewport.utility-1.0.12] shutdown
[9.697s] [ext: omni.kit.viewport.window-104.1.18] shutdown
2023-05-05 06:13:42 [9,713ms] [Warning] [omni.ext._impl._internal] omni.kit.viewport.window-104.1.18 -> <class 'omni.kit.viewport.window.extension.ViewportWindowExtension'>: extension object is still alive, something holds a reference on it. References: ["[0]:type: <class 'frame'>, id: 343270400", "[1]:type: <class 'frame'>, id: 343118608", "[2]:type: <class 'frame'>, id: 119418496"]
Error executing job with overrides: ['headless=True', 'task=Ant', 'multi_gpu=True']
Traceback (most recent call last):
File "scripts/rlgames_train.py", line 115, in parse_hydra_configs
task = initialize_task(cfg_dict, env)
File "/home/bizon/eric/OmniIsaacGymEnvs/omniisaacgymenvs/utils/task_util.py", line 72, in initialize_task
env.set_task(task=task, sim_params=sim_config.get_physics_params(), backend="torch", init_sim=init_sim)
File "/home/bizon/eric/OmniIsaacGymEnvs/omniisaacgymenvs/envs/vec_env_rlgames.py", line 51, in set_task
super().set_task(task, backend, sim_params, init_sim)
File "/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.isaac.gym/omni/isaac/gym/vec_env/vec_env_base.py", line 80, in set_task
self._world.reset()
File "/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.isaac.core/omni/isaac/core/world/world.py", line 285, in reset
task.post_reset()
File "/home/bizon/eric/OmniIsaacGymEnvs/omniisaacgymenvs/tasks/ant.py", line 86, in post_reset
LocomotionTask.post_reset(self)
File "/home/bizon/eric/OmniIsaacGymEnvs/omniisaacgymenvs/tasks/shared/locomotion.py", line 174, in post_reset
self.reset_idx(indices)
File "/home/bizon/eric/OmniIsaacGymEnvs/omniisaacgymenvs/tasks/shared/locomotion.py", line 125, in reset_idx
self.initial_dof_pos[env_ids] + dof_pos, self.dof_limits_lower, self.dof_limits_upper
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cuda:0)
Does this platform support video recording during training? This function is supported in IsaacGymEnv. Thanks!
Hello,
Thanks for providing this amazing repo.
Are there any plans to add/migrate the implementation of the HumanoidAMP task from IsaacGymEnvs repo?
If so, what's the timeline?
Hi,
What is the difference between this repo and IsaacGymEnvs?
Hi,
Thank you for your work๏ผ
Runs with an error when I make changes to the number of environments, whether the number of environments is larger or less than the default number, for example:
PYTHON_PATH scripts/rlgames_train.py task=Cartpole num_envs=64
I got this:
Error executing job with overrides: ['task=Cartpole', 'num_envs=64']
Traceback (most recent call last):
File "scripts/rlgames_train.py", line 109, in parse_hydra_configs
rlg_trainer.run()
File "scripts/rlgames_train.py", line 82, in run
'sigma': None
File "/home/jrl/.local/share/ov/pkg/isaac_sim-2022.1.0/kit/python/lib/python3.7/site-packages/rl_games/torch_runner.py", line 108, in run
self.run_train(args)
File "/home/jrl/.local/share/ov/pkg/isaac_sim-2022.1.0/kit/python/lib/python3.7/site-packages/rl_games/torch_runner.py", line 86, in run_train
agent = self.algo_factory.create(self.algo_name, base_name='run', params=self.params)
File "/home/jrl/.local/share/ov/pkg/isaac_sim-2022.1.0/kit/python/lib/python3.7/site-packages/rl_games/common/object_factory.py", line 15, in create
return builder(**kwargs)
File "/home/jrl/.local/share/ov/pkg/isaac_sim-2022.1.0/kit/python/lib/python3.7/site-packages/rl_games/torch_runner.py", line 37, in
self.algo_factory.register_builder('a2c_continuous', lambda **kwargs : a2c_continuous.A2CAgent(**kwargs))
File "/home/jrl/.local/share/ov/pkg/isaac_sim-2022.1.0/kit/python/lib/python3.7/site-packages/rl_games/algos_torch/a2c_continuous.py", line 18, in init
a2c_common.ContinuousA2CBase.init(self, base_name, params)
File "/home/jrl/.local/share/ov/pkg/isaac_sim-2022.1.0/kit/python/lib/python3.7/site-packages/rl_games/common/a2c_common.py", line 965, in init
A2CBase.init(self, base_name, params)
File "/home/jrl/.local/share/ov/pkg/isaac_sim-2022.1.0/kit/python/lib/python3.7/site-packages/rl_games/common/a2c_common.py", line 185, in init
assert(self.batch_size % self.minibatch_size == 0)
AssertionErrorSet the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
/home/jrl/.local/share/ov/pkg/isaac_sim-2022.1.0/python.sh: line 46: 8605 Segmentation fault (core dumped)$python_exe "$ @" $args
Since I set a smaller number of environments, it seems that the problem is not related to memory.
Can you give me some advice?
Can't solve the following syntax error. Running on Windows 11 cmd. I set PYTHON_PATH in the python.bat environment but now it's not letting me run rlgames_train.py
>>> PYTHON_PATH scripts/rlgames_train.py task=Cartpole
File "<stdin>", line 1
PYTHON_PATH scripts/rlgames_train.py task=Cartpole
^
SyntaxError: invalid syntax
Tried running this in and out of the python environment. Any help is greatly appreciated.
Hello,
I have a pick and place application, where a robot is grasping a cloth created using ClothPrim.
I started from isaac sim and i was able to grasp the cloth, and when i integrated everything inside this repository i have the following behavior.
The cloth parameters are exactly the same and i use the same robot model as well. So i don't know exactly what to do...
Hi, would it be possible for IsaacGymEnvs to be upgraded from gym to gymnasium? Gymnasium is the maintained version of openai gym and is compatible with current RL training libraries (rllib and tianshou have already migrated, and stable-baselines3 will soon).
This repository is currently listed in the gymnasium third party environments but we are cleaning the list up to only include maintained gymnasium-compatible repositories.
For information about upgrading and compatibility, see migration guide and gym compatibility. The main difference is the API has switched to returning truncated
and terminated
, rather than done
, in order to give more information and mitigate edge case issues.
Hi,
Thank you for your contribution.
I tested the Anymal environment and all parameters were left unchanged.
After training the reward reaches above 65, but when I test with the environment number set to 1, the reward does not go above 10 and the robot can not walk properly.
However, when num_envs > 1 everything is fine on testing. What leads to this problem, and does this cause danger to the real robot, even though the simulation is already fine?
Hi there,
As stated here, I'm currently modifying the ShadowHand example you provide in order to control the hand using efforts rather than position control, and I'd like to train RL models to achieve this using contact/force/tactile observations rather than "visual" ones (fingertip positions, angles, velocities, etc.) as done in the OpenAI examples for instance.
In order to simulate the pressure/tactile sensors of the real Shadow Hand, I want to attach a contact sensor (as defined here) to all fingertips of all envs, in order to get contact data at each step of my RL loop.
In the set_up_scene
method of my task class, I tried to create such sensors using the IsaacSensorCreateContactSensor
command with self.default_zero_env_path + "/shadow_hand/robot0_xxdistal"
as the prim path (xx being replaced by ff, mf, ...), and I then used the ContactSensorInterface
in get_observations
to get data from these sensors. I observed nothing but zero measurements, so I tried to visualize my sensors in the simulator by setting the visualize
flag to True when using the creation command, and I set the color
parameter to some kind of red. Turns out the sensors are located way above the hand (in the picture below, see the red dots outlining the spherical sensors), and do not move although the fingers do, which is not wanted.
When setting use_flatcache
to False in the config file, the hand appears standing vertically, and does not move anymore. However, the sensors' location matches in this case the position of the fingertips of this vertically standing hand.
I also tried to create a ContactSensor
object without the mentioned command, setting the prim path as stated before, and adding the sensor to the scene, but this was unsuccessful.
Could someone help me creating contact sensors attached to the controlled fingers? Thanks very much in advance!
on line 106 of simulation_context.py
if self._device is not None and "cuda" in self._device:
device_id = self._settings.get_as_int("/physics/cudaDevice")
self._device = f"cuda:{device_id}"
this will always return cuda:0 as far as I can tell. No way to pass in cuda:1 when constructing the context.
I have changed both device_id and rl_device in config.yaml in omniIsaacgymenvs and config.yaml in AnymalTerrain. So cuda: 1 seems to be used everywhere else but here.
Hi
For the Ant environment, if the training process is not perfect (in which all ants advance and there is no overlap with other environments), several environments cross and generate the following error.
2022-06-08 12:23:10 [191,143ms] [Error] [omni.physx.plugin] PhysX error: The application needs to increase PxgDynamicsMemoryConfig::totalAggregatePairsCapacity to 4120 , otherwise, the simulation will miss interactions
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpubroadphase/src/PxgAABBManager.cpp, LINE 1064
The problem seems to be in the way of filtering collisions between clones. This error only occurs in environments where there is a possibility of intersection, but not with static environments such as Cartpole or Allegro/Shadow-Hand
By changing some training configuration (AntPPO.yaml file), it is possible to reproduce the error:
normalize_input: False
normalize_value: False
schedule_type: standard
A possible solution is to increase the gpu_total_aggregate_pairs_capacity
but the most simple one is to increase the space between the environments (envSpacing
)
Thanks for the amazing project.
I want to write an environment that supports SMDP or HRL algorithms based on base_task.py.
Take HRL as an example, the RL controller may require each subenvironment in parallel to implement a skill primitive, but the step lengths of the different skill primitives are not the same, which results in all the subenvironments stepping at the same time but ending at different steps.
However, the subenvironment that finishes first continues to step according to the skill primitives provided by RL, regardless of whether the other subenvironments are still executing the previous skill primitives.
Can the current isaacgym environment achieve this asynchronous parallel effect?
Thanks for any suggestion :) @kellyguo11
Hi all,
I met one problem that the link's coordination keeps static when robot base is moving as the video.
$ PYTHON_PATH scripts/random_policy.py task=Ant num_envs=1
It causes a big problem when adding sensors to the link, for example the IMU sensor will always output lin_acc [0, 0, 9.8] and ang_vel [0, 0, 0] when adding to the torso.
Meanwhile I checked IMU example in Isaac Sim, and the coordinate worked perfectly as the video.
Failed to find the reason after checking codes. I really appreciate it if you can give some hints.
PYTHON_PATH -m pip install -e .
this command doesnt work and it gives the following : The system cannot find the path specified.
even that I can check it with doskey /m and it gives me the correct path "doskey PYTHON_PATH=C:\Users\user\AppData\Local\ov\pkg\isaac_sim-\python.bat $" where I changed user to the correct path name on my PC
Hi,
I encounter an issue when trying to run examples you provide that involve hand models, i.e. either the Allegro hand of the Shadow one. I tried to run several different commands, but I get the same error every time:
PxgCudaDeviceMemoryAllocator fail to allocate memory 2147483648 bytes!! Result = 2
I tried to run scripts/random_policy.py
and scripts/rlgames_train.py
, using num_envs=1
(and more, but I assume that 1 is the lighter choice in terms of memory), with test=True
(inference mode) as well as test=False
(training mode), and with headless=True
as well as headless=False
. For all these, I get the exact same error mentioned above (always the same number of bytes, i.e. 2 GB), followed by lots of PhysX errors and a Exception: Failed to create simulation view backend
exception at the end, and a segfault. The whole output is quite large, but I could paste it here if needed.
I tried running other examples (Anymal, AnymalTerrain, Cartpole, etc.) and they happen to work, even for multiple envs (e.g. Anymal with 50 robots). Bad luck, I'm working with robotic hands...
I did monitor my GPU to see what was happening concerning memory, and the GPU memory usage goes to a bit more than 4GB (for 6GB of available memory) then the program crashes. I assume the problem comes from that, since 2GB is thus more than available.
I don't really know where these 2GB come from, and why this seems to only happen with the two hands. This issue seems related to mine, if that can help.
I don't know if such an issue is expected considering my specs (see below), or if you experienced such problems on your end, but since I'm planning to use Isaac Gym to do RL with the shadow hand, I'd appreciate your opinion on this problem. Feel free to ask for more info, including full terminal output if needed.
Thank you in advance for your help!
My environment:
Commit: 1aaf354
Isaac Sim Version: 2022.2.0
OS: Ubuntu 20.04
GPU: RTX 3060
CUDA: 12.0
GPU Driver: 525.85.05
Hi,
Thanks for your amazing work!
I'm new to Isaac Sim, and I'm trying to train/test the ShadowHand and AllegroHand tasks on my Windows machine with the default parameters using the example commands:
PYTHON_PATH scripts/rlgames_train.py task=ShadowHand
and PYTHON_PATH scripts/random_policy.py task=ShadowHand
However, I always got the following warnings and nothing shows on Isaac Sim UI:
...
Pipeline: GPU
Pipeline Device: cuda:0
Sim Device: GPU
Obs type: full
Task Device: cuda:0
RL device: cuda:0
C:\Research\Omniverse\ov\pkg\isaac_sim-2022.1.1/exts/omni.isaac.ml_archive/pip_prebundle\gym\spaces\box.py:74: UserWarning: WARN: Box bound precision lowered by casting to float32
"Box bound precision lowered by casting to {}".format(self.dtype)
[15.269s] [ext: omni.physx.flatcache-1.4.15-5.1] startup
2022-10-13 01:01:26 [15,426ms] [Warning] [carb] Acquiring non optional plugin interface which is not listed as dependency: [omni::kit::IStageUpdate v1.0] (plugin: (null)), by client: omni.physx.flatcache.plugin. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client.
2022-10-13 01:01:29 [18,563ms] [Warning] [omni.client.plugin] Tick: authentication: Discovery(ws://localhost:3333): Auth/Credentials service not found!
2022-10-13 01:01:41 [30,303ms] [Warning] [omni.hydra.scene_delegate.plugin] Calling getBypassRenderSkelMeshProcessing for prim /World/envs/env_0/shadow_hand/robot0_mfproximal/visuals.proto_robot0_V_mfproximal_id0 that has not been populated
2022-10-13 01:01:42 [31,566ms] [Warning] [omni.physx.plugin] ParseFEMClothMaterial: UsdPrim doesn't have a PhysxSchemaPhysxDeformableSurfaceMaterialAPI2022-10-13 01:01:44 [32,999ms] [Warning] [omni.usd] Coding Error: in _ValidateEditPrim at line 1279 of C:\b\w\ca6c508eae419cf8\USD\pxr\usd\usd\stage.cpp -- Cannot create prim spec at path </__Master_4/robot0_C_forearm>; authoring to an instancing master is not allowed.
2022-10-13 01:01:44 [33,001ms] [Warning] [omni.usd] Warning: in AddAppliedSchema at line 300 of C:\b\w\ca6c508eae419cf8\USD\pxr\usd\usd\prim.cpp -- Unable to create primSpec at path </__Master_4/robot0_C_forearm> in edit target 'anon:00000189A4CFC0E0:World0.usd'. Failed to add applied API schema.
2022-10-13 01:01:44 [33,003ms] [Warning] [omni.usd] Coding Error: in _ValidateEditPrim at line 1279 of C:\b\w\ca6c508eae419cf8\USD\pxr\usd\usd\stage.cpp -- Cannot create prim spec at path </__Master_4/robot0_C_forearm>; authoring to an instancing master is not allowed.
2022-10-13 01:01:44 [33,003ms] [Warning] [omni.usd] Warning: in AddAppliedSchema at line 300 of C:\b\w\ca6c508eae419cf8\USD\pxr\usd\usd\prim.cpp -- Unable to create primSpec at path </__Master_4/robot0_C_forearm> in edit target 'anon:00000189A4CFC0E0:World0.usd'. Failed to add applied API schema.
... (above warnings repeat)
This problem didn't occur when I run other tasks such as Humanoid and Cartpole.
My system spec: Windows 10 + AMD Ryzen 9 5900HX + RTX3080 laptop
Any help will be greatly appreciated!
Hi, does anyone know how to get an rgb image from the camera that has already existed in an USD model within OmniIsaacGymEnvs?
the prim_path for the camera (at a fixed location) is '/World/envs/env_0/cabinet/Camera_Prim/Camera'. I have attached a shot to my isaac sim.
I will be appreciated if anyone can help me!
This error occurs when I tried to run the FrankaCabinet taks with headless mode. I didn't change the script under tasks/franka_cabinet.py. The command is as follows:
/home/midea/.local/share/ov/pkg/isaac_sim-*/python.sh omniisaacgymenvs/scripts/rlgames_train.py task=FrankaCabinet headless=True
Then I got:
[1.760s] [ext: omni.kit.window.title-1.1.2] startup
[1.761s] [ext: omni.isaac.gym-0.3.3] startup
[1.761s] [ext: omni.isaac.sim.python.gym.headless-2022.2.1] startup
[1.762s] Simulation App Starting
[2.268s] app ready
2023-04-19 12:38:12 [2,265ms] [Warning] [omni.client.plugin] HTTP Client: provider_http: CC-493: Request through cache failed. Retrying without cache for http://omniverse-content-production.s3-us-west-2.amazonaws.com/.cloudfront.toml
2023-04-19 12:38:12 [2,265ms] [Warning] [omni.client.plugin] HTTP Client: omniclient: CC-873: Bypassing cache until the application is restarted
[2.284s] Simulation App Startup Complete
Setting seed: 42
Sim params does not have attribute: physx
Sim params does not have attribute: franka
Sim params does not have attribute: cabinet
Sim params does not have attribute: prop
Pipeline: GPU
Pipeline Device: cuda:0
Sim Device: GPU
[2.309s] [ext: omni.inspect-1.0.1] startup
[2.316s] [ext: omni.kit.clipboard-1.0.0] startup
[2.320s] [ext: omni.kit.menu.create-1.0.8] startup
[2.327s] [ext: omni.volume-0.1.0] startup
[2.330s] [ext: omni.kit.context_menu-1.5.12] startup
[2.333s] [ext: omni.activity.core-1.0.1] startup
[2.335s] [ext: omni.hydra.rtx-0.1.0] startup
[2.342s] [ext: omni.debugdraw-0.1.1] startup
[2.348s] [ext: omni.kit.widget.stage-2.7.24] startup
[2.352s] [ext: omni.kit.window.property-1.8.2] startup
[2.354s] [ext: omni.kit.viewport.utility-1.0.14] startup
[2.355s] [ext: omni.kit.property.usd-3.18.17] startup
[2.362s] [ext: omni.kit.widget.text_editor-1.0.2] startup
[2.364s] [ext: omni.kit.widget.settings-1.0.1] startup
[2.365s] [ext: omni.kit.widget.graph-1.5.6-104_2] startup
[2.377s] [ext: omni.ui.scene-1.5.18] startup
[2.382s] [ext: omni.kit.window.preferences-1.3.8] startup
[2.480s] [ext: omni.kit.window.extensions-1.1.1] startup
[2.484s] [ext: omni.kit.widget.prompt-1.0.5] startup
[2.485s] [ext: omni.graph.ui-1.24.2] startup
[2.519s] [ext: omni.graph.scriptnode-0.10.0] startup
[2.521s] [ext: omni.graph.action-1.31.1] startup
[2.530s] [ext: omni.graph.bundle.action-1.3.0] startup
[2.530s] [ext: omni.syntheticdata-0.2.4] startup
2023-04-19 12:38:12 [2,527ms] [Warning] [omni.syntheticdata.scripts.extension] SyntheticData extension needs at least a stageFrameHistoryCount of 3
[2.546s] [ext: omni.command.usd-1.0.2] startup
[2.548s] [ext: omni.replicator.core-1.7.8] startup
2023-04-19 12:38:12 [2,553ms] [Warning] [omni.replicator.core.scripts.annotators] Annotator PostProcessDispatch is already registered, overwriting annotator template
[2.616s] [ext: omni.replicator.isaac-1.7.4] startup
Error executing job with overrides: ['task=FrankaCabinet', 'num_envs=1', 'headless=True']
2023-04-19 12:38:12 [2,822ms] [Warning] [omni.client.plugin] HTTP Client: provider_http: CC-493: Request through cache failed. Retrying without cache for http://omniverse-content-production.s3-us-west-2.amazonaws.com/
An error occurred during Hydra's exception formatting:
AssertionError()
Traceback (most recent call last):
File "/home/midea/.local/share/ov/pkg/isaac_sim-2022.2.1/kit/python/lib/python3.7/site-packages/hydra/_internal/utils.py", line 252, in run_and_report
assert mdl is not None
AssertionError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "omniisaacgymenvs/scripts/rlgames_train.py", line 143, in <module>
parse_hydra_configs()
File "/home/midea/.local/share/ov/pkg/isaac_sim-2022.2.1/kit/python/lib/python3.7/site-packages/hydra/main.py", line 52, in decorated_main
config_name=config_name,
File "/home/midea/.local/share/ov/pkg/isaac_sim-2022.2.1/kit/python/lib/python3.7/site-packages/hydra/_internal/utils.py", line 378, in _run_hydra
lambda: hydra.run(
File "/home/midea/.local/share/ov/pkg/isaac_sim-2022.2.1/kit/python/lib/python3.7/site-packages/hydra/_internal/utils.py", line 294, in run_and_report
raise ex
File "/home/midea/.local/share/ov/pkg/isaac_sim-2022.2.1/kit/python/lib/python3.7/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
return func()
File "/home/midea/.local/share/ov/pkg/isaac_sim-2022.2.1/kit/python/lib/python3.7/site-packages/hydra/_internal/utils.py", line 381, in <lambda>
overrides=args.overrides,
File "/home/midea/.local/share/ov/pkg/isaac_sim-2022.2.1/kit/python/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 111, in run
_ = ret.return_value
File "/home/midea/.local/share/ov/pkg/isaac_sim-2022.2.1/kit/python/lib/python3.7/site-packages/hydra/core/utils.py", line 233, in return_value
raise self._return_value
File "/home/midea/.local/share/ov/pkg/isaac_sim-2022.2.1/kit/python/lib/python3.7/site-packages/hydra/core/utils.py", line 160, in run_job
ret.return_value = task_function(task_cfg)
File "omniisaacgymenvs/scripts/rlgames_train.py", line 115, in parse_hydra_configs
task = initialize_task(cfg_dict, env)
File "/home/midea/wk/isaac_proj/OmniIsaacGymEnvs/omniisaacgymenvs/utils/task_util.py", line 41, in initialize_task
from omniisaacgymenvs.tasks.franka_drawer import FrankaDrawerTask
File "/home/midea/wk/isaac_proj/OmniIsaacGymEnvs/omniisaacgymenvs/tasks/franka_drawer.py", line 33, in <module>
from omni.isaac.franka import Franka
ModuleNotFoundError: No module named 'omni.isaac.franka'
2023-04-19 12:38:13 [3,097ms] [Warning] [carb.audio.context] 1 contexts were leaked
/home/midea/.local/share/ov/pkg/isaac_sim-2022.2.1/python.sh: line 41: 10311 Segmentation fault (core dumped) $python_exe "$@" $args
There was an error running python
But this error gone when I set headless=False
. How can I use headless mode with the task FrankaCabinet?
Hi,
I have tried the provided domain randomization code for the shadow hand task. I want to apply noises to object_view's material properties, especially restitution, on reset. I randomize the restitution of objects in two ranges, [0.0, 0.1]
or [0.9, 0.99]
, and launched two runs. However, the behavior of objects dropping to the ground seems identical.
More specifically, I have made the following modifications to the released code:
domain_randomization/randomization_params/rigid_prim_views/object_view/material_properties/on_reset
of the ShadowHand task config:direct
;[[0.7, 1, 0.0], [1.3, 1, 0.1]]
or [[0.7, 1, 0.9], [1.3, 1, 0.99]]
.task/shared/in_hand_manipulation.py
, I register a GeometryPrimView
for the ground plane, and apply a physics material with restitution 0.5
to it, during set_up_scene
.I launched two runs with the two different distribution parameters (one with restitution in [0.0, 0.1]
, and the other with restitution in [0.9, 0.99]
). I printed object_view._physics_view.get_material_properties()
, which is indeed some random value in [0.0, 0.1]
or [0.9, 0.99]
, but the behavior of the simulation environment seems identical for these two different settings of object restitutions. When objects drop to the ground from the hand, none of these two restitution ranges makes the object bounce more than the other.
Why are there identical behaviors under different restitution ranges? I have attached the modified code below. Is there something wrong with it?
python scripts/rlgames_train.py task=Cartpole
/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/extscache/omni.pip.torch-1_13_0-0.1.4+104.1.lx64/torch-1-13-0/torch/utils/tensorboard/init.py:5: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
tensorboard.version
/home/bizon/anaconda3/envs/issac_sim/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'config': Defaults list is missing _self_
. See https://hydra.cc/docs/upgrades/1.0_to_1.1/default_composition_order for more information
warnings.warn(msg, UserWarning)
/home/bizon/anaconda3/envs/issac_sim/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:412: UserWarning: In config: Invalid overriding of hydra/job_logging:
Default list overrides requires 'override' keyword.
See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/defaults_list_override for more information.
deprecation_warning(msg)
Error executing job with overrides: ['task=Cartpole']
Traceback (most recent call last):
File "scripts/rlgames_train.py", line 98, in parse_hydra_configs
env = VecEnvRLGames(headless=headless, sim_device=cfg.device_id, enable_livestream=cfg.enable_livestream, enable_viewport=enable_viewport)
TypeError: init() got an unexpected keyword argument 'enable_livestream'
I am running in a conda enviornment which is why I am using 'python' to launch.
Any ideas? I am completely stuck here.
Following the instructions for installing a conda enviorment. Here is what I typed:
(issac_sim) bizon@dl:/eric/OmniIsaacGymEnvs$ source ~/.local/share/ov/pkg/isaac_sim-2022.2.0/setup_conda_env.sh/eric/OmniIsaacGymEnvs$ alias PYTHON_PATH=~/.local/share/ov/pkg/isaac_sim-2022.2.0/python.sh
(issac_sim) bizon@dl:
(issac_sim) bizon@dl:/eric/OmniIsaacGymEnvs$ PYTHON_PATH -m pip install -e ./eric/OmniIsaacGymEnvs$ ls
Warning: running in conda env, please deactivate before executing this script
If conda is desired please source setup_conda_env.sh in your python 3.7 conda env and run python normally
/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/python.sh: line 41: /home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/python/bin/python3: Argument list too long
There was an error running python
(issac_sim) bizon@dl:
docker docs LICENSE.txt omniisaacgymenvs README.md setup.py
(issac_sim) bizon@dl:~/eric/OmniIsaacGymEnvs$ python
Python 3.7.16 (default, Jan 17 2023, 22:20:44)
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
quit()
Something seems to be going wrong in python.sh at the very last line. Any ideas? I am trying to move from issac gym to sim and I am completely stuck..
Is it possible to read out the depth information for each camera (mounted at the end effector of each manipulator) in headless mode?
I'm currently training an RL policy with 4096 robots in parallel. By activating the visualization (headless=False), the training process takes significantly longer.
Why is OmniIsaacGymEnvs (Omniverse Isaac Sim) version about 6x slower, compared with IsaacGymEnvs (IsaacGym Preview) version, with same hardware?
Tested on Humanoid environment, headless, with 8192 environments (RTX 3080 Laptop)
IsaacGymEnvs TPS: ~260k
OmniIsaacGymEnvs TPS: ~40k
It is also observed that OmniIsaacGymEnvs have only 5% GPU utilization, while IsaacGymEnvs reaches 70% GPU utilization
Test code for OmniIsaacGymEnvs
PYTHON_PATH scripts/random_policy.py task=Humanoid headless=True task.env.numEnvs=8192
(random_policy.py modified, added a TPS counter like below)
Test code for IsaacGymEnvs
import isaacgym
import isaacgymenvs
import torch
import time
device = "cuda:0"
env_num = 8192
total_steps = int(1e6)
envs = isaacgymenvs.make(
seed=0,
task="Humanoid",
num_envs=env_num,
sim_device=device,
rl_device=device,
headless=True
)
print("Observation space is", envs.observation_space)
print("Action space is", envs.action_space)
last_time = time.time()
last_steps = 0
obs = envs.reset()
for _ in range(total_steps):
obs, reward, done, info = envs.step(torch.rand((env_num,)+envs.action_space.shape, device=device))
last_steps += 1
if last_steps % 100 == 0:
cur_time = time.time()
tps = last_steps * env_num / (cur_time - last_time)
last_time = cur_time
last_steps = 0
print ("TPS {:.1f}".format(tps))
It seems that the contact forces of the Anymal's knees are not correct values, and in the task AnymalTerrain, the reset is not triggered while the knees contact with the terrain.
It is simple to reproduce the issue:
Change the defaultJointAngles in AnymalTerrain.yaml to:
LF_HAA: 0.03 # [rad]
LH_HAA: 0.03 # [rad]
RF_HAA: -0.03 # [rad]
RH_HAA: -0.03 # [rad]
LF_HFE: 0.4 # [rad]
LH_HFE: -0.4 # [rad]
RF_HFE: 0.4 # [rad]
RH_HFE: -0.4 # [rad]
LF_KFE: -2.8 # [rad]
LH_KFE: 2.8 # [rad]
RF_KFE: -2.8 # [rad]
RH_KFE: 2.8 # [rad]
Run the following command in the terminal:
python scripts/rlgames_train.py task=AnymalTerrain headless=false num_envs=2 test=True
We will see that the knees of the Anymal robots touches the ground but the env is not terminated, and I found that the returned value of self._anymals._knees.get_net_contact_forces(clone=False) is always 0.
Hi,
Thanks for providing this fantastic repo.
I am trying to setup a training task with deformable objects, but I found out that the deformable physics properties of the objects is not simulated during the training. But when I export the scene and simulate it in the new window, it is working. I tried to create a use.file with the deformable physics properties already set and also tried to define the deformable physics in the function set_up_scene in the task python file, but both resulted in the same issue.
Does the training environment actually support deformable physics simulation? If yes, how can I setup the correct files for that? If not, will this be considered in the future?
Thanks a lot!
When we load our own cabinet use model to train the policy, we meet some errors about omni.hydra with the following commands:
python omniisaacgymenvs/scripts/rlgames_train.py task=FrankaDrawer num_envs=1024 headless=False
From the output we can see the training is still running, but we wonder whether these errors will influence the final results of the training, e.g., the reward or the success rates.
We really appreciate you for helping us.
The output of error is:
2023-04-23 05:25:06 [26,702ms] [Error] [omni.hydra] UsdToMdl: Reached invalid assignment for parameter 'opacity_constant'. Tried to assign a 'int'(USD) to a 'float'(MDL).
The full output is:
[2.322s] Simulation App Starting
[4.564s] app ready
[4.659s] Simulation App Startup Complete
[5.270s] [ext: omni.physx.flatcache-104.2.4-5.1] startup
Passing the following args to the base kit application: ['task=FrankaDrawer', 'num_envs=1024', 'headless=False']
Warp 0.6.3 initialized:
CUDA Toolkit: 11.5, Driver: 12.0
Devices:
"cpu" | x86_64
"cuda:0" | NVIDIA RTX A6000 (sm_86)
Kernel cache: /home/rl01/.cache/warp/0.6.3
[4.614s] RTX ready
Setting seed: 42
Sim params does not have attribute: physx
Sim params does not have attribute: franka
Sim params does not have attribute: cabinet
Sim params does not have attribute: prop
Pipeline: GPU
Pipeline Device: cuda:0
Sim Device: GPU
Task Device: cuda:0
RL device: cuda:0
================= in franka drawer ===================
before set_task
in franka: tensor([0., 0., 0.]) tensor([0., 0., 0., 1.])
2023-04-23 05:25:02 [22,842ms] [Warning] [omni.hydra.scene_delegate.plugin] Calling getBypassRenderSkelMeshProcessing for prim /World/envs/env_0/franka/panda_link4/visuals.proto_visuals_id0 that has not been populated
2023-04-23 05:25:02 [23,087ms] [Warning] [carb.scenerenderer-rtx.plugin] Auto-enabling Sampled Direct Lighting because scene has 2049 lights (above 10 light threshold)
~*~*~*
~*~*~* Direct GPU Helper:
~*~*~* 2048 articulations, maxLinks=11
~*~*~*
2023-04-23 05:25:06 [26,702ms] [Error] [omni.hydra] UsdToMdl: Reached invalid assignment for parameter 'opacity_constant'. Tried to assign a 'int'(USD) to a 'float'(MDL).
2023-04-23 05:25:06 [26,704ms] [Error] [omni.hydra] UsdToMdl: Reached invalid assignment for parameter 'opacity_constant'. Tried to assign a 'int'(USD) to a 'float'(MDL).
...
2023-04-23 05:25:35 [55,957ms] [Error] [omni.hydra] UsdToMdl: Reached invalid assignment for parameter 'opacity_constant'. Tried to assign a 'int'(USD) to a 'float'(MDL).
self._drawers:<omni.isaac.core.prims.rigid_prim_view.RigidPrimView object at 0x7f28b414d810>
after set_task
Started to train
Exact experiment name requested from command line: FrankaDrawer
Box(-1.0, 1.0, (9,), float32) Box(-inf, inf, (23,), float32)
cuda:0
self.batch_size:32768, self.minibatch_size:2048
build mlp: 23
RunningMeanStd: (1,)
RunningMeanStd: (23,)
[2023-04-23 13:25:21] Running RL reset
fps step: 13540.9 fps step and policy inference: 13370.2 fps total: 12399.9 epoch: 1/1500
fps step: 14710.6 fps step and policy inference: 14516.9 fps total: 12807.5 epoch: 2/1500
fps step: 14519.3 fps step and policy inference: 14335.8 fps total: 13240.5 epoch: 3/1500
fps step: 13258.7 fps step and policy inference: 13102.7 fps total: 12144.6 epoch: 4/1500
fps step: 13544.6 fps step and policy inference: 13379.5 fps total: 12435.7 epoch: 5/1500
fps step: 13434.2 fps step and policy inference: 13271.2 fps total: 11878.2 epoch: 6/1500
Hi all,
Thanks for your awesome contribution.
When I'm trying to play the demo, it occurs the problem for both rlgames_train
and rlgames_train_mt
as:
$ PYTHON_PATH scripts/rlgames_train_mt.py task=Ant headless=True
/home/USER/.local/share/ov/pkg/isaac_sim-2022.1.0/python.sh: line 46: 60079 Segmentation fault (core dumped) $python_exe "$@" $args There was an error running python
I tried to build conda
environment as well, but the same Segmentation fault
error comes.
btw, random_policy
works well on my side.
PYTHON_PATH -m pip install -e .
is done as well.
Desktop: Ubuntu 18.04 + RTX 3080 GPU
Appreciate it if you can give some hints.
Fan
Hello,
Is there a way to randomize these parameters DURING SIMULATION ? I have tried to put this code in the pre_physics step but there is no effect on the simulation (even tho the prameter of the simulation is well setted, i have checked with the getter). Is this some jit magic ? is there a way around it ?
cfg = self._sim_config.parse_actor_config("myprim")
cfg["solver_position_iteration_count"] = 4 # my new values
cfg["solver_velocity_iteration_count"] = 1 # my new values
self._sim_config.apply_articulation_settings("myprim", get_prim_at_path(self.aru.prim_path), cfg)
Regards,
PC Configuration: Ubuntu 20.04, RTX 3060, RAM 64 gb, Cuda 11.4, Nvidia driver 470.141.03.
Note: For Cartpole and Ant simulation, same command works but not for Anymal.
I was trying the demo run for anymal robot and got the following error:
rror executing job with overrides: ['task=AnymalTerrain', 'num_envs=64', 'checkpoint=omniverse://localhost/NVIDIA/Assets/Isaac/2022.1/Isaac/Samples/OmniIsaacGymEnvs/Checkpoints/anymal_terrain.pth']
Traceback (most recent call last):
File "scripts/rlgames_play.py", line 67, in parse_hydra_configs
task = initialize_demo(cfg_dict, env)
File "/home/arghya/OmniIsaacGymEnvs/omniisaacgymenvs/utils/demo_util.py", line 46, in initialize_demo
env.set_task(task=task, sim_params=sim_config.get_physics_params(), backend="torch", init_sim=init_sim)
File "/home/arghya/OmniIsaacGymEnvs/omniisaacgymenvs/envs/vec_env_rlgames.py", line 51, in set_task
super().set_task(task, backend, sim_params, init_sim)
File "/home/arghya/.local/share/ov/pkg/isaac_sim-2022.1.1/exts/omni.isaac.gym/omni/isaac/gym/vec_env/vec_env_base.py", line 82, in set_task
self._world.reset()
File "/home/arghya/.local/share/ov/pkg/isaac_sim-2022.1.1/exts/omni.isaac.core/omni/isaac/core/world/world.py", line 281, in reset
SimulationContext.reset(self, soft=soft)
File "/home/arghya/.local/share/ov/pkg/isaac_sim-2022.1.1/exts/omni.isaac.core/omni/isaac/core/simulation_context/simulation_context.py", line 423, in reset
SimulationContext.initialize_physics(self)
File "/home/arghya/.local/share/ov/pkg/isaac_sim-2022.1.1/exts/omni.isaac.core/omni/isaac/core/simulation_context/simulation_context.py", line 403, in initialize_physics
self._physics_sim_view = omni.physics.tensors.create_simulation_view(self.backend)
File "/home/arghya/.local/share/ov/pkg/isaac_sim-2022.1.1/kit/extsPhysics/omni.physics.tensors-1.4.15-5.1/omni/physics/tensors/impl/api.py", line 11, in create_simulation_view
raise Exception("Failed to create simulation view backend")
Exception: Failed to create simulation view backend
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
/home/arghya/.local/share/ov/pkg/isaac_sim-2022.1.1/python.sh: line 40: 19204 Segmentation fault (core dumped) $python_exe "$@" $args
There was an error running python
Here is the full terminal output:
arghya@arghya-Pulse-GL66-12UEK:~/OmniIsaacGymEnvs/omniisaacgymenvs$ PYTHON_PATH scripts/rlgames_play.py task=AnymalTerrain num_envs=64 checkpoint=omniverse://localhost/NVIDIA/Assets/Isaac/2022.1/Isaac/Samples/OmniIsaacGymEnvs/Checkpoints/anymal_terrain.pth
/home/arghya/.local/share/ov/pkg/isaac_sim-2022.1.1/kit/python/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'config': Defaults list is missing `_self_`. See https://hydra.cc/docs/upgrades/1.0_to_1.1/default_composition_order for more information
warnings.warn(msg, UserWarning)
/home/arghya/.local/share/ov/pkg/isaac_sim-2022.1.1/kit/python/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:412: UserWarning: In config: Invalid overriding of hydra/job_logging:
Default list overrides requires 'override' keyword.
See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/defaults_list_override for more information.
deprecation_warning(msg)
[Warning] [omni.isaac.kit.simulation_app] Modules: ['omniisaacgymenvs', 'omniisaacgymenvs.utils', 'omniisaacgymenvs.utils.hydra_cfg', 'omniisaacgymenvs.utils.hydra_cfg.hydra_utils', 'omniisaacgymenvs.utils.hydra_cfg.reformat', 'omniisaacgymenvs.utils.demo_util', 'omniisaacgymenvs.utils.config_utils', 'omniisaacgymenvs.utils.config_utils.path_utils', 'omniisaacgymenvs.envs', 'omniisaacgymenvs.envs.vec_env_rlgames', 'omni.isaac.gym', 'omni.isaac.gym.vec_env', 'omni.isaac.gym.vec_env.vec_env_base', 'omni.isaac.gym.vec_env.vec_env_mt', 'omniisaacgymenvs.scripts', 'omniisaacgymenvs.scripts.rlgames_train', 'omniisaacgymenvs.utils.rlgames', 'omniisaacgymenvs.utils.rlgames.rlgames_utils', 'omniisaacgymenvs.utils.task_util'] were loaded before SimulationApp was started and might not be loaded correctly.
[Warning] [omni.isaac.kit.simulation_app] Please check to make sure no extra omniverse or pxr modules are imported before the call to SimulationApp(...)
Starting kit application with the fillowing args: ['/home/arghya/.local/share/ov/pkg/isaac_sim-2022.1.1/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py', '/home/arghya/.local/share/ov/pkg/isaac_sim-2022.1.1/apps/omni.isaac.sim.python.kit', '--/app/tokens/exe-path=/home/arghya/.local/share/ov/pkg/isaac_sim-2022.1.1/kit', '--/persistent/app/viewport/displayOptions=3094', '--/rtx/materialDb/syncLoads=True', '--/rtx/hydra/materialSyncLoads=True--/omni.kit.plugin/syncUsdLoads=True', '--/app/renderer/resolution/width=1280', '--/app/renderer/resolution/height=720', '--/app/window/width=1440', '--/app/window/height=900', '--/renderer/multiGpu/enabled=True', '--ext-folder', '/home/arghya/.local/share/ov/pkg/isaac_sim-2022.1.1/exts', '--ext-folder', '/home/arghya/.local/share/ov/pkg/isaac_sim-2022.1.1/apps', '--/physics/cudaDevice=0', '--portable']
Passing the following args to the base kit application: ['task=AnymalTerrain', 'num_envs=64', 'checkpoint=omniverse://localhost/NVIDIA/Assets/Isaac/2022.1/Isaac/Samples/OmniIsaacGymEnvs/Checkpoints/anymal_terrain.pth']
[Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
[Info] [carb] Logging to file: /home/arghya/.local/share/ov/pkg/isaac_sim-2022.1.1/kit/logs/Kit/Isaac-Sim/2022.1/kit_20220918_195809.log
2022-09-19 00:58:09 [4ms] [Warning] [omni.ext.plugin] [ext: omni.drivesim.sensors.nv.lidar] Extensions config 'extension.toml' doesn't exist '/home/arghya/.local/share/ov/pkg/isaac_sim-2022.1.1/exts/omni.drivesim.sensors.nv.lidar' or '/home/arghya/.local/share/ov/pkg/isaac_sim-2022.1.1/exts/omni.drivesim.sensors.nv.lidar/config'
[0.146s] [ext: omni.stats-0.0.0] startup
[0.156s] [ext: omni.gpu_foundation-0.0.0] startup
2022-09-19 00:58:09 [155ms] [Warning] [carb] FrameworkImpl::setDefaultPlugin(client: omni.gpu_foundation_factory.plugin, desc : [carb::graphics::Graphics v2.5], plugin : carb.graphics-vulkan.plugin) failed. Plugin selection is locked, because the interface was previously acquired by:
[0.162s] [ext: carb.windowing.plugins-1.0.0] startup
[0.168s] [ext: omni.assets.plugins-0.0.0] startup
[0.169s] [ext: omni.kit.renderer.init-0.0.0] startup
MESA-INTEL: warning: Performance support disabled, consider sysctl dev.i915.perf_stream_paranoid=0
2022-09-19 00:58:10 [1,588ms] [Warning] [gpu.foundation.plugin] Raytracing is not supported on GPU: 1. Skipping this device.
|---------------------------------------------------------------------------------------------|
| Driver Version: 470.141.3 | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name | Active | LDA | GPU Memory | Vendor-ID | LUID |
| | | | | | Device-ID | UUID |
|---------------------------------------------------------------------------------------------|
| 0 | NVIDIA GeForce RTX 3060 Laptop.. | Yes: 0 | | 6144 MB | 10de | 0 |
| | | | | | 2520 | f67dad92.. |
|---------------------------------------------------------------------------------------------|
| 1 | Intel(R) Graphics (ADL GT2) | | | 48011 MB | 8086 | 0 |
| | | | | | 46a6 | 42f3bec9.. |
|=============================================================================================|
| OS: Linux arghya-Pulse-GL66-12UEK, Version: 5.15.0-46-generic
| XServer Vendor: The X.Org Foundation, XServer Version: 12013000 (1.20.13.0)
| Processor: 12th Gen Intel(R) Core(TM) i7-12700H | Cores: Unknown | Logical: 20
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 64014 | Free Memory: 57490
| Total Page/Swap (MB): 9536 | Free Page/Swap: 9536
|---------------------------------------------------------------------------------------------|
2022-09-19 00:58:10 [1,692ms] [Warning] [gpu.foundation.plugin] Disabling OpenGL Interop. Device 0 mismatched with OpenGL device: Mesa Intel(R) Graphics (ADL GT2).
OpenGL can only run on the main GPU. It requires the app to run with activeDevice setting set to that GPU. On laptops, you may also need to specify which GPU or driver to run on in your system (Integrated or Discrete).
2022-09-19 00:58:10 [1,693ms] [Warning] [gpu.foundation.plugin] Realm: no OpenGL interop context.
2022-09-19 00:58:11 [1,739ms] [Warning] [carb.cudainterop.plugin] On Linux only, CUDA and the display driver does not support IOMMU-enabled bare-metal PCIe peer to peer memory copy.
However, CUDA and the display driver does support IOMMU via VM pass through. As a consequence, users on Linux,
when running on a native bare metal system, should disable the IOMMU. The IOMMU should be enabled and the VFIO driver
be used as a PCIe pass through for virtual machines.
[1.754s] [ext: omni.kit.pipapi-0.0.0] startup
[1.760s] [ext: omni.kit.pip_archive-0.0.0] startup
[1.763s] [ext: omni.isaac.core_archive-1.2.0] startup
[1.775s] [ext: omni.usd.config-1.0.0] startup
[1.777s] [ext: omni.usd.libs-1.0.0] startup
[1.877s] [ext: omni.kit.pip_torch-1_11_0-0.1.3] startup
[1.897s] [ext: omni.isaac.ml_archive-1.1.0] startup
[1.898s] [ext: omni.kit.loop-isaac-0.1.0] startup
[1.899s] [ext: omni.kit.async_engine-0.0.0] startup
[1.900s] [ext: omni.appwindow-1.0.0] startup
[1.904s] [ext: omni.client-0.1.0] startup
[1.911s] [ext: omni.kit.test-0.0.0] startup
[1.911s] [ext: omni.kit.renderer.core-0.0.0] startup
[1.985s] [ext: omni.ui-2.10.3] startup
[1.999s] [ext: carb.audio-0.1.0] startup
[2.021s] [ext: omni.kit.mainwindow-0.0.0] startup
[2.022s] [ext: omni.uiaudio-1.0.0] startup
[2.024s] [ext: omni.kit.uiapp-0.0.0] startup
[2.024s] [ext: omni.usd.schema.physics-1.0.0] startup
[2.062s] [ext: omni.usd.schema.audio-0.0.0] startup
[2.068s] [ext: omni.usd.schema.semantics-0.0.0] startup
[2.076s] [ext: omni.usd.schema.omnigraph-1.0.0] startup
[2.082s] [ext: omni.usd.schema.anim-0.0.0] startup
[2.110s] [ext: omni.kit.commands-1.2.2] startup
[2.113s] [ext: omni.timeline-1.0.2] startup
[2.115s] [ext: omni.hydra.scene_delegate-0.2.0] startup
[2.119s] [ext: omni.kit.audiodeviceenum-1.0.0] startup
[2.121s] [ext: omni.usd-1.5.3] startup
[2.448s] [ext: omni.kit.asset_converter-1.2.31] startup
[2.471s] [ext: omni.usd.schema.physx-0.0.0] startup
[2.529s] [ext: omni.usd.schema.isaac-0.2.0] startup
[2.556s] [ext: omni.usd.schema.forcefield-0.0.0] startup
[2.562s] [ext: omni.kvdb-0.0.0] startup
[2.564s] [ext: omni.usdphysics-1.4.15] startup
[2.566s] [ext: omni.graph.tools-1.4.0] startup
[2.580s] [ext: omni.localcache-0.0.0] startup
[2.582s] [ext: omni.kit.stage_templates-1.1.2] startup
[2.583s] [ext: omni.convexdecomposition-1.4.15] startup
[2.585s] [ext: omni.physics.tensors-0.1.0] startup
[2.591s] [ext: omni.physx-1.4.15-5.1] startup
[4.487s] [ext: omni.graph.core-2.29.1] startup
[4.491s] [ext: omni.kit.menu.utils-1.2.11] startup
[4.499s] [ext: omni.physx.tensors-0.1.0] startup
[4.503s] [ext: omni.graph-1.23.0] startup
[4.537s] [ext: omni.kit.numpy.common-0.1.0] startup
[4.539s] [ext: omni.kit.window.script_editor-1.6.2] startup
[4.547s] [ext: omni.kit.search_core-1.0.2] startup
[4.548s] [ext: omni.isaac.dynamic_control-1.1.0] startup
[4.554s] [ext: omni.kit.renderer.capture-0.0.0] startup
[4.557s] [ext: omni.kit.widget.filebrowser-2.2.27] startup
[4.563s] [ext: omni.kit.widget.path_field-2.0.3] startup
[4.563s] [ext: omni.kit.notification_manager-1.0.5] startup
[4.565s] [ext: omni.kit.widget.versioning-1.3.8] startup
[4.566s] [ext: omni.kit.widget.browser_bar-2.0.3] startup
[4.567s] [ext: omni.kit.window.popup_dialog-2.0.8] startup
[4.568s] [ext: omni.mdl.neuraylib-0.1.0] startup
[4.571s] [ext: omni.kit.window.filepicker-2.4.30] startup
OmniAssetFileFormat
[4.610s] [ext: omni.kit.menu.create-1.0.2] startup
[4.611s] [ext: omni.mdl-0.1.0] startup
[4.626s] [ext: omni.kit.window.file_importer-1.0.4] startup
[4.627s] [ext: omni.kit.window.file_exporter-1.0.4] startup
[4.627s] [ext: omni.kit.material.library-1.3.10] startup
[4.629s] [ext: omni.kit.window.drop_support-1.0.0] startup
[4.630s] [ext: omni.kit.window.file-1.3.16] startup
[4.631s] [ext: omni.kit.context_menu-1.3.9] startup
[4.634s] [ext: omni.kit.window.property-1.6.3] startup
[4.635s] [ext: omni.kit.window.content_browser-2.4.28] startup
[4.642s] [ext: omni.kit.widget.stage-2.6.15] startup
[4.646s] [ext: omni.isaac.version-1.0.0] startup
[4.646s] [ext: omni.kit.property.usd-3.14.9] startup
[4.668s] [ext: omni.kit.viewport.legacy_gizmos-1.0.0] startup
[4.671s] [ext: omni.hydra.rtx-0.1.0] startup
[4.678s] [ext: omni.renderer-rtx-0.0.0] startup
[4.678s] [ext: omni.hydra.engine.stats-1.0.0] startup
[4.683s] [ext: omni.debugdraw-0.1.0] startup
[4.687s] [ext: omni.kit.widget.settings-1.0.0] startup
[4.688s] [ext: omni.kit.window.viewport-0.0.0] startup
[7.739s] [ext: omni.kit.widget.prompt-1.0.1] startup
[7.739s] [ext: omni.kit.widget.graph-1.4.3] startup
[7.791s] [ext: omni.kit.window.preferences-1.2.1] startup
[7.814s] [ext: omni.ui_query-1.1.1] startup
[7.816s] [ext: omni.graph.ui-1.6.1] startup
[7.831s] [ext: omni.kit.ui_test-1.2.2] startup
[7.833s] [ext: omni.graph.action-1.18.0] startup
[7.841s] [ext: omni.kit.widget.searchfield-1.0.6] startup
[7.842s] [ext: omni.kit.usd_undo-0.1.0] startup
[7.843s] [ext: omni.graph.scriptnode-0.5.0] startup
[7.844s] [ext: omni.physx.commands-1.4.15-5.1] startup
[7.848s] [ext: omni.graph.nodes-1.26.0] startup
[7.857s] [ext: omni.command.usd-1.0.1] startup
[7.859s] [ext: omni.kit.window.extensions-1.1.0] startup
[7.863s] [ext: omni.syntheticdata-0.2.1] startup
[7.886s] [ext: omni.kit.primitive.mesh-1.0.0] startup
[7.888s] [ext: omni.warp-0.2.2] startup
Warp initialized:
Version: 0.2.2
CUDA device: NVIDIA GeForce RTX 3060 Laptop GPU
Kernel cache: /home/arghya/.cache/warp/0.2.2
[7.974s] [ext: omni.isaac.ui-0.2.1] startup
[7.976s] [ext: omni.replicator.core-1.4.3] startup
[8.093s] [ext: omni.isaac.core-1.24.3] startup
[8.148s] [ext: omni.physx.ui-1.4.15-5.1] startup
[8.194s] [ext: omni.kit.property.material-1.8.5] startup
[8.195s] [ext: omni.kit.window.toolbar-1.2.4] startup
[8.199s] [ext: omni.isaac.core_nodes-0.13.0] startup
[8.206s] [ext: omni.physx.demos-1.4.15-5.1] startup
[8.208s] [ext: omni.kit.property.physx-0.1.0] startup
2022-09-19 00:58:17 [8,240ms] [Warning] [omni.physx.plugin] Deprecated: getSimulationEventStream is deprecated, please use getSimulationEventStreamV2
[8.247s] [ext: omni.physx.tests-1.4.15-5.1] startup
[8.452s] [ext: omni.isaac.wheeled_robots-0.5.8] startup
[8.462s] [ext: omni.kit.menu.common-1.0.0] startup
[8.464s] [ext: omni.physx.vehicle-1.4.15-5.1] startup
[8.475s] [ext: omni.physx.cct-1.4.15-5.1] startup
[8.527s] [ext: omni.physx.camera-1.4.15-5.1] startup
[8.533s] [ext: omni.kit.widget.stage_icons-1.0.2] startup
[8.534s] [ext: omni.ui.scene-1.5.0] startup
[8.539s] [ext: omni.physx.bundle-1.4.15-5.1] startup
[8.539s] [ext: omni.kit.window.stage-2.3.7] startup
[8.541s] [ext: omni.replicator.composer-1.1.3] startup
[8.544s] [ext: omni.isaac.lula-1.1.0] startup
[8.552s] [ext: omni.rtx.window.settings-0.6.1] startup
[8.557s] [ext: omni.isaac.surface_gripper-0.1.2] startup
[8.560s] [ext: omni.isaac.motion_planning-0.2.0] startup
[8.566s] [ext: omni.rtx.settings.core-0.5.5] startup
[8.569s] [ext: omni.isaac.manipulators-1.0.1] startup
[8.570s] [ext: omni.isaac.motion_generation-3.6.1] startup
[8.574s] [ext: omni.kit.widget.zoombar-1.0.3] startup
[8.574s] [ext: omni.kit.graph.delegate.default-1.0.15] startup
[8.576s] [ext: omni.isaac.franka-0.3.0] startup
[8.576s] [ext: omni.kit.browser.core-2.0.12] startup
[8.580s] [ext: omni.kit.graph.editor.core-1.3.3] startup
[8.582s] [ext: omni.kit.graph.usd.commands-1.1.0] startup
[8.582s] [ext: omni.kit.browser.folder.core-1.1.13] startup
[8.584s] [ext: omni.kit.graph.widget.variables-2.0.2] startup
[8.585s] [ext: omni.kit.graph.delegate.modern-1.6.0] startup
[8.586s] [ext: omni.kit.selection-0.1.0] startup
[8.587s] [ext: omni.isaac.debug_draw-0.1.2] startup
[8.591s] [ext: omni.graph.window.core-1.23.4] startup
[8.595s] [ext: omni.graph.instancing-1.1.4] startup
[8.600s] [ext: omni.kit.menu.edit-1.0.6] startup
[8.601s] [ext: omni.isaac.isaac_sensor-1.0.2] startup
2022-09-19 00:58:17 [8,601ms] [Warning] [omni.physx.plugin] Deprecated: getSimulationEventStream is deprecated, please use getSimulationEventStreamV2
[8.609s] [ext: omni.graph.tutorials-1.1.2] startup
[8.617s] [ext: omni.graph.window.action-1.3.8] startup
[8.619s] [ext: omni.kit.widget.live-0.1.0] startup
[8.622s] [ext: omni.kit.widget.layers-1.5.17] startup
[8.630s] [ext: omni.graph.bundle.action-1.0.0] startup
[8.630s] [ext: omni.isaac.range_sensor-0.4.3] startup
[8.647s] [ext: omni.kit.property.layer-1.1.2] startup
[8.649s] [ext: omni.replicator.isaac-1.3.2] startup
[8.657s] [ext: omni.isaac.kit-0.2.1] startup
[8.658s] [ext: omni.graph.window.generic-1.3.8] startup
[8.658s] [ext: omni.isaac.utils-0.1.11] startup
[8.661s] [ext: omni.kit.property.audio-1.0.5] startup
[8.662s] [ext: omni.kit.property.skel-1.0.1] startup
[8.663s] [ext: omni.kit.property.render-1.1.0] startup
[8.664s] [ext: omni.kit.property.camera-1.0.3] startup
[8.664s] [ext: omni.kit.property.geometry-1.2.0] startup
[8.666s] [ext: omni.kit.property.light-1.0.5] startup
[8.666s] [ext: omni.kit.property.transform-1.0.2] startup
[8.668s] [ext: omni.isaac.universal_robots-0.3.0] startup
[8.669s] [ext: omni.isaac.occupancy_map-0.2.4] startup
[8.679s] [ext: omni.kit.window.console-0.2.0] startup
[8.683s] [ext: omni.kit.window.status_bar-0.1.1] startup
[8.686s] [ext: omni.kit.property.bundle-1.2.4] startup
[8.687s] [ext: omni.kit.menu.file-1.0.8] startup
[8.688s] [ext: omni.kit.manipulator.viewport-1.0.6] startup
[8.689s] [ext: omni.isaac.urdf-0.4.0] startup
[8.708s] [ext: omni.isaac.dofbot-0.3.0] startup
[8.708s] [ext: omni.kit.window.title-1.1.2] startup
[8.709s] [ext: omni.kit.profiler.window-1.4.4] startup
[8.713s] [ext: omni.graph.visualization.nodes-1.1.1] startup
[8.716s] [ext: omni.isaac.synthetic_utils-0.3.5] startup
[8.717s] [ext: semantics.schema.editor-0.2.3] startup
[8.719s] [ext: omni.isaac.sim.python-2022.1.1] startup
[8.720s] Simulation App Starting
[9.362s] app ready
[11.049s] Simulation App Startup Complete
task:
name: AnymalTerrain
physics_engine: physx
env:
numEnvs: 64
numObservations: 188
numActions: 12
envSpacing: 3.0
terrain:
staticFriction: 1.0
dynamicFriction: 1.0
restitution: 0.0
curriculum: True
maxInitMapLevel: 0
mapLength: 8.0
mapWidth: 8.0
numLevels: 10
numTerrains: 20
terrainProportions: [0.1, 0.1, 0.35, 0.25, 0.2]
slopeTreshold: 0.5
baseInitState:
pos: [0.0, 0.0, 0.62]
rot: [1.0, 0.0, 0.0, 0.0]
vLinear: [0.0, 0.0, 0.0]
vAngular: [0.0, 0.0, 0.0]
randomCommandVelocityRanges:
linear_x: [-1.0, 1.0]
linear_y: [-1.0, 1.0]
yaw: [-3.14, 3.14]
control:
stiffness: 80.0
damping: 2.0
actionScale: 0.5
decimation: 4
defaultJointAngles:
LF_HAA: 0.03
LH_HAA: 0.03
RF_HAA: -0.03
RH_HAA: -0.03
LF_HFE: 0.4
LH_HFE: -0.4
RF_HFE: 0.4
RH_HFE: -0.4
LF_KFE: -0.8
LH_KFE: 0.8
RF_KFE: -0.8
RH_KFE: 0.8
learn:
terminalReward: 0.0
linearVelocityXYRewardScale: 1.0
linearVelocityZRewardScale: -4.0
angularVelocityXYRewardScale: -0.05
angularVelocityZRewardScale: 0.5
orientationRewardScale: -0.0
torqueRewardScale: -2e-05
jointAccRewardScale: -0.0005
baseHeightRewardScale: -0.0
actionRateRewardScale: -0.01
fallenOverRewardScale: -1.0
hipRewardScale: -0.0
linearVelocityScale: 2.0
angularVelocityScale: 0.25
dofPositionScale: 1.0
dofVelocityScale: 0.05
heightMeasurementScale: 5.0
addNoise: True
noiseLevel: 1.0
dofPositionNoise: 0.01
dofVelocityNoise: 1.5
linearVelocityNoise: 0.1
angularVelocityNoise: 0.2
gravityNoise: 0.05
heightMeasurementNoise: 0.06
pushInterval_s: 15
episodeLength_s: 20
sim:
dt: 0.005
up_axis: z
use_gpu_pipeline: True
gravity: [0.0, 0.0, -9.81]
add_ground_plane: False
use_flatcache: True
enable_scene_query_support: False
enable_cameras: False
default_physics_material:
static_friction: 1.0
dynamic_friction: 1.0
restitution: 0.0
physx:
worker_thread_count: 4
solver_type: 1
use_gpu: True
solver_position_iteration_count: 4
solver_velocity_iteration_count: 0
contact_offset: 0.02
rest_offset: 0.0
bounce_threshold_velocity: 0.2
friction_offset_threshold: 0.04
friction_correlation_distance: 0.025
enable_sleeping: True
enable_stabilization: True
max_depenetration_velocity: 100.0
gpu_max_rigid_contact_count: 524288
gpu_max_rigid_patch_count: 163840
gpu_found_lost_pairs_capacity: 4194304
gpu_found_lost_aggregate_pairs_capacity: 33554432
gpu_total_aggregate_pairs_capacity: 4194304
gpu_max_soft_body_contacts: 1048576
gpu_max_particle_contacts: 1048576
gpu_heap_capacity: 134217728
gpu_temp_buffer_capacity: 33554432
gpu_max_num_partitions: 8
anymal:
override_usd_defaults: False
fixed_base: False
enable_self_collisions: True
enable_gyroscopic_forces: False
solver_position_iteration_count: 4
solver_velocity_iteration_count: 0
sleep_threshold: 0.005
stabilization_threshold: 0.001
density: -1
max_depenetration_velocity: 100.0
train:
params:
seed: 42
algo:
name: a2c_continuous
model:
name: continuous_a2c_logstd
network:
name: actor_critic
separate: True
space:
continuous:
mu_activation: None
sigma_activation: None
mu_init:
name: default
sigma_init:
name: const_initializer
val: 0.0
fixed_sigma: True
mlp:
units: [512, 256, 128]
activation: elu
d2rl: False
initializer:
name: default
regularizer:
name: None
load_checkpoint: True
load_path: /home/arghya/OmniIsaacGymEnvs/omniisaacgymenvs/checkpoints/anymal_terrain.pth
config:
name: AnymalTerrain
full_experiment_name: AnymalTerrain
device: cuda:0
device_name: cuda:0
env_name: rlgpu
ppo: True
mixed_precision: False
normalize_input: True
normalize_value: True
normalize_advantage: True
value_bootstrap: True
clip_actions: False
num_actors: 64
reward_shaper:
scale_value: 1.0
gamma: 0.99
tau: 0.95
e_clip: 0.2
entropy_coef: 0.001
learning_rate: 0.0003
lr_schedule: adaptive
kl_threshold: 0.008
truncate_grads: True
grad_norm: 1.0
horizon_length: 48
minibatch_size: 16384
mini_epochs: 5
critic_coef: 2
clip_value: True
seq_len: 4
bounds_loss_coef: 0.0
max_epochs: 2000
save_best_after: 100
score_to_win: 20000
save_frequency: 50
print_stats: True
task_name: AnymalTerrain
experiment:
num_envs: 64
seed: 42
torch_deterministic: False
max_iterations:
physics_engine: physx
pipeline: gpu
sim_device: gpu
device_id: 0
rl_device: cuda:0
num_threads: 4
solver_type: 1
test: False
checkpoint: /home/arghya/OmniIsaacGymEnvs/omniisaacgymenvs/checkpoints/anymal_terrain.pth
headless: False
wandb_activate: False
wandb_group:
wandb_name: AnymalTerrain
wandb_entity:
wandb_project: omniisaacgymenvs
Sim params does not have attribute: up_axis
Sim params does not have attribute: physx
Sim params does not have attribute: anymal
Pipeline: GPU
Pipeline Device: cuda:0
Sim Device: GPU
Task Device: cuda:0
RL device: cuda:0
/home/arghya/.local/share/ov/pkg/isaac_sim-2022.1.1/exts/omni.isaac.ml_archive/pip_prebundle/gym/spaces/box.py:74: UserWarning: WARN: Box bound precision lowered by casting to float32
"Box bound precision lowered by casting to {}".format(self.dtype)
/home/arghya/.local/share/ov/pkg/isaac_sim-2022.1.1/exts/omni.kit.pip_torch-1_11_0-0.1.3+103.1.lx64.cp37/torch-1-11-0/torch/functional.py:568: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:2228.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
[11.307s] [ext: omni.physx.flatcache-1.4.15-5.1] startup
2022-09-19 00:58:20 [11,314ms] [Warning] [carb] Acquiring non optional plugin interface which is not listed as dependency: [omni::kit::IStageUpdate v1.0] (plugin: (null)), by client: omni.physx.flatcache.plugin. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client.
2022-09-19 00:58:21 [12,357ms] [Warning] [omni.hydra.scene_delegate.plugin] Calling getBypassRenderSkelMeshProcessing for prim /World/envs/env_0/anymal/base/visuals.proto_mesh_20_id20 that has not been populated
2022-09-19 00:58:22 [12,855ms] [Warning] [omni.physx.plugin] ParseFEMClothMaterial: UsdPrim doesn't have a PhysxSchemaPhysxDeformableSurfaceMaterialAPI
PxgCudaDeviceMemoryAllocator fail to allocate memory 134217728 bytes!! Result = 2
2022-09-19 00:58:31 [21,847ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuEventRecord failed
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h, LINE 54
2022-09-19 00:58:31 [21,847ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,847ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuStreamWaitEvent failed
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h, LINE 60
2022-09-19 00:58:31 [21,847ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,847ms] [Error] [omni.physx.plugin] PhysX error: memcpy failed fail!
700, FILE /buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp, LINE 2408
2022-09-19 00:58:31 [21,847ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuEventRecord failed
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h, LINE 54
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuStreamWaitEvent failed
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h, LINE 60
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuEventRecord failed
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h, LINE 54
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuStreamWaitEvent failed
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h, LINE 60
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams failed
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h, LINE 80
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams failed
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h, LINE 80
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: GPU solveContactParallel fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1492
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: GPU solveContactParallel fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1529
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: GPU compute solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1697
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: GPU solveSelfConstraints fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 859
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: GPU artiSolveInternalTendonConstraints1T fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 879
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: GPU solveInternalConstraints fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 907
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: GPU propagate solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1784
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: GPU compute solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1697
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: GPU solveSelfConstraints fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 859
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: GPU artiSolveInternalTendonConstraints1T fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 879
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: GPU solveInternalConstraints fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 907
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: GPU propagate solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1784
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: GPU compute solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1697
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: GPU solveSelfConstraints fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 859
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: GPU artiSolveInternalTendonConstraints1T fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 879
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: GPU solveInternalConstraints fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 907
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: GPU propagate solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1784
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: GPU compute solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1697
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: GPU solveSelfConstraints fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 859
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: GPU artiSolveInternalTendonConstraints1T fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 879
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: GPU solveInternalConstraints fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpuarticulation/src/PxgArticulationCore.cpp, LINE 907
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: GPU propagate solver bodies average velocities fail to launch kernel!!
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpusolver/src/PxgTGSCudaSolverCore.cpp, LINE 1784
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams failed
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h, LINE 80
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,848ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams failed
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h, LINE 80
2022-09-19 00:58:31 [21,849ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [21,849ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams failed
, FILE /buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h, LINE 80
2022-09-19 00:58:31 [21,849ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [22,049ms] [Error] [omni.physx.plugin] PhysX error: PhysX Internal CUDA error. Simulation can not continue!, FILE /buildAgent/work/99bede84aa0a52c2/source/physx/src/NpScene.cpp, LINE 3509
2022-09-19 00:58:31 [22,049ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2022-09-19 00:58:31 [22,049ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 146
2022-09-19 00:58:31 [22,049ms] [Error] [omni.physx.tensors.plugin] CUDA context validation failed
2022-09-19 00:58:31 [22,049ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 193
2022-09-19 00:58:31 [22,049ms] [Error] [omni.physx.tensors.plugin] Failed to initialize GPU simulation data
2022-09-19 00:58:31 [22,049ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 457
2022-09-19 00:58:31 [22,049ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 458
2022-09-19 00:58:31 [22,049ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 459
2022-09-19 00:58:31 [22,049ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 460
2022-09-19 00:58:31 [22,049ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 461
2022-09-19 00:58:31 [22,049ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 462
2022-09-19 00:58:31 [22,049ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 463
2022-09-19 00:58:31 [22,049ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 464
2022-09-19 00:58:31 [22,049ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 465
2022-09-19 00:58:31 [22,049ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 466
2022-09-19 00:58:31 [22,049ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 467
2022-09-19 00:58:31 [22,050ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 468
2022-09-19 00:58:31 [22,050ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 469
2022-09-19 00:58:31 [22,050ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 471
2022-09-19 00:58:31 [22,050ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 472
2022-09-19 00:58:31 [22,050ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 474
2022-09-19 00:58:31 [22,050ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 475
2022-09-19 00:58:31 [22,050ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 476
2022-09-19 00:58:31 [22,050ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 478
2022-09-19 00:58:31 [22,050ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 479
2022-09-19 00:58:31 [22,050ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../source/extensions/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 480
Error executing job with overrides: ['task=AnymalTerrain', 'num_envs=64', 'checkpoint=omniverse://localhost/NVIDIA/Assets/Isaac/2022.1/Isaac/Samples/OmniIsaacGymEnvs/Checkpoints/anymal_terrain.pth']
Traceback (most recent call last):
File "scripts/rlgames_play.py", line 67, in parse_hydra_configs
task = initialize_demo(cfg_dict, env)
File "/home/arghya/OmniIsaacGymEnvs/omniisaacgymenvs/utils/demo_util.py", line 46, in initialize_demo
env.set_task(task=task, sim_params=sim_config.get_physics_params(), backend="torch", init_sim=init_sim)
File "/home/arghya/OmniIsaacGymEnvs/omniisaacgymenvs/envs/vec_env_rlgames.py", line 51, in set_task
super().set_task(task, backend, sim_params, init_sim)
File "/home/arghya/.local/share/ov/pkg/isaac_sim-2022.1.1/exts/omni.isaac.gym/omni/isaac/gym/vec_env/vec_env_base.py", line 82, in set_task
self._world.reset()
File "/home/arghya/.local/share/ov/pkg/isaac_sim-2022.1.1/exts/omni.isaac.core/omni/isaac/core/world/world.py", line 281, in reset
SimulationContext.reset(self, soft=soft)
File "/home/arghya/.local/share/ov/pkg/isaac_sim-2022.1.1/exts/omni.isaac.core/omni/isaac/core/simulation_context/simulation_context.py", line 423, in reset
SimulationContext.initialize_physics(self)
File "/home/arghya/.local/share/ov/pkg/isaac_sim-2022.1.1/exts/omni.isaac.core/omni/isaac/core/simulation_context/simulation_context.py", line 403, in initialize_physics
self._physics_sim_view = omni.physics.tensors.create_simulation_view(self.backend)
File "/home/arghya/.local/share/ov/pkg/isaac_sim-2022.1.1/kit/extsPhysics/omni.physics.tensors-1.4.15-5.1/omni/physics/tensors/impl/api.py", line 11, in create_simulation_view
raise Exception("Failed to create simulation view backend")
Exception: Failed to create simulation view backend
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
/home/arghya/.local/share/ov/pkg/isaac_sim-2022.1.1/python.sh: line 40: 19204 Segmentation fault (core dumped) $python_exe "$@" $args
There was an error running python
Ane help is appreciated. TIA.
// using python here because I am using conda
python -m torch.distributed.run --nnodes=1 --nproc_per_node=2 scripts/rlgames_train.py headless=True task=AnymalTerrain multi_gpu=True
WARNING:main:
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
/home/bizon/anaconda3/envs/isaac-sim/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'config': Defaults list is missing _self_
. See https://hydra.cc/docs/upgrades/1.0_to_1.1/default_composition_order for more information
warnings.warn(msg, UserWarning)
/home/bizon/anaconda3/envs/isaac-sim/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:412: UserWarning: In config: Invalid overriding of hydra/job_logging:
Default list overrides requires 'override' keyword.
See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/defaults_list_override for more information.
deprecation_warning(msg)
/home/bizon/anaconda3/envs/isaac-sim/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'config': Defaults list is missing _self_
. See https://hydra.cc/docs/upgrades/1.0_to_1.1/default_composition_order for more information
warnings.warn(msg, UserWarning)
/home/bizon/anaconda3/envs/isaac-sim/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:412: UserWarning: In config: Invalid overriding of hydra/job_logging:
Default list overrides requires 'override' keyword.
See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/defaults_list_override for more information.
deprecation_warning(msg)
[Warning] [omni.isaac.kit.simulation_app] Modules: ['omni.isaac.kit.app_framework'] were loaded before SimulationApp was started and might not be loaded correctly.
[Warning] [omni.isaac.kit.simulation_app] Please check to make sure no extra omniverse or pxr modules are imported before the call to SimulationApp(...)
Starting kit application with the following args: ['/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py', '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/apps/omni.isaac.sim.python.gym.headless.kit', '--/app/tokens/exe-path=/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/kit', '--/persistent/app/viewport/displayOptions=3094', '--/rtx/materialDb/syncLoads=True', '--/rtx/hydra/materialSyncLoads=True--/omni.kit.plugin/syncUsdLoads=True', '--/app/renderer/resolution/width=1280', '--/app/renderer/resolution/height=720', '--/app/window/width=1440', '--/app/window/height=900', '--/renderer/multiGpu/enabled=True', '--/app/fastShutdown=True', '--ext-folder', '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/exts', '--ext-folder', '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/apps', '--/physics/cudaDevice=1', '--portable', '--no-window']
Passing the following args to the base kit application: ['headless=True', 'task=AnymalTerrain', 'multi_gpu=True']
[Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
[Info] [carb] Logging to file: /home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/kit/logs/Kit/Isaac-Sim/2022.2/kit_20230512_104301.log
2023-05-12 20:43:01 [5ms] [Warning] [omni.ext.plugin] [ext: omni.sensors.nv.lidar] Extensions config 'extension.toml' doesn't exist '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.sensors.nv.lidar' or '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.sensors.nv.lidar/config'
2023-05-12 20:43:01 [6ms] [Warning] [omni.ext.plugin] [ext: omni.sensors.nv.radar] Extensions config 'extension.toml' doesn't exist '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.sensors.nv.radar' or '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.sensors.nv.radar/config'
[Warning] [omni.isaac.kit.simulation_app] Modules: ['omni.isaac.kit.app_framework'] were loaded before SimulationApp was started and might not be loaded correctly.
[Warning] [omni.isaac.kit.simulation_app] Please check to make sure no extra omniverse or pxr modules are imported before the call to SimulationApp(...)
Starting kit application with the following args: ['/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py', '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/apps/omni.isaac.sim.python.gym.headless.kit', '--/app/tokens/exe-path=/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/kit', '--/persistent/app/viewport/displayOptions=3094', '--/rtx/materialDb/syncLoads=True', '--/rtx/hydra/materialSyncLoads=True--/omni.kit.plugin/syncUsdLoads=True', '--/app/renderer/resolution/width=1280', '--/app/renderer/resolution/height=720', '--/app/window/width=1440', '--/app/window/height=900', '--/renderer/multiGpu/enabled=True', '--/app/fastShutdown=True', '--ext-folder', '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/exts', '--ext-folder', '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/apps', '--/physics/cudaDevice=0', '--portable', '--no-window']
Passing the following args to the base kit application: ['headless=True', 'task=AnymalTerrain', 'multi_gpu=True']
[Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
[Info] [carb] Logging to file: /home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/kit/logs/Kit/Isaac-Sim/2022.2/kit_20230512_104301.log
2023-05-12 20:43:01 [8ms] [Warning] [omni.ext.plugin] [ext: omni.sensors.nv.lidar] Extensions config 'extension.toml' doesn't exist '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.sensors.nv.lidar' or '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.sensors.nv.lidar/config'
2023-05-12 20:43:01 [8ms] [Warning] [omni.ext.plugin] [ext: omni.sensors.nv.radar] Extensions config 'extension.toml' doesn't exist '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.sensors.nv.radar' or '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.sensors.nv.radar/config'
[0.098s] [ext: omni.stats-0.0.0] startup
[0.103s] [ext: omni.rtx.shadercache-1.0.0] startup
[0.105s] [ext: omni.assets.plugins-0.0.0] startup
[0.106s] [ext: omni.gpu_foundation-0.0.0] startup
[0.111s] [ext: carb.windowing.plugins-1.0.0] startup
2023-05-12 20:43:01 [111ms] [Warning] [carb.windowing-glfw.gamepad] Joystick with unknown remapping detected (will be ignored): ASRock LED Controller [03000000ce260000a201000010010000]
[0.118s] [ext: omni.kit.renderer.init-0.0.0] startup
[0.101s] [ext: omni.stats-0.0.0] startup
[0.105s] [ext: omni.rtx.shadercache-1.0.0] startup
[0.107s] [ext: omni.assets.plugins-0.0.0] startup
[0.108s] [ext: omni.gpu_foundation-0.0.0] startup
[0.113s] [ext: carb.windowing.plugins-1.0.0] startup
2023-05-12 20:43:01 [105ms] [Warning] [carb.windowing-glfw.gamepad] Joystick with unknown remapping detected (will be ignored): ASRock LED Controller [03000000ce260000a201000010010000]
[0.119s] [ext: omni.kit.renderer.init-0.0.0] startup
|---------------------------------------------------------------------------------------------|
| Driver Version: 530.30.02 | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name | Active | LDA | GPU Memory | Vendor-ID | LUID |
| | | | | | Device-ID | UUID |
|---------------------------------------------------------------------------------------------|
| 0 | NVIDIA GeForce RTX 4090 | Yes: 1 | | 24810 MB | 10de | 0 |
| | | | | | 2684 | 527cadd8.. |
|---------------------------------------------------------------------------------------------|
| 1 | NVIDIA GeForce RTX 4090 | Yes: 0 | | 24810 MB | 10de | 0 |
| | | | | | 2684 | 7c9328c9.. |
|=============================================================================================|
| OS: Linux dl, Version: 5.15.0-69-generic
| XServer Vendor: The X.Org Foundation, XServer Version: 12013000 (1.20.13.0)
| Processor: AMD Ryzen Threadripper PRO 5975WX 32-Cores | Cores: Unknown | Logical: 64
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 257558 | Free Memory: 249198
| Total Page/Swap (MB): 2047 | Free Page/Swap: 2047
|---------------------------------------------------------------------------------------------|
Hi,
The simulation and examples are working well for me, thanks a lot for that. But whenever I stop the (single-threaded) training script, I run into this warning in any other terminal on the same machine:
Message from syslogd@rlsim2 at Dec 14 09:51:09 ...
kernel:[409187.093643] watchdog: BUG: soft lockup - CPU#54 stuck for 23s! [carb::livestrea:1460402]
and have to wait fairly long (20-30sec) for the process to shut down. Have you experienced this before or might know what is causing this?
I am running the FrankaCabinet task on an A6000 with Driver Version: 515.65.01 and a Threadripper PRO 3975WX which is far from being fully utilized. I run the scripts within the latest Isaac docker image.
Hi, we're trying to use this repository to train multiple robots that use Lidar sensors for detecting obstacles. For testing if the sensors work correctly, we added a cube (with the collision API enabled) to the default environment. When running the simulation with the random policy script however, the Lidars don't seem to detect any obstacles. This can be remedied by selecting the cube in the GUI and disabling and enabling the collision API of the cube again, but this workaround only works for objects in the default environment and not for any of the clones.
This is a screenshot of the scene after starting the simulation
This is a screenshot of the scene after disabling and reenabling the collision api of the cube in env_0
Is there any way to configure the scene so the Lidar sensors work properly in every environment without having to use the GUI?
Hi all,
i want to perform my new RL task and the new RL task was modified from the example cartpole. I change the default parameter from [fixed_base = false ] to [fixed_base = true ] in the Cartpole.yaml , as a result ,the segmentation fault happend. And i test the problem in oher examples, such as Ant and Allegrohand, the same error occured.
however, i used to change the fixed_base parameter in former isaacgym version [ isaacgym_preview3] which was independent of isaac sim, and no error happend.
following is the error list:
Cartpole:
override_usd_defaults: False
fixed_base: True // i change the default value
enable_self_collisions: False
enable_gyroscopic_forces: True
solver_position_iteration_count: 4
solver_velocity_iteration_count: 0
sleep_threshold: 0.005
stabilization_threshold: 0.001
density: -1
max_depenetration_velocity: 100.0
contact_offset: 0.02
rest_offset: 0.001
.local/share/ov/pkg/isaac_sim-2022.1.0/python.sh: line 46: 20430 Segmentation fault
There was an error running python
Process finished with exit code 1
Appreciate it if you can give some hints.
mao
After Installation, I tried to launch rlgames_train.py
~/OmniIsaacGymEnvs/omniisaacgymenvs$ PYTHON_PATH scripts/rlgames_train.py task=Cartpole
But here is error:
Traceback (most recent call last):
File "scripts/rlgames_train.py", line 34, in
from omniisaacgymenvs.utils.config_utils.path_utils import retrieve_checkpoint_path
File "~/OmniIsaacGymEnvs/omniisaacgymenvs/utils/config_utils/path_utils.py", line 32, in
import omni.client
File "~/.local/share/ov/pkg/isaac_sim-2022.1.0/kit/extscore/omni.client/omni/client/init.py", line 21, in
from ._omniclient import *
ImportError: ~ /.local/share/ov/pkg/isaac_sim-2022.1.0/kit/extscore/omni.usd.libs/bin/libjs.so: undefined symbol: _ZN32pxrInternal_v0_20__pxrReserved__18Tf_PostErrorHelperERKNS_13TfCallContextENS_16TfDiagnosticTypeEPKcz
There was an error running python
So I tested the simplest code
Print_pi.py
from omni.isaac.examples.hello_world.hello_world_extension import HelloWorldExtension
print('Hi")
Run
$ PYTHON_PATH print_hi.py
ERROR
Traceback (most recent call last):
File "print_hi.py", line 12, in
from omni.isaac.examples.hello_world.hello_world_extension import HelloWorldExtension
File "~/.local/share/ov/pkg/isaac_sim-2022.1.0/exts/omni.isaac.examples/omni/isaac/examples/hello_world/init.py", line 10, in
from omni.isaac.examples.hello_world.hello_world import HelloWorld
File "~/.local/share/ov/pkg/isaac_sim-2022.1.0/exts/omni.isaac.examples/omni/isaac/examples/hello_world/hello_world.py", line 9, in
from omni.isaac.examples.base_sample import BaseSample
File "~/.local/share/ov/pkg/isaac_sim-2022.1.0/exts/omni.isaac.examples/omni/isaac/examples/base_sample/init.py", line 10, in
from omni.isaac.examples.base_sample.base_sample import BaseSample
File "~/.local/share/ov/pkg/isaac_sim-2022.1.0/exts/omni.isaac.examples/omni/isaac/examples/base_sample/base_sample.py", line 9, in
from omni.isaac.core import World
File "~/.local/share/ov/pkg/isaac_sim-2022.1.0/exts/omni.isaac.core/omni/isaac/core/init.py", line 9, in
from omni.isaac.core.physics_context.physics_context import PhysicsContext
File "~/.local/share/ov/pkg/isaac_sim-2022.1.0/exts/omni.isaac.core/omni/isaac/core/physics_context/init.py", line 9, in
from omni.isaac.core.physics_context.physics_context import PhysicsContext
File "~/.local/share/ov/pkg/isaac_sim-2022.1.0/exts/omni.isaac.core/omni/isaac/core/physics_context/physics_context.py", line 12, in
from pxr import Usd, UsdGeom, Gf, Sdf, UsdPhysics, PhysxSchema, UsdShade
File "~/.local/share/ov/pkg/isaac_sim-2022.1.0/kit/extscore/omni.usd.libs/pxr/Usd/init.py", line 24, in
from . import bong_usd
ImportError: ~/.local/share/ov/pkg/isaac_sim-2022.1.0/kit/extscore/omni.usd.libs/bin/libjs.so: undefined symbol: _ZN32pxrInternal_v0_20__pxrReserved__18Tf_PostErrorHelperERKNS_13TfCallContextENS_16TfDiagnosticTypeEPKcz
There was an error running python
How can I solve it?
Here's my hardware spec
Ubuntu 20.04
Python 3.8 (I tried Python 3.7 in conda env, but I got the same result)
Intel CPU + RTX3070Ti
Hi,
Thanks for the great jobs on Omniverse Isaac Gym.
I have a question when I transfer policy from isaac gym. I used my own asset so I get a urdf file.
I use isaac sim to import it as a usd file, but there is no prim_paths on isaac sim to save.
So when I want to train by omni isaac gym, it shows wrong. How can I get prim file?
Hi,
I'm trying to run some of the examples. While the Ant task works fine, other robots lead to a segfault during the initialize_task() function. This includes both the cartpole and franke_cabinet tasks.
I am running this in the latest Isaac docker image nvcr.io/nvidia/isaac-sim:2022.1.1 on an RTX A6000 with 515.65.1 drivers
The error is
[Warning] [omni.isaac.kit.simulation_app] Modules: ['omniisaacgymenvs', 'omniisaacgymenvs.utils', 'omniisaacgymenvs.utils.hydra_cfg', 'omniisaacgymenvs.utils.hydra_cfg.hydra_utils', 'omniisaacgymenvs.utils.hydra_cfg.reformat', 'omniisaacgymenvs.utils.rlgames', 'omniisaacgymenvs.utils.rlgames.rlgames_utils', 'omniisaacgymenvs.utils.config_utils', 'omniisaacgymenvs.utils.config_utils.path_utils', 'omniisaacgymenvs.envs', 'omniisaacgymenvs.envs.vec_env_rlgames', 'omni.isaac.gym', 'omni.isaac.gym.vec_env', 'omni.isaac.gym.vec_env.vec_env_base', 'omni.isaac.gym.vec_env.vec_env_mt'] were loaded before SimulationApp was started and might not be loaded correctly.
[Warning] [omni.isaac.kit.simulation_app] Please check to make sure no extra omniverse or pxr modules are imported before the call to SimulationApp(...)
Starting kit application with the fillowing args: ['/isaac-sim/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py', '/isaac-sim/apps/omni.isaac.sim.python.kit', '--/app/tokens/exe-path=/isaac-sim/kit', '--/persistent/app/viewport/displayOptions=3094', '--/rtx/materialDb/syncLoads=True', '--/rtx/hydra/materialSyncLoads=True--/omni.kit.plugin/syncUsdLoads=True', '--/app/renderer/resolution/width=1280', '--/app/renderer/resolution/height=720', '--/app/window/width=1440', '--/app/window/height=900', '--/renderer/multiGpu/enabled=True', '--ext-folder', '/isaac-sim/exts', '--ext-folder', '/isaac-sim/apps', '--/physics/cudaDevice=0', '--portable', '--allow-root']
Passing the following args to the base kit application: ['--enable', 'omni.kit.livestream.native', '--no-window']
[Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
[Info] [carb] Logging to file: /isaac-sim/kit/logs/Kit/Isaac-Sim/2022.1/kit_20221209_085428.log
2022-12-09 08:54:28 [1ms] [Warning] [omni.ext.plugin] [ext: omni.drivesim.sensors.nv.lidar] Extensions config 'extension.toml' doesn't exist '/isaac-sim/exts/omni.drivesim.sensors.nv.lidar' or '/isaac-sim/exts/omni.drivesim.sensors.nv.lidar/config'
[0.115s] [ext: omni.stats-0.0.0] startup
[0.155s] [ext: omni.gpu_foundation-0.0.0] startup
2022-12-09 08:54:28 [153ms] [Warning] [carb] FrameworkImpl::setDefaultPlugin(client: omni.gpu_foundation_factory.plugin, desc : [carb::graphics::Graphics v2.5], plugin : carb.graphics-vulkan.plugin) failed. Plugin selection is locked, because the interface was previously acquired by:
[0.166s] [ext: carb.windowing.plugins-1.0.0] startup
2022-12-09 08:54:28 [159ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2022-12-09 08:54:28 [159ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.3]) (impl: carb.windowing-glfw.plugin)
[0.167s] [ext: omni.assets.plugins-0.0.0] startup
[0.168s] [ext: omni.kit.renderer.init-0.0.0] startup
|---------------------------------------------------------------------------------------------|
| Driver Version: 515.65.1 | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name | Active | LDA | GPU Memory | Vendor-ID | LUID |
| | | | | | Device-ID | UUID |
|---------------------------------------------------------------------------------------------|
| 0 | NVIDIA RTX A6000 | Yes: 0 | | 49386 MB | 10de | 0 |
| | | | | | 2230 | 31d81b8f.. |
|=============================================================================================|
| OS: Linux rlsim2, Version: 5.4.0-132-generic
| Processor: AMD Ryzen Threadripper PRO 3975WX 32-Cores | Cores: Unknown | Logical: 64
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 128729 | Free Memory: 119711
| Total Page/Swap (MB): 8191 | Free Page/Swap: 8191
|---------------------------------------------------------------------------------------------|
2022-12-09 08:55:02 [34,347ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2022-12-09 08:55:02 [34,347ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.3]) (impl: carb.windowing-glfw.plugin)
2022-12-09 08:55:02 [34,348ms] [Error] [carb.glinterop.plugin] GLInteropContext::init: carb::windowing is not available
2022-12-09 08:55:02 [34,348ms] [Warning] [gpu.foundation.plugin] Realm: no OpenGL interop context.
2022-12-09 08:55:02 [34,439ms] [Warning] [carb.cudainterop.plugin] On Linux only, CUDA and the display driver does not support IOMMU-enabled bare-metal PCIe peer to peer memory copy.
However, CUDA and the display driver does support IOMMU via VM pass through. As a consequence, users on Linux,
when running on a native bare metal system, should disable the IOMMU. The IOMMU should be enabled and the VFIO driver
be used as a PCIe pass through for virtual machines.
[34.456s] [ext: omni.kit.pipapi-0.0.0] startup
[34.477s] [ext: omni.kit.pip_archive-0.0.0] startup
[34.489s] [ext: omni.isaac.core_archive-1.2.0] startup
[34.533s] [ext: omni.usd.config-1.0.0] startup
[34.539s] [ext: omni.usd.libs-1.0.0] startup
[34.645s] [ext: omni.kit.pip_torch-1_11_0-0.1.3] startup
[34.713s] [ext: omni.isaac.ml_archive-1.1.0] startup
[34.720s] [ext: omni.kit.loop-isaac-0.1.0] startup
[34.721s] [ext: omni.kit.async_engine-0.0.0] startup
[34.725s] [ext: omni.appwindow-1.0.0] startup
2022-12-09 08:55:02 [34,718ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2022-12-09 08:55:02 [34,718ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.3]) (impl: carb.windowing-glfw.plugin)
[34.731s] [ext: omni.client-0.1.0] startup
[34.744s] [ext: omni.kit.test-0.0.0] startup
[34.747s] [ext: omni.kit.renderer.core-0.0.0] startup
2022-12-09 08:55:02 [34,741ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2022-12-09 08:55:02 [34,741ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.3]) (impl: carb.windowing-glfw.plugin)
2022-12-09 08:55:02 [34,742ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2022-12-09 08:55:02 [34,742ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.3]) (impl: carb.windowing-glfw.plugin)
2022-12-09 08:55:02 [34,742ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2022-12-09 08:55:02 [34,742ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.3]) (impl: carb.windowing-glfw.plugin)
2022-12-09 08:55:02 [34,743ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2022-12-09 08:55:02 [34,743ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.3]) (impl: carb.windowing-glfw.plugin)
[34.841s] [ext: omni.ui-2.10.3] startup
[34.864s] [ext: carb.audio-0.1.0] startup
[34.868s] [ext: omni.kit.mainwindow-0.0.0] startup
[34.872s] [ext: omni.uiaudio-1.0.0] startup
[34.875s] [ext: omni.kit.uiapp-0.0.0] startup
[34.875s] [ext: omni.usd.schema.physics-1.0.0] startup
[34.935s] [ext: omni.usd.schema.isaac-0.2.0] startup
[35.029s] [ext: omni.usd.schema.omnigraph-1.0.0] startup
[35.049s] [ext: omni.usd.schema.physx-0.0.0] startup
[35.150s] [ext: omni.usd.schema.anim-0.0.0] startup
[35.264s] [ext: omni.usd.schema.audio-0.0.0] startup
[35.281s] [ext: omni.usd.schema.semantics-0.0.0] startup
[35.299s] [ext: omni.usd.schema.forcefield-0.0.0] startup
[35.318s] [ext: omni.timeline-1.0.2] startup
[35.323s] [ext: omni.hydra.scene_delegate-0.2.0] startup
[35.329s] [ext: omni.kit.commands-1.2.2] startup
[35.342s] [ext: omni.kit.audiodeviceenum-1.0.0] startup
[35.344s] [ext: omni.usd-1.5.3] startup
[35.396s] [ext: omni.kit.asset_converter-1.2.31] startup
[35.418s] [ext: omni.kit.widget.path_field-2.0.3] startup
[35.427s] [ext: omni.kit.search_core-1.0.2] startup
[35.430s] [ext: omni.kit.window.popup_dialog-2.0.8] startup
[35.440s] [ext: omni.kit.widget.browser_bar-2.0.3] startup
[35.443s] [ext: omni.kit.widget.filebrowser-2.2.27] startup
[35.452s] [ext: omni.mdl.neuraylib-0.1.0] startup
[35.460s] [ext: omni.kit.menu.utils-1.2.11] startup
[35.476s] [ext: omni.kit.notification_manager-1.0.5] startup
[35.481s] [ext: omni.kit.widget.versioning-1.3.8] startup
[35.485s] [ext: omni.kit.menu.create-1.0.2] startup
[35.486s] [ext: omni.kit.window.filepicker-2.4.30] startup
OmniAssetFileFormat
[35.573s] [ext: omni.mdl-0.1.0] startup
[35.601s] [ext: omni.kit.stage_templates-1.1.2] startup
[35.606s] [ext: omni.kit.window.file_exporter-1.0.4] startup
[35.608s] [ext: omni.kit.material.library-1.3.10] startup
[35.614s] [ext: omni.kit.window.file_importer-1.0.4] startup
[35.615s] [ext: omni.kit.window.drop_support-1.0.0] startup
[35.617s] [ext: omni.kit.window.file-1.3.16] startup
[35.620s] [ext: omni.kit.context_menu-1.3.9] startup
[35.627s] [ext: omni.kit.window.property-1.6.3] startup
[35.629s] [ext: omni.kit.window.content_browser-2.4.28] startup
[35.652s] [ext: omni.kit.widget.stage-2.6.15] startup
[35.659s] [ext: omni.renderer-rtx-0.0.0] startup
[35.659s] [ext: omni.kit.property.usd-3.14.9] startup
[35.731s] [ext: omni.kit.widget.settings-1.0.0] startup
[35.736s] [ext: omni.kit.widget.prompt-1.0.1] startup
[35.737s] [ext: omni.kit.widget.graph-1.4.3] startup
[35.754s] [ext: omni.kit.window.preferences-1.2.1] startup
[35.819s] [ext: omni.hydra.engine.stats-1.0.0] startup
[35.829s] [ext: omni.hydra.rtx-0.1.0] startup
[35.837s] [ext: omni.kit.viewport.legacy_gizmos-1.0.0] startup
[35.839s] [ext: omni.graph.core-2.29.1] startup
[35.842s] [ext: omni.graph.tools-1.4.0] startup
[35.952s] [ext: omni.kit.window.viewport-0.0.0] startup
[39.336s] [ext: omni.graph-1.23.0] startup
[39.439s] [ext: omni.debugdraw-0.1.0] startup
[39.448s] [ext: omni.ui_query-1.1.1] startup
[39.454s] [ext: omni.graph.ui-1.6.1] startup
[39.488s] [ext: omni.kit.ui_test-1.2.2] startup
[39.493s] [ext: omni.kvdb-0.0.0] startup
[39.500s] [ext: omni.graph.action-1.18.0] startup
[39.517s] [ext: omni.kit.widget.searchfield-1.0.6] startup
[39.520s] [ext: omni.convexdecomposition-1.4.15] startup
[39.527s] [ext: omni.localcache-0.0.0] startup
[39.535s] [ext: omni.usdphysics-1.4.15] startup
[39.537s] [ext: omni.graph.scriptnode-0.5.0] startup
[39.541s] [ext: omni.physx-1.4.15-5.1] startup
[39.575s] [ext: omni.kit.numpy.common-0.1.0] startup
[39.584s] [ext: omni.physics.tensors-0.1.0] startup
[39.596s] [ext: omni.graph.nodes-1.26.0] startup
[39.615s] [ext: omni.isaac.dynamic_control-1.1.0] startup
[39.625s] [ext: omni.isaac.version-1.0.0] startup
[39.627s] [ext: omni.physx.tensors-0.1.0] startup
[39.631s] [ext: omni.syntheticdata-0.2.1] startup
[39.692s] [ext: omni.isaac.core-1.24.3] startup
[39.937s] [ext: omni.isaac.synthetic_utils-0.3.5] startup
[39.942s] [ext: omni.kit.window.script_editor-1.6.2] startup
[39.949s] [ext: omni.kit.usd_undo-0.1.0] startup
[39.954s] [ext: semantics.schema.editor-0.2.3] startup
[39.963s] [ext: omni.physx.commands-1.4.15-5.1] startup
[39.974s] [ext: omni.kit.renderer.capture-0.0.0] startup
[39.982s] [ext: omni.physx.ui-1.4.15-5.1] startup
[40.078s] [ext: omni.warp-0.2.2] startup
Warp initialized:
Version: 0.2.2
CUDA device: NVIDIA RTX A6000
Kernel cache: /root/.cache/warp/0.2.2
[40.508s] [ext: omni.kit.property.material-1.8.5] startup
[40.512s] [ext: omni.physx.demos-1.4.15-5.1] startup
[40.520s] [ext: omni.kit.window.toolbar-1.2.4] startup
[40.528s] [ext: omni.kit.property.physx-0.1.0] startup
2022-12-09 08:55:08 [40,674ms] [Warning] [omni.physx.plugin] Deprecated: getSimulationEventStream is deprecated, please use getSimulationEventStreamV2
[40.685s] [ext: omni.ui.scene-1.5.0] startup
[40.697s] [ext: omni.physx.tests-1.4.15-5.1] startup
[41.210s] [ext: omni.physx.vehicle-1.4.15-5.1] startup
[41.233s] [ext: omni.kit.manipulator.viewport-1.0.6] startup
[41.238s] [ext: omni.physx.camera-1.4.15-5.1] startup
[41.250s] [ext: omni.physx.cct-1.4.15-5.1] startup
[41.441s] [ext: omni.command.usd-1.0.1] startup
[41.450s] [ext: omni.kit.primitive.mesh-1.0.0] startup
[41.459s] [ext: omni.graph.visualization.nodes-1.1.1] startup
[41.465s] [ext: omni.physx.bundle-1.4.15-5.1] startup
[41.465s] [ext: omni.kit.widget.stage_icons-1.0.2] startup
[41.468s] [ext: omni.replicator.core-1.4.3] startup
[41.597s] [ext: omni.kit.widget.zoombar-1.0.3] startup
[41.599s] [ext: omni.rtx.window.settings-0.6.1] startup
[41.611s] [ext: omni.kit.window.stage-2.3.7] startup
[41.616s] [ext: omni.replicator.composer-1.1.3] startup
[41.625s] [ext: omni.kit.browser.core-2.0.12] startup
[41.635s] [ext: omni.rtx.settings.core-0.5.5] startup
[41.641s] [ext: omni.kit.window.extensions-1.1.0] startup
[41.650s] [ext: omni.kit.graph.usd.commands-1.1.0] startup
[41.656s] [ext: omni.kit.browser.folder.core-1.1.13] startup
[41.660s] [ext: omni.isaac.debug_draw-0.1.2] startup
[41.667s] [ext: omni.isaac.ui-0.2.1] startup
[41.672s] [ext: omni.kit.graph.delegate.default-1.0.15] startup
[41.673s] [ext: omni.isaac.motion_planning-0.2.0] startup
[41.681s] [ext: omni.isaac.lula-1.1.0] startup
[41.694s] [ext: omni.isaac.surface_gripper-0.1.2] startup
[41.701s] [ext: omni.kit.graph.editor.core-1.3.3] startup
[41.705s] [ext: omni.isaac.motion_generation-3.6.1] startup
[41.714s] [ext: omni.isaac.manipulators-1.0.1] startup
[41.717s] [ext: omni.isaac.core_nodes-0.13.0] startup
[41.734s] [ext: omni.kit.graph.widget.variables-2.0.2] startup
[41.737s] [ext: omni.kit.graph.delegate.modern-1.6.0] startup
[41.740s] [ext: omni.isaac.franka-0.3.0] startup
[41.741s] [ext: omni.isaac.wheeled_robots-0.5.8] startup
[41.766s] [ext: omni.graph.window.core-1.23.4] startup
[41.778s] [ext: omni.graph.instancing-1.1.4] startup
[41.784s] [ext: omni.graph.tutorials-1.1.2] startup
[41.808s] [ext: omni.graph.window.action-1.3.8] startup
[41.810s] [ext: omni.isaac.isaac_sensor-1.0.2] startup
2022-12-09 08:55:09 [41,811ms] [Warning] [omni.physx.plugin] Deprecated: getSimulationEventStream is deprecated, please use getSimulationEventStreamV2
[41.825s] [ext: omni.kit.selection-0.1.0] startup
[41.827s] [ext: omni.kit.widget.layers-1.5.17] startup
[41.850s] [ext: omni.graph.bundle.action-1.0.0] startup
[41.850s] [ext: omni.kit.profiler.window-1.4.4] startup
[41.863s] [ext: omni.kit.menu.edit-1.0.6] startup
[41.866s] [ext: omni.isaac.range_sensor-0.4.3] startup
[41.897s] [ext: omni.kit.property.layer-1.1.2] startup
[41.900s] [ext: omni.replicator.isaac-1.3.2] startup
[41.922s] [ext: omni.isaac.kit-0.2.1] startup
[41.924s] [ext: omni.graph.window.generic-1.3.8] startup
[41.926s] [ext: omni.isaac.utils-0.1.11] startup
[41.931s] [ext: omni.kit.property.audio-1.0.5] startup
[41.933s] [ext: omni.kit.property.skel-1.0.1] startup
[41.936s] [ext: omni.kit.property.render-1.1.0] startup
[41.937s] [ext: omni.kit.property.camera-1.0.3] startup
[41.939s] [ext: omni.kit.property.geometry-1.2.0] startup
[41.942s] [ext: omni.kit.property.light-1.0.5] startup
[41.944s] [ext: omni.kit.property.transform-1.0.2] startup
[41.948s] [ext: omni.isaac.universal_robots-0.3.0] startup
[41.950s] [ext: omni.isaac.occupancy_map-0.2.4] startup
[41.963s] [ext: omni.kit.window.console-0.2.0] startup
[41.969s] [ext: omni.kit.window.status_bar-0.1.1] startup
[41.974s] [ext: omni.kit.property.bundle-1.2.4] startup
[41.977s] [ext: omni.kit.menu.file-1.0.8] startup
[41.978s] [ext: omni.isaac.dofbot-0.3.0] startup
[41.980s] [ext: omni.kit.streamsdk.plugins-0.0.0] startup
[41.980s] [ext: omni.kit.livestream.core-0.0.0] startup
[41.985s] [ext: omni.kit.window.title-1.1.2] startup
2022-12-09 08:55:10 [41,979ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2022-12-09 08:55:10 [41,979ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.3]) (impl: carb.windowing-glfw.plugin)
2022-12-09 08:55:10 [41,980ms] [Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
2022-12-09 08:55:10 [41,980ms] [Error] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.3]) (impl: carb.windowing-glfw.plugin)
[41.989s] [ext: omni.isaac.urdf-0.4.0] startup
[42.023s] [ext: omni.kit.widget.live-0.1.0] startup
2022-12-09 08:55:10 [42,018ms] [Warning] [omni.ext-live.plugin] Unable to detect Omniverse Cache Server. File /root/.nvidia-omniverse/config/Cache/Omniverse.toml is not found. Consider installing it (version >= 280) for better IO performance.
[42.027s] [ext: omni.kit.menu.common-1.0.0] startup
[42.030s] [ext: omni.kit.livestream.native-0.1.4] startup
Active user not found. Using default user [kiosk]Streaming server started.
[42.118s] [ext: omni.isaac.sim.python-2022.1.1] startup
[42.120s] Simulation App Starting
[42.935s] app ready
[46.282s] Simulation App Startup Complete
task_name: ${task.name}
experiment:
num_envs:
seed: 42
torch_deterministic: False
max_iterations:
physics_engine: physx
pipeline: gpu
sim_device: gpu
device_id: 0
rl_device: cuda:0
num_threads: 4
solver_type: 1
test: False
checkpoint:
headless: False
wandb_activate: False
wandb_group:
wandb_name: test
wandb_entity: wazzup
wandb_project: isaactest
task:
name: FrankaCabinet
physics_engine: ${..physics_engine}
env:
numEnvs: 2
envSpacing: 3.0
episodeLength: 500
enableDebugVis: False
clipObservations: 5.0
clipActions: 1.0
controlFrequencyInv: 2
startPositionNoise: 0.0
startRotationNoise: 0.0
numProps: 4
aggregateMode: 3
actionScale: 7.5
dofVelocityScale: 0.1
distRewardScale: 2.0
rotRewardScale: 0.5
aroundHandleRewardScale: 10.0
openRewardScale: 7.5
fingerDistRewardScale: 100.0
actionPenaltyScale: 0.01
fingerCloseRewardScale: 10.0
sim:
dt: 0.0083
use_gpu_pipeline: True
gravity: [0.0, 0.0, -9.81]
add_ground_plane: True
use_flatcache: True
enable_scene_query_support: False
enable_cameras: False
default_physics_material:
static_friction: 1.0
dynamic_friction: 1.0
restitution: 0.0
physx:
worker_thread_count: 4
solver_type: 1
use_gpu: True
solver_position_iteration_count: 12
solver_velocity_iteration_count: 1
contact_offset: 0.005
rest_offset: 0.0
bounce_threshold_velocity: 0.2
friction_offset_threshold: 0.04
friction_correlation_distance: 0.025
enable_sleeping: True
enable_stabilization: True
max_depenetration_velocity: 1000.0
gpu_max_rigid_contact_count: 524288
gpu_max_rigid_patch_count: 33554432
gpu_found_lost_pairs_capacity: 524288
gpu_found_lost_aggregate_pairs_capacity: 262144
gpu_total_aggregate_pairs_capacity: 1048576
gpu_max_soft_body_contacts: 1048576
gpu_max_particle_contacts: 1048576
gpu_heap_capacity: 33554432
gpu_temp_buffer_capacity: 16777216
gpu_max_num_partitions: 8
franka:
override_usd_defaults: False
fixed_base: False
enable_self_collisions: False
enable_gyroscopic_forces: True
solver_position_iteration_count: 12
solver_velocity_iteration_count: 1
sleep_threshold: 0.005
stabilization_threshold: 0.001
density: -1
max_depenetration_velocity: 1000.0
contact_offset: 0.005
rest_offset: 0.0
cabinet:
override_usd_defaults: False
fixed_base: False
enable_self_collisions: False
enable_gyroscopic_forces: True
solver_position_iteration_count: 12
solver_velocity_iteration_count: 1
sleep_threshold: 0.0
stabilization_threshold: 0.001
density: -1
max_depenetration_velocity: 1000.0
contact_offset: 0.005
rest_offset: 0.0
prop:
override_usd_defaults: False
fixed_base: False
enable_self_collisions: False
enable_gyroscopic_forces: True
solver_position_iteration_count: 12
solver_velocity_iteration_count: 1
sleep_threshold: 0.005
stabilization_threshold: 0.001
density: 100
max_depenetration_velocity: 1000.0
contact_offset: 0.005
rest_offset: 0.0
train:
params:
seed: ${...seed}
algo:
name: a2c_continuous
model:
name: continuous_a2c_logstd
network:
name: actor_critic
separate: False
space:
continuous:
mu_activation: None
sigma_activation: None
mu_init:
name: default
sigma_init:
name: const_initializer
val: 0
fixed_sigma: True
mlp:
units: [256, 128, 64]
activation: elu
d2rl: False
initializer:
name: default
regularizer:
name: None
load_checkpoint: False
load_path: None
config:
name: FrankaCabinet
full_experiment_name: test
env_name: rlgpu
device: cuda:0
device_name: cuda:0
ppo: True
mixed_precision: False
normalize_input: True
normalize_value: True
num_actors: 2
reward_shaper:
scale_value: 0.01
normalize_advantage: True
gamma: 0.99
tau: 0.95
learning_rate: 5e-4
lr_schedule: adaptive
kl_threshold: 0.008
score_to_win: 100000000
max_epochs: 1500
save_best_after: 200
save_frequency: 100
print_stats: True
grad_norm: 1.0
entropy_coef: 0.0
truncate_grads: True
e_clip: 0.2
horizon_length: 16
minibatch_size: 16
mini_epochs: 8
critic_coef: 4
clip_value: True
seq_len: 4
bounds_loss_coef: 0.0001
name: FrankaCabinet
Sim params does not have attribute: physx
Sim params does not have attribute: franka
Sim params does not have attribute: cabinet
Sim params does not have attribute: prop
Pipeline: GPU
Pipeline Device: cuda:0
Sim Device: GPU
Task Device: cuda:0
RL device: cuda:0
/isaac-sim/exts/omni.isaac.ml_archive/pip_prebundle/gym/spaces/box.py:74: UserWarning: WARN: Box bound precision lowered by casting to float32
"Box bound precision lowered by casting to {}".format(self.dtype)
[46.493s] [ext: omni.physx.flatcache-1.4.15-5.1] startup
2022-12-09 08:55:14 [46,522ms] [Warning] [carb] Acquiring non optional plugin interface which is not listed as dependency: [omni::kit::IStageUpdate v1.0] (plugin: (null)), by client: omni.physx.flatcache.plugin. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client.
2022-12-09 08:55:14 [46,571ms] [Warning] [omni.client.plugin] Tick: authentication: Discovery(ws://localhost:3333): Error creating Api/Connection search: Not connected
2022-12-09 08:55:15 [47,748ms] [Warning] [omni.hydra.scene_delegate.plugin] Calling getBypassRenderSkelMeshProcessing for prim /World/envs/env_0/franka/panda_rightfinger/visuals.proto_visuals_id0 that has not been populated
2022-12-09 08:55:15 [47,798ms] [Warning] [omni.physx.plugin] ParseFEMClothMaterial: UsdPrim doesn't have a PhysxSchemaPhysxDeformableSurfaceMaterialAPI
gdb output:
Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007fa325614917 in ?? () from /isaac-sim/kit/extsPhysics/omni.physx-1.4.15-5.1/bin/libomni.physx.plugin.so
(gdb) bt
#0 0x00007fa325614917 in ?? () from /isaac-sim/kit/extsPhysics/omni.physx-1.4.15-5.1/bin/libomni.physx.plugin.so
#1 0x00007fa3256e899a in ?? () from /isaac-sim/kit/extsPhysics/omni.physx-1.4.15-5.1/bin/libomni.physx.plugin.so
#2 0x00007fa3256df6d8 in ?? () from /isaac-sim/kit/extsPhysics/omni.physx-1.4.15-5.1/bin/libomni.physx.plugin.so
#3 0x00007fa3257f85e5 in ?? () from /isaac-sim/kit/extsPhysics/omni.physx-1.4.15-5.1/bin/libomni.physx.plugin.so
#4 0x00007fa32572a9e2 in ?? () from /isaac-sim/kit/extsPhysics/omni.physx-1.4.15-5.1/bin/libomni.physx.plugin.so
#5 0x00007fa325136ed6 in ?? () from /isaac-sim/kit/extsPhysics/omni.physx-1.4.15-5.1/omni/physx/bindings/_physx.cpython-37m-x86_64-linux-gnu.so
#6 0x00007fa32513dde2 in ?? () from /isaac-sim/kit/extsPhysics/omni.physx-1.4.15-5.1/omni/physx/bindings/_physx.cpython-37m-x86_64-linux-gnu.so
#7 0x00000000004ba8f7 in _PyMethodDef_RawFastCallKeywords () at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:693
#8 0x00000000004ba6a6 in _PyCFunction_FastCallKeywords (func=func@entry=0x7fa3c4112fa0, args=args@entry=0x7f94c05a4a88, nargs=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:732
#9 0x00000000004c2c79 in _PyObject_FastCallKeywords () at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:156
#10 0x00000000004ba379 in call_function () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:4619
#11 0x00000000004b6cc2 in _PyEval_EvalFrameDefault () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3093
#12 0x00000000004c4f56 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=0x7f94c05a4910) at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:544
#13 function_code_fastcall (globals=<optimized out>, nargs=1, nargs@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, args=<optimized out>, co=<optimized out>) at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:283
#14 _PyFunction_FastCallKeywords () at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:408
#15 0x00000000004ba23f in call_function () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:4616
#16 0x00000000004b310b in _PyEval_EvalFrameDefault () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3110
#17 0x00000000004c4f56 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=0x7f94c059d050) at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:544
#18 function_code_fastcall (globals=<optimized out>, nargs=1, nargs@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, args=<optimized out>, co=<optimized out>) at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:283
#19 _PyFunction_FastCallKeywords () at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:408
#20 0x00000000004ba23f in call_function () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:4616
#21 0x00000000004b310b in _PyEval_EvalFrameDefault () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3110
#22 0x00000000004c4f56 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=0x7f94c0fbd810) at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:544
#23 function_code_fastcall (globals=<optimized out>, nargs=1, nargs@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, args=<optimized out>, co=<optimized out>) at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:283
#24 _PyFunction_FastCallKeywords () at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:408
#25 0x00000000004ba23f in call_function () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:4616
#26 0x00000000004b6cc2 in _PyEval_EvalFrameDefault () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3093
#27 0x00000000004b1411 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:547
#28 _PyEval_EvalCodeWithName () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3930
#29 0x00000000004c50ec in _PyFunction_FastCallKeywords () at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:433
#30 0x00000000004ba23f in call_function () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:4616
#31 0x00000000004b3c41 in _PyEval_EvalFrameDefault () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3139
#32 0x00000000004b1411 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:547
#33 _PyEval_EvalCodeWithName () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3930
#34 0x00000000004c50ec in _PyFunction_FastCallKeywords () at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:433
#35 0x00000000004ba23f in call_function () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:4616
#36 0x00000000004b310b in _PyEval_EvalFrameDefault () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3110
#37 0x00000000004b1411 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:547
#38 _PyEval_EvalCodeWithName () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3930
#39 0x00000000004c50ec in _PyFunction_FastCallKeywords () at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:433
#40 0x00000000004ba23f in call_function () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:4616
#41 0x00000000004b6cc2 in _PyEval_EvalFrameDefault () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3093
#42 0x00000000004b1411 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:547
#43 _PyEval_EvalCodeWithName () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3930
#44 0x00000000004c50ec in _PyFunction_FastCallKeywords () at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:433
#45 0x00000000004ba23f in call_function () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:4616
#46 0x00000000004b3c41 in _PyEval_EvalFrameDefault () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3139
#47 0x00000000004b1411 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:547
#48 _PyEval_EvalCodeWithName () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3930
#49 0x00000000004c50ec in _PyFunction_FastCallKeywords () at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:433
#50 0x00000000004ba23f in call_function () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:4616
#51 0x00000000004b2fe1 in _PyEval_EvalFrameDefault () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3124
#52 0x00000000004c4f56 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=0x623c580) at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:544
#53 function_code_fastcall (globals=<optimized out>, nargs=0, nargs@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, args=<optimized out>, co=<optimized out>) at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:283
#54 _PyFunction_FastCallKeywords () at /usr/local/src/conda/python-3.7.15/Modules/opcode_targets.h:408
---Type <return> to continue, or q <return> to quit---
#55 0x00000000004ba23f in call_function () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:4616
#56 0x00000000004b2fe1 in _PyEval_EvalFrameDefault () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3124
#57 0x00000000004b1411 in PyEval_EvalFrameEx (throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:547
#58 _PyEval_EvalCodeWithName () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3930
#59 0x00000000004b1209 in PyEval_EvalCodeEx () at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:3959
#60 0x000000000054f39b in PyEval_EvalCode (co=co@entry=0x7fa8effb0ed0, globals=globals@entry=0x7fa8f0f7b370, locals=locals@entry=0x7fa8f0f7b370) at /usr/local/src/conda/python-3.7.15/Programs/gcmodule.c:524
#61 0x000000000056a263 in run_mod () at /usr/local/src/conda/python-3.7.15/Modules/Python-ast.c:1037
#62 0x0000000000570a17 in PyRun_FileExFlags () at /usr/local/src/conda/python-3.7.15/Modules/Python-ast.c:990
#63 0x000000000056ffd4 in PyRun_SimpleFileExFlags () at /usr/local/src/conda/python-3.7.15/Modules/Python-ast.c:429
#64 0x000000000056fb5b in PyRun_AnyFileExFlags () at /usr/local/src/conda/python-3.7.15/Modules/Python-ast.c:84
#65 0x0000000000544be3 in pymain_run_file (p_cf=0x7ffd63b5ade0, filename=<optimized out>, fp=0x259f7b0) at /tmp/build/80754af9/python_1669321496543/work/build-static/python.c:470
#66 pymain_run_filename (cf=0x7ffd63b5ade0, pymain=0x7ffd63b5aef0) at /tmp/build/80754af9/python_1669321496543/work/build-static/python.c:1660
#67 pymain_run_python (pymain=0x7ffd63b5aef0) at /tmp/build/80754af9/python_1669321496543/work/build-static/python.c:2970
#68 pymain_main () at /tmp/build/80754af9/python_1669321496543/work/build-static/python.c:3517
#69 0x000000000054487c in _Py_UnixMain (argc=<optimized out>, argv=<optimized out>) at /tmp/build/80754af9/python_1669321496543/work/build-static/python.c:3552
#70 0x00007fa8f0109c87 in __libc_start_main (main=0x544830 <main>, argc=5, argv=0x7ffd63b5b048, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd63b5b038) at ../csu/libc-start.c:310
#71 0x000000000054472e in _start () at /usr/local/src/conda/python-3.7.15/Modules/pytime.c:2515
Any help would be much appreciated.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.