Not possible to run on any GPU other than cuda:0 about omniisaacgymenvs HOT 2 OPEN

Hi there, could you confirm the version of Isaac Sim you are working with? The GPU device should be set as a parameter when passed to the VecEnv class from rlgames_train.py. On the VecEnv side, this will be used when creating the Isaac Sim instance, which will set the /physics/cudaDevice value.

from omniisaacgymenvs.

Thanks! Updating to 2022.2.1 doesn't crash when running the following command:

python scripts/rlgames_train.py task=AnymalTerrain checkpoint=runs/AnymalTerrain/nn/AnymalTerrain.pth test=True num_envs=64 rl_device=cuda:1 device_id=1

But it still shows cuda:0 at 97% capacity using 136 w and is allocating memory on it. If I stop the process cuda:0 goes down to 5%. So it's definitely causing it. It is using some resources on cuda:1 but why would it be using so much more on cuda:0?

The simulation also does not behave properly. The Anymal do not move. If I don't pass these arguments it works fine.

|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 4090 On | 00000000:01:00.0 Off | Off |
| 0% 37C P2 136W / 450W| 6832MiB / 24564MiB | 97% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA GeForce RTX 4090 On | 00000000:41:00.0 Off | Off |
| 0% 33C P2 84W / 450W| 4376MiB / 24564MiB | 61% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

python scripts/rlgames_train.py task=AnymalTerrain checkpoint=runs/AnymalTerrain/nn/AnymalTerrain.pth test=True num_envs=64 rl_device=cuda:1 device_id=1
/home/bizon/anaconda3/envs/isaac-sim/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'config': Defaults list is missing _self_. See https://hydra.cc/docs/upgrades/1.0_to_1.1/default_composition_order for more information
warnings.warn(msg, UserWarning)
/home/bizon/anaconda3/envs/isaac-sim/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:412: UserWarning: In config: Invalid overriding of hydra/job_logging:
Default list overrides requires 'override' keyword.
See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/defaults_list_override for more information.

deprecation_warning(msg)
[Warning] [omni.isaac.kit.simulation_app] Modules: ['omni.isaac.kit.app_framework'] were loaded before SimulationApp was started and might not be loaded correctly.
[Warning] [omni.isaac.kit.simulation_app] Please check to make sure no extra omniverse or pxr modules are imported before the call to SimulationApp(...)
Starting kit application with the following args: ['/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py', '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/apps/omni.isaac.sim.python.kit', '--/app/tokens/exe-path=/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/kit', '--/persistent/app/viewport/displayOptions=3094', '--/rtx/materialDb/syncLoads=True', '--/rtx/hydra/materialSyncLoads=True--/omni.kit.plugin/syncUsdLoads=True', '--/app/renderer/resolution/width=1280', '--/app/renderer/resolution/height=720', '--/app/window/width=1440', '--/app/window/height=900', '--/renderer/multiGpu/enabled=True', '--/app/fastShutdown=True', '--ext-folder', '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/exts', '--ext-folder', '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/apps', '--/physics/cudaDevice=1', '--portable']
Passing the following args to the base kit application: ['task=AnymalTerrain', 'checkpoint=runs/AnymalTerrain/nn/AnymalTerrain.pth', 'test=True', 'num_envs=64', 'rl_device=cuda:1', 'device_id=1']
[Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
[Info] [carb] Logging to file: /home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/kit/logs/Kit/Isaac-Sim/2022.2/kit_20230505_080928.log
2023-05-05 18:09:28 [7ms] [Warning] [omni.ext.plugin] [ext: omni.sensors.nv.lidar] Extensions config 'extension.toml' doesn't exist '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.sensors.nv.lidar' or '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.sensors.nv.lidar/config'
2023-05-05 18:09:28 [7ms] [Warning] [omni.ext.plugin] [ext: omni.sensors.nv.radar] Extensions config 'extension.toml' doesn't exist '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.sensors.nv.radar' or '/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.sensors.nv.radar/config'
[0.105s] [ext: omni.stats-0.0.0] startup
[0.115s] [ext: omni.rtx.shadercache-1.0.0] startup
[0.119s] [ext: omni.assets.plugins-0.0.0] startup
[0.120s] [ext: omni.gpu_foundation-0.0.0] startup
[0.125s] [ext: carb.windowing.plugins-1.0.0] startup
2023-05-05 18:09:28 [123ms] [Warning] [carb.windowing-glfw.gamepad] Joystick with unknown remapping detected (will be ignored): ASRock LED Controller [03000000ce260000a201000010010000]
[0.131s] [ext: omni.kit.renderer.init-0.0.0] startup

|---------------------------------------------------------------------------------------------|
| Driver Version: 530.30.02 | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name | Active | LDA | GPU Memory | Vendor-ID | LUID |
| | | | | | Device-ID | UUID |
|---------------------------------------------------------------------------------------------|
| 0 | NVIDIA GeForce RTX 4090 | Yes: 1 | | 24810 MB | 10de | 0 |
| | | | | | 2684 | 527cadd8.. |
|---------------------------------------------------------------------------------------------|
| 1 | NVIDIA GeForce RTX 4090 | Yes: 0 | | 24810 MB | 10de | 0 |
| | | | | | 2684 | 7c9328c9.. |
|=============================================================================================|
| OS: Linux dl, Version: 5.15.0-69-generic
| XServer Vendor: The X.Org Foundation, XServer Version: 12013000 (1.20.13.0)
| Processor: AMD Ryzen Threadripper PRO 5975WX 32-Cores | Cores: Unknown | Logical: 64
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 257558 | Free Memory: 245449
| Total Page/Swap (MB): 2047 | Free Page/Swap: 2047
|---------------------------------------------------------------------------------------------|
2023-05-05 18:09:29 [955ms] [Warning] [gpu.foundation.plugin] IOMMU is enabled. Found 50 items in /sys/kernel/iommu_groups/.
2023-05-05 18:09:29 [955ms] [Warning] [gpu.foundation.plugin] On Linux only, CUDA and the display driver does not support IOMMU-enabled bare-metal PCIe peer to peer memory copy.
However, CUDA and the display driver does support IOMMU via VM pass through. As a consequence, users on Linux,
when running on a native bare metal system, should disable the IOMMU. The IOMMU should be enabled and the VFIO driver
be used as a PCIe pass through for virtual machines.
2023-05-05 18:09:29 [955ms] [Warning] [gpu.foundation.plugin]
2023-05-05 18:09:29 [955ms] [Warning] [gpu.foundation.plugin] -----------------------------------------------------------------------
2023-05-05 18:09:29 [955ms] [Warning] [gpu.foundation.plugin] !!!!! Local system validation failed! Incorrect configuration detected.
2023-05-05 18:09:29 [955ms] [Warning] [gpu.foundation.plugin] Summary below. Details above.
2023-05-05 18:09:29 [955ms] [Warning] [gpu.foundation.plugin] -----------------------------------------------------------------------
2023-05-05 18:09:29 [955ms] [Warning] [gpu.foundation.plugin]
2023-05-05 18:09:29 [955ms] [Warning] [gpu.foundation.plugin] - ECC: OK
2023-05-05 18:09:29 [956ms] [Warning] [gpu.foundation.plugin] - IOMMU: FAILED
2023-05-05 18:09:29 [956ms] [Warning] [gpu.foundation.plugin]
2023-05-05 18:09:29 [956ms] [Warning] [gpu.foundation.plugin] -----------------------------------------------------------------------
[1.461s] [ext: omni.kit.pipapi-0.0.0] startup
[1.466s] [ext: omni.kit.pip_archive-0.0.0] startup
[1.468s] [ext: omni.kit.loop-isaac-1.0.0] startup
[1.468s] [ext: omni.kit.async_engine-0.0.0] startup
[1.469s] [ext: omni.kit.test-0.0.0] startup
[1.480s] [ext: omni.usd.config-1.0.0] startup
[1.483s] [ext: omni.usd.libs-1.0.0] startup
[1.657s] [ext: omni.isaac.core_archive-2.0.1] startup
[1.663s] [ext: omni.pip.torch-1_13_1-0.1.4] startup
[1.665s] [ext: omni.isaac.ml_archive-1.1.0] startup
[1.665s] [ext: omni.client-0.1.1] startup
[1.672s] [ext: omni.appwindow-1.0.1] startup
[1.675s] [ext: omni.kit.renderer.core-0.0.0] startup
[1.927s] [ext: omni.kit.renderer.capture-0.0.0] startup
[1.928s] [ext: omni.kit.renderer.imgui-0.0.0] startup
[2.082s] [ext: carb.audio-0.1.0] startup
[2.107s] [ext: omni.ui-2.14.4] startup
[2.116s] [ext: omni.uiaudio-1.0.0] startup
[2.117s] [ext: omni.kit.mainwindow-1.0.0] startup
[2.117s] [ext: omni.kit.uiapp-0.0.0] startup
[2.118s] [ext: omni.usd.schema.physics-1.0.0] startup
[2.155s] [ext: omni.usd.schema.isaac-1.1.0] startup
[2.169s] [ext: omni.usd.schema.semantics-0.0.0] startup
[2.178s] [ext: omni.usd.schema.omnigraph-1.0.0] startup
[2.185s] [ext: omni.usd.schema.forcefield-0.0.0] startup
[2.191s] [ext: omni.usd.schema.omniscripting-1.0.0] startup
[2.197s] [ext: omni.usd.schema.geospatial-0.0.0] startup
[2.202s] [ext: omni.usd.schema.anim-0.0.0] startup
[2.250s] [ext: omni.usd.schema.audio-0.0.0] startup
[2.255s] [ext: omni.usd.schema.scene.visualization-1.0.0] startup
[2.255s] [ext: omni.usd.schema.physx-0.0.0] startup
[2.280s] [ext: omni.gpucompute.plugins-0.0.0] startup
[2.280s] [ext: omni.timeline-1.0.5] startup
[2.281s] [ext: omni.kit.search_core-1.0.2] startup
[2.282s] [ext: omni.hydra.scene_delegate-0.3.0] startup
[2.288s] [ext: omni.kit.widget.path_field-2.0.4] startup
[2.291s] [ext: omni.kit.widget.filebrowser-2.3.10] startup
[2.299s] [ext: omni.kit.widget.browser_bar-2.0.5] startup
[2.300s] [ext: omni.kit.window.popup_dialog-2.0.16] startup
[2.303s] [ext: omni.kit.actions.core-1.0.0] startup
[2.305s] [ext: omni.usd.core-1.0.4] startup
[2.308s] [ext: omni.kit.audiodeviceenum-1.0.0] startup
[2.308s] [ext: omni.kit.widget.nucleus_connector-1.0.3] startup
[2.309s] [ext: omni.kit.commands-1.4.5] startup
[2.312s] [ext: omni.hydra.usdrt_delegate-4.3.2] startup
[2.329s] [ext: omni.kit.notification_manager-1.0.5] startup
[2.331s] [ext: omni.usd-1.6.30] startup
[2.439s] [ext: omni.kit.widget.versioning-1.3.8] startup
[2.441s] [ext: omni.iray.libs-0.0.0] startup
[2.444s] [ext: omni.kit.window.filepicker-2.7.15] startup
[2.487s] [ext: omni.mdl.neuraylib-0.1.0] startup
[2.489s] [ext: omni.kit.collaboration.channel_manager-1.0.9] startup
[2.490s] [ext: omni.kit.window.file_importer-1.0.10] startup
[2.491s] [ext: omni.mdl-0.1.0] startup
[2.507s] [ext: omni.kit.usd.layers-2.0.11] startup
[2.514s] [ext: omni.kit.menu.utils-1.4.7] startup
[2.521s] [ext: omni.kit.material.library-1.3.21] startup
[2.525s] [ext: omni.kit.menu.create-1.0.8] startup
[2.526s] [ext: omni.kit.clipboard-1.0.0] startup
[2.526s] [ext: omni.kit.widget.graph-1.5.6-104_2] startup
[2.544s] [ext: omni.kit.context_menu-1.5.12] startup
[2.547s] [ext: omni.kit.window.extensions-1.1.1] startup
[2.550s] [ext: omni.inspect-1.0.1] startup
[2.553s] [ext: omni.kit.window.property-1.8.2] startup
[2.554s] [ext: omni.kit.primitive.mesh-1.0.8] startup
[2.557s] [ext: omni.graph.core-2.65.4] startup
[2.560s] [ext: omni.isaac.version-1.0.0] startup
[2.561s] [ext: omni.kit.stage_templates-1.1.13] startup
[2.562s] [ext: omni.kit.usd_undo-0.1.2] startup
[2.563s] [ext: omni.graph.tools-1.17.2] startup
[2.575s] [ext: omni.ui.scene-1.5.18] startup
[2.578s] [ext: omni.graph-1.50.2] startup
[2.613s] [ext: omni.ui_query-1.1.1] startup
[2.613s] [ext: omni.volume-0.1.0] startup
[2.615s] [ext: omni.kit.ui_test-1.2.9] startup
[2.616s] [ext: omni.hydra.rtx-0.1.0] startup
[2.622s] [ext: omni.kit.widget.searchfield-1.0.10] startup
[2.624s] [ext: omni.kit.graph.delegate.default-1.0.18] startup
[2.625s] [ext: omni.debugdraw-0.1.1] startup
[2.629s] [ext: omni.kit.graph.editor.core-1.4.7] startup
[2.631s] [ext: omni.hydra.pxr-1.1.2] startup
[2.636s] [ext: omni.activity.core-1.0.1] startup
[2.638s] [ext: omni.kit.hydra_texture-1.0.11] startup
[2.643s] [ext: omni.kit.viewport.legacy_gizmos-1.0.10] startup
[2.647s] [ext: omni.kit.viewport.registry-104.0.2] startup
[2.648s] [ext: omni.kit.widget.viewport-104.1.16] startup
[2.650s] [ext: omni.kit.window.drop_support-1.0.1] startup
[2.651s] [ext: omni.kit.window.file_exporter-1.0.10] startup
[2.651s] [ext: omni.hydra.engine.stats-1.0.0] startup
[2.655s] [ext: omni.kit.window.content_browser_registry-0.0.1] startup
[2.655s] [ext: omni.kit.window.file-1.3.32] startup
[2.657s] [ext: omni.kit.viewport.window-104.1.23] startup
[2.677s] [ext: omni.kit.widget.settings-1.0.1] startup
[2.678s] [ext: omni.kit.widget.stage-2.7.24] startup
[2.681s] [ext: omni.kit.window.content_browser-2.6.8] startup
[2.691s] [ext: omni.kit.viewport.utility-1.0.14] startup
[2.692s] [ext: omni.kit.window.preferences-1.3.8] startup
[2.716s] [ext: omni.kit.widget.text_editor-1.0.2] startup
[2.717s] [ext: omni.kit.widget.prompt-1.0.5] startup
[2.718s] [ext: omni.kit.property.usd-3.18.17] startup
[2.725s] [ext: omni.kit.graph.delegate.modern-1.6.5] startup
[2.726s] [ext: omni.graph.ui-1.24.2] startup
[2.835s] [ext: omni.physics.tensors-0.1.0] startup
[2.842s] [ext: omni.kit.graph.widget.variables-2.0.4] startup
[2.843s] [ext: omni.kit.graph.usd.commands-1.2.0] startup
[2.844s] [ext: omni.convexdecomposition-104.2.4-5.1] startup
[2.846s] [ext: omni.graph.window.core-1.43.3] startup
[2.852s] [ext: omni.kit.widget.zoombar-1.0.4] startup
[2.852s] [ext: omni.kvdb-0.0.0] startup
[2.854s] [ext: omni.usdphysics-104.2.4-5.1] startup
[2.857s] [ext: omni.localcache-0.0.0] startup
[2.858s] [ext: omni.graph.scriptnode-0.10.0] startup
[2.860s] [ext: omni.physx-104.2.4-5.1] startup
[2.894s] [ext: omni.command.usd-1.0.2] startup
[2.896s] [ext: omni.physx.tensors-0.1.0] startup
[2.901s] [ext: omni.graph.window.generic-1.3.15] startup
[2.902s] [ext: omni.kit.browser.core-2.2.2] startup
[2.906s] [ext: omni.graph.action-1.31.1] startup
[2.913s] [ext: omni.kit.browser.folder.core-1.7.3] startup
[2.915s] [ext: omni.kit.window.cursor-1.1.1] startup
[2.916s] [ext: omni.kit.widget.spinner-1.0.4] startup
[2.916s] [ext: omni.kit.browser.sample-1.2.5] startup
[2.918s] [ext: omni.graph.nodes-1.48.3] startup
[2.929s] [ext: omni.kit.viewport.menubar.core-104.4.3] startup
[2.938s] [ext: omni.kit.numpy.common-0.1.0] startup
[2.940s] [ext: omni.warp-0.6.3] startup
Warp 0.6.3 initialized:
CUDA Toolkit: 11.5, Driver: 12.1
Devices:
"cpu" | x86_64
"cuda:0" | NVIDIA GeForce RTX 4090 (sm_89)
"cuda:1" | NVIDIA GeForce RTX 4090 (sm_89)
Kernel cache: /home/bizon/.cache/warp/0.6.3
[3.103s] [ext: omni.syntheticdata-0.2.4] startup
2023-05-05 18:09:31 [3,104ms] [Warning] [omni.kit.viewport.menubar.core.menu_item.viewport_menu_item] ViewportMenuItem order argument is deprecated, use order_setting_path
[3.116s] [ext: omni.isaac.dynamic_control-1.2.3] startup
[3.125s] [ext: omni.replicator.core-1.7.8] startup
2023-05-05 18:09:31 [3,133ms] [Warning] [omni.replicator.core.scripts.annotators] Annotator PostProcessDispatch is already registered, overwriting annotator template
[3.205s] [ext: omni.isaac.core-1.46.3] startup
[3.274s] [ext: omni.kit.manipulator.viewport-104.0.7] startup
[3.275s] [ext: omni.isaac.core_nodes-0.23.2] startup
[3.287s] [ext: omni.isaac.cloner-0.4.1] startup
[3.287s] [ext: omni.isaac.lula-1.2.1] startup
[3.297s] [ext: omni.graph.visualization.nodes-2.0.0] startup
[3.403s] [ext: omni.isaac.debug_draw-0.2.3] startup
[3.409s] [ext: omni.isaac.gym-0.3.3] startup
[3.409s] [ext: omni.graph.window.action-1.5.5] startup
[3.410s] [ext: omni.isaac.ui-0.8.0] startup
[3.412s] [ext: usdrt.scenegraph-4.3.2] startup
[3.427s] [ext: omni.isaac.motion_planning-0.2.2] startup
[3.436s] [ext: omni.isaac.surface_gripper-0.4.0] startup
[3.440s] [ext: omni.isaac.range_sensor-0.8.1] startup
[3.468s] [ext: omni.isaac.motion_generation-4.5.6] startup
[3.472s] [ext: omni.isaac.manipulators-1.1.0] startup
[3.473s] [ext: omni.isaac.sensor-5.6.3] startup
2023-05-05 18:09:32 [3,465ms] [Warning] [omni.ext.plugin] [ext: omni.isaac.sensor-5.6.3] Native plugin search dir contains several files: /home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.sensor/bin . It makes dir walk slow, replace with the exact path to the plugin instead of a wildcard.
[3.510s] [ext: omni.isaac.dofbot-0.3.0] startup
[3.511s] [ext: omni.isaac.occupancy_map-0.2.7] startup
[3.521s] [ext: omni.isaac.quadruped-1.3.0] startup
[3.644s] [ext: omni.isaac.scene_blox-0.1.0] startup
[3.644s] [ext: omni.isaac.synthetic_utils-0.4.3] startup
[3.646s] [ext: omni.isaac.universal_robots-0.3.2] startup
[3.646s] [ext: omni.isaac.isaac_sensor-2.0.0] startup
2023-05-05 18:09:32 [3,638ms] [Warning] [omni.isaac.isaac_sensor] omni.isaac.isaac_sensor is deprecated and was renamed; use omni.isaac.sensor
[3.647s] [ext: omni.kit.widget.live_session_management-1.1.3] startup
[3.648s] [ext: omni.kit.hotkeys.core-1.0.8] startup
[3.652s] [ext: omni.isaac.wheeled_robots-0.6.3] startup
[3.657s] [ext: omni.kit.widget.stage_icons-1.0.2] startup
[3.658s] [ext: omni.kit.widget.layers-1.6.11] startup
[3.666s] [ext: omni.kit.selection-0.1.0] startup
[3.668s] [ext: omni.isaac.mjcf-0.3.1] startup
[3.675s] [ext: omni.kit.window.stage-2.3.12] startup
[3.677s] [ext: omni.kit.property.layer-1.1.4] startup
[3.679s] [ext: omni.kit.menu.edit-1.1.6] startup
[3.680s] [ext: omni.kit.menu.file-1.1.4] startup
[3.681s] [ext: omni.kit.menu.stage-1.2.1] startup
[3.682s] [ext: omni.kit.property.audio-1.0.6] startup
[3.683s] [ext: omni.kit.property.camera-1.0.3] startup
[3.684s] [ext: omni.kit.property.geometry-1.2.2] startup
[3.685s] [ext: omni.kit.property.light-1.0.6] startup
[3.686s] [ext: omni.hydra.scene_api-0.1.1] startup
[3.693s] [ext: omni.kit.property.material-1.8.19] startup
[3.695s] [ext: omni.kit.property.render-1.1.0] startup
[3.695s] [ext: omni.kit.property.skel-1.0.1] startup
[3.696s] [ext: omni.kit.property.transform-1.3.5] startup
[3.699s] [ext: omni.kit.manipulator.transform-104.6.12] startup
[3.700s] [ext: omni.kit.widget.toolbar-1.4.0] startup
[3.705s] [ext: omni.kit.profiler.window-2.0.5] startup
[3.752s] [ext: omni.kit.property.bundle-1.2.6] startup
[3.752s] [ext: omni.kit.manipulator.selector-1.0.1] startup
[3.753s] [ext: omni.kit.manipulator.tool.snap-1.2.4] startup
[3.758s] [ext: omni.kit.window.toolbar-1.4.0] startup
[3.759s] [ext: omni.kit.property.isaac-0.2.0] startup
[3.760s] [ext: omni.kit.stage_column.payload-1.0.8] startup
[3.762s] [ext: omni.kit.manipulator.camera-104.0.23] startup
[3.762s] [ext: omni.kit.manipulator.prim-104.10.7] startup
[3.766s] [ext: omni.kit.manipulator.selection-104.0.6] startup
[3.767s] [ext: omni.kit.viewport.menubar.display-104.1.11] startup
[3.768s] [ext: omni.kit.viewport.menubar.camera-104.1.9] startup
[3.771s] [ext: omni.rtx.window.settings-0.6.2] startup
[3.775s] [ext: omni.kit.viewport.menubar.settings-104.3.5] startup
[3.778s] [ext: omni.kit.viewport.menubar.render-104.0.20] startup
[3.779s] [ext: omni.physx.commands-104.2.4-5.1] startup
[3.783s] [ext: omni.kit.stage_column.variant-1.0.9] startup
[3.784s] [ext: omni.hydra.pxr.settings-1.0.3] startup
[3.785s] [ext: omni.kit.viewport.bundle-104.0.2] startup
[3.785s] [ext: omni.physx.ui-104.2.4-5.1] startup
[3.834s] [ext: omni.kit.viewport.menubar.lighting-104.0.9] startup
[3.836s] [ext: omni.kit.viewport.pxr-104.0.1] startup
[3.836s] [ext: omni.kit.viewport.ready-1.0.2] startup
[3.837s] [ext: omni.physx.demos-104.2.4-5.1] startup
[3.839s] [ext: omni.kit.property.physx-0.1.0] startup
[3.880s] [ext: omni.isaac.urdf-0.5.9] startup
[3.906s] [ext: omni.kit.widget.highlight_label-1.0.0] startup
[3.907s] [ext: omni.physx.supportui-104.2.4-5.1] startup
[3.919s] [ext: omni.physx.vehicle-104.2.4-5.1] startup
[3.929s] [ext: omni.physx.cct-104.2.4-5.1] startup
[3.936s] [ext: omni.physx.camera-104.2.4-5.1] startup
[3.941s] [ext: omni.kit.actions.window-1.1.0] startup
[3.944s] [ext: omni.kit.window.commands-0.2.4] startup
[3.946s] [ext: omni.kit.window.console-0.2.3] startup
[3.954s] [ext: omni.physx.bundle-104.2.4-5.1] startup
[3.954s] [ext: omni.graph.bundle.action-1.3.0] startup
[3.954s] [ext: omni.kit.window.script_editor-1.7.2] startup
[3.956s] [ext: omni.kit.hotkeys.window-1.1.3] startup
[3.960s] [ext: omni.kit.window.status_bar-0.1.5] startup
[3.964s] [ext: omni.kit.window.title-1.1.2] startup
[3.965s] [ext: omni.replicator.composer-1.2.10] startup
[3.968s] [ext: omni.replicator.isaac-1.7.4] startup
[4.111s] [ext: omni.resourcemonitor-1.0.0] startup
[4.117s] [ext: omni.rtx.settings.core-0.5.8] startup
[4.121s] [ext: omni.isaac.franka-0.4.0] startup
[4.122s] [ext: omni.kit.viewport.rtx-104.0.0] startup
[4.122s] [ext: semantics.schema.editor-0.3.3] startup
[4.125s] [ext: omni.kit.widget.live-2.0.3] startup
[4.127s] [ext: omni.isaac.utils-0.2.4] startup
[4.130s] [ext: omni.isaac.kit-1.4.1] startup
[4.130s] [ext: omni.isaac.cortex-0.3.2] startup
[4.131s] [ext: omni.kit.window.stats-0.1.2] startup
[4.131s] [ext: omni.isaac.sim.python-2022.2.1] startup
[4.132s] Simulation App Starting
[8.610s] app ready
[8.769s] RTX ready
[8.978s] Simulation App Startup Complete
task:
name: AnymalTerrain
physics_engine: physx
env:
numEnvs: 64
numObservations: 188
numActions: 12
envSpacing: 3.0
terrain:
staticFriction: 1.0
dynamicFriction: 1.0
restitution: 0.0
curriculum: True
maxInitMapLevel: 0
mapLength: 8.0
mapWidth: 8.0
numLevels: 10
numTerrains: 20
terrainProportions: [0.1, 0.1, 0.35, 0.25, 0.2]
slopeTreshold: 0.5
baseInitState:
pos: [0.0, 0.0, 0.62]
rot: [1.0, 0.0, 0.0, 0.0]
vLinear: [0.0, 0.0, 0.0]
vAngular: [0.0, 0.0, 0.0]
randomCommandVelocityRanges:
linear_x: [-1.0, 1.0]
linear_y: [-1.0, 1.0]
yaw: [-3.14, 3.14]
control:
stiffness: 80.0
damping: 2.0
actionScale: 0.5
decimation: 4
defaultJointAngles:
LF_HAA: 0.03
LH_HAA: 0.03
RF_HAA: -0.03
RH_HAA: -0.03
LF_HFE: 0.4
LH_HFE: -0.4
RF_HFE: 0.4
RH_HFE: -0.4
LF_KFE: -0.8
LH_KFE: 0.8
RF_KFE: -0.8
RH_KFE: 0.8
learn:
terminalReward: 0.0
linearVelocityXYRewardScale: 1.0
linearVelocityZRewardScale: -4.0
angularVelocityXYRewardScale: -0.05
angularVelocityZRewardScale: 0.5
orientationRewardScale: -0.0
torqueRewardScale: -2e-05
jointAccRewardScale: -0.0005
baseHeightRewardScale: -0.0
actionRateRewardScale: -0.01
fallenOverRewardScale: -1.0
hipRewardScale: -0.0
linearVelocityScale: 2.0
angularVelocityScale: 0.25
dofPositionScale: 1.0
dofVelocityScale: 0.05
heightMeasurementScale: 5.0
addNoise: True
noiseLevel: 1.0
dofPositionNoise: 0.01
dofVelocityNoise: 1.5
linearVelocityNoise: 0.1
angularVelocityNoise: 0.2
gravityNoise: 0.05
heightMeasurementNoise: 0.06
pushInterval_s: 15
episodeLength_s: 20
sim:
dt: 0.005
use_gpu_pipeline: True
gravity: [0.0, 0.0, -9.81]
add_ground_plane: False
use_flatcache: True
enable_scene_query_support: False
disable_contact_processing: True
enable_cameras: False
default_physics_material:
static_friction: 1.0
dynamic_friction: 1.0
restitution: 0.0
physx:
worker_thread_count: 4
solver_type: 1
use_gpu: True
solver_position_iteration_count: 4
solver_velocity_iteration_count: 0
contact_offset: 0.02
rest_offset: 0.0
bounce_threshold_velocity: 0.2
friction_offset_threshold: 0.04
friction_correlation_distance: 0.025
enable_sleeping: True
enable_stabilization: True
max_depenetration_velocity: 100.0
gpu_max_rigid_contact_count: 524288
gpu_max_rigid_patch_count: 163840
gpu_found_lost_pairs_capacity: 4194304
gpu_found_lost_aggregate_pairs_capacity: 33554432
gpu_total_aggregate_pairs_capacity: 4194304
gpu_max_soft_body_contacts: 1048576
gpu_max_particle_contacts: 1048576
gpu_heap_capacity: 134217728
gpu_temp_buffer_capacity: 33554432
gpu_max_num_partitions: 8
anymal:
override_usd_defaults: False
enable_self_collisions: True
enable_gyroscopic_forces: False
solver_position_iteration_count: 4
solver_velocity_iteration_count: 0
sleep_threshold: 0.005
stabilization_threshold: 0.001
density: -1
max_depenetration_velocity: 100.0
train:
params:
seed: 42
algo:
name: a2c_continuous
model:
name: continuous_a2c_logstd
network:
name: actor_critic
separate: True
space:
continuous:
mu_activation: None
sigma_activation: None
mu_init:
name: default
sigma_init:
name: const_initializer
val: 0.0
fixed_sigma: True
mlp:
units: [512, 256, 128]
activation: elu
d2rl: False
initializer:
name: default
regularizer:
name: None
load_checkpoint: True
load_path: /home/bizon/eric/OmniIsaacGymEnvs/omniisaacgymenvs/runs/AnymalTerrain/nn/AnymalTerrain.pth
config:
name: AnymalTerrain
full_experiment_name: AnymalTerrain
device: cuda:1
device_name: cuda:1
env_name: rlgpu
multi_gpu: False
ppo: True
mixed_precision: False
normalize_input: True
normalize_value: True
normalize_advantage: True
value_bootstrap: True
clip_actions: False
num_actors: 64
reward_shaper:
scale_value: 1.0
gamma: 0.99
tau: 0.95
e_clip: 0.2
entropy_coef: 0.001
learning_rate: 0.0003
lr_schedule: adaptive
kl_threshold: 0.008
truncate_grads: True
grad_norm: 1.0
horizon_length: 48
minibatch_size: 16384
mini_epochs: 5
critic_coef: 2
clip_value: True
seq_len: 4
bounds_loss_coef: 0.0
max_epochs: 2000
save_best_after: 100
score_to_win: 20000
save_frequency: 50
print_stats: True
task_name: AnymalTerrain
experiment:
num_envs: 64
seed: 42
torch_deterministic: False
max_iterations:
physics_engine: physx
pipeline: gpu
sim_device: gpu
device_id: 1
rl_device: cuda:1
multi_gpu: False
num_threads: 4
solver_type: 1
test: True
checkpoint: /home/bizon/eric/OmniIsaacGymEnvs/omniisaacgymenvs/runs/AnymalTerrain/nn/AnymalTerrain.pth
headless: False
enable_livestream: False
mt_timeout: 30
wandb_activate: False
wandb_group:
wandb_name: AnymalTerrain
wandb_entity:
wandb_project: omniisaacgymenvs
Setting seed: 42
Sim params does not have attribute: physx
Sim params does not have attribute: anymal
Pipeline: GPU
Pipeline Device: cuda:1
Sim Device: GPU
Task Device: cuda:1
RL device: cuda:1
/home/bizon/anaconda3/envs/isaac-sim/lib/python3.7/site-packages/gym/spaces/box.py:84: UserWarning: WARN: Box bound precision lowered by casting to float32
logger.warn(f"Box bound precision lowered by casting to {self.dtype}")
/home/bizon/.local/share/ov/pkg/isaac_sim-2022.2.1/extscache/omni.pip.torch-1_13_1-0.1.4+104.2.lx64/torch-1-13-1/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
[10.759s] [ext: omni.physx.flatcache-104.2.4-5.1] startup
2023-05-05 18:09:42 [14,075ms] [Warning] [omni.hydra.scene_delegate.plugin] Calling getBypassRenderSkelMeshProcessing for prim /World/envs/env_0/anymal/base/visuals.proto_mesh_12_id12 that has not been populated
*~
*~ Direct GPU Helper:
*~ 64 articulations, maxLinks=13
*~
self.seed = 42
Started to play
[BasePlayer] Creating regular env: rlgpu
{'observation_space': Box(-inf, inf, (188,), float32), 'action_space': Box(-1.0, 1.0, (12,), float32), 'agents': 1, 'value_size': 1}
build mlp: 188
build mlp: 188
RunningMeanStd: (1,)
RunningMeanStd: (188,)
=> loading checkpoint '/home/bizon/eric/OmniIsaacGymEnvs/omniisaacgymenvs/runs/AnymalTerrain/nn/AnymalTerrain.pth'
[2023-05-05 08:09:45] Running RL reset
reward: 7.2 steps: 998.0
reward: 5.6 steps: 999.0
reward: 4.4 steps: 999.0
reward: 0.0 steps: 780.0
reward: 0.6 steps: 794.0
reward: 0.1 steps: 802.0
reward: 0.5 steps: 817.0
reward: 6.7 steps: 999.0
reward: 3.0 steps: 999.0
reward: 8.8 steps: 999.0
reward: 15.8 steps: 999.0

from omniisaacgymenvs.