isaac-sim / isaaclab Goto Github PK
View Code? Open in Web Editor NEWUnified framework for robot learning built on NVIDIA Isaac Sim
Home Page: https://isaac-sim.github.io/IsaacLab
License: Other
Unified framework for robot learning built on NVIDIA Isaac Sim
Home Page: https://isaac-sim.github.io/IsaacLab
License: Other
The teleop example (source/standalone/environments/teleoperation/teleop_se3_agent.py) fails. When launched using the --cpu option or not, the script runs, but I'm not able to control the robot using the keyboard. I simply launch it in the terminal and press keys on my keyboard, making them printed in this same terminal, but not moving the robot in the simulator. When pressing keys in the simulator, 'K' does toggle the gripper (which is expected), but pressing other keys simply selects different modes in the left panel of the simulator.
Run teleoperation demo in API Demos:
./orbit.sh -p source/standalone/environments/teleoperation/teleop_se3_agent.py --task Isaac-Lift-Franka-v0 --num_envs 1 --cpu --device keyboard
No error in the terminal:
Bug first mentioned here. Potential cause of the error mentioned here.
Hi, is there hyperparameter tuning and grid search tools in Orbit?
for example, I'm looking for rl_utils
/hyperparameters_optimization()
like options for stable baseline3.
thanks!
I propose adding the gamepad as a device for teleoperation.
A gamepad will make it easier for the teleoperation of the robot arms than the keyboard. Supporting the gamepad also makes the feature set more comprehensive.
Implementations of gamepad
in source/extensions/omni.isaac.orbit/omni/isaac/orbit/devices/gamepad
, providing classes such as Se2Gamepad
and Se3Gamepad
.
No
Gamepad has already been used for camera navigation. So users need to unselect the "Gamepad Camera Control" option in the GUI before using it for teleoperation.
I have just checked the roadmap and there is no a parallel camera support.
I believe that the parallel camera support (and an example) would be very useful for an image based DRL.
Hi. I want to do a toy problem for the soft-cube grasping.
Here is my modified version of soft_prim_view
, which come from /home/bong/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.core/omni/isaac/core/prims/rigid_prim_view.py
soft_prim_view.py
class SoftPrimView(XFormPrimView):
...
def initialize(self, physics_sim_view: omni.physics.tensors.SimulationView = None) -> None:
if physics_sim_view is None:
physics_sim_view = omni.physics.tensors.create_simulation_view(self._backend)
physics_sim_view.set_subspace_roots("/")
carb.log_info("initializing view for {}".format(self._name))
self._physics_sim_view = physics_sim_view
self._physics_view = physics_sim_view.create_soft_body_view(self._regex_prim_paths.replace(".*", "*")) # CHANGED!
# self._num_shapes = self._physics_view.max_shapes # for rigid
carb.log_info("Soft Prim View Device: {}".format(self._device))
# if self._track_contact_forces: # for rigid
# self._contact_view.initialize(self._physics_sim_view)
but self._physics_view._backend
is None
, which causes the problem in api.py
api.py
class SoftBodyView:
def __init__(self, backend, frontend):
self._backend = backend
self._frontend = frontend
@property
def count(self):
return self._backend.count
currently, is function physics_sim_view.create_soft_body_view
unavailable?
...or what I missed?
thanks!
Hi, I want to run sb3/play.py with my checkpoint.
I configured conda environment, orbit, into VSCode.
Problem: --num_envs=1
option cause a problem when the first episode is done. --num_envs=2,3,4...
is ok.
+ My Guess: type of obs or reward is numpy.array
. I guess there is some dimension shrinks when --num_envs=1
.
Thanks!
Error Message
Exception has occurred: IndexError
index 1 is out of bounds for axis 0 with size 1
File "/home/bong/.local/share/ov/pkg/isaac_sim-2022.2.1/Orbit/source/standalone/workflows/sb3/play.py", line 83, in main
obs, reward, done, env_info = env.step(actions)
File "/home/bong/.local/share/ov/pkg/isaac_sim-2022.2.1/Orbit/source/standalone/workflows/sb3/play.py", line 96, in <module>
main()
IndexError: index 1 is out of bounds for axis 0 with size 1
and this is my launch.json
{
"version": "0.2.0",
"configurations": [
{
"name": "Python: Current File",
"type": "python",
"request": "launch",
"args" : ["--task","Lift-Robotiq-v0","--num_envs","1", "--checkpoint","./logs/sb3/Lift-Robotiq-v0/Apr12_07-12-28/model_16384000_steps.zip","--cpu"], // play
"program": "${file}",
"console": "integratedTerminal",
"env": {
"EXP_PATH": "${workspaceFolder}/_isaac_sim/apps",
"RESOURCE_NAME": "IsaacSim"
},
"envFile": "${workspaceFolder}/.vscode/.python.env",
"preLaunchTask": "setup_python_env"
},
{
"name": "Python: Attach (windows-x86_64/linux-x86_64)",
"type": "python",
"request": "attach",
"port": 3000,
"host": "localhost",
},
]
}
Awesome paper and work so far, I was curious about setting up environments with visual observations. Is that possible yet? If so how do you configure that? I couldn't find anything about visual observations (for RL workflows) in the docs.
Thanks
I have a question. My task involves pushing a cube on an inclined plane by training Franka Panda. The source code allows us to create a default ground plane but I want to create an inclined plane with angle as a parameter. How can I go about it?
Omniverse Streaming Client is a lightweight application allowing Users to connect to Omniverse applications deployed on the network or over the Internet.
https://docs.omniverse.nvidia.com/app_streaming-client/app_streaming-client/overview.html
It is quite useful for people who are developing on remote servers. Thus, it would help to have a pipeline to enable this functionality easily.
import carb
from omni.isaac.core.utils.extensions import enable_extension
from omni.isaac.core.utils.carb import set_carb_setting
# settings interface
carb_settings = carb.settings.get_settings()
# set values
set_carb_setting(carb_settings, "/app/livestream/enabled", True)
set_carb_setting(carb_settings, "/app/window/drawMouse", True)
set_carb_setting(carb_settings, "/app/livestream/proto", "ws")
set_carb_setting(carb_settings, "/app/livestream/websocket/framerate_limit", 120)
set_carb_setting(carb_settings, "/ngx/enabled", False)
# enable extensions for live stream
enable_extension("omni.kit.livestream.native")
enable_extension("omni.services.streaming.manager")
REMOTE_ENABLED
flag in the terminal (corresponding: PR).While developing a Docker container setup for Orbit (see here) I have encountered issues in running the demos. In particular I am seeing warnings related to accessing assets and an error generated by set_articulation_properties
in the kit.py
file. It seems like the path provided for the articulation primitive is incorrect in my setup see here.
I am posting this here for visibility and in case there is a quick fix for this or something basic I am missing. All files that I am using can be found here.
orbit_1 | [145.454s] Simulation App Startup Complete
orbit_1 | 2023-02-08 15:54:02 [147,443ms] [Warning] [omni.usd] Warning: in _ReportErrors at line 2830 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/usd/usd/stage.cpp -- Could not open asset @http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2022.1/Isaac/Props/Mounts/SeattleLabTable/table_instanceable.usd@ for reference on prim @anon:0x254cb410:World0.usd@,@anon:0x22ea3a60:World0-session.usda@</World/Table_1>. (recomposing stage on stage @anon:0x254cb410:World0.usd@ <0x2509eb30>)
orbit_1 |
orbit_1 | 2023-02-08 15:54:02 [147,447ms] [Warning] [omni.usd] Warning: in _ReportErrors at line 2830 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/usd/usd/stage.cpp -- Could not open asset @http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2022.1/Isaac/Props/Mounts/SeattleLabTable/table_instanceable.usd@ for reference on prim @anon:0x254cb410:World0.usd@,@anon:0x22ea3a60:World0-session.usda@</World/Table_2>. (recomposing stage on stage @anon:0x254cb410:World0.usd@ <0x2509eb30>)
orbit_1 |
orbit_1 | 2023-02-08 15:54:02 [147,645ms] [Warning] [omni.usd] Warning: in _ReportErrors at line 2830 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/usd/usd/stage.cpp -- Could not open asset @http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2022.1/Isaac/Samples/Orbit/Robots/FrankaEmika/panda_instanceable.usd@ for reference on prim @anon:0x254cb410:World0.usd@,@anon:0x22ea3a60:World0-session.usda@</World/Robot_1>. (recomposing stage on stage @anon:0x254cb410:World0.usd@ <0x2509eb30>)
orbit_1 |
orbit_1 | Traceback (most recent call last):
orbit_1 | File "./source/standalone/demo/play_arms.py", line 159, in <module>
orbit_1 | main()
orbit_1 | File "./source/standalone/demo/play_arms.py", line 94, in main
orbit_1 | robot.spawn("/World/Robot_1", translation=(0.0, -1.0, 0.0))
orbit_1 | File "/isaac-sim/Orbit/source/extensions/omni.isaac.orbit/omni/isaac/orbit/robots/robot_base.py", line 150, in spawn
orbit_1 | solver_velocity_iteration_count=self.cfg.articulation_props.solver_velocity_iteration_count,
orbit_1 | File "/isaac-sim/Orbit/source/extensions/omni.isaac.orbit/omni/isaac/orbit/utils/kit.py", line 274, in set_articulation_properties
orbit_1 | raise ValueError(f"No articulation schema present for prim '{prim_path}'.")
orbit_1 | ValueError: No articulation schema present for prim '/World/Robot_1'.
orbit_1 | /isaac-sim/python.sh: line 40: 23 Segmentation fault (core dumped) $python_exe "$@" $args
orbit_1 | There was an error running python
docker_orbit_1 exited with code 1
I'm trying to use LeggedMobileManipulator
to combine a quadruped and a robot arm. I use 'convert_urdf.py' to convert a URDF to USD. However, the joint indices were not as [base_dof_nums, arm_dof_nums, tool_dof_nums(which is 0 in my case)] after the robot init. (The dof indices were initialized in line 212 of articulation_view.py
.) So there may be some problems when I call robot.get_default_dof_state() or robot.apply_action(), etc.
My question is how to make sure the joint order is like [base, arm, tool] when importing the USD file? Thanks a lot!
Hi, there seems to be several environments missing in the repo right now compared to the paper. I'm wondering if they are not in the orbit_env directory, or they are yet to be released? If latter, is there any timeline on when they'll be available?
Thanks!
Hi this is my code. I modified frank.py, and created my own code.
"""
# from omni.isaac.orbit.actuators.config.franka import PANDA_HAND_MIMIC_GROUP_CFG
from omni.isaac.orbit.actuators.group import ActuatorGroupCfg, GripperActuatorGroupCfg
from omni.isaac.orbit.actuators.group.actuator_group_cfg import ActuatorControlCfg
from omni.isaac.orbit.actuators.model import ImplicitActuatorCfg
# from omni.isaac.orbit.utils.assets import ISAAC_ORBIT_NUCLEUS_DIR
from omni.isaac.orbit.robots.single_arm import SingleArmManipulatorCfg
# _FRANKA_PANDA_ARM_INSTANCEABLE_USD = f"{ISAAC_ORBIT_NUCLEUS_DIR}/Robots/FrankaEmika/panda_instanceable.usd"
# _ROBOTIQ_WRIST_INSTANCEABLE_USD = "/home/bong/.local/share/ov/pkg/isaac_sim-2022.2.1/Orbit/source/extensions/omni.isaac.orbit_envs/omni/isaac/orbit_envs/bong/robotiq_wrist.usd"
_ROBOTIQ_WRIST_INSTANCEABLE_USD = "/home/bong/.local/share/ov/pkg/isaac_sim-2022.2.1/Orbit/source/extensions/omni.isaac.orbit_envs/omni/isaac/orbit_envs/bong/robotiq_wrist_resized_direction.usd"
# _ROBOTIQ_WRIST_INSTANCEABLE_USD = "/home/bong/.local/share/ov/pkg/isaac_sim-2022.2.1/Orbit/source/extensions/omni.isaac.orbit_envs/omni/isaac/orbit_envs/bong/robotiq_wrist_resized_direction_mass.usd"
VELOCITY_LIMIT = 0.000000000000001
TORQUE_LIMIT = 100
STIFFNESS = 400
ROBOTIQ_WRIST_WITH_ROBOTIQ_CFG = SingleArmManipulatorCfg(
meta_info=SingleArmManipulatorCfg.MetaInfoCfg(
usd_path=_ROBOTIQ_WRIST_INSTANCEABLE_USD,
arm_num_dof=6,
tool_num_dof=4,
tool_sites_names=["bot_link_1", "bot_link_2", "top_link_1", "top_link_2"], # xform
),
init_state=SingleArmManipulatorCfg.InitialStateCfg( # revjoint
pos=[0.2, 0, 0.3],
rot=[0.707, 0, 0.707, 0],
dof_pos={
"trans_x": 0.0,
"trans_y": 0.0,
"trans_z": 0.0,
"rev_x": 0.0,
"rev_y": 0.0,
"rev_z": 0.0,
"bot_joint_0_p": 0.0,
"top_joint_0_p": 0.0,
"bot_joint_1_p": 0.0,
"top_joint_1_p": 0.0,
"bot_joint_2_p": 0.0,
"top_joint_2_p" : 0.0,
},
dof_vel={".*": 0.0},
),
ee_info=SingleArmManipulatorCfg.EndEffectorFrameCfg(
body_name="base", pos_offset=(0.0, 0.0, 0.15), rot_offset=(1.0, 0.0, 0.0, 0.0) # xform / head
),
rigid_props=SingleArmManipulatorCfg.RigidBodyPropertiesCfg(
# max_depenetration_velocity=5.0,
max_depenetration_velocity=0.0000000000001,
),
collision_props=SingleArmManipulatorCfg.CollisionPropertiesCfg(
contact_offset=0.005,
rest_offset=0.0,
),
articulation_props=SingleArmManipulatorCfg.ArticulationRootPropertiesCfg(
enable_self_collisions=True,
),
actuator_groups={
"wrist_trans": ActuatorGroupCfg(
dof_names=["trans_[x,y,z]"],
model_cfg=ImplicitActuatorCfg(velocity_limit=VELOCITY_LIMIT, torque_limit=TORQUE_LIMIT),
control_cfg=ActuatorControlCfg(
command_types=["p_abs"],
stiffness={".*": 400.0},
damping={".*": 20.0},
dof_pos_offset={
"trans_x": 0.0,
"trans_y": 0.0,
"trans_z": 0.0,
},
),
),
"wrist_rev": ActuatorGroupCfg(
dof_names=["rev_[x-z]"],
model_cfg=ImplicitActuatorCfg(velocity_limit=VELOCITY_LIMIT, torque_limit=TORQUE_LIMIT),
control_cfg=ActuatorControlCfg(
command_types=["p_abs"],
stiffness={".*": 400.0},
damping={".*": 20.0},
dof_pos_offset={"rev_x": 0, "rev_y": 0, "rev_z": 0.0},
),
),
"robotiq_hand": GripperActuatorGroupCfg(
dof_names=["bot_joint_0_p", "top_joint_0_p", "bot_joint_1_p", "top_joint_1_p", "bot_joint_2_p", "top_joint_2_p"],
model_cfg=ImplicitActuatorCfg(velocity_limit=VELOCITY_LIMIT, torque_limit=TORQUE_LIMIT),
control_cfg=ActuatorControlCfg(command_types=["p_abs"], stiffness={".*": 400}, damping={".*": 20}),
mimic_multiplier={"bot_joint_0_p": 1, "top_joint_0_p": -1, "bot_joint_1_p": 1, "top_joint_1_p": -1, "bot_joint_2_p": -1, "top_joint_2_p": 1},
# speed=1,
open_dof_pos=0,
close_dof_pos=0.785398,
)
},
)
"""Configuration of Franka arm with Franka Hand using implicit actuator models."""
as you can see, I set VELOCITY_LIMIT = 0.000000000000001
and max_depenetration_velocity=0.0000000000001,
.
But the robot still moves so fast.
How can I change it?
I also checked docs, but I have no idea what is wrong.
I want to limit velocity only while keeping proper torque.
thanks!
Hi,
I'm preparing a simple example to study Orbit. Currently, my robot is made up of a 6-DoF Wrist and a 1-DoF gripper with a mimic.
Problem: Policy Network returns 7-DoF currently.
What I want to do: I only need 6-DoF action for the wrist, and I want to close the gripper manually on _step_impl.
Question: How can I reduce the action space of the policy network, keeping the articulation of the gripper alive?
Thank you for your attention!
Best,
Bong.
Here are my code snippets for the robot (which is similar to franka.py)
robotiq.py: please see comments
actuator_groups={
"wrist_trans": ActuatorGroupCfg(
dof_names=["trans_[x,y,z]"],
model_cfg=ImplicitActuatorCfg(velocity_limit=VELOCITY_LIMIT, torque_limit=TORQUE_LIMIT),
control_cfg=ActuatorControlCfg(
command_types=["p_abs"],
stiffness={".*": 400.0},
damping={".*": 20.0},
dof_pos_offset={
"trans_x": 0.0,
"trans_y": 0.0,
"trans_z": 0.0,
},
),
),
"wrist_rev": ActuatorGroupCfg(
dof_names=["rev_[x-z]"],
model_cfg=ImplicitActuatorCfg(velocity_limit=VELOCITY_LIMIT, torque_limit=TORQUE_LIMIT),
control_cfg=ActuatorControlCfg(
command_types=["p_abs"],
stiffness={".*": 400.0},
damping={".*": 20.0},
dof_pos_offset={"rev_x": 0, "rev_y": 0, "rev_z": 0.0},
),
),
# If I comment on this part,
# policy network returns only six action spaces, that's good!
# However, I cannot control the gripper's end-effector. Please see _step_impl
"robotiq_hand": GripperActuatorGroupCfg(
dof_names=["bot_joint_0_p", "top_joint_0_p", "bot_joint_1_p", "top_joint_1_p", "bot_joint_2_p", "top_joint_2_p"],
model_cfg=ImplicitActuatorCfg(velocity_limit=VELOCITY_LIMIT, torque_limit=TORQUE_LIMIT),
control_cfg=ActuatorControlCfg(command_types=["p_abs"], stiffness={".*": 400}, damping={".*": 20}),
mimic_multiplier={"bot_joint_0_p": 1, "top_joint_0_p": -1, "bot_joint_1_p": 1, "top_joint_1_p": -1, "bot_joint_2_p": -1, "top_joint_2_p": 1},
# speed=1,
open_dof_pos=0,
close_dof_pos=0.785398,
)
},
Below is snippets that are parts of lift/lift_env
_step_impl please see my comments!
def _step_impl(self, actions: torch.Tensor):
...
elif self.cfg.control.control_type == "default":
self.robot_actions[:] = self.actions # Without comment in "robotiq.py" -> self.robot_actions.shape is [:,7]
# With a comment in "robotiq.py" -> self.robot_actions.shape is [:,6]
# HERE IS MY CODE!!
# Without comment "robotiq.py", it works, but there is useless action space[:,-1] on policy returns!
# With the comment "robotiq.py", I cannot access gripper's robot action!
if is_ee_close_to_object()
self.robot_actions[:, -1] = 0 # Without comment "robotiq.py" -> close the gripper
# With the comment "robotiq.py" -> it access the last joint of the wrist.
...
Hi, thanks for the great work of putting the paper, library and benchmark together.
In the paper you mention that Orbit supports both YCB and Partnet-Mobility. I found the the example to use YCB (play_ycb_objects.py) but didn't find any details on Partnet-Mobility. I also couldn't find them looking through the assets in the Isaac nucleus.
Do you have any details on how to import Partnet-Mobility into Isaac / Orbit? Thanks a lot.
In your Robotic Workflows/Reinforcement learning video (https://isaac-orbit.github.io/) you have an example of the robot opening a drawer, we also see you have an ArticulatedObject class, however we did not find any example of how to create such an object. It would be really appreciated if you could release this example, and also a sample .usd file of an articulated object.
Thank you very much in advance
Hi
There are two bugs in omni.isaac.orbit.core.utils.math
axis_angle_from_quat
doesn't properly handle quat with negative w componentsubtract_frame_transforms
have a missing tf combine when calculating the translation partI will submit a related PR to fix those issues.
Hello, I want to add dummy steps to the environment. For example, it would look like this:
Dummy action: The gripper moves 1 unit along the x-axis for 10 steps due to the dummy action, then closes to grab the object.
Action: The robot is driven by the agent. The robot performs a specific task (e.g., rotation).
Dummy action: After the rotation is performed, the gripper moves 1 unit along the y-axis due to the dummy action.
In this case, the entire environment has early termination and a multi-environment.
The important point here is that the rollout should not be contaminated. That is, when the env_0
dummy action is performed, env_0
does not provide any information to the rollout, and env_1
still collects rollouts. the env_0
and env_1
are updated together through sim.step()
.
Is there already completed work for this? I'm asking because it's challenging to code from scratch due to early termination and multi-environment.
Thank you.
Looks like this missing the exp-kernel, assuming it should be something along the lines of:
def reaching_object_position_exp(self, env: LiftEnv, sigma: float):
"""Penalize end-effector tracking position error using exp-kernel."""
error = torch.sum(torch.square(env.robot.data.ee_state_w[:, 0:3] - env.object.data.root_pos_w), dim=1)
return torch.exp(-error / sigma)
After Isaac-Velocity-Anymal-C-v0 is trained 1000 steps baesd on rsl-rl, it cannot walk, but float in the air.
on bea049b commit,
./orbit.sh -p source/standalone/workflows/rsl_rl/train.py --task Isaac-Velocity-Anymal-C-v0 --headless
./orbit.sh -p source/standalone/workflows/rsl_rl/play.py --task Isaac-Velocity-Anymal-C-v0 --num_envs 1 --checkpoint /PATH/TO/model_1000.pth
Describe the characteristic of your environment:
I have in general been able to access isaac-sim simulations that are run in headless mode within a docker container through the Omniverse Streaming Client. Should I expect the same when running the Orbit demo scripts in headless mode? Currently the Omniverse Streaming Client doesn't connect for me on some of the demos.
If you are submitting a bug report, please fill in the following details and use the tag [bug].
When running the agent training step of the Robomimic example following the documentation. I get an error saying that object 'object_positions' doesn't exist
Just follow the exact steps in Running existing scripts/Imitation Learning
in the document page.
# step a: collect data with keyboard
./orbit.sh -p source/standalone/workflows/robomimic/collect_demonstrations.py --task Isaac-Lift-Franka-v0 --num_envs 1 --num_demos 10 --device keyboard
# split data
./orbit.sh -p source/standalone//workflows/robomimic/tools/split_train_val.py logs/robomimic/Isaac-Lift-Franka-v0/hdf_dataset.hdf5 --ratio 0.2
./orbit.sh -p source/standalone/workflows/robomimic/train.py --task Isaac-Lift-Franka-v0 --algo bc --dataset logs/robomimic/Isaac-Lift-Franka-v0/hdf_dataset.hdf5
Describe the characteristic of your environment:
I think the problem is that in lift_bc.json
there is object_positions
, whereas in lift_cfg.py
it is commented out. There is object_relative_tool_positions
instead.
Was following the installation instructions of orbit from
https://isaac-orbit.github.io/orbit/source/setup/installation.html
I ran ./orbit.sh -e
and I got
kaito@pop-os:~/Documents/Orbit$ ./orbit.sh -e
[INFO] Installing extra requirements such as learning frameworks...
/home/kaito/.local/share/ov/pkg/isaac_sim-2022.2.1/kit/python/bin/python3: can't open file '/home/kaito/Documents/Orbit/source/extensions/omni.isaac.orbit_envs[all]': [Errno 2] No such file or directory
There was an error running python
I am trying to attach a camera to my robot's link using the script from play_camera.py.
I initialized my robot as:
robot = my_world.scene.add(PR2(prim_path="/World/PR2", name="pr2"))
and spawned the camera as:
camera.spawn("/World/PR2/wide_stereo_r_stereo_camera_frame")
where the prim_path is the link where I would like to attach the camera.
Instead of attaching to the provided prim_path, it is spawned at a distance from the given link ( at a different spot and height in the air than that of the link )
Kindly let me know if there is a way to attach the camera to a link. Thanks
Hello, I've found two bugs in Orbit. I'm not sure that description is certainty, but I hope this Bug Report can help you fix these bugs in the next version.
def _process_info_cfg(self) -> None:
"""Post processing of configuration parameters."""
# process parent config
super()._process_info_cfg()
# resolve regex expressions for indices
# -- end-effector body
self.ee_body_index = None
for body_index, body_name in enumerate(self.body_names):
if re.fullmatch(self.cfg.ee_info.body_name, body_name):
self.ee_body_index = body_index - 1
Traceback (most recent call last):
File "source/standalone/workflows/skrl/train.py", line 167, in <module>
main()
File "source/standalone/workflows/skrl/train.py", line 107, in main
**convert_skrl_cfg(experiment_cfg["models"]["policy"]),
TypeError: gaussian_model() argument after ** must be a mapping, not NoneType
I fix it by adding "return d" after "source/standalone/workflows/skrl/config.py", line 90.
def update_dict(d):
for key, value in d.items():
if isinstance(value, dict):
update_dict(value)
else:
if key in _direct_eval:
d[key] = eval(value)
elif key.endswith("_kwargs"):
d[key] = value if value is not None else {}
elif key in ["rewards_shaper_scale"]:
d["rewards_shaper"] = reward_shaper_function(value)
return d
Describe the characteristic of your environment:
I have defined a mobile manipulator and import a YCB object in issac orbit.
here is the code of importing YCB object
Obj_cfg = RigidObjectCfg()
Obj_cfg.meta_info.usd_path = f"{ISAAC_NUCLEUS_DIR}/Props/YCB/Axis_Aligned_Physics/006_mustard_bottle.usd"
rigid_object = RigidObject(Obj_cfg)
obj_quat = convert_quat(tf.Rotation.from_euler("XYZ", (-90, 90, 0), degrees=True).as_quat(), to="wxyz")
translation =(np.random.uniform(-0.1,0.1), np.random.uniform(-0.3,0), 0.8)
rigid_object.spawn("/World/Objects/mustard_bottle",translation=translation, orientation=obj_quat)
and when I try to control the mobile manipulator to grasp the YCB object, there is no collision between the gripper and the object.
But there is collision between the robotic arm and the object. I have check the URDF file, but there is no difference between the arm and the gripper.
the following video shows the grasp process.
https://user-images.githubusercontent.com/41771334/231722830-79a4a5e2-c3ec-407b-b2a0-b8c5db67d322.mp4
We’re trying to implement a simple gripper that works not by grasp friction but by tethering the object to the gripper when the gripper is close enough. I have used class LeggedMobileManipulator in issac orbit. I found there is a class named SurfaceGripper (https://docs.omniverse.nvidia.com/py/isaacsim/source/extensions/omni.isaac.surface_gripper/docs/index.html) can solve this problem. But how can I add SurfaceGripper to LeggedMobileManipulator in Issac orbit. And are there other solutions that can achieve this goal?
Is it possible to use Jax as learning framework without any performance/speed of simulation loss ?
This thread is for discussing renaming all "DOF" attributes to "Joints" in the core framework.
The framework uses "DOF" (degrees-of-freedom) as a convention taken from IsaacGym. For fixed arms, the joints and DOFs are equivalent. However, for floating-based systems, DOF also includes the un-actuated base (i.e. generalized coordinates).
It can be annoying to call it "DOF" since nomenclature-wise it is incorrect for robotics.
Rename the attributes that have "DOF" or "dof" to call them "Joint" or "joint" respectively.
To keep it backward compatible, we can add a check using __getattr__
call that renames the called variable name to joint and add a warning. After v1.0, we can remove this function.
This is a breaking change but useful in the long run.
simulation_app = SimulationApp(config)
import os
config = {"headless": args_cli.headless}
# load cheaper kit config in headless
if args_cli.headless:
app_experience = f"{os.environ['EXP_PATH']}/omni.isaac.sim.python.gym.headless.kit"
else:
app_experience = f"{os.environ['EXP_PATH']}/omni.isaac.sim.python.kit"
# launch the simulator
simulation_app = SimulationApp(config, experience=app_experience)
Describe the characteristic of your environment:
Add any other context about the problem here.
Hi, thank you for the great project.
The paper says that the demo in Figure 6 was collected by "Demonstration of the designed tasks using hand-crafted state machines and task-space controllers." but this agent was not found in current code base. Also, there was no specific mention of a hand crafted agent in the develeopment roadmap.
Is this something you plan to release in the future?
Thanks!
the mimic_multiplier is multiplied twice in gripper_group.py
since I applied gripper mimic in configure file for sb3,
"robotiq_hand": GripperActuatorGroupCfg(
dof_names=["bot_joint_0_p", "top_joint_0_p", "bot_joint_1_p", "top_joint_1_p", "bot_joint_2_p", "top_joint_2_p"],
model_cfg=ImplicitActuatorCfg(velocity_limit=20, torque_limit=1000),
control_cfg=ActuatorControlCfg(command_types=["p_abs"], stiffness={".*": 400}, damping={".*": 20}),
mimic_multiplier={"bot_joint_0_p":1, "top_joint_0_p":1, "bot_joint_1_p":1, "top_joint_1_p":-1, "bot_joint_2_p":0, "top_joint_2_p":1},
# speed=1,
open_dof_pos=0,
close_dof_pos=0.785398,
mimic multiplier didn't work since "top_joint_1_p":-1
is multiplied twice, in torch elementwise matmul manner. commenting second, it worked.
Thanks for releasing this, really exciting work! Just wanted to share some issues I encountered when trying it out.
When I tried to install the repo I got an error complaining 'pip' not found at this line: https://github.com/NVIDIA-Omniverse/Orbit/blob/main/orbit.sh#L114. After changing it to use the isaac_sim python binary it did work.
Also, the franka lift env doesn't seem to train (got reward ~13 with rl_games and ~18 with rsl_rl, in both case arm just moved away ignoring the object).
Depending on the configuration of the robot, it seems like translation keyboard commands affect the orientation of the robot. In my attached video, I only press the "A" and "D" keys in the collect demonstration script, but the eef orientation also moves.
I have attached a video, but the issue can be reproduced by moving the arm around using the keyboard controller, using the collect human demos script.
Describe the characteristic of your environment:
Should line 102 in mobile_manipulator.py be
self._data.ee_jacobian[:] = jacobians[:, self.ee_body_index - 1, :, self.base_num_dof : self.base_num_dof+self.arm_num_dof]
?
The original code is
self._data.ee_jacobian[:] = jacobians[:, self.ee_body_index - 1, :, self.base_num_dof : self.arm_num_dof]
but self.base_num_dof : self.arm_num_dof
seems wrong
In omni/isaac/orbit/robots/config/unitree.py, the Kp,Kd values of Unitree A1 robot are 25 and 0.5.
So what are these values in the reality.
Specifically, for various robot behavior (walking,turning, gallop,...), are these parameters uniform (in simulation or reality)?
If not, what are the approximate values of each parameter.
Thank you!
"we use
ZMQ
to send joint commands fromORBIT
to a computer running the real-time kernel forFranka Emika
robot. To abide by the real-time safety constraints, we use a quintic interpolator to upsample the60 Hz
joint commands from the simulator to1000 Hz
for execution on the robot" (orbit
paper)
This sound very efficient to overcome the impossibility to use nvidia-drivers on the PC that talks to the robot (due to the PREEMPT_RT
kernel)!
I would have two series of questions (unfortunately I could not find any additional details in the documentation / paper):
regarding the zmq
interface:
regarding the computer running the RT kernel for the Franka Emika robot:
FCI
via libfranka
, isn't it?joint position
(called qc
on this FCI
figure) passed to a joint position motion generator
, which is then processed by one of the FCI
internal controllers, e.g. the joint impedance controller
, as explained here? (green path on the attached image)torques
(called to-d
on this FCI
figure) directly processed by an external controller
? (purple path on the attached image)fig.9
the target joint position
issued at 1kHz
is converted to torques
by a PD
controller.
PD
controller the built-in one of FCI
?PD
controller also used in simulation during training (I do not think so)? And if yes are the PD
parameters identical in simulation and on the real robot?Hello, I like this framework very much, and I am also looking forward to the February function mentioned in the roadmap. May I know when the next version will be updated?
Gymnasium is the maintained version of OpenAI Gym that is now handled by Farama Foundation. It is currently the definition used in libraries such as rllib and tianshou, while others like stable-baselines3 will also update soon.
More recently, there has been an interesting proposal on RL environment definition in the framework torchrl
which would fit well for our targetted applications.
This thread aims to serve as a voting/staging on what would be the best environment definition to use.
The current Gym definition (from 0.21.0) is outdated and depends on older libraries such as importlib-meta==4.1
package. This creates conflicts with new updated packages and it would be best to switch to the new Gym definition.
Related issues: OIGE #28, Sb3 #1327
Once SB3 also upgrades to Gymnasium, then it would be best to update everything to use the latest definitions.
More information on Gymnasium:
The environment definition EnvBase
which relies heavily on using tensordict
.
The advantages of their base class are:
tensordict
makes it easier to work with given the scaling possible with Isaac Sim and should be more efficient/flexible than dict
of torch.Tensor
Related Issues: torchrl #883
Since there are wrappers for other RL frameworks (RL-Games and RSL-RL), this probably won't cause any breaking changes on that side. We will just need to adapt them based on the chosen IsaacEnv
definition.
Originally posted by Aqua12138 March 2, 2023
Why is the cube not responding physically when I run lift env, I tried controlling the franka to clip it but there doesn't seem to be any collision response
Screencast from 10.03.2023 14:19:13.webm
There is no interaction with the cube, which should be picked up by the robot after training.
Hi, I want to create some custom environment.
for example, I want to upgrade the environment franka-lift -v0
, changing rigid object and manipulator such as UR10.
what is the best way? can you recommend some docs for the custom environment?
thanks!
Currently in certain demo scripts there are uses of 'headless' to indicate whether or not rendering should be enabled for the SimulationContext.
I think this could be confusing, because 'headless' already has a potentially conflicting meaning within SimulationApp. A user may input 'headless' to one of these scripts and still want rendering. For instance, one might want to run headlessly on a remote server, and to run the Omniverse Streaming Client for a local GUI.
I think this could be resolved by simply adding another flag 'render' (or alternatively 'no-render') which will take the place of 'headless' in these evaluations and allow it to retain its pure meaning in the SimulationApp API.
We could keep the existing coercion of headless, but it wouldn't work for people hoping to stream.
Demos 1 and 2 in API Demos abort the execution due to unhandled error
Run demos 1 and 2 in API Demos
Spawn different quadrupeds, visualize feet markers, and make robots stand using position commands:
./orbit.sh -p source/standalone/demo/play_quadrupeds.py
Spawn multiple Franka arms and apply random position commands:
./orbit.sh -p source/standalone/demo/play_arms.py --robot franka_panda
Describe the characteristic of your environment:
Hi great work!
What is a possible way to clear stage and spawn a random rigid object at each episode without closing the simulator? without running into memory leaks
Thanks,
When trying to set the camera pose using the Camera.set_world_pose_ros()
, the obtained pose in the data
is different from the one being set. On digging deeper, the main issue seems to be with using the Replicator API to create the camera in the Camera.spawn()
method.
When replicator creates a camera using the rep.create.camera()
, it creates an Xform prim and a Camera prim under it. The camera prim is rotated by (90, 0, -90) relative to the parent, to have the up-axis of the camera Z instead of Y. Now, when the camera pose is computed, the transformation of the Camera prim is used. However, when setting the pose, the pose is set to the Xform prim (also called the camera rig). So in a way, the camera prim pose ends up at the input pose with an orientation offset of (90, 0, -90).
Modify the play_camera.py
script with the following addition from L-140 onwards:
# set pose
# camera.set_world_pose_from_view(eye=[5.0, 5.0, 5.0], target=[0.0, 0.0, 0.0])
position = [5.0, 5.0, 5.0]
orientation = [-0.1759199, 0.33985114, 0.82047324, -0.4247082]
camera.set_world_pose_ros(position, orientation)
This gives the output:
Camera @ '/World/CameraSensor/Camera_Xform/Camera':
data types : ['rgb', 'distance_to_image_plane', 'normals', 'motion_vectors', 'trigger_outputs']
tick rate (s): 0
timestamp (s): 0.0
frame : 33
shape : (480, 640)
position : [5. 5. 5.]
orientation : [-0.27984815 0.88047624 -0.11591691 0.36470519]
Expected output orientation to match the input orientation.
Describe the characteristic of your environment:
The camera methods currently print the warnings that they may not work.
2023-01-26 15:55:07 [14,045ms] [Warning] [omni.isaac.orbit.sensors.camera.camera] The function `set_world_pose_ros` is currently not implemented correctly.
Hello Orbit Team,
I went through the docs and were able to run the environments that I was able to find namely Cartpole, Ant, Humanoid, Velocity, Reach and Lift and I was wondering where the other examples showcased in your paper and docs (here: https://isaac-orbit.github.io/orbit/source/features/environments.html) are and how to access them.
The base examples only show the basic observations like the position and orientation of the actor, object, and target position, however a more complex example that utilizes more complex observations would be appreciated.
For example Cloth Folding or Mobile Reach would be interesting, as these require observations which would require the NN to learn spacial awareness.
I would greatly appreciate it if you publish some of the more complex examples as Isaac still has a quite meager community support for how exactly to utilize the engine to its potential.
Kind regards
Hi! I want to create a visual mesh like this.
Unfortunately, In-hand Manipulation
env. is not opened yet, I would like to edit the existing code. With these options,
I expected the same behavior in the picture.
However, in my codes, the visual_objects
still have a collider, while gravity is disabled.
Can you give me some advice?
Thanks!
Reproduce Steps======================================================
I just added some lines to the .cfg
file.
lift_cfg.py
@configclass
class VisualObjectCfg(RigidObjectCfg):
meta_info = RigidObjectCfg.MetaInfoCfg(
usd_path=f"{ISAAC_NUCLEUS_DIR}/Props/Blocks/DexCube/dex_cube_instanceable.usd", scale=(1, 1, 1))
init_state = RigidObjectCfg.InitialStateCfg(
pos=(0.4, 0.0, 0.075), rot=(1.0, 0.0, 0.0, 0.0), lin_vel=(0.0, 0.0, 0.0), ang_vel=(0.0, 0.0, 0.0))
collision_props = RigidObjectCfg.CollisionPropertiesCfg(collision_enabled=False)
rigid_props = RigidObjectCfg.RigidBodyPropertiesCfg(disable_gravity=True)
...
@configclass
class LiftEnvCfg(IsaacEnvCfg):
...
# -- object
object: ManipulationObjectCfg = ManipulationObjectCfg()
# -- VISUAL OBJECT
visual_object: VisualObjectCfg = VisualObjectCfg()
...
After that, I added two lines one my env
file.
lift_env.py
class LiftEnv(IsaacEnv):
def __init__(self, cfg: LiftEnvCfg = None, headless: bool = False):
...
self.object = RigidObject(cfg=self.cfg.object)
self.visual_object = RigidObject(cfg=self.cfg.visual_object) # Here
...
def _design_scene(self) -> List[str]:
...
self.object.spawn(self.template_env_ns + "/Object")
self.visual_object.spawn(self.template_env_ns + "/VisualObject") # Here
...
I have a usd file for my dualarm mobile robot. I want to run some RL with it. Is there any examples or descriptions on how I can get started?
Orbit has already made thoughtful design decisions that support really nice UX/UI paradigms for configuration management. Showing a few use-cases of how this might be done would be beneficial for the community as it can be difficult to decide "how to start". We would like to have one or two best-practices examples that fully leverage configuration via dataclasses
. Preferably, one which makes no commitment to configuration framework, and one which demonstrates a a few tweaks that can allow Hydra to be used at its most powerful capacity.
We'll provide an example that shows how to set up an environment(robot / scene), RL algorithm, and training script and configure all of this via dataclasses.
We'll then show how to extend this configuration paradigm via some key features of Hydra.
We can also look into what benefits other configuration frameworks have (one such being tyro), as long as they use dataclasses
as the principle data structure.
#57 and #67 is a related issue/PR pair, and it is a good start! We will build on this and provide more use-cases that leverage Hydra and are clean / easy to understand. See some of the previous discussion in #67 .
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.