mandizhao / robot-collab Goto Github PK

Dear MandiZhao，when I try to run Task: Pack Grocery，often when I get past the first round of dialogue, the LLM keeps talking about the same behavior and can't plan the right action to execute.
the feedback likes as following：

INFO:root:Step: 13 LLM plan parsed, begin RRT planning
tograsp: {'Bob': ('banana', 'banana_top', 0), 'Alice': ('apple', 'apple_top', 0)} inhand: {'Bob': ('banana', 'banana_top', 'banana_joint'), 'Alice': ('apple', 'apple_top', 'apple_joint')} {'Bob': 'PLACE banana bin_front_left PATH [(-0.25, 0.40, 0.3), (0.0, 0.40, 0.3), (0.12, 0.40, 0.3), (0.25, 0.41, 0.3)]', 'Alice': 'PLACE apple bin_front_right PATH [(-0.73, 0.40, 0.3), (-0.48, 0.40, 0.3), (-0.23, 0.40, 0.3), (0.65, 0.41, 0.3)]'}
Given 4 waypoints, found 4 valid waypoints via IK
=====================================
init-sampling! 1
init-sampling! 2
init-sampling! 3
init-sampling! 4
failed to find a path, reason: RRT failed: ReasonTimeout_time221.55805015563965_iter9

Can you give me some advices？Thank you very much！

用gpt4尝试了grocery的任务，两个机械臂只能完成两个正确的step，后续的5个step一直存在collision，没法继续进行，我的gpt4余额都快花完了：（

python run_dialog.py --task pack -llm gpt-4
File "run_dialog.py", line 35
env: MujocoSimEnv,
^
SyntaxError: invalid syntax

Ubuntu20.04 System, I want to apply the command in my OS.

When I try to run python run_dialog.py --task rope , I get the following error:

ValueError: Error: material 'stove-metal' not found in geom 145
Object name = groove_front, id = 145, line = 55, column = -1

Hello, I have an issue when run python run_dialog.py --task pack -llm gpt-4

WARNING:root:PackGroceryTask requires split parsed plans, and no failed waypoints, no direct waypoints

Traceback (most recent call last):
File "C:\Anaconda\envs\roco\run_dialog.py", line 494, in
main(args)
File "C:\Anaconda\envs\roco\run_dialog.py", line 412, in main
env = env_cl(
^^^^^^^
File "C:\Anaconda\envs\roco\rocobench\envs\task_pack.py", line 91, in init
super(PackGroceryTask, self).init(
File "C:\Anaconda\envs\roco\rocobench\envs\base_env.py", line 216, in init
self.physics = dm_mujoco.Physics.from_xml_path(filepath)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Anaconda\Lib\site-packages\dm_control\mujoco\engine.py", line 455, in from_xml_path
return cls.from_model(model)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Anaconda\Lib\site-packages\dm_control\mujoco\engine.py", line 419, in from_model
return cls(data)
^^^^^^^^^
File "C:\Anaconda\Lib\site-packages\dm_control\mujoco\engine.py", line 123, in init
self._reload_from_data(data)
File "C:\Anaconda\Lib\site-packages\dm_control\mujoco\engine.py", line 400, in _reload_from_data
model=index.struct_indexer(self.model, 'mjmodel', axis_indexers),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Anaconda\Lib\site-packages\dm_control\mujoco\index.py", line 623, in struct_indexer
attr = getattr(struct, field_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'MjModel' object has no attribute 'eq_active'. Did you mean: 'eq_active0'?

What should I do to revise it? Many thanks.

The bin and soda_can are missing in the rendered video of the pack_grocery task, and the blue cup is missing in the rendered video of the open_cabinet task.

I noticed their mesh is commented out in the XML file https://github.com/MandiZhao/robot-collab/blob/main/rocobench/envs/task_pack.xml#L140, but after I uncommented them, their physics is weird: the soda_can will roll down the table without contact, and the mug can never be placed on the coaster correctly.

There seems something wrong with the XML file, could you kindly update it to the correct one you used in your experiments or guide me on how to fix this issue? Thanks!

assert llm_source in ["gpt-4", "gpt-3.5-turbo", "claude"], f"llm_source must be one of [gpt4, gpt-3.5-turbo, claude], got {llm_source}"
AssertionError: llm_source must be one of [gpt4, gpt-3.5-turbo, claude], got gpt4

I didn't get claude api key, but I get gpt4, do I need to get both of them?thanks.

Can you take a look at this?

Traceback (most recent call last):
  File "run_dialog.py", line 494, in <module>
    main(args)
  File "run_dialog.py", line 463, in main
    runner.run(args)
  File "run_dialog.py", line 380, in run
    self.one_run(run_id)
  File "run_dialog.py", line 223, in one_run
    self.display_plan(plan, save_name=f"vis_llm_plan_{i}", save_dir=step_dir)
  File "run_dialog.py", line 156, in display_plan
    raise NotImplementedError
NotImplementedError

I can't see the video,only the picture