umass-foundation-model / co-llm-agents Goto Github PK

Hello,
I have a question about the "Logging error". After I launch the scripts/test_LMs.sh, the system shows this error but the code still keep runing. Is it normal? (I use mac os).

Python tdw module version 1.11.18
{'scene': '5a', 'layout': '0_0', 'seed': 2824, 'task': 'food'}
New Setting: True
dataset/dataset_test/5a_0_0_metadata.json saved
/Users/shengsu/Downloads/miniconda/envs/embodied/lib/python3.9/site-packages/gym/utils/passive_env_checker.py:199: UserWarning: WARN: The result returned by env.reset() should be (obs, info) by default, , where obs is a observation and info is a dictionary containing additional information.
logger.warn(
--- Logging error ---
Traceback (most recent call last):
File "/Users/shengsu/Downloads/miniconda/envs/embodied/lib/python3.9/logging/init.py", line 1083, in emit
msg = self.format(record)
File "/Users/shengsu/Downloads/miniconda/envs/embodied/lib/python3.9/logging/init.py", line 927, in format
return fmt.format(record)
File "/Users/shengsu/Downloads/miniconda/envs/embodied/lib/python3.9/logging/init.py", line 663, in format
record.message = record.getMessage()
File "/Users/shengsu/Downloads/miniconda/envs/embodied/lib/python3.9/logging/init.py", line 367, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
File "/Users/shengsu/Downloads/embodied/Co-LLM-Agents/envs/tdw_mat/tdw-gym/challenge.py", line 177, in
main()
File "/Users/shengsu/Downloads/embodied/Co-LLM-Agents/envs/tdw_mat/tdw-gym/challenge.py", line 172, in main
challenge.submit(agents, logger, args.eval_episodes)
File "/Users/shengsu/Downloads/embodied/Co-LLM-Agents/envs/tdw_mat/tdw-gym/challenge.py", line 64, in submit
agent.reset(obs = state[str(id)], goal_objects = info['goal_description'], output_dir = os.path.join(self.output_dir, str(i)), env_api = env_api, rooms_name=info['rooms_name'])
File "/Users/shengsu/Downloads/embodied/Co-LLM-Agents/envs/tdw_mat/tdw-gym/lm_agent.py", line 395, in reset
self.logger.info(self.goal_objects, goal_count)
Message: {'orange': 2, 'apple': 3, 'banana': 1, 'bread': 3, 'burger': 1}
Arguments: (10,)
--- Logging error ---
Traceback (most recent call last):
File "/Users/shengsu/Downloads/miniconda/envs/embodied/lib/python3.9/logging/init.py", line 1083, in emit
msg = self.format(record)
File "/Users/shengsu/Downloads/miniconda/envs/embodied/lib/python3.9/logging/init.py", line 927, in format
return fmt.format(record)
File "/Users/shengsu/Downloads/miniconda/envs/embodied/lib/python3.9/logging/init.py", line 663, in format
record.message = record.getMessage()
File "/Users/shengsu/Downloads/miniconda/envs/embodied/lib/python3.9/logging/init.py", line 367, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
File "/Users/shengsu/Downloads/embodied/Co-LLM-Agents/envs/tdw_mat/tdw-gym/challenge.py", line 177, in
main()
File "/Users/shengsu/Downloads/embodied/Co-LLM-Agents/envs/tdw_mat/tdw-gym/challenge.py", line 172, in main
challenge.submit(agents, logger, args.eval_episodes)
File "/Users/shengsu/Downloads/embodied/Co-LLM-Agents/envs/tdw_mat/tdw-gym/challenge.py", line 64, in submit
agent.reset(obs = state[str(id)], goal_objects = info['goal_description'], output_dir = os.path.join(self.output_dir, str(i)), env_api = env_api, rooms_name=info['rooms_name'])
File "/Users/shengsu/Downloads/embodied/Co-LLM-Agents/envs/tdw_mat/tdw-gym/lm_agent.py", line 395, in reset
self.logger.info(self.goal_objects, goal_count)
Message: {'orange': 2, 'apple': 3, 'banana': 1, 'bread': 3, 'burger': 1}
Arguments: (10,)
dict_keys(['visible_objects', 'rgb', 'seg_mask', 'depth', 'camera_matrix', 'agent', 'held_objects', 'oppo_held_objects', 'FOV', 'status', 'messages', 'valid', 'current_frames'])
[' (1000)', ' (2000)', ' (3000)', ' (4000)', ' (5000)', ' (6000)']
--- Logging error ---
Traceback (most recent call last):
File "/Users/shengsu/Downloads/miniconda/envs/embodied/lib/python3.9/logging/init.py", line 1083, in emit
msg = self.format(record)
File "/Users/shengsu/Downloads/miniconda/envs/embodied/lib/python3.9/logging/init.py", line 927, in format
return fmt.format(record)
File "/Users/shengsu/Downloads/miniconda/envs/embodied/lib/python3.9/logging/init.py", line 663, in format
record.message = record.getMessage()
File "/Users/shengsu/Downloads/miniconda/envs/embodied/lib/python3.9/logging/init.py", line 367, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
File "/Users/shengsu/Downloads/embodied/Co-LLM-Agents/envs/tdw_mat/tdw-gym/challenge.py", line 177, in
main()
File "/Users/shengsu/Downloads/embodied/Co-LLM-Agents/envs/tdw_mat/tdw-gym/challenge.py", line 172, in main
challenge.submit(agents, logger, args.eval_episodes)
File "/Users/shengsu/Downloads/embodied/Co-LLM-Agents/envs/tdw_mat/tdw-gym/challenge.py", line 64, in submit
agent.reset(obs = state[str(id)], goal_objects = info['goal_description'], output_dir = os.path.join(self.output_dir, str(i)), env_api = env_api, rooms_name=info['rooms_name'])
File "/Users/shengsu/Downloads/embodied/Co-LLM-Agents/envs/tdw_mat/tdw-gym/lm_agent.py", line 395, in reset
self.logger.info(self.goal_objects, goal_count)
Message: {'orange': 2, 'apple': 3, 'banana': 1, 'bread': 3, 'burger': 1}
Arguments: (10,)
--- Logging error ---
Traceback (most recent call last):
File "/Users/shengsu/Downloads/miniconda/envs/embodied/lib/python3.9/logging/init.py", line 1083, in emit
msg = self.format(record)
File "/Users/shengsu/Downloads/miniconda/envs/embodied/lib/python3.9/logging/init.py", line 927, in format
return fmt.format(record)
File "/Users/shengsu/Downloads/miniconda/envs/embodied/lib/python3.9/logging/init.py", line 663, in format
record.message = record.getMessage()
File "/Users/shengsu/Downloads/miniconda/envs/embodied/lib/python3.9/logging/init.py", line 367, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
File "/Users/shengsu/Downloads/embodied/Co-LLM-Agents/envs/tdw_mat/tdw-gym/challenge.py", line 177, in
main()
File "/Users/shengsu/Downloads/embodied/Co-LLM-Agents/envs/tdw_mat/tdw-gym/challenge.py", line 172, in main
challenge.submit(agents, logger, args.eval_episodes)
File "/Users/shengsu/Downloads/embodied/Co-LLM-Agents/envs/tdw_mat/tdw-gym/challenge.py", line 64, in submit
agent.reset(obs = state[str(id)], goal_objects = info['goal_description'], output_dir = os.path.join(self.output_dir, str(i)), env_api = env_api, rooms_name=info['rooms_name'])
File "/Users/shengsu/Downloads/embodied/Co-LLM-Agents/envs/tdw_mat/tdw-gym/lm_agent.py", line 395, in reset
self.logger.info(self.goal_objects, goal_count)
Message: {'orange': 2, 'apple': 3, 'banana': 1, 'bread': 3, 'burger': 1}
Arguments: (10,)
dict_keys(['visible_objects', 'rgb', 'seg_mask', 'depth', 'camera_matrix', 'agent', 'held_objects', 'oppo_held_objects', 'FOV', 'status', 'messages', 'valid', 'current_frames'])
[' (1000)', ' (2000)', ' (3000)', ' (4000)', ' (5000)', ' (6000)']
/Users/shengsu/Downloads/embodied/Co-LLM-Agents/envs/tdw_mat/tdw-gym/tdw_gym.py:710: RuntimeWarning: divide by zero encountered in divide
depth_img = Image.fromarray(100 / depth).convert('RGB')
2023-07-19 11:55:39,196 - simple_example - INFO - Environment Reset. Took 294.9135160446167 secs

could you please tell me the hardware configuration and training time?

Hi,

As per the README, when I ran ./scripts/test_LMs-gpt-4.sh in the tdw_mat I see this unity window

which is very small for me to findout what's in it. how to have a full window of this graphics?
this was ran on the Ubuntu with Nvidia GPU.

Hi， could you please tell me the inference time for tdw? On my desktop, it takes around 50 hours, is it normal?

https://github.com/UMass-Foundation-Model/Co-LLM-Agents/blob/2c3a5f02ed4f2b45751605511121b26291c777d5/envs/cwah/gen_data/init_goal_setter/init_goal_base.py#L468C28-L468C28

these two line:Unresolved reference 'apartment'

I got this error when I run ./scripts/symbolic_obs_llm_llm.sh and and python vh_demo.py --deployment local --extra_agent LLM_comm --task_group 0 --showmodal in envs/cwah:

(cwah) liyun@DESKTOP-0NAM80S:~/Co-LLM-Agents/envs/cwah$ ./scripts/symbolic_obs_llm_llm.sh
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
Init Env
['/home/liyun/Co-LLM-Agents/envs/cwah/../executable/linux_exec.v2.3.0.x86_64', '-batchmode', '-nographics', '-http-port=6314', '-logFile /home/liyun/Co-LLM-Agents/envs/cwah/Player_6314.log']
Getting connection...
error: XDG_RUNTIME_DIR not set in the environment.
episode: 0
task_goal: {0: {'on_pudding_268': 1, 'on_juice_268': 1, 'on_apple_268': 1, 'on_cupcake_268': 2}, 1: {'on_pudding_268': 1, 'on_juice_268': 1, 'on_apple_268': 1, 'on_cupcake_268': 2}}
total_goal: {'on_pudding_268': 2, 'on_juice_268': 2, 'on_apple_268': 2, 'on_cupcake_268': 2}
Resetting... Envid: 3. Taskid: 0. Index: 0
Traceback (most recent call last):
File "/home/liyun/anaconda3/envs/cwah/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/home/liyun/anaconda3/envs/cwah/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request
httplib_response = conn.getresponse()
File "/home/liyun/anaconda3/envs/cwah/lib/python3.8/http/client.py", line 1348, in getresponse
response.begin()
File "/home/liyun/anaconda3/envs/cwah/lib/python3.8/http/client.py", line 316, in begin
version, status, reason = self._read_status()
File "/home/liyun/anaconda3/envs/cwah/lib/python3.8/http/client.py", line 277, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/home/liyun/anaconda3/envs/cwah/lib/python3.8/socket.py", line 669, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/liyun/anaconda3/envs/cwah/lib/python3.8/site-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen(
File "/home/liyun/anaconda3/envs/cwah/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/home/liyun/anaconda3/envs/cwah/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/liyun/anaconda3/envs/cwah/lib/python3.8/site-packages/urllib3/packages/six.py", line 770, in reraise
raise value
File "/home/liyun/anaconda3/envs/cwah/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/liyun/anaconda3/envs/cwah/lib/python3.8/site-packages/urllib3/connectionpool.py", line 451, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/home/liyun/anaconda3/envs/cwah/lib/python3.8/site-packages/urllib3/connectionpool.py", line 340, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='127.0.0.1', port=6314): Read timed out. (read timeout=30)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/liyun/Co-LLM-Agents/envs/cwah/envs/../../virtualhome/simulation/unity_simulator/comm_unity.py", line 92, in post_command
resp = requests.post(self._address, json=request_dict, timeout=self.timeout_wait)
File "/home/liyun/anaconda3/envs/cwah/lib/python3.8/site-packages/requests/api.py", line 115, in post
return request("post", url, data=data, json=json, **kwargs)
File "/home/liyun/anaconda3/envs/cwah/lib/python3.8/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/home/liyun/anaconda3/envs/cwah/lib/python3.8/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/home/liyun/anaconda3/envs/cwah/lib/python3.8/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/home/liyun/anaconda3/envs/cwah/lib/python3.8/site-packages/requests/adapters.py", line 578, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='127.0.0.1', port=6314): Read timed out. (read timeout=30)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "testing_agents/test_symbolic_LLMs.py", line 116, in
arena.reset(episode_id)
File "/home/liyun/Co-LLM-Agents/envs/cwah/testing_agents/../algos/arena_mp2.py", line 53, in reset
ob = self.env.reset(task_id=task_id)
File "/home/liyun/Co-LLM-Agents/envs/cwah/testing_agents/../envs/unity_environment.py", line 325, in reset
s, g = self.comm.environment_graph()
File "/home/liyun/Co-LLM-Agents/envs/cwah/envs/../../virtualhome/simulation/unity_simulator/comm_unity.py", line 298, in environment_graph
response = self.post_command({'id': str(time.time()), 'action': 'environment_graph'})
File "/home/liyun/Co-LLM-Agents/envs/cwah/envs/../../virtualhome/simulation/unity_simulator/comm_unity.py", line 98, in post_command
raise UnityCommunicationException(str(e))
unity_simulator.comm_unity.UnityCommunicationException: HTTPConnectionPool(host='127.0.0.1', port=6314): Read timed out. (read timeout=30)
CLOSING PROC
CLOSING PROC
CLOSING PROC

Could you please hlep me to hanle with this ASAP? it seems the v2.3.0 virtualhome_unity problem refer to this xavierpuigf/virtualhome#42?

https://github.com/UMass-Foundation-Model/Co-LLM-Agents/blob/2c3a5f02ed4f2b45751605511121b26291c777d5/envs/cwah/testing_agents/data_collection_agent.py#L97C74-L97C74

Unresolved reference 'Random_agent'

I tried nearly every possible method to fix the error "XDG_RUNTIME_DIR not set in the environment." and "unity_simulator.comm_unity.UnityCommunicationException: HTTPConnectionPool(host='127.0.0.1', port=6310): Read timed out. (read timeout=30)" but failed.

https://github.com/UMass-Foundation-Model/Co-LLM-Agents/blob/2c3a5f02ed4f2b45751605511121b26291c777d5/envs/cwah/envs/envs.py#L33C1-L33C22

Unresolved reference 'utils_rl_agent'

I always encounter socket.timeout: timed out when I execute bash scripts/symbolic_obs_llm_llm.sh. Do you any experience to solve this problem?

After executing ./scripts/test_LM.sh, the following TypeError occurs when the agent is asked to place an target_object into a container.

Traceback (most recent call last):
File "/home/eclab/Desktop/compete-llm/compete-LLM-Agents/tdw_mat/tdw-gym/challenge.py", line 185, in
main()
File "/home/eclab/Desktop/compete-llm/compete-LLM-Agents/tdw_mat/tdw-gym/challenge.py", line 179, in main
challenge.submit(agents, logger, args.eval_episodes)
File "/home/eclab/Desktop/compete-llm/compete-LLM-Agents/tdw_mat/tdw-gym/challenge.py", line 82, in submit
state, reward, done, info = self.env.step(actions)
File "/home/eclab/anaconda3/envs/tdw_mat/lib/python3.9/site-packages/gym/wrappers/order_enforcing.py", line 37, in step
return self.env.step(action)
File "/home/eclab/anaconda3/envs/tdw_mat/lib/python3.9/site-packages/gym/wrappers/env_checker.py", line 39, in step
return self.env.step(action)
File "/home/eclab/Desktop/compete-llm/compete-LLM-Agents/tdw_mat/tdw-gym/tdw_gym.py", line 697, in step
data = self.controller.communicate([])
File "/home/eclab/Desktop/compete-llm/compete-LLM-Agents/tdw_mat/transport_challenge_multi_agent/transport_challenge.py", line 202, in communicate
return super().communicate(commands)
File "/home/eclab/Desktop/compete-llm/compete-LLM-Agents/tdw_mat/transport_challenge_multi_agent/asset_cached_controller.py", line 46, in communicate
m.on_send(resp)
File "/home/eclab/anaconda3/envs/tdw_mat/lib/python3.9/site-packages/tdw/add_ons/replicant.py", line 170, in on_send
initialization_commands = self.action.get_initialization_commands(resp=resp,
File "/home/eclab/Desktop/compete-llm/compete-LLM-Agents/tdw_mat/transport_challenge_multi_agent/put_in.py", line 96, in get_initialization_commands
return self._move_object_arm_away(resp=resp, static=static, dynamic=dynamic)
File "/home/eclab/Desktop/compete-llm/compete-LLM-Agents/tdw_mat/transport_challenge_multi_agent/put_in.py", line 166, in _move_object_arm_away
self._sub_action = ReachForTransportChallenge(target={k: v for k, v in target.items()},
File "/home/eclab/Desktop/compete-llm/compete-LLM-Agents/tdw_mat/transport_challenge_multi_agent/reach_for_transport_challenge.py", line 25, in init
super().init(target=target,
TypeError: init() got an unexpected keyword argument 'target'

these two 'goal' not found

Sometimes the agents can go through the wall, and stay outside of the rooms, this is not correct for the current code. which parameters should i set to avoid such problem?

When I ran test_H.sh, I first saw the layout of the TDW window and room objects, and then prompted the following error: AttributeError: 'NoneType' object has no attribute 'name', and then the program ended.

How I can solve it? Thank you.

There are some errors caused by out of bound index. I cannot figure it out why it is, are there some important hyperparameters that the author should mention to prevent such errors?

When I am running demo_scene.py, I get

You are using TDW 1.11.18.0 but version 1.12.0.0 is available.
To upgrade to the last version of 1.11:
pip3 install tdw==1.11.23.5
Consider upgrading to the latest version of TDW (1.12.0):
pip3 install tdw -U
Couldn't find build at /home/chris/tdw_build/TDW/TDW.x86_64
Downloading now...

and then it just paused there for about several minutes, then I quited. I'm wondering if that is a proper behavior?

I've been closely following CoELA project especially CoLLAMA . I'm really impressed by the work. I'm wondering, are there any plans to share the code and datasets about CoLLAMA? I'm super excited about the possibility of diving into it and maybe even contributing

Hello, I would like to inquire if your program is running on a local Linux computer or a remote Linux server. If I use a remote Linux server, may I not be able to see the visual interface. I am not sure how to set up and run your program

Does it work in windows

Hi,

When I tried to run experiments in the folder /cwah/scripts(e.g., ./scripts/symbolic obs llm llm.sh), I got an error saying "couldn't launch the linux_exec.v2.3.0 environment".

And I organized the folders as required:

Is it because I'm using MacOS?

Thank you:)

File "/home/jie/newdisk/AIAgent/CoELA/tdw_mat/tdw-gym/lm_agent.py", line 768, in act
self.get_new_object_list()
File "/home/jie/newdisk/AIAgent/CoELA/tdw_mat/tdw-gym/lm_agent.py", line 251, in get_new_object_list
if self.object_map[i, j] == 0:
IndexError: index 132 is out of bounds for axis 1 with size 120