trackmania-rl / tmrl Goto Github PK
View Code? Open in Web Editor NEWReinforcement Learning for real-time applications - host of the TrackMania Roborace League
Home Page: https://pypi.org/project/tmrl
License: MIT License
Reinforcement Learning for real-time applications - host of the TrackMania Roborace League
Home Page: https://pypi.org/project/tmrl
License: MIT License
I'm trying to run the pretrained AI. I have follow the installation guide and the get started guide however when I try to run it I get this issue. I can see the program resizing my window from quarter of my screen to a 8th. I have tried running the game in the lowest possible resolution with a fps cap on 30 and I'm running on a 2060 super.
Here is the command log, do you have any idea what might cause this?
Running on:
Windows 11
Python 3.9.12
(base) C:\Windows\system32>python -m tmrl --test
DEBUG:root: PATH_DATA:C:\Users\jakob\TmrlData
INFO:numexpr.utils:Note: NumExpr detected 16 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
INFO:numexpr.utils:NumExpr defaulting to 8 threads.
INFO:root:Namespace(server=False, trainer=False, worker=False, test=True, benchmark=False, record_reward=False, check_env=False, no_wandb=False, config={})
DEBUG:root:Loading model from C:\Users\jakob\TmrlData\weights\SAC_4_LIDAR_pretrained.pth
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): api.ipify.org:80
DEBUG:urllib3.connectionpool:http://api.ipify.org:80 "GET / HTTP/1.1" 200 14
INFO:root:09/14/22 11:09:44 local IP: 192.168.1.127
INFO:root:09/14/22 11:09:44 public IP: 213.66.199.113
INFO:root:09/14/22 11:09:44 server IP: 127.0.0.1
DEBUG:root: virtual joystick in use
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,C:\ProgramData\Anaconda3\lib\site-packages\rtgym\envs\real_time_env.py:357: UserWarning: Time-step timed out. Elapsed since last time-step: 1.5917513370513916
warnings.warn(f"Time-step timed out. Elapsed since last time-step: {now - self.__t_end}")
File "C:\ProgramData\Anaconda3\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\ProgramData\Anaconda3\lib\site-packages\tmrl\__main__.py", line 81, in <module>
main(arguments)
File "C:\ProgramData\Anaconda3\lib\site-packages\tmrl\__main__.py", line 42, in main
rw.run_episodes(10000)
File "C:\ProgramData\Anaconda3\lib\site-packages\tmrl\networking.py", line 956, in run_episodes
self.run_episode(max_samples_per_episode, train=train)
File "C:\ProgramData\Anaconda3\lib\site-packages\tmrl\networking.py", line 972, in run_episode
obs = self.reset(collect_samples=False)
File "C:\ProgramData\Anaconda3\lib\site-packages\tmrl\networking.py", line 867, in reset
new_obs = self.env.reset()
File "C:\ProgramData\Anaconda3\lib\site-packages\gym\core.py", line 323, in reset
return self.env.reset(**kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\gym\core.py", line 379, in reset
obs, info = self.env.reset(**kwargs)
ValueError: too many values to unpack (expected 2)
(base) C:\Windows\system32>
I have sent you guys an e-mail, maybe went to spam. Could you check it out?
Hey,
I was wondering if you're still working on this project? I'd love to contribute to it if it's possible in any way. =)
(I'm a Data Scientist who worked on GANs during my PhD and I'd love to learn about reinforcement learning!)
I tried to cleanly install TMRL again, and deleted the folder and ran the pip command as per the instructions. The pip command worked fine, but there was no TmrlData folder created.
How do I get it back?
Hi, I followed the instructions in the tutorial, but when i launch the trainer, i have got this message:
wandb: ERROR Error while calling W&B API: run tmrl/tmrl/SAC_4_imgs_pretrained was previously created and deleted; try a new run name (<Response [409]>)
Hello!
So on the wandb.ai website it says my round_time is around 200, but in reality the round times are around 20-23 while it trains etc. Also would it be possible to see how many times the ai has resat on a map/how many times it has driven it?s
I followed all instructions in the tutorial.
All tests seem to work. However, when I run the test script, nothing happens. The only thing I see is a restart every few seconds.
What I did:
When I run python -m tmrl --test
, I get the following output (nothing else).
(.venv) PS C:\PATH\tmrl> python -m tmrl --test
INFO:root:Namespace(server=False, trainer=False, worker=False, test=True, benchmark=False, record_reward=False, check_env=False, no_wandb=False, config={})
INFO:root:10/22/23 15:24:54 server IP: 127.0.0.1
C:\PATH\venv\Lib\site-packages\gymnasium\core.py:311: UserWarning: WARN: env.default_action to get variables from other wrappers is deprecated and will be removed in v1.0, to get this variable you can do `env.unwrapped.default_action` for environment variables or `env.get_wrapper_attr('default_action')` that will search the reminding wrappers.
logger.warn(
Hi while installing the env through the installation guide, I ran the validation command python -m tmrl --install
and it yielded a ModuleNotFoundError: No module named 'keyboard'
.
Doing a pip install keyboard
solved my issue, however maybe it could be fixed in the setup.py
Is there a way to be able to run the code via WSL
And run trackmania on windows?
nvm ig
This should be mentioned in the readme somewhere.
C:\Users\jake>python -m tmrl
Traceback (most recent call last):
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python39_64\lib\runpy.py", line 188, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python39_64\lib\runpy.py", line 147, in _get_module_details
return get_module_details(pkg_main_name, error)
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python39_64\lib\runpy.py", line 111, in get_module_details
import(pkg_name)
File "C:\Users\jake\AppData\Roaming\Python\Python39\site-packages\tmrl_init.py", line 35, in
from tmrl.config.config_objects import CONFIG_DICT
File "C:\Users\jake\AppData\Roaming\Python\Python39\site-packages\tmrl\config\config_objects.py", line 10, in
from tmrl.training_offline import TorchTrainingOffline
File "C:\Users\jake\AppData\Roaming\Python\Python39\site-packages\tmrl\training_offline.py", line 6, in
import torch
File "C:\Users\jake\AppData\Roaming\Python\Python39\site-packages\torch_init.py", line 141, in
raise err
OSError: [WinError 126] The specified module could not be found. Error loading "C:\Users\jake\AppData\Roaming\Python\Python39\site-packages\torch\lib\shm.dll" or one of its dependencies.
Hello. I am running the example code. I did not change the config, so the wandb info is still the same but when I run the trainer I get the following traceback.
wandb: View run jolly-violet-855 at: https://wandb.ai/tmrl/tmrl/runs/test_123
wandb: View project at: https://wandb.ai/tmrl/tmrl
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: C:\Users\Rajiv\AppData\Local\Temp\tmp12lymgnh\wandb\run-20240508_163952-test_123\logs
Exception in thread NetStatThr:
Traceback (most recent call last):
File "E:\Python\Python311\Lib\threading.py", line 1038, in _bootstrap_inner
self.run()
File "E:\Python\Python311\Lib\threading.py", line 975, in run
self._target(*self._args, **self._kwargs)
File "D:\Project\MGAIA\MGAIA_A3\venv\Lib\site-packages\wandb\sdk\wandb_run.py", line 278, in check_network_status
self._loop_check_status(
File "D:\Project\MGAIA\MGAIA_A3\venv\Lib\site-packages\wandb\sdk\wandb_run.py", line 233, in _loop_check_status
local_handle = request()
^^^^^^^^^
File "D:\Project\MGAIA\MGAIA_A3\venv\Lib\site-packages\wandb\sdk\interface\interface.py", line 884, in deliver_network_status
return self._deliver_network_status(status)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Project\MGAIA\MGAIA_A3\venv\Lib\site-packages\wandb\sdk\interface\interface_shared.py", line 504, in _deliver_network_status
return self._deliver_record(record)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Project\MGAIA\MGAIA_A3\venv\Lib\site-packages\wandb\sdk\interface\interface_shared.py", line 453, in _deliver_record
handle = mailbox._deliver_record(record, interface=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Project\MGAIA\MGAIA_A3\venv\Lib\site-packages\wandb\sdk\lib\mailbox.py", line 455, in _deliver_record
interface._publish(record)
File "D:\Project\MGAIA\MGAIA_A3\venv\Lib\site-packages\wandb\sdk\interface\interface_sock.py", line 51, in _publish
self._sock_client.send_record_publish(record)
File "D:\Project\MGAIA\MGAIA_A3\venv\Lib\site-packages\wandb\sdk\lib\sock_client.py", line 221, in send_record_publish
self.send_server_request(server_req)
File "D:\Project\MGAIA\MGAIA_A3\venv\Lib\site-packages\wandb\sdk\lib\sock_client.py", line 155, in send_server_request
self._send_message(msg)
File "D:\Project\MGAIA\MGAIA_A3\venv\Lib\site-packages\wandb\sdk\lib\sock_client.py", line 152, in _send_message
self._sendall_with_error_handle(header + data)
File "D:\Project\MGAIA\MGAIA_A3\venv\Lib\site-packages\wandb\sdk\lib\sock_client.py", line 130, in _sendall_with_error_handle
sent = self._sock.send(data)
^^^^^^^^^^^^^^^^^^^^^
ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine
This traceback does not happen when I edit out the wandb key from the config and select "don't visualize...".
But I'd like to have the visuals. I don't know if this is a TMRL issue or something on my end. But I hope someone can shed some light on this.
Hello! I'm getting the following error while running --check-envoronment
subcommand:
File "/home/vladimir/.conda/envs/tmrl/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/vladimir/.conda/envs/tmrl/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/vladimir/.conda/envs/tmrl/lib/python3.10/site-packages/tmrl/__main__.py", line 84, in <module>
main(arguments)
File "/home/vladimir/.conda/envs/tmrl/lib/python3.10/site-packages/tmrl/__main__.py", line 63, in main
check_env_tm20lidar()
File "/home/vladimir/.conda/envs/tmrl/lib/python3.10/site-packages/tmrl/tools/check_environment.py", line 16, in check_env_tm20lidar
lidar = Lidar(window_interface.screenshot())
File "/home/vladimir/.conda/envs/tmrl/lib/python3.10/site-packages/tmrl/custom/utils/window.py", line 180, in screenshot
monitor = {"top": self.x + self.x_offset, "left": self.y + self.y_offset, "width": self.w, "height": self.h}
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'
As you could see I'm running TMRL on linux inside conda environment. I've followed readme/install_linux.md
and readme/get_started.md
tutorials. I've installed the xdotool as it's requrested. Yet I'm getting the error above. Here we config I got based on mentioned tutorials:
{
"RUN_NAME": "SAC_4_LIDAR_pretrained",
"RESET_TRAINING": false,
"BUFFERS_MAXLEN": 500000,
"RW_MAX_SAMPLES_PER_EPISODE": 1000,
"CUDA_TRAINING": true,
"CUDA_INFERENCE": false,
"VIRTUAL_GAMEPAD": true,
"DCAC": false,
"LOCALHOST_WORKER": true,
"LOCALHOST_TRAINER": true,
"PUBLIC_IP_SERVER": "0.0.0.0",
"PASSWORD": "==>TMRL@UseASecurePasswordHere!<==",
"TLS": false,
"TLS_HOSTNAME": "default",
"TLS_CREDENTIALS_DIRECTORY": "",
"NB_WORKERS": -1,
"WANDB_PROJECT": "<IT'S IN HERE JUST DONT WANT TO PUBLISH IT>",
"WANDB_ENTITY": "tmrl",
"WANDB_KEY": "<IT'S IN HERE JUST DONT WANT TO PUBLISH IT>",
"PORT": 55555,
"LOCAL_PORT_SERVER": 55556,
"LOCAL_PORT_TRAINER": 55557,
"LOCAL_PORT_WORKER": 55558,
"BUFFER_SIZE": 536870912,
"HEADER_SIZE": 12,
"SOCKET_TIMEOUT_CONNECT_TRAINER": 300.0,
"SOCKET_TIMEOUT_ACCEPT_TRAINER": 300.0,
"SOCKET_TIMEOUT_CONNECT_ROLLOUT": 300.0,
"SOCKET_TIMEOUT_ACCEPT_ROLLOUT": 300.0,
"SOCKET_TIMEOUT_COMMUNICATE": 30.0,
"SELECT_TIMEOUT_OUTBOUND": 30.0,
"ACK_TIMEOUT_WORKER_TO_SERVER": 300.0,
"ACK_TIMEOUT_TRAINER_TO_SERVER": 300.0,
"ACK_TIMEOUT_SERVER_TO_WORKER": 300.0,
"ACK_TIMEOUT_SERVER_TO_TRAINER": 7200.0,
"RECV_TIMEOUT_TRAINER_FROM_SERVER": 7200.0,
"RECV_TIMEOUT_WORKER_FROM_SERVER": 600.0,
"WAIT_BEFORE_RECONNECTION": 10.0,
"LOOP_SLEEP_TIME": 1.0,
"MAX_EPOCHS": 10000,
"ROUNDS_PER_EPOCH": 10,
"TRAINING_STEPS_PER_ROUND": 1000,
"MAX_TRAINING_STEPS_PER_ENVIRONMENT_STEP": 4.0,
"ENVIRONMENT_STEPS_BEFORE_TRAINING": 1000,
"UPDATE_MODEL_INTERVAL": 1000,
"UPDATE_BUFFER_INTERVAL": 1000,
"SAVE_MODEL_EVERY": 0,
"MEMORY_SIZE": 1000000,
"BATCH_SIZE": 256,
"ALG": {
"ALGORITHM": "SAC",
"LEARN_ENTROPY_COEF":false,
"LR_ACTOR":0.00001,
"LR_CRITIC":0.00005,
"LR_ENTROPY":0.0003,
"GAMMA":0.995,
"POLYAK":0.995,
"TARGET_ENTROPY":-0.5,
"ALPHA":0.01,
"REDQ_N":10,
"REDQ_M":2,
"REDQ_Q_UPDATES_PER_POLICY_UPDATE":20,
"OPTIMIZER_ACTOR": "adam",
"OPTIMIZER_CRITIC": "adam",
"BETAS_ACTOR": [0.997, 0.997],
"BETAS_CRITIC": [0.997, 0.997],
"L2_ACTOR": 0.0,
"L2_CRITIC": 0.0
},
"ENV": {
"RTGYM_INTERFACE": "TM20LIDAR",
"WINDOW_WIDTH": 958,
"WINDOW_HEIGHT": 488,
"IMG_WIDTH": 64,
"IMG_HEIGHT": 64,
"IMG_GRAYSCALE": true,
"SLEEP_TIME_AT_RESET": 1.5,
"IMG_HIST_LEN": 4,
"RTGYM_CONFIG": {
"time_step_duration": 0.05,
"start_obs_capture": 0.04,
"time_step_timeout_factor": 1.0,
"act_buf_len": 2,
"benchmark": false,
"wait_on_done": true,
"ep_max_length": 1000,
"interface_kwargs": {"save_replays": false}
},
"REWARD_CONFIG": {
"END_OF_TRACK": 100.0,
"CONSTANT_PENALTY": 0.0,
"CHECK_FORWARD": 500,
"CHECK_BACKWARD": 10,
"FAILURE_COUNTDOWN": 10,
"MIN_STEPS": 70,
"MAX_STRAY": 100.0
}
},
"__VERSION__": "0.6.0"
}
I had to install keyboard explicitly. To record the reward I have to run the command with sudo. Thus, I've added a symlink to ~/TmrlData to /root to make config available.
I modify the original code to make my own IA (the main.py file but I change cfg and other params). the observation is now :
def get_observation_space(self):
"""
must be a Tuple
"""
speed = spaces.Box(low=0.0, high=1000.0, shape=(1, ))
poss = spaces.Box(low=0.0, high=999999999, shape=(
self.pos_hist_len,
3,
)) # historic of position
track = {
"name": spaces.Box(0, 99999999, (1, ), np.uint64),
"rotation": spaces.Box(0, 6, (1, ), np.int32),
"position": spaces.Box(-99999999, 99999999, (3, ), np.int64),
"speed": spaces.Box(-99999999, 99999999, (1, ), np.int64),
"flags": spaces.Box(0, 8000000000, (1, ), np.uint64),
"skin": spaces.Box(0, 8000000000, (1, ), np.uint64),
}
max_track_element = 50
return spaces.Tuple((speed, poss, spaces.Space((max_track_element, spaces.Dict(track)))))
(I edit to support dict and etc) but the trainer save a wrong datamodel and I don't know why but I also edit ENV_CLS and put the same for worker and it's when the worker try to load that make this error
File "D:\GitHub\trackmania-ia-with-tmrl\worker.py", line 516, in <module>
rw.run()
File "D:\GitHub\trackmania-ia-with-tmrl\worker.py", line 436, in run
self.update_actor_weights()
File "D:\GitHub\trackmania-ia-with-tmrl\worker.py", line 499, in update_actor_weights
self.actor.load_state_dict(torch.load(self.model_path, map_location=self.device))
File "C:\Users\User\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\nn\modules\module.py", line 1604, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for SquashedGaussianMLPActor:
size mismatch for net.0.weight: copying a param with shape torch.Size([256, 83]) from checkpoint, the shape in current model is torch.Size([256, 315]).
you can find the complete repo here
C:\Windows\System32>python -m tmrl --test
INFO:numexpr.utils:Note: NumExpr detected 16 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
INFO:numexpr.utils:NumExpr defaulting to 8 threads.
INFO:root:Namespace(server=False, trainer=False, worker=False, test=True, benchmark=False, record_reward=False, check_env=False, no_wandb=False, config={})
INFO:root:11/06/23 16:05:52 server IP: 127.0.0.1
C:\Users\haris\anaconda3\Lib\site-packages\gymnasium\core.py:311: UserWarning: WARN: env.default_action to get variables from other wrappers is deprecated and will be removed in v1.0, to get this variable you can do `env.unwrapped.default_action` for environment variables or `env.get_wrapper_attr('default_action')` that will search the reminding wrappers.
logger.warn(
Exception in thread Thread-2 (__client_thread):
Traceback (most recent call last):
File "C:\Users\haris\anaconda3\Lib\threading.py", line 1038, in _bootstrap_inner
self.run()
File "C:\Users\haris\anaconda3\Lib\threading.py", line 975, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\haris\anaconda3\Lib\site-packages\tmrl\custom\utils\tools.py", line 41, in __client_thread
s.connect((self._host, self._port))
ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "C:\Users\haris\anaconda3\Lib\site-packages\tmrl\__main__.py", line 82, in <module>
main(arguments)
File "C:\Users\haris\anaconda3\Lib\site-packages\tmrl\__main__.py", line 41, in main
rw.run_episodes(10000)
File "C:\Users\haris\anaconda3\Lib\site-packages\tmrl\networking.py", line 648, in run_episodes
self.run_episode(max_samples_per_episode, train=train)
File "C:\Users\haris\anaconda3\Lib\site-packages\tmrl\networking.py", line 663, in run_episode
obs, info = self.reset(collect_samples=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\haris\anaconda3\Lib\site-packages\tmrl\networking.py", line 561, in reset
new_obs, info = self.env.reset()
^^^^^^^^^^^^^^^^
File "C:\Users\haris\anaconda3\Lib\site-packages\gymnasium\core.py", line 467, in reset
return self.env.reset(seed=seed, options=options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\haris\anaconda3\Lib\site-packages\gymnasium\core.py", line 515, in reset
obs, info = self.env.reset(seed=seed, options=options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\haris\anaconda3\Lib\site-packages\gymnasium\wrappers\order_enforcing.py", line 61, in reset
return self.env.reset(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\haris\anaconda3\Lib\site-packages\rtgym\envs\real_time_env.py", line 514, in reset
elt, info = self.interface.reset(seed=seed, options=options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\haris\anaconda3\Lib\site-packages\tmrl\custom\custom_gym_interfaces.py", line 157, in reset
data, img = self.grab_data_and_img()
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\haris\anaconda3\Lib\site-packages\tmrl\custom\custom_gym_interfaces.py", line 134, in grab_data_and_img data = self.client.retrieve_data()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\haris\anaconda3\Lib\site-packages\tmrl\custom\utils\tools.py", line 72, in retrieve_data
assert t_now - t_start < timeout, f"OpenPlanet stopped sending data since more than {timeout}s."
^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: OpenPlanet stopped sending data since more than 10.0s.
I am trying to use TMRL with a keras PPO ai and I was having some problems with the input so I tried a clean install I uninstalled tmrl, then deleted the tmrlData folder, and then reinstalled tmrl, but it didn't generate the tmrlData
Can somebody help, I am on windows 11, using vscode
Hi!
When I run the command python -m tmrl --test
it gives me an error:
INFO:root:pywin32 failed to import. Attempting to fix pywin32 installation...
ERROR:root:tmrl could not fix pywin32 on your system. The following exceptions were raised:
=== Exception 1 ===
str(e1)
=== Exception 2 ===
str(e2)
Please install pywin32 manually.
Traceback (most recent call last):
File "C:\Users\isabe\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\tmrl\__init__.py", line 10, in <module>
import win32gui
ImportError: DLL load failed: Le module spécifié est introuvable.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\isabe\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\tmrl\__init__.py", line 17, in <module>
fix_pywin32()
File "C:\Users\isabe\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\tmrl\tools\init_package\init_pywin32.py", line 698, in fix_pywin32
install(destination)
File "C:\Users\isabe\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\tmrl\tools\init_package\init_pywin32.py", line 433, in install
LoadSystemModule(lib_dir, "pywintypes")
File "C:\Users\isabe\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\tmrl\tools\init_package\init_pywin32.py", line 196, in LoadSystemModule
mod = importlib.util.module_from_spec(spec)
File "<frozen importlib._bootstrap>", line 583, in module_from_spec
File "<frozen importlib._bootstrap_external>", line 1043, in create_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: DLL load failed: Le module spécifié est introuvable.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.2544.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 183, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.2544.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 142, in _get_module_details
return _get_module_details(pkg_main_name, error)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.2544.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 109, in _get_module_details
__import__(pkg_name)
File "C:\Users\isabe\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\tmrl\__init__.py", line 25, in <module>
raise RuntimeError("Please install pywin32 manually: https://github.com/mhammond/pywin32")
RuntimeError: Please install pywin32 manually: https://github.com/mhammond/pywin32
So I installed pywin32
with conda install pywin32
but when I run the command again, it gives me the same error...
I don't know what to do, I'm stuck here. Thanks for your help.
Hey im having some trouble with moving my tmrl file because when i move it i get this
PS F:\TMRL-AI> python -m tmrl --server Traceback (most recent call last): File "<frozen runpy>", line 189, in _run_module_as_main File "<frozen runpy>", line 148, in _get_module_details File "<frozen runpy>", line 112, in _get_module_details File "C:\Users\liamr\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tmrl\__init__.py", line 31, in <module> from tmrl.config.config_objects import CONFIG_DICT File "C:\Users\liamr\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tmrl\config\config_objects.py", line 8, in <module> import tmrl.config.config_constants as cfg File "C:\Users\liamr\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tmrl\config\config_constants.py", line 16, in <module> with open(CONFIG_FILE) as f: ^^^^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\liamr\\TmrlData\\config\\config.json'
and i dont know how to change the location
thank you
I am currently experimenting with the command line interface for this library, but am constantly getting thrown a TypeError.
I have tried experimenting with tweaking the parameters, restarting openplanet, etc, but to no avail.
Here are the command line logs from the trainer, the other 2 scripts appear to be functioning properly.
I am running this in powershell with python 3.10, with no wandb connection
INFO:root:Namespace(server=False, trainer=True, worker=False, test=False, benchmark=False, record_reward=False, check_env=False, no_wandb=True, config={})
INFO:root:10/08/23 23:58:30 server IP: 127.0.0.1
INFO:root:--- NOW RUNNING SAC on TrackMania ---
INFO:root:Loading checkpoint...
INFO:root: Loaded checkpoint in 0.0033698081970214844 seconds.
INFO:root:Updating checkpoint...
INFO:root:Target entropy: -0.5.
INFO:root:Max epochs changed to 100 (old: 10000).
INFO:root:Rounds per epoch changed to 5 (old: 10).
INFO:root:Checkpoint updated in 0.0 seconds.
INFO:root:=== epoch 0/100 ==== round 0/5 =======================================
INFO:root: Waiting for new samples
INFO:root: Resuming training
INFO:root:starting training
C:\Users\Yile0\AppData\Local\Programs\Python\Python310\lib\site-packages\tmrl\custom\utils\nn.py:44: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
assert b.storage().data_ptr() == a.storage().data_ptr()
Traceback (most recent call last):
File "C:\Users\Yile0\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Yile0\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\Yile0\AppData\Local\Programs\Python\Python310\lib\site-packages\tmrl\__main__.py", line 82, in <module>
main(arguments)
File "C:\Users\Yile0\AppData\Local\Programs\Python\Python310\lib\site-packages\tmrl\__main__.py", line 56, in main
trainer.run()
File "C:\Users\Yile0\AppData\Local\Programs\Python\Python310\lib\site-packages\tmrl\networking.py", line 393, in run
run(interface=self.interface,
File "C:\Users\Yile0\AppData\Local\Programs\Python\Python310\lib\site-packages\tmrl\networking.py", line 326, in run
for stats in iterate_epochs_tm(run_cls, interface, checkpoint_path, dump_run_instance_fn, load_run_instance_fn, 1, updater_fn):
File "C:\Users\Yile0\AppData\Local\Programs\Python\Python310\lib\site-packages\tmrl\networking.py", line 269, in iterate_epochs_tm
yield run_instance.run_epoch(interface=interface) # yield stats data frame (this makes this function a generator)
File "C:\Users\Yile0\AppData\Local\Programs\Python\Python310\lib\site-packages\tmrl\training_offline.py", line 153, in run_epoch
stats += pandas_dict(memory_len=len(self.memory), round_time=round_time, idle_time=idle_time, **DataFrame(stats_training).mean(skipna=True)),
File "C:\Users\Yile0\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\frame.py", line 11338, in mean
result = super().mean(axis, skipna, numeric_only, **kwargs)
File "C:\Users\Yile0\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\generic.py", line 11978, in mean
return self._stat_function(
File "C:\Users\Yile0\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\generic.py", line 11935, in _stat_function
return self._reduce(
File "C:\Users\Yile0\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\frame.py", line 11207, in _reduce
res = df._mgr.reduce(blk_func)
File "C:\Users\Yile0\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\internals\managers.py", line 1459, in reduce
nbs = blk.reduce(func)
File "C:\Users\Yile0\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\internals\blocks.py", line 377, in reduce
result = func(self.values)
File "C:\Users\Yile0\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\frame.py", line 11139, in blk_func
return op(values, axis=axis, skipna=skipna, **kwds)
File "C:\Users\Yile0\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\nanops.py", line 147, in f
result = alt(values, axis=axis, skipna=skipna, **kwds)
File "C:\Users\Yile0\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\nanops.py", line 404, in new_func
result = func(values, axis=axis, skipna=skipna, mask=mask, **kwargs)
File "C:\Users\Yile0\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\nanops.py", line 720, in nanmean
the_sum = _ensure_numeric(the_sum)
File "C:\Users\Yile0\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\nanops.py", line 1678, in _ensure_numeric
raise TypeError(f"Could not convert {x} to numeric")
TypeError: Could not convert [tensor(1851.6046) tensor(326.9721)] to numeric
Hi!
In the scope of a school project, alongside with my team we woud like to use your library to create an AI that drives in TrackMania. We were wondering if it was possible to change the map on which the AI is driving. In fact, we managed to get it run on different map, although it keeps reseting after a couple of second.
Therefore, is there a way to setup new maps for training/testing? If yes, where should we look at?
Thanks!
Hello! I am fairly new to python so I don't have much knowledge on it. I am getting a consistent error where after ten seconds of pressing e when setting up the rewards system for a new bot/AI I get this error:
AssertionError: OpenPlanet stopped sending data since more than 10.0s.
Traceback:
Traceback (most recent call last): File "C:\Users\Omar\AppData\Local\Programs\Python\Python310\Lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\Omar\AppData\Local\Programs\Python\Python310\Lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Users\Omar\AppData\Local\Programs\Python\Python310\lib\site-packages\tmrl\__main__.py", line 82, in <module> main(arguments) File "C:\Users\Omar\AppData\Local\Programs\Python\Python310\lib\site-packages\tmrl\__main__.py", line 58, in main record_reward_dist(path_reward=cfg.REWARD_PATH) File "C:\Users\Omar\AppData\Local\Programs\Python\Python310\lib\site-packages\tmrl\tools\record.py", line 30, in record_reward_dist data = client.retrieve_data(sleep_if_empty=0.01) # we need many points to build a smooth curve File "C:\Users\Omar\AppData\Local\Programs\Python\Python310\lib\site-packages\tmrl\custom\utils\tools.py", line 72, in retrieve_data assert t_now - t_start < timeout, f"OpenPlanet stopped sending data since more than {timeout}s."
What was printed before trackback:
INFO:root:Namespace(server=False, trainer=False, worker=False, test=False, benchmark=False, record_reward=True, check_env=False, no_wandb=False, config={}) INFO:root:start recording
This happens even after I reload Trackmania, reload the plugin, and switch out the Script folder in OpenplanetNext with the one in TmrlData. Please help! Thanks in advance! :)
[ ScriptEngine] [09:28:57] [Plugin_GrabData_0_1] C:/Users/Malik/OpenplanetNext/Scripts/Plugin_GrabData_0_1.as (16, 1) : INFO : Compiling void Main()
[ ScriptEngine] [09:28:57] [Plugin_GrabData_0_1] C:/Users/Malik/OpenplanetNext/Scripts/Plugin_GrabData_0_1.as (64, 21) : ERR : No matching symbol 'CGameTerminal::ESGamePlaygroundUIConfig__EUISequence::Finish'
[ ScriptEngine] [09:28:57] Script compilation failed!
Thats my openplanet log can somebody help me
Hi, very cool repo. Would it be possible for it to be upgraded from gym to gymnasium? Gymnasium is the maintained version of openai gym and is compatible with current RL training libraries (rllib and tianshou have already migrated, and stable-baselines3 will soon).
This repository is currently listed in the gymnasium third party environments but we are cleaning the list up to only include maintained gymnasium-compatible repositories.
For information about upgrading and compatibility, see migration guide and gym compatibility. The main difference is the API has switched to returning truncated
and terminated
, rather than done
, in order to give more information and mitigate edge case issues.
Hi Yann,
I really like your library, I have been trying to port it to play a much quicker game, currently OLDTV
https://store.steampowered.com/app/643270/OLDTV/ and I would like your feedback on how to setup the reward function and action_states.
I looked at the guide you provided for inputs and reward function but i am unable to apply it for something like OLDTV. I am very new to this field so any advice would be appreciated
the code i have been following is mainly this: https://github.com/trackmania-rl/tmrl/blob/master/tmrl/custom/custom_gym_interfaces.py
thank you
Victor
It's my first IA so I don't know what it is but make the error IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)
so when I try to fix he make more error so how I can fix this
the code originaly:
# Compute logprob from Gaussian, and then apply correction for Tanh squashing.
# NOTE: The correction formula is a little bit magic. To get an understanding
# of where it comes from, check out the original SAC paper (arXiv 1801.01290)
# and look in appendix C. This is a more numerically-stable equivalent to Eq 21.
# Try deriving it yourself as a (very difficult) exercise. :)
logp_pi = pi_distribution.log_prob(pi_action).sum(axis=-1)
logp_pi -= (2 * (np.log(2) - pi_action - F.softplus(-2 * pi_action))).sum(axis=1)```
pi_action is the same format as in the default code
We are running tmrl on linux. Our GPU is Nvidia RTX 3090. The server and worker seem to run just fine. However, after a few minutes of running, the trainer outputs the following error:
/home/trackmania-rl/env-linux/lib/python3.10/site-packages/torch/nn/modules/conv.py:456: UserWarning: Attempt to open cnn_infer failed: handle=0 error: /home/trackmania-rl/env-linux/lib/python3.10/site-packages/torch/lib/../../nvidia/cudnn/lib/libcudnn_cnn_infer.so.8: undefined symbol: _Z22cudnnGenericOpTensorNdILi2EE13cudnnStatus_tP12cudnnContext16cudnnGenericOp_t21cudnnNanPropagation_tPK21cudnnActivationStructPKvPK17cudnnTensorStructS9_S9_SC_S9_S9_SC_Pv, version libcudnn_ops_infer.so.8 (Triggered internally at ../aten/src/ATen/native/cudnn/Conv_v8.cpp:78.)
return F.conv2d(input, weight, bias, self.stride,
Traceback (most recent call last):
File "/home/trackmania-rl/env-linux/custom_actor_module.py", line 860, in <module>
my_trainer.run_with_wandb(entity=wandb_entity,
File "/home/trackmania-rl/env-linux/lib/python3.10/site-packages/tmrl/networking.py", line 419, in run_with_wandb
run_with_wandb(entity=entity,
File "/home/trackmania-rl/env-linux/lib/python3.10/site-packages/tmrl/networking.py", line 317, in run_with_wandb
for stats in iterate_epochs_tm(run_cls, interface, checkpoint_path, dump_run_instance_fn, load_run_instance_fn, 1, updater_fn):
File "/home/trackmania-rl/env-linux/lib/python3.10/site-packages/tmrl/networking.py", line 270, in iterate_epochs_tm
yield run_instance.run_epoch(interface=interface) # yield stats data frame (this makes this function a generator)
File "/home/trackmania-rl/env-linux/lib/python3.10/site-packages/tmrl/training_offline.py", line 127, in run_epoch
stats_training_dict = self.agent.train(batch)
File "/home/trackmania-rl/env-linux/custom_actor_module.py", line 737, in train
pi, logp_pi = self.model.actor(obs=o, test=False, compute_logprob=True)
File "/home/trackmania-rl/env-linux/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/trackmania-rl/env-linux/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/trackmania-rl/env-linux/custom_actor_module.py", line 511, in forward
net_out = self.net(obs)
File "/home/trackmania-rl/env-linux/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/trackmania-rl/env-linux/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/trackmania-rl/env-linux/custom_actor_module.py", line 355, in forward
x = F.relu(self.conv1(images))
File "/home/trackmania-rl/env-linux/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/trackmania-rl/env-linux/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/trackmania-rl/env-linux/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 460, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/home/trackmania-rl/env-linux/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 456, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: GET was unable to find an engine to execute this computation
What we have tried:
Any suggestions?
The testmodel and testmap works great. Now I am experimenting with training my own model on my own map. I have recorded a reward function using the command and when running python -m tmrl --check-environment
I can se that the reward increases when I drive faster along the path of my map so I think that is working.
However when I start the server, worker and trainer it always seem to prefer staying still and not giving any inputs. It might try driving a bit forward on the first run but then it just stays. The virtual controller is working since if I give it a push by pressing forwards it will steer.
Do you have any clue what I might be missing.
I tried renaming the SAC_4_LIDAR_pretrained
and weights in case it used some old data in case I accidentally trained it to stand still. But I got the same result.
Thank you so much for your dedication on this amazing project 🙏
Hello,
When i start my lidar training I get this message in the trainer terminal window:
"wandb:
Error while calling W&B API: run tmrl/tmr/SAC_4_imgs_pretrained was previously created and deleted; try a new run name (Response [409]>) Error while calling WAB API: run tmrl/tmrl/SAC 4 imgs pretrained was previously created and deleted; try a new run name (Response [409]>)"
After 90 seconds it times out and I get back to the normal command line. The car moves but I dont know if it actually improves. It resets every 10s-13s and starts again
Anyone know whats the problem here?
I have gotten the model to start training on a map I created. However, it would be really fun to be able to record all the AI runs to create a merged replay of all the ghosts to see its progression. Is there any way to do this?
Also, another question when training the AI will the progress it makes automatically be saved in the .pth file when I close the worker with ctrl+c so I can resume later?
This issue happens randomly and very rarely so I don't know how to reproduce it.
While saving the ckeckpoint (i.e. pickling the Trainer), it may happen that pickle crashes with a segmentation fault and no traceback.
Any additional information regarding this issue would be welcome.
When I try to turn "interface_kwargs": {"save_replays": false} in config.json to true,the worker doesn't control car and throw out [WinError 10061].
My OpenPlanet version is 1.26.25
My tmrl version is 0.6.0
Thanks!
Following an error in an uninstallation, I no longer have neffarius. I think that's why the car won't move. Can anyone help me to sort out this problem?
I get the following error when I try and import tmrl or run >python -m tmrl --test:
C:\Users\saksh>python -m tmrl --test
Traceback (most recent call last):
File "", line 189, in _run_module_as_main
File "", line 148, in _get_module_details
File "", line 112, in get_module_details
File "C:\Users\saksh\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tmrl_init.py", line 33, in
from tmrl.config.config_objects import CONFIG_DICT
File "C:\Users\saksh\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tmrl\config\config_objects.py", line 9, in
import tmrl.config.config_constants as cfg
File "C:\Users\saksh\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tmrl\config\config_constants.py", line 37, in
BUFFERS_MAXLEN = TMRL_CONFIG["BUFFERS_MAXLEN"] # Maximum length of the local buffers for RolloutWorkers, Server and TrainerInterface
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
KeyError: 'BUFFERS_MAXLEN'
I have a problem because my C disk was full so I couldn't train more because is can't save new checkpoint.
Can I move checkpoint files to another disk?
If I can, how to change the code
On going to the test track(map) and typing "python -m tmrl --test" everything is working, i get the sound like if new usb device is connected, even map restarts, but nothing happends after, i get RuntimeError: Numpy is not available
ofc numpy is installed, version is 1.22.4
python 3.10
Closed it - works after i reinstalled numpy
Dear author, I am a graduate student, and I want to use parallel genetic algorithm to train my racing AI, so I read your article. I have to say that your article is very hardcore. As a novice, I can only barely understand your "get started" part and implement the running of the test file. But then, you asked me to read the Tutorial: TMRL as a Python library part, which made me confused. Please forgive my stupidity, but you tried to teach me to write a py file, but even if I learned how to write this file, which folder should I put it in, and what file should I run? In addition, you also provided a large number of script files in the directory, such as custom, tools, etc. Where should these files be placed, and which file should I run? I have no answer to these questions. I hope you can provide more detailed explanations for beginners like me, thank you very much! !
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.