Dear Ling-Pan,

I am writing to request an update to the OMAR benchmark code to include implementation for the discrete action space. Despite our best efforts, we have been unable to locate the necessary code. Furthermore, we have found that directly applying the Gumbel-Softmax reparameterization method as the original paper has resulted in poor performance. Would it be possible for you to provide the corresponding code or update the repository accordingly?

Thank you in advance for your attention to this matter.

In main.py, there is an undefined function named prepare_output_dir.

I know it is not a big deal, but it should be fixed to improve your code.

Introduction

I successfully run the code on Ubuntu 20.04 within the following environment:

• Python == 3.7.2
• baselines == 0.1.6 (local)
• Cython == 0.29.37
• gym == 0.10.8
• mujoco-py == 2.0.2.8
• multiagent == 0.0.1 (local)
• seaborn == 0.12.2
• torch == 1.12.1+cu116

Problems during the installation mostly occured on mujoco-py and to be honest there's just no way around it. Few tips I can offer is to stay patient and it will somehow work out :)

When running the code, thing gets stricker. I tried three env_id and only one managed to "output results". Here are the problems I came across and my solutions to it:

Datasets

1. simple_tag / simple_world (didn't run out)

- In algorithms/maddpg,py/init_from_env(), line 453:

num_in_pol = obsp.shape[0]    #observation space
num_out_pol = acsp.shaep[0]   #action space

However, ascp doesn't possess the shape[0] attribute, which leads to IndexOutBounday exception (Also, code follows it comes up with access to acsp.high[0] which would cause NoAttribute exception).

I failed to find out the possible reasons by looking following the stack up to the multiagent-particle-envs/multiagent/environment.py/__init__(), where the env is created. So I guess it either involves version incompatibility of multiagent (but it only comes with one version 0.0.1 and that is the one installed), or the code might be modified to satisfy other datasets but those two.

- In utils/make_env.py/make_env():

if benchmark or scenario_name == 'simple_spread':
    env = MultiAgentEnv(world, scenario.reset_world, scenario.reward, scenario.observation, scenario.benchmark_data, discrete_action=discrete_action)
else:
    env = MultiAgentEnv(world, scenario.reset_world, scenario.reward, scenario.observation, discrete_action=discrete_action)

MultiAgentEnv() doesn't have parameter discrete_action, the implement of it only has discrete_action_space and discrete_action_input, which is set True and False by default seperately. I have no idea but to simply delete the parameter and everything seems fine to me.

2. HalfCheetah v2

I noticed that there are a few if else desinated to seperate HalfCheetah v2 apart from others, so I decided to give it a try. With the two modifcations above, I came across the following problem:

In algorithms/maddpg,py/update(), line 257:

dist = torch.distributions.Normal(self.omar_mu, self.omar_sigma)

This line will throw an exception about ValueError: Expected parameter scale of distribution Normal to satisfy the constraint GreaterThan..., which means this function can only take values that are larger than 0 as input but is called that way.

The reference of torch.distributions.Normal() is put within a for recycle and even I set self.omar_mu and self.omar_sigma to meet the requirement, it will throw the same exception right on the second recycle. Therefore, I tried torch.clamp() and it seems worked. Here is the modified code concerned:

if self.is_mamujoco:
    ...
    # Adding a tiny postive shift
    self.omar_mu += 1e-6
    self.omar_sigma += 1e-6      

    for iter_idx in range(self.omar_iters):
        dist = torch.distributions.Normal(self.omar_mu, self.omar_sigma) 
        ...
        updated_mu = torch.mean(elite_acs, dim=1)
        updated_sigma = torch.std(elite_acs, dim=1)

        # Make sure the updated parameters are bigger than 0
        updated_mu = torch.clamp(updated_mu, min=1e-6)
        updated_sigma = torch.clamp(updated_sigma, min=1e-6)
        ...

Notice: I have no idea how this could affect the code so be cautious when trying

Then the code was successfully running. Since there is no results outputing function I can only monitor the process in debug mode, eventually reached an eval_return of around 1800 in around 370k steps.

Hope this could help!

Hi, thanks for your impressive work! I'm running OMAR as a baseline of my algorithm. When I run the code(with original parameters including cql_alpha and omar_coe), I find the results of MPE is consistent with the results in the paper, but the results of mamujoco is far from the data presented in the paper. I attach some results here:

And I tried many other methods including MACQL, TD3+BC. Their best results are all lower than 1000, which results in a normalized score less than 30. Can you figure out what's wrong with my results? I think this may come from the environment difference. I use the newest mamujoco 1.1.0, and mujoco version 2.1.0. Are these configurations consistent with the dataset? The way for calculating normalized score may be another reason. Would you tell me the way of calculating the score? Is it the same as MPE? Thanks again!

Dear ling-pan:

I am a PhD student from Tsinghua University. I am writing to see if you can share the dataset for the Cooperative Navigation. I cannot download the dataset from the google drive due to the requested access.

Looking forward to your reply!

Best,
Yiqin