cts198859 / deeprl_signal_control Goto Github PK

When training large_grid

I want to change the phase and increase the number.

Which file should be modified at this time?

※ I modified the code as below.

1. large_grid.py

• line 20 : PHASE_NUM = 6
• line 40~41 : phases = ['GGrrrrGGrrrr', 'rrGrrrrrGrrr', 'rrrGGrrrrGGr', 'rrrrrGrrrrrG', 'rrrGGGrrrrrr', 'rrrrrrrrrGGG']

Is there any code I need to modify?

Thanks~

Hello

Thank you very much
I created a new folder called Results. And training

After 2 training -days, I got the results of the model. But I had errors when evaluating

python3 main.py --base-dir [base_dir] evaluate --agents [agents] --evaluation-seeds [seeds]

I have trouble with --agents [agents] . I used ia2c in config_ia2c_large.ini so I wrote

python3 main.py --base-dir Results evaluate --agents ia2c --evaluation-seeds [seeds]
Errors

Is there a PyTorch version？

Hello~

I have one question.
Is there any way to apply a3c?

Hi, I could run the trainnning of ma2c on large grid (for 5 days of trainning). But now, I want to evaluate it.
When I run this command:
python main.py --base-dir test_dir evaluate --agents ma2c --evaluate-seeds 3
I got the following error:

[ERROR] Evaluation: ma2c does not exist!

Of course, the folder ma2c really not exists in test_dir.
What have I done wrong? could you help please?

Hello?

I want to compare the difference between whether an intersection checks the state(also reward) of the neighborhood or not.

If I want to make a model that does not check the state of neighbors, can I delete all the neighbors list from the [large_grid_env.py] file after setting the ma2c model?

ex) neighbor_map['nt1'] = ['nt6','nt2'] →neighbor_map['nt1'] = []

Hello.

I have another question.

I want to give the traffic lights a minimum green duration.

How can I do it? Can you help me?

Hello.

Can I use multi-threading during training?

How can I enable multi-threading in main.py?

Hello author, may I ask why there is a sudden drop in reward for IA2C at 0.6M steps in the large grid?

Hi, it seems you update the target network every step along with the q network in IQL network. Shown as in the policies.py.

        with tf.variable_scope(self.name + '_q', reuse=True):
            q0s = self._build_net(self.S)
            q0 = tf.reduce_sum(q0s * A_sparse, axis=1)
        with tf.variable_scope(self.name + '_q', reuse=True):
            q1s = self._build_net(self.S1)
            q1 = tf.reduce_max(q1s, axis=1)
        tq = tf.stop_gradient(tf.where(self.DONE, self.R, self.R + gamma * q1))
        self.loss = tf.reduce_mean(tf.square(q0 - tq))

        wts = tf.trainable_variables(scope=self.name)
        grads = tf.gradients(self.loss, wts)
        if max_grad_norm > 0:
            grads, self.grad_norm = tf.clip_by_global_norm(grads, max_grad_norm)
        self.lr = tf.placeholder(tf.float32, [])
        self.optimizer = tf.train.AdamOptimizer(learning_rate=self.lr)
        self._train = self.optimizer.apply_gradients(list(zip(grads, wts)))

Should we try to update the target network every N(1000?) steps? Thanks!

I train the first model IA2C, another error is occurs, can you help me check it? my sumo version is 1.3.0, should I change the version of sumo?

Traceback (most recent call last):
  File "main.py", line 232, in <module>
    train(args)
  File "main.py", line 131, in train
    trainer.run()
  File "/project/wangmin/Multi-Agent-master/utils.py", line 285, in run
    ob, done, R, cur_rewards = self.explore(ob, done)
  File "/project/wangmin/Multi-Agent-master/utils.py", line 160, in explore
    next_ob, reward, done, global_reward = self.env.step(action)
  File "/project/wangmin/Multi-Agent-master/envs/env.py", line 584, in step
    self._simulate(rest_interval_sec)
  File "/project/wangmin/Multi-Agent-master/envs/env.py", line 474, in _simulate
    self.sim.simulationStep()
  File "/home/wangmin/project/sumo-1_3_0/tools/traci/connection.py", line 319, in simulationStep
    result = self._sendExact()
  File "/home/wangmin/project/sumo-1_3_0/tools/traci/connection.py", line 98, in _sendExact
    raise FatalTraCIError("connection closed by SUMO")
traci.exceptions.FatalTraCIError: connection closed by SUMO

Hi~.

There was a problem while training.

When I write the code, I get the following error. How do I fix it?

code : "python3 main.py --base-dir large_grid/ma2c train --config-dir config/config_ma2c_large.ini --test-mode no_test"
error : "Could not connect to TraCI server at localhost:8000 [Errno 111] Connection refused."

Thank you.

Hi, I try to run the deeprl_signal_control. But when I am in the step to run the build_file.py, I have this following error:

Loading configuration... done.
Error: Invalid edge id 'e:np1,nt1'.
Error: The connection-source edge 'e:np1,nt1' is not known.
Error: The connection-source edge 'e:np1,nt1' is not known.
Error: The connection-source edge 'e:np1,nt1' is not known.
Error: The connection-source edge 'e:np2,nt1' is not known.
Error: The connection-source edge 'e:np2,nt1' is not known.
Error: The connection-source edge 'e:np2,nt1' is not known.
Error: The connection-source edge 'e:np3,nt1' is not known.
Error: The connection-source edge 'e:np3,nt1' is not known.
Error: The connection-source edge 'e:np3,nt1' is not known.
Error: The connection-source edge 'e:np8,nt4' is not known.
Error: The connection-source edge 'e:np8,nt4' is not known.
Error: The connection-source edge 'e:np9,nt4' is not known.
Error: The connection-source edge 'e:np9,nt4' is not known.
Error: The connection-source edge 'e:nt1,npc' is not known.
Error: The connection-source edge 'e:nt1,npc' is not known.
Error: The connection-source edge 'e:nt1,nt2' is not known.
Error: The connection-source edge 'e:nt1,nt2' is not known.
Error: The connection-source edge 'e:nt3,nt2' is not known.
Error: The connection-source edge 'e:nt3,nt2' is not known.
Error: The connection-source edge 'e:nt1,nt6' is not known.
Error: The connection-source edge 'e:nt1,nt6' is not known.
Error: The connection-source edge 'e:nt5,nt6' is not known.
Error: The connection-source edge 'e:nt5,nt6' is not known.
Error: The connection-source edge 'e:npc,nt3' is not known.
Error: The connection-source edge 'e:npc,nt3' is not known.
Error: The connection-source edge 'e:nt4,nt3' is not known.
Error: The connection-source edge 'e:nt4,nt3' is not known.
Error: The connection-source edge 'e:npc,nt5' is not known.
Error: The connection-source edge 'e:npc,nt5' is not known.
Error: The connection-source edge 'e:nt4,nt5' is not known.
Error: The connection-source edge 'e:nt4,nt5' is not known.
Error: No edges loaded.
Quitting (on error).
Error: The network file 'exp.net.xml' is not accessible.
Quitting (on error).
Could anyone help please?

When I was training the agent, I observed that the process only used 1 core. Is multithreading supported in this code?

when I run "main.py train", the Error happen!
Traceback (most recent call last):
File "main.py", line 216, in
train(args)
File "main.py", line 88, in train
env = init_env(config['ENV_CONFIG'])
File "main.py", line 64, in init_env
return RealNetEnv(config, port=port)
File "/home/bruce/deeprl_signal_control-master/envs/real_net_env.py", line 117, in init
super().init(config, output_path, is_record, record_stat, port=port)
File "/home/bruce/deeprl_signal_control-master/envs/env.py", line 105, in init
self._init_sim(self.seed)
File "/home/bruce/deeprl_signal_control-master/envs/env.py", line 288, in _init_sim
subprocess.Popen(command)
File "/home/bruce/anaconda3/lib/python3.6/subprocess.py", line 709, in init
restore_signals, start_new_session)
File "/home/bruce/anaconda3/lib/python3.6/subprocess.py", line 1344, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'sumoD': 'sumoD'

I'm very sad, who can do me a favor?

In my computer, I run the experiment in 1600 episode and finished in four days. It spend a lot of time for me to collect data.

So, I wonder if there is any way to speed up the experiment. And is it useful to change hyperemeters like episode_length_sec and control_interval_sec?

I test ma2c model, and I only get one evaluation data eva_data\large_grid_ma2c_trip.xml file. I cannot get data like time-series data from this file. I want to know how to generate the data like your eva_data file (control.csv, traffic.csv, trip.csv).

Thanks!!

File "C:\Users\chenvy\Anaconda3\envs\chenvy\lib\configparser.py", line 959, in getitem
raise KeyError(key)
KeyError: 'ENV_CONFIG'

Hello,

Is there a simple way to compare the different models of the repo with the SUMO automatically generated TLS program?
If not, have you done it and how does it look?

I run the train.py using the large_grid and sumo version 1.3.1 but I got the following error :

Error: tcpip::Socket::accept() Unable to create listening socket: Address already in use
Quitting (on error).

Does anyone know how to avoid this?

Hi
There was a problem while training.

When I write the code, I get the following error. How do I fix it?

!python3 main.py --base-dir IA2C train --config-dir config/config_ia2c_large.ini --test-mode no_test
error :
Loading configuration ... done.
Error: File './large_grid/data/exp.net.xml' is not accessible (No such file or directory).
Quitting (on error).
Traceback (most recent call last):
File "main.py", line 229, in
train(args)
File "main.py", line 93, in train
env = init_env(config['ENV_CONFIG'])
File "main.py", line 61, in init_env
return LargeGridEnv(config, port=port)
File "/content/drive/My Drive/traffic_signal_controll/deeprl_signal_control/envs/large_grid_env.py", line 68, in init
super().init(config, output_path, is_record, record_stat, port=port)
File "/content/drive/My Drive/traffic_signal_controll/deeprl_signal_control/envs/env.py", line 109, in init
self._init_nodes()
File "/content/drive/My Drive/traffic_signal_controll/deeprl_signal_control/envs/env.py", line 209, in _init_nodes
for node_name in self.sim.trafficlight.getIDList():
File "/usr/local/lib/python3.7/dist-packages/traci/domain.py", line 196, in getIDList
return self._getUniversal(tc.TRACI_ID_LIST, "")
File "/usr/local/lib/python3.7/dist-packages/traci/domain.py", line 172, in _getUniversal
return _parse(self._retValFunc, varID, self._getCmd(varID, objectID, format, *values))
File "/usr/local/lib/python3.7/dist-packages/traci/domain.py", line 177, in _getCmd
r = self._connection._sendCmd(self._cmdGetID, varID, objID, format, *values)
File "/usr/local/lib/python3.7/dist-packages/traci/connection.py", line 181, in _sendCmd
return self._sendExact()
File "/usr/local/lib/python3.7/dist-packages/traci/connection.py", line 91, in _sendExact
raise FatalTraCIError("connection closed by SUMO")
traci.exceptions.FatalTraCIError: connection closed by SUMO

我没太看懂你们的配置，

default_base_dir = '/Users/tchu/Documents/rl_test/signal_control_results/eval_dec16/real_net’
default_config_dir = './config/config_test_large.ini'
修改成
default_base_dir = '/Users/thnx1/Desktop/硕士论文/code/deeprl_signal_control-master/control_results/real_net'
default_config_dir = '/Users/thnx1/Desktop/硕士论文/code/deeprl_signal_control-master/config/config_test_large.ini'
可以吗？

Hello.

How many steps does the neural network's lstm look past?

I just train the first model IA2C, it occurs a error, can you help me?

Traceback (most recent call last):
  File "main.py", line 230, in <module>
    train(args)
  File "main.py", line 129, in train
    trainer.run()
  File "/project/wangmin/Multi-Agent-master/utils.py", line 278, in run
    ob = self.env.reset()
  File "/project/wangmin/Multi-Agent-master/envs/env.py", line 562, in reset
    self._init_sim(seed, gui=gui)
  File "/project/wangmin/Multi-Agent-master/envs/env.py", line 301, in _init_sim
    subprocess.Popen(command)
  File "/software/google/tensorflow/python3.6-cpu/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/software/google/tensorflow/python3.6-cpu/lib/python3.6/subprocess.py", line 1295, in _execute_child
    restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory

Hi, It's still me. I want to know if there is an option to display the scenario when we have trained the agents to compare visually the difference between different scenario implemented.
Thanks.

Hi, I now want to try the real net. But when I run the buil_file.py I got the following error:
ModuleNotFoundError: No module named 'envs'
But when I append the path of the envs, I got another error so could you help please?
Another question, isn't there a config in small grid for ma2c ?

Hello.

I have a question about using [distance_map] in [ma2c].

If you look at line 610 of [env.py], you don't seem to be using [distance_map].

Is there any reason not to use [distance_map]?

It seems that [distance_map] is used in [Equation (9)] in the paper(preprint version).
(https://arxiv.org/pdf/1903.04527.pdf)

And how can I use [distance_map]?

Thank you.

Hello, thank you very much for your paper (Multi-Agent Deep Reinforcement Learning for Large-Scale Traffic Signal Control) and let me learn a lot.
After reading the experiment, I downloaded your code on git and ran it. In the results of my run, the reward of 5x5 is not good as the plot of this paper. Is this situation caused by parameters or seed?
And I test the real_net, the reward is about twice of the ma2c_real_train_reward.csv in your git-project. Is this situation caused by flow_rate ?
I'm looking forward to your reply.
Best wishes,

Hi, I am wondering if you could provide the config file for training the model under the small grid scenario as well. I tried to write a config file on my own but somehow the training keeps running into problems.

Hi,

I've gone through deepril it's very interesting and related to my research what I'm doing now.
So I tried to run the deepRL code in my local machine. I've installed Sumo and required python and tensorflow as mentioned.
But the code is throwing errors, i've used config_test_large.ini. I'm running directly in my windows machine, through pycharm.

Now i'm with the error in input and output shapes,
ValueError: Shape of a new variable (lr_0a_q/q/w) must be fully defined, but instead was (?, 5).

Could you please help me? I might be missing some of the configurations or may be the way I'm running the code is not right.

default_base_dir = '/Users/tchu/Documents/rl_test/signal_control_results/eval_sep2019/large_grid'这个文件代码中有吗？第一步就卡在了base-dir上面

Sorry, I would like to ask for some advice。
In each step, the AC algorithm is used to obtain the probability of the latest 5 phases of the neighbor. Why remove the probability of the last phase?
code：
def update_fingerprint(self, policy):
for node_name, pi in zip(self.node_names, policy):
self.nodes[node_name].fingerprint = np.array(pi)[:-1]

I am training the ma2c model in the real net scenario, but the training reward curve just start at -14 or so, then with the increase of training time the average episode reward converges to -9 or so, enventually. But the the reward curve of the original paper changes from -125 converge to -30 or so. I want to know the reason about that, may be I have a wrong flow-config? Could you give me any advice?

Hello
I am a traffic engineer. Now I am studying DRL with Sumo.
After downloading the repo, I try to run it but I can't.

python3 main.py --base-dir IA2C train --config-dir config/config_ia2c_large.ini --test-mode no_test