amar-iastate / l2rpn-using-a3c Goto Github PK

Reinforcement Learning using the Actor-Critic framework for the L2RPN challenge (https://l2rpn.chalearn.org/ & https://competitions.codalab.org/competitions/22845#learn_the_details-overview). The agent trained using this code was one of the winners of the challenge. The code runs on the pypownet environment (https://github.com/MarvinLer/pypownet). It is released under a license of LGPLv3

License: GNU Lesser General Public License v3.0

MATLAB 20.23% Python 79.77%

keras reinforcement-learning a3c l2rpn power-grids codalab lgplv3

l2rpn-using-a3c's Introduction

L2RPN-using-A3C

Reinforcement Learning using the Actor-Critic framework for the L2RPN challenge (https://l2rpn.chalearn.org/ & https://competitions.codalab.org/competitions/22845#learn_the_details-overview). The agent trained using this code was one of the winners of the challenge. The code uses the pypownet environment (https://github.com/MarvinLer/pypownet). The code is released under a license of LGPLv3.

Requirements

Python >= 3.6
Keras
pypownet
Virtual Environment (conda/venv) Recommended

Pypownet Installation and Documentation : https://github.com/MarvinLer/pypownet

Explaination of Files

PDF Files
- Amar_L2RPN_IJCNN_git.pdf - Presentation of the method at IJCNN-2019 in the L2RPN workshop. Summarizes the idea beind the approach and the training methodology.
Numpy Files
- valid_actions_array_uniq.npz - matrix of valid unique actions
- valid_actions_masking_subid_perm.npz - matrix that maps the substation-Ids to the unique valid actions to be used for masking the output of the actor
Python Files
- valid_switching_controls.py - python file that creates the numpy files explained above
- pypow_14_a3c_final.py - python file used to train the actor & critic neural networks using A3C
Chronic Datasets in public_data folder
- datasets - Original chronics data given by the L2RPN contest
- datasets_sub_4 - Subsampled chronics from the original data by 4
- datasets_sub_7 - Subsampled chronics from original data by 7
- you can create other subsamples by modifying the value of the 'sub_sample' value in the matlab file create_sub_files.m

Usage

Training your own A3C model

python pypow_14_a3c_final.py

This will create two new files

pypow_14_a3c_actor.h5 - The weights of the actor neural network
pypow_14_a3c_critic.h5 - The weights of the critic neural network

Key Hyper-Parameter Tuning for Training

To speed up the learning, the enviornment difficulty level is slowly increased and the following hyper-parameters in the code can be used to make the environment difficult or easy

game_level_global - chooses subsampled data so that the agents can see data from farther in the dataset
game_over_mode_global - controls the behavior of the lines in the environment
chronic_loop_mode_global - controls how the envirnment 'Reset' function will behave

License information

This Source Code is subject to the terms of the GNU Lesser General Public License v3.0. If a copy of the LGPL-v3 was not distributed with this file, You can obtain one at https://www.gnu.org/licenses/lgpl-3.0.html.

l2rpn-using-a3c's People

Contributors

Stargazers

Watchers

Forkers

qhuang-pnl mzy2240 yang-new nlebang eexxyy happynewya ts132000 karlzhang-hhg ahmadsuleman gradwanderer

l2rpn-using-a3c's Issues

code is not working because "AttributeError: can't set attribute. "

my python is 3.8, tf2.0, I ran this code on my conda spyder called "pypow_14_a3c_final.py" and I got a problem: Using custom reward signal CustomRewardSignal of file F:L2RPN-using-A3C-master\public_data\reward_signal.py Traceback (most recent call last): File "F\L2RPN-using-A3C-master\pypow_14_a3c_final.py", line 310, in env = set_environement() File "F:...\L2RPN-using-A3C-master\pypow_14_a3c_final.py", line 43, in set_environement game_over_mode=game_over_mode) File "...\lib\site-packages\pypownet\environment.py", line 812, in init self.reset() File "C...\lib\site-packages\pypownet\environment.py", line 831, in reset self.game.n_timesteps_horizon_maintenance) File "...\lib\site-packages\pypownet\environment.py", line 374, in init self.shape = seek_shapes(self, ()) AttributeError: can't set attribute. I want to seek the reason , can you help me please?

issue about tensorflow

what's the type of tensorflow?1.0 or 2.0?

About the training

Hi，after several days‘ training，the scores I got are always negative. Could you tell me the possible reasons for this results？I set the training mode is easy and the datasets is sub_datasets_4. Thank you very much.

Error about tensorflow

when i train model, i'm running into some problems, the tensorflow version is 1.13 and the keras version is 2.3.1

Exception in thread Thread-1:
Traceback (most recent call last):
File "/home/nesa517/anaconda3/envs/L2RPN-using-A3C-master/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "pypow_14_a3c_final.py", line 195, in run
action = self.get_action(env,state)
File "pypow_14_a3c_final.py", line 281, in get_action
policy_nn = self.actor.predict(np.reshape(state, [1, self.state_size]))[0]
File "/home/nesa517/anaconda3/envs/L2RPN-using-A3C-master/lib/python3.7/site-packages/keras/engine/training.py", line 1462, in predict
callbacks=callbacks)
File "/home/nesa517/anaconda3/envs/L2RPN-using-A3C-master/lib/python3.7/site-packages/keras/engine/training_arrays.py", line 324, in predict_loop
batch_outs = f(ins_batch)
File "/home/nesa517/anaconda3/envs/L2RPN-using-A3C-master/lib/python3.7/site-packages/tensorflow/python/keras/backend.py", line 3073, in call
self._make_callable(feed_arrays, feed_symbols, symbol_vals, session)
File "/home/nesa517/anaconda3/envs/L2RPN-using-A3C-master/lib/python3.7/site-packages/tensorflow/python/keras/backend.py", line 3019, in _make_callable
callable_fn = session._make_callable_from_options(callable_opts)
File "/home/nesa517/anaconda3/envs/L2RPN-using-A3C-master/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1471, in _make_callable_from_options
return BaseSession._Callable(self, callable_options)
File "/home/nesa517/anaconda3/envs/L2RPN-using-A3C-master/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1425, in init
session._session, options_ptr, status)
File "/home/nesa517/anaconda3/envs/L2RPN-using-A3C-master/lib/python3.7/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Tensor input_1:0, specified in either feed_devices or fetch_devices was not found in the Graph

Exception ignored in: <function BaseSession._Callable.del at 0x7f4d05194a60>
Traceback (most recent call last):
File "/home/nesa517/anaconda3/envs/L2RPN-using-A3C-master/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1455, in del
self._session._session, self._handle, status)
File "/home/nesa517/anaconda3/envs/L2RPN-using-A3C-master/lib/python3.7/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: No such callable handle: 139963925418400