[Feature Request] Weight Normalization about deer HOT 6 CLOSED

Hi,

To introduce weight normalization (or any other form of normalization such as layer normalization or the usage of SeLu), you would need to modify the neural network (e.g. deer/q_networks/NN_keras.py if you use AC_net_keras). You can then also choose appropriate hyper-parameters that are well suited for the given neural network when you instantiate one of the learning algorithms (or even write your own modifications of the provided learning algorithms).

Hope this helps.

PS: the links you provided do not have the right url.

from deer.

I appreciate your quick response!

Honestly, I was hoping that applying weight normalization to the neural network model provided by NN_keras.py would only require to make changes to AC_net_keras.py in ll. 75-82 similar to the changes made in ll. 63-69 of cifar10_cnn.py in the aforementioned example. As you probably noticed, I'm new to Deep RL. Maybe you can point out the reason(s) for why one would have to change NN_keras.py in order to introduce weight normalization.

from deer.

Right, in the code you provided they have integrated the normalization in the optimizer instead of modifying the neural network definition. So it should be possible to directly take their optimizers and use it in the learning algorithm (replacing the simple SGD+momentum in AC_net_Keras as you pointed out)

from deer.

Thanks for the clarification.

Is normalization and weight normalization in particular something you would like to add to your library at this point? If it is not, you may, of course, feel free to close this issue. Otherwise maybe someone else or I will propose an appropriate implementation and post it here at some point.

Anyway, thanks for your help!

from deer.

We have to be sure that the benefit of adding this outweigh the additional complexity in the code and for the user. If it's only a matter of adding an update rule in AC_net_keras.py in ll. 75-82, then maybe ok. But I guess we would need to add the whole weight normalization code in this library...

from deer.

Yes, I thought about that, as well. Overall, Weight Normalization might not be worth the additional complexity. One would not only have to add the whole code by OpenAI but also find a solution for the data-dependent initialization. The latter would probably make Weight Normalization only applicable to certain environments or even leave the solution of the initialization problem completely to the user.

Therefore, I decided to go with Batch Normalization for now. The implementation for Keras models is very straightforward and training seems to improve. You are welcome to take a look at my implementation of Batch Normalization for NN_keras.py.

PS: I am not sure if feeding the "training=0" argument to the layer is correct but it seems to work.

from deer.