deshpandeshrinath / deepdgp Goto Github PK

Hello.

In the file ddpg.py, there's a line that reads the following (line 154 in the current version):

self.actor_loss = -tf.reduce_mean(self.predicted_Q_on_predicted_current_action)

The self.actor_loss variable is used in a couple of lines down, that reads (line 159) the following:

self.actor_grads = tf.gradients(self.actor_loss, self.actor.trainable_vars)
actor_grads_and_vars = zip(self.actor_grads, self.actor.trainable_vars)
self.actor_train_op = tf.train.AdamOptimizer(learning_rate=self.lr_act).apply_gradients(actor_grads_and_vars)

These lines basically describe the process of updating the actor network. However, I am particularly concerned about the line where the self.actor_loss is defined:

self.actor_loss = -tf.reduce_mean(self.predicted_Q_on_predicted_current_action)

Here, a series of state-action pairs are used to calculate the Q-values, and then they are reduced to a scalar mean value by the tf.reduce_mean() operation. This is then fed into the tf.gradients operation in the next stage.

Looking at other implementations of DDPG using Tensorflow and Keras, and the DDPG paper, i think the tf.reduce_mean() operation in this graph is kind of odd. Could you please explain why you decided to implement it in this way?

By the way, the code works very well, I'm just curious to know why you decided to implement it in this way.

Thanks in advance.