Target Dropout not scaled by keep_prob about targeted-dropout HOT 5 CLOSED

That's a good point, there will be some shift in this case and TD will likely stand to benefit from performing some sort of rescaling of the neurons that fall within the "queue".

Would be nice to see some experiments comparing the effect of this.

from targeted-dropout.

By the way I just applied at FOR.ai!

from targeted-dropout.

We do this because with normal dropout the intention is to replace the deleted neurons back into the network -- which changes the expected norm of the activation -- so we need to scale the neurons so that when the deleted ones are replaced the expected norm stays about the same.

In our case, we don't intend to return the deleted weights/neurons so there's no need to perform and rescaling.

from targeted-dropout.

Thank you!

What I noticed is that targeted dropout does not preserve the same expected value of weight/neuron at train and test time:

• during training you sort under a certain rule the neurons/weights and you perform dropout on the "queue",

• during test you sort under the same rule the neurons/weights but you drop deterministically the "queue",

so the average values for neurons/weights is not the same on train and test.

from targeted-dropout.

That's great to hear! Looking forward to reviewing your application.

from targeted-dropout.