Proper CUDNN RNN initialisation about speechbrain HOT 6 CLOSED

I think allowing initializations of CNNs in CRDNN. Since the rectified activation units at the CNN blocks produce unbounded outputs, bad initializations could lead to very large or very small variance on its output distribution. This could be problematic when one is building a very deep network...

from speechbrain.

We can do it. Actually, according to my experience initialization doesn't play a crucial role. For instance, in the current RNN class we already added orthogonal init for recurrent connections and the performance was the same. What really makes a big difference is batch normalization and recurrent droput (which cannot be added properly to the cudnn models). Feel free to ask for a pull request that changes the RNN class if there is evidence of some performance improvements...

from speechbrain.

It's not much a matter of performance but a matter of trainability. A wrong initialisation scheme might just makes the convergence impossible, especially with a larger number of neutrons. Indeed, because of the summation, you'll end up in the dead spot of the TANH function, and then ... well you won't converge. @jjery2243542 seems to have this problem with 2048 neutrons per LSTM layer.

from speechbrain.

from speechbrain.

But if using batch norm it will not be the case I think. We can still do it.

from speechbrain.

I did rnn initialization in #110

from speechbrain.