BN in cell upate about bnlstm HOT 1 OPEN

Hi there!

Or is term (7) the cell update they talk about?

I'm no expert, but as I understand it they don't apply BN in equation 7 so that the cell gradient keeps the properties of only being affected by the forget gate multiplication. I find the graphics by Colah shows it very clearly visually: http://colah.github.io/posts/2015-08-Understanding-LSTMs/

Or are both BN in term (6) input-hidden updates, and BN in term (8) hidden-hidden updates?

No, I understand it the same way you do. The two BNs being discussed in section 3 are both in equation 6. The BN in 8 isn't mentioned in the discussion. Could be we're both wrong tho :)

Further down in that section they discuss storing separate statistics for each time step, this isn't done in my code yet (as discussed in #2). Would be great to replicate their results using global per timestep statistics.

from bnlstm.