Why two separate tf.matmul operations in BNLSTMCell.__call__? about bnlstm HOT 5 CLOSED

The input and hidden states maybe different sizes, yes, but it would still be possible to have a single weight matrix of shape (x + h, 4h).

Note that the W_hh is initialized with the identity for a quarter of its weights. So that initializer would have to be a bit more complicated.

I remember them keeping the two batch norms separate in the paper, however by slicing the resulting matrix I think that would still be possible with your proposed optimization.

from bnlstm.

I see now that the pr by hardmaru actually concatenated the two weight matrices before the matmul for the vanilla lstm, thus keeping the initializers as they are. Same approach could work for the bnlstm.

from bnlstm.

Concatenating would be neat for performance then. 👍

Yeah, I saw that you're using the identity for the input gate. I guess that's the fixed initialization proposed by Tim Cooijman?

from bnlstm.

Thinking about this a little more. Actually, slicing would not work. There has to be two matrix multiplications if batch norm is supposed to be applied afterwards on both terms separately.

from bnlstm.

Uh, which kinda answers my own question. Closing. 🎉

from bnlstm.