Question about .layer_index in init_weights about boilerplate HOT 2 OPEN

Excellent question; "self.layer_index" is actually the number of layers in the network overall. This is only called "layer_index" and not "num_layers" or "L" because I quickly adopted the init() code from my earlier implementation, which used b.layer_index to calculate the mult coefficient for each block's init.

I had earlier thought (and had confirmed by the authors, although this was a communication mistake) that the coefficient for each layer was based on the depth of that layer (b.layer_index), rather than the overall depth of the network (self.layer_index).

You might be able to make a case that since each layer is at the end of a b.layer_index stack, this is still a workable interpretation, but this was not the intent of the authors (or how they implemented it), even if it does still train.

from boilerplate.

Thanks for the reply.

I had another look at the paper and the paragraph just below equation 5 contains an expression that does not depend on the branch number (i), but only on the total number of branches (m).
This did surprise me a bit since it would make intuitive sense to me that the optimal equation would include something about the position in the network.

You can close the issue now, this was an interesting chat!

from boilerplate.