arunpatala / residual.mnist Goto Github PK

Hello!

I saw your MNIST writeup here. Looks really cool!

I'm also trying to implement residual networks in torch with mixed success. It would be great to compare notes.

I'm curious: the call to convunit() always adds a ReLU layer to the end of each unit it constructs. This means that each resUnit is calculating ReLU( ReLU(f(x)) + x ). If I'm interpreting the architecture correctly, it looks like this:

Doing this means that the first term of the sum can never be negative. What that means is that x will never decrease as it propagates through the network. I think this differs from the original paper: if I read correctly, they don't add that second ReLU until after the building block.

Would this affect performance at all?

(I'm not sure how much of an issue this is. My experiments on CIFAR are pretty inconclusive...)

and this value is not used in this function, is it a neceissity here?

Hi,

How we can run this on the CPU ?