Rank Expansion & Training about rexnet HOT 3 CLOSED

Hello @steve2972! Sorry for the late reply.

For the first question, the implementation using simple addition is to take advantage of a simple implementation (in Pytorch) of shortcuts without any additional parameters. I have not tested the shortcut connection with a projection using 1x1 conv with BN but guess using projection shortcut may have a possibility to improve the accuracy due to extra parameters. Notice that the inverted bottleneck with the stride of 2 does not have a shortcut.

Next, after the depthwise convolution, a pointwise convolution follows:

Best regards,
Dongyoon

모델에 관심을 가져주셔서 감사합니다! 여기 issue나 제 개인 메일을 통해서 언제든지 궁금하신 사항 남겨주시면 답변 드리겠습니다.

감사합니다.

from rexnet.

Regarding your first question - while authors probably didn't experiment with this design choice, there is a paper which proposes to use partial residuals and shows that it has some nice properties in terms of gradient propagation - Partial Residual Networks

from rexnet.

@bonlime Thanks for the suggestion. Our model is an instance model to provide see how the rank expansion affects the model performance, so one can replace the basic modules with any components. I will put the partial residual module into the list of things to do in the future.

from rexnet.