Performance on Pre-Activation ResNet about skip-connections-matter HOT 2 OPEN

Hi @S-Abdelnabi I also have the same question about how to change the backward hook to support Pre-Act ResNets. Do you or @csdongxian have any updates on this?

from skip-connections-matter.

Sincerely sorry for the late reply. Since I was busy graduating and switched the email, I missed the issue.

I still remember that I applied SGM to ResNet-v2 (PreAct ResNet in slim of TensorFlow) and it worked. For adaptation to PreAct ResNet, there are two points:

(1) Are there modules belonging to ReLU? Since the implementation in SGM first finds some modules with "relu" in their name. However, some implementation (e.g., this repo uses torch.nn.functional.relu directly, and we cannot find any module called "relu".

(2) The code in Line 33 in utils_sgm.py is to avoid the gradient vanishing, and it should be conducted in the trunk path (i.e., all gradients flow through the same path). For example, we could register the hook for BasicBlock / Bottleneck in ResNet or PreActBlock / PreActBottleneck module in PreAct ResNet. Unfortunately, the original code only supports ResNet we used. When we apply other architectures, it is better to modify the code.

from skip-connections-matter.