AttributeError: 'BatchNorm2d' object has no attribute 'output' about backpack HOT 4 OPEN

Hi Qiyao,

Beware of BatchNorm; most of the quantities returned by BackPACK are not defined when there's a batchnorm layer in the middle (see e.g. #239).

Easy checks of things that can cause something like this to happen would be calling backward twice (where the first backward clears the graph, and the second backward then crashes), and maybe missing a call to backpack.extend(model). But this doesn't seem to be the case here.

Only a rough guess looking at the stack, but the error might be specific to BatchNorm.
The error occurs after the computation of the backward pass, during cleanup.
delete_old_quantities = not self.__should_retain_backproped_quantities(module). The error 'BatchNorm2d' object has no attribute 'output' indicates that the extension needed to store additional quantities during the forward pass (the output of the layer) but did not. This is weird; I would expect it to crash much earlier. What extension are you running with batchnorm?

from backpack.

1. hmmm, is there currently an alternative to BatchNorm? I guess it would be safest to just stick to linear and conv layers + activations, although the accuracy will for sure decrease in that case
2. Yeah I don't think I am calling backward twice, and I made sure to add model = extend(model). BTW, the documentation page also recommended trying use_converter=True, but I guess that one has its own bugs so I did not dig deeper.
3. Even though I don't have the full MWE ready, the error code is easy to share

model = get_cls_net()
model = extend(model)
with backpack(BatchGrad()):
    model(torch.rand(1,3,32,32)).sum().backward()

The weird thing is, BatchNorm worked with this code when I was trying it on a smaller model, so what I am doing right now is trying to sort out the structural differences between these two models and see if I can find anything useful

from backpack.

is there currently an alternative to BatchNorm?

There are, for example GroupNorm or LayerNorm (see https://pytorch.org/docs/stable/nn.html#normalization-layers).
The problem with BatchNorm is that there are no "individual gradient"; it is not possible to isolate the contribution of one sample to the loss because BatchNorm mixes them.

What's the model (get_cls_net)?

from backpack.

Oh I thought backpack doesn't support GroupNorm

BTW I might have figured out the issue, it goes away when I do add an eval like: extend(model).eval(). Not sure why but I guess that is a fix!

from backpack.