Support for single output networks (BCELoss and MSELoss) about backpack HOT 3 CLOSED

Hi Alex,

I rolled out support for BCEWithLogitsLoss, the binary equivalent of CrossEntropyLoss, in most extensions (#279, #280, #282, #283, #284). I was favoring BCEWithLogitsLoss over BCELoss because Fisher = GGN for the former.

Feel free to close this issue if you believe it to be solved.

from backpack.

Hi Alex, thanks for the report!

MSELoss not supporting vectors is annoying if it clashes with the standard pytorch API. I'll try to look into it.

There's no technical reason for BCELoss to be missing. It's not on the priority list as CrossEntropyLoss is more general. The difference in running time would not be noticable (compared to the rest of the model) and they would give the same output as it's a reparametrization (up to floating point precision).

from backpack.

To have BCELoss included, one would need to extend the loss function and provide the second-derivatives and some other quantities I guess? Maybe I might give it a shot sometime.
I understand why it doesn't make much sense to assign high priority to it as it though is indeed almost the same as softmax for two classes.

The MSELoss problem is not so bad and can simply be avoided by reshaping the labels from N-dimensional vector to a[N,1] matrix.

from backpack.