Pytorch 1.7.0 RuntimeError about deepctr-torch HOT 5 CLOSED

1.Cause

This is because pytorch 1.7 changes Tensor iteration behavior. When weight_list is a tensor or a Parameter, list(weight_list) returns list of UnbindBackward tensors, which further lead to inplace operation error.
In previous versions (1.2-1.6), list(weight_list) returns list of SelectBackward tensors and it works.
Related pytorch issue can be seen here: pytorch/pytorch#47899

Location:

In DeepCTR-Torch, this error happens in:

2.Solution

We fix this bug by simply remove list(weight_list) operation for <class 'torch.nn.parameter.Parameter'> input in a9402a4.

Take DeepFM as example, add_regularization_weight() are called 4 times:

list(weight_list) are used to convert generator and filter to a list of tensors. For a Parameter, we can directly compute its norm and there is no need to convert it to a list.

3.Reproduction

If you are still interested, I simplify this problem and reproduce it in the codes below (works in previous versions but fails in 1.7)

import torch

x = torch.tensor([[0.5, 0.2, 0.3, 0.8]], requires_grad=True)
w = torch.tensor([[0.2, 0.5, 0.1, 0.5]], requires_grad=True)
y_true = torch.tensor([1])

# In 1.7, `list(w)` returns list of UnbindBackward tensors, which further leads to inplace operation error.
# In previous versions, `list(w)` returns list of SelectBackward tensors and it works.
weight_list = list(w)
print('weight_list',weight_list) 

for i in range(2):
    l2_norm = torch.norm(weight_list[0], 2)
    y = w.mm(x.T)

    # mse with l2 regularization
    loss = pow(y - y_true, 2) + 0.5 * pow(l2_norm, 2)
    loss.backward()

    # simulate optimizer.step()
    with torch.no_grad():
        w.add_(w.grad, alpha=1e-3)

output:
torch 1.7:

weight_list [tensor([0.2000, 0.5000, 0.1000, 0.5000], grad_fn=<UnbindBackward>)]
...
RuntimeError: Output 0 of UnbindBackward is a view and its base or another view of its base has been modified inplace. This view is the output of a function that returns multiple views. Such functions do not allow the output views to be modified inplace. You should replace the inplace operation by an out-of-place one.

Process finished with exit code 1

previous versions:

weight_list [tensor([0.2000, 0.5000, 0.1000, 0.5000], grad_fn=<SelectBackward>)]

Process finished with exit code 0

Except the solution above, if it's necessary to so such operation, using indices works:

# weight_list = list(w)
weight_list = [w[i] for i in range(len(w))]

using indices returns SelectBackward tensors instead of UnbindBackward tensors.

from deepctr-torch.

Details can be found in pytorch/pytorch#47899 and the release note of pytorch 1.7 here.

Use torch.unbind instead of a for loop

from deepctr-torch.

Thanks

from deepctr-torch.

Hi, we have solved this issue in v0.2.4, please use pip install -U deepctr-torch to upgrade.

from deepctr-torch.

Hi, we have solved this issue in v0.2.4, please use pip install -U deepctr-torch to upgrade.

Thanks, I will try it later!

from deepctr-torch.