Using a different loss function than MSE about lassonet HOT 9 CLOSED

Can you print the full error message? In particular give the full traceback so that we can know whether it's a dense model or not.

How did you implement quantile loss?

Does a simple MLP work correctly?

from lassonet.

The quantile loss is defined as follows:

def qloss(preds:torch.Tensor, target:torch.Tensor, quantile:float) -> torch.Tensor:
   error = target - preds.squeeze(1) # to make preds shape = (n, )
   return torch.max((quantile - 1) * error, quantile * error).sum()

Normal MLP Architecture (which is kind of replicating the residual network concept of lassonet):

class MLP(nn.Module):
  def __init__(self, input_dim, hidden_dim):
    super(MLP, self).__init__()
    self.layer1 = nn.Sequential(nn.Linear(input_dim, hidden_dim),
                                nn.ReLU())
    self.layer2 = nn.Linear(hidden_dim, 1)
    self.skip = nn.Linear(input_dim, 1)
      
  def forward(self, x):
    x_skip = self.skip(x)
    x = self.layer1(x)
    x = self.layer2(x)
    x = x + x_skip
    return x

Upon training with above architecture it returns satisfactory results with final quantile loss of 3.65 which is also cross checked with other models.

I am using lassonet in the following way:

from sklearn.preprocessing import StandardScaler
from lassonet import LassoNetRegressor

model = LassoNetRegressor(hidden_dims = (86,), val_size = 0, verbose = 2)
model.criterion = qloss
sc = StandardScaler()
X_train = torch.tensor(sc.fit_transform(X.T), dtype=torch.float32)
# X_train shape = (3953, 15), y_train.shape = (3953,)
y_train = torch.Tensor(y)

hist_obj = model.path(X_train, y_train)

After running for few epochs and for different lambdas it gave this error-

epoch: 13
loss: 2921739.5
epoch: 14
loss: 8718601.0
epoch: 15
loss: 11001912.0
epoch: 16
loss: 8222154.0
epoch: 0
loss: 8565492.0
epoch: 1
loss: 19698790.0
epoch: 2
loss: 8724698.0
Loss: 188280816.0
l2_regularization: 1392959.75
l2_regularization_skip: 46.82383728027344
Loss is 188280816.0
Did you normalize input?
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
 in ()
      4 y_train = torch.Tensor(y)
      5 
----> 6 hist_obj = model.path(X_train, y_train)

5 frames
/usr/local/lib/python3.9/dist-packages/lassonet/interfaces.py in path(self, X, y, X_val, y_val, lambda_seq, lambda_max, return_state_dicts, callback)
    470             if self.model.selected_count() == 0:
    471                 break
--> 472             last = self._train(
    473                 X_train,
    474                 y_train,

/usr/local/lib/python3.9/dist-packages/lassonet/interfaces.py in _train(self, X_train, y_train, X_val, y_val, batch_size, epochs, lambda_, optimizer, return_state_dict, patience)
    329                     return ans
    330 
--> 331                 optimizer.step(closure)
    332                 model.prox(lambda_=lambda_ * optimizer.param_groups[0]["lr"], M=self.M)
    333             print("epoch:", epoch)

/usr/local/lib/python3.9/dist-packages/torch/optim/optimizer.py in wrapper(*args, **kwargs)
    138                 profile_name = "Optimizer.step#{}.step".format(obj.__class__.__name__)
    139                 with torch.autograd.profiler.record_function(profile_name):
--> 140                     out = func(*args, **kwargs)
    141                     obj._optimizer_step_code()
    142                     return out

/usr/local/lib/python3.9/dist-packages/torch/optim/optimizer.py in _use_grad(self, *args, **kwargs)
     21         try:
     22             torch.set_grad_enabled(self.defaults['differentiable'])
---> 23             ret = func(self, *args, **kwargs)
     24         finally:
     25             torch.set_grad_enabled(prev_grad)

/usr/local/lib/python3.9/dist-packages/torch/optim/sgd.py in step(self, closure)
    128         if closure is not None:
    129             with torch.enable_grad():
--> 130                 loss = closure()
    131 
    132         for group in self.param_groups:

/usr/local/lib/python3.9/dist-packages/lassonet/interfaces.py in closure()
    324                             f"l2_regularization_skip: {model.l2_regularization_skip()}"
    325                         )
--> 326                         assert False
    327                     ans.backward()
    328                     loss += ans.item() * len(batch) / n_train

AssertionError:

from lassonet.

Your traceback is incomplete. Can you set verbose = 2? You should have some message "Initialized dense model" along with information about the dense model.

from lassonet.

Yeah so this is the information.

epoch: 927
loss: 29672.576171875
epoch: 928
loss: 29670.18359375
Initialized dense model
929 epochs, val_objective 2.97e+04, val_loss 2.97e+04, regularization 1.42e+00, l2_regularization 2.15e-01
epoch: 0
loss: 29667.796875
epoch: 1
loss: 6788969.5
Loss: 57149707255808.0
l2_regularization: 78822872.0
l2_regularization_skip: 6728.37109375
Loss is 57149707255808.0
Did you normalize input?
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
[](https://localhost:8080/#) in ()
      4 y_train = torch.Tensor(y)
      5 
----> 6 hist_obj = model.path(X_train, y_train)

5 frames
[/usr/local/lib/python3.9/dist-packages/lassonet/interfaces.py](https://localhost:8080/#) in path(self, X, y, X_val, y_val, lambda_seq, lambda_max, return_state_dicts, callback)
    470             if self.model.selected_count() == 0:
    471                 break
--> 472             last = self._train(
    473                 X_train,
    474                 y_train,

[/usr/local/lib/python3.9/dist-packages/lassonet/interfaces.py](https://localhost:8080/#) in _train(self, X_train, y_train, X_val, y_val, batch_size, epochs, lambda_, optimizer, return_state_dict, patience)
    329                     return ans
    330 
--> 331                 optimizer.step(closure)
    332                 model.prox(lambda_=lambda_ * optimizer.param_groups[0]["lr"], M=self.M)
    333             print("epoch:", epoch)

[/usr/local/lib/python3.9/dist-packages/torch/optim/optimizer.py](https://localhost:8080/#) in wrapper(*args, **kwargs)
    138                 profile_name = "Optimizer.step#{}.step".format(obj.__class__.__name__)
    139                 with torch.autograd.profiler.record_function(profile_name):
--> 140                     out = func(*args, **kwargs)
    141                     obj._optimizer_step_code()
    142                     return out

[/usr/local/lib/python3.9/dist-packages/torch/optim/optimizer.py](https://localhost:8080/#) in _use_grad(self, *args, **kwargs)
     21         try:
     22             torch.set_grad_enabled(self.defaults['differentiable'])
---> 23             ret = func(self, *args, **kwargs)
     24         finally:
     25             torch.set_grad_enabled(prev_grad)

[/usr/local/lib/python3.9/dist-packages/torch/optim/sgd.py](https://localhost:8080/#) in step(self, closure)
    128         if closure is not None:
    129             with torch.enable_grad():
--> 130                 loss = closure()
    131 
    132         for group in self.param_groups:

[/usr/local/lib/python3.9/dist-packages/lassonet/interfaces.py](https://localhost:8080/#) in closure()
    324                             f"l2_regularization_skip: {model.l2_regularization_skip()}"
    325                         )
--> 326                         assert False
    327                     ans.backward()
    328                     loss += ans.item() * len(batch) / n_train

AssertionError:

from lassonet.

It looks like your objective is already not good in the dense model.Initialized dense model: val_loss 2.97e+04. Maybe you need to start from there to understand what is wrong? Compare your training with our _train function.

from lassonet.

Okay I will look into the paper once again to get a better clarity of the internal procedures and also see my loss function and other stuffs. I will get back if there is anything more which can be discussed. Thank you for your help and time.

from lassonet.

My point is precisely that you don't need to dive deep in the paper. LassoNet starts the optimization from a dense model. This dense model seems to be badly fitted. Try to figure out why!

from lassonet.

Hi Louis,
I figured it out, probably there was a problem with the batch size. Initially the whole dataset was passed but when I switched to mini batches loss began to stabilize and converged at the end. I also obtained the path information in history object for various lambdas.

Thank you for your help. I guess this issue can be closed.

from lassonet.

Nice, I'm happy if I could help you!

from lassonet.