@coco90417,
Unfortunately with the code provided above, we are unable to debug the issue. Could you please provide the complete standalone code and the tensorflow version you are using. Thank you!

from tf-keras.

Thank you for looking into this.

Below is what you could reproduce the issue.

import tensorflow as tf
import numpy as np
from tensorflow import keras

1000 observations; 20 predictors, 1 of predictors is redundant of all 0s

so you would be expecting weight for the first predictor to be 0 with L1 penalty

x_training = tf.concat(
[
tf.zeros([1000, 1]),
tf.random.normal([1000, 19], seed = 1),
], axis = 1)

y_training = tf.random.normal([1000], seed = 1)

from keras import models, layers, regularizers, optimizers

lasso regression with tensorflow

model = models.Sequential()
model.add(layers.InputLayer(input_shape = (20, )))
model.add(layers.Dense(units = 1,
kernel_regularizer = regularizers.L1(l1=1000)))
model.compile(optimizer = optimizers.Adam(),
loss = "mse")

fit = model.fit(x_training, y_training, epochs = 1000)

model.get_weights()

[array([[-1.15685923e-06],
[ 5.27249262e-05],
[-1.37503495e-04],
[-1.28477444e-04],
[-2.45119820e-06],
[ 1.45760217e-04],
[ 1.54898667e-04],
[ 2.10382195e-05],
[ 1.31973664e-06],
[-1.33556408e-04],
[ 4.12663186e-06],
[ 1.44984601e-05],
[-1.64269970e-04],
[-1.78692422e-04],
[-1.08664912e-04],
[ 2.58929811e-04],
[-5.15903737e-06],
[ 2.46357895e-05],
[ 7.39442373e-06],
[-1.24024616e-04]]),
array([0.00742478])]

The weight is not even small for predictor 1. The weight constraint was what I thought might address it, but would to hear about your recommended approach.

from tf-keras.

@coco90417, Unfortunately with the code provided above, we are unable to debug the issue. Could you please provide the complete standalone code and the tensorflow version you are using. Thank you!

Please see my response above. Much appreciated.

from tf-keras.

@tilakrayal Let me know how may I help on this issue.

from tf-keras.

Below is output from standard linear regression with L1 penalty, where we can observe shrinking of coefficient to strictly 0 in comparison.

linear regression

from sklearn import linear_model
lr = linear_model.Lasso(alpha=0)
lr.fit(x_training, y_training)
lr.coef_

Out[19]: array([ 0. , -0.05187305, -0.03030285, 0.01014469, 0.05644973,
-0.00163288, -0.01867641, -0.07605389, -0.03441314, 0.03904819,
-0.04583849, -0.02413755, 0.01884066, -0.01466035, 0.02474141,
0.02456712, 0.01741131, 0.0345059 , 0.01402795, 0.02839415])

lasso regression

lr_lasso = linear_model.Lasso(alpha=0.1)
lr_lasso.fit(x_training, y_training)
lr_lasso.coef_

Out[22]: array([ 0., -0., -0., 0., 0., -0., -0., -0., -0., 0., -0., -0., 0.,
-0., 0., 0., 0., 0., 0., 0.])

from tf-keras.

@sachinprasadhs, kindly follow up on this thread. Please let us know if there is anything we can do to help.

from tf-keras.

@coco90417,
The L1 penalty in vanilla form doesn't guarantee weights to be exactly zero. While it encourages sparsity by shrinking coefficients towards zero, some small, non-zero values can persist.

This behavior is a key characteristic of L1 regularization and has both advantages and disadvantages depending on the context. The weights not being exactly 0 is an optimisation artifact. If you do linear least squares with L1 regularisation and optimise it with gradient descent you will never reach exactly 0.

Also notice that for L1, the gradient (of the L1 term) is either 1 or -1, except for when w=0. That means that L1-regularization will move any weight towards 0 with the same step size, regardless of the weight’s value. In contrast, you can see that the L2 gradient is linearly decreasing towards 0 as the weight goes towards 0. Therefore, L2-regularization will also move any weight towards 0, but it will take smaller and smaller steps as a weight approaches 0. Thank you!

from tf-keras.

This issue is stale because it has been open for 14 days with no activity. It will be closed if no further activity occurs. Thank you.

from tf-keras.

This issue was closed because it has been inactive for 28 days. Please reopen if you'd like to work on this further.

from tf-keras.