python - GLMNET gives different solutions for lasso-based algorithms when I expect the same solution - Stack Overflow

admin管理员组

文章数量:1399992

There are many ways to implement an adaptive lasso model. One of them would be to:

1. Solve an unpenalized regression model and extract the betas
2. Define w=1/abs(beta)
3. Use these weights in an adaptive lasso model and obtain beta2

Another approach would be to repeat step 1 and then do:

1. X2 = X*beta (this being column-wise product)
2. Solve a lasso model using X2 and obtain beta3
3. Re-scale the obtained coefficients as beta3 = beta3 * beta

These two approaches are mathematically equivalent, and I have verified using python libraries like asgl that both approaches produce the same solution. However, if I try to compare them using glmnet I obtain very different results.

My questions are: why? how can I obtain the same solutions? I provide here the python implementation (which produces the same solution with both approaches) and the R implementation using glment (that dont)

Python implementation

from sklearn.datasets import make_regression
from asgl import Regressor
import numpy as np

X, y = make_regression(n_samples=1000, n_features=10, n_informative=5, bias=10, noise=5, random_state=42)

model = Regressor(model='lm', penalization=None)
model.fit(X, y)

beta = model.coef_

X2 = X * beta
model = Regressor(model='lm', penalization='lasso', lambda1=0.1)
model.fit(X2, y)
beta2 = model.coef_ * beta

model = Regressor(model='lm', penalization='alasso', lambda1=0.1, individual_weights=1/np.abs(beta))
model.fit(X, y)
beta3 = model.coef_

# beta2 and beta3 are very similar

R implementation

library(glmnet)

set.seed(42)
n <- 1000  # Number of samples
p <- 10    # Number of features

# Generate synthetic data
X <- matrix(rnorm(n * p), nrow = n)
true_coef <- rep(0, p)
true_coef[1:5] <- rnorm(5)
y <- X %*% true_coef + 10 + rnorm(n, sd = 5) 

# Step 1: fit the model you want to use for your weight estimation
lm_model <- lm(y ~ X)
beta <- coef(lm_model)[-1]  # Remove intercept

# Step 2: Create X2 by multiplying X with beta
X2 <- sweep(X, 2, beta, "*") 

lambda1 = 0.1

# Step 3: Fit lasso regression on X2
lasso_model <- glmnet(X2, y, alpha = 1, lambda = lambda1, standardize = FALSE)
beta2 <- as.vector(coef(lasso_model, s = lambda1)[-1])
# Multiply back by beta 
beta2 <- unname(beta2 * beta)

# Step 4: Fit adaptive lasso on X
weights <- 1/(abs(beta)+1e-8)
# Fit adaptive lasso using penalty.factor for the weights
adaptive_lasso_model <- glmnet(X, y, alpha = 1, lambda = lambda1, penalty.factor = weights, standardize = FALSE)
beta3 <- as.vector(coef(adaptive_lasso_model, s = lambda1)[-1])

# beta2 and beta3 are very different 

本文标签：