Search Results for “lasso (statistics)”

Source: Lasso (statistics)

In statistics and machine learning, lasso (least absolute shrinkage and selection operator; also Lasso or LASSO) is a regression analysis method that performs both variable selection and regularization in order to enhance the prediction accuracy and interpretability of the resulting statistical model. The lasso method assumes that the coefficients of the linear model are sparse, meaning that few of them are non-zero. It was originally introduced in geophysics, and later by Robert Tibshirani, who coined the term.
Lasso was originally formulated for linear regression models. This simple case reveals a substantial amount about the estimator. These include its relationship to ridge regression and best subset selection and the connections between lasso coefficient estimates and so-called soft thresholding. It also reveals that (like standard linear regression) the coefficient estimates do not need to be unique if covariates are collinear.
Though originally defined for linear regression, lasso regularization is easily extended to other statistical models including generalized linear models, generalized estimating equations, proportional hazards models, and M-estimators. Lasso's ability to perform subset selection relies on the form of the constraint and has a variety of interpretations including in terms of geometry, Bayesian statistics and convex analysis.
The LASSO is closely related to basis pursuit denoising.

History

Lasso

lasso

Lasso

Basic form

= Least squares

lasso

= Orthonormal covariates

lasso

= Correlated covariates

lasso

General form

Lasso

lasso

Interpretations

= Geometric interpretation

Lasso

lasso

= Making λ easier to interpret with an accuracy-simplicity tradeoff

lasso

= Bayesian interpretation

lasso

= Convex relaxation interpretation

Lasso

lasso

Generalizations

Lasso

lasso

Lasso

= Elastic net

lasso

= Group lasso

lasso

LASSO

= Fused lasso

lasso

In contrast, variables can be clustered into highly correlated groups, and then a single representative covariate can be extracted from each cluster.
Algorithms exist that solve the fused lasso problem, and some generalizations of it. Algorithms can solve it exactly in a finite number of operations.

= Quasi-norms and bridge regression

=

Lasso, elastic net, group and fused lasso construct the penalty functions from the

ℓ

1

{\displaystyle \ell ^{1}}

and

ℓ

2

{\displaystyle \ell ^{2}}

norms (with weights, if necessary). The bridge regression utilises general

ℓ

p

{\displaystyle \ell ^{p}}

norms (

p
≥
1

{\displaystyle p\geq 1}

) and quasinorms (

0
<
p
<
1

{\displaystyle 0
). For example, for p=1/2 the analogue of lasso objective in the Lagrangian form is to solve

min

β
∈

R

p

{

1
N

‖

y
−
X
β

‖

2

2

+
λ

‖
β

‖

1

/

2

}

,

{\displaystyle \min _{\beta \in \mathbb {R} ^{p}}\left\{{\frac {1}{N}}\left\|y-X\beta \right\|_{2}^{2}+\lambda {\sqrt {\|\beta \|_{1/2}}}\right\},}

where

‖
β

‖

1

/

2

=

(

∑

j
=
1

p

|

β

j

|

)

2

{\displaystyle \|\beta \|_{1/2}={\biggl (}\sum _{j=1}^{p}{\sqrt {|\beta _{j}|}}{\biggr )}^{2}}

It is claimed that the fractional quasi-norms

ℓ

p

{\displaystyle \ell ^{p}}

(

0
<
p
<
1

{\displaystyle 0
) provide more meaningful results in data analysis both theoretically and empirically. The non-convexity of these quasi-norms complicates the optimization problem. To solve this problem, an expectation-minimization procedure is developed and implemented for minimization of function

min

β
∈

R

p

{

1
N

‖

y
−
X
β

‖

2

2

+
λ

∑

j
=
1

p

ϑ
(

β

j

2

)

}

,

{\displaystyle \min _{\beta \in \mathbb {R} ^{p}}\left\{{\frac {1}{N}}\left\|y-X\beta \right\|_{2}^{2}+\lambda \sum _{j=1}^{p}\vartheta (\beta _{j}^{2})\right\},}

where

ϑ
(
γ
)

{\displaystyle \vartheta (\gamma )}

is an arbitrary concave monotonically increasing function (for example,

ϑ
(
γ
)
=

γ

{\displaystyle \vartheta (\gamma )={\sqrt {\gamma }}}

gives the lasso penalty and

ϑ
(
γ
)
=

γ

1

/

4

{\displaystyle \vartheta (\gamma )=\gamma ^{1/4}}

gives the

ℓ

1

/

2

{\displaystyle \ell ^{1/2}}

penalty).
The efficient algorithm for minimization is based on piece-wise quadratic approximation of subquadratic growth (PQSQ).

= Adaptive lasso

=
The adaptive lasso was introduced by Zou in 2006 for linear regression and by Zhang and Lu in 2007 for proportional hazards regression.

= Prior lasso

=
The prior lasso was introduced for generalized linear models by Jiang et al. in 2016 to incorporate prior information, such as the importance of certain covariates. In prior lasso, such information is summarized into pseudo responses (called prior responses)

y
^

p

{\displaystyle {\hat {y}}^{\mathrm {p} }}

and then an additional criterion function is added to the usual objective function with a lasso penalty. Without loss of generality, in linear regression, the new objective function can be written as

min

β
∈

R

p

{

1
N

‖

y
−
X
β

‖

2

2

+

1
N

η

‖

y
^

p

−
X
β

‖

2

2

+
λ
‖
β

‖

1

}

,

{\displaystyle \min _{\beta \in \mathbb {R} ^{p}}\left\{{\frac {1}{N}}\left\|y-X\beta \right\|_{2}^{2}+{\frac {1}{N}}\eta \left\|{\hat {y}}^{\mathrm {p} }-X\beta \right\|_{2}^{2}+\lambda \|\beta \|_{1}\right\},}

which is equivalent to

min

β
∈

R

p

{

1
N

‖

y
~

−
X
β

‖

2

2

+

λ

1
+
η

‖
β

‖

1

}

,

{\displaystyle \min _{\beta \in \mathbb {R} ^{p}}\left\{{\frac {1}{N}}\left\|{\tilde {y}}-X\beta \right\|_{2}^{2}+{\frac {\lambda }{1+\eta }}\|\beta \|_{1}\right\},}

the usual lasso objective function with the responses

y

{\displaystyle y}

being replaced by a weighted average of the observed responses and the prior responses

y
~

=
(
y
+
η

y
^

p

)

/

(
1
+
η
)

{\displaystyle {\tilde {y}}=(y+\eta {\hat {y}}^{\mathrm {p} })/(1+\eta )}

(called the adjusted response values by the prior information).
In prior lasso, the parameter

η

{\displaystyle \eta }

is called a balancing parameter, in that it balances the relative importance of the data and the prior information. In the extreme case of

η
=
0

{\displaystyle \eta =0}

, prior lasso is reduced to lasso. If

η
=
∞

{\displaystyle \eta =\infty }

, prior lasso will solely rely on the prior information to fit the model. Furthermore, the balancing parameter

η

{\displaystyle \eta }

has another appealing interpretation: it controls the variance of

β

{\displaystyle \beta }

in its prior distribution from a Bayesian viewpoint.
Prior lasso is more efficient in parameter estimation and prediction (with a smaller estimation error and prediction error) when the prior information is of high quality, and is robust to the low quality prior information with a good choice of the balancing parameter

η

{\displaystyle \eta }

.

Computing lasso solutions

The loss function of the lasso is not differentiable, but a wide variety of techniques from convex analysis and optimization theory have been developed to compute the solutions path of the lasso. These include coordinate descent, subgradient methods, least-angle regression (LARS), and proximal gradient methods. Subgradient methods are the natural generalization of traditional methods such as gradient descent and stochastic gradient descent to the case in which the objective function is not differentiable at all points. LARS is a method that is closely tied to lasso models, and in many cases allows them to be fit efficiently, though it may not perform well in all circumstances. LARS generates complete solution paths. Proximal methods have become popular because of their flexibility and performance and are an area of active research. The choice of method will depend on the particular lasso variant, the data and the available resources. However, proximal methods generally perform well.
The "glmnet" package in R, where "glm" is a reference to "generalized linear models" and "net" refers to the "net" from "elastic net" provides an extremely efficient way to implement LASSO and some of its variants.
The "celer" package in Python provides a highly efficient solver for the Lasso problem, often outperforming traditional solvers like scikit-learn by up to 100 times in certain scenarios, particularly with high-dimensional datasets. This package leverages dual extrapolation techniques to achieve its performance gains. The celer package is available at GitHub.

Choice of regularization parameter

Choosing the regularization parameter (

λ

{\displaystyle \lambda }

) is a fundamental part of lasso. A good value is essential to the performance of lasso since it controls the strength of shrinkage and variable selection, which, in moderation can improve both prediction accuracy and interpretability. However, if the regularization becomes too strong, important variables may be omitted and coefficients may be shrunk excessively, which can harm both predictive capacity and inferencing. Cross-validation is often used to find the regularization parameter.
Information criteria such as the Bayesian information criterion (BIC) and the Akaike information criterion (AIC) might be preferable to cross-validation, because they are faster to compute and their performance is less volatile in small samples. An information criterion selects the estimator's regularization parameter by maximizing a model's in-sample accuracy while penalizing its effective number of parameters/degrees of freedom. Zou et al. proposed to measure the effective degrees of freedom by counting the number of parameters that deviate from zero. The degrees of freedom approach was considered flawed by Kaufman and Rosset and Janson et al., because a model's degrees of freedom might increase even when it is penalized harder by the regularization parameter. As an alternative, the relative simplicity measure defined above can be used to count the effective number of parameters. For the lasso, this measure is given by

P

^

=

∑

i
=
1

p

|

β

i

−

β

0
,
i

|

1
p

∑

l

|

b

OLS

,
l

−

β

0
,
l

|

,

{\displaystyle {\hat {\mathcal {P}}}=\sum _{i=1}^{p}{\frac {|\beta _{i}-\beta _{0,i}|}{{\frac {1}{p}}\sum _{l}|b_{{\text{OLS}},l}-\beta _{0,l}|}},}

which monotonically increases from zero to

p

{\displaystyle p}

as the regularization parameter decreases from

∞

{\displaystyle \infty }

to zero.

Selected applications

LASSO has been applied in economics and finance, and was found to improve prediction and to select sometimes neglected variables, for example in corporate bankruptcy prediction literature, or high growth firms prediction.