Consider the vector x⃗ =(1,ε)∈R2 where ε>0 is small. The l1 and l2 norms of x⃗ , respectively, are given by
Now say that, as part of some regularization procedure, we are going to reduce the magnitude of one of the elements of x⃗ by δ≤ε . If we change x1 to 1−δ , the resulting norms are
On the other hand, reducing x2 by δ gives norms
The thing to notice here is that, for an
l2
penalty, regularizing the larger term
x1
results in a much greater reduction in norm than doing so to the smaller term
x2≈0
. For the
l1
penalty, however, the reduction is the same. Thus, when penalizing a model using the
l2
norm, it is highly unlikely that anything will ever be set to zero, since the reduction in norm going from
ε
to
0
is almost nonexistent when
Another way to think of it: it’s not so much that l1 penalties encourage sparsity, but that l2 penalties in some sense discourage sparsity by yielding diminishing returns as elements are moved closer to zero.
an lα penalty for any α≤1 will also induce sparsity, but you see those less often in practice (probably because they’re non-convex). If you really just want sparsity then an l0 penalty (proportional to the number of non-zero entries) is the way to go, it just so happens that it’s a bit of a nightmare to work with.
Yes - that’s correct. There are many norms that lead to sparsity (e.g., as you mentioned, any Lp norm with p <= 1). In general, any norm with a sharp corner at zero induces sparsity. So, going back to the original question - the L1 norm induces sparsity by having a discontinuous gradient at zero (and any other penalty with this property will do so too)
Have a look on figure 3.11 (page 71) of The elements of statistical learning. It shows the position of a unconstrained β^ that minimizes the squared error function, the ellipses showing the levels of the square error function, and where are the β^ subject to constraints ℓ1(β^)<t and ℓ2(β^)<t .
This will allow you to understand very geometrically that subject to the ℓ1 constraint, you get some null components. This is basically because the ℓ1 ball {x:ℓ1(x)≤1} has “edges” on the axes.
More generally, this book is a good reference on this subject: both rigorous and well illustrated, great explanations.
I think your second paragraph is a key… at least for my intuition: an l1 “ball” is more like a diamond that’s spikey along the axes, which means that a hyperplane constrained to hit it is more likely to have a zero on the axes.
Yes, I use to imagine the optimization process as the movement of a point submitted to two forces : attraction towards the unconstrained β^ thanks to the squared error function, attraction towards 0 thaks to the ℓ1 or ℓ2 norm. Here, the “geometry” of this attraction force changes the behavior of the point. If you fix a small ℓ1 or ℓ2 ball in which it can freely move, it will slide on the border of the ball, in order to go near to β^ . The result is shown on the illustration in the aforementionned book.
With a sparse model, we think of a model where many of the weights are 0. Let us therefore reason about how L1-regularization is more likely to create 0-weights.
Consider a model consisting of the weights (w1,w2,…,wm) .
With L1 regularization, you penalize the model by a loss function L1(w) = Σi|wi| .
With L2-regularization, you penalize the model by a loss function L2(w) = 12Σiw2i
If using gradient descent, you will iteratively make the weights change in the opposite direction of the gradient with a step size η . Let us look at the gradients:
dL1(w)dw=sign(w) , where sign(w)=(w1|w1|,w1|w1|,…,w1|wm|)
dL2(w)dw=w
If we plot the loss function and it’s derivative for a model consisting of just a single parameter, it looks like this for L1:
Notice that for
L1
, the gradient is either 1 or -1, except for when
w1=0
. That means that L1-regularization will move any weight towards 0 with the same step size, regardless the weight’s value. In contrast, you can see that the
L2
gradient is linearly decreasing towards 0 as the weight goes towards 0. Therefore, L2-regularization will also move any weight towards 0, but it will take smaller and smaller steps as a weight approaches 0.
Try to imagine that you start with a model with
w1=5
and using
η=12
. In the following picture, you can see how gradient descent using L1-regularization makes 10 of the updates
w1:=w1−η⋅dL1(w)dw=w1−0.5⋅1
, until reaching a model with
w1=0
:
In constrast, with L2-regularization where
η=12
, the gradient is
w1
, causing every step to be only halfway towards 0. That is we make the update
w1:=w1−η⋅dL1(w)dw=w1−0.5⋅w1
Therefore, the model never reaches a weight of 0, regardless of how many steps we take:
Note that L2-regularization can make a weight reach zero if the step size
η
is so high that it reaches zero or beyond in a single step. However, the loss function will also consist of a term measuring the error of the model with the respect to the given weights, and that term will also affect the gradient and hence the change in weights. However, what is shown in this example is just how the two types of regularization contribute to a change in weights.