1. Convex and strictly convex
Common used notations about convexity are convex and strictly convex. Their definitions are
Definition 1: [convex]: f(x) is said to be convex if one of the following holds ∀x,y
Definition 2: [strictly convex]: f(x) is said to be strictly convex if one of the following holds ∀x,y
And there exist two equivalent definitions:
Theorem 3. [first order condition(1)]: If
f(x)
is first differentiable, then
f(x)
is convex iff
∀x,y
This equivalence holds for strictly convex for > .
proof:
necessary: If
sufficient: If the first order condition is satisfied,
combining these two together, we get:
**Theorem 4. [first order condition(2)[monotone of
∇f(x)
]]:
f(x)
is convex iff
(∇f(x)−∇f(y))⋅(x−y)≥0
.
proof: necessary:
If
f(x)
is convex, then
∀x,y
, we have
adding these two equalities:
i.e.
sufficient:
Let g(t)=f(x+t(y−x)) . Then ∇g(x)=∇f(x+t(y−x))⋅(y−x)
so ∇g(t) is monotone increasing.
So
Theorem 5. [second order condition]: If
f(x)
is second differentiable, then
f(x)
is convex iff
∀x
This equivalence holds for strictly convex for > .
proof:
For simply, we firstly prove one variable function situation:
If
sufficient:
From h′′(x)≥0 and taylor expansion, we have
and from last theorem, we know h(x) is convex.
necessary:
∀x≤z≤y , we have z=λx+(1−λy) with λ=y−zy−x
So for
t1≤x≤z≤y≤t2
, we have
letting t1→x and t2→y , we have
So h′(x) is increasing → h′′(x)≥0 .
Now we prove for multivariable function. Let
g(t)=f(x+tℓ)
be one variable function.
sufficient:
From convexity of
f(x)
,
So g(t) is convex as a one variable function. Then
So
necessary:*
Let g(t)=f(x+t(y−x)) , then
So g(t) is convex.
Then
So f(x) is convex.
From the proof, we know that the convexity of a function on a convex set is one-dimensional fact.
Intuition:
- convex says a function is convex ≥ a linear function
- strictly convex says a function is convex > a linear function
2. Strong convex
Definition 3: [strong convex]:
Then from last section, we have that:
first order condition (1):
first order condition (2)[monotone of derivative]:
seconf order condition :
Intuition: strong convex says a function is convex ≥ a quadratic function.
Theorem: If a function is strong convex then the first derivative of it is Lipschitz continuous.
proof: Firstly, we claim that the subset
S={x,f(x)≤f(x(0))}
is closed. Since
∀y∈S
, we have
And the maximum eigenvalue of
∇2f(x)
is continuous, so there exists a upper bound
M
for