Understanding the ADMM algorithm

版权声明：本文为博主原创文章，未经博主允许不得转载。https://blog.csdn.net/weixin_42444045/article/details/83512560

ADMM

This chapter explains the procedure of Alternative Direction Methods for Multipliers (ADMM), starting from the fundamental knowledge about the optimization problem.

1. What is optimization problem?

The following function represents a most common and simple optimization problem:
$$mi{n}_{x}f(x)----(1)$$

The x represents the variable needs to be optimized, i.e. by changing the value of x to make our objective function $f(x)$ achieve the minimum value.

In above equation, if we have only this function without any constrains to variable x, this problem is a simplest optimization problem, but in many circumstances, there will be many constrains to optimization variable x such as x needs to meet equality or inequality constraints, or to satisfy a set of numbers, e.g. $s.t.Ax=b,Ax>c$. So an optimization problem with equality constraints might look like this:
$$mi{n}_{x}f(x)----(2)$$ $$s.t.Ax=b$$

2. Dual Gradient Ascent

The alternating direction method of multipliers (ADMM) is an algorithm that solves convex optimization problems by breaking them into smaller pieces, each of which are then easier to handle.

Consider the question (2), $x\in {\mathbb{R}}^n,A\in {\mathbb{R}}^{m\ast n},f:{\mathbb{R}}^n\in R$ is a convex function, the Lagrangian function is :
$$L(x,y)=f(x)+y^T(Ax-b)----(3)$$
The dual function of (3) is:
$$g(y)=in{f}_{x}L(x,y)=-f^{\ast }(-A^{T}y)-b^{T}y----(4)$$
Where the $y$ is Lagrangian multiplier and dual variable, $f\ast$ is the conjugate function of $f$.

Assume a function satisfies strong duality property, then the optimal solution for original problem is same as the optimal solution for dual problem. If the optimal solution for original problem is $x\ast$, the optimal solution for dual problem is $y\ast$, then:
$$x^{\ast }=argmi{n}_{x}L(x,y^{\ast })----(5)$$
We use gradient ascend to solve dual problem, the update of dual gradient ascent is:
$$x^{k+1}=argmi{n}_{x}L(x,y^k)$$$$y^{k+1}=y^k+a_k(A{x}^k+1-b)----(6)$$
Where $a_k>0$ is the step size of the gradient ascent.

Assume the objective function is decomposable, then:
$$f(x)=\sum _{i=1}^n{f}_i(x_i)----(7)$$
Where , $x=(x_1,x_2,...,x_n),x_i\in {\mathbb{R}}^n$, split matrix A:
$$A_x=\sum _{i=1}^n{A}_i{x}_i----(8)$$
then,
$$A_x=\sum {i=1}^n{A}_i{x}_i----(9)$$
The reconstructed Lagrangian function is :
$$L(x,y)=\sum _{i=1}^n{L}_i(x_i,y)=\sum _{i=1}^n(f_i(x_i)+y^T{A}_i{x}_i-(1/N)y^{T}b)----(10)$$
The update for dual gradient ascent:
$$x_i^{k+1}=argmi{n}_{x_i}{L}_i(x_i,y^k)$$$$y^{k+1}=y^k+a^k(A{x}^{k+1}-b)----(11)$$

3. Augmented Lagrangian Method

In order to increase the robustness of dual ascent method and relax constrains of strong convexity, we introduce the augmented Lagrangian method as follow:
$$L_{\rho }(x,y)=f(x)+y^T(Ax-b)+(\rho /2)||Ax-b|{|}_2^2----(12)$$
Where is penalty parameter. Augmented Lagrangian method is actually adding a penalty term to reconstruct the original problem into:
$$mi{n}_{x}f(x)+(\rho /2)||Ax-b|{|}_2^2$$$$s.t.Ax=b----(13)$$
Dual ascent update:
$$x^{k+1}=argmi{n}_{x}L(x,y^k)$$
$$y^{k+1}=y^k+\rho (A{x}^k-b)----(14)$$

4.ADMM

The method in section 3 uses augmented Lagrangian method to update and the step size is now ρ which replaces , this is called multiplier method, it can make function be more convergence than normal dual ascent method, but the addition of second penalty term disabled the decomposable property of original problem, to address this problem, alternative direction method for multipliers (ADMM) is introduced.

For optimization problem:
$$mi{n}_{x}f(x)+g(z)----(15)$$$$s.t.Ax+Bz=c$$
$x\in {\mathbb{R}}^n,z\in {\mathbb{R}}^m,A\in {\mathbb{R}}^{p\ast n},B\in {\mathbb{R}}^{p\ast m},c\in {\mathbb{R}}^p$, the augmented Lagrangian function is:
$$L_p(x,z,y)=f(x)+g(z)+y^T(Ax+Bz-c)+(\rho /2)||Ax+Bz-c|{|}_2^2----(16)$$
The update of variables by using ADMM is:
$$x^{k+1}=argmi{n}_x{L}_{\rho }(x^{k+1},z,y^k)$$$$z^{k+1}=argmi{n}_x{L}_{\rho }(x^{k+1},z,y^k)$$$$y^{k+1}=y^k+\rho (A{x}^{k+1}-B{z}^{k+1}-c)$$
With $\rho >0$, the basic original iteration structure of ADMM method is presented.