lternating Direction Method of Multiplier(ADMM) Algorithm

Alternating Direction Method of Multipliers (ADMM) 是一种通过将凸优化问题分解为一系列的易解子问题进行求解的算法，目前它在很多领域得到了广泛的应用。 [2].

This is simplified version, specifically for the LASSO:

给定一个稀疏向量$x\in R^n$和矩阵$A\in R^{m\times n}$，
$$y=Ax+e$$
其中$e$是加性高斯白噪声。为了恢复信号$x$，我们求解如下最小化问题
$$\hat{x}=\underset{x}{\min }||y-Ax|{|}_2^2+\lambda ||x|{|}_1$$

在求解过程中，迭代地计算如下两个式子，直到满足收敛条件。
$$x^{k+1}=(A^{T}A+\rho I{)}^{-1}(A^{T}y+\rho (z-u))$$
$$z^{k+1}=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}(\hat{x})\mathrm{m}\mathrm{a}\mathrm{x}\left(0,|x|-\frac{\lambda }{\rho }\right)$$

下面是ADMM算法的PYTHON实现方式。 (http://stanford.edu/~boyd/admm.html).

import numpy as np
import matplotlib.pyplot as plt
from math import sqrt, log

def Sthresh(x, gamma):
    return np.sign(x)*np.maximum(0, np.absolute(x)-gamma/2.0)

def ADMM(A, y):

    m, n = A.shape
    w, v = np.linalg.eig(A.T.dot(A))
    MAX_ITER = 10000

    # Function to caluculate min 1/2(y - Ax) + l||x||
    # via alternating direction methods
    xhat = np.zeros([n, 1])
    zhat = np.zeros([n, 1])
    u = np.zeros([n, 1])

    # Calculate regression co-efficient and stepsize
    lamb = sqrt(2*log(n, 10))
    rho = 1/(np.amax(np.absolute(w)))

    # Pre-compute to save some multiplications
    AtA = A.T.dot(A)
    Aty = A.T.dot(y)
    Q = AtA + rho*np.identity(n)
    Q = np.linalg.inv(Q)

    for i in np.arange(1, MAX_ITER + 1):

        # x minimisation step via posterier OLS
        xhat = Q.dot(Aty + rho*(zhat - u))

        # z minimisation via soft-thresholding
        zhat = Sthresh(xhat + u, lamb/rho)

        # mulitplier update
        u = u + xhat - zhat

    return zhat, rho, lamb

def test(m=50, n=200):
    """Test the ADMM method with randomly generated matrices and vectors"""
    A = np.random.randn(m, n)

    num_non_zeros = 10
    positions = np.random.randint(0, n, num_non_zeros)
    amplitudes = 100*np.random.randn(num_non_zeros, 1)
    x = np.zeros((n, 1))
    x[positions] = amplitudes

    y = A.dot(x) + np.random.randn(m, 1)

    xhat, rho, lamb = ADMM(A, y)

    plt.plot(x, label='Original')
    plt.plot(xhat, label = 'Estimate')

    plt.legend(loc = 'upper right')

    plt.show()


if __name__ == "__main__":
    test()

参考文献：
[1] https://codereview.stackexchange.com/questions/108263/alternating-direction-method-of-multipliers
[2] http://stanford.edu/~boyd/admm.html