## 对偶上升方法

$\underset{x}{min}f\left(x\right)$
$st.Ax=b$

$L\left(x,y\right)=f\left(x\right)+{y}^{T}\left(Ax-b\right)$

${x}^{\ast }=argmi{n}_{x}L\left(x,{y}^{\ast }\right)$

${x}_{k+1}=argmi{n}_{x}L\left(x,{y}^{k}\right)$
${y}^{k+1}={y}^{k}+{\alpha }_{k}\left(A{x}^{k}+1-b\right)$

## 对偶分解性

$f\left(x\right)=\sum _{i=1}^{N}{f}_{i}\left({x}_{i}\right)$

$A=\left[{A}_{1},{A}_{2},\cdots ,{A}_{N}\right]$

$L\left(x,y\right)=\sum _{i=1}^{N}{L}_{i}\left({x}_{i},y\right)=\sum _{i=1}^{N}\left({f}_{i}\left({x}_{i}\right)+{y}^{T}{A}_{i}{x}_{i}-\left(\frac{1}{N}\right){y}^{T}b\right)$

${x}_{i}^{k+1}=argmi{n}_{{x}_{i}}{L}_{i}\left({x}_{i},{y}^{k}\right)$
${y}^{k+1}={y}^{k}+{\alpha }^{k}\left(A{x}^{k+1}-b\right)$

## 增广拉格朗日（乘子法）

${L}_{\rho }\left(x,y\right)=f\left(x\right)+{y}^{T}\left(Ax-b\right)+\frac{\rho }{2}{‖\begin{array}{c}Ax-b\end{array}‖}_{2}^{2}$

$\underset{x}{min}f\left(x\right)+\frac{\rho }{2}{‖\begin{array}{c}Ax-b\end{array}‖}_{2}^{2}$
$s.t.Ax=b$

${x}^{k+1}=argmi{n}_{x}{L}_{\rho }\left(x,{y}^{k}\right)$
${y}^{k+1}={y}^{k}+\rho \left(A{x}^{k+1}-b\right)$

$\underset{x}{min}f\left(x\right)+g\left(z\right)$
$s.t.Ax+Bz=c$

${L}_{\rho }\left(x,z,y\right)=f\left(x\right)+g\left(z\right)+{y}^{T}\left(Ax+Bz-c\right)+\frac{\rho }{2}{‖\begin{array}{c}Ax+Bz-c\end{array}‖}_{2}^{2}$
${x}^{k+1}=argmi{n}_{x}{L}_{\rho }\left(x,{y}^{k},{z}^{k}\right)$
${z}^{k+1}=argmi{n}_{z}{L}_{\rho }\left({x}^{k+1},z,{y}^{k}\right)$
${y}^{k+1}={y}^{k}+\rho \left(A{x}^{k+1}+B{z}^{k+1}-c\right)$

## 全局变量一致性优化(Global Variable Consensus Optimization)

$\underset{x}{min}\sum _{i=1}^{N}{f}_{i}\left({x}_{i}\right)$
$s.t.{x}_{i}-z=0,i=0,1,\dots ,N$

${L}_{\rho }\left({x}_{1},{x}_{2},\dots ,{x}_{N},z,y\right)=\sum _{i=1}^{N}{f}_{i}\left({x}_{i}\right)+{y}_{i}^{T}\left({x}_{i}-z\right)+\frac{\rho }{2}{‖\begin{array}{c}{x}_{i}-z\end{array}‖}_{2}^{2}$
${x}_{i}^{k+1}=argmi{n}_{x}{f}_{i}\left({x}_{i}\right)+{y}_{i}^{{T}^{k}}\left({x}_{i}-{z}^{k}\right)+\frac{\rho }{2}{‖\begin{array}{c}{x}_{i}-{z}^{k}\end{array}‖}_{2}^{2}$
${z}^{k+1}=\frac{1}{N}\sum _{i=1}^{N}\left({x}_{i}^{k+1}+\frac{1}{\rho }{y}_{i}^{k}\right)$
${y}_{i}^{k+1}={y}_{i}^{k}+\rho \left({x}_{i}^{k+1}-{z}^{k+1}\right)$

$\underset{x}{min}\sum _{i=1}^{N}{f}_{i}\left({x}_{i}\right)+g\left(z\right)$
$s.t.{x}_{i}-z=0,i=0,1,\dots ,N$

${x}_{i}^{k+1}=argmi{n}_{x}\left({f}_{i}\left({x}_{i}\right)+{y}_{i}^{{T}^{k}}\left({x}_{i}-{z}^{k}\right)+\frac{\rho }{2}{‖\begin{array}{c}{x}_{i}-{z}^{k}\end{array}‖}_{2}^{2}\right)$
${z}^{k+1}=argmi{n}_{z}\left(g\left(z\right)+\sum _{i=1}^{N}\left(-{y}_{i}^{{T}^{k}}+\frac{\rho }{2}{‖\begin{array}{c}{x}_{i}^{k+1}-z\end{array}‖}_{2}^{2}\right)\right)$
${y}_{i}^{k+1}={y}_{i}^{k}+\rho \left({x}_{i}^{k+1}-{z}_{i}^{k+1}\right)$

$\underset{x}{min}\sum _{i=1}^{N}{f}_{i}\left({x}_{i}\right)+g\left(z\right)$
$s.t.{x}_{i}-z=0,i=0,1,\dots ,N,$
${f}_{i}\left({x}_{i}\right)$$f_i(x_i)$表示划分到每一个节点上的目函数，${x}_{i}$$x_i$表示局部模型参数，$z$$z$表示全局一致性变量。我们把整个数据集划分到每一节点，各节点独立并行训练，通过迭代更新，最终收敛到一个一致的全局解。

## Lasso问题

$\underset{x}{min}\frac{1}{2}{‖\begin{array}{c}Ax-b\end{array}‖}_{2}^{2}+\lambda {‖\begin{array}{c}x\end{array}‖}_{1}$

$\underset{{x}_{i}}{min}\sum _{i=1}^{N}\frac{1}{2}{‖\begin{array}{c}Ax-b\end{array}‖}^{2}+\lambda {‖\begin{array}{c}z\end{array}‖}_{1}$
$s.t.{x}_{i}-z=0,i=0,1,\dots ,N$
${x}_{i}^{k+1}=argmi{n}_{{x}_{i}}\left(\frac{1}{2}{‖\begin{array}{c}{A}_{i}{x}_{i}-{b}_{i}\end{array}‖}^{2}+{y}_{i}^{{T}^{k}}\left({x}_{i}-{z}^{k}\right)+\frac{\rho }{2}{‖\begin{array}{c}{x}_{i}-{z}^{k}\end{array}‖}_{2}^{2}\right)$
${z}^{k+1}=argmi{n}_{z}\left(\lambda {‖\begin{array}{c}z\end{array}‖}_{1}+\sum _{i=1}^{N}\left(-{y}_{i}^{{T}^{k}}z+\frac{\rho }{2}{‖\begin{array}{c}{x}_{i}^{k+1}-z\end{array}‖}_{2}^{2}\right)\right)$
${y}_{i}^{k+1}={y}_{i}^{k}+\rho \left({x}_{i}^{k+1}-{z}^{k+1}\right)$

${A}_{i}^{T}\left({A}_{i}{x}_{i}-{b}_{i}\right)+{y}_{i}^{k}+\rho \left({x}_{i}-{z}^{k}\right)=0$

${x}_{i}^{k+1}=\left({A}_{i}^{T}{A}_{i}+\rho I{\right)}^{-1}\left({A}_{i}^{T}{b}_{i}+\rho {z}^{k}-{y}_{i}^{k}\right)$

${z}^{k+1}=argmi{n}_{z}\left(\lambda {‖\begin{array}{c}z\end{array}‖}_{1}+\frac{\rho N}{2}{‖\begin{array}{c}z-\frac{1}{\rho N}\sum _{i=1}^{N}\left(\rho {x}_{i}^{k+1}+{y}_{i}^{k}\right)\end{array}‖}_{2}^{2}\right)$

${z}^{k+1}={\mathcal{S}}_{\lambda /\rho N}\left(\frac{1}{\rho N}\sum _{i=1}^{N}\left(\rho {x}_{i}^{k+1}+{y}_{i}^{k}\right)\right)$