Chapter 5: Beyond the black-box model
- Section5.1: propose a first order method with a 1/t^2 convergence rate, despite the non-smoothness
- Section5.2: the function is the maximum of smooth functions
- Section5.3: a concise description of Interior Point Methods
Section 5.1: Sum of a smooth and a simple non-smooth term
the problem:
f is convex and beta-smooth, and g is convex.ISTA(Iterative Shrinkage-Thresholding Algorthm)
- 根据Gradient Descent on the smooth function f
xt+1的表达式 - 结合这个问题minimize f+g得到xt+1的表达式
这个就是ISTA算法的迭代式。 - 查阅论文可得到证明此算法的收敛率(函数convex并且smooth,收敛率为1/t, 函数只convex时,收敛率为1/根号t)
- 这个算法需要假设g is simple, 因为计算xt+1本身是一个凸优化问题,当假设g is simple时,计算xt+1可以通过解决n个在一维空间上的凸优化问题。
- 根据Gradient Descent on the smooth function f
FISTA(Fast ISTA)
- 结合Nesterov’s Accelerated Gradient Descent得到对应的
- 收敛率(证明查询相关论文)
- 结合Nesterov’s Accelerated Gradient Descent得到对应的
CMD and RDA
- ISTA and FISTA assume smoothness in the Euclidean metric.
- CMD and RDA use these ideas in a non-Euclidean metric
总结
- 当函数可以分解为sum of f and g(f is convex and smooth, g is convex)时,这个凸优化问题的收敛率比只知道函数为凸时的收敛率小。
未完待续