Deep Learning：Optimization for Training Deep Models(一)

How Learning Differs from Pure Optimization

Optimization algorithms used for training of deep models differ from traditional optimization algorithms in several ways.

• Machine learning usually acts indirectly. In most machine learning scenarios, we care about some performance measure P , that is defined with respect to the test set and may also be intractable. We therefore optimize P only indirectly. We reduce a different cost function J(θ) in the hope that doing so will improve P. This is in contrast to pure optimization, where minimizing J is a goal in and of itself.
• Optimization algorithms for training deep models also typically include some specialization on the specific structure of machine learning objective functions. Typically, the cost function can be written as an average over the training set, such as: Eq.(1)
  $$J(\theta)=\mathbb{E}_{(x,y)\sim \hat{p}_{data}}L(f(x;\theta),y)$$
  where $L$ is the per-example loss function, $f(x; θ)$ is the predicted output when the input is x, $\hat{p}_{data}$ is the empirical distribution.
  We would usually prefer to minimize the corresponding objective function where the expectation is taken across the data generating distribution $p_{data}$ rather than just over the finite training set: Eq.(2)