pytorch优化器Adam和Adamw
只介绍二者的weight_decay的区别
一般的梯度下降的方法是 θ j = θ j − α ∂ ∂ θ j J \theta_j=\theta_j-\alpha\frac{\partial}{\partial\theta_j}J θj=θj−α∂θj∂J
Adam
算法
给出pytorch.optim.Adam类中具体实现的算法，来自pytorch中class Adam(Optimizer)
input : γ (lr) , β 1 , β 2 (betas) , θ 0 (params) , f ( θ ) (objective) λ (weight decay) , a m s g r a d initialize : m 0 ← 0 ( first moment) , v 0 ← 0 (second moment) , v 0 ^ m a x ← 0 for t = 1 to … do g t ← ∇ θ f t ( θ t − 1 ) if λ ≠ 0 g t ← g t + λ θ t − 1 m t ← β 1 m t − 1 + ( 1 − β 1 ) g t v t ← β 2 v t − 1 + ( 1 − β 2 ) g t 2 m t ^ ← m t / ( 1 − β 1 t ) v t ^ ← v t / ( 1 − β 2 t ) if a m s g r a d v t ^ m a x ← m a x ( v t ^ m a x , v t ^ ) θ t ← θ t − 1 − γ m t ^ / ( v t ^ m a x + ϵ ) else θ t ← θ t − 1 − γ m t ^ / ( v t ^ + ϵ ) r e t u r n θ t \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma \text{ (lr)}, \beta_1, \beta_2 \text{ (betas)},\theta_0 \text{ (params)},f(\theta) \text{ (objective)} \\ &\hspace{13mm} \lambda \text{ (weight decay)}, \: amsgrad \\ &\textbf{initialize} : m_0 \leftarrow 0 \text{ ( first moment)}, v_0\leftarrow 0 \text{ (second moment)},\: \widehat{v_0}^{max}\leftarrow 0\\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}\textbf{if} \: \lambda \neq 0 \\ &\hspace{10mm} g_t \leftarrow g_t + \lambda \theta_{t-1} \\ &\hspace{5mm}m_t \leftarrow \beta_1 m_{t-1} + (1 - \beta_1) g_t \\ &\hspace{5mm}v_t \leftarrow \beta_2 v_{t-1} + (1-\beta_2) g^2_t \\ &\hspace{5mm}\widehat{m_t} \leftarrow m_t/\big(1-\beta_1^t \big) \\ &\hspace{5mm}\widehat{v_t} \leftarrow v_t/\big(1-\beta_2^t \big) \\ &\hspace{5mm}\textbf{if} \: amsgrad \\ &\hspace{10mm}\widehat{v_t}^{max} \leftarrow \mathrm{max}(\widehat{v_t}^{max}, \widehat{v_t}) \\ &\hspace{10mm}\theta_t \leftarrow \theta_{t-1} - \gamma \widehat{m_t}/ \big(\sqrt{\widehat{v_t}^{max}} + \epsilon \big) \\ &\hspace{5mm}\textbf{else} \\ &\hspace{10mm}\theta_t \leftarrow \theta_{t-1} - \gamma \widehat{m_t}/ \big(\sqrt{\widehat{v_t}} + \epsilon \big) \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} input:γ (lr),β1,β2 (betas),θ0 (params),f(θ) (objective)λ (weight decay),amsgradinitialize:m0←0 ( first moment),v0←0 (second moment),v0
最低0.47元/天 解锁文章
6332
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
