第七周学习笔记

第七周学习笔记

本周的主要学习工作

1.CS229

[课程地址]

第十九讲，微分动态规划

主要内容

• Debug强化学习算法
  假设我们要建立一个直升机的强化学习算法

• 建立一个直升机的模拟器

• 选择回报函数

• 使用强化学习算法，得到策略
  假如表现很差，怎么办？

• 诊断：

  • 如果学习到的策略在模拟器中效果不错，但在现实中表现不佳，则是模拟器的问题
  • 如果人类策略的值函数大于学习结果，则是强化学习方法的不佳
  • 如果学习结果的值函数大于人类结果，但现实中表现不佳，则是回报函数的问题
    1.升级模拟器
    2.改进R
    3.改进RL算法

• 线性二次调节控制（Linear-Quadratic Regulation Control）

• 微分动态规划（differential dynamic programming）

• Kalman滤波 线性二次高斯控制（Linear-Quadratic Gaussian Control）

第二十讲，策略搜索

主要内容

• POMDP
• 策略搜索（Policy search）
• Pegasus,固定随机模拟器的伪随机序列

2.Problem Set #1

(a)$$J(\theta )=\frac{1}{2}(X\theta -y{)}^T(X\theta -y)$$$$\nabla =\frac{\partial J}{\partial \theta }=X^T(X\theta -y)$$$$H=\frac{\partial {}^{2}J}{\partial {\theta }^2}=X^{T}X$$
(b)对于任意的初始值${\theta }_0$
$${\theta }_1={\theta }_0-H^{-1}\nabla =(X^{T}X{)}^{-1}{X}^{T}y$$
一次迭代即可收敛
2.已在之前的实验中完成
代码在这
3.
( a )$$J(\Theta )=\frac{1}{2}{\Vert {\Theta }^{T}X-y\Vert }^2$$
( b )$$\Theta =(X^{T}X{)}^{-1}{X}^{T}Y$$
( c )
对于任意j
$${\theta }_j=(X^{T}X{)}^{-1}{X}^T{y}_j$$
写成矩阵形式为
$$\Theta =(X^{T}X{)}^{-1}{X}^{T}Y$$
即分别线性回归和一起线性回归的结果相同
4.
( a )
$$\begin{array}{rl}l(\phi )& =log\prod _{i=1}^m[(1-{\varphi }_y)p(x^{(i)}|y^{(i)}=0){]}^{(1-y^{(i)})}[{\varphi }_{y}p(x^{(i)}|y^{(i)}=1){]}^{y^{(i)}}\\ & =\sum _{i=1}^m(1-y^{(i)})(log(1-{\varphi }_y)+\sum _{j=1}^n(x_{j}log{\varphi }_{j|y=0}+(1-x_j)log(1-{\varphi }_{j|y=0})\left)\right)\\ & +y^{(i)}(log{\varphi }_y+\sum _{j=1}^n{x}_{j}log{\varphi }_{j|y=1}+(1-x_j)log(1-{\varphi }_{j|y=1}))\end{array}$$
( b )
$$\frac{\partial l(\varphi )}{\partial {\varphi }_{j|y=0}}=\sum _{i=1}^m\frac{(1-y^{(i)})x_j^{(i)}}{{\varphi }_{j|y=0}}-\frac{(1-y^{(i)})(1-x_j^{(i)})}{1-{\varphi }_{j|y=0}}$$$$\frac{\partial l(\varphi )}{\partial {\varphi }_{j|y=1}}=\sum _{i=1}^m\frac{y^{(i)}{x}_j^{(i)}}{{\varphi }_{j|y=1}}-\frac{y^{(i)}(1-x_j)}{1-{\varphi }_{j|y=1}}$$$$\frac{\partial l(\varphi )}{\partial {\varphi }_y}=\sum _{i=1}^m-\frac{1-y^{(i)}}{1-{\varphi }_y}+\frac{y^{(i)}}{{\varphi }_y}$$
令以上偏导数都等于0，得
ϕ j ∣ y = 0 = ∑ i = 1 m 1 { y ( i ) = 0 } x j ( i ) ∑ i = 1 m 1 { y ( i ) = 0 } \phi_{j|y=0}=\dfrac{\sum_{i=1}^{m}1\{ y^{(i)}=0\}x^{(i)}_j}{\sum_{i=1}^{m}1\{ y^{(i)}=0\}} ϕj∣y=0​=∑i=1m​1{y(i)=0}∑i=1m​1{y(i)=0}xj(i)​​ ϕ j ∣ y = 1 = ∑ i = 1 m 1 { y ( i ) = 1 } x j ( i ) ∑ i = 1 m 1 { y ( i ) = 1 } \phi_{j|y=1}=\dfrac{\sum_{i=1}^{m}1\{ y^{(i)}=1\}x^{(i)}_j}{\sum_{i=1}^{m}1\{ y^{(i)}=1\}} ϕj∣y=1​=∑i=1m​1{y(i)=1}∑i=1m​1{y(i)=1}xj(i)​​$${\varphi }_y=\frac{{\sum }_{i=1}^m{y}^{(i)}}{m}$$
( c )
$$\begin{array}{rl}p(y=1|x)\ge p(y=0|x)& ⟺p(x|y=1)p(y)\ge p(x|y=0)p(y)\\ & ⟺p(x|y=1){\varphi }_y\ge p(x|y=0)(1-{\varphi }_y)\\ & ⟺\prod _{j=1}^{n}p(x_j|y=1){\varphi }_y\ge \prod _{j=1}^{n}p(x_j|y=0)(1-{\varphi }_y)\\ & ⟺\prod _{j=1}^n{\varphi }_{j|y=1}^{x_j}(1-{\varphi }_{j|y=1}{)}^{1-x_j}{\varphi }_y\ge \prod _{j=1}^n{\varphi }_{j|y=0}^{x_j}(1-{\varphi }_{j|y=0}{)}^{1-x_j}(1-{\varphi }_y)\\ & ⟺\sum _{j=1}^n{x}_{j}log{\varphi }_{j|y=1}+(1-x_j)log(1-{\varphi }_{j|y=1})+log\frac{{\varphi }_y}{1-{\varphi }_y}\ge \sum _{j=1}^n{x}_{j}log{\varphi }_{j|y=0}^{x_j}+(1-x_j)(1-{\varphi }_{j|y=0}{)}^{1-x_j}\\ & ⟺\sum _{j=1}^n{x}_{j}log\frac{{\varphi }_{j|y=1}}{{\varphi }_{j|y=0}}+(1-x_j)log\frac{1-{\varphi }_{j|y=1}}{1-{\varphi }_{j|y=0}}+log\frac{{\varphi }_y}{1-{\varphi }_y}\ge 0\\ & ⟺\sum _{j=1}^n{x}_{j}log\frac{{\varphi }_{j|y=1}(1-{\varphi }_{j|y=0})}{{\varphi }_{j|y=0}(1-{\varphi }_{j|y=1})}+\sum _{j=1}^{n}log\frac{1-{\varphi }_{j|y=1}}{1-{\varphi }_{j|y=0}}+log\frac{{\varphi }_y}{1-{\varphi }_y}\ge 0\end{array}$$
所以最终得到的是一个线性分类器
5.
( a )
p ( y ; ϕ ) = ( 1 − ϕ ) y − 1 ϕ = e x p { l o g ( 1 − ϕ ) y − l o g ϕ 1 − ϕ } p(y;\phi)=(1-\phi)^{y-1}\phi=exp\{log(1-\phi)y-log\dfrac{\phi}{1-\phi}\} p(y;ϕ)=(1−ϕ)y−1ϕ=exp{log(1−ϕ)y−log1−ϕϕ​}
所以
$$b(y)=1$$$$\eta =log(1-\varphi )$$$$T(y)=y$$$$a(\eta )=\eta +log(1-e^{\eta })$$
( b )$$g(\eta )=\frac{1}{1-e^{\eta }}$$$$h_{\theta }(x)=\frac{1}{1-e^{{\theta }^{T}x}}$$
( c )
$$y=\frac{1}{1-e^{{\theta }^{T}x}}$$$$l(\theta )=e^{{\theta }^{T}x(y-1)}(1-e^{{\theta }^{T}x})$$$$logl(\theta )={\theta }^{T}x(y-1)+log(1-e^{{\theta }^{T}x})$$$$\frac{\partial logl(\theta )}{\partial \theta }=(y-\frac{1}{1-e^{{\theta }^{T}x}})x$$
权值更新公式为
$${\theta }_{k+1}={\theta }_k-\alpha \frac{\partial J}{\partial \theta }$$

Problem Set #2

( a )
$$J(\theta )=\frac{1}{2}(X\theta -y{)}^T(X\theta -y)+\frac{\lambda }{2}{\theta }^T\theta$$$$\frac{\partial J}{\partial \theta }=X^T(X\theta -y)+\lambda \theta$$
$J$是凸函数，令偏导数为零$$\theta =(X^{T}X+\lambda I{)}^{-1}{X}^{T}y$$
( b )$${\theta }^{T}x=y^{T}X(\lambda I+X^{T}X{)}^{-1}x=y^T(\lambda I+X{X}^T{)}^{-1}Xx$$
以$\varphi (x^{(i)})$代替X中的每一列，则无需计算出每个$\varphi (x^{(i)})$，只要计算出$\varphi (x^{(i)})$之间的内积即可。
此外
$$(\lambda I+BA{)}^{-1}B=\frac{1}{\lambda }(\frac{BA}{\lambda }+\frac{BABA}{{\lambda }^2}+\cdot \cdot \cdot )B=\frac{B}{\lambda }(\frac{AB}{\lambda }+\frac{ABAB}{{\lambda }^2}+\cdot \cdot \cdot )=B(\lambda I+AB{)}^{-1}$$
2.
( a )
假设该最优化问题的解中，存在${\xi }_j<0$，因为
$$y^{(j)}(w^T{x}^{(j)}+b)\ge 1-{\xi }_j\ge 1$$
若固定其他参数不变，令${\xi }_j=0$可以得到使目标函数更小的解，故当前解不是局部极小值，所以${\xi }_j\ge 0$对于任意$j$成立，因此，是否增加${\xi }_j\ge 0$的约束不影响问题的解。
( b )
$$\begin{gathered} L(w,b,\xi ,\alpha )=\frac{1}{2}{\Vert w\Vert }^2+\frac{C}{2}\sum _{i=1}^m{\xi }_i^2-\sum _{i=1}^m{\alpha }_i(y^{(i)}(w^T{x}^{(i)}+b)-1+{\xi }_i) \\ {\alpha }_i\ge 0,i=1,2,..,m \end{gathered}$$
( c )
$$\begin{gathered} {\nabla }_{w}L=w-\sum _{i=1}^m{\alpha }_i{y}^{(i)}{x}^{(i)}=0 \\ {\nabla }_b=\sum _{i=1}^m{\alpha }_i{y}^{(i)}=0 \\ {\nabla }_{\xi }=C\xi -\alpha =0 \end{gathered}$$
( d )
根据( c )
$$\begin{gathered} w=\sum _{i=1}^m{\alpha }_i{y}^{(i)}{x}^{(i)} \\ \sum _{i=1}^m{\alpha }_i{y}^{(i)}=0 \\ c{\xi }_i={\alpha }_i \end{gathered}$$$$\begin{array}{rl}L(w,b,\xi ,\alpha )& =\frac{1}{2}{\Vert w\Vert }^2+\frac{C}{2}\sum _{i=1}^m{\xi }_i^2-\sum _{i=1}^m{\alpha }_i(y^{(i)}(w^T{x}^{(i)}+b)-1+{\xi }_i)\\ & =\frac{1}{2}(\sum _{i=1}^m{\alpha }_i{y}^{(i)}{x}^{(i)}{)}^T(\sum _{i=1}^m{\alpha }_i{y}^{(i)}{x}^{(i)})+\frac{1}{2C}\sum _{i=1}^m{\alpha }_i^2-\sum _{i=1}^m{\alpha }_i{y}^{(i)}(\sum _{j=1}^m{\alpha }_j{y}^{(i)}{x}^{(j)}{)}^T{x}^{(i)}+\sum _{i=1}^m{\alpha }_i-\frac{1}{C}\sum _{i=1}^m{\alpha }_i^2\\ & =-\frac{1}{2}\sum _{i=1}^m\sum _{j=1}^m{\alpha }_i{\alpha }_j{y}^{(i)}{y}^{(j)}(x^{(i)}{)}^T{x}^{(j)}-\frac{1}{2C}\sum _{i=1}^m{\alpha }_i^2+\sum _{i=1}^m{\alpha }_i\end{array}$$
原问题的对偶问题为
$$\begin{gathered} \underset{\alpha }{\max }W(\alpha )=-\frac{1}{2}\sum _{i=1}^m\sum _{j=1}^m{\alpha }_i{\alpha }_j{y}^{(i)}{y}^{(j)}(x^{(i)}{)}^T{x}^{(j)}-\frac{1}{2C}\sum _{i=1}^m{\alpha }_i^2+\sum _{i=1}^m{\alpha }_i \\ s.t.\sum _{i=1}^m{\alpha }_i{y}^{(i)}=0 \\ {\alpha }_i\ge 0,i=1,2,..,m \end{gathered}$$
3.
( a )
$\forall i$，令${\alpha }_i=1$，令$b$=0，则$\forall k$
$$\begin{array}{r}\left|f(x^{(k)})-y^{(k)}\right|=\left|\sum _{i=1,i̸=k}^m{y}^{(i)}{e}^{-\frac{\Vert x^{(i)}-x^{(k)}\Vert }{{\tau }^2}}\right|\le \sum _{i=1,i̸=k}^m\left|e^{-\frac{\Vert x^{(i)}-x^{(k)}\Vert }{{\tau }^2}}\right|\le (m-1)e^{-\frac{ϵ}{{\tau }^2}}\end{array}$$
只要取$$\tau <\sqrt{\frac{ϵ}{log(m-1)}}$$
就有
$$\left|f(x^{(k)})-y^{(k)}\right|\le (m-1)e^{-\frac{ϵ}{{\tau }^2}}<1$$
( b )问题有误，原意应该是问没有松弛变量时能否达到零误差
可以，( a )中已经找到了达成零误差的解，由于SVM损失函数是凸函数，故必定可以找到那个解。
( c )
不一定，我们可以通过减小C来使得优化过程更偏向得到间隔更大但带有一定训练误差的解