模式识别作业2——SYSU

最新推荐文章于 2021-11-09 20:21:03 发布

得浪浪

最新推荐文章于 2021-11-09 20:21:03 发布

阅读量2.5k

点赞数 1

分类专栏：课程作业文章标签：模式识别

本文链接：https://blog.csdn.net/weixin_44175879/article/details/111238662

版权

课程作业专栏收录该内容

3 篇文章 2 订阅

订阅专栏

参考教材：《模式识别》吴建鑫著

Problem1-6.5

（a）

$$
由习题6.1可知，rank(X) = rank(X^TX)\
rank(X+Y) \leq rank(X) + rank(Y)\
rank(X^TX) = rank(XX^T)\
\because S_w = \sum_i S_i\
\therefore rank(S_w) \leq \sum_i rank(S_i)\
\because S_i = \sum_{C_{ij}\in C_i}(C_{ij}-\overline C_i)(C_{ij}-\overline C_i)^T\

\therefore rank(S_i) = rank(\sum_{C_{ij}\in C_i}(C_{ij}-\overline C_i)(C_{ij}-\overline C_i)^T)\
= \sum_{C_{ij}\in C_i}rank((C_{ij}-\overline C_i)(C_{ij}-\overline C_i)^T)\
= \sum_{C_{ij}\in C_i}rank((C_{ij}-\overline C_i)^T(C_{ij}-\overline C_i))
= \sum_{C_{ij}\in C_i}1\

\therefore rank(S_w) \leq \sum_i rank(S_i) = \sum_i\sum_{C_{ij}\in C_i}1 =N < D\
\therefore 若N \leq D ,S_w可逆
$$

(b)

$$
\because S_B = \sum_{c=1}^C(m_c - \overline m)(m_c - \overline m)^T\
\overline m = \frac{1}{c}\sum_{c=1}^Cm_c\
\therefore S_B = \sum_{c=1}^C(m_c - \overline m)(m_c - \overline m)^T = \sum_{c=1}^C(m_c*m_cT - m_c*\overline m^T - m_c^T\overline m + \overline m*\overline m^T)\
= \sum_{c=1}^Cm_c*m_cT - \sum_{c=1}^C m_c*\overline m^T - \sum_{c=1}^Cm_cT\overline m + \sum_{c=1}^C\overline m*\overline m^T\
= C * \overline m*\overline m^T - 2 C\overline m*\overline m^T + \sum_{c=1}^Cm_c*m_cT\
= \sum_{c=1}^Cm_c*m_cT - C * \overline m*\overline m^T\

rank(S_B) = rank( \sum_{c=1}^Cm_c*m_cT - C * \overline m*\overline m^T) \
\leq \sum_{c=1}^{Crank(m_c*m_c}T) - rank(C * \overline m*\overline m^T)\
= \sum_{c=1}^Crank(m_cTm_c) - rank(C * \overline m^T\overline m)\
= C-1
$$

因为 $S_B$ 的秩小于等于C-1，所以广义特征值问题 $S_W^{-1}S_B\omega = \lambda S_W\omega$ 最多可以获得C-1个广义特征向量。

©

由（a）可知$ rank(S_w) \leq \sum_i rank(S_i) = \sum_i\sum_{C_{ij}\in C_i}1 =N $, 若 N > D, 则有可能使得$ rank(S_w) = D \leq N $成立，则$ S_w $可逆。若 N > > D, 则这种可能性更大，所以$ S_w$可逆的可能性也更大。

（d）

$G^{-T}Q\\ \therefore X^TS_BX = (G^{-T}Q)^TS_B(G^{-T}Q) = Q^TG^{-1}S_BG^{-T}Q\\ \because C = G^{-1}S_BG^{-T}\\ \therefore X^TS_BX = Q^TCQ\\ \because Q^TCQ为对角矩阵\\ \therefore X^TS_BX为对角矩阵\\ \\ X^TS_WX = (G^{-T}Q)^TS_W(G^{-T}Q) = Q^TG^{-1}S_WG^{-T}Q\\ \because S_W = GG^T\\ \therefore X^TS_WX = Q^TG^{-1}GG^TG^{-T}Q = Q^TQ\\ 由对角化过程可知Q为单位正交矩阵\\ \therefore X^TS_WX = I\\$

$广义特征值问题等价于求解S_W^{-1}S_B\omega = \lambda \omega\\ S_W^{-1}S_B的相似对角矩阵的对角线元素即为广义特征向量的值\\ \because X^TS_W^{-1}S_BX = (G^{-T}Q)^TS_W^{-1}S_B (G^{-T}Q) = Q^TG^{-1}S_W^{-1}S_BG^{-T}Q\\ = Q^TG^{-1}(GG^T)^{-1}S_BG^{-T}Q = Q^TG^{-1}G^{-T}G^{-1}S_BG^{-T}Q = Q^TICQ = Q^TCQ = X^TS_BX\\ \therefore 由矩阵相似知识可知，S_W^{-1}S_B和X^TS_BX相似\\ \because X^TS_BX是对角矩阵\\ \therefore X^TS_BX的对角元即为所以特征值\\ \because X^TS_W^{-1}S_BX = X^TS_BX\\ \therefore X的列即为对应的广义特征向量$

这样得到的X不是单位正交的

Problem2-7.4

（a）

$\omega^Tx\\ 那么优化问题为 argmin_y ||x-y||^2 \quad s.t. f(y) = 0\\ 拉格朗日函数为L(y,\lambda) = ||x-y||^2 -\lambda(\omega^Ty+b)\\ 令\frac{\partial L}{\partial y} = 0可得y = x + \frac{\lambda}{2}\omega\\ 带入f(y) = 0可得\omega^Tx + \frac{\lambda}{2}\omega^T\omega = 0\\ \lambda = -\frac{2\omega^Tx}{\omega^T\omega}\\ x_\perp = x + \frac{\lambda}{2}\omega = x - \frac{\omega^Tx}{\omega^T\omega}\omega\\ z = x - x_\perp = \frac{\omega^Tx}{\omega^T\omega}\omega\\ ||z|| = || \frac{\omega^Tx}{\omega^T\omega}\omega|| = \frac{|\omega^Tx|}{||\omega||}\\ 则优化问题为max_\omega\quad min_{1≤i≤n} \quad \frac{|\omega^Tx|}{||\omega||}\\ s.t. \quad y_if(x_i) > 0 1≤i≤n\\ 其中y_i=1表示正类样本，y_i = -1表示负类样本\\ 因为y_if(x_i) = |f(x_i)| = |\omega^Tx_i|\\ 所以min_{1≤i≤n} \frac{|\omega^Tx|}{||\omega||} = min_{1≤i≤n} \frac{y_i\omega^Tx}{||\omega||} = \frac{1}{||\omega||}min_{1≤i≤n}y_i\omega^Tx\\ 取c = min_{1≤i≤n} \quad y_i\omega^{*T}x，则min_{1≤i≤n}y_i(\frac{\omega*}{c})^Tx = 1 >0\\ 所以优化问题等价于max_{\omega} \frac{min_{1≤i≤n} |\omega^Tx_i|}{||\omega||} \Leftrightarrow max_\omega \frac{1}{||\omega||}\Leftrightarrow max_\omega \frac12\omega^T\omega\\ s.t. y_i\omega^Tx_i ≥1 \quad 1≤i≤n$

(b)

$由（a）已知优化问题为max_\omega \frac12\omega^T\omega\\ s.t. y_i\omega^Tx_i ≥1 \quad 1≤i≤n\\ 拉格朗日函数为L(\omega,\alpha) = \frac12\omega^T\omega - \sum_{i=1}^{n}\alpha_i(y_i(\omega^Tx_i ) - 1)\\ s.t. \alpha_i \geq 0 \quad 1 \leq i \leq n\\ \frac{\partial L}{\partial \omega} = 0 \Rightarrow \omega = \sum_{i=1}^{n}\alpha_iy_ix_i\\ 所以KKT条件为：\\ \omega = \sum_{i=1}^{n}\alpha_iy_ix_i\\ \alpha_i \geq 0\\ y_i\omega^Tx_i \geq1\\ \alpha_i(y_i\omega^Tx_i-1) = 0\\ 所以有\frac12\omega^T\omega = \frac12 \sum_{i=1}^n\sum_{j=1}^n\alpha_i\alpha_jy_iy_jx_i^Tx_j\\ \sum_{i=1}^n\alpha_iy_i\omega^Tx_i = \omega^T\omega = \sum_{i=1}^n\sum_{j=1}^n\alpha_i\alpha_jy_iy_jx_i^Tx_j\\ 所以拉格朗日函数为-\frac12 \sum_{i=1}^n\sum_{j=1}^n\alpha_i\alpha_jy_iy_jx_i^Tx_j + \sum_{i=1}^n\alpha_i\\ 对偶形式为 max_\alpha -\frac12 \sum_{i=1}^n\sum_{j=1}^n\alpha_i\alpha_jy_iy_jx_i^Tx_j + \sum_{i=1}^n\alpha_i\\ s.t. \alpha_i \geq 0 \quad 1 \leq i \leq n \\ \Leftrightarrow\\ min_\alpha \frac12 \sum_{i=1}^n\sum_{j=1}^n\alpha_i\alpha_jy_iy_jx_i^Tx_j - \sum_{i=1}^n\alpha_i\\ s.t. \alpha_i \geq 0 \quad 1 \leq i \leq n\\ 因为\alpha为有限维，所以一定存在一个最大的\alpha_i，设C = max_{1\leq i\leq n}(\alpha_i)\\ 则对偶问题为min_\alpha \frac12 \sum_{i=1}^n\sum_{j=1}^n\alpha_i\alpha_jy_iy_jx_i^Tx_j - \sum_{i=1}^n\alpha_i\\ s.t.0 \leq \alpha_i \leq C \quad 1 \leq i \leq n\\ 由KKT条件，\omega = \sum_{i=1}^{n}\alpha_iy_ix_i\\ 所以若\alpha的最优解为\alpha^*,\omega^* = \sum_{i=1}^{n}\alpha^*_iy_ix_i$

©

$通过对x增加一个额外的维度，所增加的维度总有一个常数值1，将任意x∈R^d转化为R^{(d+1)}空间上的\hat{x}\\ 之后对数据集\hat{x}进行训练学习得到一个不含偏置项的\hat\omega，那么这个\hat\omega 的最后一维的值即为在R^d空间上含偏置项的SVM的b\\ b =(\sum_{i=1}^{n}\alpha^*_iy_ix_i)_{d+1}$

Problem3-8.6

$由题意可知\\ P(x|y=1) = \frac{1}{\sqrt{2\pi}\sum}exp(-\frac{(x-\mu_1)^2}{2\sum})\\ P(x|y=2) = \frac{1}{\sqrt{2\pi}\sum}exp(-\frac{(x-\mu_2)^2}{2\sum})\\ P(y=1|x;\theta) = \frac{P(x|y=1)P(y=1)}{P(x;\theta)}\\ P(y=2|x;\theta) = \frac{P(x|y=2)P(y=2)}{P(x;\theta)}\\ 因为y^* = argmax(P(y=i|x;\theta))\\ 所以y^* = \left\{ \begin{aligned} & 1 \quad if P(y=1|x;\theta) > P(y=2|x;\theta)\\ & 2 \quad if P(y=1|x;\theta) ≤ P(y=2|x;\theta)\\ \end{aligned} \right. \\ 又因为 \left\{ \begin{aligned} & 当 P(y=1|x;\theta) > P(y=2|x;\theta)时\quad \frac{P(y=1|x;\theta)}{P(y=2|x;\theta)} > 1\\ & 当 P(y=1|x;\theta) ≤ P(y=2|x;\theta)时\quad \frac{P(y=1|x;\theta)}{P(y=2|x;\theta)} ≤ 1\\ \end{aligned} \right. \\ 所以y^* = \left\{ \begin{aligned} & 1 \quad if \frac{P(y=1|x;\theta)}{P(y=2|x;\theta)} > 1\\ & 2 \quad if \frac{P(y=1|x;\theta)}{P(y=2|x;\theta)} ≤ 1\\ \end{aligned} \right. \\ 因为\frac{P(y=1|x;\theta)}{P(y=2|x;\theta)} = \frac{ \frac{P(x|y=1)P(y=1)}{P(x;\theta)}}{\frac{P(x|y=2)P(y=2)}{P(x;\theta)}} = \frac{P(x|y=1)P(y=1)}{P(x|y=2)P(y=2)}\\ 且 P(y=1) = P(y = 2) = 0.5\\ 所以\frac{P(y=1|x;\theta)}{P(y=2|x;\theta)} = \frac{P(x|y=1)}{P(x|y=2)} \\= \frac{\frac{1}{\sqrt{2\pi}\sum}exp(-\frac{(x-\mu_1)^2}{2\sum})}{ \frac{1}{\sqrt{2\pi}\sum}exp(-\frac{(x-\mu_2)^2}{2\sum})} = exp(\frac{(x-\mu_2)^2-(x-\mu_1)^2}{2\sum})\\ 所以y^* = \left\{ \begin{aligned} & 1 \quad if\quad exp(\frac{(x-\mu_2)^2-(x-\mu_1)^2}{2\sum}) > 1\\ & 2 \quad if\quad exp(\frac{(x-\mu_2)^2-(x-\mu_1)^2}{2\sum}) ≤ 1\\ \end{aligned} \right. \\ 因为\left\{ \begin{aligned} & exp(\frac{(x-\mu_2)^2-(x-\mu_1)^2}{2\sum}) > 1 \quad if \frac{(x-\mu_2)^2-(x-\mu_1)^2}{2\sum} > 0\\ &exp(\frac{(x-\mu_2)^2-(x-\mu_1)^2}{2\sum}) ≤ 1 \quad if \frac{(x-\mu_2)^2-(x-\mu_1)^2}{2\sum} ≤ 0\\ \end{aligned} \right. \\ 所以y^* = \left\{ \begin{aligned} & 1 \quad if\quad \frac{(x-\mu_2)^2-(x-\mu_1)^2}{2\sum} > 0\\ & 2 \quad if\quad \frac{(x-\mu_2)^2-(x-\mu_1)^2}{2\sum} ≤ 0\\ \end{aligned} \right. \\ 即y^* = \left\{ \begin{aligned} & 1 \quad if\quad \frac{2(\mu_1-\mu_2)x + \mu_1^2-\mu_2^2}{2\sum} > 0\\ & 2 \quad if\quad \frac{2(\mu_1-\mu_2)x + \mu_1^2-\mu_2^2}{2\sum} ≤ 0\\ \end{aligned} \right. \\ 所以令\omega = (\frac{2(\mu_1-\mu_2)}{2\sum})^T, \quad b = \frac{\mu_1^2-\mu_2^2}{2\sum}\\ 则该预测规则可被重写成如下的的能使形式：\\ y^* = \left\{ \begin{aligned} &1 \quad if \omega^Tx + b > 0\\ &2 \quad if \omega^Tx + b ≤0 \end{aligned} \right. \\ 其中\omega = (\frac{2(\mu_1-\mu_2)}{2\sum})^T, \quad b = \frac{\mu_1^2-\mu_2^2}{2\sum}$

Problem4-9.1

$E_d^T(x-\overline{x})\\ 所以 ||y_1 - y_2||_2^2 = ||E_d^T(x_1-\overline{x_1}) - E_d^T(x_2-\overline{x_2}) ||_2^2\\ = ||E_d^T(x_1-x_2)||_2^2\\ = (E_d^Tx_1-E_d^Tx_2)^T(E_d^Tx_1-E_d^Tx_2)\\ = (x_1-x_2)^TE_dE_d^T(x_1-x_2)\\ 又因为d_A^2(x_1,x_2) = (x_1-x_2)^TA(x_1-x_2)\\ 所以欲使d^2_A(x_1,x_2) = ||y_1 - y_2||_2^2\\ 即使(x_1-x_2)^TA(x_1-x_2) = (x_1-x_2)^TE_dE_d^T(x_1-x_2)\\ 所以A = E_dE_d^T$

Problem5-10.3

离散分布和连续分布的情况类似，下面我们只讨论离散分布一种情况，连续分布只需要将就和换成积分即可。
$KL(p||q)\\由吉布斯不等式：-\sum_{i=1}^{n}p_ilog_2p_i ≦ -\sum_{i=1}^np_ilog_2q_i，等号成立当且仅当p_i = q_i \forall i\\KL(p||q) = \sum_{i=1}^np_ilog_2(\frac{p_i}{q_i}) = \sum_{i=1}^{n}p_ilog_2p_i- \sum_{i=1}^np_ilog_2q_i ≧0\\所以CE(p,q)= h(p) + KL(p||q) ≧ h(p)\\等号成立当且仅当KL(p||q) = 0,即p_i = q_i \forall i，也即p = q$

得浪浪

关注

1
点赞
踩
6

收藏

觉得还不错? 一键收藏
0
评论
模式识别作业2——SYSU

参考教材：《模式识别》吴建鑫著Problem1-6.5（a）$$由习题6.1可知，rank(X) = rank(X^TX)\rank(X+Y) \leq rank(X) + rank(Y)\rank(X^TX) = rank(XX^T)\\because S_w = \sum_i S_i\\therefore rank(S_w) \leq \sum_i rank(S_i)\\because S_i = \sum_{C_{ij}\in C_i}(C_{ij}-\overline C_i)(C
复制链接

扫一扫

专栏目录