线性判别分析(fisher) 实现

线性判别分析

线性判别分析中有降维，把数据都投影到同一条线上，然后在直线上取一个阈值，将直线分成两条射线，每一条代表一个分类。会损失一些数据信息，但如果这些信息是一些干扰信息，丢失也未尝不是好事。

线性判别分析之后的结果是一个向量，其他的不行吗？

主要指导思想(目标)：类内小，类间大。

公式推导

我们得到的是向量，为了方便计算损失，不妨设$||ww||=1$，每一个数据${XX}_i$看作一个向量。那么${XX}_{i}ww$是每个数据在$ww$方向上的投影。与$ww$的其中一个平面是划分平面。

两个不同类别分别命名为$C_1$和$C_2$，用$\mu \mu$,${\mu \mu }_{C_1}$, ${\mu \mu }_{C_2}$分别代表全部数据，$C_1$数据,$C_2$数据的均值，用$\Sigma \Sigma$,${\Sigma \Sigma }_{C_1}$, ${\Sigma \Sigma }_{C_2}$分别代表全部数据，$C_1$数据,$C_2$数据的协方差矩阵。
$\tilde{\mu }$和${\tilde{\sigma }}^2$表示投影的均值和方差。

$$\begin{array}{ccl}\mu \mu & =& \frac{1}{N}{\sum }_1^N{XX}_i\\ {\mu \mu }_{C_1}& =& \frac{1}{N_{C_1}}{\sum }_1^{N_{C_1}}{XX}_{C_{1}i}\\ {\mu \mu }_{C_2}& =& \frac{1}{N_{C_2}}{\sum }_1^{N_{C_2}}{XX}_{C_{1}i}\\ \Sigma \Sigma & =& \frac{1}{N}{\sum }_1^N({XX}_i-\mu \mu )({XX}_i-\mu \mu {)}^T\\ {\Sigma \Sigma }_{C_1}& =& \frac{1}{N_{C_1}}{\sum }_1^{N_{C_1}}({XX}_{C_{1}i}-{\mu \mu }_{C_1})({XX}_{C_{1}i}-{\mu \mu }_{C_1}{)}^T\\ {\Sigma \Sigma }_{C_2}& =& \frac{1}{N_{C_2}}{\sum }_1^{N_{C_2}}({XX}_{C_{2}i}-{\mu \mu }_{C_2})({XX}_{C_{2}i}-{\mu \mu }_{C_2}{)}^T\end{array}$$

$$\begin{array}{ccl}\tilde{\mu }& =& \frac{1}{N}{\sum }_1^N{XX}_i\theta \theta \\ {\tilde{\mu }}_{C_1}& =& \frac{1}{N_{C_1}}{\sum }_1^{N_{C_1}}{XX}_{C_{1}i}\theta \theta \\ {\tilde{\mu }}_{C_2}& =& \frac{1}{N_{C_2}}{\sum }_1^{N_{C_2}}{XX}_{C_{2}i}\theta \theta \\ {\tilde{\sigma }}^2& =& \frac{1}{N}{\sum }_1^N({XX}_i\theta \theta -\tilde{\mu }{)}^2\\ {\tilde{\sigma }}_{C_1}^2& =& \frac{1}{N_{C_1}}{\sum }_1^{N_{C_1}}({XX}_{C_{1}i}\theta \theta -{\tilde{\mu }}_{C_1}{)}^2\\ {\tilde{\sigma }}_{C_2}^2& =& \frac{1}{N_{C_2}}{\sum }_1^{N_{C_2}}({XX}_{C_{2}i}\theta \theta -{\tilde{\mu }}_{C_2}{)}^2\end{array}$$

类间：$({\tilde{\mu }}_{C_1}-{\tilde{\mu }}_{C_2}{)}^2$

类内：${\tilde{\sigma }}_{C_1}^2+{\tilde{\sigma }}_{C_2}^2$

目标函数:$J(\theta \theta )=\frac{({\tilde{\mu }}_{C_1}-{\tilde{\mu }}_{C_2}{)}^2}{{\tilde{\sigma }}_{C_1}^2+{\tilde{\sigma }}_{C_2}^2}$

$$\begin{array}{ccl}J(\theta \theta )& =& \frac{({\tilde{\mu }}_{C_1}-{\tilde{\mu }}_{C_2}{)}^2}{{\tilde{\sigma }}_{C_1}^2+{\tilde{\sigma }}_{C_2}^2}\\ 分子& =& ({\tilde{\mu }}_{C_1}-{\tilde{\mu }}_{C_2}{)}^2\\ & =& (\frac{1}{N_{C_1}}{\sum }_1^{N_{C_1}}{XX}_{C_{1}i}\theta \theta -\frac{1}{N_{C_2}}{\sum }_1^{N_{C_2}}{XX}_{C_{2}i}\theta \theta {)}^2\\ & =& (({\mu \mu }_{C_1}-{\mu \mu }_{C_2})\theta \theta {)}^2\\ & =& {\theta \theta }^T({\mu \mu }_{C_1}-{\mu \mu }_{C_2}{)}^T({\mu \mu }_{C_1}-{\mu \mu }_{C_2})\theta \theta \\ {\tilde{\sigma }}_{C_1}^2& =& \frac{1}{N_{C_1}}{\sum }_1^{N_{C_1}}({XX}_{C_{1}i}\theta \theta -{\tilde{\mu }}_{C_1}{)}^2\\ & =& \frac{1}{N_{C_1}}{\sum }_1^{N_{C_1}}({XX}_{C_{1}i}\theta \theta -\frac{1}{N_{C_1}}{\sum }_1^{N_{C_1}}{XX}_{C_{1}i}\theta \theta {)}^2\\ & =& \frac{1}{N_{C_1}}{\sum }_1^{N_{C_1}}(({XX}_{C_{1}i}-\frac{1}{N_{C_1}}{\sum }_1^{N_{C_1}}{XX}_{C_{1}i})\theta \theta {)}^2\\ & =& \frac{1}{N_{C_1}}{\sum }_1^{N_{C_1}}(({XX}_{C_{1}i}-{\mu \mu }_{C_1})\theta \theta {)}^2\\ & =& \frac{1}{N_{C_1}}{\sum }_1^{N_{C_1}}{\theta \theta }^T({XX}_{C_{1}i}-{\mu \mu }_{C_1}{)}^T({XX}_{C_{1}i}-{\mu \mu }_{C_1})\theta \theta \\ & =& {\theta \theta }^T(\frac{1}{N_{C_1}}{\sum }_1^{N_{C_1}}({XX}_{C_{1}i}-{\mu \mu }_{C_1}{)}^T({XX}_{C_{1}i}-{\mu \mu }_{C_1}))\theta \theta \\ & =& {\theta \theta }^T{\Sigma \Sigma }_{C_1}\theta \theta \\ {\tilde{\sigma }}_{C_2}^2& =& {\theta \theta }^T{\Sigma \Sigma }_{C_2}\theta \theta \\ 分母& =& {\theta \theta }^T{\Sigma \Sigma }_{C_1}\theta \theta +{\theta \theta }^T{\Sigma \Sigma }_{C_2}\theta \theta \\ & =& {\theta \theta }^T({\Sigma \Sigma }_{C_1}+{\Sigma \Sigma }_{C_2})\theta \theta \end{array}$$
$\therefore$

$$\begin{array}{ccl}J(\theta \theta )& =& \frac{{\theta \theta }^T({\mu \mu }_{C_1}-{\mu \mu }_{C_2}{)}^T({\mu \mu }_{C_1}-{\mu \mu }_{C_2})\theta \theta }{{\theta \theta }^T({\Sigma \Sigma }_{C_1}+{\Sigma \Sigma }_{C_2})\theta \theta }\end{array}$$

设$S_b=({\mu \mu }_{C_1}-{\mu \mu }_{C_2}{)}^T({\mu \mu }_{C_1}-{\mu \mu }_{C_2})$,$S_w={\Sigma \Sigma }_{C_1}+{\Sigma \Sigma }_{C_2}$

$S_b$就是类内方差

$S_w$就是类间方差

此时$J(\theta \theta )=\frac{{\theta \theta }^T{S}_b\theta \theta }{{\theta \theta }^T{S}_w\theta \theta }$

求导

$$\begin{array}{rcl}\frac{\partial J(\theta \theta )}{\partial \theta \theta }& =& \frac{\partial \frac{{\theta \theta }^T{SS}_b\theta \theta }{{\theta \theta }^T{SS}_w\theta \theta }}{\partial \theta \theta }\\ & =& \frac{\partial ({\theta \theta }^T{SS}_b\theta \theta ({\theta \theta }^T{SS}_w\theta \theta {)}^{-1})}{\partial \theta \theta }\\ & =& \frac{\partial ({\theta \theta }^T{SS}_b\theta \theta )}{\partial \theta \theta }({\theta \theta }^T{SS}_w\theta \theta {)}^{-1}+{\theta \theta }^T{SS}_b\theta \theta \frac{\partial (({\theta \theta }^T{SS}_w\theta \theta {)}^{-1})}{\partial \theta \theta }\\ & =& 2{\theta \theta }^T{SS}_b({\theta \theta }^T{SS}_w\theta \theta {)}^{-1}+{\theta \theta }^T{SS}_b\theta \theta (-\frac{1}{({\theta \theta }^T{SS}_w\theta \theta {)}^2})(2{\theta \theta }^T{SS}_w)\end{array}$$

令导数等于零

$$\begin{array}{rcl}00& =& 2{SS}_b\theta \theta ({\theta \theta }^T{SS}_w\theta \theta {)}^{-1}+{\theta \theta }^T{SS}_b\theta \theta (-\frac{1}{({\theta \theta }^T{SS}_w\theta \theta {)}^2})(2{SS}_w\theta \theta )\\ 2{SS}_b\theta \theta ({\theta \theta }^T{SS}_w\theta \theta {)}^{-1}& =& {\theta \theta }^T{SS}_b\theta \theta (\frac{1}{({\theta \theta }^T{SS}_w\theta \theta {)}^2})(2{SS}_w\theta \theta )\\ {SS}_b\theta \theta ({\theta \theta }^T{SS}_w\theta \theta )& =& ({\theta \theta }^T{SS}_b\theta \theta ){SS}_w\theta \theta \\ ({\theta \theta }^T{SS}_b\theta \theta ){SS}_w\theta \theta & =& {SS}_b\theta \theta ({\theta \theta }^T{SS}_w\theta \theta )\\ \theta \theta & =& {SS}_w^{-1}\frac{{\theta \theta }^T{SS}_w\theta \theta }{{\theta \theta }^T{SS}_b\theta \theta }{SS}_b\theta \theta \\ \theta \theta & =& {SS}_w^{-1}\frac{{\theta \theta }^T{SS}_w\theta \theta }{{\theta \theta }^T{SS}_b\theta \theta }({\mu \mu }_{C_1}-{\mu \mu }_{C_2}{)}^T({\mu \mu }_{C_1}-{\mu \mu }_{C_2})\theta \theta \end{array}$$
$\because$
$\frac{{\theta \theta }^T{SS}_w\theta \theta }{{\theta \theta }^T{SS}_b\theta \theta }$,$({\mu \mu }_{C_1}-{\mu \mu }_{C_2})\theta \theta$是一个数，不影响$\theta \theta$的方向

$\therefore$
$$\theta \theta \propto {SS}_w^{-1}({\mu \mu }_{C_1}-{\mu \mu }_{C_2}{)}^T$$

$if{SS}_w\propto II$

$\theta \theta \propto ({\mu \mu }_{C_1}-{\mu \mu }_{C_2}{)}^T$

求任意一个点的投影

$$pro{j}_{\theta \theta }(x)=x^T\theta \theta$$

求阈值

$$\begin{array}{rcl}threshold& =& \frac{N_{C_1}{\tilde{\mu }}_{C_1}+N_{C_2}{\tilde{\mu }}_{C_1}}{N_{C_1}+N_{C_2}}\\ & =& \frac{N_{C_1}{\mu \mu }_{C_1}\theta \theta +N_{C_2}{\mu \mu }_{C_1}\theta \theta }{N_{C_1}+N_{C_2}}\end{array}$$

依赖

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

人工数据集

n = 100
X = np.random.multivariate_normal((1, 1), [[0.64, 0], [0, 0.64]], size = int(n/2))
X = np.insert(X, 50, np.random.multivariate_normal((3, 3), [[0.64, 0], [0,0.64]], size = int(n/2)),0)
#X = np.insert(X, 0, 1, 1)
m = X.shape[1]
y = np.array([1]*50+[-1]*50).reshape(-1,1)
plt.scatter(X[:50, -2], X[:50, -1])
plt.scatter(X[50:, -2], X[50:, -1], c = "#ff4400")

<matplotlib.collections.PathCollection at 0x7f2b50e680d0>

X1 = X[(y==1).reshape(-1)]
X0 = X[(y==-1).reshape(-1)]
n1 = np.array([[X1.shape[0]]])
n0 = np.array([[X0.shape[0]]])
mu1 = X1.mean(axis = 0).reshape(-1,1)
mu0 = X0.mean(axis = 0).reshape(-1,1)
Sigma1 = np.cov(X1.T)
Sigma0 = np.cov(X0.T)
theta = (Sigma1 + Sigma0) @ (mu1 - mu0)
threshold = (n1*mu1 + n0*mu0).T@theta/(n1 + n0)
def getForecast(x):
    return x.T @ theta
threshold

array([[-10.45793931]])

预测

print(f'{ 1 if getForecast(np.array([[1],[1]])) > threshold else 0}')

1

分界展示

plt.scatter(X[:50, -2], X[:50, -1])
plt.scatter(X[50:, -2], X[50:, -1], c = "#ff4400")
for i in np.arange(-1,5,0.02):
    for j in np.arange(-1,5,0.02):
        if abs(getForecast(np.array([[i],[j]])) - threshold) <0.01:
            plt.scatter(i,j,c="#000000")