线性判别分析(Linear Discriminant Analysis, LDA)详解

最新推荐文章于 2025-04-02 14:37:45 发布

ningaiiii

最新推荐文章于 2025-04-02 14:37:45 发布

阅读量701

点赞数 4

分类专栏：机器学习与深度学习文章标签：机器学习 python 人工智能

本文链接：https://blog.csdn.net/m0_75139089/article/details/144867782

版权

机器学习与深度学习专栏收录该内容

73 篇文章 ¥9.90 ¥99.00

订阅专栏

超级会员免费看

线性判别分析(Linear Discriminant Analysis, LDA)详解

1. 引言

线性判别分析(LDA)是一种经典的降维方法和分类方法，它在模式识别、机器学习等领域有着广泛应用。LDA的核心思想是：将高维数据投影到低维空间中，使得同类数据尽可能紧凑，不同类数据尽可能分开。

2. 数学原理

2.1 基本思想

LDA的目标是找到一个投影方向 $w$ ，使得：

同类样本投影后尽可能近（类内方差最小）
不同类样本投影后尽可能远（类间方差最大）

2.2 数学推导

对于二分类问题：

类内散度矩阵 $S_w$ ：
$S_w = \sum_{i=1}^c \sum_{x \in X_i} (x - \mu_i)(x - \mu_i)^T$
类间散度矩阵 $S_b$ ：
$S_b = \sum_{i=1}^c N_i(\mu_i - \mu)(\mu_i - \mu)^T$
目标函数（Fisher准则）：
$\frac{w^T S_b w}{w^T S_w w}$
最优解：
$S_w^{-1}S_b w = \lambda w$

3. 算法实现

import numpy as np
from sklearn.datasets import make_classification
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.metrics import confusion_matrix, classification_report
from sklearn.model_selection import train_test_split

class LDA:
    def __init__(self, n_components=1):
        self.n_components = n_components
        self.w = None
        self.explained_variance_ratio_ = None
        
    def fit(self, X, y):
        n_samples, n_features = X.shape
        classes = np.unique(y)
        
        # 计算类内散度矩阵
        S_w = np.zeros((n_features, n_features))
        for cls in classes:
            X_cls = X[y == cls]
            mean_cls = X_cls.mean(axis=0)
            X_centered = X_cls - mean_cls
            S_w += X_centered.T @ X_centered
            
        # 计算类间散度矩阵
        mean_total = X.mean(axis=0)
        S_b = np.zeros((n_features, n_features))
        for cls in classes:
            X_cls = X[y == cls]
            mean_cls = X_cls.mean(axis=0)
            n_cls = X_cls.shape[0]
            mean_diff = (mean_cls - mean_total).reshape(-1, 1)
            S_b += n_cls * mean_diff @ mean_diff.T
            
        # 求解特征值和特征向量
        eigvals, eigvecs = np.linalg.eig(np.linalg.inv(S_w) @ S_b)
        
        # 选择最大的n_components个特征值对应的特征向量
        idx = eigvals.argsort()[::-1]
        self.w = eigvecs[:, idx[:self.n_components]]
        
        # 计算解释方差比
        self.explained_variance_ratio_ = np.real(eigvals[idx[:self.n_components]] / np.sum(eigvals))
        
    def transform(self, X):
        return X @ self.w
    
    def predict(self, X):
        # 添加预测方法
        X_transformed = self.transform(X)
        # 使用简单的阈值进行分类（针对二分类问题）
        return (X_transformed[:, 0] > 0).astype(int)

4. 实验示例

# 数据准备
X, y = make_classification(n_samples=1000, n_features=20, n_classes=2, 
                         n_informative=2, n_redundant=10, random_state=42)

# 划分训练集和测试集
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# 创建并训练LDA模型
lda = LDA(n_components=2)
lda.fit(X_train, y_train)

# 转换数据
X_train_transformed = lda.transform(X_train)
X_test_transformed = lda.transform(X_test)

# 1. 绘制训练集和测试集的投影结果
plt.figure(figsize=(15, 5))
plt.subplot(121)
plt.scatter(X_train_transformed[y_train==0, 0], X_train_transformed[y_train==0, 1], 
           c='red', label='Class 0', alpha=0.7)
plt.scatter(X_train_transformed[y_train==1, 0], X_train_transformed[y_train==1, 1], 
           c='blue', label='Class 1', alpha=0.7)
plt.xlabel('First discriminant')
plt.ylabel('Second discriminant')
plt.title('LDA Projection (Training Set)')
plt.legend()

plt.subplot(122)
plt.scatter(X_test_transformed[y_test==0, 0], X_test_transformed[y_test==0, 1], 
           c='red', label='Class 0', alpha=0.7)
plt.scatter(X_test_transformed[y_test==1, 0], X_test_transformed[y_test==1, 1], 
           c='blue', label='Class 1', alpha=0.7)
plt.xlabel('First discriminant')
plt.ylabel('Second discriminant')
plt.title('LDA Projection (Test Set)')
plt.legend()
plt.tight_layout()
plt.show()

# 2. 打印解释方差比
print("\n解释方差比:")
for i, ratio in enumerate(lda.explained_variance_ratio_):
    print(f"判别式 {i+1}: {ratio:.4f}")

# 3. 绘制第一判别式的分布
plt.figure(figsize=(10, 6))
for label in [0, 1]:
    plt.hist(X_train_transformed[y_train==label, 0], 
             bins=30, alpha=0.5, 
             label=f'Class {label}')
plt.xlabel('First Discriminant')
plt.ylabel('Frequency')
plt.title('Distribution of First Discriminant')
plt.legend()
plt.show()

# 4. 进行预测和评估
y_pred = lda.predict(X_test)

# 5. 绘制混淆矩阵
plt.figure(figsize=(8, 6))
cm = confusion_matrix(y_test, y_pred)
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues')
plt.title('Confusion Matrix')
plt.xlabel('Predicted')
plt.ylabel('True')
plt.show()

# 6. 打印分类报告
print("\n分类报告:")
print(classification_report(y_test, y_pred))

# 7. 计算并显示类间距离
class_0_mean = X_train_transformed[y_train==0].mean(axis=0)
class_1_mean = X_train_transformed[y_train==1].mean(axis=0)
distance = np.linalg.norm(class_0_mean - class_1_mean)
print(f"\n类间欧氏距离: {distance:.4f}")

# 8. 计算每个特征的判别能力
feature_importance = np.abs(lda.w)
plt.figure(figsize=(10, 6))
plt.bar(range(lda.w.shape[0]), feature_importance[:, 0])
plt.title('Feature Importance (First Discriminant)')
plt.xlabel('Feature Index')
plt.ylabel('Absolute Weight')
plt.show()