PCA和LDA

1，PCA是非监督方法，LDA（Linear discriminant analysis ）是有监督方法。二者都可以用来降维。

2，LDA可以用在多分类问题上。

What is PCA?

PCA is an unsupervised method that tries to find the directions of maximum variance in the data. It projects the data onto a new set of orthogonal axes, called principal components, that capture as much of the variability as possible. The first principal component is the one that explains the most variance, the second one is the one that explains the most variance after removing the effect of the first one, and so on. By selecting a subset of principal components, we can reduce the dimensionality of the data and keep only the most relevant features.

What is LDA?

LDA is a supervised method that tries to find the directions of maximum separation between the classes in the data. It assumes that the data within each class follows a multivariate normal distribution with a common covariance matrix. It projects the data onto a new set of axes, called linear discriminants, that maximize the ratio of the between-class variance to the within-class variance. The first linear discriminant is the one that separates the classes the most, the second one is the one that separates the classes the most after removing the effect of the first one, and so on. By selecting a subset of linear discriminants, we can reduce the dimensionality of the data and enhance the class discrimination.

How to compare PCA and LDA?

PCA and LDA have different goals and assumptions, so they may produce different results depending on the data and the problem. PCA is more suitable for exploratory data analysis, where we want to discover the main patterns and sources of variation in the data. LDA is more suitable for classification tasks, where we want to distinguish the data into different groups based on some labels. PCA does not use any class information, so it may not capture the features that are relevant for discrimination. LDA uses the class information, so it may not capture the features that are relevant for variance.

How to interpret PCA results?

Interpreting the results of PCA involves looking at eigenvalues and explained variance ratio of the principal components, eigenvectors and loadings of the principal components, as well as scores and biplots of the principal components. The eigenvalues and explained variance ratio indicate how much variance each component explains and how much information is retained by selecting a certain number of components. The eigenvectors and loadings show how each component is related to the original variables, as well as how much each variable contributes to each component. Lastly, the scores and biplots indicate how each observation is represented in the new space, as well as how the observations are clustered or separated along the components.

How to interpret LDA results?

Interpreting the results of LDA involves looking at the eigenvalues and explained variance ratio of the linear discriminants, which indicate how much separation each discriminant achieves and how much information is retained in selecting a certain number of discriminants. Additionally, the eigenvectors and coefficients of the linear discriminants show how each discriminant is related to the original variables and how much each variable contributes to each discriminant. Lastly, the scores and plots of the linear discriminants demonstrate how each observation is classified in the new space, as well as how well the classes are separated along the discriminants.

How to choose between PCA and LDA?

When deciding between PCA and LDA for a given problem, there is no definitive answer. It depends on the nature and purpose of the data, the assumptions and limitations of each method, and the trade-off between variance and discrimination. Generally, if labeled data is present for classification purposes, LDA may be more suitable than PCA as it uses class information and maximizes class separation. In contrast, if unlabeled data or exploration is desired, PCA may be a better choice as it does not require any class information and captures the most variance. Additionally, if there are more variables than observations, PCA may be more stable and robust than LDA due to overfitting and multicollinearity issues. Lastly, if there are more classes than observations, LDA may not perform well; thus, PCA can be used as a preprocessing step before applying LDA or other classifiers.

PCA vs LDA: Key Differences

Here are some key differences between PCA and LDA:

Objective: PCA is an unsupervised technique that aims to maximize the variance of the data along the principal components. The goal is to identify the directions that capture the most variation in the data. LDA, on the other hand, is a supervised technique that aims to maximize the separation between different classes in the data. The goal is to identify the directions that capture the most separation between the classes.
Supervision: PCA does not require any knowledge of the class labels of the data, while LDA requires labeled data in order to learn the separation between the classes.
Dimensionality Reduction: PCA reduces the dimensionality of the data by projecting it onto a lower-dimensional space, while LDA reduces the dimensionality of the data by creating a linear combination of the features that maximizes the separation between the classes.
Output: PCA outputs principal components, which are linear combinations of the original features. These principal components are orthogonal to each other and capture the most variation in the data. LDA outputs discriminant functions, which are linear combinations of the original features that maximize the separation between the classes.
Interpretation: PCA is often used for exploratory data analysis, as the principal components can be used to visualize the data and identify patterns. LDA is often used for classification tasks, as the discriminant functions can be used to separate the classes.
Performance: PCA is generally faster and more computationally efficient than LDA, as it does not require labeled data. However, LDA may be more effective at capturing the most important information in the data when class labels are available.

LDA vs PCA: When to use which method?

PCA is an unsupervised learning algorithm while LDA is a supervised learning algorithm. This means that PCA finds directions of maximum variance regardless of class labels while LDA finds directions of maximum class separability.

So now that you know how each method works, when should you use PCA vs LDA for dimensionality reduction? In general, you should use LDA when your goal is classification – that is, when you have labels for your data points and want to predict which label new points will have based on their feature values . On the other hand, if you don’t have labels for your data or if your goal is simply to find patterns in your data (not classification), then PCA will likely work better .

That said, there are some situations where LDA may outperform PCA even when you’re not doing classification . For example , imagine that your data has 100 features but only 10% of those features are actually informative (the rest are noise). If you run PCA on this dataset, it will identify all 100 components since its goal is simply to maximize variance . However , because only 10% of those components are actually informative, 90% of them will be useless. If you were to run LDA on this same dataset, it would only identify 10 components since its goal capturing class separability would be better served by discarding noisy features. Thus, if noise dominates your dataset then LDA may give better results even if your goal isn’t classification! Because LDA makes stronger assumptions about the structure of your data, it will often perform better than PCA when your dataset satisfies those assumptions but worse when it doesn’t.

Reference

https://www.linkedin.com/advice/0/how-do-you-interpret-results-pca-lda-terms-feature
https://vitalflux.com/pca-vs-lda-differences-plots-examples/