Feature Selection and Extraction

Reference:

Pattern recognition, by Sergios Theodoridis, Konstantinos Koutroumbas (2009)

Slides of CS4220, TUD

The Peaking Phenomenon (5.3)

More features $\stackrel{?}{⟹}$ Better performance

• If the corresponding PDFs are known, the Bayesian error goes down with more features. We can perfectly discriminate the two classes by arbitrarily increasing the number of features.
• If the PDFs are unknown and the associated parameters must be estimated using a finite training set, then we must try to keep the number of features to a relatively low number.

In practice, for a finite $N$, by increasing the number of features one obtains an initial improvement in performance, but after a critical value further increase of the number of features results in an increase of the probability of error. This phenomenon is also known as the peaking phenomenon.

What feature subset to keep? $⟹$ Feature selection / extraction

• Feature selection: select $d$ out of $p$ measurements
• Feature extraction: map $p$ measurements to $d$ measurements

What do we need?

• Criterion functions, e.g., error, class overlap, information loss…
• Optimization or “search” algorithms to find mapping for given criterion

Class Separability Measures (5.6)

Divergence

Let us recall our familiar Bayes rule. Given two classes ${\omega }_1$ and ${\omega }_2$ and a feature vector $\mathbf{x}$, we select ${\omega }_1$ if
$$P({\omega }_1|\mathbf{x})>P({\omega }_2|\mathbf{x})$$
The classification error probability depends on the difference between $P({\omega }_1|\mathbf{x})$ and $P({\omega }_2|\mathbf{x})$, hence the ratio $P({\omega }_1|\mathbf{x})/P({\omega }_2|\mathbf{x})$ can convey useful information concerning the discriminatory capabilities associated with an adopted feature vector $\mathbf{x}$. Alternatively, the same information resides in the ratio
$$\ln \frac{p(\mathbf{x}|{\omega }_1)}{p(\mathbf{x}|{\omega }_2)}\equiv D_{12}(\mathbf{x})$$
Since $\mathbf{x}$ takes different values, it is natural to consider the mean value over class ${\omega }_1$ (because $\mathbf{x}$ is classified to class ${\omega }_1$), that is
$$D_{12}={\int }_{-\infty }^{+\infty }p(\mathbf{x}|{\omega }_1)\ln \frac{p(\mathbf{x}|{\omega }_1)}{p(\mathbf{x}|{\omega }_2)}d\mathbf{x}\text{\hspace{1em}(DV.1)}$$
Similar arguments hold for class ${\omega }_2$, and we define
$$D_{21}={\int }_{-\infty }^{+\infty }p(\mathbf{x}|{\omega }_2)\ln \frac{p(\mathbf{x}|{\omega }_2)}{p(\mathbf{x}|{\omega }_1)}d\mathbf{x}\text{\hspace{1em}(DV.2)}$$
The sum
$$d_{12}=D_{12}+D_{21}\text{\hspace{1em}(DV.3)}$$
is known as the divergence and can be used as a separability measure for the classes ${\omega }_1,{\omega }_2$, with respect to the adopted feature vector $\mathbf{x}$.

For a multiclass problem, the divergence is computed for every class pair ${\omega }_i,{\omega }_j$
$$d_{ij}=D_{ij}+D_{ji}={\int }_{-\infty }^{+\infty }[p(\mathbf{x}|{\omega }_i)-p(\mathbf{x}|{\omega }_j)]\ln \frac{p(\mathbf{x}|{\omega }_i)}{p(\mathbf{x}|{\omega }_j)}d\mathbf{x}\text{\hspace{1em}(DV.4)}$$
and the average class separability can be computed using the average divergence
$$d=\sum _{i=1}^M\sum _{j=1}^{M}P({\omega }_i)P({\omega }_j)d_{ij}\text{\hspace{1em}(DV.5)}$$
If the components of the feature vector are statistically independent, then it can be shown that
$$\begin{array}{rl}{d}_{ij}(x_1,x_2,\cdots ,x_l)& ={\int }_{-\infty }^{+\infty }[p(\mathbf{x}|{\omega }_i)-p(\mathbf{x}|{\omega }_j)]\sum _{r=1}^l\ln \frac{p(x_r|{\omega }_i)}{p(x_r|{\omega }_j)}d\mathbf{x}\\ & \stackrel{a}{=}\sum _{r=1}^l{\int }_{-\infty }^{+\infty }[p(x_r|{\omega }_i)-p(x_r|{\omega }_j)]\ln \frac{p(x_r|{\omega }_i)}{p(x_r|{\omega }_j)}d{x}_r\\ & =\sum _{r=1}^l{d}_{ij}(x_r)\end{array}\text{\hspace{1em}(DV.6)}$$
where $\stackrel{a}{=}$ is due to
$${\int }_{-\infty }^{+\infty }\cdots {\int }_{-\infty }^{+\infty }p(\mathbf{x}|{\omega }_i)-p(\mathbf{x}|{\omega }_j)d{x}_1\cdots d{x}_{r-1}d{x}_{r+1}\cdots d{x}_l=p(x_r|{\omega }_i)-p(x_r|{\omega }_j)$$
Assuming now that the density functions are Gaussians $\mathcal{N}({\mathbf{\mu }}_i,{\mathbf{\Sigma }}_i)$ and $\mathcal{N}({\mathbf{\mu }}_j,{\mathbf{\Sigma }}_j)$, respectively, the computation of the divergence is simplified, and it is not difficult to show that
d i j = 1 2 t r a c e { Σ i − 1 Σ j + Σ j − 1 Σ i − 2 I } + 1 2 ( μ i − μ j ) T ( Σ i − 1 + Σ j − 1 ) ( μ i − μ j ) (DV.7) d_{ij}=\frac{1}{2}\mathrm{trace}\{\boldsymbol \Sigma_i^{-1}\boldsymbol \Sigma_j+\boldsymbol \Sigma_j^{-1}\boldsymbol \Sigma_i-2I \}+\frac{1}{2}(\boldsymbol \mu_i-\boldsymbol \mu_j)^T(\boldsymbol \Sigma_i^{-1}+\boldsymbol \Sigma_j^{-1})(\boldsymbol \mu_i-\boldsymbol \mu_j)\tag{DV.7} dij​=21​trace{ Σi−1​Σj​+Σj−1​Σi​−2I}+21​(μi​−μj​)T(Σi−1​+Σj−1​)(μi​−μj​)(DV.7)
It can be seen that a class separability measure cannot depend only on the difference of the mean values; it must also be variance dependent. If the covariance matrices of the two Gaussian distributions are equal, then the divergence is further simplified to
$$d_{ij}=({\mathbf{\mu }}_i-{\mathbf{\mu }}_j{)}^T{\mathbf{\Sigma }}^{-1}({\mathbf{\mu }}_i-{\mathbf{\mu }}_j)$$
which is nothing other than the Mahalanobis distance between the corresponding mean vectors. This has a direct relation with the Bayes error, which is a desirable property for class separation measures.

---

Chernoff Bound and Bhattacharyya Distance

The minimum attainable classification error of the Bayes classifier for two classes ${\omega }_1$