Classifiers Based on Bayes Decision Theory

Reference:

Section 2.1-2.6 of Pattern recognition, by Sergios Theodoridis, Konstantinos Koutroumbas (2009)

Slides of CS4220, TUD

Bayes Decision Theory

We will initially focus on the two-class case.

Let ${\omega }_1,{\omega }_2$ be the two classes in which our patterns belong.

• priori probabilities: $P({\omega }_1),P({\omega }_2)$

  Assumed to be known. If not, they can easily estimated as $P({\omega }_1)\approx N_1/N,P({\omega }_2)\approx N_2/N$.

• likelihood function: $p(\mathbf{x}|{\omega }_1),p(\mathbf{x}|{\omega }_2)$

  Assumed to be known. If not, they can also be estimated from the available training data.

• posteriori probabilities: $P({\omega }_1|\mathbf{x}),P({\omega }_2|\mathbf{x})$

  Can be computed by the Bayes rule:
  $$P({\omega }_i|\mathbf{x})=\frac{p(\mathbf{x}|{\omega }_i)P({\omega }_i)}{p(\mathbf{x})}\text{\hspace{1em}(BD.1)}$$
  where $p(\mathbf{x})$ is the pdf of $\mathbf{x}$ and for which we have
  $$p(\mathbf{x})=\sum p(\mathbf{x}|{\omega }_i)P({\omega }_i)\text{\hspace{1em}(BD.2)}$$

The Bayes classification rule can now be stated as
$$\begin{array}{rl}& \text{ If }P\left({\omega }_1|x\right)>P\left({\omega }_2|x\right),x\text{ is classified to }{\omega }_1\\ & \text{ If }P\left({\omega }_1|x\right)<P\left({\omega }_2|x\right),x\text{ is classified to }{\omega }_2\end{array}\text{\hspace{1em}(BD.3)}$$
Using $(BD.2)$, the decision can equivalently be based on the inequalities
$$p(\mathbf{x}|{\omega }_1)P({\omega }_1)\gtrless p(\mathbf{x}|{\omega }_2)P({\omega }_2)\text{\hspace{1em}(BD.4)}$$
It can also be described as
$$\begin{gathered} p(\mathbf{x}|{\omega }_1)P({\omega }_1)-p(\mathbf{x}|{\omega }_2)P({\omega }_2)\gtrless 0 \\ \frac{p(\mathbf{x}|{\omega }_1)P({\omega }_1)}{p(\mathbf{x}|{\omega }_2)P({\omega }_2)}\gtrless 1 \\ \log (p(\mathbf{x}|{\omega }_1)P({\omega }_1))-\log (p(\mathbf{x}|{\omega }_2)P({\omega }_2))\gtrless 0 \end{gathered}$$

---

Minimizing the classification error probability

We will show that the Bayesian classifier is optimal with respect to minimizing the classification error probability.

Define the classification error as
$$P(error)=\sum _{i=1}^{C}P(error|{\omega }_i)P({\omega }_i)\text{\hspace{1em}(BD.5)}$$

For a determined decision boundary (the green line), it could happen that $\mathbf{x}\in {\omega }_2$ is assigned to ${\omega }_1$, and that $\mathbf{x}\in {\omega }_1$ is assigned to ${\omega }_2$. Let $R_i$ be the region of the feature space in which we decide in favor of
${\omega }_i$. The classification error probability can be written as
$$\begin{array}{rl}{P}_e& =P({\omega }_1){\int }_{R_2}p\left(\mathbf{x}|{\omega }_1\right)d\mathbf{x}+P({\omega }_2){\int }_{R_1}p\left(\mathbf{x}|{\omega }_2\right)d\mathbf{x}\\ & ={\int }_{R_2}P\left({\omega }_1|\mathbf{x}\right)p(\mathbf{x})d\mathbf{x}+{\int }_{R_1}P\left({\omega }_2|\mathbf{x}\right)p(\mathbf{x})d\mathbf{x}\end{array}\text{\hspace{1em}(BD.6)}$$
which is the marked yellow area in the figure above. Then it’s intuitive that the optimal decision boundary should be
$${\mathbf{x}}_0:P\left({\omega }_1|{\mathbf{x}}_0\right)=P\left({\omega }_2|{\mathbf{x}}_0\right)\text{\hspace{1em}(BD.7)}$$

so that
$$\begin{array}{l}{R}_1:P\left({\omega }_1|\mathbf{x}\right)>P\left({\omega }_2|\mathbf{x}\right)\\ R_2:P\left({\omega }_2|\mathbf{x}\right)>P\left({\omega }_1|\mathbf{x}\right)\end{array}\text{\hspace{1em}(BD.8)}$$
By combining $(BD.6)$ and $(BD.7)$, we obtain the Bayes error ${\epsilon }^{\ast }$, which is the minimum classification error we can get. It does not depend on the classification rule that we apply, but on the distribution of data. In practical we can not obtain ${\epsilon }^{\ast }$, since we do not have the true distributions, and the high dimensional integrals are very complicated.

---

Minimizing the average risk

Sometimes misclassification of class A to class B is much more dangerous than misclassification of class B to class A. Thus, it is more appropriate to assign a penalty term to weigh each error:
$$\mathbf{r}={\lambda }_{12}P({\omega }_1){\int }_{R_2}p\left(\mathbf{x}|{\omega }_1\right)dx+{\lambda }_{21}P({\omega }_2){\int }_{R_1}p$$