Notes 2 : Probability Distribution-CSDN博客

本文链接：https://blog.csdn.net/YHmoun/article/details/79562782

by Haicang

Nonparametric Methods

Nonparametric Methods

Histogram

A general formula

For a certain region $R$ , the probability of points fall in $R$ is $P$ , and the number of points in region $R$ is $K = NP$ . The formula of probability density is as follows:

p (x) = K N V (5.1)

$p(x) = \frac{K}{NV} \tag{5.1}$
Here,

K K $K$ is the number of points fall in the region,

N

$N$ is the number of the total points, while

V V $V$ is the volume of the current region.

So there are two ways to estimate $p(x)$ . One is to fix $V$ , the volume and change $K$ (Kernel method), while the other is vice versa, namely fix $K$ , the number of points and vary $V$ the volume (Nearest-Neighbors). With $N \to \infty$ density from the two methods, both will converge to the true distribution.

Kernel density estimators

Use $k(u)$ to denote a certain kernel function. There are some constraints:

k (u) \leq 0 \int k (u) d u = 1

$k(\textbf{u}) \le 0 \\ \int k(\textbf{u}) \, d\textbf{u} = 1$

k(u) k ( u ) $k(\textbf{u})$ is like a pdf(probability density function).

then $K$ in (5.1) is as follow:

\begin{matrix} (5.2) & K = \sum_{n = 1}^{N} k (\frac{x - x_{n}}{h}) \end{matrix}

$K = \sum_{n=1}^{N} k \left(\frac{x - x_n}{h} \right) \tag{5.2}$
Substituting this expression into (5.1), we get

p (x) = 1 N 1 V \cdot K e r n e l (5.3)

$p(\textbf{x}) = \frac{1}{N} \frac{1}{V} \cdot Kernel \tag{5.3}$

But there remains something I have to make sure.

$h$ is a smoothing parameter. There is a tradeoff between sensitivity to noise at small $h$ and over-smoothing at large $h$ .

Nearest-neighbor methods

Considering a small sphere centered on the point $\textbf{x}$ , we want to estimate density $p(\textbf{x}$ . Then we allow the sphere to grow until it contains $K$ data points. $K$ is also a smoothing parameter to trade-off. A large $K$ can cause over-smoothing, while a small $K$ suffers from noise. Model produced by KNN is not a true density model because the integral over all space diverges.

KNN classifier For a point $\textbf{x}$ , it’s label will be the the most frequent label in it’s KNN. As before, $K$ is also a smoothing parameter. This can be derived from the conditional probability version of (5.1), and then comes the intuitive result.

\begin{matrix} (5.4) & p (C_{k} | x) = \frac{K_{k}}{K} \end{matrix}

$p(C_k | \textbf{x}) = \frac{K_k}{K} \tag{5.4}$
Where,

K K $K$ is the number of total points in the region, while

K_{k}

$K_k$ is the number of points labeled

k k $k$ in the same region.

KNN is very quick in learning (Some one says it’s not really learning), while the prediction process is very slow compared with over method. But an interesting property is that, when $N \to \infty$ , the error rate is never more than twice the minimum Bayes error, which thought to be the lower bound of the mismatch of a machine learning algorithm.

The sci-kit learning link of KNN can be found here.

[1] Bishop. Pattern Recognition and Machine Learning