Unsupervised Nearest Neighbors Clustering With Application to Hyperspectral Images

A dynamic niching clustering algorithm based on individual-connectedness and its application to color image segmentation

Abstract
KSEM, a stochastic extension of the $k$NN density-based clustering (KNNCLUST) method which randomly assigns objects to clusters by sampling a posterior class label distribution.

---

Notations

$X$: Dataset, i.e. $X=\{x_i\}$, $x_i\in \mathbb{R}^d$, $i=1,\dots,n$.
$C_i$: Discrete random variable corresponding to the class label held by object $x_i$.
$c_i$: Outcome label sampled from some distribution on $C_i$.
$c$: $c={[c_1,\dots, c_n]}^T$ be the vector of cluster labels.
$p\left(C_i|x_i;\{x_j,c_j\}_{j\not=i}\right)$: Local posterior distribution of $C_i$.
$\kappa(i)$: Set of indices of the $k$NNs of $x_i$.
$\Omega(i)$: $\{c_j|j\in\kappa(i)\}$.

Algorithm

The local posterior label distribution in KSEM can be modelled primarily as:

$$\begin{equation} \hat{p}\left(C_i=c_{\mathscr{L}}|x_i;\{x_j,c_j\}_{j\in\kappa(i)}\right)\propto \sum_{j\in\kappa(i)}{g(x_j,x_i)\delta_{c_j c_\mathscr{L}}}\tag{1} \end{equation}$$
$\forall c_{\mathscr{L}}\in\Omega(i)$, $1\leq i\leq n$, where $g$ is a (non negative) kernel function defined on $\mathbb{R}^d$, $\delta_{ij}$ is the Kronecker delta. Though many kernel functions can be used, in this work, they have restricted to the following Gaussian kernel:
$$\begin{equation} g(x,x_i)=\frac{1}{\left(\sqrt{2\pi}d_{k,\kappa(x_i)}\right)^d}\exp{\left(-\frac{1}{2}\frac{\|x-x_i\|_2^2}{d_{k,\kappa(x_i)}^2(x_i)}\right)},\tag{2} \end{equation}$$
where $x\in\mathscr{R}^d$, and $d_{k,S(x_i)}$ represents the distance from $x_i$ to its $k$th NN. Then they propose the estimation of posterior label distribution as follows:
$$\begin{equation} \hat{p}_\alpha\left(C_i=c_{\mathscr{L}}|x_i;\{x_j,c_j\}_{j\in\kappa(i)}\right)=\frac{\left[\sum\limits_{j\in\kappa(i)}{g(x_j,x_i)\delta_{c_jc_\mathscr{L}}}\right]^\alpha}{\sum\limits_{c_m\in\Omega(i)}\left[\sum\limits_{j\in\kappa(i)}{g(x_j,x_i)\delta_{c_jc_m}}\right]^\alpha}\tag{3} \end{equation}$$
$\forall c_{\mathscr{L}}\in \Omega(i), 1\leq i\leq n$, where $\alpha \in [1, +\infty]$ is a parameter controlling the degree of determinism in the construction of the pseudo-sample: $\alpha=1$corresponds to the SEM (stochastic) scheme, while $\alpha \rightarrow +\infty$ corresponds to the CEM (deterministic) scheme, leading to a labeling scheme which is similar to the KNNCLUST’s rule. In this work, setting $\alpha=1.2$ is recommended.
Leting $S_{c\mathscr{L}}=\{x_i\in X|c_i=c_{\mathscr{L}}\}$, teh Kozachenko-Leonenko conditional differential entropy estimate writes:
$$\begin{equation} \hat{h}(X|c_{\mathscr{L}})=\frac{d}{n_\mathscr{L}}\sum_{x_i\in S_{c_\mathscr{L}}}\ln d_{k,S_{c_\mathscr{L}}}(x_i)+\ln(n_\mathscr{L}-1)-\psi(k)+\ln V_d\tag{4} \end{equation}$$
$\forall c_\mathscr{L} \in \Omega$, where $n_\mathscr{L}=|S_{c_\mathscr{L}}|$, $\psi(k)=\Gamma'(k)/\Gamma(k)$ is the digamma function, $\Gamma(k)$ is the gamma function and $V_d=\pi^{d/2}/\Gamma(d/2+1)$ is the volume of the unit ball in $\mathbb{R}^d$. An overall clustering entropy measure can be obtained from conditional entropies (4) as:
$$\begin{equation} \hat{h}(X|c)=\frac{1}{n}\sum_{c_\mathscr{L}\in\Omega}{n_\mathscr{L}\hat{h}(X|c_\mathscr{L})}\tag{5} \end{equation}$$
This measure can be used as a stopping criterion during the iterations quite naturally. Since objects are aggregated into preciously formed clusters during the iterations, the individual class-conditional entropies can only increase, and so does the conditional entropy(5). However, when convergence is achieved, this measure reaches an upper limit, and therefore a stopping criterion can be set up from its relative magnitude variation $\Delta_h=|\hat{h}(X|c^{(t)})-\hat{h}(X|c^{(t-1)})|/\hat{h}(X|c^{(t-1)})$, where $c^{(t)}$ is the vector of cluster labels at iteration $t$. The stopping criterion $\Delta_h<{10}^{-4}$ is recommended.

Application

Despite the reduction in complexity brought by the $k$NN search, the case of image segmentation by unsupervised clustering of pixels with KSEM remains computationally difficult, which can severely lower its usage for large size images. In the particular domain of multivariate imagery (multispectral/hyperspectral), the objects of interest are primarily grouped thanks to their spectral information characteristics. To help the clustering of image pixels, one often uses the spatial information, and the fact that two neighboring pixels are likely to belong to the same cluster. So they limit the search of a pixel’s $k$NNs to a subset of its spatial neighbors, selected via a predefined sampling pattern. Specifically, the pattern has a local sampling density inversely proportional to the distance from the central (query) point (as shown in figure below).