高维数据可视化工具：t-SNE

Introduce

Base on Visualizing Data using t-SNE.
How to Use t-SNE Effectively
example code

Student t-distribution Stochastic Neighbor Embedding (t-SNE) is a kinds of vsiualize tools, also a clssifier for high-dimmetion(high-dim) data (well proform in $N<100$), which can embed and clssify the high-dim data into 2 or 3-dim plane clusters. It’s a kinds of unsupervise Mechine Learing (ML) technique.

64 pixels hand writes digits classify & flower species classify (4-input characteristic)

Classifying of data, essentially is a kinds of dimensionality reduction, ML in here is used to, mininmize something (cost function) with larger parameters and non-linear.

Stochastic Neighbor Embedding (SNE), Basis idea of SNE and it’s math

SNE is expeted to find a faithful representation in low-dim for high-dim date, which preserve the small daistant (local) structure and reflect the large daistant of high-dim data.

conditional probability

Assume the one sample of high-dim data can be write down as a vector $y_i$ and it’s mapping point in low-dim is $x_i$. Then define to conditional probability $p_{j|i},q_{j|i}$ :

$$\begin{gathered} x_i\text{high-dim vector, a fixed piont as real data in high-dim} \\ {y}_i\text{mapping low-dim vector, movable and mapping point in low-dim} \end{gathered}$$

$$p_{j\mid i}=\frac{\exp \left(-{\Vert x_i-x_j\Vert }^2/2{\sigma }_i^2\right)}{{\sum }_{k\ne i}\exp \left(-{\Vert x_i-x_k\Vert }^2/2{\sigma }_i^2\right)},p_{i\mid i}=0$$

$$q_{j\mid i}=\frac{\exp \left(-{\Vert y_i-y_j\Vert }^2\right)}{{\sum }_{k\ne i}\exp \left(-{\Vert y_i-y_k\Vert }^2\right)},\sigma =\frac{1}{\sqrt{2}},q_{i\mid i}=0$$

where $\Vert x_i-x_j\Vert$ is the Euclidean distances. The $similarity$ of datapoint $x_j$ to datapoint $x_i$ is the conditional probability, $p_{j|i}$, that $x_i$ would pick $x_j$ as its neighbor if neighbors were picked in proportion to their probability density under a Gaussian centered at $x_i$. For example, for larger distance, $p_{j|i}\sim 0$. The ${\sigma }_i$ here, perform a rescaling effect.

If $y_i$ faithful to $x_i$, that should be $q_{j\mid i}\approx p_{j\mid i}$

The Cost function

It’s Kullback- Leibler divergence (which is in this case equal to the cross-entropy up to an additive constant, also named relative entropy and well uesd in such as $Ads-CFT$ Holographic theory, is equivalent to the difference between the Shannon entropy of two probability distributions).
$$C=\sum _{i}KL\left(P_i\Vert Q_i\right)=\sum _i\sum _j{p}_{j\mid i}\log \frac{p_{j\mid i}}{q_{j\mid i}}$$
Shannon entropy is $S=-{\sum }_j{p}_j\log p_j$
$$\sum _j{p}_{j\mid i}\log \frac{p_{j\mid i}}{q_{j\mid i}}=\sum _j{p}_{j\mid i}\log p_{j\mid i}-\sum _j{p}_{j\mid i}\log q_{j\mid i}$$

if $y_i$ faithful to $x_i$, have $q_{j\mid i}\approx p_{j\mid i}$, that $C=0$.

why minimize Cost function?

It’s Larger cost by mapping the widely separated $x_i$ points to dense $y_i$ points.

For simplicity, we consider such scenario:

Right panel: high-dim $x$ plane, $x_i$ nearby $x_j$, but others far away.
Left panel: low-dim $y$ plane, $y_i$ nearby $y_k$, but others including $y_j$ far away.

For $x$ plane, set $\sigma =1/\sqrt{2}$, have
$$p_{j\mid i}\approx \frac{\exp \left(-{\Vert x_i-x_j\Vert }^2\right)}{\exp \left(-{\Vert x_i-x_j\Vert }^2\right)}=1$$
Due to large distace between the other points, $\exp \left(-{\Vert x_i-x_k\Vert }^2\right)\sim 0$.

For $y$ plane, have
$$q_{j\mid i}\approx \frac{\exp \left(-{\Vert y_i-y_j\Vert }^2\right)}{\exp \left(-{\Vert y_i-y_k\Vert }^2\right)}=0$$
Due to the small distant of $\Vert y_i-y_k\Vert$ and large distance between the other.

Then the Cost function contribute by index $j,i$ is
$$C_{ji}=p_{j\mid i}\log \frac{p_{j\mid i}}{q_{j\mid i}}=\log \frac{1}{0}\sim \infty$$
So after the minimizing of Cost function, it will not appear. In mapping $y$-plane, the near piont of $y_i$ will be $y_j$, becasue in this time, $q_{j\mid i}\approx 1$, then $C_{ji}=0$.

Entropy and perplexity

If we using probability to discribe thing, it’s natural relate it to information entropy or Bayes. The define of Shannon entropy is:
$$H\left(P_i\right)=-\sum _j{p}_{j\mid i}{\log }_2{p}_{j\mid i}$$
The perplexity is defined as
$$\mathrm{Perp}\left(P_i\right)=2^{H\left(P_i\right)}$$
How complex one thing can be?

$$\begin{gathered} \text{entropy}\approx \text{Number of bits} \\ \text{perplexity}\approx \text{Number of states} \end{gathered}$$
It dicide by how many bits are needed to describe all of its possible states. For example, coin tossing, “heads” and “tails” two state, 1-bit entropy.

Example:

Consider there are 4 coins (equal probability situation)
$$\begin{gathered} \text{entropy}=4, \\ H=16\times -\frac{1}{2^4}{\log }_2\frac{1}{2^4}=4 \\ \text{Perplexity}=2^4\text{possible states} \end{gathered}$$

SNE performs a binary search for the value of ${\sigma }_i$ that produces a $P_i$ with a fixed perplexity ($5\sim 50$) that is specified by the user.

In dense regions, a smaller value of ${\sigma }_i$ is usually more appropriate.

Futher understanding of $\text{Perplexity}$ and ${\sigma }_i$

$$\begin{gathered} \text{Perplexity of }{x}_i=\text{effective number of points neighbor }{x}_i \\ \text{The range of neighbor is set by }{\sigma }_i \end{gathered}$$
Example:

if point $x_i$ with 16 same distance neighbor points, in a specify ${\sigma }_i$, which will make $p_{k|i}=1/16$, that the Perplexity of $x_i=16$, In fact, the Perplexity is specified by the user, algorithm will find ${\sigma }_i$

Effect of cahnge ${\sigma }_i$

This distribution has an entropy which increases as ${\sigma }_i$ increases.
Consider $N$ equidistant ($d$) points, only calculate the ponits’ with distance inner $3{\sigma }_i$, assmue have $M$ point:
$$\begin{gathered} p_{j\mid i}\approx \frac{\exp \left(-d^2/{\sigma }_i\right)}{(M-1)\exp \left(-d^2/{\sigma }_i\right)}\approx \frac{1}{M} \\ H={\log }_{2}M \end{gathered}$$
Next increases the ${\sigma }_i\to {\sigma }_i^{\prime }$, the inner $3{\sigma }_i^{\prime }$ points also increases $M\to M^{\prime }$, $H={\log }_2{M}^{\prime }$.

Minimize the Cost Function

Using the gradient descent method, the gradient of Cost function is:
$$\frac{\delta C}{\delta y_i}=2\sum _j\left(p_{j\mid i}-q_{j\mid i}+p_{i\mid j}-q_{i\mid j}\right)\left(y_i-y_j\right)$$
It’s easy calculate from difinition.
After the algorithm find the ${\sigma }_i$, changing the mapping $y_i$ alone with the invert direction of gradient, will reach the minimum point of $C$.

The dynamics of $y_i$ points can analogy to $N$-degree spring system, where
$$\begin{gathered} k=2\sum _j\left(p_{j\mid i}-q_{j\mid i}+p_{i\mid j}-q_{i\mid j}\right) \\ \frac{\delta C}{\delta y_i}=k\left(y_i-y_j\right) \end{gathered}$$
if $y_i$ faithful to $x_i$, have $q_{j\mid i}\approx p_{j\mid i}$ and $q_{i\mid j}\approx p_{i\mid j}$, similarity is equal, that $k=0,\frac{\delta C}{\delta y_i}=0$. Before this, the spring system will try to cluster itself.

Updata of $y_i$

$${\mathcal{Y}}^{(t)}={\mathcal{Y}}^{(t-1)}+\eta \frac{\delta C}{\delta \mathcal{Y}}+\alpha (t)\left({\mathcal{Y}}^{(t-1)}-{\mathcal{Y}}^{(t-2)}\right)$$
where ${\mathcal{Y}}^{(t)}$ indicates the solution at iteration t, $\eta$ indicates the learning rate, and $\alpha (t)$ represents the momentum at iteration t.

How the mapping points moving in $y$-plane with $t$?

t-Distribution SNE

Difference to SNE:

1. symmetrized version of the SNE cost function (set $p_{i|j}=p_{j|i},q_{i|j}=q_{j|i}$)
2. uses a Student-t distribution rather than a Gaussian to compute the similarity ($q_{ij}$) between two points in the $y_i$ space

Symmetrized the similarity
$$\begin{gathered} p_{ij}=\frac{p_{j\mid i}+p_{i\mid j}}{2n} \\ {q}_{ij}=\frac{q_{j\mid i}+q_{i\mid j}}{2n} \end{gathered}$$
Cost function become
$$\frac{\delta C}{\delta y_i}=4\sum _j\left(p_{ij}-q_{ij}\right)\left(y_i-y_j\right)$$

“Crowding Problem”

For example, map the $m$-dim data into $2$-dim, in somewhere data dense area, the volume of $x$-plane is $\sim r^m$, but the volume in $y$-plane is $\sim r^2$.

Solve

In the process of minimizing, add some guassian noise to simulate annealing, then add a slight repulsion to all springs.

The repulsion term is $q_{ij}(-k)$, set
$$q_{ij}>\frac{2\rho }{n(n-1)}$$
The minimize process is:
Start ($\rho =0$) $\to$ introduce gaussian noise in some mid-t steps $\to$ introduce backgroup repulsion ($\rho >0$) $\to$ Gap of cluster appear.

Problem of directly optimize UNI-SNE

If add the repulsion at begin, consider large distance points $y_i,y_j$, it’s
$$q_{ij}\sim \frac{2\rho }{n(n-1)}$$
nearly same the background repulsion. Small change of $\Vert y_i-y_j\Vert$ will no influence on Cost fuction $C$, $q_{ij}$ still $\sim \frac{2\rho }{n(n-1)}$.

Student t-distribution

$$q_{ij}=\frac{{\left(1+{\Vert y_i-y_j\Vert }^2\right)}^{-1}}{{\sum }_{k\ne l}{\left(1+{\Vert y_k-y_l\Vert }^2\right)}^{-1}}$$
In high-dim ($x$-plane) still use a Gussian distribution, but in $y$-plane use the above one.

Whne $\Vert y_i-y_j\Vert$ large, than
$$q_{ij}\sim 1/{\Vert y_i-y_j\Vert }^2\sim \text{long range force }1/r^2$$
New gradient:
$$\frac{\delta C}{\delta y_i}=4\sum _j\left(p_{ij}-q_{ij}\right)\left(y_i-y_j\right){\left(1+{\Vert y_i-y_j\Vert }^2\right)}^{-1}$$

Optimization Trick

“early compression”: force the map points to stay close together at the start of the optimization, make the clusters easier run/explore the global space of possible.

“early exaggeration” (used in t-SNE): multiply all of the $p_{ij}$ by a factor, for example, 4, in the initial stages of the optimization. Data tend to form tight widely separated clusters in the map. Makes points much easier move in cluster.

computational bottleneck

The $p_{ij},q_{ij}$ are point to points, so as data points increase, computational amount will go as
$$n(n-1)/2\sim n^2$$
For the data pointa $\ge 10,000$ will be time-consuming, one solution is random sampling a sub-set.

Ramda wlk

Random walk-based affinity measure. More efficient, faster. prevent “short-cut” problem.A,B,C are equidistant, but A,B in same cluster, while C in another cluster. The single noise point between two different clusters will not influence the cluster partitioning, because the similarity of two point ($x_i,x_j$) is not by the shortest path, while is integration of all path.

The emission probability from $x_i\to x_j$ is
$$e^{-{\Vert x_i-x_j\Vert }^2}$$

Other data dimension reduction methods

1. PCA 1979
2. classical scaling. Minimze the Cost fuction
   $$C=\frac{1}{{\sum }_{ij}\Vert x_i-x_j\Vert }\sum _{i\ne j}\frac{{\left(\Vert x_i-x_j\Vert -\Vert y_i-y_j\Vert \right)}^2}{\Vert x_i-x_j\Vert }=\frac{1}{N}\sum _{i\ne j}\frac{{\sigma }_{ij}^2}{d_{ij}}$$
   if $\Vert x_i-x_j\Vert$ is large, then $\Vert y_i-y_j\Vert$ is large but also shrink by a factor $\frac{1}{\Vert x_i-x_j\Vert }$. It will shrink large distance, amplify short distance in high-dim data.
3. Sammon mapping (Sammon, 1969)
4. t-SNE.
5. Isomap
6. LLE…

Diffusion method

Diffusion-based interpretation: the mapping pioints distance $\Vert y_i-y_j\Vert$ should be same as diffusion distance.

Definiton of diffusion distance:
$$D^{(t)}\left(x_i,x_j\right)=\sqrt{\sum _k\frac{{\left(p_{ik}^{(t)}-p_{jk}^{(t)}\right)}^2}{\psi {\left(x_k\right)}^{(0)}}}$$
Cost function:
$$C=\sum _i\sum _j{\left(D^{(t)}\left(x_i,x_j\right)-\Vert y_i-y_j\Vert \right)}^2$$
where $p_{ij}^{(t)}$ represents the probability of a particle traveling from $x_i$ to $x_j$ in t timesteps with Gaussian emission probabilities, $\psi {\left(x_k\right)}^{(0)}$ is a measure for the local density of the points.

how $x_i$ particle travel to $x_j$ ?

Emission probability of $x_i\to x_j$ (arrow line) is equal $e^{-{\Vert x_i-x_j\Vert }^2/{\sigma }^2}$ .
For solid line, $x_i\to x_j$ will take 4 time-steps, but high probability.
For dashed line, $x_i\to x_j$ will take 1 time-step, but low probability.

Assign much higher importance to the large diffusion distances rather than small, not good at retaining the local structure.

Weakness of t-SNE

1. Unclear the porform in high-dim $y$. (dimension of $y>3$)
2. Sensitive to the intrinsic dimension of data (dimension of $x_i>100$ maybe less successful).
3. Not guranteed converge the Cost function.

Experimental

compare with other methods

Visualizations of 6,000 handwritten digits from the MNIST data set.

classify task

64 pixels hand writes digits

Input shape, 1797 pictures, 64 pixels.

Iris flower classify

Input characteristic of iris flower

['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']

Input shape, (150,4)