t-SNE visualization of CNN codes

Refer from http://cs.stanford.edu/people/karpathy/cnnembed/

Description

I took 50,000 ILSVRC 2012 validation images, extracted the 4096-dimensional fc7 CNN ( Convolutional Neural Network) features using Caffe and then used Barnes-Hut t-SNE to compute a 2-dimensional embedding that respects the high-dimensional (L2) distances. In other words, t-SNE arranges images that have a similar CNN (fc7) code nearby in the embedding.

Results

Embeddings where images are displayed exactly at their embedded location:

1000x1000 (240KB)

4000x4000 (3.2MB)

6000x6000 (6.6MB)

And below, embeddings where every position is filled with its nearest neighbor. Note that since the actual embedding is roughly circular, this leads to a visualization where the corners are a little "stretched" out and over-represented:

1000x1000 (307KB)

4000x4000 (4.9MB)

6000x6000 (10.8MB)

It's impossible to precisely embed 4096-dimensional space in 2 dimensions so in this final visualization, I take the 4000x4000 image and also draw the "seams", which measure the actual (L2) distance between the full 4096-dimensional codes of neighboring images in the grid. A bright red edge means the distance is high, and a black edge means the distance is low, in the original space.

Code, features, embedding

- Here's a link to the 50,000 image filenames, the 2-dimensional embeddings and my Matlab visualization code if you'd like to produce your own images. (1MB)

- And here are the raw 4096-dimensional CNN codes for the 50,000 images (as .mat file) if you'd like to re-run your own t-SNE or something else. (261MB)

Feel free to use any of the images/code anywhere. Ping me at @karpathy for questions.