20:Learning a Self-Expressive Network for Subspace Clustering

      State-of-the-art subspace clustering methods are based on the self-expressive model, which represents each data point as a linear combination of other data points. since the number of self-expressive coefficients grows quadratically with the number of data points, their ability to handle large-scale datasets is often limited ,learn the self-expressive coefficients with desired properties on the training data, but also han- dle out-of-sample data .

     SENet yields highly competitive performance on MNIST, Fashion MNIST and Extended MNIST and state-of-the-art perfor- mance on CIFAR-10.

1. Introduction

 

      With the development of data collection and storage and processing technology, the availability of computer vision in large-scale database has increased dramatically. With the development of deep learning and other modern machine learning technology, based on the analysis of large data has achieved great success, but these methods need a lot of labeled data, and the data acquisition cost are high. Never markers extracting mode and big data clustering has become an important open problem. We are assuming that each cluster are made by a higher dimensional space environment under the condition of low dimensional subspace approximation, consider large-scale unmarked data clustering problem, namely the subspace clustering. The problem in the image motion segmentation hybrid clustering system to identify cancer subtypes clustering. Hyperspectral image segmentation has a wide range of applications.

      we introduce the self-expressive network (SENet) to learn a self-expressive model for subspace clustering, which can be leveraged to handle out-of-sample data and large-scale data.

      SENet trained on a certain dataset can be used to produce self-expressive coefficients for another dataset drawn from the same data distribution, therefore the method can handle out-of-sample data effectively.

Our experiments showcase the effectiveness of our method as summarized below:

We show that the self-expressive coefficients computed by a trained SENet closely approximate those computed by solving for them directly without the network. This il- lustrates the ability of SENet to approximate the desired self-expressive coefficients.

We show that a SENet trained on (part of) the training set of MNIST and Fashion MNIST can be used to pro- duce self-expressive coefficients on the test set that give a good clustering performance. This illustrates the abil- ity of SENet to handle out-of-sample data.

We show that SENet can be used to cluster datasets con- taining 70,000+ data poins, such as MNIST, Fashion MNIST and Extended MNIST, very efficiently, achiev- ing a performance that closely matches (for MNIST, Fashion MNIST and Extended MNIST) or surpasses (for CIFAR-10) the state of the art.

2.related work

Deep Clustering : deep networks are used to extract features. our work assumes that the input data already lie in linear subspaces, and focuses on computing the self-expressive coefficients.

Scalable Subspace Clustering: several methods adopt a two-step approach for computing self-expressive coefficients:

1) construct a dictionary, either generated in random or learned/selected from data.

2) express each data point as a linear combinations of the atoms in the dictionary.

However, the output dimension hence the scale of the optimization problem increases at least quadratically with the size of the dictionary, therefore using a sufficiently large dictionary may be impossible.

Self-attention Models : The self-attention mechanism used in Graph Attention Networks (GAT) , Transformer, Non-local Neural Networks. Similar to SENet, the coefficients in the linear combination are computed with a neural network. However, unlike the self- expressive models, which use the distance between the input features and output features to define a training loss in an unsupervised manner, the self-attention methods impose a supervised learning loss on the output features. This leads to a difference in the design of the network architecture.

3. Self-Expressive Network

本文的工作，实际上提出了一个自表达网络，如果你的样本集数目非常大，n非常大的时候，基本想法是化离散为连续，不去计算离散的自表达系数，而是学习一个自表达函数。我们期待这里获得的自表达函数，能够符合子空间保持性质，特别的，我们希望所学习的自表达函数能够有可以推广到未见新样本上的自表达性质。

文章设计了一个query-key结构的网络，再配上一个可学习的软阈值激活函数。

这里面因为有L1范数，所以实际上我们计算的是次梯度，可以使用随机梯度下降法来训练。具体在实验中，用的是计算了次梯度之后，使用Adam优化器。为了优化这样一个计算梯度很花费计算量的环节，文章设计了一个在线累计计算的改进算法，这个算法在训练阶段把内存需求变成一个固定的大小，内存需求可以独立于训练集的规模。

4. Experiments

本设计是为了提高SENet的表达能力，使其具有普遍逼近能力，使其能够轻松接近式中凸式的最优解，从而具有子空间保持性。先在一个简单的数据集上来训练自表达网络，然后把不同训练迭代次数的自表达系数矩阵可视化出来，可以看到一个块对角结构。如果我们把训练好的网络直接应用在测试集上，通过网络直接计算出自表达系数，可以看到右边结果，能看到比较清晰的块对角结构。那这个验证了训练好的网络它有推广到未见新样本的能力。

为了验证所训练的自表达网络能否获得全局最优解，在更难的一组合成数据集上进行了实验。比较所训练的自表达网络和跟它完全采样一样的超参数的弹性网络子空间聚类模型EnSC，在目标函数聚类准确率(ACC)还有子空间恢复误差（SRE），代数连通性(CONN)等等指标上作比较。

为了进一步验证所训练的自表达网络具有推广能力，我们从MNIST和Fashion MNIST的训练集中，采样一部分样本来构成训练集，然后我们分布使用MNIST的独立测试集和Fashion MNIST的独立测试集作为测试集。这时可以发现这里的子空间保持误差随着随着训练集的增大，而稳定的下降，结果越来越好，然后谱聚类给出的聚类准确率也是可以接受的，随着训练集的增大，会得到更好的结果。

 

接下来，我们为了展示自表达网络处理大规模数据集的潜力，分布从MNIST和Fashion MNIST，CIFAR-10，EMNIST数据集，选择一部分样本构成数据集。这里我们训练自表达网络记录了它的训练时间和所获得的聚类准确率，这里可以看到我们的方法可以在比较短的时间内，得到相当的结果或者是要胜出的准确率。

最后，记录了其他的可扩展子空间聚类方法在四个数据集上的聚类准确率。列举了一些常用的深度聚类方法的性能，可以看到基于自表达网络的结果跟现有最好的结果相比，性能相当或者可以胜出。

本文通过学习自表达网络来做子空间聚类：

首先提出了一个query-key网络结构，加上一个可学习的软阈值激活函数来重参数化自表达模型。

其次给出了一个训练算法，它可以独立于所占用的内存需求、独立于样本集的规模。

最后给出了实验，来验证了泛化能力和用来处理大规模数据集的潜力。