Fantope Projection and Selection: A near-optimal convex relaxation of sparse PCA

1. Goal

This paper mainly deals with Sparse Principal Component Analysis(PCA) using subspace method.

2. Theorey

2.1 How to get their formulation

Notation: $\lambda_1,\lambda_2,\cdots,\lambda_p$ are in decreasing order.
From Ky Fan’s maximum principal 1, we know that

$$\sum_{i=1}^d\lambda_i(\Sigma)=\mathrm{tr}({V^*}' \Sigma V^*) =\max_{V'V=I_d} \mathrm{tr}(V' \Sigma V) =\max_{V'V=I_d} \langle\Sigma, VV'\rangle$$

If we regard the last formula as a function of $VV'$, it is linear. So if we change the constrain to its convex hull does not change the optimization problem. From the less well known observation that

$$\mathscr{F}^d_p =\{H: \text{trace}(H)=d, 0\leq H\leq I_d\} = \text{conv}(\{VV': V'V=I_d\})$$

From all the analysis, we get

$$\sum_{i=1}^d\lambda_i(\Sigma)= \langle\Sigma, \Pi\rangle =\max_{H\in \mathscr{F}^d_p} \langle\Sigma, H\rangle$$

How to introduce the sparsity? And which norm is suitable to use? The goal of this paper is to get sparse PCs, then we should choose penalty making $V^*\in \mathbb{R}^{p\times d}$ sparse. For matrix, there are two ways to get sparsity:

• columnwise sparsity: for matrix $A$, each of its column is sparse, i.e. only few elements of $A_{*i}$ are nonzero.
• row sparsity: for matrix $A$, its rows are sparse, i.e. only few rows of $A$ are sparse, which produce the group sparsity.

For sparse PCA, to select the import features, this paper uses row sparsity. An intuitive penalty is $\|V\|_{2,0}$. But in high dimensional situation, $\ell_0$ norm is NP hard to deal. A common trick is replacing $\ell_0$ with $\ell_1$. Then the penalty becomes $\|V\|_{2,1}$. But our model is function of $H = VV'$. So what sparsity on $H$ can approximate well of $\|V\|_{2,1}$. Note that

∥H∥1,1=∑ijk|Vik||V