Distance between subspacess

Distance between subspaces

1. Definition of distance between subspaces

How to describe the distance between subspaces? The canonical angles between subspaces can be used to define the distance between subspaces. And there are several equivalent ways to do this. Here I will talk is the definition used in the reference “Minimax sparse principal subspace estimation in high dimension”, which is based on the projection matrices of subpaces, because of the convenience to represent subspaces by there projector matrices.

Definition: Let $\mathscr{E}$ and $\mathscr{F}$ be d-dimensional subspaces of $\mathbb{R}^p$ with orthogonal projectors $E$ and $F$. Denote the singular values of $EF^\perp$ by $s_1, s_2,\cdots$. The canonical angles between $\mathscr{E}$ and $\mathscr{F}$ are the numbers

$$\theta_k(\mathscr{E},\mathscr{F}) = \mathrm{arc}\sin(s_k)$$
for $k=1,2,\cdots,d$ and the $angle operator$ between $\mathscr{E}$ and $\mathscr{F}$ is the $d\times d$ matrix
$$\Theta(\mathscr{E},\mathscr{F}) =\mathrm{diag}(\theta_1,\theta_2,\cdots,\theta_d)$$ .

2. Theory of the distance

Since the nonzero singular values of $E-F$ are the nonzero singular values of $EF^\perp$, each counted twice. (which is the Proposition 2.1 in “Minimax sparse principal subspace estimation in high dimension”). Thus

$$\|\mathrm{sin}\Theta(\mathscr{E},\mathscr{F})\|_F^2 = \sum_{k=1}^d s_k^2 = \|EF^\perp\|_F^2 = \frac{1}{2}\|E-F\|_F^2$$

If we also assume that $V_E$ and $V_F$ are the orthogonal bases for subspaces $\mathscr{E}$ and $\mathscr{F}$ respectively. Then from the Proposition 2.2 in “Minimax sparse principal subspace estimation in high dimension”, we have

$$\frac{1}{2}\inf_{Q \in \mathbb{V}_{d.d}}\|V_E - V_F Q\|_F^2\leq \|\mathrm{sin}\Theta(\mathscr{E},\mathscr{F})\|_F^2 \leq \inf_{Q \in \mathbb{V}_{d.d}}\|V_E - V_F Q\|_F^2$$
This is to say the distance between two subspaces is equivalent to the minimal distance between their orthogonal bases.

3. Conclusion of distance

Let $V_E\in \mathbb{V}_{p,d}$ and $E_F\in \mathbb{V}_{p,d}$ be two column orthogonal matrix, which is to say $E^tE = F^tF=I_d$, with right rotation. Then the following is true,

$$\|V_E - V_F\|_F^2\leq \|\underbrace{V_EV_E^t - V_FV_F^t}_{\text{Projection matrices}}\|_F^2$$

4. Extension

If we assume that $V_E$ and $V_F$ span the d-dimensional principal subspaces $(\mathscr{E},\mathscr{F})$ of $\Sigma_E$ and $\Sigma_F$, then from the curvature lemma (lemma 4.2) given in “Minimax sparse principal subspace estimation in high dimension” , we have

$$\|\mathrm{sin}\Theta(\mathscr{E},\mathscr{F})\|_F^2 = \frac{1}{2}\|V_EV_E^t - V_FV_F^t\|_F^2\leq \frac{\langle\Sigma_E, V_EV_E^t - V_FV_F^t\rangle}{\lambda_d(\Sigma_E) - \lambda_{d+1}(\Sigma_E)}$$
$$\|\mathrm{sin}\Theta(\mathscr{E},\mathscr{F})\|_F^2 = \frac{1}{2}\|V_EV_E^t - V_FV_F^t\|_F^2\leq \frac{\langle\Sigma_F, V_FV_F^t - V_EV_E^t\rangle}{\lambda_d(\Sigma_F) - \lambda_{d+1}(\Sigma_F)}$$
Combining these together,

$$\begin{aligned} \|V_EV_E^t - V_FV_F^t\|_F^2 & \leq \frac{1}{\delta}\langle\Sigma_E - \Sigma_F, V_EV_E^t - V_FV_F^t\rangle \\\\ & \leq \frac{1}{\delta} \|\Sigma_E - \Sigma_F\|_F\|V_EV_E^t - V_FV_F^t\|_F\\\\ \end{aligned}$$

where $\delta = \min\{\lambda_d(\Sigma_E) - \lambda_{d+1}(\Sigma_E) , \lambda_d(\Sigma_F) - \lambda_{d+1}(\Sigma_F)\}$
from which , we know

$$\|V_EV_E^t - V_FV_F^t\|_F\leq \frac{1}{\delta} \|\Sigma_E - \Sigma_F\|_F$$ .

Thus

∥VE−VFOrthogonal bases∥2F≤∥VEVtE−VFVtFProjection matrices∥2F≤1δ∥ΣE−ΣFSymmetric matrices∥F

$$\|\underbrace{V_E - V_F}_{\text{Orthogonal bases}}\|_F^2\leq \|\underbrace{V_EV_E^t - V_FV_F^t}_{\text{Projection matrices}}\|_F^2 \leq \frac{1}{\delta} \|\underbrace{\Sigma_E - \Sigma_F}_{\text{Symmetric matrices}}\|_F$$