The proof of partial theorems in ‘TENSOR-TRAIN DECOMPOSITION‘

  本文记录著名的张量 TT 分解论文 TENSOR-TRAIN DECOMPOSITION 的部分内容. 因为感觉原论文的部分证明写的不清楚, 故重写.

1. Theorem 2.1

  By the tensor train decomposition, the $n_1\times n_2\times \cdots \times n_d$ tensor $\mathcal{A}$ can be writen as$$\mathcal{A}\left(i_1,...,i_d\right)=\sum _{{\alpha }_0,...,{\alpha }_d}{\mathcal{G}}_1\left({\alpha }_0,i_1,{\alpha }_1\right){\mathcal{G}}_2\left({\alpha }_1,i_2,{\alpha }_2\right)\cdots {\mathcal{G}}_d\left({\alpha }_{d-1},i_d,{\alpha }_d\right).\text{\hspace{1em}(1.1)}$$ where ${\alpha }_0={\alpha }_d=1$. Let set the unfolding matrix of $\mathcal{A}$ as follow:$$A_k=A_k\left(\overline{i_1,...i_k},\overline{i_{k+1},...,i_d}\right)=\mathcal{A}\left(i_1,...i_k,i_{k+1},...,i_d\right).\text{\hspace{1em}(1.2)}$$ where the $\overline{i_1,...i_k}$ is the multi-index of the matrix, which is defined as $$\overline{i_1,...i_k}=i_1+\left(i_2-1\right)\cdot n_1+\cdots +\left(i_k-1\right)\cdot n_1{n}_2\cdots n_{k-1}.$$ The size of this matrix is $\left({\prod }_{s=1}^k{n}_s\times {\prod }_{s=k+1}^d{n}_s\right)$, and it can be obtained from the tensor $\mathcal{A}$ by a single call to the reshape function in MatLab:$$A_k=\text{reshape}\left(\mathcal{A},\left[\prod _{s=1}^k{n}_s,\prod _{s=k+1}^d{n}_s\right]\right).$$ Then the theorem 2.1 as follow.

  Theorem 2.1 If for each unfolding matrix $A_k$ of form $(1.2)$ of a d-dimensional tensor $\mathcal{A}$ with $$\text{rank}\left(A_k\right)=r_k,\text{\hspace{1em}(1.3)}$$ then there exists a decomposition $(1.1)$ with TT-ranks not higher than $r_k$.

  proof Consider the unfolding matrix $A_1$. As its rank is equal to $r_k$, it can be decomposed by eigenvalue decompesition as follow:$$A_1=U{V}^{\text{T}},\text{\hspace{1em}(1.4)}$$ with $U$ is a $(n_1\times r_1)$-matrix, $V^{\text{T}}$ is a $(r_1\times {\prod }_{s=2}^d{n}_s)$-matrix. Then the $(1.4)$ can be written as $$\begin{array}{rl}{A}_1\left(i_1,\overline{i_2\cdots i_d}\right)& =\sum _{{\alpha }_1=1}^{r_1}U\left(i_1,{\alpha }_1\right)V^{\text{T}}\left({\alpha }_1,\overline{i_2\cdots i_d}\right)\\ & =\sum _{{\alpha }_1=1}^{r_1}U\left(i_1,{\alpha }_1\right)V\left(\overline{i_2\cdots i_d},{\alpha }_1\right)\end{array}\text{\hspace{1em}(1.5)}$$ $V^{\text{T}}\left({\alpha }_1,\overline{i_2\cdots i_d}\right)$ can be treated as a tensor $\mathcal{V}$, so $(1.5)$ can be written as $$A_1\left(i_1,\overline{i_2\cdots i_d}\right)=\sum _{{\alpha }_1=1}^{r_1}U\left(i_1,{\alpha }_1\right)\mathcal{V}\left({\alpha }_1,i_2,\cdots ,i_d\right).\text{\hspace{1em}(1.6)}$$

  The matrix $V$ can be expressed as$$V=A_1^{\text{T}}U{\left(U^{\text{T}}U\right)}^{-1}=A_1^{\text{T}}W,$$ which means that $$\begin{array}{rl}V\left(\overline{i_2\cdots i_d},{\alpha }_1\right)& =\sum _{i_1=1}^{n_1}{A}_1^{\text{T}}\left(\overline{i_2\cdots i_d},i_1\right)W\left(i_1,{\alpha }_1\right)\\ & =\sum _{i_1=1}^{n_1}{A}_1\left(i_1,\overline{i_2\cdots i_d}\right)W\left(i_1,{\alpha }_1\right)\\ & =\sum _{i_1=1}^{n_1}\mathcal{A}\left(i_1,...,i_d\right)W\left(i_1,{\alpha }_1\right).\end{array}\text{\hspace{1em}(1.7)}$$ As $V\left(\overline{i_2\cdots i_d},{\alpha }_1\right)=V^{\text{T}}\left({\alpha }_1,\overline{i_2\cdots i_d}\right)=\mathcal{V}\left({\alpha }_1,i_2\cdots i_d\right)$, the tensor $\mathcal{V}$ can be written as $$\mathcal{V}\left({\alpha }_1,i_2\cdots i_d\right)=\sum _{i_1=1}^{n_1}\mathcal{A}\left(i_1,...,i_d\right)W\left(i_1,{\alpha }_1\right).\text{\hspace{1em}(1.8)}$$ Now the $\mathcal{V}$ can be treated as a $(d-1)$-dimensional tensor ${\mathbf{V}}_1$ with $({\alpha }_1{i}_2)$ as one long index: $${\mathbf{V}}_1\left({\alpha }_1{i}_2,i_3,\cdots ,i_d\right)=\mathcal{V}\left({\alpha }_1,i_2,i_3,\cdots ,i_d\right).$$ Then ${\mathbf{V}}_1$ is a $(r_1\times n_2)\times n_3\times \cdots \times n_d$-dimension tensor.

  Now consider its unfolding matrices $V_2,...,V_d$. We know that $A_k$ is a $\left({\prod }_{s=1}^k{n}_s,{\prod }_{s=k+1}^d{n}_s\right)$-matrix with $\text{rank}\left(A_k\right)=r_k$, then $A_k$ can be decomposed by $A_k=F_k\cdot G_k$, where $F_k$ is a $\left({\prod }_{s=1}^k{n}_s,r_k\right)$-matrix and $G_k$ is a $\left(r_k,{\prod }_{s=k+1}^d{n}_s\right)$-matrix. Therefore, $\mathcal{A}$ can be expressed as $$\begin{array}{rl}\mathcal{A}\left(i_1,...,i_d\right)& =A_k\left(\overline{i_1\cdots i_k},\overline{i_{k+1}\cdots i_d}\right)\\ & =\sum _{\beta =1}^{r_k}{F}_k\left(\overline{i_1\cdots i_k},\beta \right)G_k\left(\beta ,\overline{i_{k+1}\cdots i_d}\right)\\ & =\sum _{\beta =1}^{r_k}\mathcal{F}\left(i_1,...,i_k,\beta \right)\mathcal{G}\left(\beta ,i_{k+1},...,i_d\right)\end{array}\text{\hspace{1em}(1.9)}$$ Using equ$(1.8)$ and $(1.9)$, we obtain $$\begin{array}{rl}{V}_k& =V^{\text{T}}\left(\overline{{\alpha }_1{i}_2\cdots i_k},\overline{i_{k+1}\cdots i_d}\right)\\ & =\sum _{i_1=1}^{n_1}\mathcal{A}\left(i_1,\cdots ,i_d\right)W\left(i_1,{\alpha }_1\right)\\ & =\sum _{i_1=1}^{n_1}\left[\sum _{\beta =1}^{r_k}\mathcal{F}\left(i_1,...,i_k,\beta \right)\mathcal{G}\left(\beta ,i_{k+1},...,i_d\right)\right]W\left(i_1,{\alpha }_1\right)\\ & =\sum _{i_1=1}^{n_1}\sum _{\beta =1}^{r_k}\mathcal{F}\left(i_1,...,i_k,\beta \right)\mathcal{G}\left(\beta ,i_{k+1},...,i_d\right)W\left(i_1,{\alpha }_1\right)\\ & =\sum _{\beta =1}^{r_k}\sum _{i_1=1}^{n_1}W\left(i_1,{\alpha }_1\right)\mathcal{F}\left(i_1,...,i_k,\beta \right)\mathcal{G}\left(\beta ,i_{k+1},...,i_d\right)\\ & =\sum _{\beta =1}^{r_k}\mathcal{H}\left({\alpha }_1{i}_2,...,i_k,\beta \right)\mathcal{G}\left(\beta ,i_{k+1},...,i_d\right).\end{array}\text{\hspace{1em}(1.10)}$$ where
$$\mathcal{H}\left({\alpha }_1{i}_2,...,i_k,\beta \right)=\sum _{i_1=1}^{n_1}W\left(i_1,{\alpha }_1\right)\mathcal{F}\left(i_1,...,i_k,\beta \right).$$ Therefore, $\text{rank}(V_k)\le r_k$. The process can be continued by induction.

□

  According the process of this proof, we can construct the composition representing as equ.$(1.1)$. Let the ${\mathcal{G}}_1$ be equal to the $U$ of equ.$(1.4)$. Then $\mathcal{A}$ can be represented as $$\begin{array}{rl}\mathcal{A}\left(i_1,\cdots ,i_d\right)& =A_1\left(i_1,\overline{i_2\cdots i_d}\right)\\ & =\sum _{{\alpha }_1=1}^{r_1}U\left(i_1,{\alpha }_1\right)V^{\text{T}}\left({\alpha }_1,\overline{i_2\cdots i_d}\right)\\ & =\sum _{{\alpha }_1=1}^{r_1}U\left(i_1,{\alpha }_1\right)\mathcal{V}\left({\alpha }_1,i_2,\cdots ,i_d\right)\\ & =\sum _{{\alpha }_1=1}^{r_1}{\mathcal{G}}_1\left(i_1,{\alpha }_1\right){\mathbf{V}}_1\left({\alpha }_1{i}_2,i_3,\cdots ,i_d\right)\\ & =\sum _{{\alpha }_1=1}^{r_1}{\mathcal{G}}_1\left({\alpha }_0,i_1,{\alpha }_1\right){\mathbf{V}}_1\left({\alpha }_1{i}_2,i_3,\cdots ,i_d\right).\end{array}\text{\hspace{1em}(1.11)}$$ where ${\alpha }_0=1$.

  Now considering the tensor ${\mathbf{V}}_1$, let $V_2$ define as $$V_2\left(\overline{{\alpha }_1{i}_2},\overline{i_3\cdots i_d}\right)={\mathbf{V}}_1\left({\alpha }_1{i}_2,i_3,\cdots ,i_d\right).$$Its dimension is $(r_1\times n_2)\times (n_3\times \cdots \times n_d)$. In other words, $$V_2=\text{reshape}\left({\mathbf{V}}_1,\left[r_1{n}_2,\prod _{s=3}^d{n}_s\right]\right).$$ We set $\text{rank}(V_2)=r_2$, then $V_2$ can be expressed as $$V_2\left(\overline{{\alpha }_1{i}_2},\overline{i_3\cdots i_d}\right)=\sum _{{\alpha }_2=1}^{r_2}{U}^{\prime }\left(\overline{{\alpha }_1{i}_2},{\alpha }_2\right)V^{\prime }\left({\alpha }_2,\overline{i_3\cdots i_d}\right)\text{\hspace{1em}(1.12)}$$ Like ${\mathbf{V}}_1$, existing tensor ${\mathcal{V}}^{\prime }$ satisfies ${\mathcal{V}}^{\prime }\left({\alpha }_2,i_3,\cdots ,i_d\right)=V^{\prime }\left({\alpha }_2,\overline{i_3\cdots i_d}\right)$.And ${\mathcal{V}}^{\prime }$ can be treat as a $(d-2)$-dimension tensor ${\mathbf{V}}_2$, which can expressed as$${\mathbf{V}}_2\left({\alpha }_2{i}_3,i_4,...,i_d\right)={\mathcal{V}}^{\prime }\left({\alpha }_2,i_3,\cdots ,i_d\right).\text{\hspace{1em}(1.13)}$$ And $U^{\prime }$ in $(1.12)$ can be also treates as a tensor satisfying $${\mathcal{G}}_2\left({\alpha }_1,i_2,{\alpha }_2\right)=U^{\prime }\left(\overline{{\alpha }_1{i}_2},{\alpha }_2\right).$$Therefore, $\mathcal{A}$ can be expressed as $$\begin{array}{rl}\mathcal{A}\left(i_1,\cdots ,i_d\right)& =\sum _{{\alpha }_1=1}^{r_1}{\mathcal{G}}_1\left({\alpha }_0,i_1,{\alpha }_1\right){\mathbf{V}}_1\left({\alpha }_1{i}_2,i_3,\cdots ,i_d\right)\\ & =\sum _{{\alpha }_1=1}^{r_1}{\mathcal{G}}_1\left({\alpha }_0,i_1,{\alpha }_1\right)\left[\sum _{{\alpha }_2=1}^{r_2}{\mathcal{G}}_2\left({\alpha }_1,i_2,{\alpha }_2\right){\mathbf{V}}_2\left({\alpha }_2{i}_3,i_4,...,i_d\right)\right]\\ & =\sum _{{\alpha }_2=1}^{r_2}\sum _{{\alpha }_1=1}^{r_1}{\mathcal{G}}_1\left({\alpha }_0,i_1,{\alpha }_1\right){\mathcal{G}}_2\left({\alpha }_1,i_2,{\alpha }_2\right){\mathbf{V}}_2\left({\alpha }_2{i}_3,i_4,...,i_d\right)\end{array}.\text{\hspace{1em}(1.14)}$$
Repeating the proceduce up to ${\mathcal{G}}_d$, and we can get the TT-representation as $(1.1)$.