Gaussian Process(高斯过程) GPSS暑校笔记总计四（英文版）——多输出高斯过程

Multiple-Output Gaussian Process

Working Situation

As the picture shows, we want to learn from the three sensors (with complete signal information) to recover the fourth one.

Dependencies between processes

Multiple-independent Output GP

f 1 ( x ) ∼ G P ( 0 , k 1 ( x , x ′ ) )                      f 2 ( x ) ∼ G P ( 0 , k 2 ( x , x ′ ) ) D 1 = { ( x i , 1 , y 1 ( x i , 2 ) ) ∣ i = 1 , … , N 1 }                      D 2 = { ( x i , 2 , y 2 ( x i , 2 ) ) ∣ i = 1 , … , N 2 } y 1 ∼ N ( 0 , K 1 + σ 1 2 )                      y 2 ∼ N ( 0 , K 2 + σ 2 2 l ) \begin{aligned} f_{1}(\mathbf{x}) \sim \mathcal{G} \mathcal{P}\left(0, k_{1}\left(\mathbf{x}, \mathbf{x}^{\prime}\right)\right) &\;\;\;\;\;\;\;\;\;\; f_{2}(\mathbf{x}) \sim \mathcal{G} \mathcal{P}\left(0, k_{2}\left(\mathbf{x}, \mathbf{x}^{\prime}\right)\right) \\ D_{1}=\left\{\left(\mathbf{x}_{i, 1}, y_{1}\left(\mathbf{x}_{i, 2}\right)\right) | i=1, \ldots, N_{1}\right\} & \;\;\;\;\;\;\;\;\;\;\mathcal{D}_{2}=\left\{\left(\mathbf{x}_{i, 2}, y_{2}\left(\mathbf{x}_{i, 2}\right)\right) | i=1, \ldots, N_{2}\right\} \\ \mathbf{y}_{1} \sim \mathcal{N}\left(\mathbf{0}, \mathbf{K}_{1}+\sigma_{1}^{2}\right) & \;\;\;\;\;\;\;\;\;\;\mathbf{y}_{2} \sim \mathcal{N}\left(\mathbf{0}, \mathbf{K}_{2}+\sigma_{2}^{2} \mathbf{l}\right) \end{aligned} f1​(x)∼GP(0,k1​(x,x′))D1​={(xi,1​,y1​(xi,2​))∣i=1,…,N1​}y1​∼N(0,K1​+σ12​)​f2​(x)∼GP(0,k2​(x,x′))D2​={(xi,2​,y2​(xi,2​))∣i=1,…,N2​}y2​∼N(0,K2​+σ22​l)​

$$\left[\begin{array}{l}{\mathbf{y}}_1\\ {\mathbf{y}}_2\end{array}\right]\sim \mathcal{N}\left(\left[\begin{array}{l}0\\ 0\end{array}\right],\left[\begin{array}{cc}{\mathbf{K}}_1& 0\\ 0& {\mathbf{K}}_2\end{array}\right]+\left[\begin{array}{cc}{\sigma }_1^2\mathbf{l}& 0\\ 0& {\sigma }_2^2\mathbf{l}\end{array}\right]\right)$$

How to find the independences for kernel design

$${\mathbf{K}}_{\mathbf{f},\mathbf{f}}=\left[\begin{array}{cc}{\mathbf{K}}_1& ?\\ ?& {\mathbf{K}}_2\end{array}\right]$$

Build a cross-covariance function $cov[f_1(x),f_2(x^{{}^{\prime }})]$ such that $K_{f,f}$ is positive semi-definite.

Different input configurations of data

$$\begin{array}{ll}{\mathcal{D}}_1=\left\{{\left({\mathbf{x}}_i,f_1\left({\mathbf{x}}_i\right)\right)}_{i=1}^N\right\}& {\mathcal{D}}_1=\left\{{\left({\mathbf{x}}_{i,1},f_1\left({\mathbf{x}}_{i,1}\right)\right)}_{i=1}^{N_1}\right\}\\ {\mathcal{D}}_2=\left\{{\left({\mathbf{x}}_i,f_2\left({\mathbf{x}}_i\right)\right)}_{i=1}^N\right\}& {\mathcal{D}}_2=\left\{{\left({\mathbf{x}}_{i,2},f_2\left({\mathbf{x}}_{i,2}\right)\right)}_{i=1}^{N_2}\right\}\end{array}$$

Intrinsic Coregionalization Model

Two outputs

Sample Once

Consider two outputs $f_1(x) $$f_2(x)$ with $x\in {\mathcal{R}}^p$.

1. Sample from a GP $u(\mathbf{x})\sim \mathcal{G}\mathcal{P}\left(0,k\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)\right)$ to obtain $u^1(\mathbf{x})$
2. Obtain $f_1(x) $ and $f_2(x)$ by linearly transforming:

$$\begin{array}{rl}{f}_1(\mathbf{x})& =a_1^1{u}^1(\mathbf{x})\\ f_2(\mathbf{x})& =a_2^1{u}^1(\mathbf{x})\end{array}$$

For a fixed value $x$. we can group $f_1(x)$ and $f_2(x)$ in a vector:
$$\mathbf{f}(\mathbf{x})=\left[\begin{array}{l}{f}_1(\mathbf{x})\\ f_2(\mathbf{x})\end{array}\right]$$
and this vector will be refer as a $\mathbf{v}\mathbf{e}\mathbf{c}\mathbf{t}\mathbf{o}\mathbf{r}-\mathbf{v}\mathbf{a}\mathbf{l}\mathbf{u}\mathbf{e}\mathbf{d}\mathbf{f}\mathbf{u}\mathbf{n}\mathbf{c}\mathbf{t}\mathbf{i}\mathbf{o}\mathbf{n}$.

The covariance for $f(x)$ is computed as:
cov ⁡ ( f ( x ) , f ( x ′ ) ) = E { f ( x ) [ f ( x ′ ) ] ⊤ } − E { f ( x ) } [ E { f ( x ′ ) } ] ⊤ \operatorname{cov}\left(\mathbf{f}(\mathbf{x}), \mathbf{f}\left(\mathbf{x}^{\prime}\right)\right)=\mathbb{E}\left\{\mathbf{f}(\mathbf{x})\left[\mathbf{f}\left(\mathbf{x}^{\prime}\right)\right]^{\top}\right\}-\mathbb{E}\{\mathbf{f}(\mathbf{x})\}\left[\mathbb{E}\left\{\mathbf{f}\left(\mathbf{x}^{\prime}\right)\right\}\right]^{\top} cov(f(x),f(x′))=E{f(x)[f(x′)]⊤}−E{f(x)}[E{f(x′)}]⊤

E { [ f 1 ( x ) f 2 ( x ) ] [ f 1 ( x ′ ) f 2 ( x ′ ) ] } = [ E { f 1 ( x ) f 1 ( x ′ ) } E { f 1 ( x ) f 2 ( x ′ ) } E { f 2 ( x ) f 1 ( x ′ ) } E { f 2 ( x ) f 2 ( x ′ ) } ] E { f 1 ( x ) f 1 ( x ′ ) } = E { a 1 1 u 1 ( x ) a 1 1 u 1 ( x ′ ) } = ( a 1 1 ) 2 E { u 1 ( x ) u 1 ( x ′ ) } E { f 1 ( x ) f 2 ( x ′ ) } = E { a 1 1 u 1 ( x ) a 2 1 ( x ′ ) } = a 1 1 a 2 1 E { u 1 ( x ) u 1 ( x ′ ) } E { f 2 ( x ) f 2 ( x ′ ) } = E { a 2 1 u 1 ( x ) a 2 1 u 1 ( x ′ ) } = ( a 2 1 ) 2 E { u 1 ( x ) u 1 ( x ′ ) } \mathbb{E}\left\{\left[\begin{array}{c}{f_{1}(\mathbf{x})} \\ {f_{2}(\mathbf{x})}\end{array}\right]\left[\begin{array}{ll}{f_{1}\left(\mathbf{x}^{\prime}\right)} & {f_{2}\left(\mathbf{x}^{\prime}\right) ]}\end{array}\right\}=\left[\begin{array}{cc}{\mathbb{E}\left\{f_{1}(\mathbf{x}) f_{1}\left(\mathbf{x}^{\prime}\right)\right\}} & {\mathbb{E}\left\{f_{1}(\mathbf{x}) f_{2}\left(\mathbf{x}^{\prime}\right)\right\}} \\ {\mathbb{E}\left\{f_{2}(\mathbf{x}) f_{1}\left(\mathbf{x}^{\prime}\right)\right\}} & {\mathbb{E}\left\{f_{2}(\mathbf{x}) f_{2}\left(\mathbf{x}^{\prime}\right)\right\}}\end{array}\right]\right.\\ \begin{aligned} \mathbb{E}\left\{f_{1}(\mathbf{x}) f_{1}\left(\mathbf{x}^{\prime}\right)\right\} &=\mathbb{E}\left\{a_{1}^{1} u^{1}(\mathbf{x}) a_{1}^{1} u^{1}\left(\mathbf{x}^{\prime}\right)\right\}=\left(a_{1}^{1}\right)^{2} \mathbb{E}\left\{u^{1}(\mathbf{x}) u^{1}\left(\mathbf{x}^{\prime}\right)\right\} \\ \mathbb{E}\left\{f_{1}(\mathbf{x}) f_{2}\left(\mathbf{x}^{\prime}\right)\right\} &=\mathbb{E}\left\{a_{1}^{1} u^{1}(\mathbf{x}) a_{2}^{1}\left(\mathbf{x}^{\prime}\right)\right\}=a_{1}^{1} a_{2}^{1} \mathbb{E}\left\{u^{1}(\mathbf{x}) u^{1}\left(\mathbf{x}^{\prime}\right)\right\} \\ \mathbb{E}\left\{f_{2}(\mathbf{x}) f_{2}\left(\mathbf{x}^{\prime}\right)\right\} &=\mathbb{E}\left\{a_{2}^{1} u^{1}(\mathbf{x}) a_{2}^{1} u^{1}\left(\mathbf{x}^{\prime}\right)\right\}=\left(a_{2}^{1}\right)^{2} \mathbb{E}\left\{u^{1}(\mathbf{x}) u^{1}\left(\mathbf{x}^{\prime}\right)\right\} \end{aligned} E{[f1​(x)f2​(x)​][f1​(x′)​f2​(x′)]​}=[E{f1​(x)f1​(x′)}E{f2​(x)f1​(x′)}​E{f1​(x)f2​(x′)}E{f2​(x)f2​(x′)}​]E{f1​(x)f1​(x′)}E{f1​(x)f2​(x′)}E{f2​(x)f2​(x′)}​=E{a11​u1(x)a11​u1(x′)}=(a11​)2E{u1(x)u1(x′)}=E{a11​u1(x)a21​(x′)}=a11​a21​E{u1(x)u1(x′)}=E{a21​u1(x)a21​u1(x′)}=(a21​)2E{u1(x)u1(x′)}​

So that term could be written as:
$$\begin{gathered} \mathbb{E}\left\{\mathbf{f}(\mathbf{x}){\left[\mathbf{f}\left({\mathbf{x}}^{\prime }\right)\right]}^{\top }\right\}=\left[\begin{array}{cc}{\left(a_1^1\right)}^2\mathbb{E}\left\{u^1(\mathbf{x})u^1\left({\mathbf{x}}^{\prime }\right)\right\}& a_1^1{a}_2^1\mathbb{E}\left\{u^1(\mathbf{x})u^1\left({\mathbf{x}}^{\prime }\right)\right\}\\ a^1{a}^2\mathbb{E}\left\{u^1(\mathbf{x})u^1\left({\mathbf{x}}^{\prime }\right)\right\}& {\left(a_2^1\right)}^2\mathbb{E}\left\{u^1(\mathbf{x})u^1\left({\mathbf{x}}^{\prime }\right)\right\}\end{array}\right] \\ =\left[\begin{array}{cc}{\left(a_1^1\right)}^2& a_1^1{a}_2^1\\ a_1^1{a}_2^1& {\left(a_2^1\right)}^2\end{array}\right]\mathbb{E}\left\{u^1(\mathbf{x})u^1\left({\mathbf{x}}^{\prime }\right)\right\} \end{gathered}$$
The term E { f ( x ) } \mathbb{E}\{\mathbf{f}(\mathbf{x})\} E{f(x)} is computed as:
E { [ f 1 ( x ) f 2 ( x ) ] } = [ E { f 1 ( x ) } E { f 1 ( x ) } ] = [ E { a 1 1 u 1 ( x ) } E { a 2 1 u 1 ( x ) } ] ] = [ a 1 1 a 2 1 ] E { u 1 ( x ) } \mathbb{E}\left\{\left[\begin{array}{c}{f_{1}(\mathbf{x})} \\ {f_{2}(\mathbf{x})}\end{array}\right]\right\}=\left[\begin{array}{c}{\mathbb{E}\left\{f_{1}(\mathbf{x})\right\}} \\ {\mathbb{E}\left\{f_{1}(\mathbf{x})\right\}}\end{array}\right]=\left[\begin{array}{c}{\mathbb{E}\left\{a_{1}^{1} u^{1}(\mathbf{x})\right\}} \\ {\mathbb{E}\left\{a_{2}^{1} u^{1}(\mathbf{x})\right\}}\end{array}\right] ]=\left[\begin{array}{c}{a_{1}^{1}} \\ {a_{2}^{1}}\end{array}\right] \mathbb{E}\left\{u^{1}(\mathbf{x})\right\} E{[f1​(x)f2​(x)​]}=[E{f1​(x)}E{f1​(x)}​]=[E{a11​u1(x)}E{a21​u1(x)}​]]=[a11​a21​​]E{u1(x)}
Putting them together, the covariance for $f(x^{{}^{\prime }})$ follows as:
$$\left[\begin{array}{cc}{\left(a_1^1\right)}^2& a_1^1{a}_2^1\\ a_1^1{a}_2^1& {\left(a_2^1\right)}^2\end{array}\right]\mathbb{E}\left\{u^1(\mathbf{x})u^1\left({\mathbf{x}}^{\prime }\right)\right\}-\left[\begin{array}{c}{a}_1^1\\ a_2^1\end{array}\right][\begin{array}{cc}{a}_1^1& a_2^1]\end{array}\left\{u^1(\mathbf{x})\right\}\mathbb{E}\left\{u^1\left({\mathbf{x}}^{\prime }\right)\right\}$$
Defining $\mathbf{a}={\left[\begin{array}{ll}{a}_1^1& a_2^1\end{array}\right]}^{\top }$,
$$\begin{array}{rl}\mathrm{cov}\left(\mathbf{f}(\mathbf{x}),\mathbf{f}\left({\mathbf{x}}^{\prime }\right)\right)& ={\mathbf{a}\mathbf{a}}^{\top }\mathbb{E}\left\{u^1(\mathbf{x})u^1\left({\mathbf{x}}^{\prime }\right)\right\}-{\mathbf{a}\mathbf{a}}^{\top }\mathbb{E}\left\{u^1(\mathbf{x})\right\}\mathbb{E}\left\{u^1\left({\mathbf{x}}^{\prime }\right)\right\}\\ & ={\mathbf{a}\mathbf{a}}^{\top }\underset{k\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)}{\underbrace{\left[\mathbb{E}\left\{u^1(\mathbf{x})u^1\left({\mathbf{x}}^{\prime }\right)\right\}-\mathbb{E}\left\{u^1(\mathbf{x})\right\}\mathbb{E}\left\{u^1\left({\mathbf{x}}^{\prime }\right)\right\}\right]}}\\ & =\mathbf{a}{\mathbf{a}}^{\top }k\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)\end{array}$$
We define $\mathbf{B}={\mathbf{a}\mathbf{a}}^{\top }$, leading to
$$\mathrm{cov}\left(\mathbf{f}(\mathbf{x}),\mathbf{f}\left({\mathbf{x}}^{\prime }\right)\right)=\mathbf{B}k\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)=\left[\begin{array}{ll}{b}_{11}& b_{12}\\ b_{21}& b_{22}\end{array}\right]k\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)$$
and the $\mathbf{B}$ has rank one, since it is the result of the multiplication of two column-vector.

Sample Twice

Sample twice from a GP $u(\mathbf{x})\sim \mathcal{G}\mathcal{P}\left(0,k\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)\right)$ to obtain $u^1(\mathbf{x})\text{ and }{u}^2(\mathbf{x})$.

Adding a scaled transformation.:
$$\begin{array}{l}{f}_1(\mathbf{x})=a_1^1{u}^1(\mathbf{x})+a_1^2{u}^2(\mathbf{x})\\ f_2(\mathbf{x})=a_2^1{u}^1(\mathbf{x})+a_2^2{u}^2(\mathbf{x})\end{array}$$
**

Notice that the $u_1$ and $u_2$ are independent, although they share the same covariance $k$.
$$\mathbf{f}(\mathbf{x})=\left[\begin{array}{cc}{\left(a_1^1\right)}^{}& a_1^2\\ a_2^1& {\left(a_2^2\right)}^{}\end{array}\right]\left[\begin{array}{l}{u}^1\\ u^2\end{array}\right]$$
The vector-valued function can be written as $f(x)$, where ${\mathbf{a}}^1={\left[a_1^1{a}_2^1\right]}^{\top }\text{ and }{\mathbf{a}}^2={\left[a_1^2{a}_2^2\right]}^{\top }$

The covariance for $f(x)$ is computed as:
$$\begin{array}{rl}\mathrm{cov}\left(\mathbf{f}(\mathbf{x}),\mathbf{f}\left({\mathbf{x}}^{\prime }\right)\right)& ={\mathbf{a}}^1{\left({\mathbf{a}}^1\right)}^{\top }\mathrm{cov}\left(u^1(\mathbf{x}),u^1\left({\mathbf{x}}^{\prime }\right)\right)+{\mathbf{a}}^2{\left({\mathbf{a}}^2\right)}^{\top }\mathrm{cov}\left(u^2(\mathbf{x}),u^2\left({\mathbf{x}}^{\prime }\right)\right)\\ & ={\mathbf{a}}^1{\left({\mathbf{a}}^1\right)}^{\top }k\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)+{\mathbf{a}}^2{\left({\mathbf{a}}^2\right)}^{\top }k\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)\\ & =\left[{\mathbf{a}}^1{\left({\mathbf{a}}^1\right)}^{\top }+{\mathbf{a}}^2{\left({\mathbf{a}}^2\right)}^{\top }\right]k\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)\end{array}$$
notice that $u_1$ and $u_2$ are independent, so their variance could be added directly.

we define $\mathbf{B}={\mathbf{a}}^1{\left({\mathbf{a}}^1\right)}^{\top }+{\mathbf{a}}^2{\left({\mathbf{a}}^2\right)}^{\top }$, leading to:
$$\mathrm{cov}\left(\mathbf{f}(\mathbf{x}),\mathbf{f}\left({\mathbf{x}}^{\prime }\right)\right)=\mathbf{B}k\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)=\left[\begin{array}{ll}{b}_{11}& b_{12}\\ b_{21}& b_{22}\end{array}\right]k\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)$$
Notice that $B$ has rank two.

Observed Data:

$$\left[\begin{array}{c}{\mathbf{f}}_1\\ {\mathbf{f}}_2\end{array}\right]=\left[\begin{array}{c}{f}_1\left({\mathbf{x}}_1\right)\\ ⋮\\ f_1\left({\mathbf{x}}_N\right)\\ f_2\left({\mathbf{x}}_1\right)\\ ⋮\\ f_2\left({\mathbf{x}}_N\right)\end{array}\right]\sim \mathcal{N}\left(\left[\begin{array}{l}0\\ 0\end{array}\right],\left[\begin{array}{cc}{b}_{11}\mathbf{K}& b_{12}\mathbf{K}\\ b_{21}\mathbf{K}& b_{22}\mathbf{K}\end{array}\right]\right)$$
The matrix $\mathbf{k}\in {\mathcal{R}}^{\mathbf{N}\ast \mathbf{N}}$ has elements $k(x_i,x_j)$.

If we use Kronecker product we would get:
$$\left[\begin{array}{c}{\mathbf{f}}_1\\ {\mathbf{f}}_2\end{array}\right]=\left[\begin{array}{c}{f}_1\left({\mathbf{x}}_1\right)\\ ⋮\\ f_1\left({\mathbf{x}}_N\right)\\ f_2\left({\mathbf{x}}_1\right)\\ ⋮\\ f_2\left({\mathbf{x}}_N\right)\end{array}\right]\sim \mathcal{N}\left(\left[\begin{array}{l}0\\ 0\end{array}\right],\mathbf{B}\otimes \mathbf{K}\right)$$

General Case

Consider a set of functions { f d ( x ) } d = 1 D \left\{f_{d}(\mathbf{x})\right\}_{d=1}^{D} {fd​(x)}d=1D​.

In the ICM,
$$f_d(\mathbf{x})=\sum _{i=1}^R{a}_d^i{u}^i(\mathbf{x})$$
where the functions $u_i(x)$ are GPs sampled independently, and share the same covariance function $k(x,x^{{}^{\prime }})$.

For $\mathbf{f}(\mathbf{x})={\left[f_1(\mathbf{x})\cdots f_D(\mathbf{x})\right]}^{\top }$, the covariance is given as:
$$\mathrm{cov}\left[\mathbf{f}(\mathbf{x}),\mathbf{f}\left({\mathbf{x}}^{\prime }\right)\right]=\mathbf{A}{\mathbf{A}}^{\top }k\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)=\mathbf{B}k\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)$$
where
$$\mathbf{A}=\left[{\mathbf{a}}^1{\mathbf{a}}^2\cdots {\mathbf{a}}^R\right]$$
and the Rank of $B$ is given by $R$.

ICM: autokrigeability

If the outputs are considered to be noise-free, prediction using the ICM under an isotopic data case is equivalent to independent prediction over each output. This circumstance is also known as autokrigeability.

The prove:

Assume that we only have two outputs: $f_1,f_2$

the predicated mean could be written as:
$$\begin{gathered} \mu =K_{f_{\ast },f}(K_{f,f}{)}^{-1}f \\ {K}_{f,f}=B\otimes K \end{gathered}$$

$$\begin{array}{rl}\mu & =B\otimes K_{\ast }(B\otimes K{)}^{-1}f\\ & =B\otimes K_{\ast }(B^{-1}\otimes K^{-1})f\\ & =B{B}^{-1}\otimes K_{\ast }{K}^{-1}f\\ & =I\otimes K_{\ast }{K}^{-1}f\\ & =\left[\begin{array}{cc}{K}_{\ast }{K}^{-1}& 0\\ 0& K_{\ast }{K}^{-1}\end{array}\right]\left[\begin{array}{c}{f}_1\\ f_2\end{array}\right]\end{array}$$

it means, the prediction of $f_1$ only depends on the data set for $f_1$

Semiparametric Latent Factor Model (SLFM)

ICM uses R samples $u^i(x)$ from $u(x)$ with the same covariance function. SLFM uses Q samples from $u_q$ processes with different covariance functions.

Two Outputs

1. Sample from a GP $\mathcal{G}\mathcal{P}\left(0,k_1\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)\right)$ to obtain $u_1(x)$.

2. Sample from a GP $\mathcal{G}\mathcal{P}\left(0,k_2\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)\right)$ to obtain $u_2(x)$.

3. Adding a scaled versions:
   $$\begin{array}{l}{f}_1(\mathbf{x})=a_{1,1}{u}_1(\mathbf{x})+a_{1,2}{u}_2(\mathbf{x})\\ f_2(\mathbf{x})=a_{2,1}{u}_1(\mathbf{x})+a_{2,2}{u}_2(\mathbf{x})\end{array}$$

Similar, it can be written as:
$$\mathbf{f}(\mathbf{x})={\mathbf{a}}_1{u}_1(\mathbf{x})+{\mathbf{a}}_2{u}_2(\mathbf{x})$$
with ${\mathbf{a}}_1={\left[a_{1,1}{a}_{2,1}\right]}^{\top }\text{ and }{\mathbf{a}}_2={\left[a_{1,2}{a}_{2,2}\right]}^{\top }$

The covariance for $f(x)$ is computed as:
$$\begin{array}{rl}\mathrm{cov}\left(\mathbf{f}(\mathbf{x}),\mathbf{f}\left({\mathbf{x}}^{\prime }\right)\right)& ={\mathbf{a}}_1{\left({\mathbf{a}}_1\right)}^{\top }\mathrm{cov}\left(u_1(\mathbf{x}),u_1\left({\mathbf{x}}^{\prime }\right)\right)+{\mathbf{a}}_2{\left({\mathbf{a}}_2\right)}^{\top }\mathrm{cov}\left(u_2(\mathbf{x}),u_2\left({\mathbf{x}}^{\prime }\right)\right)\\ & ={\mathbf{a}}_1{\left({\mathbf{a}}_1\right)}^{\top }{k}_1\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)+{\mathbf{a}}_2{\left({\mathbf{a}}_2\right)}^{\top }{k}_2\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)\end{array}$$
We define ${\mathbf{B}}_1={\mathbf{a}}_1{\left({\mathbf{a}}_1\right)}^{\top }\text{ and }{\mathbf{B}}_2={\mathbf{a}}_2{\left({\mathbf{a}}_2\right)}^{\top }$, leading to:
$$\mathrm{cov}\left(\mathbf{f}(\mathbf{x}),\mathbf{f}\left({\mathbf{x}}^{\prime }\right)\right)={\mathbf{B}}_1{k}_1\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)+{\mathbf{B}}_2{k}_2\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)$$
Notice that $B_{1} $ and $B_2$ have rank one.

$$\left[\begin{array}{c}{\mathbf{f}}_1\\ {\mathbf{f}}_2\end{array}\right]=\left[\begin{array}{c}{f}_1\left({\mathbf{x}}_1\right)\\ ⋮\\ f_1\left({\mathbf{x}}_N\right)\\ f_2\left({\mathbf{x}}_1\right)\\ ⋮\\ f_2\left({\mathbf{x}}_N\right)\end{array}\right]\sim \mathcal{N}\left(\left[\begin{array}{l}0\\ 0\end{array}\right],{\mathbf{B}}_1\otimes {\mathbf{K}}_1+{\mathbf{B}}_2\otimes {\mathbf{K}}_2\right)$$

General Case:

Consider a set of functions { f d ( x ) } d = 1 D \left\{f_{d}(\mathbf{x})\right\}_{d=1}^{D} {fd​(x)}d=1D​

In the SLFM,
$$f_d(\mathbf{x})=\sum _{q=1}^Q{a}_{d,q}{u}_q(\mathbf{x})$$
where the functions $u_q(x)$ are GPs with covariance functions $k_q(x,x^{{}^{\prime }})$.

For $\mathbf{f}(\mathbf{x})={\left[f_1(\mathbf{x})\cdots f_D(\mathbf{x})\right]}^{\top }$, the covariance is given as:
$$\mathrm{cov}\left[\mathbf{f}(\mathbf{x}),\mathbf{f}\left({\mathbf{x}}^{\prime }\right)\right]=\sum _{q=1}^Q{\mathbf{A}}_q{\mathbf{A}}_q^{\top }{k}_q\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)=\sum _{q=1}^Q{\mathbf{B}}_q{k}_q\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)$$
where $A_q=a_q$.

The rank of each $B_q$ is one.

Linear model of coregionalization (LMC)

The LMC generalizes the ICM and the SLFM allowing several independent samples from GPs with different covariances.

Consider a set of functions { f d ( x ) } d = 1 D \left\{f_{d}(\mathbf{x})\right\}_{d=1}^{D} {fd​(x)}d=1D​
$$f_d(\mathbf{x})=\sum _{q=1}^Q\sum _{i=1}^{R_q}{a}_{d,q}^i{u}_q^i(\mathbf{x})$$
where the functions $u_q^i$ are GPs with zero means and covariance functions:
$$\mathrm{cov}\left[u_q^i(\mathbf{x}),u_{q^{\prime }}^{i^{\prime }}\left({\mathbf{x}}^{\prime }\right)\right]=k_q\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)$$
if $i=i^{{}^{\prime }}$ and $q=q^{{}^{\prime }}$

There are $Q$ groups of samples. For each group, there are $R_q$ samples obtained independently from the same GP with covariance $k_q(x,x^{{}^{\prime }})$.

The LMC corresponds to the sum of Q ICMs.

Suppose we have D = 2, Q = 2, and $R_q$=2. According to LMC:
$$\begin{array}{l}{f}_1(\mathbf{x})=a_{1,1}^1{u}_1^1(\mathbf{x})+a_{1,1}^2{u}_1^2(\mathbf{x})+a_{1,2}^1{u}_2^1(\mathbf{x})+a_{1,2}^2{u}_2^2(\mathbf{x})\\ f_2(\mathbf{x})=a_{2,1}^1{u}_1^1(\mathbf{x})+a_{2,1}^2{u}_1^2(\mathbf{x})+a_{2,2}^1{u}_2^1(\mathbf{x})+a_{2,2}^2{u}_2^2(\mathbf{x})\end{array}$$
For $\mathbf{f}(\mathbf{x})={\left[f_1(\mathbf{x})\cdots f_D(\mathbf{x})\right]}^{\top }$, the covariance $\mathrm{cov}\left[\mathbf{f}(\mathbf{x}),\mathbf{f}\left({\mathbf{x}}^{\prime }\right)\right]$ is given as:
$$\mathrm{cov}\left[\mathbf{f}(\mathbf{x}),\mathbf{f}\left({\mathbf{x}}^{\prime }\right)\right]=\sum _{q=1}^Q{\mathbf{A}}_q{\mathbf{A}}_q^{\top }{k}_q\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)=\sum _{q=1}^Q{\mathbf{B}}_q{k}_q\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)$$
where ${\mathbf{A}}_q=\left[{\mathbf{a}}_q^1{\mathbf{a}}_q^2\cdots {\mathbf{a}}_q^{R_q}\right]$.

The rank of each $B_q$ is $R_q$.

The matrices $B_q$ are known as the coregionalization matrices.

$$\left[\begin{array}{c}{\mathbf{f}}_1\\ {\mathbf{f}}_2\end{array}\right]=\left[\begin{array}{c}{f}_1\left({\mathbf{x}}_1\right)\\ ⋮\\ f_1\left({\mathbf{x}}_N\right)\\ f_2\left({\mathbf{x}}_1\right)\\ ⋮\\ f_2\left({\mathbf{x}}_N\right)\end{array}\right]\sim \mathcal{N}\left(\left[\begin{array}{l}0\\ 0\end{array}\right],\sum _{q=1}^Q{\mathbf{B}}_q\otimes {\mathbf{K}}_q\right)$$