Multiple-Output Gaussian Process
Working Situation
As the picture shows, we want to learn from the three sensors (with complete signal information) to recover the fourth one.
Dependencies between processes
Multiple-independent Output GP
f 1 ( x ) ∼ G P ( 0 , k 1 ( x , x ′ ) )                      f 2 ( x ) ∼ G P ( 0 , k 2 ( x , x ′ ) ) D 1 = { ( x i , 1 , y 1 ( x i , 2 ) ) ∣ i = 1 , … , N 1 }                      D 2 = { ( x i , 2 , y 2 ( x i , 2 ) ) ∣ i = 1 , … , N 2 } y 1 ∼ N ( 0 , K 1 + σ 1 2 )                      y 2 ∼ N ( 0 , K 2 + σ 2 2 l ) \begin{aligned} f_{1}(\mathbf{x}) \sim \mathcal{G} \mathcal{P}\left(0, k_{1}\left(\mathbf{x}, \mathbf{x}^{\prime}\right)\right) &\;\;\;\;\;\;\;\;\;\; f_{2}(\mathbf{x}) \sim \mathcal{G} \mathcal{P}\left(0, k_{2}\left(\mathbf{x}, \mathbf{x}^{\prime}\right)\right) \\ D_{1}=\left\{\left(\mathbf{x}_{i, 1}, y_{1}\left(\mathbf{x}_{i, 2}\right)\right) | i=1, \ldots, N_{1}\right\} & \;\;\;\;\;\;\;\;\;\;\mathcal{D}_{2}=\left\{\left(\mathbf{x}_{i, 2}, y_{2}\left(\mathbf{x}_{i, 2}\right)\right) | i=1, \ldots, N_{2}\right\} \\ \mathbf{y}_{1} \sim \mathcal{N}\left(\mathbf{0}, \mathbf{K}_{1}+\sigma_{1}^{2}\right) & \;\;\;\;\;\;\;\;\;\;\mathbf{y}_{2} \sim \mathcal{N}\left(\mathbf{0}, \mathbf{K}_{2}+\sigma_{2}^{2} \mathbf{l}\right) \end{aligned} f1(x)∼GP(0,k1(x,x′))D1={
(xi,1,y1(xi,2))∣i=1,…,N1}y1∼N(0,K1+σ12)f2(x)∼GP(0,k2(x,x′))D2={
(xi,2,y2(xi,2))∣i=1,…,N2}y2∼N(0,K2+σ22l)
[ y 1 y 2 ] ∼ N ( [ 0 0 ] , [ K 1 0 0 K 2 ] + [ σ 1 2 l 0 0 σ 2 2 l ] ) \left[\begin{array}{l}{\mathbf{y}_{1}} \\ {\mathbf{y}_{2}}\end{array}\right] \sim \mathcal{N}\left(\left[\begin{array}{l}{\mathbf{0}} \\ {\mathbf{0}}\end{array}\right],\left[\begin{array}{cc}{\mathbf{K}_{1}} & {\mathbf{0}} \\ {\mathbf{0}} & {\mathbf{K}_{2}}\end{array}\right]+\left[\begin{array}{cc}{\sigma_{1}^{2} \mathbf{l}} & {\mathbf{0}} \\ {\mathbf{0}} & {\sigma_{2}^{2} \mathbf{l}}\end{array}\right]\right) [y1y2]∼N([00],[K100K2]+[σ12l00σ22l])
How to find the independences for kernel design
K f , f = [ K 1 ? ? K 2 ] \mathbf{K}_{\mathbf{f}, \mathbf{f}}=\left[\begin{array}{cc}{\mathbf{K}_{1}} & {?} \\ {?} & {\mathbf{K}_{2}}\end{array}\right] Kf,f=[K1??K2]
Build a cross-covariance function c o v [ f 1 ( x ) , f 2 ( x ′ ) ] cov[f_1(x), f_2(x^{'})] cov[f1(x),f2(x′)] such that K f , f K_{f,f} Kf,f is positive semi-definite.
Different input configurations of data
D 1 = { ( x i , f 1 ( x i ) ) i = 1 N }            D 1 = { ( x i , 1 , f 1 ( x i , 1 ) ) i = 1 N 1 } D 2 = { ( x i , f 2 ( x i ) ) i = 1 N }            D 2 = { ( x i , 2 , f 2 ( x i , 2 ) ) i = 1 N 2 } \begin{array}{ll}{\mathcal{D}_{1}=\left\{\left(\mathbf{x}_{i}, f_{1}\left(\mathbf{x}_{i}\right)\right)_{i=1}^{N}\right\}} &\;\;\;\;\; {\mathcal{D}_{1}=\left\{\left(\mathbf{x}_{i, 1}, f_{1}\left(\mathbf{x}_{i, 1}\right)\right)_{i=1}^{N_{1}}\right\}} \\ {\mathcal{D}_{2}=\left\{\left(\mathbf{x}_{i}, f_{2}\left(\mathbf{x}_{i}\right)\right)_{i=1}^{N}\right\}} & \;\;\;\;\;{\mathcal{D}_{2}=\left\{\left(\mathbf{x}_{i, 2}, f_{2}\left(\mathbf{x}_{i, 2}\right)\right)_{i=1}^{N_{2}}\right\}}\end{array} D1={
(xi,f1(xi))i=1N}D2={
(xi,f2(xi))i=1N}D1={
(xi,1,f1(xi,1))i=1N1}D2={
(xi,2,f2(xi,2))i=1N2}
Intrinsic Coregionalization Model
Two outputs
Sample Once
Consider two outputs $f_1(x) $ f 2 ( x ) f_{2}(x) f2(x) with x ∈ R p x\in \mathcal{R}^{p} x∈Rp.
- Sample from a GP u ( x ) ∼ G P ( 0 , k ( x , x ′ ) ) u(\mathbf{x}) \sim \mathcal{G P}\left(0, k\left(\mathbf{x}, \mathbf{x}^{\prime}\right)\right) u(x)∼GP(0,k(x,x′)) to obtain u 1 ( x ) u^{1}(\mathbf{x}) u1(x)
- Obtain $f_1(x) $ and f 2 ( x ) f_{2}(x) f2(x) by linearly transforming:
f 1 ( x ) = a 1 1 u 1 ( x ) f 2 ( x ) = a 2 1 u 1 ( x ) \begin{aligned} f_{1}(\mathbf{x}) &=a_{1}^{1} u^{1}(\mathbf{x}) \\ f_{2}(\mathbf{x}) &=a_{2}^{1} u^{1}(\mathbf{x}) \end{aligned} f1(x)f2(x)=a11u1(x)=a21u1(x)
For a fixed value x x x. we can group f 1 ( x ) f_1(x) f1(x) and f 2 ( x ) f_2(x) f2(x) in a vector:
f ( x ) = [ f 1 ( x ) f 2 ( x ) ] \mathbf{f}(\mathbf{x})=\left[\begin{array}{l}{f_{1}(\mathbf{x})} \\ {f_{2}(\mathbf{x})}\end{array}\right] f(x)=[f1(x)f2(x)]
and this vector will be refer as a v e c t o r − v a l u e d    f u n c t i o n \bf{vector-valued \; function} vector−valuedfunction.
The covariance for f ( x ) f(x) f(x) is computed as:
cov ( f ( x ) , f ( x ′ ) ) = E { f ( x ) [ f ( x ′ ) ] ⊤ } − E { f ( x ) } [ E { f ( x ′ ) } ] ⊤ \operatorname{cov}\left(\mathbf{f}(\mathbf{x}), \mathbf{f}\left(\mathbf{x}^{\prime}\right)\right)=\mathbb{E}\left\{\mathbf{f}(\mathbf{x})\left[\mathbf{f}\left(\mathbf{x}^{\prime}\right)\right]^{\top}\right\}-\mathbb{E}\{\mathbf{f}(\mathbf{x})\}\left[\mathbb{E}\left\{\mathbf{f}\left(\mathbf{x}^{\prime}\right)\right\}\right]^{\top} cov(f(x),f(x′))=E{
f(x)[f(x′)]⊤}−E{
f(x)}[E{
f(x′)}]⊤
E { [ f 1 ( x ) f 2 ( x ) ] [ f 1 ( x ′ ) f 2 ( x ′ ) ] } = [ E { f 1 ( x ) f 1 ( x ′ ) } E { f 1 ( x ) f 2 ( x ′ ) } E { f 2 ( x ) f 1 ( x ′ ) } E { f 2 ( x ) f 2 ( x ′ ) } ] E { f 1 ( x ) f 1 ( x ′ ) } = E { a 1 1 u 1 ( x ) a 1 1 u 1 ( x ′ ) } = ( a 1 1 ) 2 E { u 1 ( x ) u 1 ( x ′ ) } E { f 1 ( x ) f 2 ( x ′ ) } = E { a 1 1 u 1 ( x ) a 2 1 ( x ′ ) } = a 1 1 a 2 1 E { u 1 ( x ) u 1 ( x ′ ) } E { f 2 ( x ) f 2 ( x ′ ) } = E { a 2 1 u 1 ( x ) a 2 1 u 1 ( x ′ ) } = ( a 2 1 ) 2 E { u 1 ( x ) u 1 ( x ′ ) } \mathbb{E}\left\{\left[\begin{array}{c}{f_{1}(\mathbf{x})} \\ {f_{2}(\mathbf{x})}\end{array}\right]\left[\begin{array}{ll}{f_{1}\left(\mathbf{x}^{\prime}\right)} & {f_{2}\left(\mathbf{x}^{\prime}\right) ]}\end{array}\right\}=\left[\begin{array}{cc}{\mathbb{E}\left\{f_{1}(\mathbf{x}) f_{1}\left(\mathbf{x}^{\prime}\right)\right\}} & {\mathbb{E}\left\{f_{1}(\mathbf{x}) f_{2}\left(\mathbf{x}^{\prime}\right)\right\}} \\ {\mathbb{E}\left\{f_{2}(\mathbf{x}) f_{1}\left(\mathbf{x}^{\prime}\right)\right\}} & {\mathbb{E}\left\{f_{2}(\mathbf{x}) f_{2}\left(\mathbf{x}^{\prime}\right)\right\}}\end{array}\right]\right.\\ \begin{aligned} \mathbb{E}\left\{f_{1}(\mathbf{x}) f_{1}\left(\mathbf{x}^{\prime}\right)\right\} &=\mathbb{E}\left\{a_{1}^{1} u^{1}(\mathbf{x}) a_{1}^{1} u^{1}\left(\mathbf{x}^{\prime}\right)\right\}=\left(a_{1}^{1}\right)^{2} \mathbb{E}\left\{u^{1}(\mathbf{x}) u^{1}\left(\mathbf{x}^{\prime}\right)\right\} \\ \mathbb{E}\left\{f_{1}(\mathbf{x}) f_{2}\left(\mathbf{x}^{\prime}\right)\right\} &=\mathbb{E}\left\{a_{1}^{1} u^{1}(\mathbf{x}) a_{2}^{1}\left(\mathbf{x}^{\prime}\right)\right\}=a_{1}^{1} a_{2}^{1} \mathbb{E}\left\{u^{1}(\mathbf{x}) u^{1}\left(\mathbf{x}^{\prime}\right)\right\} \\ \mathbb{E}\left\{f_{2}(\mathbf{x}) f_{2}\left(\mathbf{x}^{\prime}\right)\right\} &=\mathbb{E}\left\{a_{2}^{1} u^{1}(\mathbf{x}) a_{2}^{1} u^{1}\left(\mathbf{x}^{\prime}\right)\right\}=\left(a_{2}^{1}\right)^{2} \mathbb{E}\left\{u^{1}(\mathbf{x}) u^{1}\left(\mathbf{x}^{\prime}\right)\right\} \end{aligned} E{ [f1(x)f2(x)][f1(x′)f2(x′)]}=[E{ f1(x)f