Gaussian Process(高斯过程) GPSS暑校笔记六（英文版）——随机微分方程

Basic Ideas

Why use S§DE solvers for GPs?

• The $O(n^3)$ computational complexity is a challenge.
• What do we get:
  • $O(n)$ state-space methods for SDEs/SPDEs.
  • Sparse approximations developed for SPDEs.
  • Reduced rank Fourier/basis function approximations. Path to non-Gaussian processes.
• Downsides:
  • We often need to approximate.
  • Mathematics can become messy

Stochastic differential equations and Gaussian processes

Ornstein-Uhlenbeck process

The mean and covariance functions:
$$\begin{array}{rl}m(x)& =0\\ k\left(x,x^{\prime }\right)& ={\sigma }^2\exp \left(-\lambda \left|x-x^{\prime }\right|\right)\end{array}$$
This has a path representation as a stochastic differential equation (SDE):
$$\frac{df(t)}{dt}=-\lambda f(t)+w(t)$$
where $w(t)$ is a white noise process with $x$ relabeled as $t$.

Prove:
$$\begin{array}{rl}FT:(i\omega )\hat{f}& =-\lambda \hat{f}+\hat{\omega }\\ \hat{f}& =\frac{\hat{\omega }}{\lambda +(i\omega )}\\ SpectralDensity:\delta (\omega )& =\frac{E[|\hat{w}{|}^2]}{w^2+{\lambda }^2}=\frac{q}{w^2+{\lambda }^2}\\ IF:h(\tau )& =\frac{1}{2\pi }\int \frac{q}{w^2+{\lambda }^2}\exp (iw\tau )d\tau \end{array}$$
Consider a Gaussian process regression problem:
$$\begin{array}{rl}f(x)& \sim \mathrm{G}\mathrm{P}\left(0,{\sigma }^2\exp \left(-\lambda \left|x-x^{\prime }\right|\right)\right)\\ y_k& =f\left(x_k\right)+{\epsilon }_k\end{array}$$
this is equivalent to the state-space model:
$$\begin{array}{rl}\frac{df(t)}{dt}& =-\lambda f(t)+w(t)\\ y_k& =f\left(t_k\right)+{\epsilon }_k\end{array}$$
that is, with $fk=f(t_k)$ we have a Gauss-Markov model
$$\begin{array}{rl}{f}_{k+1}& \sim p\left(f_{k+1}|f_k\right)\\ y_k& \sim p\left(y_k|f_k\right)\end{array}$$
Solvable in $O(n)$ time using Kalman filter/smoother