DGM: A deep learning algorithm for solving partial differential equations

最新推荐文章于 2023-01-13 15:32:07 发布

pinn山里娃

最新推荐文章于 2023-01-13 15:32:07 发布

阅读量2.7k

点赞数 13

分类专栏：物理驱动深度学习论文分享文章标签：算法

本文链接：https://blog.csdn.net/weixin_45521594/article/details/108571946

版权

物理驱动深度学习论文分享专栏收录该内容

40 篇文章 38 订阅

订阅专栏

论文信息

题目：
DGM: A deep learning algorithm for solving partial differential equations

作者及单位：
Justin Sirignano∗ and Konstantinos Spiliopoulos

期刊、会议：

时间：18

论文地址：论文链接

代码：

基础

论文动机

High-dimensional partial differential equations (PDEs) are used in physics, engineering, and finance. Their numerical solution has been a longstanding challenge
This quickly becomes computationally intractable when the dimension d becomes even moderately large. We propose to solve high-dimensional PDEs using a meshfree deep learning algorithm.
The method is similar in spirit to the Galerkin method, but with several key changes using ideas from machine learning.
The Galerkin method is a widely-used computational method which seeks a reduced-form solution to a PDE as a linear combinationof basis functions.
DGM is a natural merger of Galerkin methods and machine learning

问题背景

本文方法

Approximation Power of Neural Networks for PDEs

$\begin{array}{ll} \partial_{t} u(t, x)+\mathcal{L} u(t, x)=0, & (t, x) \in[0, T] \times \Omega \\ u(0, x)=u_{0}(x), & x \in \Omega \\ u(t, x)=g(t, x), & x \in[0, T] \times \partial \Omega \end{array}$

The error function:
$J(f)=\left\|\partial_{t} f+\mathcal{L} f\right\|_{2,[0, T] \times \Omega}^{2}+\|f-g\|_{2,[0, T] \times \partial \Omega}^{2}+\left\|f(0, \cdot)-u_{0}\right\|_{2, \Omega}^{2}$

This paper explores several new innovations：

First, we focus on high-dimensional PDEs and apply deep learning advances of the past decade to this problem.
Secondly, to avoid ever forming a mesh, we sample a sequence of random spatial points.
Thirdly, the algorithm incorporates a new computational scheme for the efficient computationof neural network gradients arising from the second derivatives of high-dimensional PDEs.

DGM

在这里插入图片描述

A Monte Carlo Method for Fast Computation of Second Derivatives

包含二阶导数项得在计算高维时计算代价可能很昂贵，计算二阶导数代价（以4.2网络结构为例）是 $\mathcal{O}\left(d^{2} \times N\right)$ ，其中d为x的空间维度，N为batch size，相比之下，一阶导数为 $\mathcal{O}(d \times N)$ ，而且含有二阶导数项计算花费甚至更多，因为在梯度算法优化还会求一阶导数（相当于三阶导数），考虑到这些情况，二阶导数采用Monte Carlo方法.

假设 $\mathcal{L} f(t, x, ; \theta)$ 中二阶导数形式为 $\frac{1}{2} \sum_{i, j=1}^{d} \rho_{i, j} \sigma_{i}(x) \sigma_{j}(x) \frac{\partial^{2} f}{\partial x_{i} x_{j}}(t, x ; \theta)$ ，假设 $\left[\rho_{i, j}\right]_{i, j=1}^{d}$ 是正定矩阵同时定义 $\sigma(x)=\left(\sigma_{1}(x), \ldots, \sigma_{d}(x)\right)$ .这里 $\sigma$ 代表什么？例如，在考虑随机微分方程的期望的时候会产生偏微分方程时代表扩散系数(diffusion coefficient)。二阶项考虑如下计算：
$\sum_{i, j=1}^{d} \rho_{i, j} \sigma_{i}(x) \sigma_{j}(x) \frac{\partial^{2} f}{\partial x_{i} x_{j}}(t, x ; \theta)=\lim _{\Delta \rightarrow 0} \mathbb{E}\left[\sum_{i=1}^{d} \frac{\frac{\partial f}{\partial x_{i}}\left(t, x+\sigma(x) W_{\Delta} ; \theta\right)-\frac{\partial f}{\partial x_{i}}(t, x ; \theta)}{\Delta} \sigma_{i}(x) W_{\Delta}^{i}\right]$
其中， $W_{t} \in \mathbb{R}^{d}$ 是一个布朗运动(Brownian motion), $\Delta \in \mathbb{R}_{+}$ 是步长，二阶替代后收敛速率为 $\mathcal{O}(\sqrt{\Delta})$

从而一阶导数算子：
$\mathcal{L}_{1} f\left(t_{n}, x_{n} ; \theta_{n}\right):=\mathcal{L} f\left(t_{n}, x_{n} ; \theta_{n}\right)-\frac{1}{2} \sum_{i, j=1}^{d} \rho_{i, j} \sigma_{i}\left(x_{n}\right) \sigma_{j}\left(x_{n}\right) \frac{\partial^{2} f}{\partial x_{i} x_{j}}\left(t_{n}, x_{n} ; \theta\right)$

前面DGM定义的损失
$\begin{array}{l} G_{1}\left(\theta_{n}, s_{n}\right):=\left(\frac{\partial f}{\partial t}\left(t_{n}, x_{n} ; \theta_{n}\right)+\mathcal{L} f\left(t_{n}, x_{n} ; \theta_{n}\right)\right)^{2} \\ G_{2}\left(\theta_{n}, s_{n}\right):=\left(f\left(\tau_{n}, z_{n} ; \theta_{n}\right)-g\left(\tau_{n}, z_{n}\right)\right)^{2} \\ G_{3}\left(\theta_{n}, s_{n}\right):=\left(f\left(0, w_{n} ; \theta_{n}\right)-u_{0}\left(w_{n}\right)\right)^{2} \\ G\left(\theta_{n}, s_{n}\right):=G_{1}\left(\theta_{n}, s_{n}\right)+G_{2}\left(\theta_{n}, s_{n}\right)+G_{3}\left(\theta_{n}, s_{n}\right) \end{array}$

DGM算法中用到梯度 $\nabla_{\theta} G_{1}\left(\theta_{n}, s_{n}\right)$ 近视为 $\tilde{G}_{1}$ 估计，有固定项 $\Delta>0$

$\begin{aligned} \tilde{G}_{1}\left(\theta_{n}, s_{n}\right):=& 2\left(\frac{\partial f}{\partial t}\left(t_{n}, x_{n} ; \theta_{n}\right)+\mathcal{L}_{1} f\left(t_{n}, x_{n} ; \theta_{n}\right)+\frac{1}{2} \sum_{i=1}^{d} \frac{\frac{\partial f}{\partial x_{i}}\left(t, x_{n}+\sigma\left(x_{n}\right) W_{\Delta} ; \theta\right)-\frac{\partial f}{\partial x_{i}}\left(t, x_{n} ; \theta\right)}{\Delta} \sigma_{i}\left(x_{n}\right) W_{\Delta}^{i}\right) \\ \times & \nabla_{\theta}\left(\frac{\partial f}{\partial t}\left(t_{n}, x_{n} ; \theta_{n}\right)+\mathcal{L}_{1} f\left(t_{n}, x_{n} ; \theta_{n}\right)+\frac{1}{2} \sum_{i=1}^{d} \frac{\frac{\partial f}{\partial x_{i}}\left(t, x_{n}+\sigma\left(x_{n}\right) \tilde{W}_{\Delta} ; \theta\right)-\frac{\partial f}{\partial x_{i}}\left(t, x_{n} ; \theta\right)}{\Delta} \sigma_{i}\left(x_{n}\right) \tilde{W}_{\Delta}^{i}\right) \end{aligned}$
其中， $W_{\Delta}$ 是d维正态随机变量，且满足 $\mathbb{E}\left[W_{\Delta}\right]=0$ 以及 $\operatorname{Cov}\left[\left(W_{\Delta}\right)_{i},\left(W_{\Delta}\right)_{i}\right]=\rho_{i, i} \Delta$ . $\tilde{W}_{\Delta}$ 有和 $W_{\Delta}$ 一样的分布. $\tilde{G}_{1}\left(\theta_{n}, s_{n}\right)$ 是， $KaTeX parse error: Expected '\right', got 'EOF' at end of input: …eft(\theta_{n}是$ \nabla_{\theta} G_{1}\left(\theta_{n}, s_{n}\right) $KaTeX parse error: Expected 'EOF', got '\right' at position 23: …Carlo的估计, s_{n}\̲r̲i̲g̲h̲t̲)$ 有 $\mathcal{O}(\sqrt{\Delta})$ 偏差. 这种误差可以根据下面的估计方法改进
$\tilde{G}_{1}\left(\theta_{n}, s_{n}\right) \quad:=\quad \tilde{G}_{1, a}\left(\theta_{n}, s_{n}\right)+\tilde{G}_{1, b}\left(\theta_{n}, s_{n}\right)$
$\begin{aligned} \tilde{G}_{1, a}\left(\theta_{n}, s_{n}\right): &=\left(\frac{\partial f}{\partial t}\left(t_{n}, x_{n} ; \theta_{n}\right)+\mathcal{L}_{1} f\left(t_{n}, x_{n} ; \theta_{n}\right)+\frac{1}{2} \sum_{i=1}^{a} \frac{\frac{\partial J}{\partial x_{i}}\left(t, x_{n}+\sigma\left(x_{n}\right) W_{\Delta} ; \theta\right)-\frac{\sigma_{j}}{\partial x_{i}}\left(t, x_{n} ; \theta\right)}{\Delta} \sigma_{i}\left(x_{n}\right) W_{\Delta}^{i}\right) \\ & \times \nabla_{\theta}\left(\frac{\partial f}{\partial t}\left(t_{n}, x_{n} ; \theta_{n}\right)+\mathcal{L}_{1} f\left(t_{n}, x_{n} ; \theta_{n}\right)+\frac{1}{2} \sum_{i=1}^{d} \frac{\frac{\partial f}{\partial x_{i}}\left(t, x_{n}+\sigma\left(x_{n}\right) \tilde{W}_{\Delta} ; \theta\right)-\frac{\partial f}{\partial x_{i}}\left(t, x_{n} ; \theta\right)}{\Delta} \sigma_{i}\left(x_{n}\right) \tilde{W}_{\Delta}^{i}\right) \end{aligned}$
$KaTeX parse error: Undefined control sequence: \su at position 623: …t)-\frac{1}{2} \̲s̲u̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲ ̲m_{i=1}^{d} \fr…$
上面的估计方法有 $\mathcal{O}(\Delta)$ 的偏差

The computational cost for calculating second derivatives

在这里插入图片描述
The modified algorithm here is computationally less expensive than the original algorithm

Relevant literature

Recently, Raissi [41, 42] develop physics informed deep learning models. They estimate deep neural network models which merge data observations with PDE models. This allows for the estimation of physical models from limited data by leveraging a priori knowledge that the physical dynamics should obey a class of PDEs. Their approach solves PDEs in one and two spatial dimensions using deep neural networks.
[33] developed an algorithm for the solution of a discrete-time version of a class of free boundary PDEs.
Their algorithm, commonly called the Longstaff-Schwartz method", uses dynamic programming and approximates the solution using a separate function approximator at each discrete time (typically a linear combination of basis functions). Our algorithm directly solves the PDE, and uses a single function approximator for all space and all time.

数值实验

The Free Boundary PDE

Free boundary PDE and will satisfy

$\begin{aligned} 0 &=\frac{\partial u}{\partial t}(t, x)+\mu(x) \cdot \frac{\partial u}{\partial x}(t, x)+\frac{1}{2} \sum_{i, j=1}^{d} \rho_{i, j} \sigma\left(x_{i}\right) \sigma\left(x_{j}\right) \frac{\partial^{2} u}{\partial x_{i} \partial x_{j}}(t, x)-r u(t, x), \quad \forall\{(t, x): u(t, x)>g(x)\} \\ u(t, x) & \geq g(x), \quad \forall(t, x) \\ u(t, x) & \in C^{1}\left(\mathbb{R}_{+} \times \mathbb{R}^{d}\right), \quad \forall\{(t, x): u(t, x)=g(x)\} \\ u(T, x) &=g(x), \quad \forall x \end{aligned}$

Implementation details for the algorithm

在训练过程使用了多块GPU，为了加速训练，使用了异步随机梯度下降(asynchronous stochastic gradient descent),这是机器学习模型中应用广泛的并行训练方法. Figure 1 displays the computational setup

在这里插入图片描述

A High-dimensional Free Boundary PDE with a Semi-Analytic Solution

The accuracy of our deep learning algorithm is evaluated in up to 200 dimensions

在这里插入图片描述
We present in Figure 2 contour plots of the absolute error and percent error across time and space for the American option PDE in 20 dimensions.

Figure 2 reports both the absolute error and the percent error. The percent error $\frac{|f(t, x ; \theta)-u(t, x)|}{|u(t, x)|} \times 100 \%$ is reported for points where $∣ u (t, x) ∣ > 0.05$ . The absolute error becomes relatively large in a few areas;however, the solution u(t; x) also grows large in these areas and therefore the percent error remains small

Burgers’ equation

$\begin{aligned} \frac{\partial u}{\partial t}(t, x ; p) &=\mathcal{L}_{p} u(t, x ; p), \quad(t, x) \in[0, T] \times \Omega \\ u(t, x ; p) &=g_{p}(x), \quad(t, x) \in[0, T] \times \partial \Omega \\ u(t=0, x ; p) &=h_{p}(x), \quad x \in \Omega \end{aligned}$
A traditional approach would be to discretize the P-space and re-solve the PDE many times for many different points p. However, the total number of grid points (and therefore the number of PDEs that must be solved) grows exponentially with the number of dimensions, and P is typically high-dimensional.

We use DGM to solve the PDE as follows:
在这里插入图片描述

We consider the following burge equation:
$\frac{\partial u}{\partial t}=\nu \frac{\partial^{2} u}{\partial x^{2}}-\alpha u \frac{\partial u}{\partial x}, \quad(t, x) \in[0,1] \times[0,1]$
$u (t, x = 0) = a$
$u (t, x = 1) = b$
$\quad x \in[0,1]$
The problem setup space is $\mathcal{P}=(\nu, \alpha, a, b) \in \mathbb{R}^{4}$ , the entire space $\nu, \alpha, a, b) \in[0,1] \times[0,1] \times[10^{-2}, 10^{-1}] \times [10^{-2}, 1] \times[-1,1] \times[-1,1]$

Figure 5 compares the deep learning solution with the exact solution for
several different problem setups p

在这里插入图片描述
Figure 5 compares the deep learning solution with the exact solution for several different problem setups p. The solutions are very close; in several cases, the two solutions are visibly indistinguishable. The deep learning algorithm is able to accurately capture the shock layers and boundary layers.

Neural Network Approximation Theorem for PDEs

Figure 6 presents the accuracy of the deep learning algorithm for different times t and different choices of ν.

总结：

We prove that the neural network converges to the solution of the partial differential equation as the number of hidden units increases.
Stability analysis of deep learning and machine learning algorithms for solving PDEs is also an important question. It would certainly be interesting to study machine learning algorithms that use a more direct variational formulation of the involved PDEs. We
leave these questions for future work