Local extreme learning machines and domain decomposition for solving linear and nonlinear PDE论文阅读笔记

Local extreme learning machines and domain decomposition for solving linear and nonlinear partial differential equations

• Suchuan Dong, Zongwei Lib
• Center for Computational and Applied Mathematics, Department ofMathematics, Purdue University West Lafayette, IN, USA
• Comput. Methods Appl. Mech. Engrg. 2021

Abstract

• We present a neural network-based method for solving linear and nonlinear partial differential equations, by combining the ideas of extreme learning machines (ELM), domain decomposition and local neural networks.
• The field solution on each sub-domain is represented by a local feed-forward neural network, and $C^k$ continuity conditions are imposed on the sub-domain boundaries. Each local neural network consists of a small number of hidden layers, while its last hidden layer can be wide.

Motivation

• Several successful DNN-based PDE solvers have emerged in the past years, such as the deep Galerkin method (DGM), the physics-informed neural network (PINN), and related approaches . Neural network-based PDE solutions are smooth analytical functions, provided that smooth activation functions are used therein. The solution and its derivatives can then be computed exactly, by evaluation of the neural network or by auto -differentiation.
• While their computational performance is promising, DNN-based PDE solvers, in their current state, suffer from a number of limitations that make them numerically less than satisfactory and computationally uncompetitive.
  • The first limitation is the solution accuracy of DNN-based methods. A survey of related literature indicates
    that the absolute error of the current DNN-based methods is generally on, and rarely goes below, the level of$10^{-3}-10^{-4}$.
  • Another limitation concerns the computational cost.
    The computational cost of DNN-based PDE solvers is extremely high.
• In the current work we concentrate on the accuracy and the computational cost of neural network-based numerical
  methods.

Contribution

• Network architecture and training parameters. The current method is based on shallow feed-forward neural networks.
• Training method. The network is trained and the values for the training parameters are determined by a least squares computation, not by the back propagation (gradient descent-type) algorithm. For linear PDEs, training the neural network involves a linear least squares computation. For nonlinear PDEs, the network training involves a nonlinear least squares computation.
• Domain decomposition and local neural networks. We partition the overall domain into sub-domains, and
  represent the solution on each sub-domain locally by a shallow feed-forward neural network.
• Block time marching. For long-time simulations of time-dependent PDEs, the current method adopts a block
  time-marching strategy. （在时间上被划分为几个窗口，称为时间块。然后依次在每个时间块的时空域上分别求解偏微分方程

Main contribution：
（1）A main contribution of this work is the introduction of
an ELM-like method for nonlinear differential equations, based on domain decomposition and local neural networks. In contrast, existing ELM-based methods for differential equations have been confined to linear problems, and the neural network is limited to a single hidden layer. For nonlinear problems, to solve the nonlinear system for the training parameters, we have adopted two methods:

• a nonlinear least squares method with perturbations (referred
  to as NLSQ-perturb)
• a combined Newton/linear least squares method (referred to as Newton-LLSQ).

We find that the random perturbation in the NLSQ-perturb method is crucial to preventing the method from being
trapped to local minima with cost values exceeding some given tolerance, especially in under-resolved cases and in
long-time simulations.

（2）An other contribution of the current work is the aforementioned block time-marching scheme for long-time
simulations of time-dependent linear/nonlinear PDEs

Method

Local extreme learning machines (locELM) for representing functions

ELM

$$u_i^s(\mathbf{x})=\sum _{j=1}^M{V}_j^s(\mathbf{x})w_{ji}^s,\mathbf{x}\in {\Omega }_s,1⩽i⩽N$$

$$f_s(\mathbf{x})=\left(u_1^s,u_2^s,\dots ,u_N^s\right)$$

Remark：Apart from the above logical operations, in the implementation we incorporate an additional
normalization layer immediately behind the input layer in each of the local neural networks. For each subdomain emn, the normalization layer performs an affine mapping and normalizes the input data, $(x,y)\in {\Omega }_{emn}=[X_m,X_{m+1}]\times [Y_n,Y_{n+1}]$, such that the output data of the normalization layer fall into the domain [−1, 1]×[−1, 1]. This extra normalization layer contains no adjustable (training) parameters.

Time-independent linear PDE

$$\begin{array}{l}Lu=f(x,y),\\ u(x,y)=g(x,y),\text{ on }\partial \Omega ,\end{array}$$
$$u^{e_{mn}}(x,y)=\sum _{j=1}^M{V}_j^{e_{mn}}(x,y)w_j^{e_{mn}},(x,y)\in {\Omega }_{e_{mn}},0⩽m⩽N_x-1,0⩽n⩽N_y-1,$$
$$\begin{array}{l}{\sum }_{j=1}^M\left[L{V}_j^{e_{mn}}\left(x_p^{e_{mn}},y_q^{e_{mn}}\right)\right]w_j^{e_{mn}}=f\left(x_p^{e_{mn}},y_q^{e_{mn}}\right),\\ \text{ for }0⩽m⩽N_x-1,0⩽n⩽N_y-1,0⩽p⩽Q_x-1,0⩽q⩽Q_y-1,\end{array}$$
where， We enforce Eq on the four boundaries of the domain Ω
$$\begin{array}{l}{\sum }_{j=1}^M{V}_j^{e_{0n}}\left(a_1,y_q^{e_{0n}}\right)w_j^{e_{0n}}=g\left(a_1,y_q^{e_{0n}}\right),\\ {\sum }_{j=1}^M{V}_j^{e_{mn}}\left(b_1,y_q^{e_{mn}}\right)w_j^{e_{mn}}=g\left(b_1,y_q^{e_{mn}}\right)\\ {\sum }_{j=1}^M{V}_j^{e_{m0}}\left(x_p^{e_{m0}},a_2\right)w_j^{e_{m0}}=g\left(x_p^{e_{m0}},a_2\right)\\ {\sum }_{j=1}^M{V}_j^{e_{mn}}\left(x_p^{e_{mn}},b_2\right)w_j^{e_{mn}}=g\left(x_p^{e_{mn}},b_2\right)\end{array}$$
continuity conditions
$$\begin{array}{l}{\sum }_{j=1}^M{V}_j^{e_{mn}}\left(X_{m+1},y_q^{e_{mn}}\right)w_j^{e_{mn}}-{\sum }_{j=1}^M{V}_j^{e_{m+1,n}}\left(X_{m+1},y_q^{e_{m+1,n}}\right)w_j^{e_{m+1,n}}=0,\\ {{\sum }_{j=1}^M\frac{\partial V_j^{e_{mn}}}{\partial x}|}_{\left(X_{m+1},y_q^{e_{mn}}\right)}{w}_j^{e_{mn}}-{{\sum }_{j=1}^M\frac{\partial V_j^{e_{m+1,n}}}{\partial x}|}_{\left(X_{m+1},y_q^{e_{m+1,n}}\right)}{w}_j^{e_{m+1,n}}=0,\\ \text{ for }0⩽m⩽N_x-2,0⩽n⩽N_y-1,0⩽q⩽Q_y-1,\end{array}$$
$$\begin{array}{l}{\sum }_{j=1}^M{V}_j^{e_{mn}}\left(x_p^{e_{mn}},Y_{n+1}\right)w_j^{e_{mn}}-{\sum }_{j=1}^M{V}_j^{e_{m,n+1}}\left(x_p^{e_{m,n+1}},Y_{n+1}\right)w_j^{e_{m,n+1}}=0,\\ {{\sum }_{j=1}^M\frac{\partial V_j^{e_{mn}}}{\partial y}|}_{\left(x_p^{e_{mn}},Y_{n+1}\right)}{w}_j^{e_{mn}}-{{\sum }_{j=1}^M\frac{\partial V_j^{e_{m,n+1}}}{\partial y}|}_{\left(x_p^{e_{mn+1}},Y_{n+1}\right)}{w}_j^{e_{m,n+1}}=0,\\ \text{ for }0⩽m⩽N_x-1,0⩽n⩽N_y-2,0⩽p⩽Q_x-1,\end{array}$$

Time-dependent linear differential equations

类似，增加时间分区域
$$\begin{array}{l}{\sum }_{j=1}^M{V}_j^{e_{mnl}}\left(x_p^{e_{mnl}},y_q^{e_{mnl}},T_{l+1}\right)w_j^{e_{mnl}}-{\sum }_{j=1}^M{V}_j^{e_{mn,l+1}}\left(x_p^{e_{mn,l+1}},y_q^{e_{mn,l+1}},T_{l+1}\right)w_j^{e_{mn,l+1}}=0,\\ 0⩽m⩽N_x-1,0⩽n⩽N_y-1,0⩽l⩽N_t-2,0⩽p⩽Q_x-1,0⩽q⩽Q_y-1\end{array}$$

Nonlinear differential equations

$$\begin{array}{l}Lu+F\left(u,u_x,u_y\right)=f(x,y),\\ u(x,y)=g(x,y),\text{ on }\partial \Omega ,\end{array}（22）$$
$$\begin{array}{rl}{u}^{e_{mn}}(x,y)& =\sum _{j=1}^M{V}_j^{e_{mn}}(x,y)w_j^{e_{mn}},\frac{\partial u^{e_{mn}}}{\partial x}=\sum _{j=1}^M\frac{\partial V_j^{e_{mn}}}{\partial x}{w}_j^{e_{mn}},\frac{\partial u^{e_{mn}}}{\partial y}=\sum _{j=1}^M\frac{\partial V_j^{e_{mn}}}{\partial y}{w}_j^{e_{mn}},\\ \text{ for }0⩽m⩽N_x-1,0⩽n⩽N_y-1,& \end{array}$$
$$\begin{array}{c}{\sum }_{j=1}^M\left[L{V}_j^{e_{mn}}\left(x_p^{e_{mn}},y_q^{e_{mn}}\right)\right]w_j^{e_{mn}}+{F\left(u^{e_{mn}},u_x^{e_{mn}},u_y^{e_{mn}}\right)|}_{\left(x_p^{e_{mn}},y_q^{e_{mn}}\right)}-f\left(x_p^{e_{mn}},y_q^{e_{mn}}\right)=0,\\ \text{ for }0⩽m⩽N_x-1,0⩽n⩽N_y-1,0⩽p⩽Q_x-1,0⩽q⩽Q_y-1,\end{array}$$

• NLSQ
  
• Newton-LLSQ
  we first linearize Eq. (22a) to arrive at a linear differential equation about the increment field.

We observe that the convergence behavior of the Newton-LLSQ method is not as regular as the NLSQ-perturb method, but it appears less likely to be trapped to local minimum solutions.

思考

• 非规则区域