[技术分享]神经网络中的动态图

Paddle2中支持动态图，动态图是指神经网络的结构可以动态变化，build with run，类似于Python语言中变量不需要定义好类型就可以赋值。这样带来的好处是使得网络更加灵活，更具运行的情况来调整数据流走向，确定就是运行比较慢，这和Python的缺点一样。参考一、参考二。

例如：

import paddle
import paddle.nn.functional as F
import numpy as np


class MyModel(paddle.nn.Layer):
    def __init__(self, input_size, hidden_size):
        super(MyModel, self).__init__()
        self.linear1 = paddle.nn.Linear(input_size, hidden_size)
        self.linear2 = paddle.nn.Linear(hidden_size, hidden_size)
        self.linear3 = paddle.nn.Linear(hidden_size, 1)

    def forward(self, inputs):
        x = self.linear1(inputs)
        x = F.relu(x)

        if paddle.rand([1,]) > 0.5: 
            x = self.linear2(x)
            x = F.relu(x)

        x = self.linear3(x)
        
        return x     



total_data, batch_size, input_size, hidden_size = 1000, 64, 128, 256

x_data = np.random.randn(total_data, input_size).astype(np.float32)
y_data = np.random.randn(total_data, 1).astype(np.float32)

model = MyModel(input_size, hidden_size)
paddle.summary(model, (input_size))

loss_fn = paddle.nn.MSELoss(reduction='mean')
optimizer = paddle.optimizer.SGD(learning_rate=0.01, 
                                 parameters=model.parameters())

for t in range(200 * (total_data // batch_size)):
    idx = np.random.choice(total_data, batch_size, replace=False)
    x = paddle.to_tensor(x_data[idx,:])
    y = paddle.to_tensor(y_data[idx,:])
    y_pred = model(x)

    loss = loss_fn(y_pred, y)
    if t % 200 == 0:
        print(t, loss.numpy())

    loss.backward()
    optimizer.step()
    optimizer.clear_grad()

上述代码中第二层的隐藏层出现的概率是50%。网络的结构可能是这样：

---------------------------------------------------------------------------
 Layer (type)       Input Shape          Output Shape         Param #    
===========================================================================
   Linear-19          [[128]]               [256]             33,024     
   Linear-20          [[256]]               [256]             65,792     
   Linear-21          [[256]]                [1]                257      
===========================================================================
Total params: 99,073
Trainable params: 99,073
Non-trainable params: 0
---------------------------------------------------------------------------
Input size (MB): 0.00
Forward/backward pass size (MB): 0.00
Params size (MB): 0.38
Estimated Total Size (MB): 0.38
---------------------------------------------------------------------------

再次运行也可能是这样：

---------------------------------------------------------------------------
 Layer (type)       Input Shape          Output Shape         Param #    
===========================================================================
   Linear-25          [[128]]               [256]             33,024     
   Linear-27          [[256]]                [1]                257      
===========================================================================
Total params: 33,281
Trainable params: 33,281
Non-trainable params: 0
---------------------------------------------------------------------------
Input size (MB): 0.00
Forward/backward pass size (MB): 0.00
Params size (MB): 0.13
Estimated Total Size (MB): 0.13
---------------------------------------------------------------------------

那么网络的结构在什么时候确定下来呢？是定义模型的时候？是训练模型的时候，还是模型推理的时候？

在上述代码最后加上paddle.summary(model, (input_size))查看训练后的网络：

---------------------------------------------------------------------------
 Layer (type)       Input Shape          Output Shape         Param #    
===========================================================================
   Linear-34          [[128]]               [256]             33,024     
   Linear-35          [[256]]               [256]             65,792     
   Linear-36          [[256]]                [1]                257      
===========================================================================
Total params: 99,073
Trainable params: 99,073
Non-trainable params: 0
---------------------------------------------------------------------------
Input size (MB): 0.00
Forward/backward pass size (MB): 0.00
Params size (MB): 0.38
Estimated Total Size (MB): 0.38
---------------------------------------------------------------------------

运行以下模型推理：

x = np.random.randn(input_size).astype(np.float32)
x =  paddle.to_tensor(x)
y_infer = model(x)
print(y_infer)

paddle.summary(model, (input_size))

结果也有两种：

Tensor(shape=[1], dtype=float32, place=CUDAPlace(0), stop_gradient=False,
       [0.94865662])
---------------------------------------------------------------------------
 Layer (type)       Input Shape          Output Shape         Param #    
===========================================================================
   Linear-34          [[128]]               [256]             33,024     
   Linear-36          [[256]]                [1]                257      
===========================================================================
Total params: 33,281
Trainable params: 33,281
Non-trainable params: 0
---------------------------------------------------------------------------
Input size (MB): 0.00
Forward/backward pass size (MB): 0.00
Params size (MB): 0.13
Estimated Total Size (MB): 0.13
---------------------------------------------------------------------------

{'total_params': 33281, 'trainable_params': 33281}

或者

Tensor(shape=[1], dtype=float32, place=CUDAPlace(0), stop_gradient=False,
       [-0.08627059])
---------------------------------------------------------------------------
 Layer (type)       Input Shape          Output Shape         Param #    
===========================================================================
   Linear-34          [[128]]               [256]             33,024     
   Linear-35          [[256]]               [256]             65,792     
   Linear-36          [[256]]                [1]                257      
===========================================================================
Total params: 99,073
Trainable params: 99,073
Non-trainable params: 0
---------------------------------------------------------------------------
Input size (MB): 0.00
Forward/backward pass size (MB): 0.00
Params size (MB): 0.38
Estimated Total Size (MB): 0.38
---------------------------------------------------------------------------

这说明模型的大小在定义时是动态的，训练时是动态的，推理时还是动态的。以上代码可在百度AI Studio的notebook中运行。