dropout层:防止过拟合
nn.dropout 每次将 p 元素设置为 0,剩下的元素乘以 1/(1-p)
eval()模式不进行dropout
使用方法如下:
In [1]: import torch
In [2]: p = 0.5
In [3]: module = torch.nn.Dropout(p)
In [4]: module.training
Out[4]: True
In [5]: inp = torch.ones(3,5)
In [6]: print(inp)
tensor([[1., 1., 1., 1., 1.],
[1., 1., 1., 1., 1.],
[1., 1., 1., 1., 1.]])
In [7]: module(inp)
Out[7]:
tensor([[0., 0., 0., 2., 0.],
[2., 2., 0., 0., 2.],
[2., 0., 2., 2., 2.]])
In [10]: 1/(1-p)
Out[10]: 2.0
In [11]: module.eval()
Out[11]: Dropout(p=0.5, inplace=False)
In [12]: module.training
Out[12]: False
In [13]: module(inp)
Out[13]:
tensor([[1., 1., 1., 1., 1.],
[1., 1., 1., 1., 1.],
[1., 1., 1., 1., 1.]])
Linear层
In [1]: import torch
In [2]: module = torch.nn.Linear(10,20)
In [3]: module
Out[3]: Linear(in_features=10, out_features=20, bias=True)
In [4]: n_samples = 40
In [5]: inp_2d = torch.rand(n_samples,10)
In [6]: module(inp_2d).shape
Out[6]: torch.Size([40, 20])
In [7]: inp_3d = torch.rand(n_samples,33,10)
In [8]: module(inp_3d).shape
Out[8]: torch.Size([40, 33, 20])
In [10]: input_7d = torch.rand(n_samples,2,3,4,5,6,10)
In [12]: module(input_7d).shape
Out[12]: torch.Size([40, 2, 3, 4, 5, 6, 20])
LayerNorm属性
可学习参数为0
In [1]: import torch
In [2]: inp = torch.tensor([[0,4.],[-1,7],[3,5]])
In [3]: n_samples,n_features = inp.shape
In [4]: print(n_samples)
3
In [5]: module = torch.nn.LayerNorm(n_features, elementwise_affine=False)
In [6]: sum(p.numel() for p in module.parameters() if p.requires_grad)
Out[6]: 0
In [7]: inp.mean(-1),inp.std(-1,unbiased=False)
Out[7]: (tensor([2., 3., 4.]), tensor([2., 4., 1.]))
In [8]: module(inp).mean(-1),module(inp).std(-1,unbiased=False)
Out[8]:
(tensor([ 0.0000e+00, -2.9802e-08, 1.1921e-07]),
tensor([1.0000, 1.0000, 1.0000]))
可学习参数为4
In [9]: module = torch.nn.LayerNorm(n_features, elementwise_affine=True)
In [10]: sum(p.numel() for p in module.parameters() if p.requires_grad)
Out[10]: 4
In [11]: (module.bias,module.weight)
Out[11]:
(Parameter containing:
tensor([0., 0.], requires_grad=True),
Parameter containing:
tensor([1., 1.], requires_grad=True))
In [12]: module(inp).mean(-1),module(inp).std(-1,unbiased=False)
Out[12]:
(tensor([ 0.0000e+00, -2.9802e-08, 1.1921e-07], grad_fn=<MeanBackward1>),
tensor([1.0000, 1.0000, 1.0000], grad_fn=<StdBackward1>))
In [13]: module.bias.data += 1
In [14]: module.weight.data *= 4
In [15]: module(inp).mean(-1),module(inp).std(-1,unbiased=False)
Out[15]:
(tensor([1.0000, 1.0000, 1.0000], grad_fn=<MeanBackward1>),
tensor([4.0000, 4.0000, 4.0000], grad_fn=<StdBackward1>))
只更新n_features,样本间独立,与n_samples无关
In [16]: module(torch.rand(n_samples,2,3,4,5,6,n_features)).shape
Out[16]: torch.Size([3, 2, 3, 4, 5, 6, 2])
In [17]: module(torch.rand(n_samples,2,3,4,5,6,n_features)).mean(-1)
Out[17]:
tensor([[[[[[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000],
[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000],
[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000],
[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000],
[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000]],
[[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000],
[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000],
[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000],
[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000],
[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000]]]]]],
grad_fn=<MeanBackward1>)
In [18]: module(torch.rand(n_samples,2,3,4,5,6,n_features)).std(-1,unbiased=False)
Out[18]:
tensor([[[[[[3.9998, 3.9962, 3.9993, 3.9966, 3.9657, 3.9954],
[3.9998, 3.9997, 3.9996, 3.9981, 3.9981, 3.9983],
[3.9799, 3.9996, 3.9657, 3.9998, 3.9998, 3.9986],
[3.9990, 3.9950, 3.9885, 3.9996, 3.9582, 3.9996],
[3.9987, 3.9989, 3.9900, 3.9992, 3.9992, 3.9994]],
[[3.9996, 3.9996, 3.9972, 3.9931, 3.9998, 3.5468],
[3.9998, 3.9808, 3.9974, 3.9985, 3.9992, 3.9986],
[1.1207, 3.9993, 3.9998, 3.6870, 3.9997, 3.9981],
[3.9998, 3.9998, 3.9986, 3.9676, 3.9999, 3.9998],
[3.9993, 3.9999, 3.9998, 3.9852, 3.9993, 3.9891]]]]]],
grad_fn=<StdBackward1>)