Rezero Linear Pytorch

最新推荐文章于 2020-08-11 22:30:11 发布

coolsunxu

最新推荐文章于 2020-08-11 22:30:11 发布

阅读量242

点赞数

分类专栏： Pytorch Python 文章标签： pytorch python 深度学习

本文链接：https://blog.csdn.net/coolsunxu/article/details/105279064

版权

Python 同时被 2 个专栏收录

55 篇文章 2 订阅

订阅专栏

Pytorch

8 篇文章 2 订阅

订阅专栏


import torch
import torch.nn as nn


class Rezero_Linear(nn.Module):
	def __init__(self,inputs,outputs):
		super(Rezero_Linear,self).__init__()
		self.inputs = inputs
		self.outputs = outputs
		self.resweight = nn.Parameter(torch.Tensor([0]))
		self.fc = nn.Linear(self.inputs,self.outputs)
		
		self.init_weights() # 初始化权重
	
	def init_weights(self):
		torch.nn.init.xavier_normal_(self.fc.weight, 
						gain=torch.sqrt(torch.tensor(2.)))
						
	def forward(self,x):
		x = x + self.resweight * torch.relu(self.fc(x))
		return x

"""
a = torch.randn(32, 512)
model = Rezero_Linear(512,512)
ou = model(a)
print(ou.shape)
"""