2013-03-18干活小计

13:00 - 23.00

13点才到实验室 汗

做了一下午思政知识图谱

复现了transformer位置编码：

class Positional(nn.Module):
    def __init__(self, pos_max_len, embedding_dim, deivce):
        super().__init__()
        self.device = device
        self.position_embedding = torch.zeros(size=(pos_max_len, embedding_dim), dtype=torch.float32)
        self.omega = torch.arange(embedding_dim/2, dtype=torch.float32)
        self.omege = self.omega/embedding_dim/2
        self.omega = 1./(10000 ** self.omega)
        out = (torch.arange(1, pos_max_len+1, dtype=torch.float32)[:, None])@(self.omega[None, :] )
        emb_sin = torch.sin(out)
        emb_cos = torch.cos(out)
        self.position_embedding[:, 0::2] = emb_sin
        self.position_embedding[:, 1::2] = emb_cos
        
    def forward(self, x):
        pos = self.position_embedding.unsqueeze(dim=0).to(self.device)
        pos = pos[:, :x.shape[1], :]
        result = x+pos
        return result

注意：out = (torch.arange(1, pos_max_len+1, dtype=torch.float32)[:, None])@(self.omega[None, :] )  这句话把两个一维tensor 变成了相应的二维列行向量

我在高质量的复现transformer 所以今天做的事没那么多