LSTM:长短期记忆网络

理论

LSTM的核心是细胞状态，用贯穿细胞的水平线表示

1.计算遗忘门

决定细胞状态需要舍弃哪部分无用信息

$$f_t=\sigma {}_g(W_f{x}_t+U_f{h}_{t-1}+b_f)$$

2.计算输入门

决定细胞状态需要添加哪些有用信息

$$i_t=\sigma {}_g(W_i{x}_t+U_i{h}_{t-1}+b_i)$$

3.计算候选细胞状态

$${\tilde{c}}_t=\sigma {}_c(W_c{x}_t+U_c{h}_{t-1}+b_c)$$

4.更新细胞状态

$$c_t=f_t\circ c_{t-1}+i_t\circ {\tilde{c}}_t$$

5.计算输出门

控制细胞状态中哪些信息被输出
$$o_t=\sigma {}_g(W_o{x}_t+U_o{h}_{t-1}+b_o)$$

6.计算输出隐状态

$$h_t=o_t\circ \sigma {}_h(c_t)$$

实践

从零实现LSTM

class My_LSTM(nn. Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.hidden_size = hidden_size
        self.gates = nn.Linear(input_size + hidden_size, hidden_size * 4)
        self.sigmoid = nn.Sigmoid()
        self.tanh = nn. Tanh()
        self.output = nn.Sequential(
            nn.Linear(hidden_size, hidden_size // 2),
            nn.ReLU(),
            nn.Linear(hidden_size // 2, output_size)
        )
        for param in self.parameters():
            if param.dim() > 1:
                nn.init.xavier_uniform_(param)

    def forward(self, x):
        batch_size = x.size(0)
        seq_len = x.size(1)
        h, c = (torch.zeros(batch_size, self.hidden_size).to(x.device) for _ in range(2))
        y_list = []
        for i in range(seq_len):
            forget_gate, input_gate, output_gate, candidate_cell = \
                self.gates(torch.cat([x[:, i, :], h], dim=-1)).chunk(4, -1)
            forget_gate, input_gate, output_gate = (self.sigmoid(g)
                                                    for g in (forget_gate, input_gate, output_gate))
            c = forget_gate * c + input_gate * self.tanh(candidate_cell)
            h = output_gate * self.tanh(c)
            y_list.append(self.output(h))
        return torch.stack(y_list, dim=1), (h, c)

Pytorch实现LSTM

参数

输入

输出

lstm = nn.LSTM(input_size=input_size,hidden_size=hidden_size,num_layers=1,batch_first=True).to(device)