AI/机器学习常用公式的LaTex代码汇总

在写AI/机器学习相关的论文或者博客的时候经常需要用到LaTex的公式，然而作为资深“伸手党”的我在网上搜索的时候，居然没有找到相关现成资源@-@

那么，我就把自己经常会遇到的公式整理如下，以NLP和一些通用指标函数为主。有需要的可以自取，当然发现有问题或者遗漏的也欢迎指正和补充。（我同步到了Github上( https://github.com/blmoistawinde/ml_equations_latex )，欢迎提issue和PR，当然还有star~）

Classical ML Equations in LaTeX

A collection of classical ML equations in Latex . Some of them are provided with simple notes and paper link. Hopes to help writings such as papers and blogs.

Better viewed at https://blmoistawinde.github.io/ml_equations_latex/

• Classical ML Equations in LaTeX
  • Model
    • RNNs(LSTM, GRU)
    • Attentional Seq2seq
      • Bahdanau Attention
      • Luong(Dot-Product) Attention
    • Transformer
      • Scaled Dot-Product attention
      • Multi-head attention
    • Generative Adversarial Networks(GAN)
      • Minmax game objective
    • Variational Auto-Encoder(VAE)
      • Reparameterization trick
  • Activations
    • Sigmoid
    • Softmax
    • Relu
  • Loss
    • Regression
      • Mean Absolute Error(MAE)
      • Mean Squared Error(MSE)
      • Huber loss
    • Classification
      • Cross Entropy
      • Negative Loglikelihood
      • Hinge loss
      • KL/JS divergence
    • Regularization
      • L1 regularization
      • L2 regularization
  • Metrics
    • Classification
      • Accuracy, Precision, Recall, F1
      • Sensitivity, Specificity and AUC
    • Regression
    • Clustering
      • (Normalized) Mutual Information (NMI)
    • Ranking
      • (Mean) Average Precision(MAP)
    • Similarity/Relevance
      • Cosine
      • Jaccard
      • Pointwise Mutual Information(PMI)
  • Notes
  • Reference

Model

RNNs(LSTM, GRU)

encoder hidden state $h_t$ at time step $t$
$$h_t=RN{N}_{enc}(x_t,h_{t-1})$$

decoder hidden state $s_t$ at time step $t$

$$s_t=RN{N}_{dec}(y_t,s_{t-1})$$

h_t = RNN_{enc}(x_t, h_{t-1})
s_t = RNN_{dec}(y_t, s_{t-1})

The $RN{N}_{enc}$, $RN{N}_{dec}$ are usually either

• LSTM (paper: Long short-term memory)

• GRU (paper: Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation).

Attentional Seq2seq

The attention weight ${\alpha }_{ij}$, the $i$th decoder step over the $j$th encoder step, resulting in context vector $c_i$

$$c_i=\sum _{j=1}^{T_x}{\alpha }_{ij}{h}_j$$

$${\alpha }_{ij}=\frac{\exp (e_{ij})}{{\sum }_{k=1}^{T_x}\exp (e_{ik})}$$

$$e_{ik}=a(s_{i-1},h_j)$$

c_i = \sum_{j=1}^{T_x} \alpha_{ij}h_j

\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k=1}^{T_x} \exp(e_{ik})}

e_{ik} = a(s_{i-1}, h_j)

$a$ is an specific attention function, which can be

Bahdanau Attention

Paper: Neural Machine Translation by Jointly Learning to Align and Translate

$$e_{ik}=v^{T}tanh(W[s_{i-1};h_j])$$

e_{ik} = v^T tanh(W[s_{i-1}; h_j])

Luong(Dot-Product) Attention

Paper: Effective Approaches to Attention-based Neural Machine Translation

If $s_i$ and $h_j$ has same number of dimension.

$$e_{ik}=s_{i-1}^T{h}_j$$

otherwise

$$e_{ik}=s_{i-1}^{T}W{h}_j$$

e_{ik} = s_{i-1}^T h_j

e_{ik} = s_{i-1}^T W h_j

Finally, the output o