Classical ML Equations in LaTeX

A collection of classical ML equations in Latex . Some of them are provided with simple notes and paper link. Hopes to help writings such as papers and blogs.

Better viewed at https://blmoistawinde.github.io/ml_equations_latex/

• Classical ML Equations in LaTeX
  • Model
    • RNNs(LSTM, GRU)
    • Attentional Seq2seq
      • Bahdanau Attention
      • Luong(Dot-Product) Attention
    • Transformer
      • Scaled Dot-Product attention
      • Multi-head attention
    • Generative Adversarial Networks(GAN)
      • Minmax game objective
    • Variational Auto-Encoder(VAE)
      • Reparameterization trick
  • Activations
    • Sigmoid
    • Tanh
    • Softmax
    • Relu
    • Gelu
  • Loss
    • Regression
      • Mean Absolute Error(MAE)
      • Mean Squared Error(MSE)
      • Huber loss
    • Classification
      • Cross Entropy
      • Negative Loglikelihood
      • Hinge loss
      • KL/JS divergence
    • Regularization
      • L1 regularization
      • L2 regularization
  • Metrics
    • Classification
      • Accuracy, Precision, Recall, F1
      • Sensitivity, Specificity and AUC
    • Regression
    • Clustering
      • (Normalized) Mutual Information (NMI)
    • Ranking
      • (Mean) Average Precision(MAP)
    • Similarity/Relevance
      • Cosine
      • Jaccard
      • Pointwise Mutual Information(PMI)
  • Notes
  • Reference

Model

RNNs(LSTM, GRU)

encoder hidden state $h_t$ at time step $t$, with input token embedding $x_t$

$$h_t = RNN_{enc}(x_t, h_{t-1})$$

decoder hidden state $s_t$ at time step $t$, with input token embedding $y_t$

$$s_t = RNN_{dec}(y_t, s_{t-1})$$

h_t = RNN_{enc}(x_t, h_{t-1})
s_t = RNN_{dec}(y_t, s_{t-1})

The $RNN_{enc}$, $RNN_{dec}$ are usually either

• LSTM (paper: Long short-term memory)

• GRU (paper: Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation).

Attentional Seq2seq

The attention weight $\alpha_{ij}$, the $i$th decoder step over the $j$th encoder step, resulting in context vector $c_i$

$$c_i = \sum_{j=1}^{T_x} \alpha_{ij}h_j$$

$$\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k=1}^{T_x} \exp(e_{ik})}$$

$$e_{ij} = a(s_{i-1}, h_j)$$

c_i = \sum_{j=1}^{T_x} \alpha_{ij}h_j

\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k=1}^{T_x} \exp(e_{ik})}

e_{ij} = a(s_{i-1}, h_j)

$a$ is an specific attention function, which can be

Bahdanau Attention

Paper: Neural Machine Translation by Jointly Learning to Align and Translate

$$e_{ij} = v^T tanh(W[s_{i-1}; h_j])$$

e_{ij} = v^T tanh(W[s_{i-1}; h_j])

Luong(Dot-Product) Attention

Paper: Effective Approaches to Attention-based Neural Machine Translation

If $s_i$ and $h_j$ has same number of dimension.

$$e_{ij} = s_{i-1}^T h_j$$

otherwise

$$e_{ij} = s_{i-1}^T W h_j$$

e_{ij} = s_{i-1}^T h_j

e_{ij} = s_{i-1}^T W h_j

Finally, the output $o_i$ is produced by:

$$s_t = tanh(W[s_{t-1};y_t;c_t])$$

$$o_t = softmax(Vs_t)$$

s_t = tanh(W[s_{t-1};y_t;c_t])
o_t = softmax(Vs_t)

Transformer

Paper: Attention Is All You Need

Scaled Dot-Product attention

$$Attention(Q, K, V) = softmax(\frac{QK^T}{\sqrt{d_k}})V$$

Attention(Q, K, V) = softmax(\frac{QK^T}{\sqrt{d_k}})V

where $\sqrt{d_k}$ is the dimension of the key vector $k$ and query vector $q$ .

Multi-head attention

$$MultiHead(Q, K, V) = Concat(head_1, ..., head_h)W^O$$

where

$$head_i = Attention(Q W^Q_i, K W^K_i, V W^V_i)$$

MultiHead(Q, K, V) = Concat(head_1, ..., head_h)W^O

head_i = Attention(Q W^Q_i, K W^K_i, V W^V_i)

Generative Adversarial Networks(GAN)

Paper: Generative Adversarial Networks

Minmax game objective