Translation Model

To give some context I have been working with contextual models (I dont know if this is an official term but I mean models which have context about their previous states and can modify the future predictions along with updating the states.) like RNNs, LSTMs and GRUs for basic language modelling. I have also mentioned this in my previous Personal Note about Temperature Sampling about working with these models.

I was following the direction of d2l.ai book for learning about contextual models and was moving from from char prediction model and then moving to machine translation. I trained a decent translator model using the dataset given at manythings/anki for Eng-French translated sentences.

My Implementation : Machine Translation Notebook I have added everything in my notebook from data collection to data formatting and training.

My motivation to learn about "old" models was to get why Attention changed everything and how models actually work in-depth. I know a lot of RL stuff is also involved in training phases. But the limitation of context and recurrent models, I got to know about after experimenting with old models.

I would like to first introduce the models which I experimented with and also point out where they failed when I wanted to push the limits as this would give more context about where context actually failed despite of using Encoder-Decoder Arch with GRU Layers.

Model Experimentations

RNN

I started with RNN which has a very basic context structure. We have no option but to send context of everything forward, we don't have choice to choose which particular input is more important for future prediction or some detection (I mean the model has no option to learn because of the architecture).

It operates on hidden states per layer and a RNN layer is defined by :

H_t = \phi(X_tW_{xh} + H_{t-1}W_{hh} + b_h)

Here,

$X_t$ : Input for the sequence time step $t$
$W_{xh}$ : Weights matrix for Input
$H_{t-1}$ : Hidden state from previous timestamp
$W_{hh}$ : Weights matrix for Hidden state layer
$b_h$ : bias for the layer

We can add multiple RNN layers and we get a "deep" RNN network. One thing I personally observed after seeing this description of layer was how do calculate the gradient of a layer which is recurssive ?

--> Answer : It is complicated if calculated in vanilla style as we will have to consider all previous hidden states and so on but there are some techinques which can be used in practice in order to "estimate" the gradient calculation such as :

Truncated Time Steps
Randomized Truncation

I designed a char prediction model like a character llm (which you might be familiar with karpathy's video about charLM). So I give a small prompt and the model predicts the next $n$ chars based on the prompt provided. It worked kind of well for small sentences and if prediction length requested was small.

Example (as given in the Temperature Blog) :

Prompt: he called me and sa
=================================
Generated Text (Greedy): he called me and sa[w a minute]

But when I want to generate more text for the same prompt weird things happen :

Prompt: he called me and sa
=================================
Generated Text (Greedy): he called me and sa[w a minute of the solent at the sound the solent at the sound the solent at the sound]..... 

This goes on and on.

This is because of the limitation of RNN models to generate more text given the prompt as the context gets noisy per char generation and eventually gets bad. Even If we give longer sequences we have difficulty in actually generating something meaningful and also If sequences are too long we will see vanishing/exploding gradient problem without actually learning something meaningful.

Till now our context is bad and gradients are a bit dicey.

LSTM

Long Short Term Memory.

⣿⣛⣿⠿⣿⣿⣶⣯⣝⣢⠍⠻⢷⣯⣟⢿⣿⣿⣷⣝⡿⢙⢿⣿
⣾⣽⡻⠿⣷⣾⣯⣽⣻⠿⣿⣶⣦⣌⠻⢿⣿⢿⣿⣷⣕⡘⢞⡞
⣿⣿⣿⣷⣄⠠⣉⣙⠛⠻⢮⣟⠻⠽⣛⣢⠙⠗⠊⢿⣿⣿⣕⡹
⣿⣿⣿⣿⣿⣷⣮⡻⣿⣿⣶⣬⣑⢔⠖⢀⠠⣶⣌⢷⡻⣿⢿⡞
⣿⣿⣿⣿⣿⣿⣿⢻⣜⢻⣟⣟⡽⠁⠀⡠⢢⡼⢣⣧⠱⣽⠈⣿
⣷⣭⡻⠿⣿⣿⣿⣷⡝⠳⡻⡿⡁⣄⡌⠱⢞⣵⣿⣯⣤⢸⠀⠘
⠻⢮⣟⢿⣶⣦⣭⣭⣭⣥⡑⠻⣷⣾⣷⣶⣿⣿⣿⣿⣿⠚⢰⡇
⢷⣆⣙⡛⢮⣕⡋⠝⠻⢍⠉⠁⣐⣿⣿⣿⣿⣿⣿⣿⣧⣼⣾⡇
⠸⣿⣿⡿⡷⠀⠀⢐⢶⣶⣸⣿⣿⡿⢻⣿⣿⣿⣿⣿⣿⣿⣿⡇
⢇⢻⠿⠟⠀⠀⡈⢤⡃⣹⣿⣿⣿⣧⣼⣿⡿⡛⠋⡸⣿⣿⣿⠇
⢆⠑⠁⣰⣤⣁⣐⢟⣴⣿⣿⣿⣿⣿⣿⣿⠀⣠⣼⢧⣿⣿⡟⠀
⢿⣆⡀⠬⣭⣥⣴⣿⣿⣿⣿⣿⣿⣿⣿⣿⣶⣤⣶⣿⣿⡿⠃⢰
⣷⣿⣕⣂⠀⠈⠉⠛⠻⠿⣿⣿⣿⣿⣿⡿⠿⠛⠛⠉⠁⢀⣤⣾

So this is model that adresses the problem of learning only meaningful info while keeping gradients in check. This model introduces certain gates in order to tune what info to keep and what to forward to next layer.

                 [h_{t-1}, x_t]
                        │
          ┌─────────────┼──────────────┬──────────────┐
          │             │              │              │
          ▼             ▼              ▼              ▼
      sigmoid        sigmoid          tanh         sigmoid
          │             │              │              │
          ▼             ▼              ▼              ▼
         f_t           i_t         C̃_t              o_t
          │             │              │              │
          │             └──────┬───────┘              │
          │                    ▼                      │
          │                    ×                      │
          │                    │                      │
          ▼                    ▼                      │
c_{t-1} ──×──────────────────► +                      │
                               │                      │
                               ▼                      │
                              c_t                     │
                               │                      │
                               ▼                      │
                             tanh                     │
                               │                      │
                               └──────────┐           │
                                          ▼           ▼
                                         ┌────────────┐
                                         │     ×      │
                                         └────────┬───┘
                                                  │
                                                  ▼
                                                 h_t
                                      
LSTM Cell

× represents Hadamard Matrix Multiplication

LSTM cell is defined by these equations :

\begin{aligned} f_t &= \sigma\left(W_f [h_{t-1}, x_t] + b_f\right) \\ i_t &= \sigma\left(W_i [h_{t-1}, x_t] + b_i\right) \\ \tilde{C}_t &= \tanh\left(W_C [h_{t-1}, x_t] + b_C\right) \\ C_t &= f_t \odot C_{t-1} + i_t \odot \tilde{C}_t \\ o_t &= \sigma\left(W_o [h_{t-1}, x_t] + b_o\right) \\ h_t &= o_t \odot \tanh(C_t) \end{aligned}

You can decode the gate working from here but let me give a brief about each gate :

Forget Gate $f_t$ : Forget Gate is responsible for how much of the old cell internal state ( $c_{t - 1}$ ) we retain.
Input Gate $i_t$ : Input gate is responsible for how much of the new data we take into account. This gate works with Input Node $\tilde{C}_t$ .
Output Gate $o_t$ : Output gate is responsible for what to send to the next layer.

One thing you can notice is that activations are sigmoid and tanh so we can have a value range of $[0, 1]$ and $[-1, 1]$ respectively. So our Input gate can select a value closer to 0 to avoid taking new info and same logic applies for other gates. We get two outputs from LSTM layer which is a new Hidden State ( $h_t$ ) and a new value of Internal State ( $\tilde{C}_t$ ).

Here we solved the context forwarding problem but we increased the number of parameters by a lot.

GRU

Gated Reccurent Unit

⣿⣿⣿⣿⣿⣿⣿⣿⣿⡿⠿⣿⣿⡿⢿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⠛⠁⣼⣿⠏⣠⣤⢻⣿⣧⠄⠹⣿⣿⡇⢻⣿⣿⣿
⢸⣿⣿⠁⡆⠸⢿⡏⢸⣿⣿⠄⣿⡟⢸⡧⠘⠿⣿⠄⢻⣿⣿
⢸⣿⠃⣶⣶⡄⢢⡄⢸⣿⣿⠄⠟⢣⣠⣴⡆⠘⣿⠄⡀⢹⣿
⡈⠿⠄⢛⣋⣑⠊⣁⣼⣿⣿⣦⣶⣾⡿⢟⣃⣀⠁⣾⣧⢸⣿
⠳⠆⣴⣿⡿⠟⠳⢼⣿⣿⣿⣿⣿⣟⡱⠟⢻⣿⣶⡈⠿⢸⡏
⡇⢠⣿⣿⠁⠄⠄⢸⣿⣿⣿⣿⣿⣿⠃⠄⠄⢻⣿⣿⠄⣄⡄
⡇⢸⣿⣿⠄⠄⠄⢸⣿⣿⣿⣿⣿⣿⠄⠄⠄⢸⣿⣿⢠⣿⡇
⣿⣤⣿⣿⣦⣄⣤⣾⣿⣿⠛⣿⣿⣿⣧⣀⣰⣿⣿⣏⣼⣿⢣
⣿⣿⣿⣿⣿⡟⠛⠻⠿⠿⠿⠿⠿⠟⠛⢻⣿⣿⣿⣿⣿⡏⢸
⢻⣿⣿⣿⣿⡇⠄⠄⢀⣀⣤⣶⣶⣦⣄⢸⣿⣿⣿⣿⣿⢇⡞
⠈⠛⣿⣿⣿⣷⡀⢠⣾⣿⣿⣿⣿⣿⢇⣼⣿⣿⣿⡿⢁⣤⠅
⣷⣦⣤⡛⠻⢿⣷⣬⣛⠛⠛⠛⣛⣥⣾⣿⠟⠛⠁⣀⣤⣴⣾
⣿⣿⣿⣿⣿⠂⠄⠄⠈⠉⠉⠉⠉⠁⠄⢠⣴⣾⣿⣿⣿⣿⣿

The Goat

In order to reduce the number of params and keep the same customisation of contexts, GRU was introduced. So GRU has 3 gates and is described as follows :

                    [h_{t-1}, x_t]
                           │
                ┌──────────┴──────────┐
                │                     │
                ▼                     ▼
             sigmoid               sigmoid
                │                     │
                ▼                     ▼
               r_t                   z_t
                │                     │
                │                     │
h_{t-1} ────────┼──► ×                │
                │    ▲                │
                └────┘                │
                     │                │
                     ▼                │
              r_t ⊙ h_{t-1}           │
                     │                │
                     ├──────┐         │
                     │      │         │
x_t ─────────────────┘      │         │
                            ▼         │
                          tanh        │
                            │         │
                            ▼         │
                          h̃_t         │
                            │         │
                            │         │
                    ┌───────┴───────┐ │
                    │               │ │
                    ▼               ▼ ▼
                 (1-z_t)            z_t
                    │               │
                    ▼               ▼
                    ×               ×
                    ▲               ▲
                    │               │
                   h̃_t           h_{t-1}
                    │               │
                    └───────┬───────┘
                            ▼
                            +
                            │
                            ▼
                           h_t

GRU Layer

Layer description is given by simple equations :

\begin{aligned} r_t &= \sigma\left(W_r [h_{t-1}, x_t] + b_r\right) \\[4pt] z_t &= \sigma\left(W_z [h_{t-1}, x_t] + b_z\right) \\[4pt] \tilde{h}_t &= \tanh\left(W_h [r_t \odot h_{t-1}, x_t] + b_h\right) \\[4pt] h_t &= z_t \odot h_{t-1} + (1-z_t)\odot\tilde{h}_t \end{aligned}

Reset Gate $r_t$ : Reset gate is responsible for how much of the previous hidden state ( $h_{t-1}$ ) is used while creating the new candidate hidden state ( $\tilde{h}_t$ ).
Update Gate $z_t$ : Update gate is responsible for how much of the previous hidden state ( $h_{t-1}$ ) is retained in the new hidden state ( $h_t$ ), and how much is replaced by the candidate hidden state ( $\tilde{h}_t$ ) as seen in the equation above.
Candidate Hidden State $\tilde{h}_t$ : Candidate hidden state represents the new information computed from the current input ( $x_t$ ) and the reset-modified previous hidden state. It is then mixed with the old hidden state using the update gate.

Now we have reduced the params and also have a context information control.

Translation

So I implemented a translation model using GRUs and Encoder-Decoder Architecture. You can check out the code here in My Implementation

So I used 2 layers of GRU for Encoder and 2 Layers of GRU for decoder and used Embeddings for english and french language tokens. You can check out the data preparation in my notebook. So basically I am Teacher Forcing method on the decoder side in order to give the model context to the whole translation by shifting one token at a time. For eg :

Input to decoder     Target prediction

<BOS>                Je
Je                   suis
suis                 faim
faim                 <EOS>

<BOS> and <EOS> are Beginning and End of Sentence respectively. Teacher forcing is one of the methods which can be used to tell decoder on how to predict the tokens.

The Idea is to generate a context (= final hidden state) and hidden states from Encoder and pass the context + french embedding to the decoder as an input. Also the hidden states of the Encoder layers are passed as initial hidden state for Decoder Layers.

This Architecture results in a translation model which performs decently with BLEU score of 32.8 which means it is a good translator.

One caveat with BLEU score is I have implemented add-one smoothening so other translators should also be measured with add-one smoothening BLEU implementation

Example Translation :

Input : 
-------

The house is on the hill.
There is no one there.
I think we should go back.

Output :
--------

la maison est sur la colline.
il n'y a personne.
je pense que nous devrions y retourner.

Now we have a translator but the issue with the contextual model is that as soon as the sequences start to become large the model translation becomes very wrong and grammatically incorrect. Modern translators use Transformer based models (with attention mechanism) which can automatically choose from some sequence which element is appropriatte in current context because Decoder can look back at all hidden states which have been assigned weights $\alpha$ . We can also use attention mechanism with RNNs instead of LSTM/GRU layers.

Encoder:

x₁ ──► h₁
x₂ ──► h₂
x₃ ──► h₃
...
x_T ──► h_T

             Decoder at step t
                    │
                    ▼
          ┌─────────────────────┐
          │      Attention      │
          │                     │
          │ h₁ → weight α₁      │
          │ h₂ → weight α₂      │
          │ h₃ → weight α₃      │
          │ ...                 │
          │ h_T → weight α_T    │
          └──────────┬──────────┘
                     │
                     ▼
              context vector c_t
                     │
                     ▼
             generate next word

KEEP YOUR SHAPES IN CHECK