In this assignment I explore different strategies used in building and training a Language Translator. I use Seq2Seq learning to convert sequences from English to Hebrew. I also include techniques such as bi-directional learning and Attention mechanism which serve as the building blocks of advanced transformer-based NLP models such as GPT, Llama etc. .
I initially wanted to build a model to translate English to Tamil which happens to be my Native Language so that I could work with the translations easily. But due to less number of available sentence pairs (207) on the https://www.manythings.org/anki/ website for this language (the website used in my class) , I picked Hebrew.
Though Hebrew has been a language of fascination for me for a while now, there are a few other important reasons I picked Hebrew for learning machine translation.
-
Hebrew is written from right to left, it will be interesting to see if Bi-LSTM and Attention produces better results.
-
English and Hebrew are from completely different language-families and roots. (https://webspace.ship.edu/cgboer/languagefamilies.html) They are from completely different regions and time-periods and have isolated places of origins.
- English -> Region: West Germany, Language family: Indo-European, Root: Germanic
- Hebrew -> Region: Israel, Language family: Afro-Asiatic, Root: Semitic
-
They share almost a near-zero lexical similarity (https://en.wikipedia.org/wiki/Lexical_similarity) and less genetic relationship and linguistic interference. (https://en.wikipedia.org/wiki/Genetic_relationship_(linguistics))
-
English-Hebrew also has a good number of sentence pairs available on the https://www.manythings.org/anki/ website(127856)
Image Source: https://alternativetransport.wordpress.com/2015/05/05/34/
- Used Hebrew Tokenizer (github.com/YontiLevin/Hebrew-Tokenizer) to parse and clean Hebrew text (Canonical normalization used for English does not work)
- Converted text to sequences (reversed sequence list for Hebrew)
- Models : Seq2Seq, Bi-directional Seq2Seq (Bi-LSTM), Seq2Seq with Attention layer
- Tuning: Latent dimensions, training epochs, dropout, activation functions
- Sampling: Greedy and Multinomial (with different temperatures)
Link to Notebook