Is-Attention-all-I-need

Purpose ?

• For a long time, I have heard of this paper and always wondered to know it properly.
• This repo is one way for me for that redemption.

Method

• Read the paper, understand the maths.
• Correlate the knowledge with LSTMs and RNNs.
• Implement the vanilla architecture using pytorch.
• Train the model on the WMT2014 English to German Translation Task.
• Rinse and repeat on some more models that came after that.

Time Duration ?

• Start Date : 02/January/2022
• Will Complete by : 02/February/2022

Resources ?

1. Paper : https://arxiv.org/pdf/1706.03762.pdf
2. Data : https://huggingface.co/datasets/wmt14
3. How to write Pytorch Code : https://nn.labml.ai/transformers/mha.html

Video Resources ?

1. https://www.youtube.com/watch?v=iDulhoQ2pro