Text generation using Transformer

Natural Language Processing (NLP) leverages algorithms and models to understand, interpret, and generate human language. At the heart of modern NLP techniques is the Transformer model, a groundbreaking architecture that excels in tasks such as translation, summarization, and text generation. Texts in NLP are represented through embeddings, which convert words into dense vectors capturing semantic meaning. Transformers generate text by processing these embeddings through layers of attention mechanisms, allowing the model to weigh the importance of different words in a sequence dynamically. This process enables the generation of coherent and contextually relevant text. In this material, I delve into the intricacies of NLP and the Transformer model, explaining their inner workings and showcasing a PyTorch implementation from scratch to provide a hands-on understanding of these powerful technologies.

Contents

• Text representation
• Vectorization and Embeddings
• Reasoning with Word Vector
• Why Transformer over RNN
• Positional Encoding
• Multi-headed self-attention layer/ Encoder
• Decoder stack
• Text Generation

A Powerpoint link

https://docs.google.com/presentation/d/1SMIPGRmEDviatPP2Ed3YJ6HzI-OxlFLTFGcLW6wB24Y/edit?usp=sharing

References

1. https://arxiv.org/abs/1706.03762
2. https://stanford.edu/~shervine/teaching/cs-230/cheatsheet-recurrent-neural-networks
3. https://kazemnejad.com/blog/transformer_architecture_positional_encoding/
4. http://jalammar.github.io/illustrated-transformer/
5. https://medium.com/analytics-vidhya/encoders-decoders-sequence-to-sequence-architecture-5644efbb3392
6. https://colah.github.io/posts/2014-07-NLP-RNNs-Representations/
7. https://wingedsheep.com/building-a-language-model/