This repository is dedicated to exploring and implementing techniques in Natural Language Processing (NLP), starting with our inaugural notebook, "Transformers from Scratch." NLP is a crucial subfield of artificial intelligence that focuses on the interaction between computers and humans through natural language. The goal is to enable computers to understand, interpret, and manipulate human language, facilitating seamless human-computer interactions. This repository aims to cover a wide range of NLP topics, from foundational algorithms to advanced models like transformers, Named Entity Recognition (NER), and techniques for fine-tuning models for specific applications.
Natural Language Processing combines computational linguistics—rule-based modeling of human language—with statistical, machine learning, and deep learning models. These approaches enable computers to process and understand human (natural) languages, making it possible to execute tasks like translation, sentiment analysis, and topic extraction. NLP technologies are behind the scenes of many applications we use daily, such as virtual assistants, chatbots, and language translation services.
-
Decoding Algorithms in NLP: A Jupyter notebook that delves into the decoding strategies used in natural language processing. It covers Greedy, Beam Search, Pure Sampling, Top-K Sampling, and Top-P (Nucleus) Sampling methods, providing a mix of theoretical background, code implementations, and visual examples to demonstrate each decoding technique's impact on text generation.
-
Understanding Positional Encoding: This notebook provides an in-depth look at positional encoding mechanisms and their significance in language models, particularly in Transformers. It delves into Sinusodial Positional Encodings, Rotary Positional Embeddings (RoPE), ALiBi (Attention with Linear Biases) methods.
-
Embeddings: This section explores the transformative world of embeddings in natural language processing, detailing both word-based and context-based embedding models. Through practical examples and code snippets, it elucidates how embeddings capture semantic and syntactic nuances of language, significantly enhancing the machine's understanding of text.
-
Tokenisation: A Jupyter notebook that explores the fundamentals of tokenization in NLP, addressing its critical role in preprocessing textual data, overcoming multilingual challenges, various tokenization techniques, and practical applications.
-
Transformers from Scratch: A detailed Jupyter notebook that introduces the concept, architecture, and implementation of transformer models from the ground up. This notebook serves as a comprehensive guide for anyone looking to understand the workings of one of the most influential models in modern NLP.
-
Neural Machine Translation with LSTMs: This Jupyter notebook introduces the principles and practical implementation of Neural Machine Translation using LSTM networks. It details the design and operation of seq2seq models with LSTM cells, providing a step-by-step guide to building, training, and evaluating an NMT system capable of translating between English and French.
- Fine Tuning: A guide on how to fine-tune pre-trained models on domain-specific tasks for improved performance.
- And more topics that delve deeper into the vast field of NLP.
To dive into these notebooks:
- Clone the repository to your local machine.
- Make sure you have Jupyter Notebook or JupyterLab installed, or alternatively, use Google Colab to access the notebooks directly from the web.
- Navigate to the repository directory and launch the desired notebook using Jupyter Notebook or JupyterLab.
- Follow the instructions within each notebook to explore the implementation and application of various NLP techniques.
This repository will cover a broad spectrum of NLP topics and techniques, including but not limited to:
- Transformers: Understanding the architecture and mechanics behind transformers, including self-attention mechanisms and positional encoding.
- Named Entity Recognition (NER): Techniques and models for extracting entities from text.
- Fine-Tuning: Strategies for adapting pre-trained models to new tasks or datasets.
- Various NLP tasks such as text classification, sentiment analysis, language modeling, etc.
Our NLP Notebook Repository is designed to be a growing resource for those interested in natural language processing, whether you're a beginner or looking to expand your knowledge. Through detailed explorations and hands-on demonstrations, we aim to provide a practical understanding of NLP and its applications.
- Natural Language Processing with Python
- The Illustrated Transformer
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Contributions are welcome! If you're interested in adding to this repository, please read the CONTRIBUTING.md file for guidelines on how to contribute.
This project is licensed under the MIT License - see the LICENSE file for details.