Skip to content

Everything you need to know about Transformers! πŸ€–

Notifications You must be signed in to change notification settings

SkalskiP/transformers

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

3 Commits
Β 
Β 

Repository files navigation

transformers course

make sense logo

πŸ‘‹ hello

I'm Peter, a software engineer embarking on a fresh journey into the world of transformers, and I invite you to join me! The course is a work in progressβ€”it’s free, open-source, and we’ll be building it together, step by step. We'll explore key concepts, tackle practical exercises, and dissect seminal papers, all while learning and growing together. Using YouTube videos for clarity and Jupyter notebooks for hands-on practice, we're set for our collaborative journey into the world of transformers. Let's dive in together! πŸš€

πŸš€ Course program

πŸ”‘ Key concepts

  • Encoder-decoder architecture
  • Self-attention
  • Multi-head attention
  • Positional encoding
  • Keys, queries, and values
  • Word embeddings
  • Dynamic padding
  • Tokenization

πŸ› οΈ Practical exercises

  • Implement self-attention from scratch
  • Implement multi-head attention from scratch
  • Build a simple transformer model for a sequence-to-sequence task
  • Fine-tune a pre-trained model like BERT or GPT-2 on a specific task
  • Use a pre-trained transformer like GPT-2 for text generation
  • Train ViT on custom dataset for image classification

πŸ—žοΈ Paper reviews

  • "Attention Is All You Need" (2017) [link]
  • "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (2018) [link]
  • "ViT: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale" (2020) [link]
  • "DETR: End-to-End Object Detection with Transformers" (2020) [link]
  • "CLIP: Learning Transferable Visual Models From Natural Language Supervision" (2021) [link]
  • "GPT-3: Language Models are Few-Shot Learners" (2020) [link]

🎬 Upcoming videos

  • Introduction to the course (coming soon)
  • Self-attention (coming soon)
  • Multi-head attention (coming soon)
  • Paper review: "Attention Is All You Need" (coming soon)

🦸 Contribution

I would love your help in making this repository even better! Whether you want to correct a typo, add some new content, or if you have any suggestions for improvement, feel free to open an issue.