Pinned

1. fft-conv-pytorch fft-conv-pytorch Public

   Implementation of 1D, 2D, and 3D FFT convolutions in PyTorch. Much faster than direct convolutions for large kernel sizes.

   Python 440 53

2. yet-another-retnet yet-another-retnet Public

   A simple but robust PyTorch implementation of RetNet from "Retentive Network: A Successor to Transformer for Large Language Models" (https://arxiv.org/pdf/2307.08621.pdf)

   Python 94 15

3. transformer-from-scratch transformer-from-scratch Public

   Code implementation from my blog post: https://fkodom.substack.com/p/transformers-from-scratch-in-pytorch

   Python 89 19

4. clip-text-decoder clip-text-decoder Public

   Generate text captions for images from their embeddings.

   Python 87 3

5. grouped-query-attention-pytorch grouped-query-attention-pytorch Public

   (Unofficial) PyTorch implementation of grouped-query attention (GQA) from "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints" (https://arxiv.org/pdf/2305.13245.pdf)

   Python 78 4

6. soft-mixture-of-experts soft-mixture-of-experts Public

   PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)

   Python 57 3