🎉Awesome-SLM🎉

🌱 How to Contribute

We are welcome contributions from researchers. For detailed guidelines on how to contribute, please see our CONTRIBUTING.md file.

📜 Contents

• 🎉Awesome SLM🎉
  • 🌱 How to Contribute
  • 📜 Contents
  • 👋 Introduction
  • 🔥 Base Model
  • 💪 Pretrain datasets
  • 💡 SFT datasets
  • 🔧 synthetic datasets
  • 📦 preference dataset
  • 🌈 benchmark

👋 Introduction

🔥 Base Model

• OPT-series [paper] [code] [model]

  • release time: 2022/06
  • organzation: meta
  • model size: 125M, 350M, 1.3B, 2.7B, 6.7B, 13B, 30B, 66B, 175B

• Pythia [paper] [code] [model]

  • release time: 2023/06
  • organzation: meta
  • model size: 70M, 160M, 410M, 1.0B, 1.4B, 2.8B, 6.9B, 12B

• phi-1 [paper] [code] [model]

  • release time: 2023/06
  • organzation: mircosoft
  • model size: 1.42B

• phi-1_5 [paper] [model]

  • release time: 2023/09
  • organzation: mircosoft
  • model size: 1.42B

• phi-2 [paper] [model]

  • release time: 2023/12
  • organzation: mircosoft
  • model size: 2.78B

• phi-3-series [paper] [model]

  • release time: 2024/04
  • organzation: mircosoft
  • model series: Phi-3-mini-4k-instruct, Phi-3-mini-128k-instruct
  • model size: 3.82B

• Tinyllama [paper] [model]

  • release time: 2024/01
  • organzation: /
  • model size: 1.1B

• MiniCPM-series [paper] [code] [model]

  • release time: 2024/02
  • organzation: openbmb
  • model series: MiniCPM-1B-sft-bf16, MiniCPM-2B-sft-bf16, MiniCPM-2B-sft-fp32, MiniCPM-2B-128k, MiniCPM-MoE-8x2B
  • model size: 1.2B, 2.4B, 8X2.4B (excluding embeddings)

• H2O-Danube-1.8B [paper] [code] [model]

  • release time: 2024/04
  • organzation: h2oai
  • model series: h2o-danube2-1.8b-base, h2o-danube2-1.8b-sft, h2o-danube2-1.8b-chat
  • model size: 1.8B

• csg-wukong-series[model]

  • release time: 2024/04
  • organzation: opencsg
  • model series: csg-wukong-1B, csg-wukong-1B-VL, csg-wukong-1B-chat
  • model size: 1B

• CT-LLM-Base[paper] [code] [model]

  • release time: 2024/04
  • organzation: opencsg
  • model series: csg-wukong-1B, csg-wukong-1B-VL, csg-wukong-1B-chat
  • model size: 1B

💪 Pretrain Datasets

• SlimPajama-627B [paper] [code] [dataset]

  • release time: 2023/06
  • dataset size: 895 GB
  • token size: 627B
  • language: Primarily English, with some non-English files in Wikipedia

• dolma [paper] [code] [dataset]

  • release time: 2024/04
  • dataset size: 4.5TB
  • token size: 1.7T
  • language: Primarily English, with some non-English files in Wikipedia

• RedPajama-Data-1T [paper] [code] [dataset]

  • release time: 2023/04
  • token size: 627B

• C4 [paper] [code] [dataset]

  • release time: 2022/01
  • dataset size: en: 305GB, en.noclean: 2.3TB, en.noblocklist: 380GB, realnewslike: 15GB, multilingual (mC4): 9.7TB (108 subsets, one per language)

💡 SFT Datasets

• ultrachat [code] [dataset]

  • release time: 2023/04
  • dataset size: 2.5GB
  • language: en

• ultrachat_200k [code] [dataset]

  • release time: 2023/10
  • dataset size: 1.6GB
  • language: en

🔧 synthetic datasets

• cosmopedia [code] [dataset]
  • release time: 2024/02
  • dataset size: 92.2GB
  • language: en

📦 preference dataset

• UltraFeedback [code] [dataset]
  • release time: 2023/09
  • dataset size: 0.94GB
  • language: en

🌈 benchmark