pre-training_cook (Reading list)

Other

• Let's Think Dot by Dot: Hidden Computation in Transformer Language Models
• A Survey on Self-Evolution of Large Language Models
• Microsfot phi
• Predicting Emergent Abilities with Infinite Resolution Evaluation v

KAN

• KAN: Kolmogorov–Arnold Networks v

Uncategorized

• JetMoe

Mup

• Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
• Feature Learning in Infinite-Width Neural Networks

Scaling Law

• MiniCPM, MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies v
• Compression Represents Intelligence Linearly

RLHF

• Proximal Policy Optimization Algorithms
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Iterative Reasoning Preference Optimization

Data

• Advancing LLM Reasoning Generalists with Preference Trees: v sft, preference tuning

Reasoning: weaker-to-stronger, sel-tuning, self-learning

• Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking v
• STaR: Self-Taught Reasoner Bootstrapping Reasoning With Reasoning