LLaMA-Paper-List

Collection of papers using LLaMA as backbone model.

Contributors

Table of Contents

• About LLaMA and related theory
• LLaMA with parameter efficiency
• Fine-tune LLaMA on downstream tasks
• LLaMA with retrieval
• LLaMA using reinforcement learning
• Quantitative analysis of LLaMA
• Prompting LLaMA

Papers

About LLaMA and related theory

• LLaMA: Open and Efficient Foundation Language Models. arxiv 2023. paper. code
  Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample
• Llama 2: Open Foundation and Fine-Tuned Chat Models. Meta AI 2023. paper. code
  Hugo Touvron, Louis Martin, Kevin Stone et al.
• Training Compute-Optimal Large Language Models. NeurIPS 2022. paper.
  Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, Laurent Sifre
• Root Mean Square Layer Normalization. NeurIPS 2019. paper. code
  Biao Zhang, Rico Sennrich
• GLU Variants Improve Transformer. arxiv 2020. paper. code
  Noam Shazeer
• RoFormer: Enhanced Transformer with Rotary Position Embedding. arxiv 2021. paper. code
  Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu
• Decoupled Weight Decay Regularization. ICLR 2019. paper. code
  Ilya Loshchilov, Frank Hutter
• Self-attention Does Not Need $O(n^2)$ Memory. arxiv 2021. paper. code
  Markus N. Rabe and Charles Staats
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. arxiv 2022. paper. code
  Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré
• Reducing Activation Recomputation in Large Transformer Models. arxiv 2022. paper.
  Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, Bryan Catanzaro

LLaMA with parameter efficiency

• LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention. arxiv 2023. paper. code
  Zhang, Renrui and Han, Jiaming and Zhou, Aojun and Hu, Xiangfei and Yan, Shilin and Lu, Pan and Li, Hongsheng and Gao, Peng and Qiao, Yu
• LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model. arxiv 2023. paper. code
  Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, Yu Qiao
• LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models. arxiv 2023. paper.
  Zhiqiang Hu, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, Roy Ka-Wei Lee, Lidong Bing, Xing Xu, Soujanya Poria
• A Simple and Effective Pruning Approach for Large Language Models. arxiv 2023. paper. code
  Mingjie Sun, Zhuang Liu, Anna Bair, J. Zico Kolter
• LLM-Pruner: On the Structural Pruning of Large Language Models. arxiv 2023. paper. code
  Xinyin Ma, Gongfan Fang, Xinchao Wang

Fine-tune LLaMA on downstream tasks

• ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge.. arxiv 2023. paper.
  Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, You Zhang
• Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca. arxiv 2023. paper. code
  Yiming Cui, Ziqing Yang, Xin Yao
• PMC-LLaMA: Further Finetuning LLaMA on Medical Papers. arxiv 2023. paper.
  Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, Weidi Xie
• Dr. LLaMA: Improving Small Language Models on PubMedQA via Generative Data Augmentation. arxiv 2023. paper.
  Zhen Guo, Peiqi Wang, Yanwei Wang, Shangdi Yu
• Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks. arxiv 2023. paper.
  Tiedong Liu, Bryan Kian Hsiang Low
• WizardLM: Empowering Large Language Models to Follow Complex Instructions. arxiv 2023. paper. code
  Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Daxin Jiang
• Enhancing Chat Language Models by Scaling High-quality Instructional Conversations. arxiv 2023. paper. code
  Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, BoWen Zhou
• LongForm: Optimizing Instruction Tuning for Long Text Generation with Corpus Extraction. arxiv 2023. paper. code
  Abdullatif Köksal, Timo Schick, Anna Korhonen, Hinrich Schütze
• In-Context Learning User Simulators for Task-Oriented Dialog Systems. arxiv 2023. paper. code
  Silvia Terragni, Modestas Filipavicius, Nghia Khau, Bruna Guedes, André Manso, Roland Mathis
• NetGPT: A Native-AI Network Architecture Beyond Provisioning Personalized Generative Services. arxiv 2023. paper. code
  Yuxuan Chen, Rongpeng Li, Zhifeng Zhao, Chenghui Peng, Jianjun Wu, Ekram Hossain, Honggang Zhang
• On decoder-only architecture for speech-to-text and large language model integration. arxiv 2023. paper. code
  Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, Yu Wu

LLaMA with retrieval

• Polyglot or Not? Measuring Multilingual Encyclopedic Knowledge Retrieval from Foundation Language Models. arxiv 2023. paper. code
  Tim Schott, Daniel Furman, Shreshta Bhat
• ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models paper. code
  Binfeng Xu, Zhiyuan Peng, Bowen Lei, Subhabrata Mukherjee, Yuchen Liu, Dongkuan Xu
• Landmark Attention: Random-Access Infinite Context Length for Transformers. arxiv 2023. paper. code
  Amirkeivan Mohtashami, Martin Jaggi

LLaMA using reinforcement learning

• LIMA: Less Is More for Alignment. arxiv 2023. paper. code
  Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, Omer Levy
• RRHF: Rank Responses to Align Language Models with Human Feedback without tears. paper. code
  Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, Fei Huang

Quantitative analysis of LLaMA

• SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression. arxiv 2023. paper. code
  Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, Dan Alistarh
• SqueezeLLM: Dense-and-Sparse Quantization. arxiv 2023. paper. code
  Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, Kurt Keutzer

Prompting LLaMA

• Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognition.. arxiv 2023. paper.
  Yuang Li, Yu Wu, Jinyu Li, Shujie Liu

How to contribute

Contributions are welcome! Please refer to CONTRIBUTING.md for contribution guidelines.