- Source codes for arxiv preprint.
- Teaching Specific Scientific Knowledge into Large Language Models through Additional Training
- Clone this repo.
git clone https://github.com/KanHatakeyama/Additional-training-Llama2.git
- Download dataset from HuggingFace
git lfs clone https://huggingface.co/datasets/kanhatakeyama/nature-family-CC-papers mv nature-family-CC-papers/database/ . mv nature-family-CC-papers/smallDB/ .
- Create env (conda)
conda env create -f environment.yml
- Demo code and model for additional training with scientific texts
- Fictional datasets
- Result analysis
- Training
- Full-parameter
- LoRA
- 7b model
- 7,13,70b models
- Selected adapter layers
- Full adapter layers
- Scientific papers
- Kan Hatakeyama
- Tokyo Tech., Japan