resume_from_checkpoint 学习率的恢复

[Lyu6PosHao]

在1000 steps时，我保存了checkpoint-1000的检查点，使用resume_from_checkpoint恢复训练。
因为有lr_scheduler的存在，学习率应该从1000 steps时的值 继续变化。
但实际上，学习率又从头（也就是step=0的时候）开始变化了。
有人遇到这个问题吗？

[cqwyj2000]

能把你脚本贴上来看一下吗

[Lyu6PosHao]

运行脚本前请仔细阅读wiki(https://github.com/ymcui/Chinese-LLaMA-Alpaca-2/wiki/pt_scripts_zh)

Read the wiki(https://github.com/ymcui/Chinese-LLaMA-Alpaca-2/wiki/pt_scripts_zh) carefully before running the script

lr=2e-4
lora_rank=8
lora_alpha=32
lora_trainable="q_proj,v_proj,k_proj,o_proj,gate_proj,down_proj,up_proj"
modules_to_save="embed_tokens,lm_head"
lora_dropout=0.05

pretrained_model=/remote-home/lvliuzh/llama/models_hf/7B
chinese_tokenizer_path=/remote-home/lvliuzh/temp/jiandan_merged_tokenizer_hf
dataset_dir=/remote-home/share/uniref50/dataset/part_to_train/
data_cache=/remote-home/share/uniref50/cache_CN/
per_device_train_batch_size=12
gradient_accumulation_steps=8
block_size=512
output_dir=/remote-home/share/uniref50/ckpts_CN/

ckpt=/remote-home/share/uniref50/ckpts_CN/checkpoint-1000/

export NCC_P2P_LEVEL=NVL
deepspeed_config_file=ds_zero2_no_offload.json
export OMP_NUM_THREADS=64
torchrun --nproc_per_node 8 run_clm_pt_with_peft.py
--deepspeed ${deepspeed_config_file}
--model_name_or_path ${pretrained_model}
--tokenizer_name_or_path ${chinese_tokenizer_path}
--dataset_dir ${dataset_dir}
--data_cache_dir ${data_cache}
--validation_split_percentage 0.001
--per_device_train_batch_size ${per_device_train_batch_size}
--do_train
--seed 42
--num_train_epochs 1
--lr_scheduler_type linear
--learning_rate ${lr}
--warmup_ratio 0.05
--weight_decay 0.01
--logging_strategy steps
--logging_steps 2
--save_strategy steps
--save_total_limit 3
--save_steps 200
--gradient_accumulation_steps ${gradient_accumulation_steps}
--preprocessing_num_workers 16
--block_size ${block_size}
--output_dir ${output_dir}
--ddp_timeout 30000
--logging_first_step True
--lora_rank ${lora_rank}
--lora_alpha ${lora_alpha}
--trainable ${lora_trainable}
--lora_dropout ${lora_dropout}
--torch_dtype float16
--load_in_kbits 16
--bf16
--gradient_checkpointing
--ddp_find_unused_parameters False
--flash_attn
--max_eval_samples 128
--modules_to_save ${modules_to_save}
--resume_from_checkpoint ${ckpt}
#--overwrite_output_dir
@cqwyj2000

[Lyu6PosHao]

我发现在deepspeed的config文件中加入如下代码，能解决这个问题，要修改"last_bach_iteration"。但是我看很多代码仓库，都没有设置这个选项，不知道为啥。我如果不这么设置，那学习率就接续不上。
"scheduler": {
"type": "WarmupDecayLR",
"params": {
"last_batch_iteration": 1000,
"total_num_steps": "auto",
"warmup_min_lr": "auto",
"warmup_max_lr": "auto",
"warmup_num_steps": "auto"
}
}