Inconsistent number of forget samples when evaluating the retain model (forget10 task)

[LetheSec]

Hi, thanks for the great work!

For the forge10 task, the forget set should be 400 samples , as shown in forget10.json . However, there are only 300 samples are used for the evaluation, as shown in eval_log_forget.json.

tofu/data/ft_epoch5_lr1e-05_llama2-7b_retain90_wd0.01/eval_results/ds_size300/eval_log_forget.json

Line 2404 in 3681105

"299": [

Can you explain why you do this? Thank you so much.

[zhilif]

In the config file https://github.com/locuslab/tofu/blob/main/config/eval_everything.yaml#L30, we specify the maximum number of samples for evaluation to be 300, for evaluation speed.

[LetheSec]

Thank you for your reply.