When eval the model on amazon datasets, the result fluctuate.

[AegeanYan]

Hi @rktamplayo , thank you for your great work!
I followed the README.md to reproduce the experimental results on Amazon dataset and use the check point you released to evaluate the test.json. But I didn't get the same result as you claim in the paper and the result I got is fluctuating, which is I evaluate 5 times or more but every time I got diffrent result. I wonder that whether there is some other settings I missed.

[AegeanYan]

I've solved this question, the 234 line sum_tokens[token_ids] += tokens[tindex] have potential randomness. You should change the writing style or set torch.use_deterministic_algorithms(True) to avoid this eval fluctuation.