Update inference.py

[angelala00]

Add eos_token_id from the generation config file so that Llama3 can perform inference correctly.

Why are these changes needed?

The addition of eos_token_id from the generation config file to the stop_token_ids is crucial for ensuring correct inference termination in frameworks like Llama3. Typically, the eos_token_id used for model inference can be sourced from the tokenizer's config file and is usually identical to the one in the model's generation config. However, for certain models like Llama3, discrepancies between these two can prevent inference from stopping correctly. By explicitly specifying the eos_token_id from the generation config, our framework can handle inference more accurately and support a wider range of models.

Related issue number (if applicable)

N/A

Checks

• I've run format.sh to lint the changes in this PR.
• I've included any doc changes needed.
• I've made sure the relevant tests are passing (if applicable).

[angelala00]

Hi @infwinston Could you please review this PR when you have a moment?

[Puupuls]

This saved me :D, Could not get llama 3 to run.
Patching in site packages fixed it!