Hi @mgerstgrasser, thanks for reporting.

Not sure I got the issue, did you mean that once enabled, wandb will upload the model in all subsequent script runs even if --report_to wandb is not specified?

Hi @mgerstgrasser, thanks for reporting.

Not sure I got the issue, did you mean that once enabled, wandb will upload the model in all subsequent script runs even if --report_to wandb is not specified?

No, I mean that right now in 4.41.0, enabling --report_to wandb will upload the entire initial model checkpoint to wandb right at the start of training, irrespective of whether you've enabled WANDB_LOG_MODEL or not. I.e. for any training run in 4.41.0 that has wandb enabled, the callback will upload gigabytes of data to wandb immediately.

+1 on this observation - I recently upgraded my transformers version and noticed this issue. Can we turning off the uploading of model weights by default, and require an explicit parameter to enable it? This will introduce a lot of bandwidth consumption unknowingly to the end user, and was quite an unpleasant surprise that took me a day to figure out unfortunately :\

@mgerstgrasser Would you like to add a flag in your PR #30897 to control this behaviour? As we haven't heard from @parambharat, this issue is not being flagged by others (thanks @wongjingping!), and it seems like something we generally might not want to do I'd say lets add and we can always change the future default behaviour if necessary.

@mgerstgrasser Would you like to add a flag in your PR #30897 to control this behaviour? As we haven't heard from @parambharat, this issue is not being flagged by others (thanks @wongjingping!), and it seems like something we generally might not want to do I'd say lets add and we can always change the future default behaviour if necessary.

Done!

Update: we're going to be reverting #30135 in a patch release, that'll be released soon.

The callback changes will stay on main, so:

• Stable releases will respect previous wandb behaviour
• Development branch will have the current behaviour + @mgerstgrasser's PR on top once merged

This will give us time to review/test and make sure this integration is working well for all. Thanks for iterating on this so quickly @mgerstgrasser!