Fix for Resuming Training gives CheckpointMismatchError

[pablo-sanchez-sony]

Link to the relevant Bug(s)

This Pull request is linked to #1300. It has been resolved by modifying the file src/pykeen/training/training_loop.py.

I've also updated minor things in the following two tracker files:

• https://github.com/pablo-sanchez-sony/pykeen/blob/19e1c97feeefb560ec0687fded136034f73a9105/src/pykeen/trackers/file.py
• https://github.com/pablo-sanchez-sony/pykeen/blob/19e1c97feeefb560ec0687fded136034f73a9105/src/pykeen/trackers/wandb.py

Description of the Change

The problem comes when using the scheduler object from PyTorch. We can observe in the constructor whenever last_epoch=-1 the initial_lr of the optimizer is updated.

https://github.com/pytorch/pytorch/blob/a5d841ef01e615e2a654fb12cf0cd08697d12ccf/torch/optim/lr_scheduler.py#L38

Basically, this makes str(self.optimizer).encode("utf-8") to be different, given that we have not yet reloaded the optimizer nor the scheduler.

I believe the issue can be solved by moving the checksum comparison to the end of the method _load_state

 if checkpoint["checksum"] != self.checksum: 
     raise CheckpointMismatchError( 
         f"The checkpoint file '{path}' that was provided already exists, but seems to be " 
         f"from a different training loop setup.", 
     ) 

Other changes

• I updated the _write method of the JSONResultTracker object such that it dumps the name of the object instead of the object itself, which is not parseable https://github.com/pablo-sanchez-sony/pykeen/blob/19e1c97feeefb560ec0687fded136034f73a9105/src/pykeen/trackers/file.py
• I updated https://github.com/pablo-sanchez-sony/pykeen/blob/19e1c97feeefb560ec0687fded136034f73a9105/src/pykeen/trackers/wandb.py such that the parameter allow_val_change allows for changing the parameters during a run.

Possible Drawbacks

Verification Process

I checked the introduced changes worked in my particular case.

Release Notes

• Fixed an issue in which loading a Checkpoint with an LR scheduler was giving an error because the learning rate was not properly loaded.
• Fixed an issue in which the JSONResultTracker was not able to dump python objects
• Fixed an issue in which allow_val_change of wandb Tracker was a dummy variable.

[cthoyt]

Please add tests that illustrate where this might be applicable