Hi Amit,

Sockeye's loss functions are implemented in loss.py. You could use an existing loss function (e.g., MSELoss as a template for creating your own. You could also trace instances of MSELoss through the code to see where and how loss functions are connected into the training logic.

If your metrics require decoding, you could add them to the checkpoint decoder's decode_and_evaluate method (after BLEU, chrF, etc.). These metrics will be reported for every checkpoint but Sockeye does not currently support optimizing them directly.

Best,
Michael

By optimizing in my case I meant early-stopping. I'll try to add to the decode and evaluate and see if I can then early stop on those

You should be able to add your metric as a choice for --optimized-metric to enable using it for early stopping. One approach would be to do this first (add the name of your metric), start a training run with a tiny model, and see where it hits an error since your metric isn't yet implemented. This can identify where the next code change would be needed to support your metric.