Add Prodigy, SophiaG optimizers

[Kimiko-AI]

Description

https://arxiv.org/pdf/2306.06101.pdf
https://arxiv.org/abs/2305.14342

Motivation and Context

How has this been tested?

No

Screenshots (if appropriate)

Types of changes

Social Handles (Optional)

[winglian]

thie update_fn function is redefined below on line 106.

[winglian]

bs you've set this as an int, but below you call bs.is_cuda()

[Kimiko-AI]

https://wandb.ai/nruaif/Test_optim?workspace=user-nruaif
Some tests, with prodigy we don't need to change lr, just set it to 1

[winglian]

I took a first pass at fixing the pylint errors, but mypy has still found a few issues

src/axolotl/custom_optim/sophia.py:233: error: "float" has no attribute "neg"  [attr-defined]
src/axolotl/custom_optim/lion.py:161: error: "None" not callable  [misc]
src/axolotl/custom_optim/lion.py:165: error: Invalid type 'Any' for *expr (iterable expected)  [misc]
src/axolotl/custom_optim/lion.py:165: error: Need more than 4 values to unpack (6 expected)  [misc]
src/axolotl/custom_optim/lion.py:175: error: Cannot determine type of "state"  [has-type]
src/axolotl/custom_optim/lion.py:176: error: Cannot determine type of "state"  [has-type]
src/axolotl/custom_optim/lion.py:178: error: Cannot determine type of "state"  [has-type]
src/axolotl/custom_optim/lion.py:180: error: Cannot determine type of "grad"  [has-type]
src/axolotl/custom_optim/lion.py:180: error: Cannot determine type of "lr"  [has-type]
src/axolotl/custom_optim/lion.py:180: error: Cannot determine type of "wd"  [has-type]
src/axolotl/custom_optim/lion.py:180: error: Cannot determine type of "beta1"  [has-type]
src/axolotl/custom_optim/lion.py:180: error: Cannot determine type of "beta2"  [has-type]

[hammoudhasan]

@Kimiko-AI I really think this pull request is worth finishing! Very useful - would love to see how Prodigy would perform on LLM training after I used it before on Stable Diffusion fine-tuning and it worked pretty well!