You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Support Maximal Update Parametrization as it could help get optimal HPs from small proxy models and apply them on large LLMs
Is there any plan for adding this feature?
More details, please refer to : https://github.com/microsoft/mup
Description
It will acclerate the LLMs development as optimal HPs could be found by small proxy models and apply them on large LLMS. And this will save a lot of training cost. So there are quite a lot benefits for the LLM community.
Alternatives (optional)
No response
Additional context (optional)
No response
The text was updated successfully, but these errors were encountered:
Motivation
Support Maximal Update Parametrization as it could help get optimal HPs from small proxy models and apply them on large LLMs
Is there any plan for adding this feature?
More details, please refer to : https://github.com/microsoft/mup
Description
It will acclerate the LLMs development as optimal HPs could be found by small proxy models and apply them on large LLMS. And this will save a lot of training cost. So there are quite a lot benefits for the LLM community.
Alternatives (optional)
No response
Additional context (optional)
No response
The text was updated successfully, but these errors were encountered: