You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For most schedules, the total number of steps is specified through the transition_steps parameter, but in some cases (e.g., optax.cosine_decay_schedule, optax.warmup_cosine_decay_schedule but confusingly not optax.cosine_onecycle_schedule) it's called decay_steps instead.
The name sgdr_schedule is not descriptive of what the schedule actually does.
Most warm-up learning rates like linear_onecycle_schedule and cosine_onecycle_schedule specify the length of the warm-up phrase using parameter pct_start , but warmup_cosine_decay_schedule instead specifies it through a parameter warmup_steps
In the documentation:
5. In the API reference https://optax.readthedocs.io/en/latest/api/optimizer_schedules.html there's a section "Schedules with warm-up". I would consider optax.cosine_onecycle_schedule to have warm-up, yet it's not in this section. My recommendation would be to remove the section ""Schedules with warm-up" and put optax.warmup_cosine_decay_schedule in the Cosine decay schedule section and optax.warmup_exponential_decay_schedule in the exponential decay section
The text was updated successfully, but these errors were encountered:
end_value
, but for cosine_decay it's calledalpha
. : deprecate kwarg alpha in cosine_decay_schedule in favor of end_value #870transition_steps
parameter, but in some cases (e.g.,optax.cosine_decay_schedule
,optax.warmup_cosine_decay_schedule
but confusingly notoptax.cosine_onecycle_schedule
) it's calleddecay_steps
instead.sgdr_schedule
is not descriptive of what the schedule actually does.linear_onecycle_schedule
andcosine_onecycle_schedule
specify the length of the warm-up phrase using parameterpct_start
, butwarmup_cosine_decay_schedule
instead specifies it through a parameterwarmup_steps
In the documentation:
5. In the API reference https://optax.readthedocs.io/en/latest/api/optimizer_schedules.html there's a section "Schedules with warm-up". I would consider
optax.cosine_onecycle_schedule
to have warm-up, yet it's not in this section. My recommendation would be to remove the section ""Schedules with warm-up" and putoptax.warmup_cosine_decay_schedule
in the Cosine decay schedule section and optax.warmup_exponential_decay_schedule in the exponential decay sectionThe text was updated successfully, but these errors were encountered: