Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Could u add Nesterov momentum in SGD #190

Open
georgekasa opened this issue Apr 9, 2023 · 0 comments
Open

Could u add Nesterov momentum in SGD #190

georgekasa opened this issue Apr 9, 2023 · 0 comments

Comments

@georgekasa
Copy link

Hello,
as i saw you are using SGD with momentum (default 0.9) could u add a feature to add Nesterov momentum

line: 375-376:
optimizer = torch.optim.SGD(params, lr=args.lr, momentum=args.momentum,
weight_decay=args.weight_decay,
nesterov=args.nesterov)

as Karpathy told in CS231n:
Nesterov Momentum is a slightly different version of the momentum update that has recently been gaining popularity. It enjoys stronger theoretical converge guarantees for convex functions and in practice it also consistenly works slightly better than standard momentum.

thank you in advance

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant