Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SGDP default settings are "incorrect" (and the visualization is uncharacteristic for SGDP) #354

Open
torridgristle opened this issue Aug 16, 2021 · 0 comments

Comments

@torridgristle
Copy link

One of SGDP's main points is the use of momentum, currently the default settings shown paired with the visualization have it disabled. This has a pretty large impact on its speed, I did a little test seeing how long it'd take to go from randn noise to a goal image and it handled it in 3 steps reaching a squared sum difference of 7.0072e-10 with a learning rate of 0.1, momentum set to 0.9888544, and nesterov disabled. With 0 momentum, the poor thing takes 27 steps to reach 0.7700. Plus, the project page for AdamP / SGDP shows the same test as the visualizations used here, but they use momentum with various settings and it definitely reaches the goal. The github page for AdamP also gives default parameters of "SGDP(params, lr=0.1, weight_decay=1e-5, momentum=0.9, nesterov=True)", and this seems closer to the demonstrations on their project page, even though their own code defaults momentum to 0. Thus, I believe the values on their page showing an example of importing and using SGDP should be used for the default values on this page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant