Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gradient clipping by global norm #315

Merged
merged 6 commits into from
May 19, 2024
Merged

Conversation

ngc92
Copy link
Contributor

@ngc92 ngc92 commented May 1, 2024

one new kernel that calculates the overall norm of the gradient, and updates to the adam kernel.
Still TODO:

  • clip value is hardcoded at function call site
  • error handling for broken gradients would fit nicely into this change, but it currently only very rudimentary
  • there is a debug print statement there. This debug print is in the kernel because we never move gradient norms to the host, but really, it should be removed once the code is ready for merge.
  • python reference needs to be updated

@ngc92 ngc92 changed the title first draft for gradient clipping by global norm gradient clipping by global norm May 2, 2024
@ngc92 ngc92 force-pushed the gradient-clipping branch 2 times, most recently from 9cf1ba7 to c80ca17 Compare May 7, 2024 22:13
@karpathy karpathy merged commit da9f4e2 into karpathy:master May 19, 2024
8 checks passed
@ngc92 ngc92 deleted the gradient-clipping branch May 25, 2024 21:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants