Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FSDP2] Fixed 2D clip grad norm test #126497

Closed
wants to merge 5 commits into from
Closed

[FSDP2] Fixed 2D clip grad norm test #126497

wants to merge 5 commits into from

Conversation

awgu
Copy link
Contributor

@awgu awgu commented May 17, 2024 •
edited

Stack from ghstack (oldest at bottom):

This fixes #126484.

We change from transformer to MLP stack since transformer seems to introduce slight numeric differences when using TP. We include a sequence parallel layer norm module in the MLP stack to exercise (S(0), R) placement.

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @penguinwu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @yf225 @chauhang @d4l3k

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented May 17, 2024 •
edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/126497

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit e09cfec with merge base 4e6673e (image):

FLAKY - The following job failed but was likely due to flakiness present on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added oncall: distributed Add this issue/PR to distributed oncall triage queue release notes: distributed (fsdp) release notes category labels May 17, 2024
awgu added a commit that referenced this pull request May 17, 2024
@awgu
[FSDP2] Fixed 2D clip grad norm test
4a061ad 
ghstack-source-id: 7c452970d09805f883a6f292efce651bb81bb166
Pull Request resolved: #126497
@awgu
Update on "[FSDP2] Fixed 2D clip grad norm test"
af2619c 
cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k

[ghstack-poisoned]
awgu added a commit that referenced this pull request May 17, 2024
@awgu
[FSDP2] Fixed 2D clip grad norm test
0c133fb 
ghstack-source-id: 92373ca426a449391ca19b703c60dda0b6de2352
Pull Request resolved: #126497
@awgu awgu added release notes: distributed (fsdp2) release notes category and removed release notes: distributed (fsdp) release notes category labels May 17, 2024
@awgu awgu requested review from wanchaol, wz337 and weifengpy May 17, 2024 01:16
@awgu awgu marked this pull request as ready for review May 17, 2024 01:16
@awgu awgu requested a review from a team as a code owner May 17, 2024 01:16
@awgu
Update on "[FSDP2] Fixed 2D clip grad norm test"
102e22d 
This fixes #126484.

We change from transformer to MLP stack since transformer seems to introduce slight numeric differences when using TP. We include a sequence parallel layer norm module in the MLP stack to exercise `(S(0), R)` placement.

cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k

[ghstack-poisoned]
awgu added a commit that referenced this pull request May 17, 2024
@awgu
[FSDP2] Fixed 2D clip grad norm test
1663a38 
ghstack-source-id: b4e373a5ccc4772ef719beae550b9977d762c4dd
Pull Request resolved: #126497
@weifengpy
Copy link
Contributor

switching to fsdp2 release note is really some labor work

wz337
wz337 approved these changes May 17, 2024
View reviewed changes
Copy link
Contributor

@wz337 wz337 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for looking into it so quickly.

@awgu
Copy link
Contributor Author

awgu commented May 17, 2024

Failure are all inductor-related, not FSDP2-related.

@awgu
Copy link
Contributor Author

awgu commented May 17, 2024

@pytorchbot merge -i

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label May 17, 2024
@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged while ignoring the following 9 checks: inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor, 1, 1, linux.g5.4xlarge.nvidia.gpu, unstable), inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu, unstable), inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu, unstable), inductor / cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu, unstable), inductor / cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu, unstable), inductor / rocm6.1-py3.8-inductor / test (inductor, 1, 1, linux.rocm.gpu.2), inductor-periodic / cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu), inductor-periodic / cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu), inductor-periodic / cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

@pytorchmergebot
Copy link
Collaborator

@awgu your PR has been successfully reverted.

pytorchmergebot added a commit that referenced this pull request May 17, 2024
@pytorchmergebot
Revert "[FSDP2] Fixed 2D clip grad norm test (#126497)"
d782e43 
This reverts commit 3f28906.

Reverted #126497 on behalf of https://github.com/jeanschmidt due to reverting to check if might have introduced inductor cuda 12 issues ([comment](#126497 (comment)))
ZelboK pushed a commit to ZelboK/pytorch that referenced this pull request May 19, 2024
@awgu @ZelboK
[FSDP2] Fixed 2D clip grad norm test (pytorch#126497)
9a47caa 
This fixes pytorch#126484.

We change from transformer to MLP stack since transformer seems to introduce slight numeric differences when using TP. We include a sequence parallel layer norm module in the MLP stack to exercise `(S(0), R)` placement.

Pull Request resolved: pytorch#126497
Approved by: https://github.com/weifengpy, https://github.com/wz337
ZelboK pushed a commit to ZelboK/pytorch that referenced this pull request May 19, 2024
@pytorchmergebot @ZelboK
Revert "[FSDP2] Fixed 2D clip grad norm test (pytorch#126497)"
c81bf77 
This reverts commit 3f28906.

Reverted pytorch#126497 on behalf of https://github.com/jeanschmidt due to reverting to check if might have introduced inductor cuda 12 issues ([comment](pytorch#126497 (comment)))
@awgu
Copy link
Contributor Author

awgu commented May 20, 2024

@pytorchbot rebase -s

@pytorchmergebot
Copy link
Collaborator

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

@awgu
Update
8a1cce1 
[ghstack-poisoned]
@pytorchmergebot
Copy link
Collaborator

Successfully rebased gh/awgu/588/orig onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via ghstack checkout https://github.com/pytorch/pytorch/pull/126497)

pytorchmergebot pushed a commit that referenced this pull request May 20, 2024
@awgu
[FSDP2] Fixed 2D clip grad norm test
1468a91 
ghstack-source-id: d6fbfab126e41561805c4c8cd4057769b1381d54
Pull Request resolved: #126497
@awgu
Update on "[FSDP2] Fixed 2D clip grad norm test"
e09cfec 
This fixes #126484.

We change from transformer to MLP stack since transformer seems to introduce slight numeric differences when using TP. We include a sequence parallel layer norm module in the MLP stack to exercise `(S(0), R)` placement.

cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k

[ghstack-poisoned]
awgu added a commit that referenced this pull request May 21, 2024
@awgu
[FSDP2] Fixed 2D clip grad norm test
381b892 
ghstack-source-id: dbf18ac97ab7a5eddef93664404af300827c29e7
Pull Request resolved: #126497
awgu
awgu commented May 21, 2024
View reviewed changes
.ci/pytorch/test.sh
@@ -326,6 +326,7 @@ test_inductor_distributed() {
python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_frozen.py --verbose
python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_mixed_precision.py -k test_compute_dtype --verbose
python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_mixed_precision.py -k test_reduce_dtype --verbose
python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py -k test_clip_grad_norm_2d --verbose
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test name was incorrect before (fixed to include trailing _).

@awgu
Copy link
Contributor Author

awgu commented May 21, 2024

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

Merge failed

Reason: 1 mandatory check(s) failed. The first few are:

Dig deeper by viewing the failures on hud

Details for Dev Infra team Raised by workflow job

Failing merge rule: Core Maintainers

@awgu
Copy link
Contributor Author

awgu commented May 21, 2024

@pytorchbot rebase -s

@pytorchmergebot
Copy link
Collaborator

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

@pytorchmergebot
Copy link
Collaborator

Tried to rebase and push PR #126497, but it was already up to date. Try rebasing against main by issuing:
@pytorchbot rebase -b main

pytorchmergebot pushed a commit that referenced this pull request May 21, 2024
@awgu
[FSDP2] Fixed 2D clip grad norm test
4ee89c4 
ghstack-source-id: dbf18ac97ab7a5eddef93664404af300827c29e7
Pull Request resolved: #126497
@awgu
Copy link
Contributor Author

awgu commented May 21, 2024

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wz337 wz337 wz337 approved these changes

@weifengpy weifengpy weifengpy approved these changes

@wanchaol wanchaol Awaiting requested review from wanchaol

Assignees
No one assigned
Labels
ciflow/inductor ciflow/trunk Trigger trunk jobs on your pull request Merged oncall: distributed Add this issue/PR to distributed oncall triage queue release notes: distributed (fsdp2) release notes category Reverted
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@awgu @weifengpy @pytorchmergebot @jeanschmidt @wz337