NVLSTree: Do not reduce chunkSize to 64KiB on large number of nodes #1112
+0
−1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This heuristic was introduced by commit 8c6c595 ("2.19.3-1") with no explanation in commit message or comment in code other than "H800/H100 fixes and tuning". In practice, we have noticed this line to cause a significant performance reduction when using NVLSTree algorithm when running nccl-tests all-reduce on a cluster of 8 AWS P5 nodes. nccl-tests 4GB all-reduce output before this change: out-of-place in-place size count type redop root time algbw busbw #wrong time algbw busbw #wrong (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) 4294967296 1073741824 float sum -1 81146 52.93 104.20 N/A 80945 53.06 104.46 N/A output after this change: out-of-place in-place size count type redop root time algbw busbw #wrong time algbw busbw #wrong (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) 4294967296 1073741824 float sum -1 30222 142.11 279.78 N/A 30309 141.70 278.98 N/A output on commit f9c3dc2 ("2.19.1-1") prior to introducing this line: out-of-place in-place size count type redop root time algbw busbw #wrong time algbw busbw #wrong (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) 4294967296 1073741824 float sum -1 30196 142.24 280.03 N/A 30131 142.54 280.63 N/A Fixes: 8c6c595 ("2.19.3-1") Signed-off-by: Liran Alon <liran@amazon.com>
liralon
force-pushed
the
nvls_tree_fix
branch
from
133dafa
to
505aa4b
Compare
wangjiezhe
added a commit
to wangjiezhe/gentoo-local
that referenced
this pull request
Feb 4, 2024
Need by pytorch-2.2.0 https://github.com/pytorch/pytorch/releases/tag/v2.2.0 pytorch/pytorch#117748 NVIDIA/nccl#1112 Signed-off-by: wangjiezhe <wangjiezhe@gmail.com>
rajachan
added a commit
to rajachan/aws-ofi-nccl
that referenced
this pull request
Feb 15, 2024
NCCL v2.19.3 reduced the chunk size used when running NVLS Tree algorithm on greater than 4 nodes to 64KiB. This drastically impacted performance on AWS (Ref: NVIDIA/nccl#1112 for some data). NCCL v2.20.3 has made this a tunable. Based on empirical testing, a max chunk size of 512KiB recovers from the regression and was also observed to be the default in v2.19.3. Setting this unconditionally without relying on ncclGetVersion symbol being available, since the parameter did not exist in versions prior to v2.20. Signed-off-by: Raghu Raja <raghunch@amazon.com>
rajachan
added a commit
to aws/aws-ofi-nccl
that referenced
this pull request
Feb 15, 2024
NCCL v2.19.3 reduced the chunk size used when running NVLS Tree algorithm on greater than 4 nodes to 64KiB. This drastically impacted performance on AWS (Ref: NVIDIA/nccl#1112 for some data). NCCL v2.20.3 has made this a tunable. Based on empirical testing, a max chunk size of 512KiB recovers from the regression and was also observed to be the default in v2.19.3. Setting this unconditionally without relying on ncclGetVersion symbol being available, since the parameter did not exist in versions prior to v2.20. Signed-off-by: Raghu Raja <raghunch@amazon.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This heuristic was introduced by commit 8c6c595 ("2.19.3-1") with no explanation in commit message or comment in code other than "H800/H100 fixes and tuning".
In practice, we have noticed this line to cause a significant performance reduction when using NVLSTree algorithm when running nccl-tests all-reduce on a cluster of 8 AWS P5 nodes.
nccl-tests 4GB all-reduce output before this change:
output after this change:
output on commit f9c3dc2 ("2.19.1-1") prior to introducing this line:
Fixes: 8c6c595 ("2.19.3-1")