Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize the speed of concurrent get of pytorch models #1884

Closed
dashanji opened this issue May 7, 2024 · 0 comments · Fixed by #1891
Closed

Optimize the speed of concurrent get of pytorch models #1884

dashanji opened this issue May 7, 2024 · 0 comments · Fixed by #1891
Assignees
@dashanji
Labels
performance Issues that related to the performance of vineyardd and vineyard SDKs.

Comments

@dashanji
Copy link
Member

dashanji commented May 7, 2024 •
edited

Describe your problem

Currently, getting a pytorch module at high concurrency is very slow as follows. The test machines's max bandwidth are both 30Gbps.

Vineyard

Concurrencies Time of getting Observed Network Bandwith from Dstat
1 2.57s around 2000Mi
6 7.73s around 3800Mi
13 14.58s around 3800Mi
27 29.32s around 3800Mi

Iperf

Concurrencies Observed Network Bandwith from Dstat Total Network bandwidth
1 around 1470Mi 12Gbits/s (1500Mib/s)
6 around 3700Mi 31.1Gbit/s (3888Mib/s)
13 around 3650Mi 30.9Gbit/s (3863Mib/s)
27 around 3650Mi 30.9Gbit/s (3863Mib/s)

Solution

In the actual scenery, the pytorch models used to be loaded in the machine with GPU, which always have high- performance networks. Thus, the bandwidth of vineyardd instance is the bottleneck. We can distribute the PyTorch model blobs among different Vineyard instances to increase network bandwidth.
@dashanji dashanji added the performance Issues that related to the performance of vineyardd and vineyard SDKs. label May 7, 2024
@dashanji dashanji self-assigned this May 10, 2024
@dashanji dashanji mentioned this issue May 15, 2024
Merged
sighingnow pushed a commit that referenced this issue Jun 5, 2024
@dashanji
Support to put a torch module into all vineyard instances dispersedly (…
3e9ff47 
…#1891)

Fixes #1884

Signed-off-by: Ye Cao <caoye.cao@alibaba-inc.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dashanji dashanji

Labels
performance Issues that related to the performance of vineyardd and vineyard SDKs.
Projects
Vineyard Releases
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support to put a torch module into all vineyard instances dispersedly dashanji/v6d
1 participant
@dashanji