-
Notifications
You must be signed in to change notification settings - Fork 48
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug] How do you handle graph breaks coming from Dynamo? #431
Comments
Hi @tbaggu , The dynamo custom backends can not control how the torch model is partitioned and converted to fx graph. The torch dynamo dispatch the fx graph to custom backend and the backend compiles the fx graph to some executable and returns to dynamo. The compilation only happens once and the compiled executable will be used many times. As long as the compiled executable is efficient, the overhead will not be very large. |
Hi
Thanks for the quick response, Is it each sub graph compiled and executed then the second will compile ??
Is that flow is n sequence?
Get Outlook for Android<https://aka.ms/AAb9ysg>
…________________________________
From: Yaoyao Ding ***@***.***>
Sent: Friday, February 23, 2024 12:58:15 PM
To: hidet-org/hidet ***@***.***>
Cc: Tirupathi Rao Baggu ***@***.***>; Mention ***@***.***>
Subject: Re: [hidet-org/hidet] [Bug] How do you handle graph breaks coming from Dynamo? (Issue #431)
Caution: This email originated from outside of the organization. Please take care when clicking links or opening attachments. When in doubt, contact your IT Department
Hi @tbaggu<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_tbaggu&d=DwMCaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=gq-inQZcrlLntwcvIjjLIH_HIhFbKD0Qm_YSVPXFWYE&m=ZC4zaNjHruoICMj8o_Zo5qwbgveusL9RZskQW2w8A1KgxSL6oInP6y-Ex5HuY9gp&s=4_Z96Qo5vny3sSw6vm4IyQCZTRobTc3rD8tKRg0schw&e=> ,
The dynamo custom backends can not control how the torch model is partitioned and converted to fx graph.
The torch dynamo dispatch the fx graph to custom backend and the backend compiles the fx graph to some executable and returns to dynamo. The compilation only happens once and the compiled executable will be used many times. As long as the compiled executable is efficient, the overhead will not be very large.
—
Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_hidet-2Dorg_hidet_issues_431-23issuecomment-2D1960847409&d=DwMCaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=gq-inQZcrlLntwcvIjjLIH_HIhFbKD0Qm_YSVPXFWYE&m=ZC4zaNjHruoICMj8o_Zo5qwbgveusL9RZskQW2w8A1KgxSL6oInP6y-Ex5HuY9gp&s=MqsJKff43jwbudwrspJOiSF1GuW41WH3S6wCLxG90gE&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_A4BNIQPD7MKQSZMQJKSLJALYVBAI7AVCNFSM6AAAAABDWFYL36VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRQHA2DONBQHE&d=DwMCaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=gq-inQZcrlLntwcvIjjLIH_HIhFbKD0Qm_YSVPXFWYE&m=ZC4zaNjHruoICMj8o_Zo5qwbgveusL9RZskQW2w8A1KgxSL6oInP6y-Ex5HuY9gp&s=KMT6SPIoBZHfgg4XIazK4WMrWj_Uke6Xciv6yteoHrk&e=>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Yes. model_opt = torch.compile(model, backend='custom-backend')
model_opt(x) # sub graph will be compiled and executed
model_opt(x) # the cached compiled executable will be used, no compilation at all |
So for each subgraph result should comeback to CPU and again it goes back to device right? Which is latency over head right?
Get Outlook for Android<https://aka.ms/AAb9ysg>
…________________________________
From: Yaoyao Ding ***@***.***>
Sent: Friday, February 23, 2024 1:19:48 PM
To: hidet-org/hidet ***@***.***>
Cc: Tirupathi Rao Baggu ***@***.***>; Mention ***@***.***>
Subject: Re: [hidet-org/hidet] [Bug] How do you handle graph breaks coming from Dynamo? (Issue #431)
Caution: This email originated from outside of the organization. Please take care when clicking links or opening attachments. When in doubt, contact your IT Department
Yes.
model_opt = torch.compile(model, backend='custom-backend')
model_opt(x) # sub graph will be compiled and executed
model_opt(x) # the cached compiled executable will be used, no compilation at all
—
Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_hidet-2Dorg_hidet_issues_431-23issuecomment-2D1960869256&d=DwMCaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=gq-inQZcrlLntwcvIjjLIH_HIhFbKD0Qm_YSVPXFWYE&m=NYlKO7jDMD63GsvK19tQ-JTXAGjnwoobE8ZiR5paV2kWfRyUvFYLswhWGrMy7c-M&s=yNyynTt-0EY6muA2axQ7QbOw334I8WQLlH6Ev7Smh-E&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_A4BNIQPJNZRMHG2BB4WMO2LYVBCZZAVCNFSM6AAAAABDWFYL36VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRQHA3DSMRVGY&d=DwMCaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=gq-inQZcrlLntwcvIjjLIH_HIhFbKD0Qm_YSVPXFWYE&m=NYlKO7jDMD63GsvK19tQ-JTXAGjnwoobE8ZiR5paV2kWfRyUvFYLswhWGrMy7c-M&s=odv07tt-4qJQryU3koLan4MauB9i2j2LYOdN-QIgZdM&e=>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
No, the result will stay in the same device (cpu or gpu) as the eager execution of the orginal model. |
Closing as this issue is not directly related to Hidet. |
Hi
I am taking this repo as reference to implement code for our custom back-end. During the development process most of the cases when i use hugging-face models directly i see lot of graph breaks, in Fx graph,
My understanding on the inductor side is, each subgraph compiled and ran the inference sent the results to CPU, and the next subgraph will start executing which is time consuming?
So my question is ,
Have you seen such graph breaks if so how Hidet handles them?
similar to below case
https://discuss.pytorch.org/t/stitching-together-graph-breaks-for-large-compilation-units/194793/5
The text was updated successfully, but these errors were encountered: