Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨[Feature] Autogen TRT Plugins #2817

Open
narendasan opened this issue May 6, 2024 · 0 comments
Open

✨[Feature] Autogen TRT Plugins #2817

narendasan opened this issue May 6, 2024 · 0 comments
Assignees
Labels
feature request New feature or request

Comments

@narendasan
Copy link
Collaborator

Is your feature request related to a problem? Please describe.

There are some cases where taking the extra step of wrapping a torch layer or pytorch custom op in a plugin to embed in at TRT engine may improve the performance of the model. However, there is a ton of boilerplate needed to actually access the operator through TensorRT. It would be great if this could get abstracted away for users.

Describe the solution you'd like

Given a functional torch operator and an FakeTensor implementation, autogenerate the TensorRT plugin code to allow that op to be embedded in a TRT engine.

Describe alternatives you've considered

This could be done in C++ as well but may be more complicated than handling this in Python.

Additional context

https://github.com/pytorch/TensorRT/blob/main/examples/dynamo/custom_kernel_plugins.py

@narendasan narendasan added the feature request New feature or request label May 6, 2024
@narendasan narendasan self-assigned this May 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant