Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better handling of Prefect __getitem__ calls #2070

Open
Andrew-S-Rosen opened this issue Apr 30, 2024 · 0 comments
Open

Better handling of Prefect __getitem__ calls #2070

Andrew-S-Rosen opened this issue Apr 30, 2024 · 0 comments
Labels
enhancement New feature or request help wanted

Comments

@Andrew-S-Rosen
Copy link
Member

Andrew-S-Rosen commented Apr 30, 2024

What new feature would you like to see?

We have monkeypatched __getitem__ calls on PrefectFuture and State in the __init__.py to ensure proper waiting on dictionaries with futures in them. However, this creates a dedicated job every time it's done, which increases the number of calls per minute (API rate limit more likely) and clogs up the UI. We should try to see if we can come up with a workaround.

if SETTINGS.WORKFLOW_ENGINE == "prefect":
from prefect.futures import PrefectFuture
from prefect.states import State
def _patched_getitem(self, index):
@job
def _getitem(future, index_):
return future[index_]
return _getitem(self, index)
PrefectFuture.__getitem__ = _patched_getitem
State.__getitem__ = _patched_getitem

@Andrew-S-Rosen Andrew-S-Rosen added enhancement New feature or request help wanted labels Apr 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted
Development

No branches or pull requests

1 participant