Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

High memory issue with RunLogger #77

Open
1 of 2 tasks
leo-schick opened this issue May 18, 2022 · 1 comment
Open
1 of 2 tasks

High memory issue with RunLogger #77

leo-schick opened this issue May 18, 2022 · 1 comment

Comments

@leo-schick
Copy link
Member

leo-schick commented May 18, 2022

The RunLogger class caches the Output event and writes it to a local variable (node_output). There are two issues with the current implementation which might cause high memory usage during pipeline execution:

  • when you have a long running task / parallel task, a long output chain might be cached. There should be a solution implemented which writes the cached output to the db after e.g. 500 entries.
  • the node_output dict is not freed after writing the output to the dabase. This will result in that the whole pipeline ouput is cached in RAM until class RunLogger is disposed (!!)
@leo-schick leo-schick changed the title Memory issue with RunLogger High memory issue with RunLogger May 18, 2022
@leo-schick
Copy link
Member Author

I noted that the ChatNotifier has a similar issue: It fees the node_output only on RunFinished, not on NodeFinished but this might be wanted. This is out of scope for this ticket.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant