Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DumpExporter fills up RAM #1226

Open
magula opened this issue Feb 11, 2023 · 0 comments
Open

DumpExporter fills up RAM #1226

magula opened this issue Feb 11, 2023 · 0 comments

Comments

@magula
Copy link
Contributor

magula commented Feb 11, 2023

cmsDumpExporter will put everything in a tempdir (i.e. usually /tmp) before packing it up in an archive at the desired path:

export_dir = os.path.join(tempfile.mkdtemp(),
archive_info["basename"])

On some systems, including Arch, /tmp is in RAM by default, and limited to half the RAM's size. At least on personal computers, this will usually cause the system to freeze or crash. Of course, CMS writes to /tmp on other occasions, but in normal operation, it will cleanup shortly afterwards and never write something as large as the whole database at once.

It seems that by setting one of the TMPDIR, TEMP, or TMP environment variables, one can specify another location, so circumventing the issue is technically easy. However, not everyone will know this and think of it beforehand, and a system freeze is annoying, as is finding out the cause of the problem.

Thus, I would like to suggest improving this somehow.

  • The files have to be written somewhere of course, and using tempfile.mkdtemp makes sense. Easy and still helpful would be a warning, so at least one will know why the system froze (or be able to hit STRG+C if fast enough...). This should include suggesting to set TMPDIR to somewhere big enough.
  • Maybe it could even explicitly ask along the lines of Path to temporary directory (default is system temp dir; sufficient free space is needed):
  • Or it could use the directory where it shall export the archive. Drawback: If the exporter crashes, it won't be cleaned up.

This sounds similar to #322, but I think that was more about importing and exporting specific contests and is unrelated.

Happy to hear your thoughts!

PS: This has been in the back of my head since forever and I haven't seen someone else mention it, so who knows if I'm the only one with this problem. 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

1 participant