Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

md5sum on uploaded videos before moving them from temporary directory #937

Open
KyleMaas opened this issue Dec 15, 2023 · 1 comment · May be fixed by #963
Open

md5sum on uploaded videos before moving them from temporary directory #937

KyleMaas opened this issue Dec 15, 2023 · 1 comment · May be fixed by #963
Assignees

Comments

@KyleMaas
Copy link
Contributor

Describe the issue
I think this is part of the cause of #852. I noticed the other day when MediaCMS was processing an upload, it was running an md5sum on the uploaded video before it responded to the client with a success message. In tracking this down, it looks like this is being done here:

cmd = ["md5sum", input_file]

And going up the call stack:

ret = helpers.media_file_info(self.media_file.path)

self.set_media_type()

Which I think is coming from here:

instance.media_init()

For those of us with large slow primary video data storage but fast /tmp on a tmpfs, this really should be happening locally on the server processing the upload rather than making the uploading client wait for it to go to the slow disk and then be read back again for the md5sum.

To Reproduce
The impact of this is not particularly noticeable unless you're processing huge video files and your storage is slow, so this may not be easy to duplicate without uploading videos >1 hour long. However, if you can set up the conditions correctly:

  1. Upload a long video - at least an hour
  2. In a separate window, open a terminal and watch ps or top for an md5sum command
  3. Notice that this happens very early in the process

Expected behavior
The md5sum should be calculated on the uploaded file before being moved from /tmp to primary storage.

Screenshots
N/A

Environment (please complete the following information):

  • OS: Ubuntu Linux
  • Installation method: Single server install
  • Browser, if applicable: N/A
@KyleMaas
Copy link
Contributor Author

new = Media.objects.create(media_file=myfile, user=self.request.user)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants