File-like objects are closed after uploading

[JordonPhillips]

Source boto/boto3#929

This is because we wrap them with the InterruptReader which closes them on exit.

[kyleknap]

Yeah looks like it will only happen for the non-multipart upload case because the provided file object is used. Will probably have to figure out a way to safeguard against accidentally closing the file.

[ramurti]

Any workaround to prevent the file being closed? I'm using this function with unnamed temporary files, so I can't simply reopen the file. I'd like to do something else with this file (for example, rotate it to local disk) in case upload_fileobj fails, but it closes the file.... (sigh)

[micahfay]

The workaround by leonsmith here worked for me- matthewwithanm/django-imagekit#391

[ofassley]

Same issue here. I'm also using Imagekit.

self = <ImageCacheFile: CACHE/avatars/a0cbd7767bf54e649f4952e8c3407418/9dd51a8dd80342ae7576d1c376b51e43.jpg>

    def _generate(self):
        # Generate the file
        content = generate(self.generator)

        actual_name = self.storage.save(self.name, content)

        # We're going to reuse the generated file, so we need to reset the pointer.
>       content.seek(0)
E       ValueError: I/O operation on closed file.

actual_name = 'CACHE/avatars/a0cbd7767bf54e649f4952e8c3407418/9dd51a8dd80342ae7576d1c376b51e43.jpg'
content    = <File: None>
self       = <ImageCacheFile: CACHE/avatars/a0cbd7767bf54e649f4952e8c3407418/9dd51a8dd80342ae7576d1c376b51e43.jpg>

../../../.virtualenvs/myapp/lib/python3.5/site-packages/imagekit/cachefiles/__init__.py:102: ValueError

[mgrdcm]

I can't tell - is this a bug? I'll be happy to take a stab at a PR if it is, and if nobody else wants to handle it. I'm also suffering from the django-imagekit issue.

[micahfay]

I've stopped using django-storages and moved to this for django-s3 stuff- lighter weight, and seems to be better supported. Also, it doesn't require that workaround :) https://github.com/etianen/django-s3-storage switching was easy even with existing data.

[joshma]

@kyleknap when you get a chance, could you chime in on whether calling self._fileobj.close() in InterruptReader is necessary? We're running into the same issue, and it's particularly difficult because we're accepting a few kinds of file-like objects so we can't just re-open it later.

[freemanlevinetang]

Ran into this issue recently while attempting to generate images, upload them to s3 for backup, and then attach them to an email.
Workaround was to pull the file back down from s3 after uploading, which seems pretty inefficient.

[leogzyl]

I'm using the following workaround:

from io import BufferedReader

class NonCloseableBufferedReader(BufferedReader):
    def close(self):
        self.flush()

buffer = NonCloseableBufferedReader(myfile)
client.upload_fileobj(Fileobj=buffer, Bucket=bucket, Key=name)
buffer.detach()
assert not myfile.closed

[zachliu]

@mgrdcm I think it is a bug. The control should be relinquished to the end user instead of closing it prematurely.

[zachliu]

@leogzyl A near perfect workaround except mypy complains if I pass in tempfile.TemporaryFile() object:

error: Argument 1 to "NonCloseableBufferedReader" has incompatible type "IO[Any]"; expected "RawIOBase"

Any solutions other than # type: ignore?

[ncvc]

@zachliu I added an init method with a cast to work around the mypy issue:

class NonCloseableBufferedReader(BufferedReader):
    def __init__(self, raw: IO[bytes], *args: Any, **kwargs: Any):
        super().__init__(cast(RawIOBase, raw), *args, **kwargs)

    def close(self) -> None:
        self.flush()

NonCloseableBufferedReader(myfile)  # No mypy errors

[al-muammar]

3 years passed... Can it be either fixed or declared to be not a bug, so other libraries should build workarounds for it?

[jannotti]

I mean, it's clearly a bug, right? You can't specify that a method maybe, sometimes, occasionally closes your files.

[arthurzenika]

We're affected by this too. The NonCloseableBufferedReader workaround seems to do the trick, but hopefully we can remove it once this is fixed.

[rooom13]

The issue still appears on March 2020, @leogzyl 's workarounds solves the issue

[zivkovic]

There is many of us with this issue, something should be done with it.

[diego200052]

I got this same issue on April 2021 using boto3 and upload_fileobj() function.

[jvanasco]

This behavior is not documented. This behavior is not expected. The first version of boto did not implement this behavior. I've been using Python for over 20 years and have never seen a library close a file in this context. There is no reason why this function should close the file.

The only thing more ridiculous than this bug existing, is that it has not been fixed in 4.5 years. If keeping the file open causes an issue, then the library has serious underlying problems.

Aside from the numerous issues this creates with on-disk tempfiles (as one other user mentioned above) which can be nearly impossible to re-open, file-like objects in Python routinely include SpooledTemporaryFile and StringIO objects. This absurd, unexpected and undocumented behavior completely destroys these file-like objects and renders them unrecoverable.

[iamahuman]

Suspected regression commit: 2f3d12c

Diff: 2f3d12c#diff-fb5195c2216d4f5a452b31d518beae30b39667b0655d8a8fd1663a183575aa0aL167-R165

-        client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
+        with osutil.open_file_chunk_reader(
+                fileobj, 0, size, progress_callbacks) as body:
+            client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)

This is because we wrap them with the InterruptReader which closes them on exit.

It appears that InterruptReader has nothing to do with the issue itself. It just so happens that its close() method is in the traceback. Also, patching the issue that way won't last if InterruptReader gets replaced with something else or removed in the future.

Put another way, InterruptReader is just a simple wrapper around a file object, and is not the right abstraction layer to put exceptional logic such as ignoring file close requests (which violates Python's file-like object wrapper protocol).

[saranglakare]

5 years anniversary of this issue and it's still open! I just wasted a couple of hours figuring out my problem and it came to this bug! Why isn't anyone working on this?

[mgrdcm]

5 years anniversary of this issue and it's still open! I just wasted a couple of hours figuring out my problem and it came to this bug! Why isn't anyone working on this?

It's tradition now.

[Natgho]

Tradition is continue :(

[yswtrue]

Same here

[antoniosarosi]

@saranglakare Almost 7 years and still counting 😂, just run into this issue today.

[sangshuduo]

I encountered this bug on May 3, 2024.

[1mamute]

Kinda disappointing this issue is almost 8 years old and not even a single mention in the documentation was made.

Here's another hacky workaround:

def patched_close():
    print("good try boto3")

file = BytesIO()

# Save the original close function
original_close = file.close

# Patch the close function with ours
file.close = patched_close

client.upload_fileobj(Fileobj=file, Bucket=bucket, Key=name)

# Unpatch it
file.close = original_close