"The file cannot be reopened" when used with Django imagekit about django-s3-storage HOT 8 CLOSED

Heh, no worries, closing an issue is it's own reward.

If nobody steps forth to claim it, maybe throw the money at a worthy charity.

from django-s3-storage.

Thanks for the report?

Could you give me the full stack trace?

Do you get the same error using filesystemstorage?
On Tue, 4 Oct 2016 at 05:35, Mitchel Humpherys [email protected]
wrote:

I'm using django-s3-storage along with django-imagekit
https://github.com/matthewwithanm/django-imagekit, but there seems to
be a problem with the way django-s3-storage opens files. It's easy to
reproduce, and I have a full django app and repro instructions for the
issue, here: https://github.com/mgalgs/iktest

I was a bit thrown off for a while because django-s3-storage isn't
anywhere in the stack trace of the erroneous situation, but @scottmx81
https://github.com/scottmx81 figured out that the test app works fine
with django-storages, or by monkey-patching pilkit.open_image like so:

from pilkit.lib import Imagefrom pilkit import utils

def open_image(target):
try:
target.seek(0)
return Image.open(target)
except ValueError:
return Image.open(target.storage.open(target.name))

utils.open_image = open_image

With django-storages no monkey-patch is required...

/cc @scottmx81 https://github.com/scottmx81

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#33, or mute the
thread
https://github.com/notifications/unsubscribe-auth/AAJFCFOW5X3hoqUjkjrTfPm2ZifHngb5ks5qwdeQgaJpZM4KNS0u
.

from django-s3-storage.

Okay, I don't think that this is an error with django-s3-storage.

Check out the source of your error, here:

https://github.com/django/django/blob/master/django/core/files/base.py#L131

Some part of your code is closing the file, then reopening it, which is not supported by in-memory files. Look for a with statement or close being called on the file somewhere, or maybe an option to have the pilkit not close the file when it's done.

I think it's working with django-storages because they seem to stream stuff to disk for all file operations, while django-s3-storage uses memory until the file exceeds 10MB. Files on disk can be reopened, hence the difference in behaviour. django-s3-storage will perform better for lots of small files (such as use as a staticfiles storage), but you have to be careful about not closing files until you're done with them.

from django-s3-storage.

@etianen thanks, you are right that there is a close being called on the file somewhere. The django-imagekit library that we are using generates thumbnails from a source image. After generating each thumbnail, it closes the source image, and then opens it again when generating the next thumbnail. I have confirmed that by commenting out the line of code where the source image is closed, the test app referenced by @mgalgs will work correctly.

The problem is that it does not seem like calling close on an image field, and then opening it again, is a bug in the other code. This does work without problem when using filesystemstorage.

Is there any possibility that support for reopening a file could be included in django-s3-storage? I understand that it is the Django File class where the error is coming from, and that it is using os.path.exists to check for the file in the server filesystem, and not through the Django storage system that it was originally opened with. But what about if instead of returning the File class from Django when the file is opened through django-s3-storages, it returned a subclass of File that was aware of S3, and tried opening itself again through django-s3-storages, instead of using os.path.exists and Python built-in open function?

from django-s3-storage.

You make a very good point. I'll look into this next week.

from django-s3-storage.

OR, I'll take a pull request! :)

from django-s3-storage.

Apologies for the delay, I was massively bogged down with other work.

S3 files can now be re-opened. Of course, better still to keep them open, as it's more efficient, but at least being inefficient works now! :)

from django-s3-storage.

Fix confirmed! Thank you!!!

If you'd like, feel free to reap this small bounty :).

from django-s3-storage.