Error "encoder error -2 when writing image file" occurs for some JPEG images about pilbox HOT 13 CLOSED

A quick Google search brought this up: python-pillow/Pillow#169, which, at a minimum, seems to indicate that you should have a more descriptive error sent to STDOUT. If you see it there, hopefully that will be more helpful in diagnosing the root cause.

from pilbox.

Unfortunately since it's running on the web I do not get anything else on stdout. All I get is:

[W 150108 01:20:40 web:1404] 415 GET /?deg=auto&filter=bilinear&mode=clip&op=resize&opt=1&prog=1&q=keep&url=https%3A%2F%2Fplic-staging.s3.amazonaws.com%2Fphotos%2F981f6f66-6f06-4a15-a7a4-1f29496cb30d%2Foriginal%2FFlag%2520-%2520Candid%2520R.jpg%3FAWSAccessKeyId%3DAKIAJAMRRGAHI5LJPB6A%26Expires%3D1423271467%26Signature%3DGMoTpEr6RalUdhu3UtKc5decGgf%253D&w=200&sig=859ed3b1fc3ee2a6704f95750f11f6cdb1dc3132 (10.93.25.70): encoder error -2 when writing image file 

Is it possible that Pilbox is catching the other error or not forwarding it to the logs?

from pilbox.

I can't really speak to how you're redirecting STDOUT/STDERR since I don't really know your setup. But can you confirm that you're redirecting both STDOUT and STDERR to your expected logs?

from pilbox.

It's running on Heroku, and Heroku always directs STDOUT and STDERR to the logs. I tried adding -u to the Python command and set PYTHONUNBUFFERED=True in the ENV just in case there was a buffering issue (suggested here) but I still don't see any more error details in the logs.

Additionally, we just now discovered by accident that if we add the rotate op it no longer produces the error.

• op=resize&mode=clip&filter=bilinear&opt=1&prog=1&q=keep&w=200 - FAIL (What we're trying to use)
• op=resize&mode=clip&filter=bilinear&opt=1&q=keep&w=200 - FAIL (one param removed from original)
• op=resize&mode=clip&opt=1&q=keep&w=200 - FAIL (two params removed from original)
• op=resize,rotate&deg=0&mode=clip&filter=bilinear&opt=1&prog=1&q=keep&w=200 - No Error (rotate and deg added)
• mode=clip&q=keep&w=200 - No Error
• w=200 - No Error

There may be other combinations, but each specific query produces either a failure or a success consistently with the same image.
It's weird that adding the rotate op or removing other params stops the error. For now we can workaround by only using known-good query combinations by hit-and-miss testing.

from pilbox.

Not the same error, but I'm able to see these messages in my log:

/usr/local/lib/python2.7/dist-packages/PIL/TiffImagePlugin.py:464: UserWarning: Possibly corrupt EXIF data.  Expecting to read 19660800 bytes but only got 0. Skipping tag 0
  "Skipping tag %s" % (size, len(data), tag))
/usr/local/lib/python2.7/dist-packages/PIL/TiffImagePlugin.py:464: UserWarning: Possibly corrupt EXIF data.  Expecting to read 2684485632 bytes but only got 0. Skipping tag 0
  "Skipping tag %s" % (size, len(data), tag))
/usr/local/lib/python2.7/dist-packages/PIL/TiffImagePlugin.py:464: UserWarning: Possibly corrupt EXIF data.  Expecting to read 1575056 bytes but only got 785. Skipping tag 0
  "Skipping tag %s" % (size, len(data), tag))

For

/?url=https%3A%2F%2Fcamo.githubusercontent.com%2Fc1c9d345ca82c833ce2ee6e3e954fc97f1ba70ef%2F68747470733a2f2f662e636c6f75642e6769746875622e636f6d2f6173736574732f333038383737332f323038363939322f33383039313233612d386536332d313165332d383338362d6564353435343532626132372e6a7067&op=rotate&deg=auto

Which is the example image that has corrupted EXIF data. However, they only seem to be getting reported on the first failure -- no idea how/why that would happen. If the same is true of your error, maybe it's possible that it's buried somewhere in your logs. Maybe bounce your server, reproduce the error and check your logs.

If you send me the image that is failing, I might have a better chance of reproducing this, but without it, can't really do too much more to help.

from pilbox.

I checked the logs front to back after a full restart but still nothing new. I'm afraid that since I can't reproduce on my local dev machine, chances for you also reproducing are also slim. But here's the failing image to try: https://drive.google.com/open?id=0Bx-VFyfqu1DlaVRZamYxYmtzTm8

I apologize for requiring a Google sign-in, but when I uploaded to an image sharing service it altered the image file.

from pilbox.

I get the following when I resize your image:

Suspension not allowed here
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/vagrant/pilbox/image.py", line 448, in <module>
    main()
  File "/vagrant/pilbox/image.py", line 442, in main
    progressive=options.progressive)
  File "/vagrant/pilbox/image.py", line 245, in save
    raise errors.ImageSaveError(str(e))
pilbox.errors.ImageSaveError: HTTP 415: Unsupported Media Type (encoder error -2 when writing image file)

Using:

python -m pilbox.image --operation=resize --mode=clip --filter=bilinear --optimize=1 --quality=keep --width=200 test.jpg 

Not 100% sure if you're getting the same, but Suspension not allowed here indicates some kind of buffer size problem. I'm no Pillow expert, but looking at the Pillow JPEG plugin code, when quality is equal to or above 95, it will double the buffer size, however since keep is being used that condition is not reached:

if "optimize" in info or "progressive" in info or "progression" in info:
    if quality >= 95:
        bufsize = 2 * im.size[0] * im.size[1]
    else:
        bufsize = im.size[0] * im.size[1]

This seems consistent with your tests of varying resizing parameters. As a work around, you can either disable optimize and progressive or set quality to a numeric value.

from pilbox.

Okay, thanks for the workaround details. It looks like "Suspension not allowed here" is also in my logs, but I didn't spot it near the error I reported.

from pilbox.

Sure, no problem. If you decide to set quality to a numeric value, I'd recommend not setting it above 95 as you may notice some performance problems when setting it above that.

from pilbox.

Proposed fix for this in: python-pillow/Pillow#1079 . All it does is bump up the buffersize when quality=keep to the same level as when quality >= 95. If the increased bufsize isn't enough, then there are still going to be issues. Please let me know if that's the case.

from pilbox.

Here is another image that we have failing with the same error (Pilbox 1.1.6). Is the proposed fix in Pillow merged into Pilbox's deps?

https://drive.google.com/file/d/0B8C_eTkaHeOdZGpvUjBPS1hmS28/view?usp=sharing

from pilbox.

The fix has only been applied to the Pillow master branch, but has not yet been cut as an official release. When that does happen, I'll update the dependency.

from pilbox.

Okay, thanks.

from pilbox.