cshum / imagor Goto Github PK

View Code? Open in Web Editor NEW

3.2K 24.0 119.0 10.96 MB

Fast, secure image processing server and Go library, using libvips

License: Apache License 2.0

Go 92.15% Makefile 0.17% Dockerfile 0.58% C 7.10%

docker jpeg png libvips resize-images crop-image watermark gif webp avif

imagor's People

Contributors

Stargazers

Watchers

Forkers

isgasho shatgupt jrz dougperkes achillesxu antonlinderer orinocoz kokizzu gihnius doytsujin shaunstanislauslau lyrl tianyaliu robert-joscelyne m8e bryanweingarten sgntechhub laplacekorea spread0x r0bdiabl0 icodein nuan123 c4x deepexpert-chenkai shenjiquanzi coosir forcomwork lijinjf19800404 duhn sunan-irystec caik13 nyaaarlathotep yijia2413 yiflylian mistshi 108518 sanyuesiyuewuyue gitobjects developgo seejser priestd09 jackwiy liflife pletessier yuzhoustayhungry yusongying samthehai m301 asyrawih srlk gangdiedao eudaimoniatech eudaimonia-tech parera10 lorenries l3dlp-sandbox hilli jvahldick chinnakornp zyx134 syrinscape funsunme super-rain cailiang daniel-007 kkdai gikeihi kldewar forkmeplease mrtmexx ruhuansanmei thomasf duan-nguyen ohsevin suofiya popstreamllc vinodborole nicholascapo finiliar-studios gg-big-org shankantino axioscode ario-eth xuweiguo huangxuelun alimy showsmall nestor-sk wzlove binrebin angstormnet patricktoca darevski astrolox cholleybc magiczml p2004a blesswinsamuel georgemei-eslite dangkaka

imagor's Issues

Loading an image directly from S3 Storage

Hello, first of all, thank you for the great tool!

I have a question and hope you could help me.

I have a minio server and two buckets in it, the first one is named source, the second one is result
I upload a random image into source bucket, and it is directly accessible via this link, for example http://localhost:9000/source/ship.jpg
So my question is, is it possible to pass only an image name (or its directory + image name) as an argument to imagor and the image will be fetched directly from S3 and after processing stored in result bucket?

For example, like this: http://localhost:8000/unsafe/200x200/ship.jpg

These are my docker env parameters

    environment:
      - 'PORT=8000'
      - 'AWS_REGION=us-east-1'
      - 'AWS_ACCESS_KEY_ID=***'
      - 'AWS_SECRET_ACCESS_KEY=***'
      - 'S3_ENDPOINT=http://minio2:9000/'
      - 'S3_STORAGE_BUCKET=source'
      - 'S3_RESULT_STORAGE_BUCKET=result'
      - 'SERVER_ACCESS_LOG=1'
      - 'IMAGOR_UNSAFE=1'
      - 'DEBUG=1'

Add source whitelist

We have a large number of services and sharing the secret between the services has been proven difficult. As an alternative we are using ALLOWED_SOURCES in thumbor:

ALLOWED_SOURCES: >
            [".*\.foobar\.com","my\.foobar\.com", "sendbird-ap-9\.s3\.amazonaws\.com"]

This will only accept and process images from the listed sources.

Looking at something similar in imagor.

Is it possible to get bucket url?

If you are using public s3 bucket for storing the processed image, is it possible to get the direct s3 URL of the image?

Is there a way to restrict files so that they are an image?

I am noticing I can pass google.com as a path to the example and it will show or render google.com. Is there a way to restrict the input so that it only resizes images or certain file types?

How to activate file saving?

Great tool!
The only question I have is how to activate file saving: I have defined the FILE_STORAGE_BASE_DIR in the docker-compose.yml file yet nothing happens when triggering the image treatments from the browser.

Resizing Not Working If Increasing Image Size W/O Stretch Param

As we are testing imagor as a true drop-in replacement for Thumbor, we have noticed that the basic resizing of an image does not work the same when trying to make an image larger.

For example, with Thumbor, if I want to make an image that is 640x480 increase to a width of 780, I can simply use these minimal URL parameters:

http://localhost:8000/unsafe/780x0/photos%2Fb%2Fef88c521cbbb57e05c12a2dcc696724f12c9155f.jpg

Using that same URL in imagor displays the image in it's original size.

I know that imagor does not have all of the features as Thumbor as of yet but wanted to point this out because it is a basic way to increase image sizes that I'm sure I'm not the only one doing this. Even though imagor offers a stretch param that works, to be a true drop in replacement, it would be nice to not have to use this as it may not be easy to change the url params in whatever backend code utilizes it.

compression?

Hey @cshum it's me again :)

This is more of a request/wish than anything, but I'd be curious if it's possible to push the limits a bit... I wonder if the quality filter or some other potential compression filter can be applied to make images smaller?

For example, I took the gopher image from the examples using

http://localhost:8000/unsafe/filters:hue(290):saturation(100):fill(yellow):format(jpeg):quality(80)/raw.githubusercontent.com/cshum/imagor/master/testdata/gopher.png

and I get this image (136kb)

If I then compress it with tinypng.com it compresses it to 104kb ... not sure if it's lossy or lossless though

I played around with some of our images, and the difference in size can be even bigger...

Is it purely a question of fine-tuning the quality? or is there some other compression that can be used?

PDF render support

Can Imagor support PDF file processing, is it possible to render specific page of a PDF?

Support for path style S3 URL

Hi,
I am trying to use a S3 object storage which doesn't support subdomain style URL like bucket.namespace.domain.tld. It supports only path style URL like namespace.domain.tld/bucket. Can you please make changes to the s3storage adapter so that it could support path style URL too?
Maybe a new config option for s3 storage. Sometjing like S3_URL=path (default url) would be nice. I am not familiar with GO. So, I can't make a PR.

Thanks

traefik (or another reverse proxy) configuration

there is a way to configure imagor under traefik or another reverse proxy?
thank you

Access rights considerations in a docker installation

I have an image present at the right location (I checked), with the right authorizations, with a fairly standard name: IMG_20180408_153807553_HDR.jpg

And yet it is not treated by the Imagor server with this error:
{"Op":"Get","URL":"https://pho/tst/IMG_20180408_153807553_HDR.jpg","Err":{"Op":"dial","Net":"tcp","Source":null,"Addr":null,"Err":{"Err":"no such host","Name":"pho","Server":"127.0.0.11:53","IsTimeout":false,"IsTemporary":false,"IsNotFound":true}}}

Other images with underscores or uppercase letters are being treated fine in the exact same environment.
The only difference is that the image weighs 3MB but that should not be an issue and is not what the error reports anyway.

Please let me know what I am doing wrong.

Question :: Support to image borders

Hi there,
First of all, congrats on the project. It seems really promising.

I am currently trying to add borders to the image, however, it seems that the project does not support it right now.
Is there any plan on including it? or do you know what is the best way to achieve it?

I tried a couple of possibilities to achieve it with a single URL such as

using padding and applying "round corner"
applying filters multiple times
/filters:round_corner(310,310,ff0000)/filters:round_corner(305:305)
/filters:round_corner(310,310,ff0000):round_corner(305:305)

For some reason, the padding does not seem to work fully with the crop/round corner.

同学，您这个项目引入了53个开源组件，存在1个漏洞，辛苦升级一下

检测到 cshum/imagor 一共引入了53个开源组件，存在1个漏洞

漏洞标题：Gin-Gonic Gin 环境问题漏洞
缺陷组件：github.com/gin-gonic/[email protected]
漏洞编号：CVE-2020-28483
漏洞描述：Gin-Gonic Gin是Gin-Gonic团队的一个基于Go语言的用于快速构建Web应用的框架。
github.com/gin-gonic/gin 全部版本存在安全漏洞，该漏洞源于可以通过设置X-Forwarded-For头来欺骗客户端的IP。
影响范围：(∞, 1.7.7)
最小修复版本：1.7.7
缺陷组件引入路径：github.com/cshum/imagor@->github.com/gin-gonic/[email protected]

另外还有几个漏洞，详细报告：https://mofeisec.com/jr?p=ab77bc

Automatic webp conversion

If the browser has the capability the image should be returned in WebP, instead of JPEG.

thumbor related option: AUTO_WEBP: "True"

Benchmarks

Hi again :)

I'm trying to run some simple benchmarks to compare thumbor performance to imagor. For some reason I'm not sure the VIPS_CONCURRENCY_LEVEL env has an effect? It seems like it's always using all available CPUs ... Also interesting is that whilst it seems to run faster compared it to a single-process thumbor, when I run it with concurrency of 4 against thumbor (with the same concurrency), thumbor out-performs imagor (without caching, only keeping the original image in file storage) ...

Perhaps it's due to the way I've setup the benchmark? maybe it doesn't keep the original image in storage and always fetches it? or some other trivial misconfiguration? because otherwise I would expect libvips to outperform thumbor ...

You can see the benchmark code I'm running at https://github.com/MinimalCompact/thumbor/tree/imagor-thumbor-benchmark/benchmarks/locust (it's based on an older benchmark I created for optimizing thumbor on a multi-proc environment, so it's a bit messy unfortunately, but hope it's still useful)

I tested it on a DigitalOcean droplet with 4 cpus. Here's a sheet with the stats

confusing 404 warnings

I just setup imagor and it's working great, but I see a lot of log lines like this

{"level":"warn","caller":"imagor/imagor.go:305","ts":1646775438.21022,"key":"0x241:5017x3165/326x190/filters:strip_exif()/<escaped_http_url>","msg":"load","error":"imagor: 404 not found"}

Shortly after I see the 200 response in the access log

{"status":200,"ts":1646775438.541976,"level":"info","ip":"xxx","caller":"server/handler.go:98","took":0.331850614,"uri":"/unsafe/0x241:5017x3165/326x190/filters:strip_exif()/<escaped_http_url>","msg":"access","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:37.0) Gecko/20100101 Firefox/37.0","method":"GET"}

I assume the 404 is coming from the result storage? I'm running with

-file-result-storage-base-dir=/tmp -file-result-storage-expiration=1h

My intent with the result storage is to use it as a short-term cache, so misses are expected. Is there a way to silence this message?

Also, if my assumption is correct, using a "404 not found" message for a file not found on the filesystem was confusing and somewhat alarming for images I knew should be returning a 200. It would be nice to have a message that told me it didn't exist on the filesystem instead (maybe with the full path?).

Feature Request: minio(s3 protocol compatible) storage engine support.

First of all, thanks the author make this amazing project.

Could you please add support for minio?

Ability to disable responding with version

Due to security concerns, the response of / should be customisable or at a minimum, the version should be removable.

Status Quo:

{"imagor":{"version":"0.8.15"}}

Proxy for idempotent behaviour

With a proxy and TTL cache when we make the same request again it will use the cache.

is there anything like this currently in the code base ?

The image on Docker Hub is not multi-arch.

But the GitHub Actions workflow appears to be attempting to build and push a multi-arch image.

Perhaps the config is building both platforms with the same tag and overwriting the arm64 image with the amd64 image, instead of pushing two separate images with a multi-arch manifest?

Access log

Hi,

How can I enable the HTTP access log for the imagor? If it is possible how can I change the log format?

ffmpeg backend, thoughts?

So I've noticed Thumbor has https://github.com/theatlantic/thumbor-video-engine

Was wondering if @cshum has any thoughts on this, or if there is any ongoing effort to add ffmpeg as a backend (or something else that can do video, alongside vips) and/or what steps would be involved to do this?

Would be crazy useful to have the same basic interface with some different options for video.

Could just call whatever version of ffmpeg is on the host to Keep It Simple. 🤔

Thanks and keep up the amazing work.

Crop params in percent size

Hi,
Thanks for this tool. It looks amazing !

My use case is to crop images by giving left-top and right-bottom coords as percent of width and height of the image.
So the params would be float values between 0 and 1, such as "0.10x0.08:0.26x0.43".

Of course, I don't have image width and height infos on the client side, otherwise it would have been easy to compute pixels-coords automatically.

Is there any way to achieve this with the current version of Imagor ? If not, would you be open to a PR ?

best way to handle large gifs?

anyone have recommendations on how to handle large gifs. I find the processing time slow and i do notice that the resize makes the gifs have empty frames at the end...

Question about max_bytes

I was just trying the new max_bytes filter and I found the following that the following URL gives http 406 status
http://localhost:8000/unsafe/fit-in/2000x0/filters:max_bytes(270000):format(webp)/https%3A%2F%2Fstatic.travelconline.com%2Fheaderimages%2FuRzH8mtvVvUQ-zwjpegjpegjpeg.jpeg

I have to use a bigger max_bytes until it fits. then I get a 200 status. I suspect that this filter does not work with jpeg images, is this right?

Thanks!

Saved images are not treated

I am using version 0.2.4
When saving images, they are identical to the original and do not bear the treatments applied on the fly.

Imagor stops working after a few days

We've been using imagor in production . After some days (undetermined, maybe 5 or maybe 2) it stops giving 408 status for any image that is not in the cache or result cache (cached images work fine). I haven't seen any other warnings or errors. It starts working normally after a container restart.

Process memory was at about 380M before last restart, I don't think memory is the problem here. System load also looks fine. I've also checked that the images do load from that host.

Where can I look at? I can't enable debug since it's production and it happens after many hours of use

This is the docker-compose conf (it was working with 0.8.13 since yesterday):

version: '3'
services:
  imagor:
    container_name: imagor
    image: shumc/imagor:0.8.15
    restart: unless-stopped
    environment:
      IMAGOR_UNSAFE: 1
      VIPS_MAX_WIDTH: 2500
      VIPS_MAX_HEIGHT: 2500
      VIPS_MAX_ANIMATION_FRAMES: 1
      VIPS_CONCURRENCY: 12
      IMAGOR_PROCESS_CONCURRENCY: 12
      IMAGOR_REQUEST_TIMEOUT: 15s
      IMAGOR_LOAD_TIMEOUT: 12s
      IMAGOR_SAVE_TIMEOUT: 8s
      IMAGOR_PROCESS_TIMEOUT: 8s
      FILE_STORAGE_BASE_DIR: /data/storage
      FILE_STORAGE_EXPIRATION: 240h
      FILE_RESULT_STORAGE_BASE_DIR: /data/result-storage
      FILE_RESULT_STORAGE_EXPIRATION: 240h
      SERVER_ACCESS_LOG: 1
    volumes:
      - /imagor:/data
    ports:
      - 8088:8000
    logging:
      options:
        max-size: 10m

Here is the log:

{"level":"info","ts":1648990752.961404,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/480x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2Fe5aba067_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.001026158}
{"level":"info","ts":1648990752.97457,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/480x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2F12057946_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000918045}
{"level":"info","ts":1648990752.9747136,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/480x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2F2fbcadc9_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000146382}
{"level":"info","ts":1648990752.9756482,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2Fa36005b9_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.00070613}
{"level":"info","ts":1648990752.9778147,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2F0a85c357_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000178921}
{"level":"info","ts":1648990752.9778247,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/480x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2Ffab73574_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.001116044}
{"level":"info","ts":1648990752.9778512,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2F8d3795f7_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000416675}
{"level":"info","ts":1648990752.9778626,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2Fae5027ef_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000943989}
{"level":"info","ts":1648990752.977896,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2F56f5e24e_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.00055589}
{"level":"info","ts":1648990752.9778981,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2F0f292ddf_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.001085486}
{"level":"info","ts":1648990752.977921,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2F2762b899_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000723866}
{"level":"info","ts":1648990752.9779384,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2F624a2897_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000044393}
{"level":"info","ts":1648990752.979009,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2F6b5d6976_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000531184}
{"level":"info","ts":1648990752.979026,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2Fc9b02d54_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000618871}
{"level":"info","ts":1648990752.979019,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2F37552625_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000116353}
{"level":"info","ts":1648990752.9790409,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2Fw3839h2554x0y0-5fb828c1_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000258686}
{"level":"info","ts":1648990752.98013,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2F6e80c4d3_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000358459}
{"level":"info","ts":1648990752.9801385,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2F0813f02a_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000672276}
{"level":"info","ts":1648990752.980162,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2Fd956af88_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000455944}
{"level":"info","ts":1648990752.981225,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2F67b1479a_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000981561}
{"level":"info","ts":1648990752.9812527,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2F3fc9e583_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000999404}
{"level":"info","ts":1648990752.9813356,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2F75496e68_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000046782}
{"level":"info","ts":1648990752.9813657,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2F3ebf1254_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000071378}
{"level":"info","ts":1648990752.9813797,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2F56af8140_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000087917}
{"level":"info","ts":1648990752.9823616,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2Fa9027343_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000345702}
{"level":"info","ts":1648990752.9823604,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2Ff06c20f0_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000557614}
{"level":"info","ts":1648990752.982379,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2F2fbcadc9_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000870566}
{"level":"info","ts":1648990752.9823644,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2F025f3466_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000773003}
{"level":"info","ts":1648990752.9824026,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2Faa73ad0f_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000748786}
{"level":"info","ts":1648990752.984616,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2Fa7a4a11c_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000802961}
{"level":"info","ts":1648990752.9867806,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2Fb85c17b0_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.00060075}
{"level":"info","ts":1648990752.9909296,"caller":"server/handler.go:98","msg":"access","status":408,"method":"GET","uri":"/unsafe/fit-in/2000x0/filters:quality(75)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F2000000%2F1130000%2F1121700%2F1121636%2F24077b87_z.jpg","ip":"37.133.66.93","user-agent":"Mozilla/5.0 (iPad; CPU OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/99.0.4844.59 Mobile/15E148 Safari/604.1","took":15.000794574}

S3 paths with url encoding not working

I am currently testing locally imagor as a drop in replacement for our Thumbor implementation. We currently use an S3 loader and we're trying to emulate that with our local testing. However, we are getting errors when trying to implement it. I believe it has to do with url encoding special characters in the file path.

Here is the url we are using:

http://localhost:8888/unsafe/400x200/smart/photos/j/006b8bbe2c1baa8f2d12c06988a6efa7a15f048d.jpg

Here is the result:

{"message":"Get \"https://photos/j/006b8bbe2c1baa8f2d12c06988a6efa7a15f048d.jpg\": dial tcp: lookup photos on 127.0.0.11:53: no such host","status":500}

If I url encode the path of the photo, then it works:

photos%2Fj%2F006b8bbe2c1baa8f2d12c06988a6efa7a15f048d.jpg

But if the url encoded path has special characters in the filename, such as { to %7B, it gets an error:

photos%2F{1089856636549d026d369fb}.jpg

photos%2F%7B1089856636549d026d369fb%7D.jpg

{"message":"Get \"https://photos/%7B1089856636549d026d369fb%7D.jpg\": dial tcp: lookup photos on 127.0.0.11:53: no such host","status":500}

I'm assuming this is an issue because when querying s3 for the file, imagor is passing the url-encoded string instead of the actual filename. imagor should probably url-decode the string before making the request to s3. Thoughts?

animated gif support?

Hi again :)

Just wondering if there are any plans to support animated gifs? It seems like imagor returns a still image for gifs...

Typical memory usage for docker instance

We are running a single docker container with s3 loader and storage and memory usage hangs out around 2 Gib (last I checked 1.819GiB of total 7.667GiB). And it seems the memory grows and is not freed when an image is not found in the cache. Is that a reasonable/expected memory usage?

http-loader does not forward http host header despite `-http-loader-forward-all-headers`

We want imagor to fetch images via internal network to make sure to avoid cdn optimizations and really load the original image.

The upstream server has multiple vhosts configured to separate images for different domains:

10.1.1.1
vhost img.example.com -> /var/www/img.example.com/
vhost img.another.net -> /var/www/img.antoher.net/

We enabled -http-loader-forward-all-headers and tried requests like this:

curl -H "Host: img.another.net" -H "Foo: bar" http://<imagor_ip>/unsafe/fit-in/800x600/http://10.1.1.1/pic.jpg

This will produce the following request on the upstream server (which we tested with netcat -ltp80)

GET /pic.jpg HTTP/1.1
Host: 10.1.1.1
User-Agent: curl/7.58.0
Accept: */*
Foo: bar

As you can see the Foo-Header will be passed along due to -http-loader-forward-all-headers but the Host-Header is not.

This can be fixed with a /etc/hosts entry for each domain:

10.1.1.1 img.example.com
10.1.1.1 img.another.net

But i think it's at least misleading that -http-loader-forward-all-headers does not really forward all headers.

docker compose with minio

Any chance that this can be added to the repo ?
https://github.com/nhost/hasura-storage/tree/main/build/dev/docker

The docker compose uses a docker out of each folder.

The root make file does all the hard work..

Rotate filter applied too late compared to other image manipulation

I tried to use the rotate filter with a crop.
With a value of 180 degree, it works perfectly, but with a 90 degree rotation I obtained a squared image.

As you can see, rotation is correctly applied, but the crop with final dimensions is already applied, which results in generating a square.
It seems rotation is applied too late.

Here is the original image I used : http://coyote.vtt.free.fr/vw/MireCouleur16-9.jpg

180 degrees rotation:

http://localhost:8000/unsafe/125x772/filters:rotate(180):fill(white)/http://coyote.vtt.free.fr/vw/MireCouleur16-9.jpg

here is the result:

90 degrees rotation:

http://localhost:8000/unsafe/125x772/filters:rotate(90):fill(white)/http://coyote.vtt.free.fr/vw/MireCouleur16-9.jpg

here is the result:

Image files that have a name containing "+" are not treated

With the same setting as #2 , I have tried to treat images from the map folder.
They are named according to another scheme, for example: map/air-lanes-jfk-lga-2018-5b+6DVXbdWRw86PJv+0.png

When I try to treat the above image I get an error:

the URL: http://mdi.lan.ourquietplaces.com:7123/unsafe/fit-in/-500x500/10x10/filters:hue(290):saturation(100):fill(yellow):format(jpeg):quality(80)/map/air-lanes-jfk-lga-2018-5b+6DVXbdWRw86PJv+0.png
the error I get: {"Op":"Get","URL":"https://'map/air-lanes-jfk-lga-2018-5b%206DVXbdWRw86PJv%200.png%27","Err":{"Op":"dial","Net":"tcp","Source":null,"Addr":null,"Err":{"Err":"no such host","Name":"'map","Server":"","IsTimeout":false,"IsTemporary":false,"IsNotFound":true}}}

signature mismatch if `imagor-signer-truncate` not between 26 and 28

Version: 0.9.10

It appears that the path parser only detects hashes with a length between 26 and 30 characters as seen below:

imagor/imagorpath/parse.go

Line 15 in 5dff3f6

"((unsafe/)|([A-Za-z0-9-_=]{26,30})/)?" +

Further, it checks if that segment is less or equal to 28 characters:

imagor/imagorpath/parse.go

Lines 67 to 71 in 5dff3f6

 if match[index+1] == "unsafe/" { 

 p.Unsafe = true 

 } else if len(match[index+2]) <= 28 { 

 p.Hash = match[index+2] 

 }

Is this intended?

The example here sets imagor-signer-truncate to 32, but that will not work with the behavior above.

Any advice on using imagor with lambda

Hi @cshum

Would you be open to accepting PRs that add lambda support?

Have you perhaps done any experimentation with imagor in a AWS lambda environment? It'd probably involve integrating https://github.com/aws/aws-lambda-go support. https://aws.amazon.com/solutions/implementations/serverless-image-handler/ / https://github.com/aws-solutions/serverless-image-handler also uses libvips, so it should be reasonably possible to do so. That tool, however, doesn't support smart resizing without additional AWS services.

Additionally:

One of our use cases is that we serve images at common breakpoints. We've found that pre-caching images at the breakpoints improves end-user performance. Given this, I was wondering if you'd be open to accepting PRs that also change the operating model a bit? In short: I'd like a lambda that accepts an SQS request in Thumbor parameter format, fetches the origin image, resizes it, and then stores the result in an S3 result cache. This way, when a web browser makes a request for the same file with the same breakpoint it is already pre-built and can be returned by the http version of imagor. Or, if it's a miss, it'll perform the resize.

Thanks!

Oskar

Create Heroku support

Hi
I love your contribution, and it would be great if this tool could be easy to host on Heroku.
Thank you

Comparison with Imgproxy

How does this compare to imgproxy ?

Compatibility with Google Cloud Storage

Hi, just found this library and it looks like a great alternative to imaginary.

Just wondering if I'm able to swap out S3 storage with Google Cloud Storage.

From the docs, I'm guessing it's not possible right now because you're putting just the bucket name

possible optimisation

Hello, one of the libvips maintainers here, congrats on this nice project!

I had a very quick look through the code (perhaps I missed it) but I'm not sure you're using Thumbnail on a non-image source, and this means (I think) you're not exploiting shrink-on-load. This can give very nice performance improvements, especially with JPEG images.

For example, theo.jpg is a 6k x 4k jpg image. If I resize with vips_resize() I see:

$ /usr/bin/time -f %M:%e vips resize theo.jpg x.jpg 0.033
88048:0.30

So 90MB of memory and 300ms, but if I resize with vips_thumbnail() it's:

$ /usr/bin/time -f %M:%e vips thumbnail theo.jpg x.jpg 200
40304:0.06

40MB and 60ms -- five times faster and half the memory use.

The vips_thumbnail() and vips_thumbnail_buffer() operations resize from a file, or from a memory area containing the encoded image, combining open and resize in one operation. Because open knows the final desired size, it can exploit tricks in many image format libraries (jpg, webp, jp2k, tiff, svg, pdf. openslide, etc.). It's well worth reorganising your code to hit these fast paths if you can.

I also noticed you're not premultiplying before resize (again, perhaps I missed it). You'll probably see haloing in transparent elements in PNG images. The fix is to premultiply / resize / unpremultiply in this case. Again, the thumbnail operations will handle this for you.

watermark position

followup from #16

http://localhost:8000/unsafe/fit-in/1200x800/filters:fill(yellow):watermark(https://i.imgur.com/71LaQJU.png,-10,-10,0):format(jpeg)/upload.wikimedia.org/wikipedia/commons/a/a9/St_Francis_Seminary.jpg

using thumbor:

using imagor:

/healthcheck returns 403

Not sure if I'm doing something wrong here or if this is a bug:

$ docker run --rm --detach -p 8888:8888 shumc/imagor:0.8.11 -imagor-unsafe -port=8888
0a30471e0f1175e2cee52665651bda2d709321b77cdce192c7f3dda5d5112456
$ curl -I localhost:8888/healthcheck
HTTP/1.1 403 Forbidden
Date: Tue, 08 Mar 2022 18:58:28 GMT
Content-Length: 49
Content-Type: text/plain; charset=utf-8

How to get image metadata？

Disable default caching

Hi, I tried using this as a docker image but it has problem with caching.

I don't want the cache header because of CDN caching, but I could not disable it.

I'm using docker-compose with these variables but still getting cache header with default values
Cache-Control: public, s-maxage=86400, max-age=86400, no-transform

IMAGOR_CACHE_HEADER_NO_CACHE: 1
IMAGOR_CACHE_HEADER_TTL: 0h0m0s
IMAGOR_CACHE_HEADER_TTL: 0h0m0s

Please disable the cache for default.

Results not saved (imagor: 400 pass)

Hi,

I'm trying a basic setup to replace thumbor but I can't manage to make it save any results.
The configuration seems ok:

        - name: PORT
          value: "8080"
        - name: DEBUG
          value: "1"
        - name: IMAGOR_UNSAFE
          value: "0"
        - name: IMAGOR_AUTO_WEBP
          value: "1"
        - name: IMAGOR_SECRET
          value: "hzieQh8AJkwKaOx2Oj4ywYWYt9ScoS2mlM6sjLrpETVU5cdYNVXA"
        - name: FILE_STORAGE_BASE_DIR
          value: "/data/store"
        - name: FILE_STORAGE_PATH_PREFIX
          value: 'store'
        - name: FILE_RESULT_STORAGE_BASE_DIR
          value: "/data/store"
        - name: FILE_RESULT_STORAGE_PATH_PREFIX
          value: 'results'

But in the logs I see:

2022-04-13T10:45:01.424Z    WARN    imagor/imagor.go:355    save    {"key": "fit-in/1280x0/filters:format(webp)/https://some-domain.com/uploads/2022/04/some-file.png", "error": "imagor: 400 pass"}

Looking in the code, this error can happen (if I understand correctly) if the path doesn't begin with / or if it matches /.. The initial / is prepended by the file storage and I don't see any /. anywhere so I'm a bit stumped.

fill filter compatibility with thumbor?

Excited to see imagor ! We're pretty heavy users of thumbor, and I also maintain the docker versions of thumbor (+ caching layer) at https://github.com/MinimalCompact/thumbor/

I took imagor for a quick spin as a drop-in replacement, and it seems to work quite well out of the box. One thing I already found however. It seems like the fill filter isn't entirely compatible with thumbor?

We have a URL that looks ilke this /{hash}/fit-in/205x205/filters:fill(FFFFFF,true):format(jpeg)/path/to/image.png. With thumbor, the background is white, whereas it is black with imagor

Would we need to change the fill params? or are you guys planning on making it compatible with thumbor?

Security around image bombs

Thank you for your amazing work on this project! Was wondering if there's anything addressing or designing around mallicious images? Imgproxy mentions it here: https://github.com/imgproxy/imgproxy#security

imgproxy checks the image type and its “real” dimensions when downloading. The image will not be fully downloaded if it has an unknown format or if the dimensions are too big (you can set the max allowed dimensions). This is how imgproxy protects from so called "image bombs”

Thank you

Clarification for different storage buckets

I don't quite understand what the difference between result-loader and result-storage would be? Can this be the same bucket?
The same applies to loader and storage. Is this to guard against deletion in the loader bucket?

In my simple mind the server would load something and then serve a result (optionally storing it). Thus I don't see where exactly the other two buckets come into play.

Thank you for your great work

local minio server not support https protocol

{"level":"warn","ts":1647773368.5157983,"caller":"imagor/imagor.go:331","msg":"save","key":"fit-in/300x290/http://minio:9000/example-bucket/xxxximgpath.png","error":"RequestError: send request failed\ncaused by: Put \"https://minio:9000/example-bucket/path/rets/fit-in/300x290/http%253A/minio%253A9000/example-bucket/xxxximgpath.png\": http: server gave HTTP response to HTTPS client"}

406 VipsJpeg: Premature end of input file

Hi, I recently upgraded the imagor to v1.0.0 and I saw this error in docker container log many times for different images:

{
    "level": "warn",
    "ts": 1660161069.1208205,
    "caller": "imagor/imagor.go:326",
    "msg": "process",
    "params": {
        "path": "fit-in/1280x720/filters:format(webp)/xxxx",
        "image": "xxxx",
        "hash": "xxxxx",
        "fit_in": true,
        "width": 1280,
        "height": 720,
        "filters": [
            {
                "name": "format",
                "args": "webp"
            }
        ]
    },
    "error": "imagor: 406 VipsJpeg: Premature end of input file"
}

	if match[index+1] == "unsafe/" {
	p.Unsafe = true
	} else if len(match[index+2]) <= 28 {
	p.Hash = match[index+2]
	}