webp-sh / webp_server_go Goto Github PK

If I set nginx like this

location ^~ /wp-content/uploads/ {
        proxy_pass http://127.0.0.1:3333;
}

And webp-server doesn't support gif.

Every time when I access some gif file, it would send File extension not allowed! message, and the pic is broken on the webpage.

I think we should not only return the warning message but also return the unsupported raw image.

Use nginx to filter some kinds of file is a workaround, but that means enduser need to set two configs. So I'm not in flavor of this.

By the way, the default support for gif should be temporarily remove due to limitations of webp-sh/webp

一定得是根目录吗?如果图片目录下面有很多目录是否可以呢?

After the transparent PNG format file is converted, can it continue to be transparent?

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Add support for unit test and integration test, includes

• WebpEncoder: encoder test from all the five samples we have except for the gif one.
• GetFileContentType、GenWebpAbs、CheckUA、ImageExists
• PrefetchImages
• Convert: basic function
• autoUpdate

I can run webp-server --config /opt/webps/config.json successfully

But I run systemctl start webps.service and systemctl status webps.service

Process: 29802 ExecStart=/opt/webps/webp-server --config /opt/webps/config.json (code=exited, status=218/CAPABILITIES)

........
3 10 22:27:07 MyCloudServer systemd[1]: webps.service: main process exited, code=exited, status=218/CAPABILITIES
3 10 22:27:07 MyCloudServer systemd[1]: Unit webps.service entered failed state.
3 10 22:27:07 MyCloudServer systemd[1]: webps.service failed.

........

why?

ios device display webp error
device:
iphone se ios14.2

webp_server_go version:0.3.0

request:
Accept: image/webp,image/png,image/svg+xml,image/*;q=0.8,video/*;q=0.8,*/*;q=0.5
User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 14_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.1 Mobile/15E148 Safari/604.1
Accept-Language: zh-cn
Accept-Encoding: gzip, deflate
Connection: keep-alive

response:
Response Code  404 Not Found

hello,Webp server is a great tool...

and it would be much more fine if we can use docker image on armv7/armv8/x86 by docker cli.......

hope it could be searched on hub.docker.com......

Just a suggestion... as i see it, currently, some (if not all, i have not checked in depth) image files are set with the executable bit... I believe it would be best to get red of it, dno't you think?

I found a directory traversal vulnerability in the system. Can traverse the files on the server.

Solution:
Restrict special symbols in the path, such as ../

Example link here

FROM golang:alpine as builder

ENV HOME=/
RUN apk update && apk add --no-cache make ca-certificates tzdata && mkdir /build
COPY go.mod /build
RUN cd /build && go mod download
COPY . /build
RUN cd /build &&make static


FROM scratch

ENV TZ=Asia/Shanghai

COPY --from=builder /build/Gakki /Gakki
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /usr/share/zoneinfo /usr/share/zoneinfo
WORKDIR /

ENTRYPOINT ["/Gakki"]

Hello, finally i can run the webp-server but i when i run it, i'll get this error:
[16:main.autoUpdate.func1()] Download error: runtime error: invalid memory address or nil pointer dereference

Can I convert WebP to JPG?

In some scenarios users may want to pre-convert their images to the webp format (just like cache-prefetch), this can increase load speed in production use.

I suggest adding a parameter in config.json like "prefetch":true.

Describe the bug
Some JPEG images are converted (exists in the /opt/exhaust directory), but the image/jpeg version is used in browser. But on the same page some JPEG images are fine.

To Reproduce
(unknown)

Expected behavior
All supported images (JPEG, PNG, etc) are converted and shown in image/webp

Screenshots and logs
Some images are in WEBP, but the selected one is in JPEG:

The file is converted indeed:

Original JPEG version:

$    
identify -verbose Factory-TOTAL-2020-RIGOLLOT-VINCENT-1.jpg 
Image:
  Filename: Factory-TOTAL-2020-RIGOLLOT-VINCENT-1.jpg
  Format: JPEG (Joint Photographic Experts Group JFIF format)
  Mime type: image/jpeg
  Class: DirectClass
  Geometry: 2000x1333+0+0
  Resolution: 300x300
  Print size: 6.66667x4.44333
  Units: PixelsPerInch
  Colorspace: sRGB
  Type: TrueColor
  Base type: Undefined
  Endianness: Undefined
  Depth: 8-bit
  Channel depth:
    red: 8-bit
    green: 8-bit
    blue: 8-bit
  Channel statistics:
    Pixels: 2666000
    Red:
      min: 0  (0)
      max: 255 (1)
      mean: 143.982 (0.564636)
      standard deviation: 91.1945 (0.357626)
      kurtosis: -1.55821
      skewness: -0.323965
      entropy: 0.927611
    Green:
      min: 0  (0)
      max: 255 (1)
      mean: 143.743 (0.5637)
      standard deviation: 83.3982 (0.327052)
      kurtosis: -1.52817
      skewness: -0.19354
      entropy: 0.952896
    Blue:
      min: 0  (0)
      max: 255 (1)
      mean: 143.162 (0.561419)
      standard deviation: 80.4569 (0.315517)
      kurtosis: -1.34815
      skewness: -0.185536
      entropy: 0.954943
  Image statistics:
    Overall:
      min: 0  (0)
      max: 255 (1)
      mean: 143.629 (0.563252)
      standard deviation: 85.0166 (0.333398)
      kurtosis: -1.47776
      skewness: -0.244404
      entropy: 0.94515
  Rendering intent: Perceptual
  Gamma: 0.454545
  Chromaticity:
    red primary: (0.64,0.33)
    green primary: (0.3,0.6)
    blue primary: (0.15,0.06)
    white point: (0.3127,0.329)
  Background color: white
  Border color: srgb(223,223,223)
  Matte color: grey74
  Transparent color: black
  Interlace: None
  Intensity: Undefined
  Compose: Over
  Page geometry: 2000x1333+0+0
  Dispose: Undefined
  Iterations: 0
  Compression: JPEG
  Quality: 78
  Orientation: Undefined
  Profiles:
    Profile-8bim: 236 bytes
    Profile-exif: 916 bytes
    Profile-icc: 3144 bytes
    Profile-iptc: 154 bytes
      City[1,90]: 0x00000000: 254700                                        -%G
      unknown[2,0]: 
      Keyword[2,25]: Ecran
      Keyword[2,25]: TDF
      Keyword[2,25]: Total
      Keyword[2,25]: accueil
      Keyword[2,25]: projet terminé
      Created Date[2,55]: 20210406
      Created Time[2,60]: 110529+0100
      unknown[2,62]: 20210406
      unknown[2,63]: 110529+0100
      Copyright String[2,116]: Vincent Rigollot
    Profile-xmp: 14140 bytes
  Properties:
    date:create: 2022-07-27T02:20:52+00:00
    date:modify: 2022-07-27T02:20:49+00:00
    exif:ApertureValue: 6/1
    exif:BrightnessValue: 5274/2560
    exif:ColorSpace: 1
    exif:Contrast: 0
    exif:Copyright: Vincent Rigollot
    exif:CustomRendered: 0
    exif:DateTime: 2021:08:19 16:29:12
    exif:DateTimeDigitized: 2021:04:06 11:05:29
    exif:DateTimeOriginal: 2021:04:06 11:05:29
    exif:DigitalZoomRatio: 16/16
    exif:ExifOffset: 244
    exif:ExifVersion: 48, 50, 51, 49
    exif:ExposureBiasValue: 0/10
    exif:ExposureMode: 1
    exif:ExposureProgram: 1
    exif:ExposureTime: 1/10
    exif:FileSource: 3
    exif:Flash: 16
    exif:FNumber: 8/1
    exif:FocalLength: 220/10
    exif:FocalLengthIn35mmFilm: 22
    exif:FocalPlaneResolutionUnit: 3
    exif:FocalPlaneXResolution: 54965632/32768
    exif:FocalPlaneYResolution: 54965632/32768
    exif:LensModel: E 17-28mm F2.8-2.8
    exif:LensSpecification: 170/10, 280/10, 28/10, 28/10
    exif:LightSource: 0
    exif:Make: SONY
    exif:MaxApertureValue: 760/256
    exif:MeteringMode: 5
    exif:Model: ILCE-7M3
    exif:OffsetTime: +02:00
    exif:OffsetTimeDigitized: +01:00
    exif:OffsetTimeOriginal: +01:00
    exif:PhotographicSensitivity: 500
    exif:RecommendedExposureIndex: 500
    exif:Saturation: 0
    exif:SceneCaptureType: 0
    exif:SceneType: 1
    exif:SensitivityType: 2
    exif:Sharpness: 0
    exif:ShutterSpeedValue: 3321928/1000000
    exif:Software: Adobe Photoshop Lightroom Classic 10.3 (Macintosh)
    exif:WhiteBalance: 0
    icc:copyright: Copyright (c) 1998 Hewlett-Packard Company
    icc:description: sRGB IEC61966-2.1
    icc:manufacturer: IEC http://www.iec.ch
    icc:model: IEC 61966-2.1 Default RGB colour space - sRGB
    jpeg:colorspace: 2
    jpeg:sampling-factor: 2x2,1x1,1x1
    signature: 2cd11f244ed9d6a9d609517d9942594fe667eccecef9000bec228b500956dd43
  Artifacts:
    filename: Factory-TOTAL-2020-RIGOLLOT-VINCENT-1.jpg
    verbose: true
  Tainted: False
  Filesize: 166027B
  Number pixels: 2666000
  Pixels per second: 174.281MB
  User time: 0.010u
  Elapsed time: 0:01.015
  Version: ImageMagick 6.9.11-60 Q16 x86_64 2021-01-25 https://imagemagick.org

Converted WEBP version:

$ identify -verbose Factory-TOTAL-2020-RIGOLLOT-VINCENT-1.jpg.1629383389.webp 
Image:
  Filename: Factory-TOTAL-2020-RIGOLLOT-VINCENT-1.jpg.1629383389.webp
  Format: WEBP (WebP Image Format)
  Mime type: image/webp
  Class: DirectClass
  Geometry: 2000x1333+0+0
  Units: Undefined
  Colorspace: sRGB
  Type: TrueColor
  Base type: Undefined
  Endianness: Undefined
  Depth: 8-bit
  Channel depth:
    red: 8-bit
    green: 8-bit
    blue: 8-bit
  Channel statistics:
    Pixels: 2666000
    Red:
      min: 0  (0)
      max: 255 (1)
      mean: 144.195 (0.565471)
      standard deviation: 91.3854 (0.358374)
      kurtosis: -1.56604
      skewness: -0.321738
      entropy: 0.941662
    Green:
      min: 0  (0)
      max: 255 (1)
      mean: 143.897 (0.564302)
      standard deviation: 83.587 (0.327792)
      kurtosis: -1.53354
      skewness: -0.194684
      entropy: 0.96099
    Blue:
      min: 0  (0)
      max: 255 (1)
      mean: 143.266 (0.561826)
      standard deviation: 80.3674 (0.315166)
      kurtosis: -1.35531
      skewness: -0.179986
      entropy: 0.968523
  Image statistics:
    Overall:
      min: 0  (0)
      max: 255 (1)
      mean: 143.786 (0.563866)
      standard deviation: 85.1133 (0.333778)
      kurtosis: -1.48481
      skewness: -0.242287
      entropy: 0.957058
  Rendering intent: Perceptual
  Gamma: 0.454545
  Chromaticity:
    red primary: (0.64,0.33)
    green primary: (0.3,0.6)
    blue primary: (0.15,0.06)
    white point: (0.3127,0.329)
  Background color: white
  Border color: srgb(223,223,223)
  Matte color: grey74
  Transparent color: black
  Interlace: None
  Intensity: Undefined
  Compose: Over
  Page geometry: 2000x1333+0+0
  Dispose: Undefined
  Iterations: 0
  Compression: Undefined
  Orientation: Undefined
  Properties:
    date:create: 2022-07-27T02:32:17+00:00
    date:modify: 2022-07-27T02:32:17+00:00
    signature: a2e148f7272a59383823b85de232f5e4c01535ff6faec45b10504b40de55a8c7
  Artifacts:
    filename: Factory-TOTAL-2020-RIGOLLOT-VINCENT-1.jpg.1629383389.webp
    verbose: true
  Tainted: False
  Filesize: 242796B
  Number pixels: 2666000
  Pixels per second: 73.4086MB
  User time: 0.030u
  Elapsed time: 0:01.036
  Version: ImageMagick 6.9.11-60 Q16 x86_64 2021-01-25 https://imagemagick.org

Environment (please complete the following information):

• OS: Ubuntu 20.04.4 LTS
• version or branch: Docker image webpsh/webp-server-go latest 920f8293ead8 5 weeks ago 35MB

Additional context
Tested on Ubuntu 22.04, Chrome 103.
Website is served via Cloudflare. Development mode enabled. Cache flushed.

example

the link is return state 404 "Image not found!"

i wanna return state 200 and a default pic.

CentOS Linux release 7.1.1503 (Core)
Linux localhost.localdomain 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
webp-server0.4.3

[root@localhost webps]# ./webp-server -dump-config > config.json
./webp-server: error while loading shared libraries: libaom.so.0: cannot open shared object file: No such file or directory

run yum install libaom libaom-devel, sucess
but this problem has always existed

The maximum pixel dimensions of a WebP image is 16383 x 16383, but many long images will exceed this limit.
The WebP server will return an error and keep taking lots of memory when the length of the original image is over 16383px.
Could we check the dimension of the image before converting or just return the original image when encountering a conversion error?

Logs:

time="2021-12-04 00:44:53" level=error msg="webpEncodeRGB: failed" func="[main.webpEncoder()]"
time="2021-12-04 00:44:53" level=error msg="webpEncodeRGB: failed" func="[main.convert()]"
00:44:53 | 500 |  1.687s |       127.0.0.1 | GET     | /for/example.jpg  | webpEncodeRGB: failed

The second problem I found is that the conversion will return an error if the image does not end correctly.
As we know, any JPEG file must start with the SOI (Start-of-Image) marker (the two bytes 0xFF, 0xD8) and end with the EOI (End-of-Image) marker (two bytes, 0xFF, 0xD9). Due to buggy software or network problems, the EOI marker (0xFF, 0xD9) can be accidentally lost. Modern browsers and picture software will ignore this error for compatibility, which also causes users to continue using faulty images without realizing it, and then have trouble using this program.
I wonder if it is possible to return the original images when whatever converting errors then we could fix this problem and maybe more trifle in the future. This problem is not a bug of this program but it could help some people if we deal with it.

Logs:

time="2021-12-04 03:19:12" level=error msg="image file example.jpg is corrupted or not supported" func="[main.convert()]"
03:19:11 | 500 |    61ms |       127.0.0.1 | GET     | /for/example.jpg  | image file example.jpg is corrupted or not supported

Here are some samples:
img_over_16383px (https://user-images.githubusercontent.com/45689550/144671266-cc929535-ec15-4001-b14c-3169e77de669.jpg)
jpg_without_eoi (https://user-images.githubusercontent.com/45689550/144673094-9e315731-e996-411f-aa48-dd47b6356c8f.jpg)

Environment:
0.4.5 with remote backend.

Describe the bug
These two images cause errors and do not return content when using the remote backend.

webp-server[1949]: time="2022-08-02 03:33:51" level=error msg="image file /www/invalid_checksum.png is corrupted: png: invalid format: invalid checksum" func="[86:main.readRawImage()]"

webp-server[1949]: time="2022-08-02 03:36:52" level=error msg="image file /www/missing_0xff00.jpg is corrupted: invalid JPEG format: missing 0xff00 sequence" func="[86:main.readRawImage()]"

invalid_checksum ( https://user-images.githubusercontent.com/45689550/182232953-58f76dde-9fe6-46a1-b669-f4087527c3ff.png )
missing_0xff00 ( https://user-images.githubusercontent.com/45689550/182232958-d85044b1-a9b9-426e-a8af-953e99996c6a.jpg )

project layout

config.json file content

{ "HOST": "127.0.0.1", "PORT": "3333", "QUALITY": "80", "IMG_PATH": "/mnt/disk/macro", "EXHAUST_PATH": "/mnt/www/img", "ALLOWED_TYPES": ["jpg","png","jpeg","bmp"] }

About Error
First:

Next: I make a right config.json and try to run it.

To Reproduce
I cann't use Releases(0.4.1 & 0.4.0) , but i can use Releases(0.3.2).

Environment (please complete the following information):
CentOS Linux release 8.5.2111

What should i do? Thank you~

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots and logs
If applicable, add screenshots and logs to help explain your problem.

Environment (please complete the following information):

• OS: [e.g. Linux]
• version or branch[e.g. 0.0.3, master]

Additional context
Add any other context about the problem here.

Describe the bug
If an image is requested that does not exists, webp server return status code 500 instead of 404 (Not Found).

To Reproduce
Request an image that does not exists.

Expected behavior
Return status code 404.

Screenshots and logs

 location ~* \.webp$ {
    rewrite ^(.*)\.webp$ $1 break;
    proxy_pass http://localhost:20090; # webp server is listening here
    proxy_set_header X-Real-IP $remote_addr;
    proxy_hide_header X-Powered-By;
    proxy_set_header HOST $http_host;
    add_header Cache-Control 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0';
  }

Environment:

• OS: Debian 10/Docker
• v0.4.4
• Default Config

Additional context
I have checked the code and it's actually correct !?

I haven't seen any relevant content on the document yet.

1. when use remote backend services,Server consumes more bandwidth.Although webp is not dynamically generated for each request,but every request will visit the remote backend picture address.
2. It is recommended to judge whether the webp exists first, and there is no need to request the original picture if it exists, instead of requesting the original picture first, find the webp file according to etag.

Describe the bug
Popular forum softwares occasionally encode images as base64, in order for the image to be read properly we have to decode the base64 image then process it.

To Reproduce
process an image that is base64 encoded.

Additional context

time="2022-06-16 23:25:43" level=info msg="[email protected]%: /opt/pics/attachments/234/234533-048307689f64cf58c51749e3f99edbb9.jpg->exhaust/attachments/234/234533-048307689f64cf58c51749e3f99edbb9.jpg.1655331758.avif 8992->23862 265.37% deflated" func="[153:main.convertLog()]"
time="2022-06-16 23:25:43" level=info msg="[email protected]%: /opt/pics/attachments/234/234533-048307689f64cf58c51749e3f99edbb9.jpg->exhaust/attachments/234/234533-048307689f64cf58c51749e3f99edbb9.jpg.1655331758.webp 8992->15896 176.78% deflated" func="[153:main.convertLog()]"
23:25:43 | 200 |    24ms |      172.20.0.1 | GET     | /attachments/234/234533-048307689f64cf58c51749e3f99edbb9.jpg 
23:25:46 | 200 |      0s |      172.20.0.1 | GET     | /attachments/234/234590-1d24e2020ba4ab7db23779011292af4e.jpg 
time="2022-06-16 23:25:46" level=error msg="image file /opt/pics/attachments/234/234590-1d24e2020ba4ab7db23779011292af4e.jpg is corrupted: invalid JPEG format: missing SOI marker" func="[86:main.readRawImage()]"
time="2022-06-16 23:25:46" level=warning msg="exhaust/attachments/234/234590-1d24e2020ba4ab7db23779011292af4e.jpg.1655421235.webp not found on filesystem" func="[234:main.findSmallestFiles()]"
23:25:49 | 200 |     4ms |      172.20.0.1 | GET     | /avatars/l/45/45480.jpg 
time="2022-06-16 23:25:50" level=error msg="image file /opt/pics/avatars/m/45/45628.jpg is corrupted: invalid JPEG format: missing SOI marker" func="[86:main.readRawImage()]"
time="2022-06-16 23:25:50" level=error msg="image file /opt/pics/avatars/m/45/45628.jpg is corrupted: invalid JPEG format: missing SOI marker" func="[86:main.readRawImage()]"
time="2022-06-16 23:25:50" level=warning msg="exhaust/avatars/m/45/45628.jpg.1629355149.webp not found on filesystem" func="[234:main.findSmallestFiles()]"
23:25:50 | 200 |     1ms |      172.20.0.1 | GET     | /avatars/m/45/45628.jpg 
time="2022-06-16 23:25:53" level=error msg="image file /opt/pics/attachments/234/234590-1d24e2020ba4ab7db23779011292af4e.jpg is corrupted: invalid JPEG format: missing SOI marker" func="[86:main.readRawImage()]"
time="2022-06-16 23:25:53" level=warning msg="exhaust/attachments/234/234590-1d24e2020ba4ab7db23779011292af4e.jpg.1655421235.webp not found on filesystem" func="[234:main.findSmallestFiles()]"
23:25:53 | 200 |      0s |      172.20.0.1 | GET     | /attachments/234/234590-1d24e2020ba4ab7db23779011292af4e.jpg 
time="2022-06-16 23:25:53" level=error msg="image file /opt/pics/attachments/234/234595-a4484b29bd5d3a2e9554eda9ac9a8a74.jpg is corrupted: invalid JPEG format: missing SOI marker" func="[86:main.readRawImage()]"
time="2022-06-16 23:25:53" level=warning msg="exhaust/attachments/234/234595-a4484b29bd5d3a2e9554eda9ac9a8a74.jpg.1655421943.webp not found on filesystem" func="[234:main.findSmallestFiles()]"
23:25:53 | 200 |      0s |      172.20.0.1 | GET     | /attachments/234/234595-a4484b29bd5d3a2e9554eda9ac9a8a74.jpg 
time="2022-06-16 23:25:55" level=error msg="image file /opt/pics/resource_icons/2/2105.jpg is corrupted: invalid JPEG format: missing SOI marker" func="[86:main.readRawImage()]"
time="2022-06-16 23:25:55" level=error msg="image file /opt/pics/resource_icons/2/2105.jpg is corrupted: invalid JPEG format: missing SOI marker" func="[86:main.readRawImage()]"
time="2022-06-16 23:25:55" level=warning msg="exhaust/resource_icons/2/2105.jpg.1583166938.webp not found on filesystem" func="[234:main.findSmallestFiles()]"
23:25:55 | 200 |     3ms |      172.20.0.1 | GET     | /resource_icons/2/2105.jpg 

Describe the bug
WebP Server 0.2.0 :

tail -f www.output.log
time="2020-08-24 10:32:36" level=info msg="WebP Server 0.2.0 127.0.0.1:3102" func="[main.main()]"
time="2020-08-24 10:32:36" level=info msg="Prefetching using 2 cores" func="[main.PrefetchImages()]"
time="2020-08-24 10:32:37" level=error msg="Download error: runtime error: invalid memory address or nil pointer dereference" func="[main.autoUpdate.func1()]"
time="2020-08-24 10:32:38" level=warning msg="Gif support is not perfect!" func="[main.WebpEncoder()]"
time="2020-08-24 10:32:39" level=warning msg="Gif support is not perfect!" func="[main.WebpEncoder()]"
time="2020-08-24 10:32:46" level=warning msg="Gif support is not perfect!" func="[main.WebpEncoder()]"
time="2020-08-24 10:32:46" level=warning msg="Gif support is not perfect!" func="[main.WebpEncoder()]"
time="2020-08-24 10:32:46" level=warning msg="Gif support is not perfect!" func="[main.WebpEncoder()]"

To Reproduce
nohup ./webp-server-go -config ./config_go_www.json -prefetch >>www.output.log 2>&1 &

Expected behavior

Is this error message missing a configset of a disable upgrade download feature or a bug?

Screenshots and logs

" level=error msg="Download error: runtime error: invalid memory address or nil pointer dereference" func="[main.autoUpdate.func1()]"

Environment (please complete the following information):

• OS: [centos 7.6 ]
• version or branch[e.g. 0.2.0]

Since we want to add an verbose in param, I think we should get some dig into log level.

Basically there are these levels:

• debug
• info
• warning
• error
• critical

The question is, which default level do we want to put our program on? And if we turn on verbose, what level should it be?

In my humble opinion, I think that it should stays on info level, and downgrade to debug level if verbose is on.

To be more accurate, prefetch logs belong to debug, and access convert log is info.

BTW, there is logrus, I think we could use this and add more logs for easy debug purpose.

However, if we just wanna silence the warning while prefetching, a simple wrapper should be enough. After all, codes' should be short and easy to understand.

What do you say?

Describe the bug
I have Apache reverse-proxy to webp-server-go. When a path of a file-not-exist is requested, an error 500 is trigger. It seems the webp-server-go is restarted because of this.

i.e.

$ docker logs -f webp-server-go
logs...
logs...
error 500
$

$ docker ps
webp-server-go          Up x seconds

To Reproduce
Requesting a file location where file does not exist

Expected behavior
Return a error 404 (?)

Screenshots and logs

Environment (please complete the following information):

• OS: Ubuntu Server 22.04
• Branch: master

Describe the bug
No svg suffix is configured in the ALLOWED_TYPES parameter, but accessing svg images throws an exception File extension not allowed! xxx.svg

To Reproduce

Docker:

docker run -d --name halo.run.webps.server -p 3333:3333 -v ~/.halo.run:/opt/pics  --restart=always webpsh/webps 

Expected behavior
If the expected suffix is not configured in ALLOWED_TYPES, then images that access this suffix should return the original image.

Screenshots and logs

06:00:36 | 400 |      0s |      172.17.0.1 | GET     | /upload/2021/12/support-team-ba6262a355c045379723c5af671177d2.svg
time="2022-03-17 06:00:37" level=warning msg="File extension not allowed! support-team-ba6262a355c045379723c5af671177d2.svg" func="[34:main.convert()]"

Environment (please complete the following information):

• OS: Linux VM-0-8-centos 4.11.8-1.el7.elrepo.x86_64
• version or branch[e.g. 0.0.3, master]: 0.4.2

Describe the bug
Services running in docker containers are often stopped abnormally and an error is reported after viewing the log. fatal error: concurrent map read and map write

To Reproduce
Personal guesswork is associated with concurrency

Expected behavior
Services will no longer be abnormally shut down due to this problem

Screenshots and logs
fatal error: concurrent map read and map write
time="2022-01-11 15:04:32" level=warning msg="exhaust/upload/2022/01/image-ffc82761be954da7bb36b14548044614-thumbnail.png.1641562072.avif not found on filesystem" func="[230:main.findSmallestFiles()]"
time="2022-01-11 15:04:32" level=warning msg="exhaust/upload/2022/01/image-e9c118d4b35f4b6f86dbd03b6d5e2cc7-thumbnail.png.1641562035.avif not found on filesystem" func="[230:main.findSmallestFiles()]"

goroutine 12 [running]:
runtime.throw({0x8560a8, 0x0})
/usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc000181338 sp=0xc000181308 pc=0x43d691
runtime.mapaccess1(0xc0000d7900, 0xc000021d10, 0x1b00)
/usr/local/go/src/runtime/map.go:411 +0x1f7 fp=0xc000181378 sp=0xc000181338 pc=0x418797
github.com/h2non/filetype.Match({0xc0000d7900, 0x4, 0x1227})
/go/pkg/mod/github.com/h2non/[email protected]/match.go:28 +0x22c fp=0xc000181500 sp=0xc000181378 pc=0x70ccec
main.getFileContentType({0xc0000d7900, 0x1226, 0x1227})
/build/helper.go:38 +0x14e fp=0xc000181658 sp=0xc000181500 pc=0x74c4ae
main.convert(0xc000262000)
/build/router.go:91 +0x82f fp=0xc000181818 sp=0xc000181658 pc=0x74ee6f
github.com/gofiber/fiber/v2.(*App).next(0xc000124c60, 0xc000262000)
/go/pkg/mod/github.com/webp-sh/fiber/[email protected]/router.go:126 +0x1d8 fp=0xc0001818b8 sp=0xc000181818 pc=0x740b98
github.com/gofiber/fiber/v2.(*Ctx).Next(0x6da2e5)
/go/pkg/mod/github.com/webp-sh/fiber/[email protected]/ctx.go:637 +0x53 fp=0xc0001818d8 sp=0xc0001818b8 pc=0x73c953
github.com/gofiber/fiber/v2/middleware/logger.New.func2(0xc000262000)
/go/pkg/mod/github.com/webp-sh/fiber/[email protected]/middleware/logger/logger.go:154 +0x22b fp=0xc000181b70 sp=0xc0001818d8 pc=0x7478eb
github.com/gofiber/fiber/v2.(*App).next(0xc000124c60, 0xc000262000)
/go/pkg/mod/github.com/webp-sh/fiber/[email protected]/router.go:126 +0x1d8 fp=0xc000181c10 sp=0xc000181b70 pc=0x740b98
github.com/gofiber/fiber/v2.(*App).handler(0xc000124c60, 0x4b2377)
/go/pkg/mod/github.com/webp-sh/fiber/[email protected]/router.go:154 +0xe5 fp=0xc000181c48 sp=0xc000181c10 pc=0x740da5
github.com/gofiber/fiber/v2.(*App).handler-fm(0xc000126f00)
/go/pkg/mod/github.com/webp-sh/fiber/[email protected]/router.go:142 +0x2c fp=0xc000181c68 sp=0xc000181c48 pc=0x74462c
github.com/valyala/fasthttp.(*Server).serveConn(0xc000128b40, {0x8d5cd0, 0xc0000106e8})
/go/pkg/mod/github.com/valyala/[email protected]/server.go:2168 +0xf2b fp=0xc000181ec8 sp=0xc000181c68 pc=0x6f912b
github.com/valyala/fasthttp.(*Server).serveConn-fm({0x8d5cd0, 0xc0000106e8})
/go/pkg/mod/github.com/valyala/[email protected]/server.go:1959 +0x39 fp=0xc000181ef0 sp=0xc000181ec8 pc=0x7023b9
github.com/valyala/fasthttp.(*workerPool).workerFunc(0xc00011d360, 0xc000057840)
/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:223 +0xa9 fp=0xc000181fa0 sp=0xc000181ef0 pc=0x700669
github.com/valyala/fasthttp.(*workerPool).getCh.func1()
/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:195 +0x38 fp=0xc000181fe0 sp=0xc000181fa0 pc=0x7003f8
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000181fe8 sp=0xc000181fe0 pc=0x46cbe1
created by github.com/valyala/fasthttp.(*workerPool).getCh
/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:194 +0x1b5

goroutine 1 [IO wait]:
internal/poll.runtime_pollWait(0x7f7104f6aa48, 0x72)
/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00014eb00, 0xc00002a000, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc00014eb00)
/usr/local/go/src/internal/poll/fd_unix.go:402 +0x22c
net.(*netFD).accept(0xc00014eb00)
/usr/local/go/src/net/fd_unix.go:173 +0x35
net.(*TCPListener).accept(0xc00000e6f0)
/usr/local/go/src/net/tcpsock_posix.go:140 +0x28
net.(*TCPListener).Accept(0xc00000e6f0)
/usr/local/go/src/net/tcpsock.go:262 +0x3d
github.com/valyala/fasthttp.acceptConn(0xc000128b40, {0x8cfe28, 0xc00000e6f0}, 0xc0000c7c70)
/go/pkg/mod/github.com/valyala/[email protected]/server.go:1788 +0x62
github.com/valyala/fasthttp.(*Server).Serve(0xc000128b40, {0x8cfe28, 0xc00000e6f0})
/go/pkg/mod/github.com/valyala/[email protected]/server.go:1703 +0x4e5
github.com/gofiber/fiber/v2.(*App).Listen(0xc000124c60, {0xc000026ea4, 0xc00018fdc8})
/go/pkg/mod/github.com/webp-sh/fiber/[email protected]/app.go:574 +0xb5
main.main()
/build/webp-server.go:117 +0x607

goroutine 28 [sleep]:
time.Sleep(0x12a05f200)
/usr/local/go/src/runtime/time.go:193 +0x12e
github.com/valyala/fasthttp.(*FS).initRequestHandler.func1()
/go/pkg/mod/github.com/valyala/[email protected]/fs.go:403 +0x50
created by github.com/valyala/fasthttp.(*FS).initRequestHandler
/go/pkg/mod/github.com/valyala/[email protected]/fs.go:400 +0x3e5

goroutine 7 [sleep]:
time.Sleep(0x1dcd6500)
/usr/local/go/src/runtime/time.go:193 +0x12e
github.com/gofiber/fiber/v2/middleware/logger.New.func1()
/go/pkg/mod/github.com/webp-sh/fiber/[email protected]/middleware/logger/logger.go:99 +0x75
created by github.com/gofiber/fiber/v2/middleware/logger.New
/go/pkg/mod/github.com/webp-sh/fiber/[email protected]/middleware/logger/logger.go:97 +0x325

goroutine 29 [sleep]:
time.Sleep(0x3b9aca00)
/usr/local/go/src/runtime/time.go:193 +0x12e
github.com/valyala/fasthttp.updateServerDate.func1()
/go/pkg/mod/github.com/valyala/[email protected]/header.go:1438 +0x1e
created by github.com/valyala/fasthttp.updateServerDate
/go/pkg/mod/github.com/valyala/[email protected]/header.go:1436 +0x27

goroutine 8 [sleep]:
time.Sleep(0x2540be400)
/usr/local/go/src/runtime/time.go:193 +0x12e
github.com/valyala/fasthttp.(*workerPool).Start.func2()
/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:66 +0x56
created by github.com/valyala/fasthttp.(*workerPool).Start
/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:58 +0xde

goroutine 30 [runnable]:
github.com/h2non/filetype.Match({0xc00024c800, 0x1598, 0x1599})
/go/pkg/mod/github.com/h2non/[email protected]/match.go:21 +0x4ab
main.getFileContentType({0xc00024c800, 0x1598, 0x1599})
/build/helper.go:38 +0x14e
main.convert(0xc00012edc0)
/build/router.go:91 +0x82f
github.com/gofiber/fiber/v2.(*App).next(0xc000124c60, 0xc00012edc0)
/go/pkg/mod/github.com/webp-sh/fiber/[email protected]/router.go:126 +0x1d8
github.com/gofiber/fiber/v2.(*Ctx).Next(0x6da2e5)
/go/pkg/mod/github.com/webp-sh/fiber/[email protected]/ctx.go:637 +0x53
github.com/gofiber/fiber/v2/middleware/logger.New.func2(0xc00012edc0)
/go/pkg/mod/github.com/webp-sh/fiber/[email protected]/middleware/logger/logger.go:154 +0x22b
github.com/gofiber/fiber/v2.(*App).next(0xc000124c60, 0xc00012edc0)
/go/pkg/mod/github.com/webp-sh/fiber/[email protected]/router.go:126 +0x1d8
github.com/gofiber/fiber/v2.(*App).handler(0xc000124c60, 0x4b2377)
/go/pkg/mod/github.com/webp-sh/fiber/[email protected]/router.go:154 +0xe5
github.com/valyala/fasthttp.(*Server).serveConn(0xc000128b40, {0x8d5cd0, 0xc0000106e0})
/go/pkg/mod/github.com/valyala/[email protected]/server.go:2168 +0xf2b
github.com/valyala/fasthttp.(*workerPool).workerFunc(0xc00011d360, 0xc0004520c0)
/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:223 +0xa9
github.com/valyala/fasthttp.(*workerPool).getCh.func1()
/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:195 +0x38
created by github.com/valyala/fasthttp.(*workerPool).getCh
/go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:194 +0x1b5

goroutine 25 [IO wait]:
internal/poll.runtime_pollWait(0x7f7104f6a960, 0x72)
/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00014ec00, 0xc000284000, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00014ec00, {0xc000284000, 0xe03, 0xe03})
/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00014ec00, {0xc000284000, 0x893, 0xc00049b2c0})
/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00008a048, {0xc000284000, 0x5d5279, 0xc0002577f8})
/usr/local/go/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc00047c888, {0xc000284000, 0x0, 0x413fed})
/usr/local/go/src/crypto/tls/conn.go:777 +0x3d
bytes.(*Buffer).ReadFrom(0xc0000f0278, {0x8c9820, 0xc00047c888})
/usr/local/go/src/bytes/buffer.go:204 +0x98
crypto/tls.(*Conn).readFromUntil(0xc0000f0000, {0x8c9fa0, 0xc00008a048}, 0xc000284575)
/usr/local/go/src/crypto/tls/conn.go:799 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc0000f0000, 0x0)
/usr/local/go/src/crypto/tls/conn.go:606 +0x112
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:574
crypto/tls.(*Conn).Read(0xc0000f0000, {0xc000285000, 0x1000, 0xc0000ac000})
/usr/local/go/src/crypto/tls/conn.go:1277 +0x16f
bufio.(*Reader).Read(0xc0004b5800, {0xc0004c8e38, 0x9, 0xc000080660})
/usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x8c9700, 0xc0004b5800}, {0xc0004c8e38, 0x9, 0x9}, 0x9)
/usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
/usr/local/go/src/io/io.go:347
net/http.http2readFrameHeader({0xc0004c8e38, 0x9, 0xc0004d17d0}, {0x8c9700, 0xc0004b5800})
/usr/local/go/src/net/http/h2_bundle.go:1555 +0x6e
net/http.(*http2Framer).ReadFrame(0xc0004c8e00)
/usr/local/go/src/net/http/h2_bundle.go:1813 +0x95
net/http.(*http2clientConnReadLoop).run(0xc000257f98)
/usr/local/go/src/net/http/h2_bundle.go:8600 +0x130
net/http.(*http2ClientConn).readLoop(0xc00019a000)
/usr/local/go/src/net/http/h2_bundle.go:8523 +0x6f
created by net/http.(*http2Transport).newClientConn
/usr/local/go/src/net/http/h2_bundle.go:7317 +0xb85

Environment (please complete the following information):

• OS: CentOS 7.9
• Docker: 20.10.12-3
• webp_server_go: 0.4.1

set EXHAUST_PATH in config.json.
If it's empty, use working dir

Hi, i know it's kinda an amateur question but i have a question how can i (wget) the right webp-server binary file.

if the image is webp ,the server will return is corrupted or not supported
could you help to return original image？

https://caniuse.com/?search=webp

Describe the bug
When I try to run webp-server executable on Ubuntu 18.04 server I get the error:
./webp-server: error while loading shared libraries: libaom.so.0: cannot open shared object file: No such file or directory

To Reproduce
Simply run the executable on Ubuntu 18.04

Expected behavior
It should run smoothly.

Screenshots and logs
user@ubuntu-server-18:~/webp-server$ ./webp-server --config config.json
./webp-server: error while loading shared libraries: libaom.so.0: cannot open shared object file: No such file or directory

Environment (please complete the following information):

• OS: Ubuntu 18.01

image map: support multiple IMG_PATH
config sample dump: add cli param to generate a sample config. @n0vad3v Which param should we choose? Didn't find a good one.

config.json

add mode

• 1 stands for lossless, quality parameter will be ignored
• 2 stands for near lossless, quality parameter will be used to reduce the size of output image
• 3 stands for lossy, quality parameter will be used to reduce the size of output image

multidir customization(if possible)

After #15, add support for read config in exhaust/img_path dir.

similar to git config --global and git config

Investigate.

Question: if I have 2 core 4 threads, should I choose 2 or 4?

Describe the bug
Docker WebP Server throws "main.imageExists" and restarts.
It seems like it happens when a image is beeing is accessed concurrently.

To Reproduce
2 WebP Server on different Nodes.
Shared "exhaust" folder between nodes (glusterfs).

Expected behavior
Normal behavior.

Screenshots and logs

webp_1_257cddb385a8 | panic: runtime error: invalid memory address or nil pointer dereference
webp_1_257cddb385a8 | [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x74eb37]
webp_1_257cddb385a8 |
webp_1_257cddb385a8 | goroutine 54 [running]:
webp_1_257cddb385a8 | main.imageExists({0xc0003b6b90, 0x4f})
webp_1_257cddb385a8 |   /build/helper.go:64 +0xb7
webp_1_257cddb385a8 | main.convertFilter({0xc0003b6a50, 0x41}, {0xc0003b6b90, 0x4f}, {0xc0003b6b40, 0x4f}, 0x0)
webp_1_257cddb385a8 |   /build/encoder.go:26 +0x46
webp_1_257cddb385a8 | main.convert(0xc0003d8000)
webp_1_257cddb385a8 |   /build/router.go:68 +0x2a5
webp_1_257cddb385a8 | github.com/gofiber/fiber/v2.(*App).next(0xc0001c4c60, 0xc0003d8000)
webp_1_257cddb385a8 |   /go/pkg/mod/github.com/webp-sh/fiber/[email protected]/router.go:126 +0x1d8
webp_1_257cddb385a8 | github.com/gofiber/fiber/v2.(*Ctx).Next(0x30)
webp_1_257cddb385a8 |   /go/pkg/mod/github.com/webp-sh/fiber/[email protected]/ctx.go:637 +0x53
webp_1_257cddb385a8 | github.com/gofiber/fiber/v2/middleware/logger.New.func2(0xc0003d8000)
webp_1_257cddb385a8 |   /go/pkg/mod/github.com/webp-sh/fiber/[email protected]/middleware/logger/logger.go:154 +0x22b
webp_1_257cddb385a8 | github.com/gofiber/fiber/v2.(*App).next(0xc0001c4c60, 0xc0003d8000)
webp_1_257cddb385a8 |   /go/pkg/mod/github.com/webp-sh/fiber/[email protected]/router.go:126 +0x1d8
webp_1_257cddb385a8 | github.com/gofiber/fiber/v2.(*App).handler(0xc0001c4c60, 0x4b2377)
webp_1_257cddb385a8 |   /go/pkg/mod/github.com/webp-sh/fiber/[email protected]/router.go:154 +0xe5
webp_1_257cddb385a8 | github.com/valyala/fasthttp.(*Server).serveConn(0xc00016ab40, {0x8d95f0, 0xc0003960d0})
webp_1_257cddb385a8 |   /go/pkg/mod/github.com/valyala/[email protected]/server.go:2338 +0x11ae
webp_1_257cddb385a8 | github.com/valyala/fasthttp.(*workerPool).workerFunc(0xc0001b92c0, 0xc0003aa040)
webp_1_257cddb385a8 |   /go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:224 +0xa9
webp_1_257cddb385a8 | github.com/valyala/fasthttp.(*workerPool).getCh.func1()
webp_1_257cddb385a8 |   /go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:196 +0x38
webp_1_257cddb385a8 | created by github.com/valyala/fasthttp.(*workerPool).getCh
webp_1_257cddb385a8 |   /go/pkg/mod/github.com/valyala/[email protected]/workerpool.go:195 +0x1b5

Environment (please complete the following information):

• OS: Debian 10, Docker
• Webp Server Go - v0.4.4

Describe the bug
当上传目录下存在其他媒体文件如mp3/mp4时报File extension not allowed！.

To Reproduce
当页面中插入媒体文件后，打开页面，媒体文件都无法访问

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots and logs

time="2022-03-15 22:49:04" level=warning msg="File extension not allowed! 六下Unit2复习课-7aeb24e18d4c456d92e5c705e2771fd6.mp4" func="[34:main.convert()]"
time="2022-03-15 22:49:04" level=warning msg="File extension not allowed! 《匆匆》第1课时-e184d45908294518a9cc4f561f67e2a5.mp4" func="[34:main.convert()]"
time="2022-03-15 22:49:06" level=warning msg="File extension not allowed! 数学比例的意义-d8f083fa34eb4d7a9cd778830626012b.mp4" func="[34:main.convert()]"
time="2022-03-15 22:49:06" level=warning msg="/my/cache/upload/2022/02/Minecraft Creeper-27f341abb9f24836af53c6acae81ffcf.png.1645433906.avif not found on filesystem" func="[230:main.findSmallestFiles()]"
time="2022-03-15 22:49:10" level=warning msg="File extension not allowed! 六下Unit2复习课-7aeb24e18d4c456d92e5c705e2771fd6.mp4" func="[34:main.convert()]"
time="2022-03-15 22:49:13" level=warning msg="File extension not allowed! 六下Unit2复习课-7aeb24e18d4c456d92e5c705e2771fd6.mp4" func="[34:main.convert()]"

Environment (please complete the following information):

• OS: [Ubuntu 20.04]
• version or branch[0.4.2]

Is your feature request related to a problem? Please describe.
The memory consumption of webp-server may grow to >500MB when processing 5 concurrent requests that require webp conversion. Such workload causes a great burden on conventional virtual servers (with 1GB ram) and may trigger the OOM killer.

Describe the solution you'd like
Implement a limit on the number of concurrent webp conversion tasks.

Describe alternatives you've considered
The memory usage problem can be partially mitigated with the following options:

1. Limit the concurrent request numbers in HTTP servers.
   This is a common strategy to prevent the backend from overloading. However, it's ill-suited for webp-server, as conversion requests will block subsequent requests that can be directly fulfilled by returning the original image file.

2. Convert every image file to webp in advance with -prefetch
   However, users still need to restart webp-server to perform pre-conversion every time they add some image files, or some concurrent requests might still cause excessive memory usage.

Both methods are not optimal enough for real-world usage, hence I think this problem should be addressed in webp-server itself.

Internet Explorer doesn't support WebP images, but the server response a WebP image to it.
Maybe we could identify IE11 by checking whether the UA contains rv:11.0, and the keyword MSIE would work for IE7-10.

So I want to start from scratch and delete all the cache.

Do I delete all the contents of remote-raw and cache and that's it?
Also, do webp files expire at some point?