Comments (6)
That's, actually, not totally correct. libvips does use your drive space to store decoded images that are too big to be securely stored in RAM. As you noticed correctly, 7360*4912*3
≈ 104 MB, and many popular image processing tools like ImageMagick would store this amount of data on the drive.
Anyway, libvips removes all unneeded decoded data, so it won't eat all your space.
If you allow processing such big images on the fly, you should know the cost of it. They will either use a lot of your memory or drive space.
from imgproxy.
This makes sense, but there are some problems:
-
Those files are deleted from the file system after request, but as you can see from
lsof
output, they are still opened by the imgproxy process and that is why OS can't free this space. So yes, during my tests imgproxy actually ate 4.8G of free space and left 0 bytes. -
Disk storage is very-very slow compared to memory even with SSD. There are already limits for maximum image resolution, so imgproxy shouldn't use more memory than defined by those limits. There are no reasons to store any data on the disk if it fits into memory. And if the data doesn't fit, OS already has mechanisms to use disk storage if necessary: swap.
-
If this storage should prevent from high memory usage, why after the end of the short benchmark (
ab -c 10 -n 50
, all requests are finished) imgproxy consumes 13% (0.94Gb) of memory?
from imgproxy.
-
It seems like libvips reuses temporary files, I don;t see any other explanation of this behavior for now. I'll continue digging into it.
Anyway, I made some changes in imgproxy, and now it uses sequential access mode instead of random access in most cases. The sequential mode doesn't create a temporary file and saves some memory. The only problem is that sequential mode isn't compatible with smartcrop, so I have to use random access mode with it. -
Disk storage is slow, for sure. But RAM is a more valuable thing. There are a lot of reasons to not use all the memory we have. There may be other processes that need RAM. Or there may be RAM limits for the process, and it'll be killed instead of swapping. So it's not such a bad idea to use your disk instead of eating all the memory. Sure, wasting the disk with temp files isn't nice, but again, if you allow processing such big files on the fly, there will be consequences.
If you still want to use your memory, you can set env variableVIPS_DISC_THRESHOLD=1000MB
, to libvips won't save images of less than 1000 Mb decompressed size to disk. -
imgproxy and vips still need memory to download and process files. The garbage collector in Go doesn't instantly return memory to the operating system, it uses allocated memory for the new objects. I made some optimizations and enforced GC to return memory more often. Now it looks much better to me.
Btw, could you share the image which you use for testing?
from imgproxy.
Btw, could you share the image which you use for testing?
Sure, here it is: https://en.wikipedia.org/wiki/File:0000140_Wat_Arun_01.jpg
from imgproxy.
Results with the current master:
- No disk swapping
- ≈400% CPU load during the test (unlike the previous version with only ≈150% CPU load).
- 2 times more throughput
- only 320 megabytes memory usage after the same test
So, everything is definitely much better )
from imgproxy.
Nice, thanks for testing!
from imgproxy.
Related Issues (20)
- Asking for Help: Nginx Configuration for Rewrites HOT 2
- Extract 9:16 thumbs from video & using VTT seems to lose the resize HOT 12
- ICO files failing with "Cant load from ICO" error HOT 7
- Base64 encoding of local url doesn't work HOT 1
- Resizing with "fit" mode leads to blurry images in some cases HOT 2
- invalid TIFF format: image dimensions are not specified HOT 6
- Support EPS files HOT 1
- "is not a video" error when trying to source PSD file HOT 2
- How to get width and height of original img? HOT 1
- Feature request: support using data urls as watermarkurl HOT 1
- Feature request: Gradient improvements HOT 5
- Text is illegible when a PDF has non-embedded CID TrueType and no character map HOT 4
- Feature Request: Option to limit max dimension of output image HOT 3
- Error: heifsave: image too large HOT 2
- Using imgproxy via Proxy for Amazon S3 HOT 3
- Option to report source image errors HOT 4
- Style transformation for SVG broken since 3.23 HOT 3
- Error: SVG detection does not work when it uses namespaces HOT 2
- Can't download source image: invalid JPEG format: missing SOF marker HOT 2
- ERROR png2vips: unable to read source source HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from imgproxy.