imazen / imageflow Goto Github PK

Currently we only have input buffer/output buffer

It seems like decent heuristics for determining the ideal image format should exist.

Here's a theoretical one:

1. If multi-frame, -> GIF
2. If the alpha channel is actually in use, -> PNG
3. No? Why not try to compress a few randomly distributed scanlines with DEFLATE. If the ratio sucks, use JPEG, otherwise use PNG.

Ok, using GIF? We need to decide the color table size, and whether to preserve transparency or support animation. We may consult the original GIF, and look at the operation graph to prove that no new colors were introduced, then re-use that table.

Using PNG? Oh, so many knobs and settings. 8-bit? 24-bit? 32-bit? Size of the color table?

JPEG? What subsampling format? What quality? Should quality decrease with resolution at a certain point (higher res usually means smaller pixels are being displayed)? Should we use trellis?

@pornel, I imagine you've thought about this quite a bit. Other than exhaustive attempts, are you aware of any good techniques for "doing the right thing" by default for the unopinionated user? We would want to budget <50ms for this, preferably <5ms for small images.

Usage example of parameters to libjpeg:
https://github.com/jcupitt/libvips/blob/01238a0fd554d765e5c1437580cf94a14e69aa55/libvips/foreign/jpeg2vips.c#L722-L725

https://blog.kitware.com/create-dlls-on-windows-without-declspec-using-new-cmake-export-all-feature/

http://stackoverflow.com/a/226017/166893

It's also possible to make gcc mimic msvc, in order to catch errors earlier. We should do that unless we want to force cmake 3.4 (which would be a bad idea until 2018)

Allow nodes to prohibit outbound edges.

Hi Nathanael, I was alerted to this rather interesting repo via your recent questions about libjpeg-turbo.

One thing that caught my eye was the statement in the readme about "trusted data":

"...all the libraries that I've reviewed are insecure. Some assume all images are trusted data (libvips)..."

libvips relies on the "standard" image decoding libraries such as libjpeg-turbo, libpng etc., most of which have seen the recent attention of fuzzing tools such as AFL.

Are you able to explain a little more about what is meant by "insecure" and "trusted data" in this context? What would you expect libraries to be doing in order to become "secure"?

My best guess is that this might relate to "trusted metadata", where fake headers could result in large memory allocations, assuming that data was able to get through the decoding library's own checks first.

Cheers,
Lovell

/cc @jcupitt

V4 cannot make the jump, as we want to retain backwards compatibility as much as possible with V3 - but we can start using it internally, perhaps.

Given image characteristics, we can translate descriptive URL commands into a graph (usually just a linked list) of operation placeholders.

Pipelines can report their support level for the given operation graph.

The abstract graph is converted to implementations.

And executed.

In theory this should let us perform a much wider range of optimizations (and enable better plugin support). By adding converters between implementations of non-compatible pipelines, we will also greatly extend the available abilities (being a super-set, not a subset of any leveraged dependency).

https://stackoverflow.com/questions/8073115/how-do-i-get-dot-to-display-an-image-for-a-node

Yeah, this means double the code paths to test. Anyone have some 64-bit PNGs with far-out color profiles?

https://github.com/imazen/imageflow/blob/master/tests/test_visuals.cpp#L110-L150

Produces significantly different results under clang 6.0/6.1 on OS X (but not on new versions of clang or any gcc version!)

0.00000569000000000000

https://travis-ci.org/imazen/imageflow/builds/121315237

https://github.com/imazen/resizer/tree/features/better-blur

By design, one can set up a job and resources, then run multiple graphs through them (as long as output resources are not re-used - only input resource codecs permit re-use at the moment).

We should have test coverage for this scenario so as to not inadvertently break it.

A reverse LUT of ~4000 will round things differently from pow. See how much, and if it's acceptable for general use.

• Ensure graph is acyclic
• Ensure nodes and edges are not orphaned
• Ensure node def constraints are upheld

API design required

Use of infobyte/node/edge pointers, even temporarily, is problematic, as that code is often intermixed with create_node calls, which can replace the underlying graph.

Block scaling in the decoder can leave edge pixels over-emphasized.
I.e, scale 9x9 image down to 25% - that's 3x3, not 2x2. But the bottom and right row of pixels have actually not been scaled at all! Unless they are weighted in a later scaling, they will remain an artefact.

We can use a hash function to generate the error code.
Static strings are easy, but what if we want to include debug info?
That's only possible when there is enough memory, so it needs to be able to fall back to just static numbers.

Fix travis build

• One must loop through all frames and expand the 'screen size' to (left + width), (top + height) for each frame. Negative left/top coordinate handling unexplored.
• Perhaps we should quantize in linear light?
• Global color map optimization seems challenging

Verify that if more than one outbound edge exists for a node, that neither of those nodes mutate the bitmap. Either insert a Clone, or fail the graph.

This should reveal a lot of segfaults. We need to add guards to nearly every function in graph.c and node.c

Render1D should expand into a HalveHorizontal and Render1D when there is a huge downsampling factor.

HalveHorizontal and HalveVerticle can be shifted left until they meet, then combined into a Halve2D. Halve2D can be shifted left to the decoder (if the decoder supports built-in scaling).

The job will have to represent some optimization rules, such as permitting perceptual scaling instead of linear scaling for the first 1/3rd of the job, or halving prior to colorspace application.

For filesize, one wants to use delta-encoding for image frames (do not dispose (dispose=1)).
For transparency support, (restore to background (dispose=2)) needs to be used.

(do not dispose) may have transparent pixels, but intermediate frames could hide these.

Perhaps the default if any transparency is detected should be (restore to background), but let users override with &force_gif_delta?

http://ivy.cs.berkeley.edu/ivywiki/uploads/deputy-quickref.html

We should offer multiple types of estimable output, and multiple kinds of results.

• scanline-based function pointer reading
• sRGB 32-bit frame
• arbitrary metadata

Graph optimization is a challenging problem. We should review primitives to see if higher-level constructs are easier to optimize, and how small we can make the equivalency matrix.