In my quest for ever-faster image data pipelines for training neural nets, I've been playing around with the source code to figure out how to reduce allocations when decoding images. I'm writing here to see if my assumptions about how one could go about this are correct and to clear up some questions.

It seems there are two allocations made:

• in jpeg_decode, a matrix out is created as the Julia representation of the image data which is returned to the caller
• in _jpeg_decode!, a UInt8-vector buf is created which JpegTurbo writes to

Copying some code from jpeg_decode, I've managed to make a method that takes in an out of the correct size and type and uses that instead of allocating it. There are some segfaults when transposing is not handled correctly or the size and type of the buffer aren't correct, but I assume these can be fixed. In any case, removing this allocation cuts memory usage in half. I assume that something similar could be done for the buf allocation; or for CTs that are based on UInt8s anyway (like RGB{N0f8}), maybe even a view will do.

As for the API to use buffered data loading, I was thinking it may be safest to have a Buffer struct that holds a out and buf. Since one often wants to reuse this buffer to load images of differing sizes, these buffers could be grown to the largest encountered image size; and images smaller than the current out buffer returned as views. This could be used something like:

buffer = JpegTurbo.Buffer(RGB{N0f8}, initialsize)
img = JpegTurbo.jpeg_decode!(buffer, file|data)
parent(img) === buffer.out

Does this approach make sense? Is there a simpler way? Am I missing something when it comes to the transposing?

Would there be a way to get the error messages thrown by C at the Julia level?
For instance I would wish to be able to control the logging flow of stuff like:

Corrupt JPEG data: 283 extraneous bytes before marker 0xd9

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

This is needed to support JPEG Compression for TiffImages.

See section "Abbreviated datastreams and multiple images" in https://raw.githubusercontent.com/libjpeg-turbo/libjpeg-turbo/main/libjpeg.txt

cc: @tlnagy

Just read that scale_ratio can be passed when decoding, which sounds awesome! Is it also somehow possible (maybe even with an external package) to read the size (h, w) of an image without loading first.

My use case is a high-throughput scenario, loading images for a deep learning pipeline. Often many of these images are saved in a much larger, but variable, resolution and are scaled and cropped to the same size (e.g. (256, 256)) before being batched and run through the model. It would be super useful to know the image size before decoding the image, so that I can select the smallest possible scale_ratio such that we still have (h>256 ,w>256). Still need to benchmark how this compares in load speed to just loading each image, downscaling and saving it once. Even if it only offers some load speed improvements, avoiding the extra resizing preprocessing step could also help reduce quality degradation, though.

The current pipeline consists of two stages:

1. first allocate a C-side buffer in memory and stores the encoded bytes, and
2. unsafe_wrap the buffer in Julia side and use Julia’s IO operation to write to the output file

function jpeg_encode(filename::AbstractString, img; kwargs...)
    open(filename, "w") do io
        jpeg_encode(io, img; kwargs...)
    end
end
jpeg_encode(io::IO, img; kwargs...) = write(io, jpeg_encode(img; kwargs...))

If we can fuse two stages, then we can get a faster version.

For jpeg_encode(filename, img) specifically, it's also possible to wrap the C stdio Libc.FILE version.

See the "Progressive JPEG support" section in https://raw.githubusercontent.com/libjpeg-turbo/libjpeg-turbo/main/libjpeg.txt

See the "Partial image decompression" section in https://raw.githubusercontent.com/libjpeg-turbo/libjpeg-turbo/main/example.txt

JPEG backends comparison

Even though JpegTurbo.jl provides more advanced and efficient in-memory features, the benchmark only tests the filename version because all other backends don't support this.

using JpegTurbo
using BenchmarkTools
using TestImages

img = testimage("cameraman");
filename = "tmp.jpg"

jpeg_encode(filename, img);
data = jpeg_encode(img);
@assert read(filename) == data;

@btime jpeg_encode($img); # 855.914 μs (7 allocations: 306.49 KiB)
@btime jpeg_encode(filename, $img); # 1.064 ms (20 allocations: 307.19 KiB)

@assert jpeg_decode(filename) == jpeg_decode(data)
@btime jpeg_decode($data); # 795.819 μs (18 allocations: 514.66 KiB)
@btime jpeg_decode(filename); # 836.992 μs (45 allocations: 630.62 KiB)

Generally speaking, for the filename version, JpegTurbo.jl and OpenCV (python) are the fastest versions since they are both backed by libjpeg-turbo.

Julia versioninfo:
Julia Version 1.8.0-DEV.1434
Commit 4abf26eec8 (2022-01-30 20:04 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.7.0)
  CPU: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.0 (ORCJIT, skylake)
Environment:
  JULIA_NUM_THREADS = 8

JpegTurbo.jl versioninfo:
JpegTurbo.jl version: 0.1.0
libjpeg version: 62
libjpeg-turbo version: 2.1.0
bit mode: 8
SIMD: enabled

OpenCV version: 4.5.5
OpenCV libjpeg-turbo version: 2.1.2-62

Scikit-image version: 0.19.1

use BenchmarkTools and PkgBenchmark

When we tried to add JpegTurbo in julia v1.7.3, we got the follow error message. What could be the problem?

  Downloaded artifact: JpegTurbo
ERROR: Unable to automatically install 'JpegTurbo' from '/home/zhaoli9/.julia/packages/JpegTurbo_jll/x2k7x/Artifacts.toml'
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] ensure_artifact_installed(name::String, meta::Dict{String, Any}, artifacts_toml::String; platform::Base.BinaryPlatforms.Platform, verbose::Bool, quiet_download::Bool, io::Base.TTY)
    @ Pkg.Artifacts ~/works/public_code/julia-1.7.3/share/julia/stdlib/v1.7/Pkg/src/Artifacts.jl:441
  [3] download_artifacts(env::Pkg.Types.EnvCache; platform::Base.BinaryPlatforms.Platform, julia_version::VersionNumber, verbose::Bool, io::Base.TTY)
    @ Pkg.Operations ~/works/public_code/julia-1.7.3/share/julia/stdlib/v1.7/Pkg/src/Operations.jl:617
  [4] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Set{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations ~/works/public_code/julia-1.7.3/share/julia/stdlib/v1.7/Pkg/src/Operations.jl:1182
  [5] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
    @ Pkg.API ~/works/public_code/julia-1.7.3/share/julia/stdlib/v1.7/Pkg/src/API.jl:268
  [6] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API ~/works/public_code/julia-1.7.3/share/julia/stdlib/v1.7/Pkg/src/API.jl:149
  [7] add(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API ~/works/public_code/julia-1.7.3/share/julia/stdlib/v1.7/Pkg/src/API.jl:144
  [8] do_cmd!(command::Pkg.REPLMode.Command, repl::REPL.LineEditREPL)
    @ Pkg.REPLMode ~/works/public_code/julia-1.7.3/share/julia/stdlib/v1.7/Pkg/src/REPLMode/REPLMode.jl:407
  [9] do_cmd(repl::REPL.LineEditREPL, input::String; do_rethrow::Bool)
    @ Pkg.REPLMode ~/works/public_code/julia-1.7.3/share/julia/stdlib/v1.7/Pkg/src/REPLMode/REPLMode.jl:385
 [10] do_cmd
    @ ~/works/public_code/julia-1.7.3/share/julia/stdlib/v1.7/Pkg/src/REPLMode/REPLMode.jl:376 [inlined]
 [11] (::Pkg.REPLMode.var"#24#27"{REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::IOBuffer, ok::Bool)
    @ Pkg.REPLMode ~/works/public_code/julia-1.7.3/share/julia/stdlib/v1.7/Pkg/src/REPLMode/REPLMode.jl:549
 [12] #invokelatest#2
    @ ./essentials.jl:716 [inlined]
 [13] invokelatest
    @ ./essentials.jl:714 [inlined]
 [14] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
    @ REPL.LineEdit ~/works/public_code/julia-1.7.3/share/julia/stdlib/v1.7/REPL/src/LineEdit.jl:2493
 [15] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
    @ REPL ~/works/public_code/julia-1.7.3/share/julia/stdlib/v1.7/REPL/src/REPL.jl:1232
 [16] (::REPL.var"#49#54"{REPL.LineEditREPL, REPL.REPLBackendRef})()
    @ REPL ./task.jl:429

I'm not even sure if this doable, see "Error handling" section in libjpeg.txt and
the "JPEG DECOMPRESSION SAMPLE INTERFACE" part in example.c

It's unclear to me how to create jmp_buf from the julia side..

The following two methods are missing in #3

jpeg_decode(io::IO)
jpeg_decode(bytes::Vector{UInt8})