running AddressSanitizer on IlmCtlTest reports SEGV on unknown address 0x000000000008 in /usr/src/CTL/lib/IlmCtlSimd/CtlSimdInst.h:9 about ctl HOT 7 CLOSED

The AddressSanitizer error does not occur if optimization is disabled by changing -O1 to -O0 in the CMakeLists asan build type as follows:

set(CMAKE_CXX_FLAGS_ASAN
    "-fsanitize=address -fno-optimize-sibling-calls -fsanitize-address-use-after-scope -fno-omit-frame-pointer -g -O0"
    CACHE STRING "Flags used by the C++ compiler during AddressSanitizer builds."
    FORCE) 

from ctl.

I've noticed some calls to alloca() in CTL, which is a function that allocates memory on the stack instead of heap. I also noticed this note about silent failure in the "bugs" section of alloca() man page https://man7.org/linux/man-pages/man3/alloca.3.html

alloca() is used by CTL in these files:

.\ctlrender\tiff_file.cc
.\ctlrender\transform.cc
.\lib\dpx\dpx_util.cc
.\lib\IlmCtl\CtlExc.cpp
.\lib\IlmCtl\CtlTypeStorage.cpp

As described above, AddressSantizer gives SEGV error (which is the same error mentioned in the bugs section of alloca() man page) when running on the CTL's unit test IlmCtlTest,

=================================================================
==2075==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000008 (pc 0x7fd0caafde64 bp 0x7ffd9bd418c0 sp 0x7ffd9bd41630 T0)
==2075==The signal is caused by a READ memory access.
==2075==Hint: address points to the zero page.
    #0 0x7fd0caafde63 in Ctl::SimdInst::lineNumber() const /usr/src/CTL/lib/IlmCtlSimd/CtlSimdInst.h:93
    #1 0x7fd0caafde63 in Ctl::SimdInst::executePath(Ctl::SimdBoolMask&, Ctl::SimdXContext&) const /usr/src/CTL/lib/IlmCtlSimd/CtlSimdInst.cpp:175
...

from ctl.

The reported AddressSanitizer error does not occur if lines 61-62 of file /lib/IlmCtlSimd/CtlSimdInst.cpp are changed

from:

#include <CtlSimdInst.h>
#include <sstream>

to:

#pragma GCC push_options
#pragma GCC optimize ("O0")
#include <CtlSimdInst.h>
#pragma GCC pop_options
#include <sstream>

to disable the compiler's optimization of CtlSimdInst.h

from ctl.

The reported AddressSanitizer error does not occur if lines 87-88 of file /lib/IlmCtlSimd/CtlSimdInst.h are compiled without optimization by changing

    void		executePath (SimdBoolMask &mask,
				     SimdXContext &xcontext) const;

to

#pragma GCC push_options
#pragma GCC optimize ("O0")
    void		executePath (SimdBoolMask &mask,
				     SimdXContext &xcontext) const;
#pragma GCC pop_options

from ctl.

The reported AddressSanitizer error occurs when unittest IlmCtlTest processes line 96 of /unittest/IlmCtl/testVSArrays.ctl that calls the CTL code function empty()

int emptyA[1][2][3];
empty(emptyA);

CTL function empty() on lines 56-60 /unittest/IlmCtl/testVSArrays.ctl is an empty ctl function

void
empty(int a[][][])
{

}

So perhaps the compiler optimization when using -O1 or higher is causing the useless function empty() to be eliminated, thus the function pointer is zero, and disabling the compiler optimization with -O0 keeps the empty function and thus the function pointer is not zero.

from ctl.

To avoid attempting to call a function that has been optimized out, like CTL function empty() on lines 56-60 /unittest/IlmCtl/testVSArrays.ctl , a check can be added to SimdCallInst::execute() before calling executePath() at line 449 of file /lib/IlmCtlSimd/CtlSimdInst.cpp

for example the line:

_callPath->executePath (callMask, xcontext);

can be changed to:

if( 0 != _callPath ) 
{
	_callPath->executePath (callMask, xcontext);
}

from ctl.

A similar type of check can be added to SimdCallBranchInst::execute() on line 240 of /lib/IlmCtlSimd/CtlSimdInst.cpp

where _truePath->execute(mask, xcontext) appears, change it to

if( 0 != _truePath )
     _truePath->executePath (mask, xcontext);

where _falsePath->execute(mask, xcontext) appears, change it to

if( 0 != _falsePath )
     _falsePath->executePath (mask, xcontext);

from ctl.