[BUG] Deadlock in OpenMP Parallel Loop When Invoking LAPACK Functions in Cython>=3.0 about cython HOT 4 CLOSED

I've install Scipy 1.10.1 in a virtual environment and I'm struggling to find "lapack_api.pxi" so I think there's some details I'm missing.

However - here's my working theory:

in compute_wrapper the nogil says that it can run without the GIL, not that it actually releases the GIL. You don't release the GIL and thus the whole thing runs with the GIL locked..

I wouldn't be surprised if it worked if you changed

cpdef void compute_wrapper(
        double[:] d, double[:] e, double[:] v, int n) noexcept:

    cdef double* dp = &d[0];
    cdef double* ep = &e[0];
    cdef double* vp = &v[0];

    with nogil:
        compute(dp, ep, vp, n);

to explicitly release the GIL.

I'm not sure what's going on beyond that (since I think the mystery .pxi file is important). But it might be that in Cython 3 the implementation of lapack_dstev has added an exception check somewhere which tries to reacquire the GIL.

from cython.

I added lapack_api.pxi to the original post. Thanks for noting the missing file.

from cython.

This is happening on linux, so probably not related directly to scipy/scipy#20294 which is windows-specific

from cython.

So:

In that version of scipy dstev is written as:

cdef void dstev(char *jobz, int *n, d *d, d *e, d *z, int *ldz, d *work, int *info) nogil

(I'm fairly sure it has noexcept in more recent versions of Scipy). Cython 3 interprets the definition without noexcept to mean "might throw a Python exception". Therefore, the call to dstev generates to code

__pyx_f_5scipy_6linalg_13cython_lapack_dstev(__pyx_v_jobz, __pyx_v_n, __pyx_v_d, __pyx_v_e, __pyx_v_z, __pyx_v_ldz, __pyx_v_work, __pyx_v_info); if (unlikely(__Pyx_ErrOccurredWithGIL()))

i.e. it always re-acquires the GIL in order to check for an exception.

As I said earlier, compute_wrapper doesn't release the GIL - it can simply run without the GIL. Therefore the deadlock is that all your OpenMP threads are trying to acquire the GIL when something else already holds it, hence the block.

If you modify compute_wrapper as I suggested to explicitly release the GIL then it works correctly.

We ran into a few issues like this with the release of Cython 3. The one differs because the parallel loop isn't Cython-generated, so it can't be fixed by changing the GIL handling from Cython.

from cython.