[BUG] Generators after being cythonized cannot be inspect as generators about cython HOT 13 CLOSED

I initiate the discussion of this issue here.

from cython.

It works for me on Cython 3.0.9 with Python 3.11.5.

Have you set the directive # cython: binding = False somewhere? That turns off a lot of introspection and means that inspect.isgeneratorfunction won't work on them. It's on by default though.

from cython.

Fun fact: I found on the repository a comment that was not made by me but never called to my attention:

ext_modules = cythonize(
    ["**/*.pyx"],
    compiler_directives={
        "binding": True,  # so inspect function works properly on cythonized functions
        "language_level": language_level,
    },  # avoid cython warnings
)

So yes, explicitly the biding is set as True.

I look for the line # cython: binding = False or just cython: binding but no place else we define that.

I can try with the most recent Cython version and let you know the outcome.

from cython.

No changes with cython==3.0.9. Let me know if a more explanatory example is needed.

from cython.

To clarify, what I tested was

# gen.py
def _my_generator():
    value = 6
    print("my_generator:", value)
    yield value
    value = value + 6
    print("my_generator:", value)

and then tested it with:

# test_gen.py
import gen
import inspect
import os

print(os.path.split(gen.__file__)[1])
print(gen._my_generator)
print(gen._my_generator.__code__.co_flags)
print(inspect.isgeneratorfunction(gen._my_generator))

That gives

gen.py
<function _my_generator at 0x7fa38359cd60>
35
True

Then if I do cythonize -if gen.py and run it again I get

gen.cpython-311-x86_64-linux-gnu.so
<cyfunction _my_generator at 0x7f286ca69e50>
35
True

I don't think your example that you show in the issue is quite what you actually run - for example the module and function names don't match.

from cython.

The only step that I miss explaining is that the my_generator function is built as a package. Cythonized using the follow lines on setup.py file:

ext_modules = cythonize(
    ["**/*.pyx"],
    compiler_directives={
        "binding": True,  # so inspect function works properly on cythonized functions
        "language_level": language_level,
    },  # avoid cython warnings
)

Then, the module is installed and imports the function from the cythonized wheel. I can create a small repository with the example and attach it to our discussion with a tox file for execution.

from cython.

I managed to pack nicely in a repository (cy_wheel_gens_tests).

from cython.

The issue is your wrapper in __init__.py which doesn't have anything to do with Cython. Here's a reduced version of your example (no Cython necessary):

def _my_generator():
    yield 1

def my_generator():
    return _my_generator()

import inspect

print(inspect.isgeneratorfunction(my_generator))

This prints False because of the wrapper.

from cython.

Aha, I see. I will make a better example because the issue that we are having is on pluggy verification of pytest hooks.

I will investigate more and if persists I will open a new ticket. Thank you for your time.

from cython.

Please, if is possible can you try to take a look at example again ? In the same repository here. I upadted the README.md with the updates made on implementation and on the test.

from cython.

Works for me on 3.10 and 3.11:

====================================================== 1 passed in 0.01s ======================================================
  py38: FAIL code 1 (5.00=setup[0.04]+cmd[0.54,2.93,1.04,0.45] seconds)
  py39: FAIL code 1 (4.75=setup[0.01]+cmd[0.51,2.87,0.91,0.44] seconds)
  py310: OK (4.61=setup[0.01]+cmd[0.46,2.81,0.94,0.40] seconds)
  py311: OK (4.04=setup[0.01]+cmd[0.45,2.47,0.89,0.21] seconds)

On 3.8, 3.9 inspect.isfunction requires return isinstance(object, types.FunctionType) which we cannot meet.

The changes in inspect that make it work right came in Python 3.10.6 and 3.11.0. (python/cpython#94050)

There's no way for us to make it work on earlier versions of Python.

from cython.

Understood. Anyway, thank you once more for your time.

from cython.

Your other option would be to monkey-patch inspect. I think we do it ourselves inside runtests.py (

At one point Cython did this specifically for isgenerator in some circumstances. That wouldn't have helped you specifically, although the basic idea could have been used. I think it got removed - possibly semi-mistakenly mainly because it wasn't tested. I don't think Cython should return to doing that in Cython itself though.

from cython.