ilevkivskyi / typing_inspect Goto Github PK

As opposed to previous posts, this time I created an issue directly in pytypes first: Stewori/pytypes#31

However I had some afterthoughts: maybe this feature is actually enough 'core'-related (and resembles too much __subclasses__()) to be implemented outside of typing_inspect?

Feel free to close if it is too 'high-level'.

The current version of typing_inspect supports typing_extensions 3.7.4. Running on Python 3.9.0/1 with that version of typing_extensions will result in the following error:

venv/lib/python3.9/site-packages/typing_inspect.py:17: in <module>
    from typing_extensions import _TypedDictMeta as _TypedDictMeta_TE
E   ImportError: cannot import name '_TypedDictMeta' from 'typing_extensions'

The problem stems from typing_extensions 3.7.4 lacking the required Python < 3.9.2 fixes.

Could you support lists in your api?
In example, have such functions:

def is_list_type(tp) -> bool:
    pass

is_list_type(List[int]) # True
is_list_type(Tuple[int, str]) # False

I presume this issue has something to do with typing_extensions, but not exactly sure how.

$ pip install typing-inspect==0.8.0
$ python
Python 3.10.8 (main, Dec 11 2022, 10:51:27) [Clang 11.0.3 (clang-1103.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from typing_inspect import typed_dict_keys
>>> from typing import TypedDict
>>> class A(TypedDict):
...   x: bool
... 
>>> typed_dict_keys(A)
>>> 

On Python 3.9 and 3.11, I get the expected return value of {'x': <class 'bool'>}

I suggest to add support for Final. I think it's a quite useful type hint. 🙂

Steps:

• Add is_final_type(tp)
• Extend get_args(tp, evaluate=None) to support Final
• Add tests

There was a regression in release 0.7.0 from 0.6.0.

Currently,

>>> from typing import List, Set
>>> from typing_inspect import get_args
>>> get_args(list[str])  # Yay this works! Great to see support for Python 3.9!
(<class 'str'>,)
>>> get_args(List)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jessemichel/Dependencies/miniconda3/envs/py39/lib/python3.9/site-packages/typing_inspect.py", line 471, in get_args
    res = tp.__args__
  File "/Users/jessemichel/Dependencies/miniconda3/envs/py39/lib/python3.9/typing.py", line 694, in __getattr__
    raise AttributeError(attr)
AttributeError: __args__

Previously,

>>> get_args(List)
(~T,)

I believe the fix would be changing this line:

res = tp.__args__ if hasattr(tp, '__args__') else ()

Based on README, making a request. :-)

Using get_generic_bases I can get generic bases for my type class. But it would be great if there would be utility function to get all ancestor classes in MRO order including generic bases. So similar to inspect.getmro, just that also parameterized generic type classes would be there.

This method would return True if

• t is the NoneType (t is type(None))
• t is a (possibly nested) Union to NoneType (this can be the result of using the Optional keyword or of an explicit Union[type(None), ....]
• t is a TypeVar with either a bound or a constraints set, and one of them is nonable

Not sure these need to be in typing_inspect, but for I am currently relying on the private names so I guess there is maybe something to do in the api here?

def is_forward_ref(typ):
    """
    Returns true if typ is a typing ForwardRef
    :param typ:
    :return:
    """
    return isinstance(typ, _ForwardRef)


class InvalidForwardRefError(Exception):
    """ Raised whenever some forward reference can not be evaluated """
    
    def __init__(self, invalid: _ForwardRef):
        self.invalid_ref = invalid

    def __str__(self):
        return "Invalid PEP484 type hint: forward reference {} could not be resolved with the current stack's " \
               "variables".format(self.invalid_ref)


def eval_forward_ref(typ: _ForwardRef):
    """
    Climbs the current stack until the given Forward reference has been resolved, or raises an InvalidForwardRefError.
    :param typ: the forward reference to resolve
    :return:
    """
    for frame in stack():
        try:
            return typ._eval_type(frame[0].f_globals, frame[0].f_locals)
        except NameError:
            pass

    raise InvalidForwardRefError(typ)

As always if this is too high-level / specific, feel, free to suggest to move it to pytypes or equivalent libs.
For example the eval_forward_ref above could be replaced with the low-level

def eval_forward_ref(typ, globs, locs):
    return typ._eval_type(globs, locs)

It depends on Python but starting from 3.6 TypedDict can be exported via typing_extensions package and on 3.8 we have it builtin. Also typing_extensions variant is 3.8 compatible while mypy_extensions is not. What happens now is that we have typed_dict_keys() function that tries to get an entries of TypedDict, however it expects TypedDict from mypy_extensions thus isintance check will never work.

typing_inspect doesn't really work with Python 3.9 right now. There are issues like #60 and #64, but a variety of things need to be updated.
My initial survey (just guessing from poking at implementations; it would be better to have someone review/expand this list who is more intimately familiar with the code-level changes):

• The typing module in 3.9 introduces a new "_BaseGenericAlias" class that might more correctly correspond to typing_inspect's current uses of "_GenericAlias."
• Add support and tests for the new plain type hints (e.g. "list[int]" rather than "List[int]"). There seems to be a new "GenericAlias" class (that extends no other classes) for these kind of annotations.
• Get the current test suite to pass under 3.9.
• Add 3.9 to the travis config for regression testing.

Any volunteers willing to give it a try? @ilevkivskyi - any additional guidance? What's wrong/missing in my above list?

On python 3.6.3 import of private '_gorg' is failing.
See https://travis-ci.org/lhupfeldt/multiconf/jobs/284627902
This module should be merged into typing or typing should provide public APIs for the methods needed by this module.

typing has switched from _gorg(obj) to obj._gorg, this should be taken into account.

Hi,

I'm seeing different behavior with get_origin(typing.List) and get_origin(list):

[ins] In [1]: from typing_extensions import get_origin

[ins] In [2]: import typing

[ins] In [3]: get_origin(list[int])
Out[3]: list

[ins] In [4]: get_origin(list)

[ins] In [5]: get_origin(typing.List)
Out[5]: list

In [1]: import typing_inspect; import typing

In [2]: t = typing.Iterable[int]

In [3]: typing_inspect.get_origin(t)
Out[3]: collections.abc.Iterable

I would expect it to return typing.Iterable instead.

Why? I am using the get_origin and get_args to take apart to walk the type tree, transforming certain nodes and collecting some data (to match a template tree against an actual tree). Similar to something you would do in the AST package or with any other traversal of a nested data structure.

This means I want to be be able to recursively fold over the types, which means I have to be able to deconstruct them and put them back together again. Since collections.abc.Iterable[int] is invalid, this breaks this use case.

This is likely the same issue as #27, but I think I have a slightly different use case. If you have general suggestions on better ways to do this kind of fold, instead of using get_origin and get_args, I am welcome to suggestions.

cc @tonyfast who is also doing some type traversal stuff.

The most useful introspection methods I'd like to use from the typing module are basically the functions that were removed in python/typing#283: checking whether a concrete object is compatible with a given type. This would be useful for simple runtime type assertions, for exploring the implications of PEP484 in the REPL, and annotations-based dispatch.

Please let me know what you think; I'd be very interested in helping to implement this. I can also submit a PR if that makes discussion easier.

Objections

Should this be in external tools?

@1st1 objected to this on the grounds that it should be left to external tools, but it seems like implementing a reference for cases which are unambiguous according to PEP484 would be quite useful, and even helpful to building external tools of that kind.

For cases where something other than recursive isinstance calls are required (e.g. type variables), a helpful error can be raised at runtime.

Mutable values

There was an objection to __instancecheck__ and __subclasscheck__ by BDFL-delegate @markshannon here:

For example,
List[int] and List[str] and mutually incompatible types, yet
isinstance([], List[int]) and isinstance([], List[str))
both return true.

This is a bit tricky for mutable types, since [] is a member of List[X] for all X, but the same list object might cease to be in List[X] when elements are added to it. For immutable types, the answer is unambiguous, so Tuple[*] could always deliver an unambiguous answer.

For List[], Dict[] and other mutable types, my preference would be to check the types of the elements right now, when the method is called, and not at some future time when it may have been mutated. This would be a distinction between mypy and the runtime type checking, but this behavior seems more in keeping with the principle of least surprise, and would accord with the main intended use cases (REPL-based exploration of the type system, and simple runtime type assertions).

Implementation

I think the implementation would be roughly like the following:

def is_member_of_type(obj, tp):
    if has_type_variables(tp):
        raise ValueError('Type variables cannot be determined at runtime')
    if is_union(tp):
        return any(is_member_of_type(obj, s) for s in tp.__args__)
    elif is_tuple(tp):
        ...
    elif ...:
        ...
    else:
        return isinstance(obj, tp)

Include issubtype to match issubclass?

issubtype could make sense as an addition, but it would involve introducing much more of the sorts of logic a typechecker should implement, e.g. the following:

>>> from typing import *
>>> class MyTuple(NamedTuple):
...     foo: int
...     bar: str
... 
>>> issubtype(MyTuple, Tuple[int, str])
True

For this reason, I think it would make more sense to leave issubtype out of typing_inspect.

2.7 reached EOL.
Will this library move away from it and focus on 3.6 as starting version?

Currently only 3.6.1 is supported but maybe it makes sense to support also older versions.

In one of my projects I had to implement this function, I guess it could be useful to share in typing_inspect, and also to provide the equivalent for subclass tests, what do you think ? (I did not find any equivalent in typing nor in typing_inspect)

from typing import Any, Type
from typing_inspect import is_union_type, get_args, get_origin

def robust_isinstance(inst: Any, typ: Type):
    """
    Similar to isinstance, but if 'typ' is a parametrized generic Type, it is first transformed into 
    its base generic class so that the instance check works. It is also robust to Union and Any.

    :param inst: the instance to check against the type
    :param typ: the type
    :return:
    """
    if typ is Any:
        return True
    elif is_union_type(typ):
        typs = get_args(typ)
        if len(typs) > 0:
            return any(robust_isinstance(inst, t) for t in typs)
        else:
            return False
    else:
        return isinstance(inst, get_base_generic_type(typ))

Note that we could also provide a boolean parameter to enable finer-grain subclass/isinstance checks for example for parametrized generic types. For example to make this return False: robust_isinstance([1], List[str])

I'd like to propose

def is_new_type(tt: t.Type[t.Any]) -> bool:
    return getattr(tt, '__supertype__', None) is not None

as an addition to this API. Perhaps it would make sense to also have a way to retrieve that supertype. I have tested that code (currently indirectly) in Python 3.{6-9} and seems to be working.

Opinions :)

I'm wondering if you can create a new release to include the new is_optional_type API 😃

Using the following versions:

python  3.9.1
mypy                   0.812
mypy-extensions        0.4.3
typed-ast              1.4.2
typing-extensions      3.7.4.3
typing-inspect         0.7.1

I get the following import error. I saw a the pull request #76, but it doesn't look like simply moving typing-extensions to 3.7.4.2 will solve it.

  File "C:\Users\<USER>\Miniconda3\envs\jupyter\lib\site-packages\tap\tap.py", line 12, in <module>
    from typing_inspect import is_literal_type, get_args
  File "C:\Users\<USER>\Miniconda3\envs\jupyter\lib\site-packages\typing_inspect.py", line 17, in <module>
    from typing_extensions import _TypedDictMeta as _TypedDictMeta_TE
ImportError: cannot import name '_TypedDictMeta' from 'typing_extensions' (C:\Users\<USER>\Miniconda3\envs\jupyter\lib\site-packages\typing_extensions.py)

I was hoping that using this package would make it easier to migrate between Python versions as it changes typing implementation. So I started using it with Python 3.6 but now that I tried to use working code on Python 3.6 with Python 3.7 it seems there are differences. For example, the following code:

import typing
import typing_inspect

A = typing.TypeVar('A')
B = typing.TypeVar('B')
C = typing.TypeVar('C')

class Base(typing.Generic[A, B]):
    pass

class Foo(Base[A, None]):
    pass

class Bar(Foo[A], typing.Generic[A, C]):
    pass

class Baz(Bar[float, int]):
    pass

print("Bar", typing_inspect.get_origin(Bar))
print("Baz", typing_inspect.get_origin(Baz))
print("Base", typing_inspect.get_origin(Base))
print("Base[float, int]", typing_inspect.get_origin(Base[float, int]))
print("Foo", typing_inspect.get_origin(Foo))
print("Foo[float]", typing_inspect.get_origin(Foo[float]))

In Python 3.6 outputs:

Bar __main__.Bar
Baz __main__.Baz
Base __main__.Base
Base[float, int] __main__.Base
Foo __main__.Foo
Foo[float] __main__.Foo

While in Python 3.7:

Bar None
Baz None
Base None
Base[float, int] <class '__main__.Base'>
Foo None
Foo[float] <class '__main__.Foo'>

I think ideally they should behave the same.

If typing_inspect will become part of typing in the future (or some parts of it), then we can merge it with the typing stub.

I'm not sure if this is appropriate but:

1. You've put "The API is still experimental, if you have ideas/suggestions please open an issue on tracker."
2. I found this library half-way through developing my own type inspections library, and so thought we could bounce ideas, or more.

Would it be better to add a layer of abstraction over __origin__ and __extra__ and have a function, say get_typing, return both? (get_typing(Mapping) == (Mapping, collections.abc.Mapping))

Why I thought it would:

1. In Python 3.7 __origin__ is a different type and so makes users have to use the following to get the same type. (This has been raised here as well #27)

   get_origin = get_extra if sys.version_info[:2] < (3, 7) else get_origin

2. It doesn't seem like 3.7 has an __extra__ or any easy way to get typing.X from typing.X[T]. And so there's no easy way to get the 'typing type', without the library explicitly allowing that.

In my library I only have get_typing rather than get_origin and get_last_origin.
Am I being short sighted by thinking you don't need get_origin in Py3.6?

Likewise with get_last_args.

I've implemented get_bases, get_parents, get_mro and get_mro_orig which would fix #6. But unfortunately uses the other functions that may not work exactly the same way yours do. And also uses some code you may not like (links.py and _conv).

Would you be interested in merging our efforts?

We'd like to use the features I added in #39 and #40 within Instagram, and it'd be easier to pull the changes in if there's a PyPI release containing them. Are there plans to cut a 0.4.1 or 0.5.0 release?

Hi! Thank you for the great package!

get_origin(Dict) returns None only in Python 3.9. Is this an expected behaviour? 🤔

Python 3.9.0 (default, Nov 11 2020, 21:02:15)
[Clang 12.0.0 (clang-1200.0.32.21)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> from typing_inspect import get_origin
>>> from typing import Dict
>>> get_origin(Dict)
>>> print(get_origin(Dict))
None
>>> print(get_origin(Dict[int, str]))
<class 'dict'>

The approximate idea is to do something like this:

def reveal_type(obj):
    print(get_generic_type(obj))
    return obj

Then in the stub for typing_inspect we can write something like:

from builtins import reveal_type as reveal_type

Any chance you could publish a new release? I noticed master contains additional support for Python 3.10 features like types.UnionType.

    from typing import (
ImportError: cannot import name 'CallableMeta' from 'typing' (/usr/lib64/python3.7/typing.py)

I assume this is because of typing module changes.

Hello,

is this

import typing
import typing_inspect
typing_inspect.is_generic_type(typing.SupportsInt) is True

supposed to happen?
Thanks

In 3.7:

>>> typing_inspect.get_args(typing.List[typing.Union[str, bytes]])
(typing.Union[str, bytes],)

In 3.6:

>>> typing_inspect.get_args(typing.List[typing.Union[str, bytes]])
((typing.Union, <class 'str'>, <class 'bytes'>),)

In [7]: sys.version                                                                                                                                                                  
Out[7]: '3.9.0b1 (v3.9.0b1:97fe9cfd9f, May 18 2020, 20:39:28) \n[Clang 6.0 (clang-600.0.57)]'

In [8]: from typing import List                                                                                                                                                      

In [9]: typing_inspect.is_generic_type(List)                                                                                                                                         
Out[9]: False

Apparently in 3.9 List became an instance of _SpecialGenericAlias instead of _GenericAlias, which is what typing_inspect checks for.

In is_new_type there is an explicit check for the module name being either typing or typing_extensions, so types created via marshmallow_dataclass.NewType are no longer recognised, and therefore cannot be used in marshmallow dataclasses.

This check on __module__ makes it impossible to define and use any custom NewType implementations, even if they are intended to be accepted as NewType.

As the pre-3.10 implementations of typing.NewType define only two extra attributes __name__ and __supertype__, I'd suggest a check on these two (besides __qualname__), and on their types:

... isinstance(getattr(tp, '__supertype__', None), type) and isinstance(getattr(tp, '__name__'), str) and ...

This would check everything that we can be sure of, and wouldn't block the custom NewType-implementations.

Is typing_inspect.get_args expected to also return metadata args for Annotated?

Python 3.9.5:

>>> import typing
>>>
>>> A = typing.Annotated[int, "some_metadata"]
>>>
>>> typing.get_args(A)
(<class 'int'>, 'some_metadata')

Python 3.8.6:

>>> import typing_extensions  # 3.10.0.2
>>> import typing_inspect  # 0.7.1
>>> 
>>> A = typing_extensions.Annotated[int, "some_metadata"]
>>>
>>> typing_inspect.get_args(A)
(<class 'int'>,)

In Python 3.6.6, get_origin(typing.Dict) returns typing.Dict, but in Python 3.7.0 it returns dict.
Is that by design?

Hey, any chance we can get a new release of this on pip with the 3.7 support a bit early?
3.7-rc is out - i cant see Dataclasses changing, and it would save me a lot of work if was released (am building 3.7 updates and setup.py does not play well with git+url for dependencies of dependencies)

I had to modify your code so that it runs correctly on python 3.5.3 (one of our production environment runs on AzureML where the conda environment was 3.5.3).

I take this opportunity to submit a PR in case you're interested.

If I want to compare two types which are like:

class Parent:
  pass

class Child(Parent):
  pass

class Foo(Sequence[Child]):
  pass

compare(Foo, Sequence[Parent])

It seems hard to do any comparison:

• inspect.getmro of Foo does not return Sequence[str]
• I can use get_generic_bases to manually traverse parent classes, but doing that in MRO order is tricky (#6)
• but what would ideally be is that I would have a method which would return the hierarchy of types and their arguments, so that I could find out that Foo inherits from Sequence[Child], and that there is an argument Child and then compare that with Sequence[Parent]; it seems pretty tricky with current typing_inspect to get from Foo to Child as an argument, no?

It should determine if t is meaningful to use in an isinstance check.

We've got some code in 3.6 that was simply doing:

isinstance(t, type)

for this purpose. It worked. typing.List as well as all your basic types passed correctly.

It blows up in 3.7 onwards because the type of typing.List and friends has changed from type to typing._GenericAlias or typing._SpecialGenericAlias.

Not having a simple way to answer the question of is something is a type that is meaningfully usable for isinstance checks from other things is frustrating. (I'd argue that such a basic thing should even go in the Python stdlib, but this module may be the best playground for that to start in)

It would be nice to somehow be able to inspect the specified type annotations of a TypedDict class instance. Let's say you have the following typed dictionary:

class FooRequired(TypedDict):
    bar: str

class Foo(FooRequired, total=False):
    baz: int

Providing:

• required_dict_keys(Foo) == {'bar': str}
• optional_dict_keys(Foo) == {'baz': int}
• dict_keys(Foo) == {'bar': str, 'baz': int}

Not necessarily this API (it was just the first thing that I imagined), but something that would result in the same capabilities.

The rational is that this could enable TypedDict classes to be used as schema specification for untrusted JSON/YAML/config sources.

Thanks in advance!

Collection types are an important special case of Generic types. In parsyfiles I ended up writing functions to easily access

• the type of the key in a Mapping
• the type of the items in Mapping (values, not keys), and Iterable (but I should probably use Container instead)

The indent was to at least support Dict[KT, VT], List[VT], Set[VT], and Tuple[VT1, VT2, VT3...].

Maybe such as API could be of some interest for typing_inspect users ?

For details about how I used it, please see _extract_collection_base_type in https://github.com/smarie/python-parsyfiles/blob/master/parsyfiles/type_inspection_tools.py. Some tests are available here

However the way I did it is probably not the best for an api, probably it would be better to provide individual functions such as get_mapping_inner_types and get_container_inner_types. An interesting thing to keep, though, is the capability to detect underspecified types such as Dict (and not Dict[A, B]) and to throw a TypeInformationRequiredError. See above file for details.

After updating typing_extensions from 4.0.1 to 4.1.1 the following test started failing.

_____________ GetUtilityTestCase.test_typed_dict_typing_extension ______________

self = <test_typing_inspect.GetUtilityTestCase testMethod=test_typed_dict_typing_extension>

    @skipUnless(PY36, "Python 3.6 required")
    def test_typed_dict_typing_extension(self):
        TDOld = TETypedDict("TDOld", {'x': int, 'y': int})
>       self.assertEqual(typed_dict_keys(TDE), {'x': int, 'y': int})
E       AssertionError: None != {'x': <class 'int'>, 'y': <class 'int'>}

test_typing_inspect.py:451: AssertionError

is_generic_type returns True for a non-generic Protocol:

Tested on Python 3.8.2 and Python 3.6.10:

from typing import Generic

from typing_extensions import Protocol, runtime
from typing_inspect import NEW_TYPING, get_parameters, is_generic_type


@runtime
class SomeProtocol(Protocol):
    def a_method(self) -> str:
        ...


assert is_generic_type(SomeProtocol)

if NEW_TYPING:
    assert isinstance(SomeProtocol, type)
    assert issubclass(SomeProtocol, Generic)
else:
    from typing import GenericMeta
    assert isinstance(SomeProtocol, GenericMeta)


# typing._check_generic raises 
# TypeError: <class '__main__.SomeProtocol'> is not a generic class
SomeProtocol[int] 

implementation of is_generic_type

python 3.7

>>> from typing import List
>>> from typing_inspect import get_parameters
>>> get_parameters(List)
(~T,)

python 3.10.6

>>> from typing import List
>>> from typing_inspect import get_parameters
>>> get_parameters(List)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/james.robson/code/pio_utilities/.venv/lib/python3.10/site-packages/typing_inspect.py", line 414, in get_parameters
    return tp.__parameters__
  File "/usr/lib/python3.10/typing.py", line 984, in __getattr__
    raise AttributeError(attr)
AttributeError: __parameters__

When you don't provide type parameters the typing classes lacks the __parameters__, _paramspec_tvars and _typevar_types members.

Hi ilevkivskyi,

It seems is_optional_type returns True for Union[Optional[xxx], yyy] as shown below.

>>> from typing_inspect import is_optional_type
>>> from typing import Union, Optional
>>> is_optional_type(Union[Optional[int], str])
True

Tested with typing_inspect 0.4.0

Is this an intended behaviour or bug? How should I do to distinguish Union[Optional[xxx], yyy] from Optional[zzz] by is_optional_type and is_union_type?

Thank you,

Python version 3.7.

Minimal example

from typing import Generic, TypeVar
import typing_inspect
    
T = TypeVar( "T" )
    
class MyGenericBase( Generic[T] ):
    pass
    
class MyGeneric( MyGenericBase[int] ):
    pass
    
print( typing_inspect.get_args( MyGeneric ) )

Actual result

( )

Expected result

( int, )

Is it possible to use typing_inspect to achieve the following?

from typing import Any, List, TypeVar

import typing_inspect


class C:
    x: int

    def __init__(self, x: int) -> None:
        self.x = x


T = TypeVar("T")


class L(List[T]):
    def append_new(self, *args: Any, **kw: Any) -> T:
        item_class = ???(T)
        self.append(item_class(*args, **kw))
        return self[-1]


l = L[C]()
print(l.append_new(42).x)