google / pytype Goto Github PK
View Code? Open in Web Editor NEWA static type analyzer for Python code
Home Page: https://google.github.io/pytype
License: Other
A static type analyzer for Python code
Home Page: https://google.github.io/pytype
License: Other
When analyzing the script http://paste.fedoraproject.org/422337/30903421/ (yes, it is wrong), pytype
crashes (see attached output with -v 4
). I don't think crash is very a good idea, is it?
From https://travis-ci.org/python/typeshed/jobs/212532837:
Running pytype tests...
pytd error processing "stdlib/2and3/cgi.pyi":
File: "stdlib/2and3/cgi.pyi", line 107
def values(self, key: Any) -> Any: ... # type: ignore
^
ParseError: syntax error, unexpected TYPECOMMENT, expecting DEF or '@'
Type comments should be allowed here.
I just got some weird results while running typeshed suite, see travis job, for typeshed#418.
pytd error processing "stdlib/2and3/mmap.pyi":
Error while parsing pyi:
File "stdlib/2and3/mmap.pyi", line 0
# Stubs for mmap
^
SyntaxError: Incompatible templates: (TemplateItem(NamedType('mmap')),) missing parameters from (TemplateItem(NamedType('bytes')),)
pytd error processing "stdlib/2.7/StringIO.pyi":
Error while parsing pyi:
File "stdlib/2.7/StringIO.pyi", line 0
# Stubs for StringIO (Python 2)
^
SyntaxError: Name 'typing.AnyStr' must be defined as a TypeVar
Ran pytype with 226 pyis, got 2 errors.
Hi, cool project! (As is any effort to bring static typing to Python, in my book.) What's the status of this project?
I tried throwing it at a couple of real code bases and it doesn't seem to be working. I can start debugging / posting issues, or if it's just not ready / meant for public consumption yet, I can save everyone's time and hold off. Or if there's some well-defined scope / set of limitations you can summarize, that would be immensely useful.
Also, is there some roadmap or timeline defined for this project? Esp. how that timeline compares to those of other on-going efforts such as mypy.
Lastly - is this a 20%/side project?
Thanks!
Python 3.5
Traceback (most recent call last):
File "/usr/local/bin/pytype", line 19, in
from pytype import infer
File "/usr/local/lib/python3.5/site-packages/pytype/infer.py", line 6, in
import StringIO
ImportError: No module named 'StringIO'
In PEP 484, there is a section on conditional stubs.
For example, the next code is a valid stub code
import sys
if sys.platform == 'win32':
def windows_def_only() -> None: ...
But currently, running pytd
on it will fail with
Error while parsing pyi:
File "/tmp/asd.pyi", line 3
if sys.platform == 'win32':
^
SyntaxError: Unexpected 'NAME'
$ pip install -U git+git://github.com/google/pytype
Collecting git+git://github.com/google/pytype
Cloning git://github.com/google/pytype to /tmp/pip-SFKRsw-build
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-SFKRsw-build/setup.py", line 22, in <module>
assert 'typeshed/stdlib/2.7/*.pyi' in typeshed
AssertionError
Suppose I have a classes.pyi
like this:
class A(object):
def foo(self) -> A: ...
class B(A):
pass
and then analyze code like this:
import classes
classes.B.foo()
Now, pytype will complain that self
is of the wrong type.
ryan@DevPC-LX /media/ryan/stuff/pytype master $ cat x.py
print(f'{a}')
ryan@DevPC-LX /media/ryan/stuff/pytype master $ pytype -V 3.6 x.py
Traceback (most recent call last):
File "/usr/local/bin/pytype", line 272, in <module>
sys.exit(main(sys.argv) or 0)
File "/usr/local/bin/pytype", line 224, in main
return _run_pytype(options)
File "/usr/local/bin/pytype", line 265, in _run_pytype
errors_csv_file=errors_csv_file)
File "/usr/local/bin/pytype", line 172, in process_one_file
options=options)
File "/usr/local/bin/pytype", line 52, in check_pyi
cache_unknowns=options.cache_unknowns)
File "/usr/local/lib/python2.7/dist-packages/pytype/infer.py", line 585, in check_types
py_src, py_filename, init_maximum_depth, run_builtins)
File "/usr/local/lib/python2.7/dist-packages/pytype/vm.py", line 544, in run_program
node, f_globals, f_locals, builtin_names = self.preload_builtins(node)
File "/usr/local/lib/python2.7/dist-packages/pytype/vm.py", line 514, in preload_builtins
builtins_code = self.compile_src(src)
File "/usr/local/lib/python2.7/dist-packages/pytype/vm.py", line 499, in compile_src
filename=filename, mode=mode)
File "/usr/local/lib/python2.7/dist-packages/pytype/pyc/pyc.py", line 139, in compile_src
code = parse_pyc_string(pyc_data)
File "/usr/local/lib/python2.7/dist-packages/pytype/pyc/pyc.py", line 110, in parse_pyc_string
return parse_pyc_stream(StringIO.StringIO(data))
File "/usr/local/lib/python2.7/dist-packages/pytype/pyc/pyc.py", line 90, in parse_pyc_stream
python_version = magic.magic_word_to_version(magic_word)
File "/usr/local/lib/python2.7/dist-packages/pytype/pyc/magic.py", line 105, in magic_word_to_version
return PYTHON_MAGIC[magic_word]
KeyError: 3377
ryan@DevPC-LX /media/ryan/stuff/pytype master $
When running the latest pytype on this file I get a huge traceback:
Traceback (most recent call last):
File "/Users/guido/v/bin/pytype", line 202, in <module>
sys.exit(main(sys.argv) or 0)
File "/Users/guido/v/bin/pytype", line 196, in main
ret = process_one_file(input_filename, output_filename, options)
File "/Users/guido/v/bin/pytype", line 151, in process_one_file
options=options)
File "/Users/guido/v/bin/pytype", line 80, in generate_pytd
maximum_depth=(1 if options.quick else None))
File "/Users/guido/v/lib/python2.7/site-packages/pytype/infer.py", line 576, in infer_types
loc, defs, builtin_names = tracer.run_program(src, filename, run_builtins)
File "/Users/guido/v/lib/python2.7/site-packages/pytype/vm.py", line 953, in run_program
node, f_globals, _ = self.run_bytecode(node, code, f_globals, f_locals)
File "/Users/guido/v/lib/python2.7/site-packages/pytype/vm.py", line 910, in run_bytecode
node, _ = self.run_frame(frame, node)
File "/Users/guido/v/lib/python2.7/site-packages/pytype/vm.py", line 252, in run_frame
state = self.run_instruction(op, state)
File "/Users/guido/v/lib/python2.7/site-packages/pytype/vm.py", line 207, in run_instruction
state = bytecode_fn(state, op)
File "/Users/guido/v/lib/python2.7/site-packages/pytype/vm.py", line 2021, in byte_CALL_FUNCTION
return self.call_function_from_stack(state, op.arg, [])
File "/Users/guido/v/lib/python2.7/site-packages/pytype/vm.py", line 1079, in call_function_from_stack
state, func, posargs, namedargs, starargs, starstarargs)
File "/Users/guido/v/lib/python2.7/site-packages/pytype/vm.py", line 1010, in call_function_with_state
starargs, starstarargs, fallback_to_unsolvable)
File "/Users/guido/v/lib/python2.7/site-packages/pytype/vm.py", line 1037, in call_function
node, funcv, posargs, namedargs or {}, starargs, starstarargs)
File "/Users/guido/v/lib/python2.7/site-packages/pytype/abstract.py", line 1974, in call
node_after_call, ret = self.vm.run_frame(frame, node)
File "/Users/guido/v/lib/python2.7/site-packages/pytype/vm.py", line 252, in run_frame
state = self.run_instruction(op, state)
File "/Users/guido/v/lib/python2.7/site-packages/pytype/vm.py", line 207, in run_instruction
state = bytecode_fn(state, op)
File "/Users/guido/v/lib/python2.7/site-packages/pytype/vm.py", line 1638, in byte_LOAD_ATTR
state, val = self.load_attr(state, obj, name)
File "/Users/guido/v/lib/python2.7/site-packages/pytype/vm.py", line 1184, in load_attr
node, result = self._retrieve_attr(state.node, obj, attr)
File "/Users/guido/v/lib/python2.7/site-packages/pytype/vm.py", line 1167, in _retrieve_attr
node2, attr_var = val.data.get_attribute(node, attr, val)
File "/Users/guido/v/lib/python2.7/site-packages/pytype/abstract.py", line 2190, in get_attribute
mod = self.vm.import_module(full_name, 0) # 0: absolute import
File "/Users/guido/v/lib/python2.7/site-packages/pytype/vm.py", line 1322, in import_module
ast = self.loader.import_name(name)
File "/Users/guido/v/lib/python2.7/site-packages/pytype/load_pytd.py", line 201, in import_name
mod = self._load_builtin("stdlib", module_name)
File "/Users/guido/v/lib/python2.7/site-packages/pytype/load_pytd.py", line 162, in _load_builtin
typeshed.parse_type_definition(subdir, module_name, version))
File "/Users/guido/v/lib/python2.7/site-packages/pytype/pytd/typeshed.py", line 77, in parse_type_definition
python_version=python_version).Replace(name=module)
File "/Users/guido/v/lib/python2.7/site-packages/pytype/pytd/utils.py", line 342, in ParsePyTD
ast = visitors.LookupClasses(ast, builtins.GetBuiltinsPyTD())
File "/Users/guido/v/lib/python2.7/site-packages/pytype/pytd/parse/visitors.py", line 542, in LookupClasses
module.Visit(VerifyLookup())
File "/Users/guido/v/lib/python2.7/site-packages/pytype/pytd/parse/node.py", line 261, in _VisitNode
new_child = _VisitNode(child, visitor, *args, **kwargs)
File "/Users/guido/v/lib/python2.7/site-packages/pytype/pytd/parse/node.py", line 226, in _VisitNode
new_child = _VisitNode(child, visitor, *args, **kwargs)
File "/Users/guido/v/lib/python2.7/site-packages/pytype/pytd/parse/node.py", line 261, in _VisitNode
new_child = _VisitNode(child, visitor, *args, **kwargs)
File "/Users/guido/v/lib/python2.7/site-packages/pytype/pytd/parse/node.py", line 276, in _VisitNode
new_node = visitor.Visit(new_node, *args, **kwargs)
File "/Users/guido/v/lib/python2.7/site-packages/pytype/pytd/parse/visitors.py", line 82, in Visit
self, node, *args, **kwargs)
File "/Users/guido/v/lib/python2.7/site-packages/pytype/pytd/parse/visitors.py", line 558, in VisitClassType
raise ValueError("Unresolved class: %r" % node.name)
ValueError: Unresolved class: 'False'
I have a test file like this (x.py):
class C:
def foo(self):
return 0
class I:
def bar(self):
return 0
The generated stub is:
class C:
I = ... # type: type
def foo(self) -> int: ...
I would have expected
class C:
class I:
def bar(self) -> int: ...
def foo(self) -> int: ...
Currently, pytype generates pyi e.g. like this:
def f(x: some.module.Foo) -> Any
this is missing the imports
from typing import Any
import some.module
.
pytype can't be packaged without tags or releases
def f(a: int, *, b: int = ...) -> None: ...
gives with pytd
Error while parsing pyi:
File "asd3.pyi", line 1
def f(a: int, *, b: int = ...) -> None: ...
^
SyntaxError: Unexpected 'COMMA'
But this notation was introduced in PEP 3102, and IMO should be valid in stub files (it is already used in typeshed).
It ought to be possible to install pytype using pip by running
python -m pip install -U .
but doing so causes an error message.
I'm attempting to use pytype and it appears it doesn't support Python 3( complains about no StringIO module). Any plans to support Python 3? or, perhaps add some documentation about which version of Python are supported.
In typeshed, there is some stubs which use @abstractmethod
to specify that a child class should implement a given method. For now, it fails with
Error while parsing pyi:
File "stdlib/2and3/distutils/cmd.pyi", line 10
@abstractmethod
^
SyntaxError: Decorator 'abstractmethod' not supported
in typeshed#418.
It is used elsewhere, grep -r @abstractmethod | wc -l
in typeshed gives 168 results.
I have a function like this:
def foo(a):
return a[:10].lower()
The generated stub is:
def foo(a: bytes or str or unicode or Dict[slice, bytearray or bytes or str or unicode or List[?, ...] or Tuple[?, ...]]) -> bytearray or bytes or str or unicode
Now, it's cute that it figures out that if the argument were a dict with slice keys and string values it might work, but in fact slice does not implement hash() so Dict[slice, ...]
is nonsense.
I'm also unclear on why it thinks that the value could be a List or Tuple, since neither of those has a lower() method.
This is causing the typeshed CI to reject the following valid code:
https://github.com/markshannon/typeshed/blob/362bedecd9778196e44d4c88acc39313bb38a6ba/stdlib/2/os/path.pyi
My typeshed branch https://github.com/JelleZijlstra/typeshed/tree/fileinput crashes pytype with
Traceback (most recent call last):
File "/home/travis/virtualenv/python2.7.9/bin/pytd", line 88, in <module>
main()
File "/home/travis/virtualenv/python2.7.9/bin/pytd", line 64, in main
parsed = parser.parse_string(sourcecode, filename=filename_in)
File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/pytype/pyi/parser.py", line 833, in parse_string
src, name, filename)
File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/pytype/pyi/parser.py", line 293, in parse
defs = parser_ext.parse(self, src)
File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/pytype/pyi/parser.py", line 556, in new_type
return self._parameterized_type(base_type, parameters)
File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/pytype/pyi/parser.py", line 631, in _parameterized_type
return pytd.GenericType(base_type=base_type, parameters=parameters)
File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/pytype/pytd/parse/node.py", line 107, in __init__
self._CHECKER.check(*args, **kwargs)
File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/pytype/pytd/parse/preconditions.py", line 139, in check
self._check_arg(condition, name, value)
File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/pytype/pytd/parse/preconditions.py", line 146, in _check_arg
raise PreconditionError("argument=%s: %s." % (name, e.message))
pytype.pytd.parse.preconditions.PreconditionError: argument=base_type: (actual=CallableType, expected=NamedType) or (actual=CallableType, expected=ClassType).
I just see this kind of declaration being generated:
def to_dataframe(self, ...) -> Any: ...
def to_file(self, path, ...) -> Any: ...
def to_view(self, view_name) -> Any: ...
I..e. with optional args it just generates '...' and with positional args it just outputs the arg name, and the return types are just Any. This seems to be so generic as to not be useful at all. What am I missing here?
I'm just running this like:
pytype foo.py -o foo.pyi
pytype sometimes emits two function definitions for one in the input. E.g. input:
def bar(x):
return x+1
bar(0)
and output:
def bar(x: int) -> int
def bar(x: bool or complex or float or int or long) -> bool or complex or float or int or long
from typing import Mapping, TypeVar
A = TypeVar('A', Mapping[str, str])
gives with pytd
Error while parsing pyi:
File "/tmp/asd2.py", line 3
A = TypeVar('A', Mapping[str, str])
^
SyntaxError: Unexpected 'LBRACKET'
Found while doing typeshed#433.
E.g. in ast.pyi, "parse" is defined like this:
def parse(source, filename = ..., mode = ..., *args, **kwargs) -> AST: ...
However, when analyzing the code
import ast
ast.parse("True")
pytype complains about two missing arguments (1 instead of 3).
pytype is complaining about invalid syntax for use of print('something') in Python 3. It's been installed on Python 2.7, of course.
pytype doesn't generate valid Python code for pyi. E.g. it generates
def f() -> None
instead of
def f() -> None: ...
.
Also, it uses ?
for Any
, omits certain imports, and uses x or y
instead of Union[x, y]
.
pytype tends to complain about functions being passed as arguments for a "Callable" parameter:
File "test.py", line 43, in parse:
Function tokenize.generate_tokens was called with the wrong arguments
Expected: (readline: typing.Callable)
Actually passed: (readline: readline)
Simplified example that causes the same warning:
import tokenize
def f():
pass
tokenize.generate_tokens(f)
Any reason this isn't released on PyPI? Would make it easier to install for users (auto install dependencies via pip
).
Why would you ever write "bool or int" (or "Union[bool, int]" in PEP 484 speak)? Since bool is a subclass of int, this can be simplified to just "int".
I have a module containing the line
from __future__ import print_function
and I invoke pytype as
pytype ./yo.py -o -
Now the generated stub has invalid imports:
import ..__future__
print_function = ... # type: ..__future__._Feature
https://github.com/rowillia/werkzeug/blob/master/werkzeug/datastructures.py
$ pytype werkzeug/datastructures.py:werkzeug/datastructures.pyi
Traceback (most recent call last):
File "/usr/local/bin/pytype", line 271, in <module>
sys.exit(main(sys.argv) or 0)
File "/usr/local/bin/pytype", line 228, in main
return _run_pytype(options)
File "/usr/local/bin/pytype", line 264, in _run_pytype
errors_csv_file=errors_csv_file)
File "/usr/local/bin/pytype", line 182, in process_one_file
options=options)
File "/usr/local/bin/pytype", line 89, in generate_pyi
cache_unknowns=options.cache_unknowns)
File "/usr/local/lib/python2.7/site-packages/pytype/infer.py", line 637, in infer_types
tracer.exitpoint = tracer.analyze(loc, defs, builtin_names, maximum_depth)
File "/usr/local/lib/python2.7/site-packages/pytype/infer.py", line 183, in analyze
self.analyze_toplevel(node, defs, ignore)
File "/usr/local/lib/python2.7/site-packages/pytype/infer.py", line 173, in analyze_toplevel
node2 = self.analyze_class(value, node)
File "/usr/local/lib/python2.7/site-packages/pytype/infer.py", line 152, in analyze_class
node2 = self.analyze_method_var(name, b, node)
File "/usr/local/lib/python2.7/site-packages/pytype/infer.py", line 119, in analyze_method_var
node2 = self.analyze_method(val, node)
File "/usr/local/lib/python2.7/site-packages/pytype/infer.py", line 111, in analyze_method
node, fvar, args, kws, starargs, starstarargs)
File "/usr/local/lib/python2.7/site-packages/pytype/infer.py", line 93, in call_function_in_frame
state, var, args, kwargs, starargs, starstarargs)
File "/usr/local/lib/python2.7/site-packages/pytype/vm.py", line 630, in call_function_with_state
starargs, starstarargs, fallback_to_unsolvable)
File "/usr/local/lib/python2.7/site-packages/pytype/vm.py", line 657, in call_function
node, funcv, posargs, namedargs or {}, starargs, starstarargs)
File "/usr/local/lib/python2.7/site-packages/pytype/abstract.py", line 2717, in call
namedargs, starargs, starstarargs)
File "/usr/local/lib/python2.7/site-packages/pytype/abstract.py", line 2558, in call
node_after_call, ret = self.vm.run_frame(frame, node)
File "/usr/local/lib/python2.7/site-packages/pytype/vm.py", line 186, in run_frame
state = self.run_instruction(op, state)
File "/usr/local/lib/python2.7/site-packages/pytype/vm.py", line 141, in run_instruction
state = bytecode_fn(state, op)
File "/usr/local/lib/python2.7/site-packages/pytype/vm.py", line 1721, in byte_CALL_FUNCTION
return self.call_function_from_stack(state, op.arg, [])
File "/usr/local/lib/python2.7/site-packages/pytype/vm.py", line 707, in call_function_from_stack
state, func, posargs, namedargs, starargs, starstarargs)
File "/usr/local/lib/python2.7/site-packages/pytype/vm.py", line 630, in call_function_with_state
starargs, starstarargs, fallback_to_unsolvable)
File "/usr/local/lib/python2.7/site-packages/pytype/vm.py", line 657, in call_function
node, funcv, posargs, namedargs or {}, starargs, starstarargs)
File "/usr/local/lib/python2.7/site-packages/pytype/abstract.py", line 1632, in call
starargs, starstarargs)
File "/usr/local/lib/python2.7/site-packages/pytype/abstract.py", line 1690, in _call_with_view
starargs, starstarargs, record_call=True)
File "/usr/local/lib/python2.7/site-packages/pytype/abstract.py", line 1749, in find_matching_signature
starargs, starstarargs, record_call)
File "/usr/local/lib/python2.7/site-packages/pytype/abstract.py", line 1397, in call_with_view
r = self._call_with_values(node, arg_dict, view)
File "/usr/local/lib/python2.7/site-packages/pytype/abstract.py", line 1433, in _call_with_values
subst = self._compute_subst(node, arg_dict, view)
File "/usr/local/lib/python2.7/site-packages/pytype/abstract.py", line 1465, in _compute_subst
subst = _match_value_against_type(actual, formal, subst, node, view)
File "/usr/local/lib/python2.7/site-packages/pytype/abstract.py", line 137, in _match_value_against_type
return left.match_against_type(other_type, subst, node, view)
File "/usr/local/lib/python2.7/site-packages/pytype/abstract.py", line 471, in match_against_type
raise NotImplementedError("Matching not implemented for %s", type(self))
NotImplementedError: ('Matching not implemented for %s', <class 'pytype.abstract.SuperInstance'>)
Pytype crashed on the following code snippet when run with pytype test.py -V 3.5
with open("a.txt", "w") as f:
f.write("Hello")
Traceback (most recent call last):
File "/usr/local/bin/pytype", line 263, in <module>
sys.exit(main(sys.argv) or 0)
File "/usr/local/bin/pytype", line 210, in main
return _run_pytype(options)
File "/usr/local/bin/pytype", line 251, in _run_pytype
errors_csv_file=errors_csv_file)
File "/usr/local/bin/pytype", line 139, in process_one_file
options=options)
File "/usr/local/bin/pytype", line 51, in check_pyi
cache_unknowns=options.cache_unknowns)
File "/usr/local/lib/python2.7/site-packages/pytype/infer.py", line 640, in check_types
py_src, py_filename, init_maximum_depth, run_builtins)
File "/usr/local/lib/python2.7/site-packages/pytype/vm.py", line 580, in run_program
code = self.compile_src(src, filename=filename)
File "/usr/local/lib/python2.7/site-packages/pytype/vm.py", line 532, in compile_src
return blocks.process_code(code)
File "/usr/local/lib/python2.7/site-packages/pytype/blocks.py", line 220, in process_code
return pyc.visit(code, OrderCodeVisitor())
File "/usr/local/lib/python2.7/site-packages/pytype/pyc/pyc.py", line 192, in visit
return visitor.visit_code(c)
File "/usr/local/lib/python2.7/site-packages/pytype/blocks.py", line 216, in visit_code
return order_code(code)
File "/usr/local/lib/python2.7/site-packages/pytype/blocks.py", line 206, in order_code
bytecodes = opcodes.dis_code(code)
File "/usr/local/lib/python2.7/site-packages/pytype/pyc/opcodes.py", line 1045, in dis_code
co_firstlineno=code.co_firstlineno)
File "/usr/local/lib/python2.7/site-packages/pytype/pyc/opcodes.py", line 1033, in dis
return _dis(data, mapping, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/pytype/pyc/opcodes.py", line 999, in _dis
cls = mapping[opcode]
KeyError: 82
For this python3.5 script, I was getting urllib
errors. Even when I used --nofail
, I was still getting inspect
errors. Also, some messages printed by pytype
, such as about super()
, etc. were not relevant. Note that the script currently doesn't have any package dependencies, and so it's trivial to test. Thanks.
Given this function:
def foo(a):
return a[:10].lower()
pytype produces:
def foo(a: Union[str, unicode, Dict[slice, Union[bytearray, str, unicode, List[Any], Tuple[Any, ...]]]]) -> Union[bytearray, str, unicode]: ...
List[Any]
and Tuple[Any, ...]
are incorrect.
They stem from the following equations:
if ~unknown2 == bytearray then ~unknown4 == bytearray
if ~unknown2 == unicode then ~unknown4 == unicode
if ~unknown2 == str then ~unknown4 == str
if ~unknown0 == unicode then ~unknown2 == unicode
if ~unknown0 == list then (~unknown2 == list & ~unknown2.list.T == ~unknown0.list.T)
if ~unknown0 == dict then (~unknown0.dict.K == slice & ~unknown2 == ~unknown0.dict.V)
if ~unknown0 == tuple then (~unknown2 == tuple & ~unknown2.tuple.T == ~unknown0.tuple.T)
if ~unknown0 == str then ~unknown2 == str
It doesn't apply the filtering ~unknown2 == ~unknown0.dict.V
, otherwise the value parameter of the outer dict would be limited to bytearray, unicode, str
. The reason it doesn't apply that filtering is because the ~unknown0.dict.V
clause isn't necessary to solve the system. (~unknown0
might be something other than a dict
)
The solver needs to establish the relationship between ~unknown0 == dict
and all ~unknown0.dict.*
implications, possibly by moving the latter into the ground truth.
% ~/Programming/pytype/env/bin/pytype mercurial/util.py
Traceback (most recent call last):
File "/Users/augie/Programming/pytype/env/bin/pytype", line 264, in
sys.exit(main(sys.argv) or 0)
File "/Users/augie/Programming/pytype/env/bin/pytype", line 218, in main
return _run_pytype(options)
File "/Users/augie/Programming/pytype/env/bin/pytype", line 258, in _run_pytype
ret = process_one_file(input_filename, output_filename, options)
File "/Users/augie/Programming/pytype/env/bin/pytype", line 177, in process_one_file
options=options)
File "/Users/augie/Programming/pytype/env/bin/pytype", line 86, in generate_pyi
maximum_depth=(1 if options.quick else None))
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/infer.py", line 586, in infer_types
loc, defs, builtin_names = tracer.run_program(src, filename, run_builtins)
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/vm.py", line 906, in run_program
node, f_globals, _ = self.run_bytecode(node, code, f_globals, f_locals)
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/vm.py", line 863, in run_bytecode
node, _ = self.run_frame(frame, node)
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/vm.py", line 226, in run_frame
state = self.run_instruction(op, state)
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/vm.py", line 181, in run_instruction
state = bytecode_fn(state, op)
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/vm.py", line 2054, in byte_IMPORT_NAME
name, abstract.get_atomic_python_constant(level))
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/vm.py", line 1292, in import_module
ast = self.loader.import_name(name)
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/load_pytd.py", line 175, in import_name
self._lookup_all_classes()
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/load_pytd.py", line 133, in _lookup_all_classes
self._finish_ast(module.ast)
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/load_pytd.py", line 119, in _finish_ast
ast.Visit(visitors.VerifyLookup())
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/pytd/parse/node.py", line 261, in _VisitNode
new_child = _VisitNode(child, visitor, _args, *_kwargs)
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/pytd/parse/node.py", line 226, in _VisitNode
new_child = _VisitNode(child, visitor, _args, *_kwargs)
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/pytd/parse/node.py", line 261, in _VisitNode
new_child = _VisitNode(child, visitor, _args, *_kwargs)
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/pytd/parse/node.py", line 226, in _VisitNode
new_child = _VisitNode(child, visitor, _args, *_kwargs)
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/pytd/parse/node.py", line 261, in _VisitNode
new_child = _VisitNode(child, visitor, _args, *_kwargs)
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/pytd/parse/node.py", line 226, in _VisitNode
new_child = _VisitNode(child, visitor, _args, *_kwargs)
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/pytd/parse/node.py", line 261, in _VisitNode
new_child = _VisitNode(child, visitor, _args, *_kwargs)
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/pytd/parse/node.py", line 276, in _VisitNode
new_node = visitor.Visit(new_node, _args, *_kwargs)
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/pytd/parse/visitors.py", line 84, in Visit
self, node, _args, *_kwargs)
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/pytd/parse/visitors.py", line 669, in VisitClassType
raise ValueError("Unresolved class: %r" % node.name)
ValueError: Unresolved class: 'bytes'
A one-line test case ut.py
import unittest
resulted in this error::
File "ut.py", line 1, in <module>: Couldn't import pyi for 'unittest' [pyi-error]
File: "/pxfs/dev/nodakai/og-py27/lib/python2.7/site-packages/pytype/typeshed/stdlib/2.7/unittest.pyi", line 30
shouldStop = False
^
ParseError: syntax error, unexpected NAME, expecting NUMBER or ELLIPSIS
pytype Git HEAD 4e164cc
From Guido:
pytype -V 3.1 basic.py
OSError: [Errno 2] No such file or directory
pytype -V 3.2 basic.py
IndexError: tuple index out of range
pytype -V 3.3 basic.py
pytype.vm.ConversionError: Only some types are supported: <class 'pytype.abstract.Instance'>
pytype -V 3.4 basic.py
AttributeError: 'dict' object has no attribute '__build_class__'
pytype -V 3.5 basic.py
KeyError: 3350
Might also be due to different files being picked up from typeshed, depending on the version.
See https://www.python.org/dev/peps/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code
In particular this should be supported:
def send_email(address, # type: Union[str, List[str]]
sender, # type: str
cc, # type: Optional[List[str]]
bcc, # type: Optional[List[str]]
subject='',
body=None # type: List[str]
):
# type: (...) -> bool
"""Send an email message. Return True if successful."""
<code>
pytype uses List[T, ...]
where PEP 484 just has List[T]
.
(Explanation: PEP 484 does use Tuple[T, ...]
for a homogeneous tuple of Ts, since Tuple[T]
means a tuple of length 1 containing a T, so that we also have Tuple[T, S]
meaning a tuple of length 2 containing a T and an S. But we don't support that for List or other Sequence types -- it's purely a Tuple thing.)
import sys
if sys.version_info >= (3, 5):
a = ... # type: None
elif sys.version_info >= (3,):
a = ... # type: None
else:
a = ... # type: None
gives
Error while parsing pyi:
File "/tmp/asd2.py", line 5
elif sys.version_info >= (3,):
^
SyntaxError: Unexpected 'NAME'
$ pytype --python_version=3.5 foo.py
Traceback (most recent call last):
File "/usr/host/bin/pytype", line 258, in <module>
sys.exit(main(sys.argv) or 0)
File "/usr/host/bin/pytype", line 201, in main
return _run_pytype(options)
File "/usr/host/bin/pytype", line 246, in _run_pytype
print_errors=True)
File "/usr/host/bin/pytype", line 131, in process_one_file
options=options)
File "/usr/host/bin/pytype", line 49, in check_pyi
cache_unknowns=options.cache_unknowns)
File "/usr/lib/python2.7/site-packages/pytype/infer.py", line 675, in check_types
py_src, py_filename, init_maximum_depth, run_builtins)
File "/usr/lib/python2.7/site-packages/pytype/vm.py", line 689, in run_program
node, f_globals, f_locals = self.preload_builtins(node)
File "/usr/lib/python2.7/site-packages/pytype/vm.py", line 652, in preload_builtins
builtins_code = self.compile_src(src)
File "/usr/lib/python2.7/site-packages/pytype/vm.py", line 637, in compile_src
filename=filename, mode=mode)
File "/usr/lib/python2.7/site-packages/pytype/pyc/pyc.py", line 158, in compile_src
code = parse_pyc_string(pyc_data)
File "/usr/lib/python2.7/site-packages/pytype/pyc/pyc.py", line 129, in parse_pyc_string
return parse_pyc_stream(StringIO.StringIO(data))
File "/usr/lib/python2.7/site-packages/pytype/pyc/pyc.py", line 109, in parse_pyc_stream
python_version = magic.magic_word_to_version(magic_word)
File "/usr/lib/python2.7/site-packages/pytype/pyc/magic.py", line 105, in magic_word_to_version
return PYTHON_MAGIC[magic_word]
KeyError: '3351\nFile: foo.py'
Works fine with --python_version=3.4
though.
The file examined is:
def foo1(a: int) -> str:
return (lambda x: x + 1)(a)
def foo2(a: int) -> str:
return a + 1
def foo3(a: str) -> int:
return (lambda x: x + 1)(a)
I was running the following command with Python 2.7.5
$ pytype hello.py
File "hello.py", line 33: invalid syntax [python-compiler-error]
This is the line where the error happens:
def bold(text : str) -> str:
But this is exactly as specified in PEP 484 , for example:
def greeting(name: str) -> str:
return 'Hello ' + name
So I think pytype does not work yet with PEP 484.
I tried using pytype to generate stubs for https://github.com/dateutil/dateutil/blob/master/dateutil/relativedelta.py. It pegged a core, I had to kill the process after waiting ~10 minutes.
The example below is inferring amount as Iterable
and claiming that balance
is an unknown attribute as well as claiming that Account.deposit
returns a list
.
Potentially multiple different issues. Or just a need for actual documentation to describe the proper getting started code base bootstrapping process and expected odd results on entirely un-annotated code. :)
#!/usr/bin/python3
from typing import Dict, Optional, Union
class Account:
"""Only a horrible bank uses a float for account balances."""
def __init__(self, opening_balance: float = 0.0):
self.balance = opening_balance # expect float
def deposit(self, amount): # expect (float) -> float
self.balance += amount
return self.balance
class Bank:
def __init__(self, bank_owner="Unladen Swallows"):
self._accounts = {} # expect Dict[str, Account]
self._accounts[bank_owner] = Account(3.5)
def deposit(self, name, amount): # expect (float) -> Optional[float]
"""Multiple return types is a horrible API!"""
account = self._accounts.get(name)
if account:
return account.deposit(amount)
else:
return None
def main():
bank = Bank()
balance = bank.deposit("Unladen Swallows", 5)
print("balance:", balance)
assert balance == 8.5
if __name__ == '__main__':
main()
rainbow_unicorn$ pytype -V 3.5 --output - example.py
from typing import Any, Dict, Iterable, Optional
class Account:
__doc__ = ... # type: str
balance = ... # type: Any
def __init__(self, opening_balance: float = ...) -> Any: ...
def deposit(self, amount: Iterable) -> list: ...
class Bank:
_accounts = ... # type: Dict[Any, Account]
def __init__(self, bank_owner = ...) -> None: ...
def deposit(self, name, amount: Iterable) -> Optional[list]: ...
def main() -> None: ...
File "example.py", line 12, in deposit: No attribute 'balance' on Account [attribute-error]
File "/usr/lib64/python2.7/site-packages/pytype/pyc/opcodes.py", line 1045, in dis_code
co_firstlineno=code.co_firstlineno)
File "/usr/lib64/python2.7/site-packages/pytype/pyc/opcodes.py", line 1033, in dis
return _dis(data, mapping, _args, *_kwargs)
File "/usr/lib64/python2.7/site-packages/pytype/pyc/opcodes.py", line 999, in _dis
cls = mapping[opcode]
KeyError: 82
Opcode 82 appears to be WITH_CLEANUP_FINISH https://hg.python.org/cpython/file/tip/Include/opcode.h#l60
When generating e.g.
def foo() -> NoneType
it should really be
def foo() -> None
There is no builtin (not even in some stdlib module) named NoneType -- that's just the name you see printed when you ask for type(None)
.
~/Programming/pytype/env/bin/pytype mercurial/error.py
Traceback (most recent call last):
File "/Users/augie/Programming/pytype/env/bin/pytype", line 264, in
sys.exit(main(sys.argv) or 0)
File "/Users/augie/Programming/pytype/env/bin/pytype", line 218, in main
return _run_pytype(options)
File "/Users/augie/Programming/pytype/env/bin/pytype", line 258, in _run_pytype
ret = process_one_file(input_filename, output_filename, options)
File "/Users/augie/Programming/pytype/env/bin/pytype", line 177, in process_one_file
options=options)
File "/Users/augie/Programming/pytype/env/bin/pytype", line 86, in generate_pyi
maximum_depth=(1 if options.quick else None))
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/infer.py", line 597, in infer_types
ast = convert_structural.convert_pytd(ast, tracer.loader.concat_all())
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/convert_structural.py", line 243, in convert_pytd
mapping, result = solve(ast, builtins_pytd)
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/convert_structural.py", line 172, in solve
return TypeSolver(ast, builtins_pytd).solve(), extract_local(ast)
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/convert_structural.py", line 133, in solve
self.match_unknown_against_complete(factory, solver, unknown, complete)
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/convert_structural.py", line 61, in match_unknown_against_complete
solver.implies(booleq.Eq(unknown.name, complete.name), implication)
File "/Users/augie/Programming/pytype/env/lib/python2.7/site-packages/pytype/pytd/booleq.py", line 454, in implies
assert e.right not in self.implications[e.left]
AssertionError
pytype still has a handful of its own builtins, since it can't parse the corresponding typeshed files yet. They're in pytype/pytd/builtins/*.pytd
and pytype/pytd/stdlib/*.pytd
.
__builtins__.pytd
might be an exception, since it models the behaviour of mutable types more precisely than builtins.pyi
in typeshed. collections.pyi
and itertools.pyi
we might want to keep, as well.
All the other .pytd
files should be removed. Files like os.pytd
and codes.pytd
are shadowing a much better definition in typeshed
.
The command line help says
-c, --check Verify against existing "output" pytd files.
But I don't understand how to use it. Suppose I have basic.py containing some self-contained code (no imports), I can generate basic.pytd by running
pytd basic.py:basic.pytd
Now suppose I edit basic.py -- the -c
options suggests that I could verify that the modified basic.py still conforms to the original basic.pytd, but all I get is errors:
$ pytype -c basic.py
Traceback (most recent call last):
File "/Users/guido/v/bin/pytype", line 202, in <module>
sys.exit(main(sys.argv) or 0)
File "/Users/guido/v/bin/pytype", line 196, in main
ret = process_one_file(input_filename, output_filename, options)
File "/Users/guido/v/bin/pytype", line 146, in process_one_file
options=options)
File "/Users/guido/v/bin/pytype", line 32, in check_pytd
with open(output_filename, "r") as fi:
TypeError: coercing to Unicode: need string or buffer, NoneType found
or
$ pytype -c basic.py:basic.pytd
Traceback (most recent call last):
File "/Users/guido/v/bin/pytype", line 202, in <module>
sys.exit(main(sys.argv) or 0)
File "/Users/guido/v/bin/pytype", line 196, in main
ret = process_one_file(input_filename, output_filename, options)
File "/Users/guido/v/bin/pytype", line 146, in process_one_file
options=options)
File "/Users/guido/v/bin/pytype", line 45, in check_pytd
maximum_depth=(1 if options.quick else None))
TypeError: check_types() got an unexpected keyword argument 'import_pytd_ext'
In PEP 484, there is a section on conditional stubs.
For example, the next code is a valid stub code
import sys
if sys.version_info <= (3,):
def old_code_stub() -> None: ...
But currently, running pytd
on it will fail with
Error while parsing pyi:
File "/tmp/asd.py", line 3
if sys.version_info <= (3,):
^
SyntaxError: Unexpected 'DOT'
Found while doing typeshed#345.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.