ambv / retype Goto Github PK

View Code? Open in Web Editor NEW

139.0 6.0 19.0 158 KB

Re-apply type annotations from .pyi stubs to your codebase.

License: MIT License

Python 100.00%

python python3 typing type-checking stubs mypy pep484 python2

retype's Introduction

retype

!!! Note this project is no longer supported/maintained as of 2022 August 14th !!!

Re-apply type annotations from .pyi stubs to your codebase.

Usage

Usage: retype [OPTIONS] [SRC]...

  Re-apply type annotations from .pyi stubs to your codebase.

Options:
  -p, --pyi-dir DIRECTORY     Where to find .pyi stubs.  [default: types]
  -t, --target-dir DIRECTORY  Where to write annotated sources.  [default:
                              typed-src]
  -i, --incremental           Allow for missing type annotations in both stubs
                              and the source.
  -q, --quiet                 Don't emit warnings, just errors.
  -a, --replace-any           Allow replacing Any annotations.
  --hg                        Post-process files to preserve implicit byte
                              literals.
  --traceback                 Show a Python traceback on error.
  --version                   Show the version and exit.
  --help                      Show this message and exit.

When you run retype, it goes through all files you passed as SRC, finds the corresponding .pyi files in the types/ directory, and re-applies typing annotations from .pyi to the sources, using the Python 3 function and variable annotation syntax. The resulting combined sources are saved in typed-src/.

You can also pass directories as sources, in which case retype will look for .py files in them recursively.

It's smart enough to do the following:

reapply typing imports
reapply function argument annotations
reapply function return value annotations
reapply method argument and return value annotations
reapply function-level variable annotations
reapply module-level name annotations
reapply module-level type aliases
reapply class-level field annotations
reapply instance-level field annotations
validate existing source annotations against the .pyi file
validate source function signatures against the .pyi file
read function signature type comments in .pyi files
read variable type comments in .pyi files
consider existing source type comments as annotations
remove duplicate type comments from source when annotations are applied
normalize remaining type comments in the source to annotations; this is done even if the corresponding .pyi file is missing

List of things to be done

add a --backward option to output type comments instead of annotations
handle if sys.version_info and sys.platform checks in stubs

Design principles

it's okay for a given .pyi file to be incomplete (gradual typing, baby!)
it's okay for functions and classes to be out of order in .pyi files and the source
it's an error for a function or class to be missing in the source
it's an error for a function's signature to be incompatible between the .pyi file and the source
it's an error for an annotation in the source to be incompatible with the .pyi file

Known limitations

Line numbers in the annotated source will no longer match original source code; this is because re-application of types requires copying typing imports and alias definitions from the .pyi file.
While formatting of the original source will be preserved, formatting of the applied annotations might differ from the formatting in .pyi files.
The source where type annotations get re-applied cannot use the legacy print statement; that wouldn't work at runtime.
Class attribute annotations in __init__() methods are moved verbatim to the respective __init__() method in the implementation. They are never translated into class-level attribute annotations, so if that method is missing, the translation will fail. Similarly, class-level attribute annotations are never applied to __init__() methods.
Forward references in .pyi files will only be properly resolved for type aliases and type vars (by inserting them right before they're used in the source). Other forms of forward references will not work in the source code due to out-of-order class and function definitions. Modify your .pyi files to use strings. retype will not automatically discover failing forward references and stringify them.
Local variable annotations present in the .pyi file are transferred to the body level of the given function in the source. In other words, if the source defines a variable within a loop or a conditional statement branch, retype will create an value-less variable annotation at the beginning of the function. Use a broad type and constrain types in relevant code paths using assert isinstance() checks.
Because of the above, existing source variable annotations and type comments buried in conditionals and loops will not be deduplicated (and mypy will complain that a name was already defined).
An async function in the stub will match a regular function of the same name in the same scope and vice versa. This is to enable annotating async functions spelled with @asyncio.coroutine.

Tests

Just run:

tox

OMG, this is Python 3 only!

Relax, you can run retype as a tool perfectly fine under Python 3.6+ even if you want to analyze Python 2 code. This way you'll be able to parse all of the new syntax supported on Python 3 but also effectively all the Python 2 syntax at the same time.

By making the code exclusively Python 3.6+, I'm able to focus on the quality of the checks and re-use all the nice features of the new releases (check out pathlib or f-strings) instead of wasting cycles on Unicode compatibility, etc.

Note: to retype modules using f-strings you need to run on Python 3.6.2+ due to bpo-23894.

License

MIT

Change Log

20.10.0

Mark python3.8 and python3.9 compatible

19.9.0

add a module entry-point, now you can call it via python -m retype
automatically all files excluded by .gitignore on merge of folders
support for ast3.num
fix a bug that meant the merge was not recursive in paths
use setup.cfg based packaging configuration
add PEP-517/8 declaration via pyproject.toml
include license in both wheel and sdist
this projects code base is now formatted with black, import ordered via isort, and uses Azure Pipelines instead of Travis (also testing on Windows and macOs)

17.12.0

support --replace-any to allow replacing pre-existing Any annotations without raising errors
bugfix: don't re-apply # type: ignore as an annotation if followed by another comment. Original patch by Shannon Zhu.

17.6.3

bugfix: don't try to re-apply # type: ignore as a function annotation
bugfix: support arbitrary source file encodings, patch by Michael Overmeyer.
bugfix: support missing newlines at the end of the file, patch by Michael Overmeyer.
bugfix: in --incremental, format default values according to PEP 8 (no spaces around the = sign if the type is missing)

17.6.2

bugfix: --incremental didn't work with multiple arguments before

17.6.1

support --incremental stub application (i.e. allow for both stubs and the source to be missing annotations for some arguments and/or return value)

17.6.0

support async functions
support --traceback for getting more information about internal errors

17.4.0

first published version
date-versioned

Authors

Glued together by Łukasz Langa. Multiple improvements by Michael Overmeyer and Bernat Gabor.

retype's People

Contributors

Stargazers

Watchers

Forkers

movermeyer shannonzhu tonyfast pombredanne bollwyvl huaihuai0406 maggiemoss couchbaselabs aneeshusa aklajnert mingyuguan jairoandrescastaneda findus23 terencehonles gaborbernat python-repository-hub bidaya0

retype's Issues

Mermaid diagram svg sizing issues - fixed height

Version: Retype 1.7.0

I have a large diagram defined in mermaid and it which gets resized to fit. The horizontal size is as I'd expect, however there is a large amount of empty vertical space above and below. See the following image.

The above page was generated by the following md file. Note I've had to insert a / before the ``` to avoid github rendering it

---
order: 2
---

# Blah 1234

### Lifecycle
Preceding text

/```mermaid

sequenceDiagram
	participant User
	participant GitRepo
	participant Server
	participant ResourceA
	participant ResourceB
	participant ResourceC
	
	User->>GitRepo: Commit changes
	loop
		Server->>GitRepo: Check for changes
	end
	User->>Server: Manually Trigger
	note right of Server: Something pipeline scheduled
	
	rect rgba(0,255,0,.1)
		note over Server,ResourceB: Blah blah blah
		activate ResourceA
		GitRepo->>ResourceA: Blah blah blah
		note right of ResourceA: Blah blah blah:<br/>Blah blah blah
		ResourceA->>Server: Blah blah blah
		deactivate ResourceA	
		
		Server->>ResourceB: something something something
		activate ResourceB
		note left of ResourceB: something:<br/>- something something something<br/>- Adds/generates stuff<br/>-Does things<br/>-Does more things<br/>-Hey, more things
		ResourceB->>Server: something something something something
		deactivate ResourceB	
	end
	
	note right of Server: Something something something<br/>Something something something
	
	rect rgba(0,255,0,.1)
		note over Server,ResourceB: something something
		Server->>ResourceB: Does something
		activate ResourceB
		note left of ResourceB: Something something something
		note left of ResourceB: Something something something
		
		ResourceB->>Server: Something something something
		deactivate ResourceB	
	end
	
	note right of Server: something something completed<br/>Schedule something something
	
	rect rgba(0,255,0,.1)
		note over Server,ResourceC: Blah something more
		Server->>ResourceC: Downloads something artifacts
		activate ResourceC
		note left of ResourceC: Blah blah blahBlah blah blah<br/>produces something something
		ResourceC->>Server: Blah blah blahBlah blah blah
		deactivate ResourceC	
	end
	
	note over User,Server: Blah blah bla<br/>Blah blah blah
	

/```

Following text

On further inspection you can see the height is fixed:

Removing that attribute fixes is for this particular case.

Checkboxes in readme render as raw HTML on PyPI

The readme could have a simple list without checkboxes (it’s not an interactive form after all!).

Maybe also worth reporting to warehouse.

Support overloaded definitions.

It seems like at this point retype doesn't support overloaded annotations. Tested with example annotation (foo.pyi):

from typing import overload

@overload
def f(x: str) -> str: ...
@overload
def f(x: int) -> int: ...

and source file (foo.py)

def f(x):
    if isinstance(x, (str, int)):
        return x
    else:
        raise TypeError(f"x should be str or int, got {type(x)}")

Executing retype results in

error: /path/to/foo.py: Annotation problem in function 'f': 1:1: incompatible existing annotation for {arg.arg!r}. Expected: 'int', actual: 'str'

Expected output would be

from typing import overload

@overload 
def f(x: str) -> str: ...
@overload
def f(x: int) -> int: ...
def f(x):
    if isinstance(x, (str, int)):
        return x
    else:
        raise TypeError(f"x should be str or int, got {type(x)}")

or equivalent.

Incompatible existing annotations when referencing classes in other modules

Hiya,

I came across your tool when trying out monkeytype and found it's very useful but bombs out on most of my files with errors like:

Annotation problem in function 'blah': 36:1: incompatible existing annotation for...

The incompatible annotations appear to be the difference between B and a.B. I tend to import modules and then reference classes within the modules rather than importing the classes directly (in order to avoid namespace pollution). Should retype support this situation but doesn't yet, or is there something else going on? Happy to debug further, just wanted to figure out if this was expected behaviour or not... 5:)

Is there a flag to unilaterally write the stub's type information into the code, ignoring the existing annotations?

Class not found in source with @type_check_only

Hello,

I think that classes decorated with @type_check_only should be allowed even if they are not present in the source file (since they are not available at runtime). Using them currently results in a Class not found in source error:

foo.pyi:

from typing import type_check_only
from typing_extensions import Protocol

@type_check_only
class A(Protocol):
    def __call__(self, u: str) -> int: ...

def f(a: A) -> int: ...

foo.py:

def f(a):
    return a(u='abc')

retype outputs error: /path/to/foo.py: Class 'A' not found in source.

I was expecting an output similar to this:

from typing import TYPE_CHECKING
from typing_extensions import Protocol

if TYPE_CHECKING:
    class A(Protocol):
        def __call__(self, u: str) -> int: ...

def f(a: 'A') -> int:
    return a(u='abc')

retype does not seem to handle `Callable` correctly

pyi file:

def set_log_callback(fn: Callable[[str], None]) -> None: ...

py file:

def set_log_callback(fn: Callable[[str], None]) -> None:

error:

Annotation problem in function 'set_log_callback': 120:1: incompatible existing annotation for {arg.arg!r}. Expected: 'Callable[[str], None]', actual: 'Callable[[str], None]'

It looks like the expected and actual nodes don't match:

(Pdb) p expected_annotation
'Callable[[str], None]'
(Pdb) p expected
Node(power, [Leaf(1, 'Callable'), Node(trailer, [Leaf(9, '['), Node(subscriptlist, [Node(atom, [Leaf(9, '['), Node(listmaker, [Leaf(1, 'str')]), Leaf(10, ']')]), Leaf(12, ','), Leaf(1, 'None')]), Leaf(10, ']')])])
(Pdb) p actual
Node(power, [Leaf(1, 'Callable'), Node(trailer, [Leaf(9, '['), Node(subscriptlist, [Node(atom, [Leaf(9, '['), Leaf(1, 'str'), Leaf(10, ']')]), Leaf(12, ','), Leaf(1, 'None')]), Leaf(10, ']')])])
(Pdb)

Information: is there any public Python API?

Hello, thank you very much for the project, it is a very useful tool!

I was wondering if there is any public Python API that allows using retype from a Python script itself (instead of relying on shell scripting).

For example, can I consider the functions retype_file and retype_path part of the public Python API?

This would allow scripting by installing retype and importing retype_file or retype_path directly instead of having to use a Popen/subprocess-based solution...

allow ignoring folders during rewrite

prefer .gitignore fallback manual 👍

lib2to3_parse assumes that the ParseError will always refer to an existing line

There seems to be a case where ParseError will report the line number after the last line number, causing an IndexError in retype:

Example file (core.py):

def get_message():
    return '123'

Example stub (types/core.pyi):

def get_message() -> str: ...

$>retype --traceback core.py
error: core.py: list index out of range
Traceback (most recent call last):
  File "retype.py", line 110, in retype_path
    retype_file(src, pyi_dir, targets, quiet=quiet, hg=hg)
  File "retype.py", line 132, in retype_file
    src_node = lib2to3_parse(src_txt)
  File "retype.py", line 161, in lib2to3_parse
    faulty_line = src_txt.splitlines()[lineno - 1]
IndexError: list index out of range

I haven't gone digging yet to see why the Driver is failing to parse this, but it seems that this should be fixed as well.

Tested using:

Python 3.6.1 (v3.6.1:69c0db5, Mar 21 2017, 17:54:52) [MSC v.1900 32 bit (Intel)] on win32
Python 3.6.1 (default, May 11 2017, 22:14:44) [GCC 4.9.2] on linux

Doesn't support property setters

related to #20
example.py:

class Testclass:
    value = None

    @property
    def someprop(self):
        return self.value

    @someprop.setter
    def someprop(self, value):
        self.value = value

example.pyi (generated with stubgen and added -> Any)

from typing import Any


class Testclass:
    value: Any = ...

    @property
    def someprop(self) -> Any: ...

    @someprop.setter
    def someprop(self, value: Any) -> None: ...

output:

retype -p . .
error: /path/example.py: Annotation problem in function 'someprop': 5:1: missing regular argument 'value' in source

Redefining aliases that use typing module fails

Thanks for this useful tool. Unfortunately while using it I've stumbled upon a weird error.

Consider the following example:

import typing

OPTIONAL_STR = typing.Optional[str]

This is content of example.py and example.pyi. After running retype against them, I get a following error:

example.py: incompatible existing alias 'OPTIONAL_STR'. Expected: 'typing.Optional[str]', actual: 'typing.Optional[str]'

However, when you slightly change the example, everything passes:

from typing import Optional

OPTIONAL_STR = Optional[str]

This behavior is the same for python 3.6, 3.7 and 3.8. Tested on Ubuntu.
Changing imports is a workaround, but it would be nice to use import typing especially if you use this module extensively.

retype doesn't check source encodings before parsing

I was trying to retype a file, but the file had a comment that contained Unicode characters.

Example file (core.py):

#This is a comment with unicode characters: "Афон"
foo = "bar"

Example stub (types/core.pyi):

foo = ... # type: str

$>retype --traceback core.py
error: core.py: 'charmap' codec can't decode byte 0x90 in position 72: character maps to <undefined>
Traceback (most recent call last):
  File "retype.py", line 110, in retype_path
    retype_file(src, pyi_dir, targets, quiet=quiet, hg=hg)
  File "retype.py", line 131, in retype_file
    src_txt = src_file.read()
  File "Python36-32\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 72: character maps to <undefined>

I would have expected it to not crash on files with Unicode characters, especially since the characters are within comments.

Tested using:

Python 3.6.1 (v3.6.1:69c0db5, Mar 21 2017, 17:54:52) [MSC v.1900 32 bit (Intel)] on win32

quote forward references

Unquoted forward references are valid in stub files, but not at runtime; when applying stubs to a module, retype should quote them when necessary to avoid creating a module that fails to run.

In light of PEP 563, this won't be necessary in Python 3.7, which probably reduces the motivation to implement it. But in any case, this issue should exist to document the issue and whatever resolution is decided on.

track line numbers between sources and target

Per the PyCon 2019 Typing summit, we agreed that probably the best way to merge stubs and sources is to use this tool. This is needed for mypy issue python/mypy#5028. To be able to use it for that we'll need to preserve/generate sources to output line transformation (so the type checker error reports are not totally off).

I'll work on getting this done, created this issue as a tracker for it and discussion platform.

single and double quotes in literals not handled correctly

pyi file:

MODE = Literal["r", "rb", "w", "wb"]

py file:

MODE = Literal["r", "rb", "w", "wb"]

error:

incompatible existing alias 'MODE'. Expected: "Literal['r', 'rb', 'w', 'wb']", actual: 'Literal["r", "rb", "w", "wb"]'

Looks like it always expects single quotes but does not normalize the types.

ambv / retype Goto Github PK

retype's Introduction

retype

Usage

List of things to be done

Design principles

Known limitations

Tests

OMG, this is Python 3 only!

License

Change Log

20.10.0

19.9.0

17.12.0

17.6.3

17.6.2

17.6.1

17.6.0

17.4.0

Authors

retype's People

Contributors

Stargazers

Watchers

Forkers

retype's Issues

Recommend Projects

Recommend Topics

Recommend Org