mbdevpl / horast Goto Github PK
View Code? Open in Web Editor NEWHuman-oriented abstract syntax tree (AST) parser/unparser for Python 3 that doesn't discard comments.
License: Apache License 2.0
horast's People
Contributors
Stargazers
Watchers
horast's Issues
docstring not parsed correctly
Following simple test fails:
import horast
import inspect
class Test:
"""
multiline
docstring
"""
original_code = inspect.getsource(Test)
tree = horast.parse(original_code)
unparsed_code = horast.unparse(tree)
assert original_code == unparsed_codebecause tree looks like this:
Module(
body=[ClassDef(
name='Test',
bases=[],
keywords=[],
body=[Expr(value=Str(
s='\n multiline\n docstring\n ',
kind=''))],
decorator_list=[])],
type_ignores=[])Would it be possible to fix this somehow? I can probably use some temporary workaround, but it would be really nice to have this working out of the box. I'm going to generate classes/methods with docstrings.
Make it PEP 561 compatible
Would you mind to make it PEP 561 compatible (see https://mypy.readthedocs.io/en/latest/installed_packages.html)? It basically means to create file py.typed in the package and add it to package_data - I would do it myself and create PR, but it somehow does not work with your setup_boilerplate - probably some trivial problem :)
Btw, thanks for your great work - horast was exactly what I needed...
Issues with fluent apis
Horast fails to parse correctly for inline comments embedded in fluent-style blocks
horast.parse("""
x = (spark_rdd
# Increate by one
.map(lambda x: x+1)
# Sum it all
.reduce(lambda x, y: x +y)
)
""")
AST visitor example?
Can you please provide an example of an ast.NodeVisitor equivalent in horast? This library looks very useful because it includes comments. I want to use it to automatically generate documentation from source, since tools like sphinx only seem to include function doc strings.
# type: ignore leads to AssertionError
If the following code:
class foo:
def bar(self): # whatever comment
pass
bar.__baz__ = None # type: ignore
is passed to horast.parse, an AssertionError is raised:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/zdenal/.local/lib/python3.6/site-packages/horast/parser.py", line 19, in parse
tree = insert_comment_tokens(code, tree, comment_tokens)
File "/home/zdenal/.local/lib/python3.6/site-packages/horast/ast_comments.py", line 27, in insert_comment_tokens
path_to_anchor, before_anchor = find_in_ast(code, tree, nodes, scope)
File "/home/zdenal/.local/lib/python3.6/site-packages/horast/ast_tools.py", line 144, in find_in_ast
scopes = get_ast_node_scopes(code, nodes)
File "/home/zdenal/.local/lib/python3.6/site-packages/horast/ast_tools.py", line 72, in get_ast_node_scopes
assert len(ast_nodes) == len(nodes), (len(ast_nodes), len(nodes))
AssertionError: (12, 13)
If I replace # type: ignore by normal comment, it works. Also, if # whatever comment is removed, it works. Tested with horast version 0.3.1. Any ideas? Many thanks!
AssertionError: assert path[-2].node is within_node, (path[-2].node, within_node)
I was getting an AssertionError because of this comment. With the comment gone, the error was also gone. Also, everything seems to be ok with the comment on its original place and disabled assertions.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/zdenal/arcor2_repos/arcor2/build-support/.venv/lib/python3.6/site-packages/horast/parser.py", line 19, in parse
tree = insert_comment_tokens(code, tree, comment_tokens)
File "/home/zdenal/arcor2_repos/arcor2/build-support/.venv/lib/python3.6/site-packages/horast/ast_comments.py", line 87, in insert_comment_tokens
tree = insert_comment_token(token, code, tree, nodes=nodes, scopes=scopes)
File "/home/zdenal/arcor2_repos/arcor2/build-support/.venv/lib/python3.6/site-packages/horast/ast_comments.py", line 61, in insert_comment_token
path_to_anchor, before_anchor = find_in_ast(code, tree, scope, nodes=nodes, scopes=scopes)
File "/home/zdenal/arcor2_repos/arcor2/build-support/.venv/lib/python3.6/site-packages/horast/ast_tools.py", line 236, in find_in_ast
assert path[-2].node is within_node, (path[-2].node, within_node)
AssertionError: (<typed_ast._ast3.ClassDef object at 0x7f5a7fe5ef60>, <typed_ast._ast3.Dict object at 0x7f5a7fea0470>)
While trying to construct a minimal example, I got a NotImplementedError...
>>> import horast
>>> code = """
d = {
1: 2,
3: 4, # comment
}
"""
>>> horast.parse(code)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/zdenal/arcor2_repos/arcor2/build-support/.venv/lib/python3.6/site-packages/horast/parser.py", line 19, in parse
tree = insert_comment_tokens(code, tree, comment_tokens)
File "/home/zdenal/arcor2_repos/arcor2/build-support/.venv/lib/python3.6/site-packages/horast/ast_comments.py", line 87, in insert_comment_tokens
tree = insert_comment_token(token, code, tree, nodes=nodes, scopes=scopes)
File "/home/zdenal/arcor2_repos/arcor2/build-support/.venv/lib/python3.6/site-packages/horast/ast_comments.py", line 61, in insert_comment_token
path_to_anchor, before_anchor = find_in_ast(code, tree, scope, nodes=nodes, scopes=scopes)
File "/home/zdenal/arcor2_repos/arcor2/build-support/.venv/lib/python3.6/site-packages/horast/ast_tools.py", line 222, in find_in_ast
scopes_containing_target_scope))
NotImplementedError: inconsistent results for target Scope(start=Location(lineno=4, offset=12), end=Location(lineno=4, offset=21)) in:
"""
d = {
1: 2,
3: 4, # comment
}
after 0, before None, within [(<typed_ast._ast3.Module object at 0x7f7caa11acf8>, Scope(start=Location(lineno=1, offset=0), end=Location(lineno=5, offset=1))), (<typed_ast._ast3.Assign object at 0x7f7caa11ae80>, Scope(start=Location(lineno=2, offset=0), end=Location(lineno=5, offset=1))), (<typed_ast._ast3.Dict object at 0x7f7caa11ada0>, Scope(start=Location(lineno=2, offset=4), end=Location(lineno=5, offset=1)))]"""Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.