Comments (5)
https://docs.python.org/2/library/ast.html
这个可以提供一下代码吗
请问最后是怎么解决的呀 我也遇到这个问题了
from rencos.
Hi,
It seems that this is a common problem, so I paste the code for parsing python dataset here. Hope it helps.
def parse_python(code):
import ast,re, asttokens
def str_node(node):
if isinstance(node, ast.AST):
fields = [(name, str_node(val)) for name, val in ast.iter_fields(node) if isinstance(val, str)]
name = fields[0][1] if fields else ''
rv = '%s %s' % (node.__class__.__name__, name.replace('\'', ''))
return rv
else:
return repr(node)
def ast_visit(node, level=0):
tokens.extend(str_node(node).split())
# print(str_node(node))
for field, value in ast.iter_fields(node):
if isinstance(value, list):
for item in value:
if isinstance(item, ast.AST) and not isinstance(item, ast.Load) and not isinstance(item, ast.Store):
ast_visit(item, level=level + 1)
elif isinstance(value, ast.AST) and not isinstance(value, ast.Load) and not isinstance(value, ast.Store):
ast_visit(value, level=level + 1)
return tokens
code = re.sub(r"\t", r"\\t", code)
code = re.sub(r"\n", r"\\n", code)
code = re.sub(r"DCNL\sDCSP", "\n", code)
code = re.sub(r"DCNL\s\s", "\n", code)
code = re.sub(r"DCNL\s", "\n", code)
code = re.sub(r"DCSP", '', code)
# print(code)
tokens = []
try:
tree = ast.parse(code)
ast_visit(tree)
return ' '.join(tokens)
except SyntaxError:
print('Error')
return 'Error'
from rencos.
got it
from rencos.
from rencos.
@satinewee
For people who need to parse the Java dataset, please see the code below.
def parse_java(code):
import javalang
from javalang.ast import Node
def get_token(node):
token = ''
if isinstance(node, str):
token = node.replace('\"|\'', '')
elif isinstance(node, set):
token = 'Modifier' # node.pop()
elif isinstance(node, Node):
token = node.__class__.__name__
return token
def get_children(root):
if isinstance(root, Node):
children = root.children
elif isinstance(root, set):
children = list(root)
else:
children = []
def expand(nested_list):
for item in nested_list:
if isinstance(item, list):
for sub_item in expand(item):
yield sub_item
elif item:
yield item
return list(expand(children))
def get_sequence(node, sequence):
token, children = get_token(node), get_children(node)
sequence.append(token)
for child in children:
get_sequence(child, sequence)
tokens = javalang.tokenizer.tokenize(code)
parser = javalang.parser.Parser(tokens)
tree = parser.parse_member_declaration()
seq = []
get_sequence(tree, seq)
return ' '.join(seq)
from rencos.
Related Issues (8)
- Dataset used after duplication removal and setting code length and comment length to 300 and 30 respectively?
- Upload Trained Model Weights HOT 1
- Could you please upload the code of how to generate train/test.ast.src HOT 3
- run preprocess error
- Is there a problem with the test set data? HOT 1
- Attention information
- It looks like there is a bug in dataset_base.py
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rencos.