This repository includes the code and experimental data in our paper entitled "A Novel Neural Source Code Representation based on Abstract Syntax Tree" that will be presented at ICSE'2019. It can be used to encode code fragments into supervised vectors for various source code related tasks. We have applied our neural source code representation to two common tasks: source code classification and code clone detection. It is also expected to be helpful in more tasks.
- python 3.6
- pandas 0.20.3
- gensim 3.5.0
- scikit-learn 0.19.1
- pytorch 0.3.1
- pycparser 2.18
- javalang 0.11.0
- RAM 16GB or more
- GPU with CUDA support is also needed
- BATCH_SIZE should be configured based on the GPU memory size
Install all the dependent packages via pip:
$ pip install pandas==0.20.3 gensim==3.5.0 scikit-learn==0.19.1 pycparser==2.18 javalang==0.11.0
Install pytorch 0.3.1 with cuda:
$ pip install https://download.pytorch.org/whl/cu80/torch-0.3.1-cp36-cp36m-linux_x86_64.whl
The version of pytorch 0.3.1 is mandatory. Higher versions may lead to errors for our exisiting code, and we will improve our code in the future. See https://pytorch.org/get-started/previous-versions/ for more options according to your CUDA version.
cd astnn
- run
python pipeline.py
to generate preprocessed data. - run
python train.py
for training and evaluation
cd clone
- run
python pipeline.py --lang c
orpython pipeline.py --lang java
to generate preprocessed data for the two datasets. - run
python train.py --lang c
to train on OJClone,python train.py --lang java
on BigCLoneBench respectively.