PSIMiner — a tool for processing PSI trees from the IntelliJ Platform.
PSI trees contain code syntax trees as well as functions to work with them,
and therefore can be used to enrich code representation using static analysis algorithms of modern IDEs.
PSIMiner is a plugin for IntelliJ IDEA that runs it in a headless mode and creates datasets for ML pipelines.
The complete documentation of different parts is stored in docs folder.
PSIMiner requires Java 11 for correct work.
Check gradle will use the correct version.
All other dependencies will be installed automatically.
Use ./gradlew build (or gradlew.bat build on Windows) to build the tool.
There are already predefined configurations compatible with the IntelliJ IDEA. Open or import project in it and run tool on test data or start tests. You can modify these configurations to suit your needs.
However, it is possible to run the tool through CLI. It is better to use predefined shell script (only for Unix system)
./psiminer.sh $dataset_path $output_folder $JSON_configPSIMiner automatically store logs in home directory of user on each run.
Check ~/psiminer.log (or something like C:\Users\yourusername\psiminer.log for Windows) and share it to describe
your
problem.
PSIMiner completely configured by JSON.
Check examples in the configs folder.
Logically PSIMiner consist of the following parts.
There are a full documentation for them in docs folder:
- Tree transformations — this is an interface for enriching trees with new information and other useful manipulations, e.g. resolve types or exclude whitespaces.
- Filters — this is an interface for removing bad trees from the data, e.g. trees that are too big.
- Label extractor — this is an interface to define the correct extraction of labels from raw trees, e.g. extract method name for each method.
- Storage — this is an interface to define how tree should be saved on the disk, e.g. code2seq format or JSONL format.
There are also a few fields to define a parser and pipeline options.
For example, setting up Language.
If you turn on additional preprocessing:
- ✅ more projects will be opened successfully by IDEA
⚠️ files in your original dataset will be changed
More about additional preprocessing
Currently, PSIMiner supports Java and Kotlin datasets.
But we developed the tool with the possibility to extend it to new languages.
And since PSI trees supports big amount of languages,
adding new language into the tool requires only implementing few interfaces.
Be aware that multiple tree transformations can't be adopted to new languages automatically. And therefore, require manual work to add support for the new language.
If you would like to see new languages, don't hesitate to create issues with their request. Or even implement them yourself and create a pull request.
You can reuse different parts of the PSIMiner inside your one tool, e.g. plugin for model inference.
To add core part of the tool (without dependency to CLI) add following code into your gradle.kts file:
dependencies {
implementation("org.jetbrains.research.psiminer:psiminer-core") {
version {
branch = "main"
}
}
}
Remember that PSIMiner is plugin for IntelliJ IDEA and, therefore, can be integrated only in another plugin.
The paper
dedicated to the PSIMiner was published in MSR'21.
If you use PSIMiner in your academic work, please, cite it.
@inproceedings{spirin_psiminer,
author={Spirin, Egor and Bogomolov, Egor and Kovalenko, Vladimir and Bryksin, Timofey},
booktitle={2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)},
title={PSIMiner: A Tool for Mining Rich Abstract Syntax Trees from Code},
year={2021},
pages={13-17},
doi={10.1109/MSR52588.2021.00014}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent