avast / retdec-regression-tests-framework Goto Github PK

View Code? Open in Web Editor NEW

23.0 7.0 13.0 542 KB

A framework for writing and running regression tests for RetDec and related tools.

License: MIT License

Makefile 0.15% Python 99.71% Shell 0.05% Dockerfile 0.09%

retdec-regression-tests-framework's Introduction

RetDec Regression Tests Framework

A framework for writing and running regression tests for RetDec and related tools.

Requirements

To run regression tests, you must have:

Python >= 3.4
Clang 3.9.1 (exactly this version; download a pre-built package from here and extract it somewhere)
Cloned our retdec repository, built and installed RetDec.
Cloned our retdec-regression-tests repository that contains test cases.

Additionally, on Microsoft Windows, you need a 32-bit and 64-bit GCC compiler from MSYS2:

Download and run an MSYS2 installer:
- 64-bit Windows: http://sourceforge.net/projects/msys2/files/Base/x86_64/msys2-x86_64-20150916.exe/download
Open an MSYS2 shell.
Install a 32-bit GCC compiler via pacman -S mingw-w64-i686-gcc.
Install a 64-bit GCC compiler via pacman -S mingw-w64-x86_64-gcc.

Additionally, to run tests for our IDA plugin, you must have:

IDA 7.x
Cloned our retdec-idaplugin repository, built and installed our IDA plugin.

Installation

Clone the repository.
Step inside it:
```
$ cd retdec-regression-tests-framework
```

Install all the needed third-party Python packages, either into a virtual environment:

$ python -m venv virtualenv
$ source virtualenv/bin/activate
$ pip install -r requirements.txt

or into your home directory:

$ pip install --user -r requirements.txt

Verify that all dependencies have been installed by running
```
python check_dependencies.py
```

Create a new file config_local.ini with the following content (you will need to adjust the paths; use Windows style paths on Windows, e.g. C:\path\to\clang):

[runner]
; Path to the extracted Clang package containing subdirectories such as bin, include, lib, share.
clang_dir = /path/to/clang
; Path to the cloned repository containing regression tests.
tests_root_dir = /path/to/retdec-regression-tests
; Path to the RetDec's installation directory.
retdec_install_dir = /path/to/retdec/installed

Additionally, if you plan to run tests for our IDA plugin, you have to also include the following settings into the [runner] section of the config_local.ini file:

idaplugin_tests_enabled = 1
; Path to the IDA 7.x directory.
idaplugin_ida_dir = /path/to/ida
; Path to our script run-ida-decompilation from the retdec-idaplugin repository.
idaplugin_script = /path/to/retdec-idaplugin/scripts/run-ida-decompilation.py

Use

Use MSYS2 shell on Microsoft Windows.

To run all tests, execute

$ python runner.py

To run only tests in the given directory, execute

$ python runner.py path/to/directory

For more information, execute

$ python runner.py --help

Documentation

The documentation is available online. To generate it locally, execute

$ make docs

Then, open docs/_build/html/index.html in your favorite web browser.

License

The framework includes several third-party libraries, whose code and licensing information is provided in the deps subdirectory.

Contributing

See RetDec contribution guidelines.

retdec-regression-tests-framework's People

Contributors

Stargazers

Watchers

Forkers

mewbak decomp-mirror terry2012 silverf0x idkwim mehrdad-shokri sopticek mr-wrmsr andyholst pombredanne xiaobo996 classicvalues retdec-build

retdec-regression-tests-framework's Issues

Make it possible to configure the compilation command used for RetDec output compilation

Some regression tests check that the compiled result of RetDec decompilation produces the expected output when executed. Right now, the compilation command are hard-coded in the regression tests framework. Most notably, usage of 32-bit MinGW on Windows - which implies the need for MSYS2.

Make it possible for a user to specify (configure) a custom compilation command used in this scenario. This would mean that the framework no longer require MSYS2 and MSVC can be used.

Ability to get function-call expression from function body

It would be handy if it was possible to get function call expression from a given function body. E.g.

call = func.calls['myfunc']  # return a CallExpr instance.

Nondeterministic C source code parsing between platforms

The C sources that are being tested are parsed using clang. The problem is that the result of this parsing (AST) is not always the same on all the supported platforms (Linux, Windows, macOS). Difference can probably occur even between machines using the same platform. Even if the same version of clang is used, there can be differences. It looks like system includes play a role here. The problem is most prominent in call expression parsing, but probably can occur in other situations as well.

Example:

#include <stdlib.h>

#include <arpa/inet.h>
#include <ctype.h>
#include <errno.h>
#include <fcntl.h>
#include <netinet/in.h>
#include <signal.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stropts.h>
#include <sys/prctl.h>
#include <sys/select.h>
#include <sys/socket.h>
#include <sys/wait.h>
#include <time.h>
#include <unistd.h>

int main()
{
        int32_t set;
        sigaddset((struct _TYPEDEF_sigset_t *)&set, SIGINT):
}

Linux parses it ok and recognizes sigaddset call.
macOS does not parse the call at all - it completely ignores it.
macOS without the long list of includes parses it ok and recognizes sigaddset call.
macOS without the type cast (i.e. sigaddset(&set, SIGINT)) parses the call as __sigbits.

Another example is parsing of strlcpy() call (without proper type signature). Linux parses it ok, but macOS does not. It parses it only if the function (and its calls) have the full signature of

size_t strlcpy(char * restrict dst, const char * restrict src, size_t dstsize);

Solutions:

Can we force clang to use some custom set of includes that would be the same everywhere?
Can we remove the #include statements from C sources before parsing it? (=> I don't think so, without them, some other function calls may not get parsed.)
Are includes the only problem?

Regression tests fail with "undefined symbol: clang_CXXRecord_isAbstract"

When I run the regression tests via runner.py or run tests via make tests, they fail with the following error:

clang.cindex.LibclangError: /path/to/clang/lib/libclang.so: undefined symbol: clang_CXXRecord_isAbstract. Please ensure that your python bindings are compatible with your libclang.so version

The Clang in /path/to/clang is 3.9.1.

Make it possible to configure a test to use 32/64-bit compilation command

Related to #6.

Some regression tests check that the compiled result of RetDec decompilation produces the expected output when executed. Right now, all these compilations are using 32-bit compilers and produce 32-bit binaries. When we add decompilation of 64-bit binaries, we will also need to make sure their decompilation results are compiled to 64-bit executables in the regression tests (pointer Arithmetics, etc.).

Make it possible to somehow specify what kind of binary (32/64-bit) shall be produced and executed from the RetDec result.