unittestbot / utbotcpp Goto Github PK

Tool that generates unit test by C/C++ source code, trying to reach all branches and maximize code coverage

License: Apache License 2.0

CMake 0.89% C 9.02% C++ 67.75% Shell 1.38% JavaScript 0.57% CSS 0.33% HTML 0.51% Batchfile 0.03% TypeScript 8.94% Python 0.12% Makefile 0.03% Kotlin 10.43%

software-analysis test-generation symbolic-execution c cpp unit-testing static-analysis klee

utbotcpp's Introduction

What is UTBotCpp?

UTBot C/C++ generates test cases by code, trying to cover a maximum number of statements and execution paths. We treat source code as a source of truth, assuming that behavior is correct and corresponds to initial user demand. Generated tests are placed in the so-called regression suite. Thus, we fix current behavior with the help of generated test cases. Using UTBot for C/C++, developers obtain full control of their code. No future change can break the code without being noticed once it's covered with tests generated by UTBot. This way, modifications made by developers to an existing code are much safer. Hence, with the help of generated unit tests, UTBot provides dramatic code quality improvement.

Features demonstration in 5 min (click to see on YouTube):

Try UTBot demo online here!

More info on wiki

How to install and use UTBot

For now, you can only use UTBot under Ubuntu 20.04 and above. Navigate to the Releases GitHub page and download last version of UTBot.

UTBot is distrbuted as an archive that contains:

A pack utbot_distr.tar.gz that contains UTBot binary and its dependencies;
UTBot plugin for Visual Studio code — utbot_plugin.vsix;
A version version.txt;
A run script unpack_and_run_utbot.sh.

To launch UTBot, unzip the archive and run the ./unpack_and_run_utbot.sh command (we recommend doing it in a fresh directory to make UTBot removing easier). To remove UTBot, simply delete this directory.

To install UTBot VSCode plugin, use VSCode Install from VSIX command.

How to contribute to UTBot

See Contributing guidelines and Developer guide

utbotcpp's People

Contributors

Stargazers

Watchers

utbotcpp's Issues

CMake Options are ignored

I tried to configure CRC32C project with CMake options from README.

So I assigned -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DCMAKE_EXPORT_LINK_COMMANDS=ON -DCRC32C_BUILD_TESTS=0 -DCRC32C_BUILD_BENCHMARKS=OFF -DCRC32C_USE_GLOG=0 to CMake Options setting in Workspace tab and run action UTBot: Configure project. It failed and provided error in logs.

Hoewever, if I comment all code in if(CRC32C_BUILD_TESTS), if(CRC32C_BUILD_BENCHMARKS) and if(CRC32C_USE_GLOG) from CMakeLists.txt, the project is configured successfully.

Move developer documentation to wiki

[c++] Generation tests for class with private members

Add ability to generate tests for classes with private member.
Access to private in klee and tests by hliberacki/cpp-member-accessor.
For check access right add AccessSpecifier to field and fill by clang getAccess().

planned klee file code

MEMBER_ACCESSOR(point_x, point, x, int)
MEMBER_ACCESSOR(point_y, point, y, int)

int klee_entry__private_set_zero(int utbot_argc, char ** utbot_argv, char ** utbot_envp) {
    class point t;
    klee_make_symbolic(&t, sizeof(t), "t");
    klee_prefer_cex(&t, accessor::accessMember<point_x>(t) >= -10  & accessor::accessMember<point_x>(t) <= 10);
    klee_prefer_cex(&t, accessor::accessMember<point_y>(t) >= -10  & accessor::accessMember<point_y>(t) <= 10);
    //////////////////////////////////////////// 
    class point t_post;
    klee_make_symbolic(&t_post, sizeof(t_post), "t_post");
set_zero(t);
    klee_assume(accessor::accessMember<point_x>(t) == accessor::accessMember<point_x>(t_post));
    klee_assume(accessor::accessMember<point_y>(t) == accessor::accessMember<point_y>(t_post));
    return 0;
}

planned result test code (probably MEMBER_ACCESSOR will move to header)

MEMBER_ACCESSOR(point_x, point, x, int)
MEMBER_ACCESSOR(point_y, point, y, int)

TEST(regression, set_zero_test_1)
{
    class point t = {0, 0};
    set_zero(t);
    class point expected_t = {0, 0};
    EXPECT_EQ(accessor::accessMember<point_x>(expected_t), accessor::accessMember<point_x>(t));
    EXPECT_EQ(accessor::accessMember<point_y>(expected_t), accessor::accessMember<point_y>(t));
}

Auto fill UTBot source folders

current realization
sourcePaths = Paths::filterPathsByDirNames(sourcePathsCandidates, requestSourcePaths, { ".c", ".cpp", ".cc" });
change to auto detect source path by compile_commands.json and link_commands.json

Error suite must throw exceptions

utbot_server_restart.sh doesn't shutdown previous running instance

Now restart script just run new instance of utbot server, without kill previous running instance.

Provide warning message about Ubuntu version

User needs to get warning message about unsupported Ubuntu if it's version less than 18.04

Release run test not working

Step to reproduce:

Run release version
Generate tests for calc.c
Run tests

/home/wx1061625/utbot-supplements/utbot_source/UTBotCpp/integration-tests/c-example/tests/stubs/lib/pointers/function_pointers_stub.h:104:15: error: two or more data types in declaration specifiers
  104 | typedef float _Float32;

Format logs - like java

Generate tests for overloading function

Add possibility to generate tests for function with parameters overloading.
like

int sum(int a, int b) {
    return a + b;
}

int sum(int a, int b, int c) {
    return a + b + c;
}

hint: also add human readable test name for such functions

Track mutation in class members

Add track mutation of class members, then generate tests for methods.

Add c++ future feature list on wiki

"No tests generated. Maybe you tried to generate tests placing cursor on invalid line." error when generating for class

Version v1.0.31-alpha
project cpp-example

Steps to reproduce:

Launch vs-code plugin
Open cpp-example project
Go to SimpleClass.h
Place cursor inside class body -> generate for current class

Then the following error is shown:

Add interactive mode to klee

When we generate tests for all functions in a file (or in a project), we run klee as many times as there are functions in the file (or in the project). The file, which we send to klee, doesn't change between launches add we lose time on preprocessing, which launches every time.
Solution: send to klee array of entry_points and run preprocessing one time, after that run ordinary process to generate tests

'stddef.h' file not found when running Clang tool

Another problem based on #77 requires the separate issue, I think.

As for now, we are getting the following error while running Clang tool:

In file included from ../src/crc32c_portable.cc:5:
In file included from ../src/./crc32c_internal.h:10:
/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/cstddef:50:10: fatal error: 'stddef.h' file not found
#include <stddef.h>

So, let's compare include paths for Clang tool and Clang compiler. To do that, just add "-v" option to both of them.

Clang tool

#include "..." search starts here:
#include <...> search starts here:
 /home/utbot/crc32c/build/include
 /home/utbot/crc32c/include
 .
 /utbot_distr/klee/include
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward
 /usr/local/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.

Clang compiler

#include "..." search starts here:
#include <...> search starts here:
 /home/utbot/crc32c/build/include
 /home/utbot/crc32c/include
 .
 /utbot_distr/klee/include
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward
 /usr/local/include
 /utbot_distr/install/lib/clang/10.0.1/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.

The only difference is that compiler command has extra path /utbot_distr/install/lib/clang/10.0.1/include. Although, the documentation states the following

default location to look for builtin headers is in a path $(dirname /path/to/tool)/../lib/clang/3.3/include relative to the tool binary

Apparently, Clang tool expects to be executed from the directory that contains tools' binaries (/utbot_distr/install/bin/ in our case). And since it is not, we need to pass this path manually by appending -internal-isystem /utbot_distr/install/lib/clang/10.0.1/include option to somewhere, I guess. At least it should work for clang, don't know about g++.

What do you reckon about the above?

Add integration tests for plugin to github actions

ld failed then try running generated tests

Release package version tag 1.0.93
undefined reference to __gcov_init

Introduce automated checks for build artifacts

This script is intended to perform the following checks as a smoke test for each UTBot CPP build result:

The installation script is able to unarchive the server tar.gz
The run script sets all the necessary environment variables and paths correctly and the server is able to start normally based on the environment set up
The run script can stop the server process - TBC!

Log such as "Generation finished for file <f> with 5 tests in error suite and 20 test in regression suite" (support hyperlinks)

Assure klee logs is in server logs

Support test generation for lists, trees, graphs in C++

Client can see ALL logs

Make running tests portable (switch to relative paths, isolation environment)

In the current solution we can't run tests if we pass them to another pc

Now

Use absolute path to source in MakeFile
Write intermediate files into tmp directory.

Solution

Move MakeFile for run tests into tests folder
Replace tmp directory by directory inside ${project}/${build}/utbot_build
Change to relative paths in MakeFile

If UTBotCPP plugin installed into VSCode, it spams client log with message "UNAVAILABLE: No connection established"

Current behavior: Write message to log every second "UNAVAILABLE: No connection established"

Expected behavior: Still ping every second but show in long only moments when connection is 1) broken and 2) reestablished.

Update readme for the project

The existing documentation is very limited… I would like to get more information from it 🙏

use more c++ like code for c++ tests

In test files write ClassName expected(0, 0) instead of class ClassName expected = {0, 0};

"Couldn't link any file" error when generating tests for project

Version v1.0.31-alpha
Test project

Steps to reproduce:

Select lib folder
Right click -> UTbot: Generate Tests for Project

Actual -> FAILED_PRECONDITION: Couldn't link any files

Expected -> Test should be generated

Old GCC version in docker base_env

Docker base_env image now has gcc-7 by default, it should be gcc-9

CMake for lib in release package

*.cmake files In release package without library.

gtest for example

CMake Error at /home/pavely/utbot-release-1.0.49/utbot_distr/install/lib/cmake/GTest/GTestTargets.cmake:99 (message):
The imported target "GTest::gtest" references the file
"/home/pavely/utbot-release-1.0.49/utbot_distr/install/lib/libgtest.a"
but this file does not exist.  Possible reasons include:

Use operator equal for class compare

In class that has operator== use it for assert in test and klee_assume in *_klee.cpp files.

probably add some config for it.

log files has incorrect symbols

red ESC symbol in VSCode, then open log files

Run tests fail

makefile for tests use default gcc and link utbot x86_64-linux-gnu

Use binary format to represent pointers in test

Now we are getting warnings like those:

/__w/UTBotCpp/UTBotCpp/server/test/suites/syntax/tests/linked_list_test.cpp:97:84: warning: integer literal is too large to be represented in a signed integer type, interpreting as unsigned [-Wimplicitly-unsigned-literal]
    struct HardNode head = {{0, {NULL, 0}, 0.000000e+00}, 'c', (struct HardNode *) 17940362863843014904};

Add possibility to generate tests for function with rvalue parameters

//function example
int double_abs_rvalue_reference(int&& a) {
    if (a < 0) {
        return -2 * a;
    }
    return a;
}

Add libcxx into base_env docker

Failed to generate test for code with external function

Problem:
Failed to generate test for absent global variable definition and absent function with default settings.

Example for absent function definition:

// Some tests for function 'get_sign_struct' were skipped, as execution of function leads KLEE to the internal error. See console log for more details.

KLEE logs:

Error: failed external call: unnamed_func

If you run utbot with LD_PRELOAD with library that defines variable or function, test is generated. Seems like klee option --external-calls=all works only when klee sees the function. However, in order to see it, you need to manually link the code with function or use LD_PRELOAD

To reproduce:
main.c

int get_sign_struct(int a) {
    a = unnamed_func();
}

Library libfoo.so:

cd /home/utbot/mylib
clang -c -Wall -Werror -fpic foo.c
clang -shared -o libfoo.so foo.o

foo.c:

int unnamed_func() {
        return 5;
}

Part of CMakeLists.txt for compiling project:

link_directories(/home/utbot/mylib)
add_executable(simple main.c)
target_link_libraries(simple foo)

Launching utbot with LD_PRELOAD=/home/utbot/mylib/libfoo.so generates test, but doesn't generate without it.

What to do:
Need to generate test somehow without LD_PRELOAD.
For this, the external library should be linked somehow.

There is a 10-year-ago solution with --load klee option. However klee doesn't have this option now.
https://klee-dev.keeda.stanford.narkive.com/1mUJw9yN/how-to-make-klee-work-on-the-external-libraries-in-native-bitcode

b) If you have a library in native format, pass --load=</path/to/library> to KLEE (usual restrictions involving externals apply).

`UTBot: Prepare project model` action in command palette

If project model configuration is failed by one of the reasons:

build folder is not create
compile_commands.json and link_commands.json are failed to generate from CMake/Make
source folders are not marked as src

then user have no chance to configure project.

We need explicit command in VSCode command pallete that will repeat aforementioned actions. Also we should create a special section in Troubleshooting wiki to tell user when that he is able to invoke this command.

link_commands.json is different for Bear and CMake

From what I have seen, Bear and CMake provide different link_commands.json on the test project from that commit. In short, it seems like Bear doesn't treat the installed library like a file needed for linking, while CMake does. Please, see examples below

Bear

[
    {
        "arguments": [
            "/utbot_distr/install/bin/clang",
            "-fuse-ld=gold",
            "/utbot_distr/install/lib/libz3.so.4.8.7.0",
            "-o",
            "installed",
            "CMakeFiles/installed.dir/main.c.o"
        ],
        "directory": "/home/utbot/UTBotCpp/server/test/suites/run/installed/build",
        "files": [
            "CMakeFiles/installed.dir/main.c.o"
        ]
    }
]

CMake

[
	{
		"command" : "/utbot_distr/install/bin/clang -fuse-ld=gold CMakeFiles/installed.dir/main.c.o -o installed  /utbot_distr/install/lib/libz3.so.4.8.7.0",
		"directory" : "/home/utbot/UTBotCpp/server/test/suites/run/installed/build-cmake",
		"files" : 
		[
			"/home/utbot/UTBotCpp/server/test/suites/run/installed/build-cmake/CMakeFiles/installed.dir/main.c.o",
			"/utbot_distr/install/lib/libz3.so.4.8.7.0"
		]
	}
]

Generate tests for class without constructor from all fields

In current realization we use constructor from initializer list for tests

Point p = {0, 0};

And default constructor for *_klee.cpp files

Point p;

it's work in class that has constructor from all fields like

class point {
     int x;
     int y;
     point(int x, int y) : x(x), y(y) {}
};

but some class hasn't this constructors, or hasn't any constructor

class point {
    int x;
    int y;
    double dist;
    point() = delete;
    point(int x, int y) : x(x), y(y) {
        dist = sqrt(x*x + y*y);
    }
};

`utbot_server_restart.sh` doen't notify if port is already in use

Current behavior:

d00555580@src-gpu-220:~/utbot_distr$ ./utbot_server_restart.sh
[2021-12-07 18:23:17] Starting a new server process; logs are written into [/home/d00555580/logs/utbot] folder
[2021-12-07 18:23:17] INFO Starting new [utbot] process right now
[2021-12-07 18:23:17] INFO New [utbot] instance with pid [34009] has been started, process options are [server --port 2121 --log=/home/d00555580/logs/utbot --tmp=/home/d00555580/logs/utbot/tmp], pid file created: [/home/d00555580/utbot_distr/server-install/utbot.pid]; STDOUT and STDERR redirected to [/home/d00555580/logs/utbot/utbot-20211207182317.log]
d00555580@src-gpu-220:~/utbot_distr$
d00555580@src-gpu-220:~/utbot_distr$ cat /home/d00555580/logs/utbot/utbot-20211207182317.log
2021-12-07 18:23:17.547 [UnitTestBot     ]             Server.cpp:45    INFO| UnitTestBot Server, build DEVBUILD
2021-12-07 18:23:17.547 [UnitTestBot     ]             Server.cpp:46    INFO| Log path: "/home/d00555580/logs/utbot"
2021-12-07 18:23:17.547 [UnitTestBot     ]             Server.cpp:47    INFO| Tmp directory path: "/home/d00555580/logs/utbot/tmp"
2021-12-07 18:23:17.547 [UnitTestBot     ]             Server.cpp:48    INFO| Executable path: "/home/d00555580/utbot_distr"
2021-12-07 18:23:17.547 [UnitTestBot     ]             Server.cpp:58    INFO| address: 0.0.0.0:2121

E1207 18:23:17.549110976   34009 server_chttp2.cc:40]        {"created":"@1638890597.548986389","description":"No address added out of total 1 resolved","file":"/utbot_distr/grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":397,"referenced_errors":[{"created":"@1638890597.548982867","description":"Failed to add any wildcard listeners","file":"/utbot_distr/grpc/src/core/lib/iomgr/tcp_server_posix.cc","file_line":341,"referenced_errors":[{"created":"@1638890597.548957891","description":"Unable to configure socket","fd":5,"file":"/utbot_distr/grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1638890597.548952991","description":"Address already in use","errno":98,"file":"/utbot_distr/grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]},{"created":"@1638890597.548981973","description":"Unable to configure socket","fd":5,"file":"/utbot_distr/grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1638890597.548978571","description":"Address already in use","errno":98,"file":"/utbot_distr/grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]}]}]}
/home/d00555580/utbot_distr/server-install/utbot(_ZN4llvm3sys15PrintStackTraceERNS_11raw_ostreamE+0x2a)[0x423657a]
/home/d00555580/utbot_distr/server-install/utbot(_ZN4llvm3sys17RunSignalHandlersEv+0x34)[0x4234504]
/home/d00555580/utbot_distr/server-install/utbot[0x4234648]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7f3dd7b79980]
/home/d00555580/utbot_distr/server-install/utbot(_ZN6Server3runEt+0x4e9)[0x10859c9]
/home/d00555580/utbot_distr/server-install/utbot(_ZN8CLIUtils5parseEiPPcRN3CLI3AppE+0x1d04)[0x1026d04]
/home/d00555580/utbot_distr/server-install/utbot(main+0x10a)[0x100408a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7f3dd48bbbf7]
/home/d00555580/utbot_distr/server-install/utbot(_start+0x2a)[0x1003eba]

Expected behavior:
utbot_server_restart.sh failed with explicit message in stderr that port is occupied and user need to specify another port (show command line how to specify it).

Divide tests into error/regression via regions

No tests generated for crc32c/src/crc32c_portable.cc file in CRC32C project

I tried to generate tests for crc32c/src/crc32c_portable.cc file in CRC32C project in dev docker and it failed.

See logs for more details.

GCC errors on running `calc_test.ccp`

Version 1.0.57
Test project

Steps to reproduce:

Generate test for calc.c in src
Try to run calc_test.ccp

Actual ->
gcc: error: unrecognized argument to -fsanitize= option: ‘pointer-overflow’; did you mean ‘signed-integer-overflow’?
gcc: error: unrecognized argument to -fsanitize= option: ‘builtin’

Expected -> Test should be run with coverage

Create simple C++ project and document step-by-step how it works in SFTP scenario

Vulnerabilities in vscode-plugin

We are getting warnings when building vscode-plugin

utbot@c02d389377c3:~/UTBotCpp/vscode-plugin$ ./build.sh 
(node:26208) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
(Use `node --trace-warnings ...` to show where the warning was created)

up to date, audited 561 packages in 3s

33 packages are looking for funding
  run `npm fund` for details

26 vulnerabilities (17 moderate, 6 high, 3 critical)

Please, pay attention to report of known vulnerabilities.

/home/utbot/UTBotCpp/integration-tests/c-example/tests/stubs/lib/pointers/function_pointers_stub.h:112:21: error: two or more data types in declaration specifiers
  112 | typedef long double _Float64x;
      |                     ^~~~~~~~~
/home/utbot/UTBotCpp/integration-tests/c-example/tests/stubs/lib/pointers/function_pointers_stub.h:112:1: warning: useless type name in empty declaration
  112 | typedef long double _Float64x;
      | ^~~~~~~

Change or delete unavailable links

Change or delete unavailable links to internal server like http://jira-msc.rnd.huawei.com, http://utbot.inhuawei.com, https://confluence-msc.rnd.huawei.com

unittestbot / utbotcpp Goto Github PK

utbotcpp's Introduction

What is UTBotCpp?

Try UTBot demo online here!

How to install and use UTBot

How to contribute to UTBot

utbotcpp's People

Contributors

Stargazers

Watchers

Forkers

utbotcpp's Issues

Clang tool

Clang compiler

Now

Solution

Bear

CMake

Recommend Projects

Recommend Topics

Recommend Org