rwth-hpc / cmake-codecov Goto Github PK
View Code? Open in Web Editor NEWCMake module for code coverage
License: BSD 3-Clause "New" or "Revised" License
CMake module for code coverage
License: BSD 3-Clause "New" or "Revised" License
Clang and gcov may support different interfaces. This can cause warnings and even crashes. FindGcov.cmake should implement a check to check if all compilers are compatible to gcov.
Another way could be to check if the compiler has an own gcov implementation, e.g. clang ships a binary named llvm-gcov which may be used as alternative to gcov.
I couldn't find a way to use the macros for a header-only library - I wanted to have a few executables exercising the coverage only in the header - I tried this:
add_library(dummy STATIC foo2.h)
set_target_properties(dummy PROPERTIES LINKER_LANGUAGE CXX)
add_coverage(dummy)
and had a few tests added with add_test
but their targets had not add_coverage()
called for them (intentionally)
Steps to reproduce:
CMakeLists.txt:
cmake_minimum_required(VERSION 3.4)
project(test CXX)
set(CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake/codecov/cmake" ${CMAKE_MODULE_PATH})
find_package(codecov)
add_executable(foo test.cc)
add_coverage(foo)
test.cc:
#include <iostream>
using namespace std;
int main(int argc, char **argv)
{
cout << "Hello World!\n";
return 0;
}
Setup:
$ git submodule add https://github.com/RWTH-ELP/CMake-codecov.git cmake/codecov
$ mkdir build
$ cd build
$ CXX=clang++ CC=clang cmake -G Ninja .. -DENABLE_COVERAGE=On
$ ninja-build -v foo
$ ./foo
$ ninja-build -v foo-gcov
Actual result:
$ ninja-build -v foo-gcov
[1/1] cd /home/juser/cmake-codecov-test/build/CMakeFiles/foo.dir && LLVM_COV_BIN=/usr/bin/llvm-cov /home/juser/cmake-codecov-test/cmake/codecov/cmake/llvm-cov-wrapper /home/juser/cmake-codecov-test/build/CMakeFiles/foo.dir/test.cc.gcno > /dev/null
../test.cc: No such file or directory
$ ninja-build -v foo-gcov
ninja: no work to do.
Expected result:
A .gcov
file under in the build directory, i.e. CMakeFiles/foo.dir/test.cc.gcov
.
I tested it on Fedora 23.
When compiling with the CMake makefile generator the gcov report is generated as expected.
Hi, i just forked this project in order to add some support for windows platforms. When i added the project into travis i saw that the build fails for the "clang-builds" (on branch master).
Would you please take a look into this and maybe rerun your builds as well?
https://travis-ci.org/farrrb/CMake-codecov/builds/335606460
Thank you,
Fabian
I am trying to use CMake-codecov
to capture code coverage for Catch and after some bumbling around I got it to work properly under Linux for both GCC and Clang.
However, I ran into trouble when I was trying to integrate it with OS X builds. The scripts properly detect that AppleClang supports coverage, finds lcov
, but then is unable to find gcov
, with this output
CMake Warning at CMake/FindGcov.cmake:129 (message):
No coverage evaluation binary found for AppleClang.
Call Stack (most recent call first):
CMake/Findcodecov.cmake:248 (add_gcov_target)
CMake/Findcodecov.cmake:38 (add_coverage_target)
CMakeLists.txt:308 (add_coverage)
CMake Warning at CMake/FindLcov.cmake:154 (message):
No coverage evaluation binary found for AppleClang.
Call Stack (most recent call first):
CMake/FindLcov.cmake:39 (lcov_capture_initial_tgt)
CMake/Findcodecov.cmake:249 (add_lcov_target)
CMake/Findcodecov.cmake:38 (add_coverage_target)
CMakeLists.txt:308 (add_coverage)
CMake Warning at CMake/FindLcov.cmake:245 (message):
No coverage evaluation binary found for AppleClang.
Call Stack (most recent call first):
CMake/FindLcov.cmake:42 (lcov_capture_tgt)
CMake/Findcodecov.cmake:249 (add_lcov_target)
CMake/Findcodecov.cmake:38 (add_coverage_target)
CMakeLists.txt:308 (add_coverage)
The full travis output can be found here.
Btw, thanks for the script. After I stopped being dumb about using it on Linux, it worked nicely.
I just try to use CMake-codecov (I'm new to lcov, just learning, so sorry if my question is out of scope)
I want to automatically call check target before calling lcov.
after calling coverage_evaluate()
at the end of main CMakeLists.txt
, I added add_dependencies(lcov check)
But I have one issue, I want to "zerocounter" lcov data before tests, to have fresh results in case of rebuild.
I have done
add_custom_command(TARGET lcov-zerocounter COMMAND ${LCOV_BIN} --quiet --zerocounter --directory ${CMAKE_BINARY_DIR})
add_dependencies(check lcov-zerocounter)
But I'm asking myself should this be present directly in CMake-codecov ? or is there a better way to handle lcov data cleanup before capture.
Thanks.
Gcov prints a warning, if a gcno file contains no functions. Lcov aborts in this situation.
A possible solution to solve this issue may be to delete the gcno file after compilation, if it contains no functions and lcov will ignore this (non existing) file.
HI,
Where is the ENABLE_COVERAGE_ALL
option applied?
I can't see a reference to it in the source.
This module should be checked for compatibility with older CMake versions and a badge for the compatibility added.
Are there variables exposed to generate branch coverage as it appears to be disabled by default?
The same example provided in issue #6 shows that also the support for lcov is failing when using the Ninja generator.
$ cmake -G Ninja -DCMAKE_BUILD_TYPE=Debug -DENABLE_COVERAGE=ON ..
$ ninja -v foo
[1/2] /usr/bin/c++ -g -O0 -g -fprofile-arcs -ftest-coverage -MD -MT CMakeFiles/foo.dir/test.cc.o -MF CMakeFiles/foo.dir/test.cc.o.d -o CMakeFiles/foo.dir/test.cc.o -c ../test.cc
[2/2] : && /usr/bin/c++ -g -O0 -g -fprofile-arcs -ftest-coverage CMakeFiles/foo.dir/test.cc.o -o foo && :
$ ./foo
Hello World!
$ ninja -v foo-genhtml
[1/8] cd /home/giaco/tmp/lcov-cmake/build && /usr/bin/geninfo --quiet --base-directory /home/giaco/tmp/lcov-cmake --initial --gcov-tool /usr/bin/gcov-7 --output-filename /home/giaco/tmp/lcov-cmake/build/CMakeFiles/foo.dir/test.cc.info.init --no-external /home/giaco/tmp/lcov-cmake/build/CMakeFiles/foo.dir/test.cc.gcno
geninfo: WARNING: could not open /home/giaco/tmp/test.cc
geninfo: WARNING: some exclusion markers may be ignored
[2/8] cd /home/giaco/tmp/lcov-cmake/build && cat /home/giaco/tmp/lcov-cmake/build/CMakeFiles/foo.dir/test.cc.info.init > /home/giaco/tmp/lcov-cmake/build/lcov/data/init/foo.info.raw
[3/8] cd /home/giaco/tmp/lcov-cmake/build && /usr/bin/lcov --quiet -a /home/giaco/tmp/lcov-cmake/build/lcov/data/init/foo.info.raw --output-file /home/giaco/tmp/lcov-cmake/build/lcov/data/init/foo.info --base-directory /home/giaco/tmp/lcov-cmake --initial && /usr/bin/lcov --quiet -r /home/giaco/tmp/lcov-cmake/build/lcov/data/init/foo.info --output-file /home/giaco/tmp/lcov-cmake/build/lcov/data/init/foo.info --initial
FAILED: lcov/data/init/foo.info
cd /home/giaco/tmp/lcov-cmake/build && /usr/bin/lcov --quiet -a /home/giaco/tmp/lcov-cmake/build/lcov/data/init/foo.info.raw --output-file /home/giaco/tmp/lcov-cmake/build/lcov/data/init/foo.info --base-directory /home/giaco/tmp/lcov-cmake --initial && /usr/bin/lcov --quiet -r /home/giaco/tmp/lcov-cmake/build/lcov/data/init/foo.info --output-file /home/giaco/tmp/lcov-cmake/build/lcov/data/init/foo.info --initial
lcov: ERROR: no valid records found in tracefile /home/giaco/tmp/lcov-cmake/build/lcov/data/init/foo.info.raw
ninja: build stopped: subcommand failed.
At least the first error (geninfo: WARNING: could not open /home/giaco/tmp/test.cc
) can be fixed passing ${CMAKE_BINARY_DIR}
instead of ${CMAKE_SOURCE_DIR}
to the --base-directory
option of geninfo
. This because Ninja works with paths relative to the binary directory.
Everything works fine with the Makefile generator, which works with absolute paths.
Instead of printing the same message over and over again (e.g. if the compiler doesn't support code coverage, but FindCodecov.cmake
is called more than once), the find_package_message()
function should be used to print these messages only once.
Some source code layout guidelines suggest to name test source files with the combined extension .t.cpp
. Unfortunately the use of get_filename_component
at
CMake-codecov/cmake/Findcodecov.cmake
Line 136 in 9a24e83
.t.cpp
instead of just .cpp
, which makes the source file not recognizable.
An alternative would be the use of a regular expression, i.e. instead of:
get_filename_component(FILE_EXT "${FILE}" EXT)
string(TOLOWER "${FILE_EXT}" FILE_EXT)
string(SUBSTRING "${FILE_EXT}" 1 -1 FILE_EXT)
one could use
string(REGEX REPLACE "^.*\\.([^\\.]+)$" "\\1" FILE_EXT "${FILE}")
string(TOLOWER "${FILE_EXT}" FILE_EXT)
The latest version of cmake warns at
CMake-codecov/cmake/FindLcov.cmake
Line 56 in 2682f89
[cmake] CMake Warning (dev) at /usr/local/share/cmake-3.17/Modules/FindPackageHandleStandardArgs.cmake:272 (message):
[cmake] The package name passed to `find_package_handle_standard_args` (lcov) does
[cmake] not match the name of the calling package (Lcov). This can lead to
[cmake] problems in calling code that expects `find_package` result variables
[cmake] (e.g., `_FOUND`) to follow a certain pattern.
[cmake] Call Stack (most recent call first):
[cmake] cmake/FindLcov.cmake:56 (find_package_handle_standard_args)
[cmake] cmake/Findcodecov.cmake:271 (find_package)
[cmake] CMakeLists.txt:49 (find_package)
[cmake] This warning is for project developers. Use -Wno-dev to suppress it.
The fix is simply to pass Lcov
instead of lcov
.
The .gcno file generated at compile time should be deleted when the object file will be deleted.
The .gcda file generated at execution time should be deleted when the object file will be deleted. If one object of a target gets recompiled all .gcda files of the target should be deleted. (I think this could be done with add_custom_command(... PRE_BUILD ...)
and find -name *.gcda
). This should also be done with dependent targets.
.gcov files and lcov output should be handled in the same way as .gcda files.
All of these files should be deleted when make clean is called.
I am using CMake-covedoc on Windows 10 with MinGW-w64. It works great but the script includes some logs that made me think there were issues at first and which might be silenced to avoid confusion:
-- Try GNU code coverage flag = [-O0 -g -fprofile-arcs -ftest-coverage]
-- Performing Test COVERAGE_FLAG_DETECTED
-- Performing Test COVERAGE_FLAG_DETECTED - Success
-- Try code coverage flag = [-O0 -g -fprofile-arcs -ftest-coverage]
CMake Warning at cmake/Findcodecov.cmake:121 (message):
Code coverage is not available for compiler. Targets using this compiler
will be compiled without it.
Call Stack (most recent call first):
testsuite/CMakeLists.txt:42 (find_package)
-- Try code coverage flag = [-O0 -g --coverage]
CMake Warning at cmake/Findcodecov.cmake:121 (message):
Code coverage is not available for compiler. Targets using this compiler
will be compiled without it.
Call Stack (most recent call first):
testsuite/CMakeLists.txt:42 (find_package)
What is not obvious from these logs is that after successfully finding MinGW-w64 g++, Findcodecov.cmake
looks for a compiler for RC
(which is among the languages in ENABLED_LANGUAGES
), fails to find a compiler (there is no CMAKE_RC_COMPILER_ID
) and as a result doesn't detect any flag.
To avoid those logs, I see two solutions:
RC
when found in ENABLED_LANGUAGES
.${COMPILER}
is empty.Both solutions could probably live side by side in order to avoid the proposed warning anyway when RC
isn't found.
Thanks again for the project, it's really nice!
LLVM clang also understands the gcc options for enabling coverage collection (e.g. --coverage
and -fprofile-arcs -ftest-coverage
). But - in that compatibility mode - llvm targets a certain gcc/gcov version. For example one that isn't supported anymore by current gcov
/lcov
(yielding inaccuracies and lcov errors).
Thus, to increase the portability of the cmake codecov module, it would be useful to also add support for llvm-cov
.
The project looks quite good. However, I can't use it in most of my github projects due to the restriction to GPL only license. Would it be considerable for you to change the license to something very liberal, so it becomes a no brainer for most project maintainers to add your cmake code to their projects?
I am facing a Coverage disabled
warning while trying to use CMake-codecov
.
I am building executable with TARGET_OBJECTS
option, that prevent CMake-codecov
to discover the language used. The internal variable NUM_COMPILERS
is equal to zero in Findcodecov.cmake
. Would it be possible to investigate this development?
ADD_EXECUTABLE(my_test_prog $<TARGET_OBJECTS:my_test_prog_obj>)
add_coverage(my_test_prog)
Below is a CMakeLists.txt, that repoduces the problem (one needs to have codecov
available to CMake
)
CMAKE_MINIMUM_REQUIRED(VERSION 2.8.8)
PROJECT(hello_world)
IF(NOT DEFINED CMAKE_BUILD_TYPE)
SET(CMAKE_BUILD_TYPE Coverage)
ENDIF()
STRING(TOUPPER "${CMAKE_BUILD_TYPE}" CMAKE_BUILD_TYPE_UPPER)
IF(${CMAKE_BUILD_TYPE_UPPER} STREQUAL "COVERAGE")
MESSAGE(STATUS "Adding coverage")
SET(CMAKE_MODULE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/cmake) # ;${CMAKE_MODULE_PATH}
FIND_PACKAGE(codecov)
ENDIF()
FILE(WRITE "${CMAKE_CURRENT_BINARY_DIR}/my_test_prog.cpp"
"
#include <iostream>
using namespace std;
int main()
{
cout << \"Hello World!\" << endl;
return 0;
}
")
ADD_LIBRARY(my_test_prog_obj OBJECT ${CMAKE_CURRENT_BINARY_DIR}/my_test_prog.cpp)
ADD_EXECUTABLE(my_test_prog $<TARGET_OBJECTS:my_test_prog_obj>)
add_coverage(my_test_prog)
The log is here
-- Adding coverage
-- Try code coverage flag = [-O0 -g -fprofile-arcs -ftest-coverage]
-- Try code coverage flag = [-O0 -g --coverage]
CMake Warning (dev) at cmake/Findcodecov.cmake:174 (message):
Coverage disabled for target my_test_prog because there is no sanitizer
available for target sources.
Call Stack (most recent call first):
cmake/Findcodecov.cmake:34 (add_coverage_target)
CMakeLists.txt:25 (add_coverage)
This warning is for project developers. Use -Wno-dev to suppress it.
-- Configuring done
-- Generating done
-- Build files have been written to: buildCoverage
#make
Scanning dependencies of target my_test_prog_obj
[ 50%] Building CXX object CMakeFiles/my_test_prog_obj.dir/my_test_prog.cpp.obj
[ 50%] Built target my_test_prog_obj
Scanning dependencies of target my_test_prog
[100%] Linking CXX executable my_test_prog.exe
[100%] Built target my_test_prog
Guillaume
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.