nickbruun / hayai Goto Github PK

View Code? Open in Web Editor NEW

364.0 364.0 54.0 1.13 MB

C++ benchmarking framework

Home Page: https://bruun.co/2012/02/07/easy-cpp-benchmarking

License: Other

C++ 89.44% Shell 0.31% CMake 4.09% Python 6.16%

hayai's People

Contributors

Stargazers

Watchers

hayai's Issues

Warnings on Visual Studio 2013

warning C4244: 'return' : conversion from 'const uint64_t' to 'double', possible loss of data hayai_test_result.hpp line 52

warning C4244: 'return' : conversion from 'const uint64_t' to 'double', possible loss of data hayai_test_result.hpp line 73

warning C4244: 'return' : conversion from 'const uint64_t' to 'double', possible loss of data hayai_test_result.hpp line 80

How to access benchmark test results after running the tests?

Very simple piece of example code:
https://bpaste.net/show/7df01dce723f

How can I access benchmark values after running the tests? For example if I needed average iteration times from multiple tests for further analysis?

Request for guidance regarding license.

I've refactored hayai's build instructions and reorganized the files a bit, but haven't changed any actual code. This was to support making your library available to the ROS build system (www.ros.org). How would you like the license to be handled? I'm thinking just leave it as is, but add a new line for a URL pointing to my fork? I don't expect you to pull these changes back in. I'd appreciate some feedback/guidance on how you'd like me to proceed.

Overflow

It would be nice if you could detect overflows and you are doing your calculations, I ended up having negative value and even very small ones in cases where the test lasts for a while (on low performance hardware).
I suggest starting by using uint64_t instead of signed values, and detect overflows when computing the average times.
I also think you should keep us and s seperated.

Measuring fixture setup and benchmark separately?

I noticed that the json output only lists runs, which I assume includes the setup times. Is is possible to measure the actual benchmark iteration separately? In the console view, is the average iteration time is just (average runtime) / (number of iterations)?

Improved command line with support for multiple outputters

Inclusion filters

Is there an example of the inclusion filters? I'd love an automatic --hayai_filter=... commandline option feature, which is parsed by a standard provided hayai::InitHayai(&argc, argv) function, much like the googletest filters.

Licensing

Hi,
this is a very interesting project. I would like to try and use at work. However, personal licenses are not compatible with our work policies. Would you be willing to change the license to something well established? For example one of these? The Apache License 2.0 seems to be what you are aiming for... Unfortunately without the beer 😄

Thanks for considering!

Support for experiments with baseline and alternative benchmarks

Homebrew formula for version 1.0.0

Specifying number of runs/iterations at runtime

The only way I was able to do it is by hacking and removing const here:

hayai/src/hayai_benchmarker.hpp

Line 287 in 9bdca1c

static std::vector<const TestDescriptor*> ListTests()

Otherwise, those numbers are set at startup time and can't be changed later. Or am I missing something?

Odd option to save benchmark results

Would it be possible to add an option to hayai to save the benchmark results, e.g. in XML or JSON format?

Do not use absolute paths for install

Absolute paths breaks CPack, any project including hayai with add_subdirectory would not be able to use CPack.

hayai defaults to clock implementation gettimeofday even though higher resolution clocks avaiable

Problem Summary:
Default clock implementation is always gettimeofday because _POSIX_TIMERS is not found when generating libhayai_main.a even thought getconf -a clearly shows that _POSIX_TIMERS is defined and set.

Expected Results:
sample/sample --help sets clock implementation to clock_gettime(CLOCK_MONOTONIC_RAW)

Actual Results:
sample/sample --help sets clock implementation to gettimeofday

Environement:
Ubuntu 16.04

getconf -a ouput:

LINK_MAX                           65000
_POSIX_LINK_MAX                    65000
MAX_CANON                          255
_POSIX_MAX_CANON                   255
MAX_INPUT                          255
_POSIX_MAX_INPUT                   255
NAME_MAX                           255
_POSIX_NAME_MAX                    255
PATH_MAX                           4096
_POSIX_PATH_MAX                    4096
PIPE_BUF                           4096
_POSIX_PIPE_BUF                    4096
SOCK_MAXBUF                        
_POSIX_ASYNC_IO                    
_POSIX_CHOWN_RESTRICTED            1
_POSIX_NO_TRUNC                    1
_POSIX_PRIO_IO                     
_POSIX_SYNC_IO                     
_POSIX_VDISABLE                    0
ARG_MAX                            2097152
ATEXIT_MAX                         2147483647
CHAR_BIT                           8
CHAR_MAX                           127
CHAR_MIN                           -128
CHILD_MAX                          63645
CLK_TCK                            100
INT_MAX                            2147483647
INT_MIN                            -2147483648
IOV_MAX                            1024
LOGNAME_MAX                        256
LONG_BIT                           64
MB_LEN_MAX                         16
NGROUPS_MAX                        65536
NL_ARGMAX                          4096
NL_LANGMAX                         2048
NL_MSGMAX                          2147483647
NL_NMAX                            2147483647
NL_SETMAX                          2147483647
NL_TEXTMAX                         2147483647
NSS_BUFLEN_GROUP                   1024
NSS_BUFLEN_PASSWD                  1024
NZERO                              20
OPEN_MAX                           1024
PAGESIZE                           4096
PAGE_SIZE                          4096
PASS_MAX                           8192
PTHREAD_DESTRUCTOR_ITERATIONS      4
PTHREAD_KEYS_MAX                   1024
PTHREAD_STACK_MIN                  16384
PTHREAD_THREADS_MAX                
SCHAR_MAX                          127
SCHAR_MIN                          -128
SHRT_MAX                           32767
SHRT_MIN                           -32768
SSIZE_MAX                          32767
TTY_NAME_MAX                       32
TZNAME_MAX                         6
UCHAR_MAX                          255
UINT_MAX                           4294967295
UIO_MAXIOV                         1024
ULONG_MAX                          18446744073709551615
USHRT_MAX                          65535
WORD_BIT                           32
_AVPHYS_PAGES                      70513
_NPROCESSORS_CONF                  8
_NPROCESSORS_ONLN                  8
_PHYS_PAGES                        4089825
_POSIX_ARG_MAX                     2097152
_POSIX_ASYNCHRONOUS_IO             200809
_POSIX_CHILD_MAX                   63645
_POSIX_FSYNC                       200809
_POSIX_JOB_CONTROL                 1
_POSIX_MAPPED_FILES                200809
_POSIX_MEMLOCK                     200809
_POSIX_MEMLOCK_RANGE               200809
_POSIX_MEMORY_PROTECTION           200809
_POSIX_MESSAGE_PASSING             200809
_POSIX_NGROUPS_MAX                 65536
_POSIX_OPEN_MAX                    1024
_POSIX_PII                         
_POSIX_PII_INTERNET                
_POSIX_PII_INTERNET_DGRAM          
_POSIX_PII_INTERNET_STREAM         
_POSIX_PII_OSI                     
_POSIX_PII_OSI_CLTS                
_POSIX_PII_OSI_COTS                
_POSIX_PII_OSI_M                   
_POSIX_PII_SOCKET                  
_POSIX_PII_XTI                     
_POSIX_POLL                        
_POSIX_PRIORITIZED_IO              200809
_POSIX_PRIORITY_SCHEDULING         200809
_POSIX_REALTIME_SIGNALS            200809
_POSIX_SAVED_IDS                   1
_POSIX_SELECT                      
_POSIX_SEMAPHORES                  200809
_POSIX_SHARED_MEMORY_OBJECTS       200809
_POSIX_SSIZE_MAX                   32767
_POSIX_STREAM_MAX                  16
_POSIX_SYNCHRONIZED_IO             200809
_POSIX_THREADS                     200809
_POSIX_THREAD_ATTR_STACKADDR       200809
_POSIX_THREAD_ATTR_STACKSIZE       200809
_POSIX_THREAD_PRIORITY_SCHEDULING  200809
_POSIX_THREAD_PRIO_INHERIT         200809
_POSIX_THREAD_PRIO_PROTECT         200809
_POSIX_THREAD_ROBUST_PRIO_INHERIT  
_POSIX_THREAD_ROBUST_PRIO_PROTECT  
_POSIX_THREAD_PROCESS_SHARED       200809
_POSIX_THREAD_SAFE_FUNCTIONS       200809
_POSIX_TIMERS                      200809
TIMER_MAX                          
_POSIX_TZNAME_MAX                  6
_POSIX_VERSION                     200809
_T_IOV_MAX                         
_XOPEN_CRYPT                       1
_XOPEN_ENH_I18N                    1
_XOPEN_LEGACY                      1
_XOPEN_REALTIME                    1
_XOPEN_REALTIME_THREADS            1
_XOPEN_SHM                         1
_XOPEN_UNIX                        1
_XOPEN_VERSION                     700
_XOPEN_XCU_VERSION                 4
_XOPEN_XPG2                        1
_XOPEN_XPG3                        1
_XOPEN_XPG4                        1
BC_BASE_MAX                        99
BC_DIM_MAX                         2048
BC_SCALE_MAX                       99
BC_STRING_MAX                      1000
CHARCLASS_NAME_MAX                 2048
COLL_WEIGHTS_MAX                   255
EQUIV_CLASS_MAX                    
EXPR_NEST_MAX                      32
LINE_MAX                           2048
POSIX2_BC_BASE_MAX                 99
POSIX2_BC_DIM_MAX                  2048
POSIX2_BC_SCALE_MAX                99
POSIX2_BC_STRING_MAX               1000
POSIX2_CHAR_TERM                   200809
POSIX2_COLL_WEIGHTS_MAX            255
POSIX2_C_BIND                      200809
POSIX2_C_DEV                       200809
POSIX2_C_VERSION                   200809
POSIX2_EXPR_NEST_MAX               32
POSIX2_FORT_DEV                    
POSIX2_FORT_RUN                    
_POSIX2_LINE_MAX                   2048
POSIX2_LINE_MAX                    2048
POSIX2_LOCALEDEF                   200809
POSIX2_RE_DUP_MAX                  32767
POSIX2_SW_DEV                      200809
POSIX2_UPE                         
POSIX2_VERSION                     200809
RE_DUP_MAX                         32767
PATH                               /bin:/usr/bin
CS_PATH                            /bin:/usr/bin
LFS_CFLAGS                         
LFS_LDFLAGS                        
LFS_LIBS                           
LFS_LINTFLAGS                      
LFS64_CFLAGS                       -D_LARGEFILE64_SOURCE
LFS64_LDFLAGS                      
LFS64_LIBS                         
LFS64_LINTFLAGS                    -D_LARGEFILE64_SOURCE
_XBS5_WIDTH_RESTRICTED_ENVS        XBS5_LP64_OFF64
XBS5_WIDTH_RESTRICTED_ENVS         XBS5_LP64_OFF64
_XBS5_ILP32_OFF32                  
XBS5_ILP32_OFF32_CFLAGS            
XBS5_ILP32_OFF32_LDFLAGS           
XBS5_ILP32_OFF32_LIBS              
XBS5_ILP32_OFF32_LINTFLAGS         
_XBS5_ILP32_OFFBIG                 
XBS5_ILP32_OFFBIG_CFLAGS           
XBS5_ILP32_OFFBIG_LDFLAGS          
XBS5_ILP32_OFFBIG_LIBS             
XBS5_ILP32_OFFBIG_LINTFLAGS        
_XBS5_LP64_OFF64                   1
XBS5_LP64_OFF64_CFLAGS             -m64
XBS5_LP64_OFF64_LDFLAGS            -m64
XBS5_LP64_OFF64_LIBS               
XBS5_LP64_OFF64_LINTFLAGS          
_XBS5_LPBIG_OFFBIG                 
XBS5_LPBIG_OFFBIG_CFLAGS           
XBS5_LPBIG_OFFBIG_LDFLAGS          
XBS5_LPBIG_OFFBIG_LIBS             
XBS5_LPBIG_OFFBIG_LINTFLAGS        
_POSIX_V6_ILP32_OFF32              
POSIX_V6_ILP32_OFF32_CFLAGS        
POSIX_V6_ILP32_OFF32_LDFLAGS       
POSIX_V6_ILP32_OFF32_LIBS          
POSIX_V6_ILP32_OFF32_LINTFLAGS     
_POSIX_V6_WIDTH_RESTRICTED_ENVS    POSIX_V6_LP64_OFF64
POSIX_V6_WIDTH_RESTRICTED_ENVS     POSIX_V6_LP64_OFF64
_POSIX_V6_ILP32_OFFBIG             
POSIX_V6_ILP32_OFFBIG_CFLAGS       
POSIX_V6_ILP32_OFFBIG_LDFLAGS      
POSIX_V6_ILP32_OFFBIG_LIBS         
POSIX_V6_ILP32_OFFBIG_LINTFLAGS    
_POSIX_V6_LP64_OFF64               1
POSIX_V6_LP64_OFF64_CFLAGS         -m64
POSIX_V6_LP64_OFF64_LDFLAGS        -m64
POSIX_V6_LP64_OFF64_LIBS           
POSIX_V6_LP64_OFF64_LINTFLAGS      
_POSIX_V6_LPBIG_OFFBIG             
POSIX_V6_LPBIG_OFFBIG_CFLAGS       
POSIX_V6_LPBIG_OFFBIG_LDFLAGS      
POSIX_V6_LPBIG_OFFBIG_LIBS         
POSIX_V6_LPBIG_OFFBIG_LINTFLAGS    
_POSIX_V7_ILP32_OFF32              
POSIX_V7_ILP32_OFF32_CFLAGS        
POSIX_V7_ILP32_OFF32_LDFLAGS       
POSIX_V7_ILP32_OFF32_LIBS          
POSIX_V7_ILP32_OFF32_LINTFLAGS     
_POSIX_V7_WIDTH_RESTRICTED_ENVS    POSIX_V7_LP64_OFF64
POSIX_V7_WIDTH_RESTRICTED_ENVS     POSIX_V7_LP64_OFF64
_POSIX_V7_ILP32_OFFBIG             
POSIX_V7_ILP32_OFFBIG_CFLAGS       
POSIX_V7_ILP32_OFFBIG_LDFLAGS      
POSIX_V7_ILP32_OFFBIG_LIBS         
POSIX_V7_ILP32_OFFBIG_LINTFLAGS    
_POSIX_V7_LP64_OFF64               1
POSIX_V7_LP64_OFF64_CFLAGS         -m64
POSIX_V7_LP64_OFF64_LDFLAGS        -m64
POSIX_V7_LP64_OFF64_LIBS           
POSIX_V7_LP64_OFF64_LINTFLAGS      
_POSIX_V7_LPBIG_OFFBIG             
POSIX_V7_LPBIG_OFFBIG_CFLAGS       
POSIX_V7_LPBIG_OFFBIG_LDFLAGS      
POSIX_V7_LPBIG_OFFBIG_LIBS         
POSIX_V7_LPBIG_OFFBIG_LINTFLAGS    
_POSIX_ADVISORY_INFO               200809
_POSIX_BARRIERS                    200809
_POSIX_BASE                        
_POSIX_C_LANG_SUPPORT              
_POSIX_C_LANG_SUPPORT_R            
_POSIX_CLOCK_SELECTION             200809
_POSIX_CPUTIME                     200809
_POSIX_THREAD_CPUTIME              200809
_POSIX_DEVICE_SPECIFIC             
_POSIX_DEVICE_SPECIFIC_R           
_POSIX_FD_MGMT                     
_POSIX_FIFO                        
_POSIX_PIPE                        
_POSIX_FILE_ATTRIBUTES             
_POSIX_FILE_LOCKING                
_POSIX_FILE_SYSTEM                 
_POSIX_MONOTONIC_CLOCK             200809
_POSIX_MULTI_PROCESS               
_POSIX_SINGLE_PROCESS              
_POSIX_NETWORKING                  
_POSIX_READER_WRITER_LOCKS         200809
_POSIX_SPIN_LOCKS                  200809
_POSIX_REGEXP                      1
_REGEX_VERSION                     
_POSIX_SHELL                       1
_POSIX_SIGNALS                     
_POSIX_SPAWN                       200809
_POSIX_SPORADIC_SERVER             
_POSIX_THREAD_SPORADIC_SERVER      
_POSIX_SYSTEM_DATABASE             
_POSIX_SYSTEM_DATABASE_R           
_POSIX_TIMEOUTS                    200809
_POSIX_TYPED_MEMORY_OBJECTS        
_POSIX_USER_GROUPS                 
_POSIX_USER_GROUPS_R               
POSIX2_PBS                         
POSIX2_PBS_ACCOUNTING              
POSIX2_PBS_LOCATE                  
POSIX2_PBS_TRACK                   
POSIX2_PBS_MESSAGE                 
SYMLOOP_MAX                        
STREAM_MAX                         16
AIO_LISTIO_MAX                     
AIO_MAX                            
AIO_PRIO_DELTA_MAX                 20
DELAYTIMER_MAX                     2147483647
HOST_NAME_MAX                      64
LOGIN_NAME_MAX                     256
MQ_OPEN_MAX                        
MQ_PRIO_MAX                        32768
_POSIX_DEVICE_IO                   
_POSIX_TRACE                       
_POSIX_TRACE_EVENT_FILTER          
_POSIX_TRACE_INHERIT               
_POSIX_TRACE_LOG                   
RTSIG_MAX                          32
SEM_NSEMS_MAX                      
SEM_VALUE_MAX                      2147483647
SIGQUEUE_MAX                       63645
FILESIZEBITS                       64
POSIX_ALLOC_SIZE_MIN               4096
POSIX_REC_INCR_XFER_SIZE           
POSIX_REC_MAX_XFER_SIZE            
POSIX_REC_MIN_XFER_SIZE            4096
POSIX_REC_XFER_ALIGN               4096
SYMLINK_MAX                        
GNU_LIBC_VERSION                   glibc 2.23
GNU_LIBPTHREAD_VERSION             NPTL 2.23
POSIX2_SYMLINKS                    1
LEVEL1_ICACHE_SIZE                 32768
LEVEL1_ICACHE_ASSOC                8
LEVEL1_ICACHE_LINESIZE             64
LEVEL1_DCACHE_SIZE                 32768
LEVEL1_DCACHE_ASSOC                8
LEVEL1_DCACHE_LINESIZE             64
LEVEL2_CACHE_SIZE                  262144
LEVEL2_CACHE_ASSOC                 8
LEVEL2_CACHE_LINESIZE              64
LEVEL3_CACHE_SIZE                  8388608
LEVEL3_CACHE_ASSOC                 16
LEVEL3_CACHE_LINESIZE              64
LEVEL4_CACHE_SIZE                  0
LEVEL4_CACHE_ASSOC                 0
LEVEL4_CACHE_LINESIZE              0
IPV6                               200809
RAW_SOCKETS                        200809
_POSIX_IPV6                        200809
_POSIX_RAW_SOCKETS                 200809

Making sample compilation and installation optional for better use in project

When using hayai as a submodule, I think we should be able to disable the installation of the library and the compilation of the sample target. I've never contributed to anything on github but I can take care of it myself if you want.

Adding simple options would do.

Improvement of actual timings for test body without call overhead?

Hi,

I have been looking over the framework - but not actually tried any thing with it yet. Hence, if the below does not make sense, then you may close the issue.

From inspection it looks like the TestBody() timing includes overhead for both calls to the clock and the loop consideration.

If I look at the code in hayai_test.hpp:49:

startTime = Clock::Now();

// Run the test body for each iteration.
std::size_t iteration = iterations;
while (iteration--)
   TestBody();

// Get the ending time.
endTime = Clock::Now();

endTime - startTime would also represent the time taken to return from the functions Clock::Now() , processing of iteration-- and the function call to TestBody.

Would it not be better to:

Bench Clock::Now() to find the function call hit for it, then subtract that from the time
Use --iteration (prefix is generally faster), but also bench the time taken (probably very small, so might be too negligible) to do the subtraction and the branch consideration (i.e. the while condition)
Bench a "dummy" function call for TestBody() that can negate the call time over a virtual function call

In my mind, currently:
Test Time = Time in body + Time in framework

While Time in framework may stay relatively static, there is no way to know how much time is in the framework so exact timings of the TestBody() call is not known.

If you take consideration of the 1,2 and 3 points, the time can be removed from the final calculation to formulate a more accurate timing.

Would this be worthwhile?

Regards,
Richard

Always 0.000% percentage difference for fastest/slowest iteration

When running hayai, in the [ITERATIONS] section, the fastest and slowest always have a delta percentage of 0.000 %.

I am currently working on a PR that fixes this.

Exclude noise from measured time

Exclude noise from measured time.

I'd like to measure time of some function. This function take variable of user defined type and modify it. So for repeated calling of this function I need restore input parameters for each iteration. As result, the measured time of benchmark it time which was taken for execution of function (useful) and for restoring input parameter (noise).

Is there way to exclude this noise from report?

Conan package

Hello,
Do you know about Conan?
Conan is modern dependency manager for C++. And will be great if your library will be available via package manager for other developers.

Here you can find example, how you can create package for the library.

If you have any questions, just ask :-)

Release hayai-v1.0.2 displays version number as 1.0.1

constantine@******:~/hayai-1.0.2$ ./build/sample/sample --help
Usage: sample [OPTIONS]

  Runs the benchmarks for this project.

Benchmark selection options:
  -l, --list
    List the names of all benchmarks instead of running them.
  -f, --filter <pattern>
    Run only the tests whose name matches one of the positive patterns but
    none of the negative patterns. '?' matches any single character; '*'
    matches any substring; ':' separates two patterns.
Benchmark execution options:
  -s, --shuffle
    Randomize benchmark execution order.

Benchmark output options:
  -o, --output <format>[:<path>]
    Output results in a specific format. If no path is specified, the output
    will be presented on stdout. Can be specified multiple times to get output
    in different formats. The supported formats are:

    console
      Standard console output.
    json
      JSON.
    junit
      JUnit-compatible XML (very restrictive.)

    If multiple output formats are provided without a path, only the last
    provided format will be output to stdout.
  --c, --color (yes|no)
    Enable colored output when available. Default yes.

Miscellaneous options:
  -?, -h, --help
    Show this help information.

hayai version: 1.0.1
Clock implementation: gettimeofday

Steps to Reproduce:

wget https://github.com/nickbruun/hayai/archive/v1.0.2.tar.gz
cd hayai-1.0.2
cmake -H. -Bbuild
make -C build
./build/sample/sample --help

Expected hayai version: 1.0.2
Actual hayai version: 1.0.1

How to Stop/Fail a Benchmark if fixture SetUp() fails?

Hi,

It seems there's not a way to stop/fail a benchmark whenever it is not possible to continue during SetUp().

class TestFixture : public ::hayai::Fixture
{
 public:
    virtual void SetUp() {
        test = new Test();
        if (!test->canContinue()) {
            // SHOULD STOP HERE!
        }
    }
    virtual void TearDown() {
        delete(test);
    }
    Test *test;
};

BENCHMARK_F()'s connected to the fixture above still continues to execute.

Cheers!

Replace googletest with Catch

Would you be interested in getting rid of the googletest behemot and replace it with header-only https://github.com/philsquared/Catch?

How to set iterations manually by command arguments

I go through the source codes and can not find the usage about set iterations by command arguments.

Is there a plan for it?

Fixture setup with arguments

I was trying to create a fixture so that the size of the input of the test would be different among benchmarks, but with the same fixture.

Hawai doesn't do well with template fixtures, I expected as much. However I think it would be interesting to give SetUp, TearDown etc a void* parameter so that we can configure those steps per benchmark.

HTML outputter

Iteration statistics are an average of each run

As each iteration result is not kept in

hayai/src/hayai_test_result.hpp

Lines 292 to 301 in 9ca0833

 private: 

 std::vector<uint64_t> _runTimes; 

 std::size_t _iterations; 

 uint64_t _timeTotal; 

 uint64_t _timeRunMin; 

 uint64_t _timeRunMax; 

 double _timeStdDev; 

 double _timeMedian; 

 double _timeQuartile1; 

 double _timeQuartile3;

, the statistics that are outputted for ITERATIONS are only an average of each run. This renders the information rather useless as ITERATIONS are only a multiple of the RUNS result.

For example the following benchmark you would expect to see a large variation in each iteration.

#include "hayai.hpp"

int g_sleep = 0;

class SleepFixture : public ::hayai::Fixture {
public:
    virtual void SetUp() {
        g_sleep += 10000;
        std::cout << "SetUp " << g_sleep << std::endl;
        usleep(g_sleep);
    }

    virtual void TearDown() {
        g_sleep += 10000;
        std::cout << "TearDown " << g_sleep << std::endl;
        usleep(g_sleep);
    }

    void foo() {
        g_sleep += 10000;
        std::cout << "Sleep " << g_sleep << std::endl;
        usleep(g_sleep);
    }
};

BENCHMARK_F(SleepFixture, Sleeping, 1, 10) {
    foo();
}

Whereas the result shows.

[==========] Running 1 benchmark..
[ RUN      ] SleepFixture.Sleeping (1 run, 10 iterations per run)
SetUp 10000
Sleep 20000
Sleep 30000
Sleep 40000
Sleep 50000
Sleep 60000
Sleep 70000
Sleep 80000
Sleep 90000
Sleep 100000
Sleep 110000
TearDown 120000
[     DONE ] SleepFixture.Sleeping (676.609770 ms)
[   RUNS   ]        Average time: 676609.770 us (~nan us)
                    Fastest time: 676609.770 us (0.000 us / 0.000 %)
                    Slowest time: 676609.770 us (0.000 us / 0.000 %)
                     Median time: 0.000 us (1st quartile: 676609.770 us | 3rd quartile: 676609.770 us)
                                  
             Average performance: 1.47796 runs/s
                Best performance: 1.47796 runs/s (0.00000 runs/s / 0.00000 %)
               Worst performance: 1.47796 runs/s (0.00000 runs/s / 0.00000 %)
              Median performance: inf runs/s (1st quartile: 1.47796 | 3rd quartile: 1.47796)
                                  
[ITERATIONS]        Average time: 67660.977 us (~nan us)
                    Fastest time: 67660.977 us (0.000 us / 0.000 %)
                    Slowest time: 67660.977 us (0.000 us / 0.000 %)
                     Median time: 0.000 us (1st quartile: 67660.977 us | 3rd quartile: 67660.977 us)
                                  
             Average performance: 14.77957 iterations/s
                Best performance: 14.77957 iterations/s (0.00000 iterations/s / 0.00000 %)
               Worst performance: 14.77957 iterations/s (0.00000 iterations/s / 0.00000 %)
              Median performance: inf iterations/s (1st quartile: 14.77957 | 3rd quartile: 14.77957)
[==========] Ran 1 benchmark..

There is no variation in the iterations as this has been averaged away. An example of this can be seen in

hayai/src/hayai_test_result.hpp

Lines 243 to 246 in 9ca0833

 inline double IterationTimeMinimum() const 

 { 

 return _timeRunMin / double(_iterations); 

 }

Is this a design decision or would it be more ideal to keep the data from each iteration and use this information to collate the minimum, maximum, etc?

Debian package

Not able to install hayai from Ubuntu ppa (I run Ubuntu 20.04 focal fossa)

Trying with sudo apt-add-repository ppa:bruun/hayai
Yielding error :
Err:5 http://ppa.launchpad.net/bruun/hayai/ubuntu focal Release 404 Not Found [IP: 185.125.190.52 80] ... then further down:
E: The repository 'http://ppa.launchpad.net/bruun/hayai/ubuntu focal Release' does not have a Release file. N: Updating from such a repository can't be done securely, and is therefore disabled by default.

	private:
	std::vector<uint64_t> _runTimes;
	std::size_t _iterations;
	uint64_t _timeTotal;
	uint64_t _timeRunMin;
	uint64_t _timeRunMax;
	double _timeStdDev;
	double _timeMedian;
	double _timeQuartile1;
	double _timeQuartile3;

	inline double IterationTimeMinimum() const
	{
	return _timeRunMin / double(_iterations);
	}

nickbruun / hayai Goto Github PK

hayai's People

Contributors

Stargazers

Watchers

Forkers

hayai's Issues

Recommend Projects

Recommend Topics

Recommend Org