snim2 / benchmarkr Goto Github PK

Any benchmarks I invoke runs 50 times, which is a maximal number of runs.

It seems that the condition on
https://github.com/snim2/benchmarkr/blob/master/benchmarkr/benchmarkr.py#L156
cannot be True.

The reason is that interval_percentage is in percentages, but the confidence interval is in [0-1].

I do not understand the computation of interval_percentage either.

Hi there!

I have read the oopsla paper of javastats and I would like to know whether the project is still working.. And, if so, if is there a chance to get stats from my own class, rather than dacapo benchmark. I have tried by myself, but with no luck at all.

NikemPro:benchmarkr Nikem$ ./javastats -c javastats.conf
{'suites': None, 'benchmarks': None, 'inputs': None, 'steady': False, 'startup': True, 'simulate': None, 'performance': None, 'config': 'javastats.conf', 'vms': None}
[]
DEBUG: suites -> ['specjvm98', 'dacapo']
DEBUG: vms -> ['sun']
DEBUG: benchmarks -> {'dacapo': ['avrora', 'batik', 'eclipse', 'fop', 'h2', 'jython', 'luindex', 'lusearch', 'pmd', 'sunflow', 'tomcat', 'xalan'], 'specjvm98': ['_201_compress', '_202_jess', '_209_db', '_213_javac', '_222_mpegaudio', '_227_mtrt', '_228_jack']}
DEBUG: inputs -> {'dacapo': ['small', 'default', 'large'], 'specjvm98': ['s1', 's10', 's100']}
DEBUG: tracefile -> /tmp/javastats_trace
DEBUG: command = cd /Users/Nikem/Documents/workspace/dacapo;ulimit -t 300;/usr/bin/time -p /usr/bin/java -jar dacapo.jar -n 1 -s small avrora> /tmp/javastats_trace 2>&1;
DEBUG: value found = 2.59
DEBUG: len(values): 1, min runs: 10, max runs: 50
DEBUG: value found = 2.58
DEBUG: len(values): 2, min runs: 10, max runs: 50
DEBUG: value found = 2.62
DEBUG: len(values): 3, min runs: 10, max runs: 50
DEBUG: value found = 2.55
DEBUG: len(values): 4, min runs: 10, max runs: 50
DEBUG: value found = 2.91
DEBUG: len(values): 5, min runs: 10, max runs: 50
DEBUG: value found = 2.75
DEBUG: len(values): 6, min runs: 10, max runs: 50
DEBUG: value found = 2.73
DEBUG: len(values): 7, min runs: 10, max runs: 50
DEBUG: value found = 2.87
DEBUG: len(values): 8, min runs: 10, max runs: 50
DEBUG: value found = 2.74
DEBUG: len(values): 9, min runs: 10, max runs: 50
DEBUG: value found = 2.82
DEBUG: len(values): 10, min runs: 10, max runs: 50
Traceback (most recent call last):
File "./javastats", line 379, in
sys.exit(main())
File "./javastats", line 368, in main
execute_run(vm, benchmark, performance, trace, stats)
File "./javastats", line 329, in execute_run
termination_condition = confidence_reached(stats_information, performance_data)
File "./javastats", line 278, in confidence_reached
(interval, mean, sdev, interval_percentage) = confidence(values, stats_information['confidence level'])
File "./javastats", line 247, in confidence
mean = stats.mean(samples)
AttributeError: 'module' object has no attribute 'mean'