polyfractal / athletic Goto Github PK
View Code? Open in Web Editor NEWPHP Benchmarking Framework
PHP Benchmarking Framework
Hi!
I've been looking into benchmarking test suites to integrate into ProxyManager (see Ocramius/ProxyManager#62) and I think athletic is the best one I've found so far in terms of code organization.
There's a couple of things that I'd like to ask as clarification and/or new feature (if possible).
@group some-test
and @baseLine
, for example) and then group results and compare them Like jeremyFreeAgent/Clutch does? (note: Clutch is un-suited for this stuff because of the introduced xhprof overhead)I will try integrating athletic as it is right now, and eventually try to patch together a result printer, but I hope these features can eventually be put into discussion.
I generally LOVE how athletic keeps tests organized, but Clutch is generally better in terms of details/tests comparison. I'd actually suggest integrating Clutch itself and open an issue there to ask if xhprof could be a problem.
Today I refactored some of this package to support:
This required the refactor of these files:
See my commits here: https://github.com/oytuntez/athletic/commits/master.
I didn't issue a PR as couldn't know how to organize the changes into tags/branches.
In Ocramius/ProxyManager#114 I implemented some hacks to have data-providers support
Could be ported back to athletic
The composer.json says the license is MIT, is there a corresponding LICENSE for this repo?
The current implementation of Athletic\Discovery\Parser
does not allow discovered events to inherit from classes that do not contain AthleticEvent
in their name.
I'm not sure why this limitation is in place, but the parser looks quite simplistic, and honestly I don't have a quickfix for it right now.
The idea is to use a more sophisticated parser, but that would of course mean having a much heavier logic.
Is this intended or bogus behaviour?
Hi!
I'm trying to evaluate athletic and I'm having issues getting even a simple example to run. The "sample event" from the readme is obviously not intended to be used as-is, so I
I'm trying to run using "vendor/bin/athletic -p ./SleepingEvent.php", which fails with "ERROR: Class \SleepingEvent does not exist". I've tried various other combinations, but to no avail. The class name in the error message changes slightly, but that's all. I'm using PHP 5.6, in case that matters.
Some annotations that one could set to a "desired running time" would be nice. E.g.
"this" method should preferably run below "0.0002" at average
This way, the benchmarking could also lead to some form of tests. We get a quick feedback of methods not running in the preferred time window.
<?php
class TestEvent {
/**
* @iterations 1000
* @maxTime 0.05
* @avgTime 0.002
*/
}
This would translate as:
the method should run 1000 iterations, not once allowed above the maximum time limit, and the average time should be less than 0.002
The terminal should update its progress, letting us know what the status is. Really useful when you have lots of benchmarks, each taking quite a bit of time.
If you have a lot of iterations memory skyrockets when used. Maybe results could be temp saved in a file somewhere when doing the calculations?
Hi there,
Is there any way to have athletic display more information when an exception occurs. It only displays the exception message, but sometimes this isn't enough :(
For example here, I'm benchmarking a lib which throws me an exception (with a very poor message) but then I'm stuck for resolving it:
% php -n vendor/bin/athletic -p benchmarks -b vendor/autoload.php
ERROR: Benchmark\Fixture\Foo
Test grouping allows you to group tests inside of an Event and compare against an optional baseline value. This feature introduces two new annotations:
@group groupName
specifies the name of the group that this particular test belongs to. All tests with this group name will be compared against each other and displayed together in the results@baseline
specifies which test should be the baseline value which other tests are compared against. Each group may have only one baseline.An example Event setup:
/**
* @iterations 100
* @group small
* @baseline
*/
public function slowIndexingAlgoSmall()
{}
/**
* @iterations 100
* @group small
*/
public function fastIndexingAlgoSmall()
{}
/**
* @iterations 100
* @group large
* @baseline
*/
public function slowIndexingAlgoLarge()
{}
/**
* @iterations 100
* @group large
*/
public function fastIndexingAlgoLarge()
{}
Presently, the default formatter does not support grouping. To display grouping appropriately, you must use the new GroupedFormatter
. To do this, pass an additional -f
or --formatter
flag on the command line:
./vendor/bin/athletic -b <path_to_benchmark> -f GroupedFormatter
Results will then look something like this:
Elasticsearch\Benchmarks\IndexingEvent
small
Method Name Iterations Average Time Ops/s Relative
------------------------------ ---------- ------------ -------------- --------- ---------
slowIndexingAlgoSmall : [Baseline] [100 ] [0.0059354090691] [168.48038]
fastIndexingAlgoSmall : [100 ] [0.0013290691376] [752.40631] [22.39%]
large
Method Name Iterations Average Time Ops/s Relative
------------------------------ ---------- ------------ -------------- --------- ---------
slowIndexingAlgoLarge : [Baseline] [100 ] [0.0022656822205] [441.36816]
fastIndexingAlgoLarge : [100 ] [0.0032706809044] [305.74673] [144.36%]
I run http://stickler-ci.com which is a service aimed at improving code quality by simplifying code review by automating code style feedback in pull requests. As a fellow open source maintainer I found I was spending a significant amount of time giving feedback to contributors on how code should be formatted, and thought there had to be a better way.
Sticker-CI is my attempt at building a better way. By making code style errors as pull request comments and a build status it is easy for both maintainers and contributors to know when a pull request matches a project's style.
I wanted to know if you were interested in trying out stickler-ci. If you do, I can submit a pull request with the configuration file, but a maintainer will need to enable webhooks by logging into https://stickler-ci.com and enabling the webhook.
Add file output support; Should function similar to piping the output to a file.
I've just found out that setUp
is called only before the iterations start. Here's the pseudo-code of the currently implemented logic:
$this->setUp();
for($i = 0; $i < $iterations; $i += 1) {
$this->benchmarkMethod();
}
This is the correct logic in my opinion:
for($i = 0; $i < $iterations; $i += 1) {
$this->setUp();
$this->benchmarkMethod();
}
I'm not sure if this is really necessary in vanilla PHP, at least it won't hurt. HHVM uses a JIT and would benefit from a warm up phase.
Right now, the error handler registered by athletic stops execution on warnings/notices (which in my case don't depend on my codebase). This should probably be removed.
For a reference, see Ocramius/GeneratedHydrator#14 and related travis builds
--path is a required flag at the moment, but Athletic should default to the current working directory if none is provided by --path.
The AthleticEvent
should change its private methods to protected, or:
$currentMethod
, $currentIteration
and $totalIterations
to setUp
and tearDown
setUpMethod
, tearDownMethod
(with argument $currentMethod
)A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.