Run framework tests only if there are fresh changes in the repository about frameworkbenchmarks HOT 11 CLOSED

The same suggestion has already been discussed in issue #5925 (and other places) before. Another reason to keep things as they are now is that it makes it possible to obtain statistics, which is an important aspect of any performance work. Also, in addition to kernel updates, all tests should be rerun whenever there is a change in the Docker version.

from frameworkbenchmarks.

This is very difficult, of course that together we need to find better ways to be faster and useful.

Why is difficult ?
For example, the many security mitigations in the kernels.
The frameworks didn't change, but results change completely.

Or you don't change the framework, but change the server, and we are again with the same problem.

from frameworkbenchmarks.

... many security mitigations in the kernels ... frameworks didn't change, but results change completely ... don't change the framework, but change the server

In this case, it should be possible to run all tests regardless of the freshness of the data in the repository.

This is very difficult ...

Why is that difficult?

from frameworkbenchmarks.

I'll show only all the frameworks, working with Nginx, it's the same with others.

No framework using Nginx, have any change.
And it isn't a problem with Nginx, was with the security mitigations in the kernel, for the CPUs.

from frameworkbenchmarks.

Not only that, a lot of frameworks use Nginx as a Proxy, in real applications.

from frameworkbenchmarks.

It's for that than I say that is NOT easy.
And we'll try to be better, but is not too easy.

from frameworkbenchmarks.

A lot of people, see the bench like a competition.

Please use it like a tool, to fix the problem in the frameworks.

from frameworkbenchmarks.

And exist bigger problems, like cache the response in fortunes.
And ....

from frameworkbenchmarks.

@volyrique

are now is that it makes it possible to obtain statistics, which is an important aspect of any performance work.

Imagine that framework X has not been changed for a whole year. For such frameworks, you can make a forced launch every 45 days, regardless of the freshness of the data in the repository. That is, in a year there will be 8 launches of framework tests even without creating a pull request in the repository.

Also, in addition to kernel updates, all tests should be rerun whenever there is a change in the Docker version.

It's OK. Within 45 days, a test with a new version of docker will be launched.

from frameworkbenchmarks.

Thanks for the suggestion, but this isn't something we're going to spend any time trying to accomplish right now. The fact that we have continuous benchmarking running is a huge improvement. Instead of this, I'd really like to cut down on the number of superfluous permutations. Also, I'm ok with removing test implementations that haven't been touched in years and are out of date; assuming the people involved are pinged that it's happening and given the chance to refresh the implementation.

from frameworkbenchmarks.

but this isn't something we're going to spend any time trying to accomplish right now.

Someday in TFB there will be 1...2 new tests and you will have to think about running tests that have no changes more rarely.

from frameworkbenchmarks.