Generating tests fails for all if a single test fails about magento2-functional-testing-framework HOT 13 CLOSED

We discussed this today in grooming and we could not agree on how to implement it. We need your feedback.

Context
Someone else's tests that fail to generate, such as those in another module, should not stop my tests from generating. The idea is, if we have 1000 tests and 2 tests have mistakes then we will generate 998 tests and output a log message about the other 2 tests instead of failing the entire process like we do today.

But we can't decide between:

a) Add a flag, like generate:tests --ignore-failing
b) Make it the default behavior of generate:tests
c) generate:tests will return a nonzero exit code like today, but still generate every test that it could

Option A -- add a flag

• Pros
  • Does not require any changes to our internal CICD pipeline
• Cons
  • We want to avoid adding lots of flags to our commands

Option B -- make it the default

• Pros
  • More intuitive that it just works
• Cons
  • We have to change our internal CICD to fail the build if tests don't generate. This can be a lot of work.

Option C -- nonzero exit code, but still generate all else

• Pros
  • Maintains current functionality of nonzero exit code
  • But still generates everything else
  • Best of both worlds
• Cons
  • None that I can think of.

What do you think?

For option A put a 👍
For option B put a ❤️
For option C put a 🚀

from magento2-functional-testing-framework.

@tomreece it would be good to have an option for this (e.g. --skip-failed or --ignore-failing)

from magento2-functional-testing-framework.

There was an idea posted on twitter that I quite liked, and that I think would pair nicely with all of the above:

When a test cannot be generated, generate a stub for the test that fails - detailing the generation error.

from magento2-functional-testing-framework.

Internal ticket
https://jira.corp.magento.com/browse/MQE-1800

@magicsunday Thank you for your issue even though it has been almost a year now. Sorry about that. We are improving our interactions with Github.

I am interested in this idea. In the past we've had reasons to fail-fast, but we need to re-evaluate those reasons. I have created an internal ticket and we discuss further.

from magento2-functional-testing-framework.

@tomreece and @okolesnyk

Could you provide the final expectations according to "fail-fast"?
I can address the issue, but need to have confirmed what is the final version?

from magento2-functional-testing-framework.

The problem is if some tests aren't generated then they won't be run in our CICD environment. This would be bad.

I think we should keep MFTF the same, fail to generate if any test is bad.

from magento2-functional-testing-framework.

@lbajsarowicz @tomreece @vasylmalanka
I think we will end up generating all tests and put into logs a list of tests which had issues.
We also should introduce static check to be able to fail build when something is not generating.

Also I believe we should output error when we execute generate test for specific test. It will simplify development process.

from magento2-functional-testing-framework.

The internal ticket https://jira.corp.magento.com/browse/MQE-1800 is ready for development for next release which is targeted toward the end of August. @lbajsarowicz will you be able to work on this issue with this schedule? Please let us know. Thanks!

from magento2-functional-testing-framework.

AC

• Generate:tests generates every *Cest file that it can
• It does not stop when it encounters a bad test. Log testnames and return number of failed test generations in the console
  "X tests failed to generate, please see for details"
• If any tests failed to generate, return non-zero exit code. Build should still fail if any tests fail to generate.
• Unit Tests

from magento2-functional-testing-framework.

@lbajsarowicz Could you please let us know by end of this week if you will be able to work on this or not? Thanks!

from magento2-functional-testing-framework.

We are going to work on this ticket internally.

from magento2-functional-testing-framework.

I am really sorry, but I won't be able to maintain enough time to perform this change, @jilu1 as I'm still stuck with my main job tasks. I regret that the decisions weren't made before :-( (when I had time to do these changes).

from magento2-functional-testing-framework.

This feature has just been released in 3.2.0.

from magento2-functional-testing-framework.