Question: how do you feel about Selenium for integration tests? about react-server HOT 11 OPEN

PR for the first step of integrating streaming into React core.

✨ Awesome! ✨

/me furiously clicks emoji on that PR.

from react-server.

I'm not opposed to investigating a switchover, for sure. I love the ease of use of zombie, but I definitely don't feel confident that it's going to catch anything browser-specific.

re: cons -

A lot slower: Each command to the browser is run independently, over the Internet, as opposed to in-process JavaScript with Zombie. It's fairly easy to parallelize different testing different browsers in Travis, but parallelizing individual tests is harder.

I'd be interested to get an idea of how much slower they are. I know they're slower than running in zombie, but our tests aren't doing that much. Hopefully it wouldn't be awful?

More confusing to run tests locally: ... Selenium would require either a webdriver/selenium server local installation, or that the dev have Sauce Lab credentials and set up Sauce Connect.

How much more confusing? I haven't ever had to set up selenium tests locally, but internally we have Java-based selenium tests that are run just like JUnit tests; basically no additional configuration needed. I would hope that we could get something working like that in this case? I think not being able to easily run tests locally would probably be a deal-breaker to getting people to write new ones.

from react-server.

To make this discussion more concrete, as an experiment I converted the (very simple) HelloWorld test spec to Selenium/Sauce; you can see it here. A few thoughts:

• The tests look pretty good: I think the actual test code in the spec looks pretty good/readable, especially considering it all became much more async. Might be even better if we integrated jasmine-promises, which allows you to just return a Promise from a test rather than mucking around with done().
• Write once, test on many renders works: I really liked the strategy in the existing tests of writing a test once that can be run three times, once each on the server render, client render, and page-to-page transition. That strategy is still doable on Selenium/Sauce.
• Added complexity has a cost: As with all projects like this, there were integration oddities that took a while to figure out; I think most of the complexity is hidden now from the test writer, but it could definitely rear its head again.
• OSX tests are pokey, but mobile tests are SLOOOOOOOOOOOOOOOOOW: I've observed a huge perf difference depending on the OS/browser combo (rough times per test in the HelloWorldSpec):
  • Linux/Chrome: 3-6s
  • Win 10/Chrome: 5-9s
  • Linux/Firefox: 8-11s
  • Win 10/IE 11: 8-12s
  • OSX/Safari: 12-15s
  • OSX/Chrome: 12-15s
  • Win 10/Edge: 10-12s (with a couple test failures)
  • Android/Android Browser: 53-67s
  • iOS/Safari: 58-65s

Those mobile numbers are eye-popping; it would take hours to run even a reasonably modest test suite on mobile browsers. According to this thread, the startup times aren't unusual, and it's that the simulator frequently boots slowly and crashes. There's a potential that you could start up just one simulator for a whole Spec or Suite, although that obviously runs a risk of introducing test dependencies.

from react-server.

I'd be interested to get an idea of how much slower they are. I know they're slower than running in zombie, but our tests aren't doing that much.

Was working on that last post for like the last 2 hours; collided in midair with your question.

How much more confusing?

Haven't tried it yet, but I think it involves installing the Selenium Server jar, the Chrome WebDriver, and then starting up the jar with Java before running the tests. Or something like that.

from react-server.

Actually, I just found webdriver-manager, which claims to be an npm installable package that handles all you need to get a local Selenium Server going. It's forked out of protractor, which seems to be a well-used project.

from react-server.

I don't think this needs to be an either/or question. I'd like to see a setup where during local development we run the integration tests against a headless browser, and on the CI server we run against Selenium in real browsers. This allows local development to continue to be as fast or faster, while improving our safety on PRs.

from react-server.

@doug-wade Thanks for the response! I want to make sure I understand what you're saying, though. Do you mean:

1. that there should be a disjoint set of tests A & B, where A tests run on Selenium and only run in CI and B tests don't use Selenium at all and run both locally and in CI? or
2. that there should be a single set of tests that all use Selenium, but locally we run them against a headless browser and in CI we run them against a browser matrix?

And a follow-on question if you meant 2: do you have a suggestion for a headless browser? AFAIK Zombie doesn't support Selenium, which is why I was thinking of it as an either-or kind of thing. Thanks!

from react-server.

iiuc, selenium automates any browser that has a driver, headless or otherwise, which means we could run our selenium tests on phantomjs locally, and on real browsers for ci. I think the Phantom driver would work okay, though we could try to HtmlUnit driver or EventFiringWebDriver, which both seem headless.

At second reading, my first response was needlessly confusing. What I meant was: I think moving to Selenium is a good idea because we could still use headless browsers to do local testing, which should be relatively quick.

I also believe that we can make setup pretty simple, using webdriver-manager and phantom-prebuilt and selenium-webdriver, such that you wouldn't have to leave npm and gulp to do your testing/setup.

from react-server.

Thanks, @doug-wade, that clarifies things for me, and I think we're basically on the same page.

One question for you: how do you feel about testing with a non-headless Firefox? I've been experimenting with webdriver-manager, and I'm pretty impressed at how well it seems to work with Firefox and Chrome, and Travis supports running Firefox locally with xvfb. I even just opened an experimental PR that runs some of the existing tests against FF/xvfb on Travis, and the build passed (much to my surprise).

It also seems to work locally, although it may require Firefox to be on the dev box. Thoughts on that?

from react-server.

Looks like #102 is related to this.

from react-server.

Indeed. It was (afaict) totally working, but then it started to be a little flaky, so I didn't think it should merge yet. Then since last Wednesday, I've been sidetracked by my PR for the first step of integrating streaming into React core.

from react-server.