Running multiple testcafe runs over a grid mess things up about testcafe-browser-provider-selenium HOT 6 CLOSED

Hi, can you provide some more information as to how things start to mess up?
Have you also configured MaxInstances? can you post some of your grid config

from testcafe-browser-provider-selenium.

Hi @alexschwantes
Lets say that I have a grid with 100 nodes connected to it and I set the concurrency to 100 as well.
Now I'm starting another testcafe job, but because there aren't free nodes he doesn't wait in a queue and it also stops the first running job (it is interfering with it) so it hangs.

Am I clear enough?

Hub config:

Config for the hub :
browserTimeout : 18000
debug : false
jettyMaxThreads : -1
host : 100.101.248.165
port : 4444
role : hub
timeout : 18000
cleanUpCycle : 5000
capabilityMatcher : org.openqa.grid.internal.utils.DefaultCapabilityMatcher
newSessionWaitTimeout : -1
throwOnCapabilityNotPresent : true
registry : org.openqa.grid.internal.DefaultGridRegistry

Node config:

browserTimeout: 18000
debug: false
jettyMaxThreads: -1
host: IP
port: 5555
role: node
timeout: 18000
cleanUpCycle: 5000
maxSession: 3
capabilities: Capabilities {applicationName: , browserName: chrome, maxInstances: 1, platform: LINUX, platformName: LINUX, seleniumProtocol: WebDriver, server:CONFIG_UUID: 81362dd6-9c1a-4011-a300-8cd..., version: 87.0.4280.66}
downPollingLimit: 2
hub: http://selenium-selenium-hub:4444/grid/register
id: http://IP:5555
nodePolling: 5000
nodeStatusCheckTimeout: 5000
proxy: org.openqa.grid.selenium.proxy.DefaultRemoteProxy
register: true
registerCycle: 5000
remoteHost: http://IP:5555
unregisterIfStillDownAfter: 60000

from testcafe-browser-provider-selenium.

From the node config, you have maxSessions set to 3 which says that the node can run up to 3 sessions on any of the configured browsers at once, but you have maxInstances for chome set to only 1. This will limit the node to only ever be able to run 1 chrome instance at a time. To run 3 chrome browsers on that node you need to set maxInstances to be 3.

ie.

maxSession: 3
capabilities: Capabilities {applicationName: , browserName: chrome, maxInstances: 3, ....

The following part is perhaps a bit unexpected though. I would not expect the second job to interfere with the first job...

Now I'm starting another testcafe job, but because there aren't free nodes he doesn't wait in a queue and it also stops the first running job (it is interfering with it) so it hangs.

Anyway, give the modified node config a try and let me know how you go

What you want to see in the grid console is something like this, but with 3 chrome icons for each node. At the moment with your configuration it would only show 1 chrome icon per node.

from testcafe-browser-provider-selenium.

Hi @alexschwantes, I'm working with @doronsever and it seems that raising the maxInstances to 3 didn't solved the problem. To give more details:
Once we're running another testcafe job, we're seeing the following excpetion:
unhandledRejection (caught in mgmt sdk) --> {"code":"E1038","data":["selenium:chrome","WebDriverError: Error forwarding the new session Empty pool of VM for setup Capabilities {browserName: chrome, goog:chromeOptions: {args: [headless, --use-fake-ui-for-media-stream, --use-fake-device-for-media..., --allow-http-screen-capture, --disable-web-security, --allow-insecure-localhost, --window-size=1440,900]}}\n at Object.checkLegacyResponse (/<JENKINS_WORKSPACE>/node_modules/selenium-webdriver/lib/error.js:553:15)\n at parseHttpResponse (/<JENKINS_WORKSPACE>/node_modules/selenium-webdriver/lib/http.js:647:13)\n at Executor.execute (/<JENKINS_WORKSPACE>/node_modules/selenium-webdriver/lib/http.js:581:28)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (internal/process/task_queues.js:93:5)"]}

It seems that an exception is thrown if testcafe can't find a "free" browser to use. Because of that, the first job also fails because the second one sometimes manages to "steal" free browsers.
Besides, adding nodes, do we have the option to tell testcafe to wait (until X minutes timeout) for a free browser/node?

from testcafe-browser-provider-selenium.

Hi so I ran a little test to check how this would work. Fos simplicity I ran everything locally.
I spun up a selenium hub and node with 5 chrome instances on the same node, all without specifying any custom configuration.
I have two simple tests in the test suite that take about 13 senconds each to run.

1. If I run one testcafe -c 2 selenium:chrome tests/test.js, I can see in the selenium grid console web page that it is utilising two chrome browsers.
2. If I then run 2 of these tests simultaneously, I can see that it is utilising 4 chrome instances.
3. If I run 3 of these tests simultaneously, I can see that it uses all 5 available instances, and that it is queuing one remainder test as there are not enough free browsers.
4. I also tried this with two nodes (each with 5 chrome instances) and the results were the same, but with the tests running across both nodes.

I ran this test with selenium 3, what version are you using?

Perhaps you could try testing something like this with a smaller number of total nodes to test it out?

I also did some digging and found That exception "Error forwarding the new session Empty pool of VM" means that your Selenium Grid doesn't have any node available with a browser that has the desired capabilities. Also someone having the issue when using the selenium plugin for Jenkin.

Might need some more information on how exactly you have jenkins, selenium and testcafe configured. Are you using the selenium plugin for jenkins or running your own selenium hub/nodes. How/where are they run? Is testcafe run from a jenkins job that is using the same configuration each time its run?

Are you running all the jobs from Jenkins? Is the testcafe command executed from the Jenkins host? If so are you able to monitor the cpu/ram/network performance of Jenkins when running your tests to make sure you aren't hitting a bottleneck.

Are you able to attach a screenshot of your selenium console http://[selenium-hub-ip]:4444/grid/console

from testcafe-browser-provider-selenium.

Closing issue as it has been inactive for quite some time.

from testcafe-browser-provider-selenium.