jmreidy / grunt-mocha-webdriver Goto Github PK

Been debugging a case where the grunt task never quit. It hang forever.

I found out that on line https://github.com/jmreidy/grunt-mocha-webdriver/blob/master/tasks/grunt-mocha-wd.js#L100 the kill() command did never fire the close event on the line https://github.com/jmreidy/grunt-mocha-webdriver/blob/master/tasks/grunt-mocha-wd.js#L96

This made the grunt task hang forever. No idea why the phantom child process is never closed. Looks like a bug ...

Could you please publish 1.0.0 to npm? Thanks for making this tool, looking forward to checking it out!

Not just PhantomJS, also with Chrome + Firefox?

If so, what should the configuration look like?

Task:

mochaWebdriver: {
      options: {
        reporter: 'spec'
      },
      sauce: {
        src: ['app/javascripts/test/index.html'],
        options: {
          username: 'Jester',
          key: '...',
          testName: 'test',
          concurrency: 1,
          browsers: [
            {browserName: 'internet explorer', platform: 'Windows 7', version: '9'},
            {browserName: 'internet explorer', platform: 'Windows 7', version: '8'},
            {browserName: 'chrome', platform: 'Windows 7', version: ''}
          ]
        }
      }
    },

Log:
vagrant@precise32:/vagrant$ grunt -verbose test:crossbrowser
...
Running "mochaWebdriver:sauce" (mochaWebdriver) task
Verifying property mochaWebdriver.sauce exists in config...OK
Files: app/javascripts/test/index.html
=> Connecting to Saucelabs ...
=> Sauce Labs trying to open tunnel
Fatal error: spawn ENOENT

This plugin has matured far beyond its original implementation; it can now run a mocha test suite against Phantom, Sauce Labs, and Selenium, with configuration options for each.

I'm wondering if this plugin should be refactored to remove its grunt dependency. That is to say, remove the core functionality into a moch-integration lib, that can be invoked by this task (via grunt), via a different task runner (gulp-mocha-integration?), or even be used standalone. What do people think? (Especially you, @saadtazi?)

So that I can use "prerun": "http://url.to/your/file.bat" to adjust some registry settings of Internet Explorers, i.E. to enable SSL/TLS before each test?

It looks like my e2e tests do not kill the PhantomJS process. How can we exit PhantomJS every time, no matter if the tests were successful or crashed in the middle?

Here an example:

Running webdriver tests against PhantomJS.
>> Error starting PhantomJS
Warning: [ERROR - 2014-04-27T04:32:14.022Z] GhostDriver - main.fail - {"message":"Could not start Ghost Driver","line":82,"sourceId":153982448,"sourceURL":":/ghostdriver/main.js","stack":"Error: Could not start Ghost Driver\n    at :/ghostdriver/main.js:82","stackArray":[{"sourceURL":":/ghostdriver/main.js","line":82}]}
 Use --force to continue.

I had to find the process and manually kill PhantomJS so that I can run it again.

On a side note, I never call browser.quit() in my tests because I were told that this grunt task is doing it automatically. Or am I wrong?

Thanks for the awesome plugin!

Sauce provides a configuration for specifying screen resolutions (see here, and search for "Screen Resolution"):

https://saucelabs.com/docs/additional-config

Just wondering if there was any way to pass this configuration via grunt-mocha-webdriver.

Thanks!

See admc/wd#232 (comment)

Would be great, so that I can access to the logs ... thx!

Currently, the log output gets mangled as tests are run concurrently.

Seems redundant to symlink the two. npm falls back to .gitignore by default if there is no .npmignore. I'd recommend moving that file to .gitignore and removing .npmignore.

• https://www.npmjs.org/doc/misc/npm-developers.html

Hi,
According to this FAQ item, you can send encrypted selenium commands (jsonWire calls) through the tunnel instead of http://ondemand.saucelabs.com/ (not encrypted).

I propose the following implementation, let me know if you're ok:

• Add a secureCommandsoption (boolean - default false). (not sure I like the option name... if you have a suggestion, feel free to share)
• Change

     else if (opts.hostname) {
        runTestsOnSelenium(fileGroup, opts, next);
      }

by

     else if (opts.hostname && !secureCommands) {
        runTestsOnSelenium(fileGroup, opts, next);
      }

to fallback to saucelabs when secureCommands is true

• Use hostname and port options instead of ondemand.saucelabs.com when secureCommands is true

Thank you

All my tests are broken because of ariya/phantomjs#10522

Either upgrade PhantomJS or use https://www.npmjs.org/package/function-bind internally please

I try to write functional tests in mocha-webdriver that simulate keyboard inputs in text fields and then test the value of the affected fields. I implemented the tests using the Promise API. Unfortunately, only one test runs successful, the other one dies saying:

  1) GMSC provides a login screen for the application accepts input in the passw
ord field:
     Error: Element is not currently interactable and may not be manipulated

Here is my code - what is my mistake?

function typeEachChar(el, text){
    var chars = text.split('');
    var typePromise = el.type(chars[0]);
    for (var i = 1; i < chars.length; i++){
        typePromise = typePromise.type(chars[i]);
    }
    return typePromise;
}

function acceptsInput (el, testText){
    return el.then(function(el){
         return typeEachChar(el, testText);
    }).then(function(){
        return el.getAttribute('value');
    }).then(function (val){
        return assert.equal(val, testText);
    })
}

describe('MyApp', function () {

    describe('provides a login screen for the application', function () {

        it('accepts input in the password field', function (done) {
            var browser = this.browser;
            var loginPage = browser.get('http://localhost/myapp/');
            acceptsInput(loginPage.elementById('Login'),'login').then(done, done);
        });

        it('accepts input in the password field', function (done) {
            var browser = this.browser;
            var loginPage = browser.get('http://localhost/myapp/');
            acceptsInput(loginPage.elementById('Password'),'password').then(done, done);
        });

    });
});

IMO Mocha tests must be respawned as a child process each time to ensure test are executed in independent environments, see pghalliday/grunt-mocha-test#64 (comment)

Any chance this could be introduced here as a new option?

attempting to install grunt-mocha-webdriver from behind the firewall at work causes failures as the git:// protocol is blocked. attempted to use the following:

git config --global url."https://github.com/".insteadOf [email protected]:
git config --global url."https://".insteadOf git://

however npm still tries to pull down the dependency over git://

changing the dependency to git+https:// seems like it would resolve the problem

https://theintern.github.io/leadfoot/ seems to be a much better driver than wd.js. What do people thinking of switching wd for leadfoot?

wd 0.2.2 has a new way of working with chai-as-promised, which is advertised in their examples as being the "preferred" way.

See this example: https://github.com/admc/wd/blob/master/examples/promise/mocha-specs.js#L19 . I'm not sure if this type of example is compatible with grunt-mocha-webdriver.

I tried to get this running in the current version (with wd manually upgraded to 0.2.2), but ran into errors, probably because I wasn't able to get a handle to the wd object to call chaiAsPromised.transferPromiseness = wd.transferPromiseness;

Similar like

        self.browser.on('status', function(info){
          console.log('\x1b[36m%s\x1b[0m', info);
        });

I want to read any console output the browser makes. No matter if console.log, console.warn, console.error etc.

How can we do this?

I'm trying to run the sauce grunt task, but gets this when it is trying to set up sauce tunnel:

File "pyclasspath/java_urllib2.py", line 41, in open

IOException: java.io.IOException: Server returned HTTP response code: 401 for URL: https://saucelabs.com/rest/v1/jquery-caret/tunnels?full=1

Any ideas?

I get this in my e2e tests without a stack trace. Difficult to find the bug.

Any chance we could tweak the grunt plugin to add additional debugging information when a fatal error occurs?

Could you release a patch version with the updates?

Thanks for merging them in and making this really useful grunt plugin!

I've noticed that if any asynchronous assertions error then the process exits and is not handled by Mocha.

For example:

beforeEach(function (done) {
  this.browser.get('http://www.google.com', done);
});

it('should fail', function (done) {
  this.browser.title(function (error, title) {
    assert.equal(title, "Not Google");
    done();
  });
});

It seems to be related to this plugin since I can't seem to recreate with grunt-mocha-test.

Whenever one of my PhantomJS based e2e tests fails, I can see that PhantomJS is still running with:

$ ps aux | grep phantom
1000     18230  6.6  2.2 2234488 136576 pts/3  Sl   16:24   0:09 /home/michael.heuberger/projects/binarykitchen/code/xxx/node_modules/phantomjs/lib/phantom/bin/phantomjs --webdriver 4444 --ignore-ssl-errors yes

This means, whenever I try to test again, I cannot run the tests without killing it first.

If I don't kill, then I see this error:

Running "mochaWebdriver:client" (mochaWebdriver) task
Running webdriver tests against PhantomJS.
>> Error starting PhantomJS
Warning: [ERROR - 2013-12-10T03:25:41.118Z] GhostDriver - Main - Could not start Ghost Driver => {
  "message": "Could not start Ghost Driver",
  "line": 82,
  "sourceId": 140310677564928,
  "sourceURL": ":/ghostdriver/main.js",
  "stack": "Error: Could not start Ghost Driver\n    at :/ghostdriver/main.js:82",
  "stackArray": [
    {
      "sourceURL": ":/ghostdriver/main.js",
      "line": 82
    }
  ]
}
 Use --force to continue.

Aborted due to warnings.

Can't we configure PhantomJS to kill whenever a test fails?

Wondering how I can pass this onto the webdriver for PhantomJS?

Sometimes it's extremely slow, takes ages to open a simple site: www.google.co.nz

Sauce Lab's support commented on this in a ticket of mine:

Whenever tests using Sauce Connect don't work against any URL, including a URL that is on the Internet, it could be a problem with Sauce Connect, and often restarting Sauce Connect resolves this issue. Are you still seeing this?

So, can we somehow set a timeout and restart Sauce Connect when URLs take too long to load?

Currently I duplicate these lines in every test file:

        self.browser.on('status', function(info){
          console.log('\x1b[36m%s\x1b[0m', info);
        });

        self.browser.on('command', function(meth, path, data){
          console.log(' > \x1b[33m%s\x1b[0m: %s', meth, path, data || '');
        });

This is not so nice. I would love to add that once at the beginning for all tests. In a init test file or in the grunt config if that's possible. Any ideas?

Hello there

Would it be possible to display Sauce Lab's remote selenium log locally whenever a test has failed?

It looks like they're attempting to kill each other's sauce tunnels. This error is manifested on Travis

Hello again

Not sure if this can be done, but according to http://support.saucelabs.com/entries/35517624-Test-did-not-see-a-new-command-for-90-seconds-Timing-out- driver.quit() should be always called to avoid time out error messages.

But when I abort test runs with CTRL-C, then this won't happend and the next run will be very weird, timeouts can happen by random. Not sure what to do here and how to call quit() in those cases?

Or should I call quit() at the top of my grunt script to ensure it's really exited before the init process?

For long running E2E tests with Saucelabs this would be very nice.

Currently, when you run a test that takes more than 8 minutes, all you see is just:

Running "mochaWebdriver:sauce" (mochaWebdriver) task
=> Connecting to Sauce Labs ...
>> Connected to Sauce Labs.

and nothing else. And after 8 minutes, lots of log lines appear and you begin to analyse. Not user-friendly. Hence I think a progress bar would be a nice-to-have-feature.

According to http://code.google.com/p/selenium/wiki/JsonWireProtocol#POST_/session/:sessionId/log I tried

this.browser.log('browser', cb);

// later in async
console.log(logs);

But I am getting this error in Grunt

Uncaught AssertionError: expected {"message":"Not JSON response","data":"Invalid Command Method - Request => {\"headers\":{\"Accept\":\"application/json\",\"Connection\":\"keep-alive\",\"Content-Length\":\"18\",\"Content-Type\":\"application/json; charset=UTF-8\",\"host\":\"127.0.0.1:4444\"},\"httpVersion\":\"1.1\",\"method\":\"POST\",\"post\":\"{\\\"type\\\":\\\"browser\\\"}\",\"url\":\"/log\",\"urlParsed\":{\"anchor\":\"\",\"query\":\"\",\"file\":\"log\",\"directory\":\"/\",\"path\":\"/log\",\"relative\":\"/log\",\"port\":\"\"... to equal undefined

All I want is to read the console.log outputs from PhantomJS ... any clues?

I'd like to read these for debugging purposes ...

Also posted this question there but am not sure where the right place is:
admc/wd#296

With the latest PhantomJS v1.9.10, setting the windows size does not work anymore. Here my snippet:

    wd.addPromiseMethod(
        'isLogged',
        function() {
            return this
                .setWindowSize(e2eConfig.tablet.width, e2eConfig.tablet.height)
                .waitForElementByCssSelector('nav .myAccount .button').text() // test fails here
                .then(function(text) {
                    expect(text).to.equal('My Account');
                })
                .elementsByCssSelector('nav menu li ul li')
                .then(function(elements) {
                    expect(elements).to.be.an('array');
                    expect(elements).to.have.length(3);
                });
        }
    );

The test fails because it is still stuck with its initial size (mobile). There is no resize effect. Why?

In my mocha options (grunt) I have a timeout of 3000 I want to reuse for the waitForElementByCss calls.

So that I can adjust that number at one place only for 30 test files.

And in the test files, it would be great if I can access that number somehow like that:

this.browser.waitForElementByCss('nav', this.options.timeout, cb);

Any chance the mocha options could be exposed?

label:improvement

Suggested Improvement (RFC?)

I stumbled upon this usage of mocha-as-promised in wd's examples.

It's currently impossible to use mocha-as-promised without modifying the mocha runner to get access to the mocha instance.

What I'm proposing is to either:

a. expose Mocha somehow
b. add a flag to instrument mocha with mocha-as-promised as part of the task config. Example

I'm available to do the work but wanted to check with you if one of the solutions is preferable.

Thanks

When doing this:

browser.uploadFile(VIDEO_PATH + 'hamster-andele.wmv').nodeify(done);

I get this error

     Error: connect ECONNREFUSED
    at errnoException (net.js:904:11)
    at Object.afterConnect [as oncomplete] (net.js:895:19

I am running local PhantomJS tests here. All I want is to upload that video file. It exists and the above path is correct.

Any ideas why it is not working?

I came across this project as I was re-working a custom grunt task for running functional mocha tests, but the one feature that was missing was the ability to run tests against a local instance of Selenium.

I was able to quickly throw together a grunt task for doing that, which ended up borrowing some code from this project and one of the other grunt mocha test runners.

Since the grunt config for the task I wrote and this project are nearly identical, I was thinking that it would be a good idea to add in a local Selenium option to this project, but wanted to check before I spent any time putting together a pull request.

Feedback appreciated!

This is interesting. When an ECONNREFUSED occurs, all other tests cases come with the same error and the grunt task never stops:

  1) Create video upload file to server:
     Error: connect ECONNREFUSED
    at errnoException (net.js:904:11)
    at Object.afterConnect [as oncomplete] (net.js:895:19


  2) Create video logout:
     Error: connect ECONNREFUSED

     at errnoException (net.js:904:11)
      at Object.afterConnect [as oncomplete] (net.js:895:19)

  3) Delete account (user) login:
     Error: connect ECONNREFUSED

     at errnoException (net.js:904:11)
      at Object.afterConnect [as oncomplete] (net.js:895:19)

  4) Delete account (user) delete:
     Error: connect ECONNREFUSED
    at errn
  Exception (net.js:904:11)
      at Object.afterConnect [as oncomplete] (net.js:895:19)

  5) Delete account (user) unable to login:
     Error: connect ECONNREFUSED
    at errno
  xception (net.js:904:11)
      at Object.afterConnect [as oncomplete] (net.js:895:19)

[WAITING FOREVER HERE, NOTHING HAPPENS, HANGING]

I just started with grunt-mocha-webdriver yesterday and saw the update for using the 2.0 syntax today, cool! I switched my basic tests over and they worked in PhantomJS. When I run against SauceLabs though, I now get errors like:

>> Connected to Saucelabs. Fatal error: Object [object Promise] has no method 'elementById'

Is this a sign that my stuff isn't written correctly or could there be a deeper issue? Thanks.

The idea is to be able to specify custom reporter when test are run in saucelabs or selenium mode.
For example, I would like to create a teamcity reporter, similar to this, but I will probably have to adapt it so it works with grunt-mocha-webdriver:

• pass the browser,
• if (suite.root) then output test suite started on browserName.

That team-city reporter doesn't have to live in grunt-mocha-webdriver. Please let me know your preference... I think outside is better...

Once the first one is done, then it would be easy for anyone to create other reporters (jenkins...).

I can give it a try but I would like to have your input first. any thoughts/limitations/concerns?

looks like out of date strings in the package.json.

The wd object is useful for a few reasons:

1. Ability to monkey patch custom functions onto the browser object
2. Ability to use constants such as wd.SPECIAL_KEYS.Tab when sending keystrokes

It would be great if the wd object could be set on the mocha context, so it could be accessed via this.wd just in the same way that we can access the browser object using this.browser.

If it could be exposed in such a way that it would be available to a "setup" file (used in the "requires" declaration as a prerequisite) that would be even better, since this is where I prefer to do my monkey-patching.

The docs say:

secureCommands

Type: Boolean (Default: false)

If true, it will use saucelabs, with default hostname set to 127.0.0.1 and port set to 4445
in order to send selenium commands through Sauce Connect tunnel (more info
here).

However even when this is set to false, Sauce Connect is used. Is there a way to connect to SauceLabs and run the tests there without Sauce Connect?

I tried it like https://github.com/admc/wd#http-configuration--base-url on the browser instance. Have you ever tried it too?

Just wondering ... I need that functionality for debugging

Better for everyone and keeps README.md slim. And change the order, newest versions on top. Thanks :)

Just wondering if I can use this plugin to execute specs in a specific order. When I set up the tunnel manually and manually execute my mocha tasks, it works. But through this plugin, it seems to execute out of order.

For instance (pseudocode):

it("should be able to log in")
it("should open the profile page")
it("should edit the user email")
it("should save the page")
it("should log out")

When I look at the sauce report page, it is showing that it tried to execute the 2nd "it" before the 1st.

Is this implementation compatible with Grunt 0.3.x versions?
I'm currently running 0.3.17

Currently I'm getting an error:

TypeError: Cannot read property 'length' of undefined

at this section of grunt-mocha-wd.js:

grunt.util.async.forEachSeries(this.files, function (fileGroup, next) {
if (opts.usePhantom) {
runTestsOnPhantom(fileGroup, opts, next);
}
}, this.async());