Look into android latest (7.1) failure about browsers HOT 37 CLOSED

@lpinca you're not wrong. For the time being though, we have to deduplicate (in airtap) because airtap currently ignores the device property. So the input browsers are distinct but when we send it to Sauce Labs they're all the same.

from browsers.

Haven't looked too deep, but my guess is this is related to the following caveat:
"Android 7.0 only comes on the GoogleAPI flavor, so make sure that you have your desired capability set to deviceName: Android GoogleAPI Emulator." seen in the release notes

from browsers.

Another potential problem is that sauce-browsers / airtap only dedups browsers by OS, not device.

@vweevers Lets dedup this in airtap.

from browsers.

Also, I think we should do something about the translations I mentioned above.

Long-term, definitely. But for lack of unit tests, we should be careful to change too much at once.

from browsers.

Yup I'm using v0.1.0

I might take a look into this soonish if nobody else does.

from browsers.

Plus, we should be sending it to Sauce Labs like so:

caps['deviceName'] = 'Android GoogleAPI Emulator';
caps['browserName'] = 'Chrome';
caps['platformVersion'] = '7.1';
caps['platformName'] = 'Android';

Rather than what we do now (which Sauce Labs in fact translates into the above):

caps['deviceName'] = 'Android GoogleAPI Emulator';
caps['browserName'] = 'android';
caps['version'] = '7.1';
caps['platform'] = 'Linux';

from browsers.

OK, so it works if we set deviceName: Android GoogleAPI Emulator, but that specific string is never returned from the sauce labs rest api (https://saucelabs.com/rest/v1/info/browsers/webdriver) as a long_name value.

On the other hand, more specific names like 'Samsung Galaxy S8 GoogleAPI Emulator' are returned. We could choose one of the available names arbitrarily, or hardcode 'Android GoogleAPI Emulator' somewhere (not ideal). Thoughts?

from browsers.

In which way is this broken?

from browsers.

Any test that uses android 7 fails:

/Users/jhiesey/Projects/node/airtap/node_modules/wd/lib/webdriver.js:134
        err = new Error('The environment you requested was unavailable.' + jsonData.value ? ('\n' + jsonData.value.message) : '');
                                                                                    ^

TypeError: Cannot read property 'value' of undefined
    at /Users/jhiesey/Projects/node/airtap/node_modules/wd/lib/webdriver.js:134:85
    at Request._callback (/Users/jhiesey/Projects/node/airtap/node_modules/wd/lib/http-utils.js:88:7)
    at Request.self.callback (/Users/jhiesey/Projects/node/airtap/node_modules/request/request.js:186:22)
    at Request.emit (events.js:160:13)
    at Request.<anonymous> (/Users/jhiesey/Projects/node/airtap/node_modules/request/request.js:1081:10)
    at Request.emit (events.js:160:13)
    at IncomingMessage.<anonymous> (/Users/jhiesey/Projects/node/airtap/node_modules/request/request.js:1001:12)
    at Object.onceWrapper (events.js:255:19)
    at IncomingMessage.emit (events.js:165:20)
    at endReadableNT (_stream_readable.js:1101:12)

The exception thrown in webdriver itself when trying to generate the error message is a separate issue.

from browsers.

Some digging (and annoying multiple property translations) just to get some more information on what it looks like now.

When asking the sauce labs api we get the following meta data:

{
  "short_version": "7.1",
  "long_name": "Samsung Galaxy Tab A 10 GoogleAPI Emulator",
  "api_name": "android",
  "long_version": "7.1.",
  "device": "GalaxyTabA10GoogleAPI",
  "latest_stable_version": "",
  "automation_backend": "webdriver",
  "os": "Linux"
}

Current functionality is to pick .api_name, which gets translated to .name here:

https://github.com/airtap/airtap/blob/d0dd74a8326602d0bdc4d40f9d3f23baee824f49/lib/get-saucelabs-browsers.js#L42-L56

Which then gets translated to .browser here:

https://github.com/airtap/airtap/blob/master/lib/airtap.js#L55

Which then gets translated to .browserName here:

https://github.com/airtap/airtap/blob/master/lib/sauce-browser.js#L59

from browsers.

I'm thinking the ideal solution is to be able to use the first info struct we get from their api and just pass that back to sauce labs when requesting a test and get rid of all these translations in the first place. But that would require interaction with sauce labs and get their support to improve on this.

from browsers.

Some more information. When using wd.init(initOpts) we pass in a DesiredCapabilities which has browserName, version and platform, which doesn't give us much to go on. If we can't do browserName='android' and version='7.0' it seems to me we're out of luck.

Please correct me if I'm wrong.

Also, the status on the JsonWireProtocol for Selenium is OBSOLETE. Not sure what that exactly means but I'm guessing sort of oldish and soon-to-be-deprecated-or-killed? Maybe want to take a look into the W3C specification. All we're doing anyway is picking a browser and telling it to load a page and no fancy DOM stuff, so seems to me it should be fairly easy to switch to something more up to date.

from browsers.

Some more information. When using wd.init(initOpts) we pass in a DesiredCapabilities which has browserName, version and platform, which doesn't give us much to go on. If we can't do browserName='android' and version='7.0' it seems to me we're out of luck.

If we can't get that ^ to work it seems to me that the problem lies at Sauce Labs.

from browsers.

Oh apparently JsonWireProtocol is the way to go. I think obsolete above just means that the api docs are old. Not really any use of switching from wd to webdriverio then

from browsers.

admc/wd#522 was submitted to fix the jsonData error.

from browsers.

I just noticed the jsonData error starting to happen with iPhone latest as well :-(

browsers:
  - name: iphone
    version: latest

from browsers.

If I manually update wd to greater than 1.6.1 to fix the jsonData bug, then I get this error which is still preventing iPhone 11.3 tests from running:

airtap browser error ([email protected]: [init({"name":"arch","browserName":"iphone","version":"11.3","platform":"Mac 10.13"})] The environment you requested was unavailable.: Appium 1.8.0 does not support Mac 10.13. Please check our platform configurator (https://saucelabs.com/docs/platforms)), restarting +165ms

What does this mean?

Appium 1.8.0 does not support Mac 10.13. Please check our platform configurator (https://saucelabs.com/docs/platforms)

Any ideas?

from browsers.

I had issues with the same browser yesterday: airtap/airtap#152 (comment). But after a few retries it did succeed.

from browsers.

I just merged airtap/airtap#165 which upgrades wd so at least the jsonData bug is fixed on master.

This still doesn't fix Android 7. Still getting:

airtap browser error ([email protected]: [init({"name":"arch","browserName":"android","version":"7.1","platform":"Linux"})] The environment you requested was unavailable.: Unsupported OS/browser/version/device combo: OS: 'unspecified', Browser: 'android', Version: '7.1.', Device: 'Android Emulator'), restarting +186ms

from browsers.

@vweevers Retrying does not help for the iPhone error I'm getting. It seems similar to the Android 7 issue (i.e. it's a browser configuration issue):

Appium 1.8.0 does not support Mac 10.13. Please check our platform configurator (https://saucelabs.com/docs/platforms)

from browsers.

Seeing the same thing now on memdown. To verify that appium is the problem, maybe we can try SAUCE_APPIUM_VERSION=X airtap test.js? Where X is an older version (I don't know which version though).

See https://github.com/airtap/airtap/blob/67246f20b52fcf465331b0adfbde18edc06bde0e/lib/sauce-browser.js#L59-L61

from browsers.

Tried SAUCE_APPIUM_VERSION=1.7.2. Same error:

[init({"build":"432","name":"memdown","browserName":"iphone","version":"11.3","platform":"Mac 
10.13","appium-version":"1.7.2","tunnel-identifier":"432.1"})] The environment you requested was 
unavailable.: Appium 1.7.2 does not support Mac 10.13. Please check our platform configurator 
(https://saucelabs.com/docs/platforms)

from browsers.

Another potential problem is that sauce-browsers / airtap only dedups browsers by OS, not device. The Sauce Labs API returns multiple iphones with the same version and os:

 {
  "short_version": "11.3",
  "long_name": "iPhone 5s Simulator",
  "api_name": "iphone",
  "long_version": "11.3.",
  "device": "iPhone 5s",
  "latest_stable_version": "",
  "automation_backend": "webdriver",
  "os": "Mac 10.13"
}, {
  "short_version": "11.3",
  "long_name": "iPhone X Simulator",
  "api_name": "iphone",
  "long_version": "11.3.",
  "device": "iPhone X",
  "latest_stable_version": "",
  "automation_backend": "webdriver",
  "os": "Mac 10.13"
}

This can cause airtap to queue the same browser more than once:

airtap:sauce:iphone:11.3 browser conf: {"name":"memdown","build":"432","browser":"iphone","version":"11.3","platform":"Mac 10.13"}
airtap:sauce:iphone:11.3 browser conf: {"name":"memdown","build":"432","browser":"iphone","version":"11.3","platform":"Mac 10.13"}

from browsers.

Also, I think we should do something about the translations I mentioned above.

from browsers.

@feross in the mean time you can downgrade to iphone 11.2:

  - name: iphone
    version: 11.2
    platform: Mac 10.12

from browsers.

(beware though, because of the dedup issue, this will start 12 identical jobs!)

from browsers.

in the mean time you can downgrade to iphone 11.2

Scratch that. iphone 11.2 and ipad are now also failing.

from browsers.

Another potential problem is that sauce-browsers / airtap only dedups browsers by OS, not device.

I think this is by design, if you don't specify platform, browsers with the same version are deduplicated. On the other hand if you specify platform you want all browsers for that platform to be tested. For example

- name: iphone
    version: 11.2
    platform: Mac 10.12

asks to test all browsers whose name matches iphone, have version 11.2 and platform Mac 10.12, not just one.

from browsers.

@vweevers It's basically this key that should be unique, right?

https://github.com/airtap/airtap/blob/c760dac878943dbf66745097edf254ac67eda49c/bin/airtap.js#L150-L151

Damnit! Sometimes I can't get permalinks to work.

from browsers.

No, nevermind.

from browsers.

Well, actually, yes 😆

These property translations are painful to me. In other words we should dedupe on .os and .api_name properties in the original data we get from sauce-browsers.

They translate into:

• .os -> .platform
• .api_name -> .browser

But isn't the the final .version property important as well? So maybe it's the data to zuul.browser() that should be unique:

https://github.com/airtap/airtap/blob/c760dac878943dbf66745097edf254ac67eda49c/bin/airtap.js#L153-L157

from browsers.

Sometimes I can't get permalinks to work.

Those don't work cross-repo, sadly.

Let's continue discussion at airtap/airtap#168?

from browsers.

Those don't work cross-repo, sadly.

Makes sense now!

from browsers.

I'm still getting something similar:

Error: [email protected]: [init({"build":"280","name":"[secure]","browserName":"android","version":"7.0","platform":"Linux","tunnel-identifier":"280.1"})] The environment you requested was unavailable.: Misconfigured -- Unsupported OS/browser/version/device combo: OS: 'unspecified', Browser: 'android', Version: '7.0.', Device: 'Android Emulator'
    at /home/travis/build/jhiesey/[secure]/node_modules/wd/lib/webdriver.js:140:15
    at Request._callback (/home/travis/build/jhiesey/[secure]/node_modules/wd/lib/http-utils.js:89:7)
    at Request.self.callback (/home/travis/build/jhiesey/[secure]/node_modules/request/request.js:186:22)
    at Request.emit (events.js:182:13)
    at Request.<anonymous> (/home/travis/build/jhiesey/[secure]/node_modules/request/request.js:1163:10)
    at Request.emit (events.js:182:13)
    at IncomingMessage.<anonymous> (/home/travis/build/jhiesey/[secure]/node_modules/request/request.js:1085:12)
    at Object.onceWrapper (events.js:273:13)
    at IncomingMessage.emit (events.js:187:15)
    at endReadableNT (_stream_readable.js:1092:12)

Is this the same issue?

from browsers.

@jhiesey if you're on latest airtap, then yes. It seems we haven't gotten to the root cause yet. We've updated wd (airtap/airtap#165) and fixed duplicates (airtap/airtap#181), but android 7 is something else

from browsers.

Finally figured this out for real!

For Android and iOS, we and Sauce Labs use the appium automation backend. However, sauce-browsers fetches /rest/v1/info/platforms/webdriver rather than /rest/v1/info/platforms/all. The latter also includes appium-only browsers, including the one we need for android 7+: "Android GoogleAPI Emulator" (which is hardcoded in airtap atm).

from browsers.

Wrote https://github.com/airtap/sauce-browsers to handle this (will replace sauce-browsers).

from browsers.