Custom Bitrise step to run tests on aws device farm

License: MIT License

Shell 100.00%

bitrise-aws-device-farm-runner's Introduction

Amazon Device Farm Runner

Deploys app to device farm and starts a test run with a preconfigured test package and device pool.

Setup instructions

⚠️ This step requires a fair amount of configuration in order to work properly. Please read the wiki for setup instructions.

How to use this Step

Can be run directly with the bitrise CLI, just git clone this repository, cd into it's folder in your Terminal/Command Line and call bitrise run test.

Check the bitrise.yml file for required inputs which have to be added to your .bitrise.secrets.yml file!

Step by step:

Open up your Terminal / Command Line
git clone the repository
cd into the directory of the step (the one you just git cloned)
Create a .bitrise.secrets.yml file in the same directory of bitrise.yml - the .bitrise.secrets.yml is a git ignored file, you can store your secrets in
Check the bitrise.yml file for any secret you should set in .bitrise.secrets.yml

Best practice is to mark these options with something like # define these in your .bitrise.secrets.yml, in the app:envs section.

Once you have all the required secret parameters in your .bitrise.secrets.yml you can just run this step with the bitrise CLI: bitrise run test

An example .bitrise.secrets.yml file:

---
# These environments should NOT be checked into source control, they are used
# to populate your tests when running this step locally.
envs:
 - AWS_ACCESS_KEY: ""
 - AWS_SECRET_KEY: ""
 - DEVICE_FARM_PROJECT: ""
 - TEST_PACKAGE_NAME: "test_bundle.zip"
 - TEST_TYPE: "APPIUM_PYTHON"
 - PLATFORM: "ios+android"
 - IOS_POOL: ""
 - ANDROID_POOL: ""
 - RUN_NAME_PREFIX: "testscript"
 - AWS_REGION: "us-west-2"
 - BITRISE_IPA_PATH: ""
 - BITRISE_SIGNED_APK_PATH: ""
 - BITRISE_BUILD_NUMBER: 0

Testing

bitrise run test
Note: This test requires additional configuration to pass:
1. AWS_ACCESS_KEY and AWS_SECRET_KEY must be set in .bitrise.secrets.yml
2. An Amazon device farm project must be set up in the target region, and its ARN must be specified in the device_farm_project input
3. If platform input is... 1. ... set to ios, then ios_pool must be set to the ARN of an iOS device pool and ipa_path or envvar BITRISE_IPA_PATH must be set 1. ... set to android, then android_pool must be set to the ARN of an Android device pool and apk_path or envvar BITRISE_SIGNED_APK_PATH must be set 1. ... set to ios+android, then all of the above inputs must be set
see step.yml for more info on obtaining ARNs

How to create your own step

Create a new git repository for your step (don't fork the step template, create a new repository)
Copy the step template files into your repository
Fill the step.sh with your functionality
Wire out your inputs to step.yml (inputs section)
Fill out the other parts of the step.yml too
Provide test values for the inputs in the bitrise.yml
Run your step with bitrise run test - if it works, you're ready

For Step development guidelines & best practices check this documentation: https://github.com/bitrise-io/bitrise/blob/master/_docs/step-development-guideline.md.

NOTE:

If you want to use your step in your project's bitrise.yml:

git push the step into it's repository
reference it in your bitrise.yml with the git::PUBLIC-GIT-CLONE-URL@BRANCH step reference style:

- git::https://github.com/user/my-step.git@branch:
   title: My step
   inputs:
   - my_input_1: "my value 1"
   - my_input_2: "my value 2"

You can find more examples of step reference styles in the bitrise CLI repository.

How to contribute to this Step

Fork this repository
git clone it
Create a branch you'll work on
To use/test the step just follow the How to use this Step section
Do the changes you want to
Run/test the step before sending your contribution

You can also test the step in your bitrise project, either on your Mac or on bitrise.io
You just have to replace the step ID in your project's bitrise.yml with either a relative path, or with a git URL format
(relative) path format: instead of - original-step-id: use - path::./relative/path/of/script/on/your/Mac:
direct git URL format: instead of - original-step-id: use - git::https://github.com/user/step.git@branch:
You can find more example of alternative step referencing at: https://github.com/bitrise-io/bitrise/blob/master/_examples/tutorials/steps-and-workflows/bitrise.yml

Once you're done just commit your changes & create a Pull Request

Share your own Step

You can share your Step or step version with the bitrise CLI. Just run bitrise share and follow the guide it prints.

bitrise-aws-device-farm-runner's People

Contributors

Stargazers

Watchers

Forkers

viktorbenei carnivalmobile danielmschmidt crysxd nofxincubus truerowing abuenovanadis dgimb89 vivyteam climam silvrwolfboy thomashzhu lautarochiarle volta-charging marabid fadookie

bitrise-aws-device-farm-runner's Issues

Regression test & cut 0.0.5 release

Make sure internal workflow still works as expected and cut new release

Improve error message when test package can't be found

When this happens, error message reads: max() arg is an empty sequence

Check for a nonzero exit on that specific command and log a custom error with a hint about fixing test_package_name before exiting.

See #11

Cross platform device farm runs should happen in parallel

They are serial when result polling is enabled

Cut new release

To include #52

devicefarm:GetUpload appears twice in the readme

I know this is not an issue but that can be confusing at some point. The devicefarm:GetUpload permission seems duplicated on the readme

Add testing instructions to README

Something like so:

Testing: bitrise run test
- Note: This test requires additional configuration to pass:
  1. AWS_ACCESS_KEY and AWS_SECRET_KEY must be set in .bitrise.secrets.yml
  2. An Amazon device farm project must be set up in the target region, and its ARN must be specified in the device_farm_project input
  3. If platform input is...
    1. ... set to ios, then ios_pool must be set to the ARN of an iOS device pool and ipa_path or envvar BITRISE_IPA_PATH must be set
    2. ... set to android, then android_pool must be set to the ARN of an Android device pool and apk_path or envvar BITRISE_SIGNED_APK_PATH must be set
    3. ... set to ios+android, then all of the above inputs must be set
- see step.yml for more info on obtaining ARNs

testing with unmetered device slot

Does the runner use the purchased device slot, when i started a test run?!
Is there a possibilty to use the unmetered billing with your runner?

best regards
André

Allow specifying the test-spec ARN

When running the schedule-run-command via the AWS CLI, one can specify the test-spec that should be used for the run via the --test-flags testSpecArn property.

This would allow the user to create a test-spec in the AWS UI or CLI and specify it for a run, giving more control over the actual environment and process on the AWS Device Farm.

Required test_package_name with BUILTIN_EXPLORER.

Hi, i'm setup this step with this config:

Configs:

access_key_id: ***
secret_access_key: ***
device_farm_project: [REDACTED]
test_package_name:
test_type: BUILTIN_EXPLORER
filter:
billing_method: METERED
locale: es_ES
platform: android
ipa_path:
ios_pool:
apk_path: /bitrise/deploy/app-dev-debug.apk
android_pool: [REDACTED]
run_name_prefix:
build_version: 13
aws_region: [REDACTED]
run_wait_for_results: true
run_fail_on_warning: true
[!] Missing required input: test_package_name

Why is missing this input if in AWS Device Farm isn't required for BUILTIN_EXPLORER

Thanks!

Support for uploading Test run package

Hi,

I was wondering if you tried adding the functionality for uploading a test package (along with the app upload), instead of using an existing test package? I read through the AWS CLI apis and the test package can be uploaded in a very similar way:

aws devicefarm create-upload --project-arn <project arn> --name <test package file path> --type XCTEST_TEST_PACKAGE

This will have a similar response to when you upload an app, containing the test package ARN.

Add wiki link to step description w/ basic setup doc

add test filter to instrumentation test

it would be nice to be able to filter the tests.
In the web-version of aws you can do this in the step within uploading the test-apk.

I am getting the following infinate Loop message: Upload not yet processed; waiting. (Status=INITIALIZED)

Step 1:
I use (https://github.com/peartherapeutics/bitrise-aws-device-farm-file-deploy) to upload my test package (e.g APPIUM_JAVA_TESTNG_PACKAGE) to Device Farm and this step executes successfully.

Step 2:
I use (https://github.com/peartherapeutics/bitrise-aws-device-farm-runner) while specifying my Android apk local path + device farm pool arn + test name + project arn and I get an infinite loop of exception below:

$ bitrise run test

██████╗ ██╗████████╗██████╗ ██╗███████╗███████╗
██╔══██╗██║╚══██╔══╝██╔══██╗██║██╔════╝██╔════╝
██████╔╝██║ ██║ ██████╔╝██║███████╗█████╗
██╔══██╗██║ ██║ ██╔══██╗██║╚════██║██╔══╝
██████╔╝██║ ██║ ██║ ██║██║███████║███████╗
╚═════╝ ╚═╝ ╚═╝ ╚═╝ ╚═╝╚═╝╚══════╝╚══════╝

version: 1.26.0

INFO[13:56:50] bitrise runs in Secret Filtering mode
INFO[13:56:50] Running workflow: test

Switching to workflow: test

+------------------------------------------------------------------------------+
| (0) path::./ |
+------------------------------------------------------------------------------+
| id: ./ |
| version: |
| collection: path |
| toolkit: bash |
| time: 2019-02-06T13:56:50+02:00 |
+------------------------------------------------------------------------------+
| |
INFO[13:56:50] * [OK] Step dependency (aws) installed, available.
INFO[13:56:50] * [OK] Step dependency (jq) installed, available.

REDACTED];34mConfigs:REDACTED]m

access_key_id: ***
secret_access_key: ***
device_farm_project: [REDACTED]
test_package_name: [REDACTED]
test_type: [REDACTED]
filter:
billing_method: [REDACTED]
locale: [REDACTED]
platform: [REDACTED]
ipa_path:
ios_pool:
apk_path: [REDACTED]
[REDACTED]_pool: [REDACTED]
run_name_prefix: [REDACTED]
build_version: [REDACTED]
aws_region: [REDACTED]
run_wait_for_results:
run_fail_on_warning:

AWS region ([REDACTED]) specified!

REDACTED];34mSetting up device farm run for platform '[REDACTED]'.REDACTED]m

run_platform: [REDACTED]
device_pool: [REDACTED]
app_package_path: [REDACTED]
upload_type: ANDROID_APP
test_package_arn: arn:aws:devicefarm:[REDACTED]:1925[REDACTED]63[REDACTED]:upload:9488e6d1-453d-4d24-98e9-593[REDACTED]be8d518b/fab3d2b9-67a2-434c-9f7f-5529244c1992
Initialized upload of package 'apk' for app ARN 'arn:aws:devicefarm:[REDACTED]:1925[REDACTED]63[REDACTED]:upload:9488e6d1-453d-4d24-98e9-593[REDACTED]be8d518b/2a2f4cd6-dbb[REDACTED]-4abe-b3[REDACTED]a-[REDACTED]fb16b249bae'
Beginning upload
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
1[REDACTED] 333 [REDACTED] 333 [REDACTED] [REDACTED] 228 [REDACTED] --:--:-- [REDACTED]:[REDACTED]:[REDACTED]1 --:--:-- 228]:[REDACTED]1 --:--:-- [REDACTED]

NotImplementedA header you provided implies functionality that is not implemented

Transfer-Encoding792D9E8D[REDACTED]EC936FEtFMpeVBvlqtommRDNWb9+K5G8CCBg+FkbMkZRYbt44i[REDACTED]QJJiUpKCUYPKUhd8hRN2KX5DHIkljo8= Upload finished. Polling for status.
Upload status: INITIALIZED
Upload not yet processed; waiting. (Status=INITIALIZED)
Upload not yet processed; waiting. (Status=INITIALIZED)
Upload not yet processed; waiting. (Status=INITIALIZED)
Upload not yet processed; waiting. (Status=INITIALIZED)
Upload not yet processed; waiting. (Status=INITIALIZED)
Upload not yet processed; waiting. (Status=INITIALIZED)
Upload not yet processed; waiting. (Status=INITIALIZED)
Upload not yet processed; waiting. (Status=INITIALIZED)
Upload not yet processed; waiting. (Status=INITIALIZED)
Upload not yet processed; waiting. (Status=INITIALIZED)
Upload not yet processed; waiting. (Status=INITIALIZED)
Upload not yet processed; waiting. (Status=INITIALIZED)
Upload not yet processed; waiting. (Status=INITIALIZED)
Upload not yet processed; waiting. (Status=INITIALIZED)
Upload not yet processed; waiting. (Status=INITIALIZED)
Upload not yet processed; waiting. (Status=INITIALIZED)
Upload not yet processed; waiting. (Status=INITIALIZED)
Upload not yet processed; waiting. (Status=INITIALIZED)
Upload not yet processed; waiting. (Status=INITIALIZED)

aws-device-farm-runner (exit code: 255)

We are facing and issue while configuring the aws device farm with the trigger in the bitrise.
following is the log from the bitrise.

`
+------------------------------------------------------------------------------+

| (4) aws-device-farm-runner |
+------------------------------------------------------------------------------+
| id: aws-device-farm-runner |
| version: 0.0.7 |
| collection: https://github.com/bitrise-io/bitrise-steplib.git |
| toolkit: bash |
| time: 2019-02-04T14:22:37Z |
+------------------------------------------------------------------------------+
| |
INFO[14:22:37] * [OK] Step dependency (aws) installed, available.
INFO[14:22:37] Start installing (jq) with apt-get
sudo "apt-get" "update"
Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
Get:2 https://packages.cloud.google.com/apt cloud-sdk-xenial InRelease [6,372 B]
Hit:3 http://archive.ubuntu.com/ubuntu xenial InRelease
Get:4 https://deb.nodesource.com/node_8.x xenial InRelease [4,646 B]
Get:5 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
Get:6 https://download.docker.com/linux/ubuntu xenial InRelease [66.2 kB]
Hit:7 https://dl.yarnpkg.com/debian stable InRelease
Get:8 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [779 kB]
Get:9 http://security.ubuntu.com/ubuntu xenial-security/main i386 Packages [658 kB]
Get:10 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [535 kB]
Get:11 http://security.ubuntu.com/ubuntu xenial-security/universe i386 Packages [464 kB]
Get:12 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]
Hit:13 https://packagecloud.io/github/git-lfs/ubuntu xenial InRelease
Get:14 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [1,175 kB]
Get:15 http://archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages [1,032 kB]
Get:16 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [936 kB]
Get:17 http://archive.ubuntu.com/ubuntu xenial-updates/universe i386 Packages [856 kB]
Fetched 6,837 kB in 1s (3,730 kB/s)
Reading package lists...
(jq) isn't installed, installing...

[OK] jq installed
INFO[14:22:46] * [OK] Step dependency (jq) installed, available.
Configs:
access_key_id: ***
secret_access_key: ***
device_farm_project: [REDACTED]
test_package_name: [REDACTED]-development-debug-androidTest.apk
test_type: INSTRUMENTATION
filter:
billing_method: METERED
locale: en_US
platform: android
ipa_path:
ios_pool:
apk_path: /bitrise/deploy/[REDACTED]-development-debug-androidTest.apk
- run_name_prefix:
build_version: 6128
aws_region: [REDACTED]
run_wait_for_results: true
run_fail_on_warning: false
AWS region ([REDACTED]) specified!
Setting up device farm run for platform 'android'.
run_platform: android
- app_package_path: /bitrise/deploy/[REDACTED]-development-debug-androidTest.apk
upload_type: ANDROID_APP
Initialized upload of package '[REDACTED]-development-debug-androidTest.apk' for app ARN 'ARN'
Beginning upload
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 2024k 0 0 100 2024k 0 2442k --:--:-- --:--:-- --:--:-- 2444k
Upload finished. Polling for status.
Upload status: SUCCEEDED
Upload successful! Starting run...
An error occurred (NotFoundException) when calling the ScheduleRun operation: One or more of the resources for this request are not available.
| |
+---+---------------------------------------------------------------+----------+
| x | aws-device-farm-runner (exit code: 255) | 14 sec |
+---+---------------------------------------------------------------+----------+
| Issue tracker: ...com/peartherapeutics/bitrise-aws-device-farm-runner/issues |
| Source: https://github.com/peartherapeutics/bitrise-aws-device-farm-runner |
+---+---------------------------------------------------------------+----------+
`

Make boolean inputs an option dropdown in step.yml

undocumented input cause fail?

Trying to use this build step in my CI env., but it fails. It's probably because I don't understand what I should use as test_package_name.

It responds with max() arg is an empty sequence

+------------------------------------------------------------------------------+
| (12) [email protected]                                            |
+------------------------------------------------------------------------------+
| id: aws-device-farm-runner                                                   |
| version: 0.0.2                                                               |
| collection: https://github.com/bitrise-io/bitrise-steplib.git                |
| toolkit: bash                                                                |
| time: 2016-12-06T17:20:58Z                                                   |
+------------------------------------------------------------------------------+
|                                                                              |
INFO[17:20:58] Start installing (awscli) with apt-get       
INFO[17:20:58]  * [OK] Step dependency (aws) installed, available. 

Configs:
  * access_key_id: ***
  * secret_access_key: ***
  * device_farm_project: arn:aws:devicefarm:us-west-2:791829917870:project:***
  * test_package_name: /bitrise/deploy/app-debug-androidTest.apk
  * test_type: INSTRUMENTATION
  * platform: android
  * ipa_path: 
  * ios_pool: 
  * apk_path: /bitrise/deploy/app-debug-androidTest.apk
  * android_pool: arn:aws:devicefarm:us-west-2:791829917870:devicepool:***
  * run_name_prefix: 
  * build_version: 131
  * aws_region: us-west-2

  AWS region (us-west-2) specified!

max() arg is an empty sequence
|                                                                              |
+---+---------------------------------------------------------------+----------+
| x | [email protected] (exit code: 255)                 | 1.74 sec |
+---+---------------------------------------------------------------+----------+
| Issue tracker: ...com/peartherapeutics/bitrise-aws-device-farm-runner/issues |
| Source: https://github.com/peartherapeutics/bitrise-aws-device-farm-runner   |
+---+---------------------------------------------------------------+----------+

peartherapeutics / bitrise-aws-device-farm-runner Goto Github PK