microsoft / powerappstestautomation Goto Github PK

License: MIT License

C# 95.77% JavaScript 4.23%

powerappstestautomation's Introduction

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

powerappstestautomation's People

Contributors

Stargazers

Watchers

Forkers

sachchinannam nprice99 britecloud craig-blowfield tylerol biyengar s-kawara madevauto aengusheaney hitachi-murph jackgilbride999 didierjeanson daleyon jingweizhanggenzeon sandeep7492 cadilu bevenpage harikrishnancs pbadnora veenaefj dragonraelms gangwardeepak pratikvparab spaetig jonasspaetig anjuram25 rogertovalle itdexter24 funinemo dowijnands papariedel maniltester toanpham1009 meghnasharma25 aptean-for-food-and-beverage taffywrinkle claudiusgonzo hangmai cleamgd samyoon08 lanorioelaine sandipdakhare1 smacrico angelagastal angelaprovoke 2thukali pocs-at oidc-joe mitawinata lauol dsabater hdaniel1 srinathpegacrm matthewflynn84 brauliovargas sonika140 sapaca sammilanese gianfy98 nickjgill james-mckie v-sguo vsco365-jack jin-trial amitavasinha hendrikriesmeyerutilitas liamchapmanhw alaabitar huahuaxianren lauramayjjc bgovernanti thanasismak russell-goodbaby-us pranav-kh jesea123 superarnold guestdj cristianhdezz xian1001 aliyoussefi nohemix scberr pjuarezs90 matthewlee0904 waelhamze ldphilli wayno4 khaledaltawallbeh vikashsharma04 narendradhavale berenice513 shin-john88 inesnatera nittingit jeremiasbila davydrbekaert stone-pei zhyuuibm shonakm fikrihasny

powerappstestautomation's Issues

Support Multi-Factor Authentication

There is a current known limitation to signing in with MFA as documented here. This has been solved for EasyRepro which we could reuse?

PR implementing MFA: microsoft/EasyRepro#735

where we can find the steps on how to use this repository to execute a sample test in powerapp automation?

Getting started page of this documentation is blank. i would like to know the steps to use this repo to execute a sample test in powerapps automation.

System.Exception: Login page failed using Cloud Identity Login.

Hello,

I'm trying to automate my canvas app tests
I followed instruction from here: https://learn.microsoft.com/en-us/power-apps/maker/canvas-apps/test-studio-classic-pipeline-editor
Unfortunatelly I got error during execution of 'Run Power Apps Test Automation Tests via Firefox':
(same issue is visible for Chrome)

A total of 1 test files matched the specified pattern.
  Failed RunTestAutomation [7 m 32 s]
  Error Message:
   Test method Microsoft.PowerApps.TestAutomation.Tests.TestAutomation.RunTestAutomation threw exception: 
System.InvalidOperationException: Login failed after 3 attempts. Exception Details: System.Exception: Login page failed using Cloud Identity Login.
   at Microsoft.PowerApps.TestAutomation.Api.OnlineLogin.<>c.<Login>b__12_3(IWebDriver f) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Api\Pages\OnlineLogin.cs:line 160
   at Microsoft.PowerApps.TestAutomation.Browser.SeleniumExtensions.WaitUntilVisible(IWebDriver driver, By by, TimeSpan timeout, Action`1 successCallback, Action`1 failureCallback) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Browser\Extensions\SeleniumExtensions.cs:line 737
   at Microsoft.PowerApps.TestAutomation.Api.OnlineLogin.Login(IWebDriver driver, Uri uri, SecureString username, SecureString password, Action`1 redirectAction) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Api\Pages\OnlineLogin.cs:line 141
   at Microsoft.PowerApps.TestAutomation.Browser.DelegateBrowserCommand`5.ExecuteCommand(IWebDriver driver, Object[] params) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Browser\DelegateBrowserCommand.cs:line 134
   at Microsoft.PowerApps.TestAutomation.Browser.BrowserCommand`1.Execute[T1,T2,T3,T4,T5,T6,T7,T8,T9](IWebDriver driver, T1 p1, T2 p2, T3 p3, T4 p4, T5 p5, T6 p6, T7 p7, T8 p8, T9 p9) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Browser\BrowserCommand.cs:line 136
   at Microsoft.PowerApps.TestAutomation.Browser.BrowserCommand`1.Execute[T1,T2,T3,T4](IWebDriver driver, T1 p1, T2 p2, T3 p3, T4 p4) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Browser\BrowserCommand.cs:line 60
   at Microsoft.PowerApps.TestAutomation.Browser.BrowserPage.Execute[TResult,T1,T2,T3,T4](BrowserCommandOptions options, Func`6 delegate, T1 p1, T2 p2, T3 p3, T4 p4) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Browser\BrowserPage.cs:line 230
   at Microsoft.PowerApps.TestAutomation.Api.OnlineLogin.Login(Uri uri, SecureString username, SecureString password) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Api\Pages\OnlineLogin.cs:line 73
   at Microsoft.PowerApps.TestAutomation.Tests.TestAutomation.RunTestAutomation() in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Tests\TestAutomation\RunTestAutomation.cs:line 113
  Stack Trace:
      at Microsoft.PowerApps.TestAutomation.Tests.TestAutomation.RunTestAutomation() in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Tests\TestAutomation\RunTestAutomation.cs:line 189
  Standard Output Messages:
 Login.MainPage failed to load in 2 minutes using Cloud Identity Login.
 Exception on Attempt #1: System.Exception: Login page failed using Cloud Identity Login.
    at Microsoft.PowerApps.TestAutomation.Api.OnlineLogin.<>c.<Login>b__12_3(IWebDriver f) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Api\Pages\OnlineLogin.cs:line 160
    at Microsoft.PowerApps.TestAutomation.Browser.SeleniumExtensions.WaitUntilVisible(IWebDriver driver, By by, TimeSpan timeout, Action`1 successCallback, Action`1 failureCallback) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Browser\Extensions\SeleniumExtensions.cs:line 737
    at Microsoft.PowerApps.TestAutomation.Api.OnlineLogin.Login(IWebDriver driver, Uri uri, SecureString username, SecureString password, Action`1 redirectAction) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Api\Pages\OnlineLogin.cs:line 141
    at Microsoft.PowerApps.TestAutomation.Browser.DelegateBrowserCommand`5.ExecuteCommand(IWebDriver driver, Object[] params) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Browser\DelegateBrowserCommand.cs:line 134
    at Microsoft.PowerApps.TestAutomation.Browser.BrowserCommand`1.Execute[T1,T2,T3,T4,T5,T6,T7,T8,T9](IWebDriver driver, T1 p1, T2 p2, T3 p3, T4 p4, T5 p5, T6 p6, T7 p7, T8 p8, T9 p9) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Browser\BrowserCommand.cs:line 136
    at Microsoft.PowerApps.TestAutomation.Browser.BrowserCommand`1.Execute[T1,T2,T3,T4](IWebDriver driver, T1 p1, T2 p2, T3 p3, T4 p4) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Browser\BrowserCommand.cs:line 60
    at Microsoft.PowerApps.TestAutomation.Browser.BrowserPage.Execute[TResult,T1,T2,T3,T4](BrowserCommandOptions options, Func`6 delegate, T1 p1, T2 p2, T3 p3, T4 p4) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Browser\BrowserPage.cs:line 230
    at Microsoft.PowerApps.TestAutomation.Api.OnlineLogin.Login(Uri uri, SecureString username, SecureString password) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Api\Pages\OnlineLogin.cs:line 73
    at Microsoft.PowerApps.TestAutomation.Tests.TestAutomation.RunTestAutomation() in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Tests\TestAutomation\RunTestAutomation.cs:line 113

I verified my credentials few time and it workd.
Any hints where the problem is or what shall I check as a next step?

Trying run a PowerApps test automated

I'm getting some errors, someone can help?

First when is trying to Run the test I get this :
Test run detected DLL(s) which were built for different framework and platform versions. Following DLL(s) do not match current settings, which are .NETFramework,Version=v4.5 framework and X86 platform.
Microsoft.PowerApps.TestAutomation.Tests.dll is built for Framework .NETFramework,Version=v4.6 and Platform AnyCPU.

at the end :
##[error]Error: The process 'D:\a_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\3.225.0\Modules\DTAExecutionHost.exe' failed with exit code 1
##[error]Vstest failed with error. Check logs for failures. There might be failed tests.

User authentication and MFA

Hello guys,

at first I've searched an issue on test side or directly in the app. But with a simple "HelloWorld" PowerApp I have same errors (reproducible in the log:
##[warning]Vstest failed with error. Check logs for failures. There might be failed tests.
##[error]Error: The process 'D:\a_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\3.225.0\Modules\DTAExecutionHost.exe' failed with exit code 1
##[error]Vstest failed with error. Check logs for failures. There might be failed tests.

As output in the the tests, I've seen this image:

But I'm not able to get an application user with deactivated MFA. Is there a way to solve the authentication issue?

Many thanks and regards
MAK46

TypeError: Core.Telemetry is undefined

I followed the guide over at https://docs.microsoft.com/en-us/powerapps/maker/canvas-apps/test-studio-classic-pipeline-editor and I get the following error while running my pipeline:

Test method Microsoft.PowerApps.TestAutomation.Tests.TestAutomation.RunTestAutomation threw exception: OpenQA.Selenium.WebDriverException: TypeError: Core.Telemetry is undefined

Any idea what I'm doing wrong?

Cannot test by Chrome browser

The test by Chrome browser fails because Chrome browser does not allow third party's cookie in private mode.

TestCaseResult

The testcaseresult is not reported to DevOps. It is not included in the jsonresultstring. The testsuiteresult is working fine.

Problems parsing override parameters

I've been experimenting with your promising project, but have run into trouble when running my pipeline.
The pipeline runs OK until it starts the tests for the browsers, when I receive a warning concerning failure to parse override parameters.

[warning]Unable to parse the override run parameters string: -OnlineUsername *** -OnlinePassword *** -BrowserType "Chrome" -OnlineUrl https://make.powerapps.com -UsePrivateMode true -TestAutomationURLFilePath D:\a\1\s\JamesPowerAppTesting\testURLs.json -DriversPath C:\SeleniumWebDrivers\ChromeDriver -LoginMethod CloudIdentity

The hard reference to Selenium doesn't appear to feature in the pipeline templates nor in the azure-pipeline.yml.

No service connector allowed

Hi,

I am trying to use this project to execute my automated tests for my PowerApps from an Azure DevOps pipeline.

My PowerApps use 2FA. For tasks other than VSTest@2 we are using service connectors however this doesn't seem to be able using VSTest@2. I am getting the following error:

An error occurred during Test Run #1 of 1: System.InvalidOperationException: Login failed after 3 attempts. Exception Details: System.Exception: Login page failed on login retry.

Any ideas on how to overcome this issue?

Any help would be appreciated.

Thanks and regards

Add Test Studio Test Cases to an existing pipeline (pipeline has Azure Repos GIT source)

I am attempting to add test cases created in PowerApp Test Studio to a Azure DevOps Pipeline. Instructions here show how to add test cases to a new pipeline:
https://learn.microsoft.com/en-us/power-apps/maker/canvas-apps/test-studio-yaml-pipeline

The project I am working on already has a pipeline set up, that references a Azure Repos GIT source.

What would be the best way to set up test cases into an existing repo such as above? Perhaps the code in this repo would be added in the existing Azure Repo?

Inconsistent Results in Pipeline

Actions:

Test Studio test are working
Created test user for accessing the tests with access to app and dataverse (CRUD)
MFA and Security defaults disabled
Repository cloned in Azure DevOps
Updated TestURLs.json
Service connection setup
Pipeline setup with variables
Disabled Firefox task (only running Test on Chrome)

Issue:

Running the Test from the pipeline produces inconsistent results (Same tests in test studio, same commit in repo)
Sometimes it is successful, but mostly fails. (only 21.43% pass rate)

Majority of failed runs report the following prompt:

Error Message:
   Test method Microsoft.PowerApps.TestAutomation.Tests.TestAutomation.RunTestAutomation threw exception: 
OpenQA.Selenium.JavaScriptException: javascript error: Cannot read properties of undefined (reading 'Log')
  (Session info: chrome=99.0.4844.74)
  Stack Trace:
      at OpenQA.Selenium.WebDriver.UnpackAndThrowOnError(Response errorResponse)
   at OpenQA.Selenium.WebDriver.Execute(String driverCommandToExecute, Dictionary`2 parameters)
   at OpenQA.Selenium.WebDriver.ExecuteScriptCommand(String script, String commandName, Object[] args)
   at OpenQA.Selenium.WebDriver.ExecuteScript(String script, Object[] args)
   at Microsoft.PowerApps.TestAutomation.Browser.SeleniumExtensions.ExecuteScript(IWebDriver driver, String script, Object[] args) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Browser\Extensions\SeleniumExtensions.cs:line 165
   at Microsoft.PowerApps.TestAutomation.Api.TestAutomation.InitiateTest(IWebDriver driver, Uri uri) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Api\Pages\TestAutomation.cs:line 142
   at Microsoft.PowerApps.TestAutomation.Api.TestAutomation.<>c__DisplayClass1_0.<ExecuteTestAutomation>b__0(IWebDriver driver) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Api\Pages\TestAutomation.cs:line 35
   at Microsoft.PowerApps.TestAutomation.Browser.DelegateBrowserCommand`1.ExecuteCommand(IWebDriver driver, Object[] params) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Browser\DelegateBrowserCommand.cs:line 28
   at Microsoft.PowerApps.TestAutomation.Browser.BrowserCommand`1.Execute[T1,T2,T3,T4,T5,T6,T7,T8,T9](IWebDriver driver, T1 p1, T2 p2, T3 p3, T4 p4, T5 p5, T6 p6, T7 p7, T8 p8, T9 p9) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Browser\BrowserCommand.cs:line 136
   at Microsoft.PowerApps.TestAutomation.Browser.BrowserCommand`1.Execute(IWebDriver driver) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Browser\BrowserCommand.cs:line 36
   at Microsoft.PowerApps.TestAutomation.Browser.BrowserPage.Execute[TResult](BrowserCommandOptions options, Func`2 delegate) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Browser\BrowserPage.cs:line 182
   at Microsoft.PowerApps.TestAutomation.Api.TestAutomation.ExecuteTestAutomation(Uri uri, Int32 testRunNumber, Int32 maxWaitTimeInSeconds) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Api\Pages\TestAutomation.cs:line 32
   at Microsoft.PowerApps.TestAutomation.Tests.TestAutomation.RunTestAutomation() in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Tests\TestAutomation\RunTestAutomation.cs:line 193
  Standard Output Messages:
 Power Apps  Test Automation Execution Starting Test #1 of 1
 An error occurred during Test Run #1 of 1: OpenQA.Selenium.JavaScriptException: javascript error: Cannot read properties of undefined (reading 'Log')
  (Session info: chrome=99.0.4844.74)
    at OpenQA.Selenium.WebDriver.UnpackAndThrowOnError(Response errorResponse)
    at OpenQA.Selenium.WebDriver.Execute(String driverCommandToExecute, Dictionary`2 parameters)
    at OpenQA.Selenium.WebDriver.ExecuteScriptCommand(String script, String commandName, Object[] args)
    at OpenQA.Selenium.WebDriver.ExecuteScript(String script, Object[] args)
    at Microsoft.PowerApps.TestAutomation.Browser.SeleniumExtensions.ExecuteScript(IWebDriver driver, String script, Object[] args) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Browser\Extensions\SeleniumExtensions.cs:line 165
    at Microsoft.PowerApps.TestAutomation.Api.TestAutomation.InitiateTest(IWebDriver driver, Uri uri) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Api\Pages\TestAutomation.cs:line 142
    at Microsoft.PowerApps.TestAutomation.Api.TestAutomation.<>c__DisplayClass1_0.<ExecuteTestAutomation>b__0(IWebDriver driver) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Api\Pages\TestAutomation.cs:line 35
    at Microsoft.PowerApps.TestAutomation.Browser.DelegateBrowserCommand`1.ExecuteCommand(IWebDriver driver, Object[] params) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Browser\DelegateBrowserCommand.cs:line 28
    at Microsoft.PowerApps.TestAutomation.Browser.BrowserCommand`1.Execute[T1,T2,T3,T4,T5,T6,T7,T8,T9](IWebDriver driver, T1 p1, T2 p2, T3 p3, T4 p4, T5 p5, T6 p6, T7 p7, T8 p8, T9 p9) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Browser\BrowserCommand.cs:line 136
    at Microsoft.PowerApps.TestAutomation.Browser.BrowserCommand`1.Execute(IWebDriver driver) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Browser\BrowserCommand.cs:line 36
    at Microsoft.PowerApps.TestAutomation.Browser.BrowserPage.Execute[TResult](BrowserCommandOptions options, Func`2 delegate) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Browser\BrowserPage.cs:line 182
    at Microsoft.PowerApps.TestAutomation.Api.TestAutomation.ExecuteTestAutomation(Uri uri, Int32 testRunNumber, Int32 maxWaitTimeInSeconds) in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Api\Pages\TestAutomation.cs:line 32
    at Microsoft.PowerApps.TestAutomation.Tests.TestAutomation.RunTestAutomation() in D:\a\1\s\Microsoft.PowerApps.TestAutomation.Tests\TestAutomation\RunTestAutomation.cs:line 162
 Current results directory location: D:\a\_temp\..\TestResults\TestResults\Deploy_VssAdministrator 2022-03-24 12_50_24\In

Selenium.WebDriverException in InitiateTest method

Exception thrown in InitiateTest method "TestAutomation.cs" . Exception: OpenQA.Selenium.WebDriverException: javascript error: Core is not defined .
Can you help me resolve the issue?

Parameter 'UsePrivateMode' is not working

If I set the 'UsePrivateMode' parameter to false, the browser will still be in private mode.
The RunTestSettings.Initialize method is not called, so the options are not overwritten.