Giter Club home page Giter Club logo

xappium.uitest's Introduction

Xappium.UITest

Writing Cross Platform (X-Plat) apps can be a challenge. Finding a way to more easily write and run integrated UI Tests can be even more challenging. Xappium.UITest aims to help solve some of these issues by building an additional layer of abstraction over Appium that makes it easier to write X-Plat UI Tests. This project is a combination of the work done independently by @dansiegel & @redth taking the best pieces of both.

This is meant to make it easier for Developers to write and integrate UI Tests with their mobile apps.

Xappium UITest!

Current State

This repo is currently evolving and may experience breaking API changes. Community feedback and contributions are welcome and encouraged. We hope that this project will serve both as a model and backbone for UI Testing for .NET Developers with classic Xamarin / Xamarin.Forms applications as well as newer .NET 6 / .NET MAUI applications.

Roadmap

Description Status
iOS Support Done
Android Support Done
WinUI Support PR Welcome
macOS Support PR Welcome
GTK Support PR Welcome
Tizen Support PR Welcome
CLI tool In process

NOTE: Android tools are tested and do work on Mac, however they do not currently work on Windows hosts. If you're an Android Guru and would like to help get this working on Windows hosts I'd love the PR!

Configuration

The UI Test AppManager is configured through a combination of local, embedded, and environment. This gives you a wide variety of configuration options and ways to customize your build. For example you may provide a uitest.json that contains common attributes for all builds... this will be automatically copied to the UI Test output directory. You may also conditionally embed a uitest.ios.json or uitest.android.json with the logical name uitest.json. Finally you may use various Environment variables to set the configuration dynamically in a CI build. The order of precedence is as follows:

  • Local uitest.json on disk (this can be overridden in the CLI)
  • Embedded uitest.json in the UITest project assembly
  • Environment Variables

Any of these properties can be set by you, however the Xappium.Cli will automatically help you build out a completed config.

{
  "platform": "Android",
  "appPath": "/<Project Path>/UITest/bin/com.avantipoint.testapp-Signed.apk",
  "appId": "com.avantipoint.testapp",
  "deviceName": "sdk_gphone_x86",
  "udid": "emulator-5554",
  "osVersion": "29",
  "settings": {
    "name": "Dan"
  },
  "capabilities": {
    "adbPort": "5037"
  }
}

Settings vs Capabilities

At first it may be confusing that the configuration has both settings and capabilities. The Capabilities are explicit to the Appium Driver configuration. It is worth noting that if you need an explicit override to the way that Xappium is configuring a particular capability you have the ability to override that here. Note that the adbPort shown above is provided as an example and should not be required for Android.

Settings on the other hand have nothing specifically to do with Xappium.UITest or Appium, but rather are exposed to try to make it easier for you to provide values that you can use for your UI Tests such as user credentials.

Writing UI Tests with Xappium

Xappium provides helpers for MSTest, NUnit, & Xunit. These each have a XappiumTestBase class that will properly handle starting and stopping the app between each test. This ensures that Appium assigns a new Test Session between each of your tests. As a result the screenshots you take will be protected from accidentally being overwritten from one test to the next.

We'll set up our tests using the Page-Object-Model. You can get more information on how this works from Sweeky's talk at the Xamarin Developer Summit. To do this we'll use a helper class from Xappium.UITest:

public class LoginPage : BasePage
{
    protected override string Trait => "LoginButton";
}

You'll notice that we provide a trait. This is generally the AutomationId of an element on the page that should be unique that page. Now let's add a test class to validate that our App Starts up and is on the LoginPage.

public class AppTests : XappiumTestBase
{
    [Fact]
    public void AppLaunches()
    {
        new LoginPage();
    }
}

Picking a Unit Test Framework

Xappium UITest actually has no requirements on which test framework you need to use as long as it's compatible with dotnet test. As a result you are able to write your UI Tests using your favorite testing frameworks.

Using the Cli Tool

NOTE: The Cli has moved to a new repo. Xappium.UITest works with or without the Cli. The Cli is meant to make running tests a breeze by taking care of all complexities of running tests and even build the UI Test and App Head if needed.

xappium.uitest's People

Contributors

dansiegel avatar dependabot[bot] avatar stephen-hawley avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

xappium.uitest's Issues

[Bug] XappiumTestBase doesn't properly implement IDisposable

This object should have a destructor and should follow this pattern:

public class SomeObj : IDisposable {
~SomObj () {
    Dispose(false);
}
public void Dispose ()
{
    Dispose (true);
}
bool disposed;
void Dispose(bool disposing)
{
    if (disposed)
       return;
    if (disposing)
       DisposeManagedResources ();
    DisposeUnmanagedResources ();
    disposed = true;
}

Is this project alive?

Hi, I am thinking about using this nugget package. But I do not see any recent activity. Is this package still in development?

Azure DevOps ndk in inconsistent location on macOS agent

Description

AvdManager Emulator Install test fails on macOS agent due to Warning: Observed package in inconsistent location.

[xUnit.net 00:00:18.59]     Xappium.Cli.Tests.AvdManagerTests.InstallsEmulator [FAIL]
  X Xappium.Cli.Tests.AvdManagerTests.InstallsEmulator [14s 996ms]
  Error Message:
   System.Exception : Warning: Observed package id 'ndk;21.4.7075529' in inconsistent location '/Users/runner/Library/Android/sdk/ndk-bundle' (Expected '/Users/runner/Library/Android/sdk/ndk/21.4.7075529')
Error: Package path is not valid. Valid system image paths are:
null
  Stack Trace:
     at Xappium.Android.AvdManager.ExecuteInternal(Action`1 configure, CancellationToken cancellationToken, PipeSource stdInput) in /_/src/Xappium.Cli/Android/Tools/AvdManager.cs:line 98
   at Xappium.Android.AvdManager.InstallEmulator(Int32 sdkVersion, CancellationToken cancellationToken) in /_/src/Xappium.Cli/Android/Tools/AvdManager.cs:line 32
   at Xappium.Cli.Tests.AvdManagerTests.InstallsEmulator() in /_/tests/Xappium.Cli.Tests/AvdManagerTests.cs:line 28
--- End of stack trace from previous location where exception was thrown ---

macOS Agent Android Home directory has both an ndk & ndk-bundle sub-directory:

--- ndk ---

18.1.5063045
21.4.7075529
22.0.7026061

--- ndk-bundle ---

CHANGELOG.md
NOTICE
NOTICE.toolchain
README.md
build
meta
ndk-build
ndk-gdb
ndk-stack
ndk-which
package.xml
platforms
prebuilt
python-packages
shader-tools
simpleperf
source.properties
sources
sysroot
toolchains
wrap.sh

--- Android Environment Variables ---

ANDROID_HOME=/Users/runner/Library/Android/sdk
ANDROID_SDK_ROOT=/Users/runner/Library/Android/sdk
ANDROID_NDK_ROOT=/Users/runner/Library/Android/sdk/ndk-bundle
ANDROID_NDK_LATEST_HOME=/Users/runner/Library/Android/sdk/ndk/22.0.7026061
ANDROID_NDK_18R_PATH=/Users/runner/Library/Android/sdk/ndk/18.1.5063045
ANDROID_NDK_HOME=/Users/runner/Library/Android/sdk/ndk-bundle

PATH=/usr/local/opt/pipx_bin:/Users/runner/.cargo/bin:/usr/local/lib/ruby/gems/2.7.0/bin:/usr/local/opt/[email protected]/bin:/usr/local/opt/curl/bin:/usr/local/bin:/usr/local/sbin:/Users/runner/bin:/Users/runner/.yarn/bin:/Users/runner/Library/Android/sdk/tools:/Users/runner/Library/Android/sdk/platform-tools:/Users/runner/Library/Android/sdk/ndk-bundle:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/usr/bin:/bin:/usr/sbin:/sbin:/Users/runner/.dotnet/tools:/Users/runner/.ghcup/bin:/Users/runner/hostedtoolcache/stack/2.5.1/x64

WebDriver times out on Azure Pipelines hosted agent

Description

While the UI Tests work perfectly fine locally they fail on Azure Pipelines with a timeout after 60 seconds. From the output we know that Appium is indeed running and that there is some communication occurring.

[debug] [iOS] Setting did not need to be updated
[debug] [iOSSim] Setting preferences of 222BD45F-9B21-4165-B3A7-AEDEFAC0D254 Simulator to {"ConnectHardwareKeyboard":false}
[debug] [iOSSim] Setting common Simulator preferences to {"RotateWindowWhenSignaledByGuest":true,"ConnectHardwareKeyboard":false,"PasteboardAutomaticSync":false}
[debug] [iOSSim] Updated 222BD45F-9B21-4165-B3A7-AEDEFAC0D254 Simulator preferences at '/Users/runner/Library/Preferences/com.apple.iphonesimulator.plist' with {"RotateWindowWhenSignaledByGuest":true,"ConnectHardwareKeyboard":false,"PasteboardAutomaticSync":false,"DevicePreferences":{"222BD45F-9B21-4165-B3A7-AEDEFAC0D254":{"ConnectHardwareKeyboard":false}}}
[iOSSim] Booting Simulator with UDID '222BD45F-9B21-4165-B3A7-AEDEFAC0D254' in headless mode. All UI-related capabilities are going to be ignored
[iOSSim] Booting Simulator with UDID '222BD45F-9B21-4165-B3A7-AEDEFAC0D254'...
[debug] [WebDriverAgent] Parsed BUILD_DIR configuration value: '/Users/runner/Library/Developer/Xcode/DerivedData/WebDriverAgent-ciegwgvxzxdrqthilmrmczmqvrgu/Build/Products'
[debug] [WebDriverAgent] Got derived data root: '/Users/runner/Library/Developer/Xcode/DerivedData/WebDriverAgent-ciegwgvxzxdrqthilmrmczmqvrgu'
[iOSSim] Simulator with UDID 222BD45F-9B21-4165-B3A7-AEDEFAC0D254 booted in 49.884s
[debug] [BaseDriver] Event 'simStarted' logged at 1613670658330 (17:50:58 GMT+0000 (Coordinated Universal Time))
[HTTP] <-- POST /wd/hub/session - - ms - -
[HTTP] 
[xUnit.net 00:01:09.75]     TestApp.UITests.AppTests.AppLaunches [FAIL]
[xUnit.net 00:01:09.80]     TestApp.UITests.AppTests.LogsIntoMainPage [FAIL]
[debug] [IOSSimulatorLog] Starting log capture for iOS Simulator with udid '222BD45F-9B21-4165-B3A7-AEDEFAC0D254' using simctl

Despite all of the output that we get we still end up with a timeout after 60 seconds.

Failed TestApp.UITests.AppTests.AppLaunches [1 ms]
  Error Message:
   System.AggregateException : One or more errors occurred. (The HTTP request to the remote WebDriver server for URL http://127.0.0.1:4723/wd/hub/session timed out after 60 seconds.) (The following constructor parameters did not have matching fixture data: AppFixture fixture)
---- OpenQA.Selenium.WebDriverException : The HTTP request to the remote WebDriver server for URL http://127.0.0.1:4723/wd/hub/session timed out after 60 seconds.
-------- System.Net.WebException : The operation has timed out.
---- The following constructor parameters did not have matching fixture data: AppFixture fixture
  Stack Trace:
  
----- Inner Stack Trace #1 (OpenQA.Selenium.WebDriverException) -----
   at OpenQA.Selenium.Appium.Service.AppiumCommandExecutor.Execute(Command commandToExecute)
   at OpenQA.Selenium.Remote.RemoteWebDriver.Execute(String driverCommandToExecute, Dictionary`2 parameters)
   at OpenQA.Selenium.Appium.AppiumDriver`1.Execute(String driverCommandToExecute, Dictionary`2 parameters)
   at OpenQA.Selenium.Remote.RemoteWebDriver.StartSession(ICapabilities desiredCapabilities)
   at OpenQA.Selenium.Remote.RemoteWebDriver..ctor(ICommandExecutor commandExecutor, ICapabilities desiredCapabilities)
   at OpenQA.Selenium.Appium.AppiumDriver`1..ctor(ICommandExecutor commandExecutor, ICapabilities appiumOptions)
   at OpenQA.Selenium.Appium.AppiumDriver`1..ctor(Uri remoteAddress, ICapabilities appiumOptions, TimeSpan commandTimeout)
   at OpenQA.Selenium.Appium.AppiumDriver`1..ctor(Uri remoteAddress, ICapabilities appiumOptions)
   at OpenQA.Selenium.Appium.iOS.IOSDriver`1..ctor(Uri remoteAddress, DriverOptions driverOptions)
   at Xappium.UITest.XappiumTestEngine.SetupDriver() in /Users/runner/work/1/s/src/Xappium.UITest/XappiumTestEngine.cs:line 62
   at Xappium.UITest.XappiumTestEngine..ctor(UITestConfiguration config) in /Users/runner/work/1/s/src/Xappium.UITest/XappiumTestEngine.cs:line 29
   at Xappium.UITest.AppManager.StartApp(UITestConfiguration testConfig) in /Users/runner/work/1/s/src/Xappium.UITest/AppManager.cs:line 57
   at Xappium.UITest.AppManager.StartApp() in /Users/runner/work/1/s/src/Xappium.UITest/AppManager.cs:line 25
   at TestApp.UITests.AppFixture..ctor() in /Users/runner/work/1/s/sample/TestApp.UITests/AppFixture.cs:line 10
----- Inner Stack Trace -----
   at System.Net.HttpWebRequest.GetResponse()
   at OpenQA.Selenium.Remote.HttpCommandExecutor.MakeHttpRequest(HttpRequestInfo requestInfo)
----- Inner Stack Trace #2 (Xunit.Sdk.TestClassException) -----

Add support to run Android tests on Windows agent

Description

The Xappium.Client currently only supports running on a Mac host. While this requirement will always need to remain for iOS, support should be added to run tests on Windows agents for Android UI Tests.

[Bug] XappiumTestEngineBase needs to implement IDisposable correctly

Follow the pattern:

public class SomeObj : IDisposable {
~SomObj () {
    Dispose(false);
}
public void Dispose ()
{
    Dispose (true);
}
bool disposed;
void Dispose(bool disposing)
{
    if (disposed)
       return;
    if (disposing)
       DisposeManagedResources ();
    DisposeUnmanagedResources ();
    disposed = true;
}

Add Test App with SingleProject

Description

While there is theoretical support for the new .NET 6 apps, a .NET 6 sample app should be added with integration tests added to CI

IsRunningOnMacInternal could be better

IsRunningOnMacInternal could be written portably without pinvokes:

static bool IsRunningOnMacInternal ()
{
    return RuntimeInformation.IsOSPlatform(OSPlatform.OSX);
}

Android UI Test fails to run

Description

When running the tests locally we get an error that we could not proxy the command to the remote server.

Error Message:
   System.AggregateException : One or more errors occurred. (A new session could not be created. Details: An unknown server-side error occurred while processing the command. Original error: Could not proxy command to the remote server. Original error: socket hang up. Check https://github.com/appium/appium-espresso-driver#troubleshooting regarding advanced session startup troubleshooting. (SessionNotCreated)) (The following constructor parameters did not have matching fixture data: AppFixture fixture)
---- System.InvalidOperationException : A new session could not be created. Details: An unknown server-side error occurred while processing the command. Original error: Could not proxy command to the remote server. Original error: socket hang up. Check https://github.com/appium/appium-espresso-driver#troubleshooting regarding advanced session startup troubleshooting. (SessionNotCreated)
---- The following constructor parameters did not have matching fixture data: AppFixture fixture
  Stack Trace:
  
----- Inner Stack Trace #1 (System.InvalidOperationException) -----
   at OpenQA.Selenium.Remote.RemoteWebDriver.UnpackAndThrowOnError(Response errorResponse)
   at OpenQA.Selenium.Remote.RemoteWebDriver.Execute(String driverCommandToExecute, Dictionary`2 parameters)
   at OpenQA.Selenium.Appium.AppiumDriver`1.Execute(String driverCommandToExecute, Dictionary`2 parameters)
   at OpenQA.Selenium.Remote.RemoteWebDriver.StartSession(ICapabilities desiredCapabilities)
   at OpenQA.Selenium.Remote.RemoteWebDriver..ctor(ICommandExecutor commandExecutor, ICapabilities desiredCapabilities)
   at OpenQA.Selenium.Appium.AppiumDriver`1..ctor(ICommandExecutor commandExecutor, ICapabilities appiumOptions)
   at OpenQA.Selenium.Appium.AppiumDriver`1..ctor(Uri remoteAddress, ICapabilities appiumOptions, TimeSpan commandTimeout)
   at OpenQA.Selenium.Appium.AppiumDriver`1..ctor(Uri remoteAddress, ICapabilities appiumOptions)
   at OpenQA.Selenium.Appium.Android.AndroidDriver`1..ctor(Uri remoteAddress, DriverOptions driverOptions)
   at Xappium.UITest.XappiumTestEngine.SetupDriver() in /Users/dansiegel/repos/Libraries/Xappium.UITest/src/Xappium.UITest/XappiumTestEngine.cs:line 81
   at Xappium.UITest.XappiumTestEngine..ctor(UITestConfiguration config) in /Users/dansiegel/repos/Libraries/Xappium.UITest/src/Xappium.UITest/XappiumTestEngine.cs:line 29
   at Xappium.UITest.AppManager.StartApp(UITestConfiguration testConfig) in /Users/dansiegel/repos/Libraries/Xappium.UITest/src/Xappium.UITest/AppManager.cs:line 57
   at Xappium.UITest.AppManager.StartApp() in /Users/dansiegel/repos/Libraries/Xappium.UITest/src/Xappium.UITest/AppManager.cs:line 25
   at TestApp.UITests.AppFixture..ctor() in /Users/dansiegel/repos/Libraries/Xappium.UITest/sample/TestApp.UITests/AppFixture.cs:line 10
----- Inner Stack Trace #2 (Xunit.Sdk.TestClassException) -----

Android - adb fails when multiple devices are attached

Description

Currently there is no validation on android devices. After started the emulator we need to do a better check as the following command will fail if multiple devices are connected...

> adb wait-for-device shell getprop sys.boot_completed 

This will fail with the error:

error: more than one device/emulator

Add logger with Verbosity Flag

Description

Currently the Xappium CLI writes most output directly to the console. Instead a new Logger class should be implemented.

This should support a new cli parameter for Verbosity which will control what the Logger will and will not actually output to the Console while ensuring that all logs are written to a file for review if needed.

Verbosity Level Description
Minimal This will only write essential step by step information
Normal This will contain more Tool output
Detailed This will contain most Tool output other than Appium
Diagnostic This will contain all Tool output including Appium

[Bug] Xappium Targets are not packed

Description

Xappium has targets that are meant to be packed which will automatically copy the uitest.json to the output directory. It appears that this is not currently being packed. This will need to be packed in future releases. This should also be moved to buildTransitive so that when Xappium.UITest is referenced transitively from one of the Test Framework packages you will still get the targets.

Steps to Reproduce

  1. Install Xappium.UITest
  2. Add a uitest.json file
  3. Build your test project

Expected Behavior

uitest.json should be copied to the output directory

Actual Behavior

The uitest.json is not copied to the output directory

Basic Information

  • Version with issue: 1.0.159-beta
  • Last known good version: n/a

Screenshots

n/a

Reproduction Link

n/a

Workaround

Add the following to the csproj of the test project

<ItemGroup>
  <None Include="uitest.json">
    <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
  </None>
</ItemGroup>

Brew Tap throws exception

Description

Brew oddly is writing to the Error output when tapping a new cask... I may need a keg for this one...

Tapped 1 cask and 1 formula (31 files, 1.6MB).
   at Xappium.Tools.Brew.ExecuteInternal(Action`1 configure, CancellationToken cancellationToken) in /_/src/Xappium.Cli/Tools/Brew.cs:line 70
   at Xappium.Tools.Brew.InstallAppleSimUtils(CancellationToken cancellationToken) in /_/src/Xappium.Cli/Tools/Brew.cs:line 34
   at Xunit.Record.ExceptionAsync(Func`1 testCode) in C:\Dev\xunit\xunit\src\xunit.core\Record.cs:line 77
  Stack Trace:
     at Xappium.Cli.Tests.Tools.BrewTests.InstallsAppleSimUtils() in /_/tests/Xappium.Cli.Tests/Tests/Tools/BrewTests.cs:line 21

DotNet test fails to run in the CLI

Description

After the CLI has successfully built and prepared all of the resources required for testing including getting Appium installed and running, the DotNet Test command fails. It does this with zero output, standard or error. It's unclear why this is failing as the command arguments work as expected when running it from the CLI independent of the Xappium Cli.

See VSTest issue.

Win32Exception on Appium install

Description

When running on Windows the CLI tool will get to the point of installing appium and then throw a Win32Exception.

Installing/running Appium...
Appium is not currently installed
Node: v12.18.3 installed
System.ComponentModel.Win32Exception (2): The system cannot find the file specified.
   at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at CliWrap.Utils.ProcessEx.Start() in /_/CliWrap/Utils/ProcessEx.cs:line 65
   at CliWrap.Command.ExecuteAsync(ProcessEx process, CancellationToken cancellationToken) in /_/CliWrap/Command.cs:line 363
   at Xappium.Tools.Node.InstallPackage(String packageName, CancellationToken cancellationToken) in C:\repos\OSS\xappium.uitest\src\Xappium.Cli\Tools\Node.cs:line 56
   at Xappium.Tools.Appium.Install(CancellationToken cancellationToken) in C:\repos\OSS\xappium.uitest\src\Xappium.Cli\Tools\Appium.cs:line 64
   at Xappium.Program.OnExecuteAsync(CancellationToken cancellationToken) in C:\repos\OSS\xappium.uitest\src\Xappium.Cli\Program.cs:line 144

[Enhancement] Publish Xappium to nuget.org

Summary

I understand there might be some work in progress,
But is it possible to publish Xappium to nuget.org ?

Use case

Currently, to run UI test with Xappium, I have to download the whole project and add preference locally. A nuget package (preview) would make it much simpler to use Xappium.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.