microsoft / testbase Goto Github PK

Samples and Tools for Test Base for M365.

License: MIT License

C# 4.31% PowerShell 49.44% Python 20.34% JavaScript 14.35% HTML 1.14% Vue 10.41%

test-automation windows-10 windows-11 compatibility-test samples sdk

testbase's Introduction

Important: Test Base for Microsoft 365 will transition to end-of-life (EOL) on May 31, 2024.

We're committed to working closely with each customer to provide support and guidance to make the transition as smooth as possible. If you have any questions, concerns, or need assistance, submit a support request.

Test Base for Microsoft 365

Test Base for Microsoft 365 is an Azure service that facilitates data-driven testing of applications. Backed by the power of data and the cloud, it enables IT professionals to take advantage of intelligent testing from anywhere in the world. It will help you test your applications without the hassle, time commitment, and expenditure of setting up and maintaining complex test environments. Most importantly, it will give you access to pre-release Windows Updates on secure virtual machines (VMs) and world-class intelligence for your applications.

This repository contains various samples and utilities to build a Test Base package.

Best Practices

Provide you with some useful samples of Test Base integration, which are best practices to use Test Base, such as

Construct Test Base scripts with your project
Build and upload test base package
Monitor the test results of Test Base

Sample app source code

It contains the source code of the sample app. This app is a sample/test app to try out the functionality of Test Base.

Sample List

Two types of samples are included in this repository: Package and SDK.

Package Samples

We provide many package samples with different test types and different languages. Most of the samples use Calculator as the test target.

Out of Box

An OOB test performs an install, launch, close, and uninstall of your application. After the install, the launch-close routine is repeated 30 times before a single uninstall is run. The OOB test provides you with standardized telemetry on your package to compare across Windows builds.

Two OOB samples are provided, click the links to find more details:

Functional

A Functional test executes your uploaded test script(s) on your package. The scripts are run in the sequence you specified and a failure in a particular script will stop subsequent scripts from executing.

Appium is levaraged to do UI automation test. And MSTest is levaraged to do CLI test. Also C#, Java, Python are used in the samples.

Here is the list of current functional test samples, click the links to find more details:

Flow Driven

Flow driven test allows you to arrange your test scripts with enhanced flow control. To help you comprehensively validate the impact of an in-place Windows upgrade, you can use flow driven tests to execute your tests on both the baseline OS and target OS with a side-by-side test result comparison.

The Flow driven test samples will be available soon.

SDK Samples

Test Base provides APIs/SDK to help you manage Test Base resources, get test results programmatically, and integrate them with our CI tools. SDK samples show how to use Test Base SDK in different ways.

Samples of how to use Python SDK

Utilities

Utilities contain tools and scripts to help integrate with CICD, build and verify the Test Base package.

Issues and Feedback

To report an issue, visit Issues page.

To provide feedback, make feature proposals, or participate in polls, visit Discussions page.

Useful links

Test Base Blog - Get the latest news about Test Base
Test Base for Microsoft 365 documentation - Microsoft documentation on details of Test Base.

testbase's People

Contributors

Stargazers

Watchers

Forkers

fenhan isabella232 bingogo123 dingshouqing skywing918 youri970410 schenardie seanpm2001 guden htlcalbbs gyz0072004 v-shengpan kkamegawa hxdhttk xiaoyihuang pbo-linaro

testbase's Issues

Azure Pipeline CI failed to build

I forked this repo and set up CI in Azure Pipelines.
I can build local environment, but Azure Pipelines can't build.

Syntax Error: Error: You forgot to add 'mini-css-extract-plugin' plugin (i.e. { plugins: [new MiniCssExtractPlugin()] }), please read https://github.com/webpack-contrib/mini-css-extract-plugin#getting-started

How to fix this?

2023-12-10T07:37:11.8224565Z ##[section]Starting: Build App
2023-12-10T07:37:11.8332999Z ==============================================================================
2023-12-10T07:37:11.8333341Z Task         : PowerShell
2023-12-10T07:37:11.8333445Z Description  : Run a PowerShell script on Linux, macOS, or Windows
2023-12-10T07:37:11.8333627Z Version      : 2.232.0
2023-12-10T07:37:11.8333723Z Author       : Microsoft Corporation
2023-12-10T07:37:11.8333870Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell
2023-12-10T07:37:11.8334047Z ==============================================================================
2023-12-10T07:37:13.0112538Z Generating script.
2023-12-10T07:37:13.0793548Z ========================== Starting Command Output ===========================
2023-12-10T07:37:13.1001596Z ##[command]"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'D:\a\_temp\1cfff0dc-5793-490e-8e40-26910e0bbe68.ps1'"
2023-12-10T07:37:22.2935976Z MSBuild version 17.8.3+195e7f5a3 for .NET
2023-12-10T07:37:46.5139855Z   Determining projects to restore...
2023-12-10T07:38:08.3088810Z   Restored D:\a\1\s\Sample-App-src\Calculator\CalculatorCLI\CalculatorCLI.csproj (in 20.91 sec).
2023-12-10T07:38:32.5185456Z   CalculatorCLI -> D:\a\1\s\Sample-App-src\Calculator\CalculatorCLI\bin\Release\net6.0\win-x64\CalculatorCLI.dll
2023-12-10T07:38:32.5417488Z   CalculatorCLI -> D:\a\1\s\Sample-App-src\Calculator\CalculatorCLI\drop\
2023-12-10T07:39:22.8419200Z npm WARN EBADENGINE Unsupported engine {
2023-12-10T07:39:22.8518525Z npm WARN EBADENGINE   package: '@achrinza/[email protected]',
2023-12-10T07:39:22.8519620Z npm WARN EBADENGINE   required: { node: '8 || 10 || 12 || 14 || 16 || 17' },
2023-12-10T07:39:22.8520962Z npm WARN EBADENGINE   current: { node: 'v18.18.2', npm: '9.8.1' }
2023-12-10T07:39:22.8521385Z npm WARN EBADENGINE }
2023-12-10T07:39:35.7012780Z npm WARN deprecated [email protected]: See https://github.com/lydell/source-map-url#deprecated
2023-12-10T07:39:35.9721972Z npm WARN deprecated @npmcli/[email protected]: This functionality has been moved to @npmcli/fs
2023-12-10T07:39:38.7232426Z npm WARN deprecated [email protected]: Please see https://github.com/lydell/urix#deprecated
2023-12-10T07:39:40.0363420Z npm WARN deprecated @hapi/[email protected]: This version has been deprecated and is no longer supported or maintained
2023-12-10T07:39:40.2671859Z npm WARN deprecated [email protected]: this library is no longer supported
2023-12-10T07:39:40.2717891Z npm WARN deprecated [email protected]: Modern JS already guarantees Array#sort() is a stable sort, so this library is deprecated. See the compatibility table on MDN: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/sort#browser_compatibility
2023-12-10T07:39:40.3191061Z npm WARN deprecated @babel/[email protected]: This proposal has been merged to the ECMAScript standard and thus this plugin is no longer maintained. Please use @babel/plugin-transform-class-properties instead.
2023-12-10T07:39:40.7764559Z npm WARN deprecated @hapi/[email protected]: This version has been deprecated and is no longer supported or maintained
2023-12-10T07:39:40.7986648Z npm WARN deprecated [email protected]: Please upgrade to consolidate v1.0.0+ as it has been modernized with several long-awaited fixes implemented. Maintenance is supported by Forward Email at https://forwardemail.net ; follow/watch https://github.com/ladjs/consolidate for updates and release changelog
2023-12-10T07:39:40.8134435Z npm WARN deprecated [email protected]: See https://github.com/lydell/source-map-resolve#deprecated
2023-12-10T07:39:40.8436803Z npm WARN deprecated [email protected]: https://github.com/lydell/resolve-url#deprecated
2023-12-10T07:39:41.4663786Z npm WARN deprecated [email protected]: Chokidar 2 does not receive security updates since 2019. Upgrade to chokidar 3 with 15x fewer dependencies
2023-12-10T07:39:41.7917122Z npm WARN deprecated @hapi/[email protected]: Moved to 'npm install @sideway/address'
2023-12-10T07:39:41.7962282Z npm WARN deprecated [email protected]: Please use @electron/asar moving forward.  There is no API change, just a package name change
2023-12-10T07:39:42.3815419Z npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
2023-12-10T07:39:42.4116513Z npm WARN deprecated [email protected]: Please use @electron/osx-sign moving forward. Be aware the API is slightly different
2023-12-10T07:39:42.4599380Z npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
2023-12-10T07:39:43.2395420Z npm WARN deprecated @hapi/[email protected]: This version has been deprecated and is no longer supported or maintained
2023-12-10T07:39:43.5983237Z npm WARN deprecated @hapi/[email protected]: Switch to 'npm install joi'
2023-12-10T07:42:01.9144253Z 
2023-12-10T07:42:01.9146483Z > [email protected] postinstall
2023-12-10T07:42:01.9147874Z > electron-builder install-app-deps
2023-12-10T07:42:01.9148727Z 
2023-12-10T07:42:02.6639763Z   • electron-builder  version=22.14.13
2023-12-10T07:42:02.7435077Z   • rebuilding native dependencies  dependencies=@bitdisaster/[email protected] platform=win32 arch=x64
2023-12-10T07:42:08.8976389Z 
2023-12-10T07:42:08.8977648Z added 1629 packages, and audited 1630 packages in 3m
2023-12-10T07:42:08.8978182Z 
2023-12-10T07:42:08.8978690Z 135 packages are looking for funding
2023-12-10T07:42:08.8979151Z   run `npm fund` for details
2023-12-10T07:42:08.9622864Z 
2023-12-10T07:42:08.9624037Z 25 vulnerabilities (13 moderate, 12 high)
2023-12-10T07:42:08.9624505Z 
2023-12-10T07:42:08.9624971Z To address issues that do not require attention, run:
2023-12-10T07:42:08.9625456Z   npm audit fix
2023-12-10T07:42:08.9625903Z 
2023-12-10T07:42:08.9626336Z To address all issues (including breaking changes), run:
2023-12-10T07:42:08.9627043Z   npm audit fix --force
2023-12-10T07:42:08.9627415Z 
2023-12-10T07:42:08.9627817Z Run `npm audit` for details.
2023-12-10T07:42:10.0376619Z 
2023-12-10T07:42:10.0377742Z > [email protected] pack
2023-12-10T07:42:10.0378448Z > npm run electron:build && node ./build_installer.js
2023-12-10T07:42:10.0378920Z 
2023-12-10T07:42:11.1406003Z 
2023-12-10T07:42:11.1407152Z > [email protected] electron:build
2023-12-10T07:42:11.1407985Z > vue-cli-service electron:build
2023-12-10T07:42:11.1408368Z 
2023-12-10T07:42:13.0440751Z  INFO  Bundling render process:
2023-12-10T07:42:13.0836152Z All browser targets in the browserslist configuration have supported ES module.
2023-12-10T07:42:13.0837335Z Therefore we don't build two separate bundles for differential loading.
2023-12-10T07:42:13.0837809Z 
2023-12-10T07:42:13.1112878Z 
2023-12-10T07:42:13.1120526Z -  Building for production...
2023-12-10T07:42:22.4919966Z  ERROR  Failed to compile with 1 error7:42:22 AM
2023-12-10T07:42:22.4921532Z 
2023-12-10T07:42:22.4926309Z  error  in ./src/App.vue?vue&type=style&index=0&id=5133d30b&lang=css
2023-12-10T07:42:22.4926978Z 
2023-12-10T07:42:22.4927803Z Syntax Error: Error: You forgot to add 'mini-css-extract-plugin' plugin (i.e. `{ plugins: [new MiniCssExtractPlugin()] }`), please read https://github.com/webpack-contrib/mini-css-extract-plugin#getting-started
2023-12-10T07:42:22.4928323Z 
2023-12-10T07:42:22.4928633Z 
2023-12-10T07:42:22.4932398Z  ERROR  Vue CLI build failed. Please resolve any issues with your build and try again.
2023-12-10T07:42:22.8441278Z ##[error]PowerShell exited with code '1'.
2023-12-10T07:42:22.8827537Z ##[section]Finishing: Build App

Issue retrieving lists of available OSs for FeatureUpdate using Microsoft.TestBase API

Hello,

I am encountering an issue while trying to retrieve lists of available OSs for FeatureUpdate using the Microsoft.TestBase API. The API endpoint I am using is:

GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.TestBase/testBaseAccounts/{testBaseAccountName}/availableOSs?osUpdateType={osUpdateType}&api-version=2022-04-01-preview

While I can successfully retrieve lists of available OSs for SecurityUpdate using the same API, when I attempt to retrieve lists for FeatureUpdate, I receive a response code of 200, but the response body is blank.

It is important to note that a month ago, when I tried the same API endpoint for FeatureUpdate, it was working correctly and I could see the expected output in the response body.

I have followed the documentation provided at [https://learn.microsoft.com/en-us/rest/api/testbase/available-os/list?tabs=HTTP#osupdatetype]

Could you please assist me in understanding why I am not getting any response data when retrieving lists of available OSs for FeatureUpdate? Is there any additional configuration or step required to retrieve this information successfully?

Thank you for your assistance.

Error occurred when building "Sample-App-Src/Calculator"

I have cloned this project and compeleted the Prerequisites mentioned in "Sample-App-Src/Calculator",

.NET SDK
Node.js
WiX Toolset After installation, add path of wix toolset (e.g, C:\Program Files (x86)\WiX Toolset v3.14\bin) to system variables

but when I run build.ps1, the following error occurred at dotnet publish "./CalculatorCLI" -r win-x64 -c Release -o "./CalculatorCLI/drop".

PS C:\Users\t-haibinzang\Desktop\microsoft\testbase\Sample-App-Src\Calculator> .\build.ps1
用于 .NET 的 Microsoft (R) 生成引擎版本 17.2.0+41abc5629
版权所有(C) Microsoft Corporation。保留所有权利。       

  正在确定要还原的项目…
C:\Users\t-haibinzang\Desktop\microsoft\testbase\Sample-App-Src\Calculator\CalculatorCLI\CalculatorCLI.csproj : error NU1101: 找不到包 Flee。源 Microsoft Visual Studio Offline Packages 中不存在具有此 ID 
的包
  未能还原 C:\Users\t-haibinzang\Desktop\microsoft\testbase\Sample-App-Src\Calculator\CalculatorCLI\CalculatorCLI.csproj (用时 185 ms)。

exit code 3010 returned from the sample Calculator and cause sample package ran failed on Test Base

Hi,

When I try to run the sample Calculator Out of Box Package on Test Base, the uninstall script report below error code:
Uninstalling Application Error: Uninstallation failed as 3010 Unistallation script finished as 3010
Seems 3010 indicates a process has operations pending and that a reboot is required to complete the process
but for this Calculator seems does not need a reboot.
Another thing is there is a typo issue in the log "Unistallation" -> "Uninstallation"
"Check registy:" => "Check registry:" ?

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

I'm trying to upload package to azure test base using Rest API and package get's successfully uploaded but every time package gets fail in verification due some blob path related issue anyone help me to to find exact blob path.

I'm trying to upload my package to azure test base using azure rest API, Every time package get's uploaded successfully to test base service but package get failed (every time verification status is verification failed ) showing some blob related error message.("The specified blob does not exist.") I have taken blob path from file upload URL of a Test Base Account.(I have stored my package inside storage account with a container have public access)

When I run rest API the Response Code: 201

Error in JSON File after failure in verification.

"packageStatus": "Error",
"validationResults": [
{
"validationName": "Sanity Checks Validator",
"isValid": false,
"errors": [
"The specified blob does not exist."
]
}
],
"validationRunStatus": "Failed"

Test Type Items fails validation

Hi There,

When we use "Test Types - Get" API, it returns the following 3 items:

Out-of-Box-Test
Functional-Test
Flow-Driven-Test

We use these values to generate our JSON file. However, when we create a package with the setting "OutofBoxTest" ("-" chars are removed), we get the following validation error response:

"code":"EnumCaseMismatch","message":"Enum does not match case for: OutofBoxTest

When we check the packages that are already created, they contain the following value "OutOfBoxTest" ("-" chars are removed). When we compare the value returned from "Test Types - Get" API, we saw that the "OutofBoxTest" is not exactly as "OutOfBoxTest" and the "O" letter in "of" is in lower case. And this issue causes a validation error when we create a package via the API.
Therefore, the values that are returned from "Test Types - Get" should be rectified as :

Out-Of-Box-Test
Functional-Test
Flow-Driven-Test

Thanks in advance.