Facilitates setting up integration test suits with Unreal Engine 4 Gauntlet.

Home Page: https://www.daedalic.com

License: MIT License

C# 3.52% C++ 95.08% C 0.37% HTML 1.03%

ue4-test-automation's People

Contributors

Stargazers

Watchers

ue4-test-automation's Issues

Allow multiple test map paths

Latent Actions cause tests to early terminate

I am having an issue when using the base Delay and SetTimerByEvent BP nodes in tests.
The issue presents itself only when I run the tests from the SessionFrontend. When I run any test with one of the above BP nodes, the first time the test is run after launching the editor, everything works fine. However, all subsequent runs of the test early terminates the test. It still gets marked as 'Passed' but if I place a breakpoint after the Delay or on the event attached to the SetTimerByEvent, the breakpoint is never hit. I also placed a breakpoint in C++ at FinishAct, and I never hit that breakpoint either.

I am running on 4.26.

Networking Support

Add support for automating tests involving servers and clients, much like the "Networked Integration Testing" by Rare, Ltd.: https://www.gdcvault.com/play/1026366/Automated-Testing-of-Gameplay-Features

Gauntlet run batch file fails

Hello,

I'm using Unreal Engine 4.26

When running test with Gauntlet, following the guide, I get this error

Batch file:

"D:\MyUnrealProject\Engine\Build\BatchFiles\RunUAT.bat"
RunUnreal
-project="D:\MyUnrealProject\MyGame\MyGame.uproject"
-scriptdir="D:\MyUnrealProject\MyGame"
-platform=Win64
-configuration=Development
-build=editor
-test="DaedalicTestAutomationPlugin.Automation.DaeGauntletTest"

Cmd output:

Running AutomationTool...
Parsing command line: -compile
ERROR: Failed to find scripts to execute in the command line params.
AutomationTool exiting with ExitCode=1 (Error_Unknown)
BUILD FAILED

Any hint?

Thanks

Gauntlet: Array dimensions exceeded supported range

After testing >300 tests we get the following error from Gauntlet:

Error: Array dimensions exceeded supported range..

   at System.String.SplitInternal(Char[] separator, Int32 count, StringSplitOptions options)
   at Gauntlet.UnrealLogParser.CreateSummary() in D:\Workspace\UnrealEngine\Nightly\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\Utils\Gauntlet.UnrealLogParser.cs:line 205
   at Gauntlet.UnrealLogParser.GetSummary() in D:\Workspace\UnrealEngine\Nightly\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\Utils\Gauntlet.UnrealLogParser.cs:line 189
   at Gauntlet.UnrealRoleArtifacts.get_LogSummary() in D:\Workspace\UnrealEngine\Nightly\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\Base\Gauntlet.UnrealSession.cs:line 424
   at Gauntlet.UnrealTestNode`1.<>c.<GetWarnings>b__8_0(UnrealRoleArtifacts A) in D:\Workspace\UnrealEngine\Nightly\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\Base\Gauntlet.UnrealTestNode.cs:line 116
   at System.Linq.Enumerable.<SelectManyIterator>d__17`2.MoveNext()
   at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source)
   at Gauntlet.TextExecutor.<>c.<ReportMasterSummary>b__22_1(TestExecutionInfo T) in D:\Workspace\UnrealEngine\Nightly\Engine\Source\Programs\AutomationTool\Gauntlet\Framework\Gauntlet.TestExecutor.cs:line 566
   at System.Linq.Enumerable.WhereListIterator`1.MoveNext()
   at System.Linq.Enumerable.Count[TSource](IEnumerable`1 source)
   at Gauntlet.TextExecutor.ReportMasterSummary(Int32 CurrentPass, Int32 NumPasses, TimeSpan Duration, IEnumerable`1 AllInfo) in D:\Workspace\UnrealEngine\Nightly\Engine\Source\Programs\AutomationTool\Gauntlet\Framework\Gauntlet.TestExecutor.cs:line 566
   at Gauntlet.TextExecutor.ExecuteTests(TestExecutorOptions InOptions, IEnumerable`1 RequiredTests) in D:\Workspace\UnrealEngine\Nightly\Engine\Source\Programs\AutomationTool\Gauntlet\Framework\Gauntlet.TestExecutor.cs:line 440
   at Gauntlet.RunUnreal.ExecuteTests(UnrealTestOptions Options, IEnumerable`1 TestList) in D:\Workspace\UnrealEngine\Nightly\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\RunUnreal.cs:line 307
AutomationTool exiting with ExitCode=152 (Error_TestFailure)

Performance Testing

Add utility actors and functions for testing the performance of a game, e.g. by rotating the camera around a specific location and measuring performance, then taking a screenshot of the location for reference.

See whether we can borrow anything from UAutomationPerformaceHelper and AScreenshotFunctionalTestBase.

Specifying Launch Configuration

Hey folks, this looks like a really great framework, thanks for taking the time to open source your efforts 👏.

I just finished reading the tiny Gauntlet docs and stumbled on this project while searching for implementation examples. So I apologize if I'm asking Gauntlet newb questions, I haven't yet read through its source.

Right now I am manually launching my test maps by setting specific PIE options based on the map, however, I really want to move to CI. I was hoping your framework might help here but I am a little confused about where the plugin sits in the launch process with Gauntlet in the mix.

For instance, I see that you can specify console parameters in the plugin, but can I also specify launch configuration in the plugin? For instance, when I test using PIE, I might set these options for one of my test maps:

4 clients
dedicated server
emulate bad network

Can I do that via the plugin or does that happen via some undocumented Gauntlet switches?

thanks

hey,

Just checked out your new lord of the rings game when I was searching for automation tools for ue4. Very neat plugin, haven't checked out the plugin entirely yet but it looks pretty neat so far. Thanks for the release. Do you guys plan to keep it updated?

Also gratz on the game, looking really good

DaeTestReportWriterPerformance.cpp needs #include "DaeTestLogCategory.h"

When compiling DaeTestReportWriterPerformance.cpp the following errors are reported (below). These errors are usually masked by Unreal "Unity" compilation. These errors are fixed by inserting #include "DaeTestLogCategory.h"

...\Plugins\DaedalicTestAutomationPlugin\Source\DaedalicTestAutomationPlugin\Private\DaeTestReportWriterPerformance.cpp(22): Error C2653 : 'FLogCategoryLogDaeTest': is not a class or namespace name
...\Plugins\DaedalicTestAutomationPlugin\Source\DaedalicTestAutomationPlugin\Private\DaeTestReportWriterPerformance.cpp(22): Error C2065 : 'CompileTimeVerbosity': undeclared identifier
...\Plugins\DaedalicTestAutomationPlugin\Source\DaedalicTestAutomationPlugin\Private\DaeTestReportWriterPerformance.cpp(22): Error C2065 : 'LogDaeTest': undeclared identifier
...\Plugins\DaedalicTestAutomationPlugin\Source\DaedalicTestAutomationPlugin\Private\DaeTestReportWriterPerformance.cpp(61): Error C2653 : 'FLogCategoryLogDaeTest': is not a class or namespace name
...\Plugins\DaedalicTestAutomationPlugin\Source\DaedalicTestAutomationPlugin\Private\DaeTestReportWriterPerformance.cpp(61): Error C2065 : 'CompileTimeVerbosity': undeclared identifier
...\Plugins\DaedalicTestAutomationPlugin\Source\DaedalicTestAutomationPlugin\Private\DaeTestReportWriterPerformance.cpp(61): Error C2065 : 'LogDaeTest': undeclared identifier
...\Plugins\DaedalicTestAutomationPlugin\Source\DaedalicTestAutomationPlugin\Private\DaeTestReportWriterPerformance.cpp(117): Error C2653 : 'FLogCategoryLogDaeTest': is not a class or namespace name
...\Plugins\DaedalicTestAutomationPlugin\Source\DaedalicTestAutomationPlugin\Private\DaeTestReportWriterPerformance.cpp(117): Error C2065 : 'CompileTimeVerbosity': undeclared identifier
...\Plugins\DaedalicTestAutomationPlugin\Source\DaedalicTestAutomationPlugin\Private\DaeTestReportWriterPerformance.cpp(117): Error C2065 : 'LogDaeTest': undeclared identifier
...\Plugins\DaedalicTestAutomationPlugin\Source\DaedalicTestAutomationPlugin\Private\DaeTestReportWriterPerformance.cpp(118): Error C2653 : 'FLogCategoryLogDaeTest': is not a class or namespace name
...\Plugins\DaedalicTestAutomationPlugin\Source\DaedalicTestAutomationPlugin\Private\DaeTestReportWriterPerformance.cpp(118): Error C2065 : 'CompileTimeVerbosity': undeclared identifier
...\Plugins\DaedalicTestAutomationPlugin\Source\DaedalicTestAutomationPlugin\Private\DaeTestReportWriterPerformance.cpp(118): Error C2065 : 'LogDaeTest': undeclared identifier
...\Plugins\DaedalicTestAutomationPlugin\Source\DaedalicTestAutomationPlugin\Private\DaeTestReportWriterPerformance.cpp(131): Error C2653 : 'FLogCategoryLogDaeTest': is not a class or namespace name
...\Plugins\DaedalicTestAutomationPlugin\Source\DaedalicTestAutomationPlugin\Private\DaeTestReportWriterPerformance.cpp(131): Error C2065 : 'CompileTimeVerbosity': undeclared identifier
...\Plugins\DaedalicTestAutomationPlugin\Source\DaedalicTestAutomationPlugin\Private\DaeTestReportWriterPerformance.cpp(131): Error C2065 : 'LogDaeTest': undeclared identifier

DaedalicTestAutomationPlugin.Automation.csproj needs migrating to .NET Core for UE5

DaedalicTestAutomationPlugin.Automation is a .NET Framework project, but UE5 now uses .NET Core.

Migration can be partially completed with https://github.com/dotnet/try-convert

4.25.3 -> 4.26.0: DaeTestPerformanceBudgetActor.cpp: cannot convert from 'const float [4]' to 'float'

Migrating to version 4.26:

1>E:/UnrealEngineProjects2/AutomatedTestingTDD/Plugins/DaedalicTestAutomationPlugin/Source/DaedalicTestAutomationPlugin/Private/DaeTestPerformanceBudgetActor.cpp(193): error C2440: 'initializing': cannot convert from 'const float [4]' to 'float' 1> E:/UnrealEngineProjects2/AutomatedTestingTDD/Plugins/DaedalicTestAutomationPlugin/Source/DaedalicTestAutomationPlugin/Private/DaeTestPerformanceBudgetActor.cpp(193): note: There is no context in which this conversion is possible

Throw Error When Using Gauntlet to Run Test

Followed the readme document to run a demo test case by Gauntlet, but get an error, what this mean and what should I do?

Run Commond: .\RunUAT.bat RunUnreal -project="G:\DemoProject\Demo1\Demo1.uproject" -scriptdir="G:\DemoProject\Demo1" -platform=Win64 -configuration=Development -build=editor -test="DaedalicTestAutomationPlugin.Automation.DaeGauntletTest"

Automation.ParseCommandLine: Parsing command line: RunUnreal -project=G:\DemoProject\Demo1\Demo1.uproject -scriptdir=G:\DemoProject\Demo1 -platform=Win64 -configuration=Development -build=editor -test=DaedalicTestAutomationPlugin.Automation.DaeGauntletTest
Automation.Process: Setting up command environment.
InternalUtils.SafeFileExists: SafeFileExists D:\UE4\UE_4.23\Engine\Binaries\DotNET\AutomationTool.exe=True
InternalUtils.GetEnvironmentVariable: GetEnvironmentVariable uebp_LOCAL_ROOT=
InternalUtils.GetEnvironmentVariable: GetEnvironmentVariable uebp_LOCAL_ROOT=
InternalUtils.GetEnvironmentVariable: GetEnvironmentVariable uebp_EngineSavedFolder=
CommandUtils.SetEnvVar: SetEnvVar uebp_EngineSavedFolder=D:/UE4/UE_4.23/Engine/Programs/AutomationTool/Saved
InternalUtils.GetEnvironmentVariable: GetEnvironmentVariable uebp_EngineSavedFolder=D:/UE4/UE_4.23/Engine/Programs/AutomationTool/Saved
InternalUtils.GetEnvironmentVariable: GetEnvironmentVariable uebp_CSVFile=
InternalUtils.GetEnvironmentVariable: GetEnvironmentVariable uebp_LogFolder=
InternalUtils.GetEnvironmentVariable: GetEnvironmentVariable uebp_LOCAL_ROOT=D:/UE4/UE_4.23
CommandUtils.SetEnvVar: SetEnvVar uebp_LogFolder=C:/Users/yanlin.wu/AppData/Roaming/Unreal Engine/AutomationTool/Logs/D+UE4+UE_4.23
InternalUtils.SafeDeleteFile: SafeDeleteFile C:\Users\yanlin.wu\AppData\Roaming\Unreal Engine\AutomationTool\Logs\D+UE4+UE_4.23\Log.txt
InternalUtils.GetEnvironmentVariable: GetEnvironmentVariable uebp_FinalLogFolder=
CommandUtils.SetEnvVar: SetEnvVar uebp_FinalLogFolder=C:/Users/yanlin.wu/AppData/Roaming/Unreal Engine/AutomationTool/Logs/D+UE4+UE_4.23
InternalUtils.SafeFileExists: SafeFileExists C:\WINDOWS\system32\robocopy.exe=True
InternalUtils.SafeFileExists: SafeFileExists C:\WINDOWS\system32\mount.exe=False
InternalUtils.SafeFileExists: SafeFileExists C:\WINDOWS\Sysnative\mount.exe=False
InternalUtils.SafeFileExists: SafeFileExists C:\WINDOWS\system32\cmd.exe=True
CommandUtils.SetEnvVar: SetEnvVar MallocNanoZone=0
InternalUtils.GetEnvironmentVariable: GetEnvironmentVariable uebp_UATChildInstance=0
WindowsPlatform.FindVSInstallDirs: Found Visual Studio installation: f:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise (Product=Microsoft.VisualStudio.Product.Enterprise, Version=15.9.28307.718, Sort=0)
AndroidPlatformFactory.RegisterBuildPlatforms: UnrealBuildTool.AndroidPlatformSDK has no valid SDK
IOSPlatformFactory.RegisterBuildPlatforms: UnrealBuildTool.IOSPlatformSDK using manually installed SDK
HTML5PlatformFactory.RegisterBuildPlatforms: UnrealBuildTool.HTML5PlatformSDK using manually installed SDK
WindowsPlatformFactory.RegisterBuildPlatforms: UnrealBuildTool.WindowsPlatformSDK using manually installed SDK
Log.OutputMessage: Added device Win64:default to pool
Log.WriteException: ==============================================================================
Log.WriteException: ERROR: Unable to find type DaedalicTestAutomationPlugin.Automation.DaeGauntletTest in assemblies. Namespaces= System.Linq.Enumerable+WhereSelectArrayIterator2[System.String,System.String]. Log.WriteException: (see C:\Users\yanlin.wu\AppData\Roaming\Unreal Engine\AutomationTool\Logs\D+UE4+UE_4.23\Log.txt for full exception trace) Log.WriteException: Log.WriteException: AutomationException: Unable to find type DaedalicTestAutomationPlugin.Automation.DaeGauntletTest in assemblies. Namespaces= System.Linq.Enumerable+WhereSelectArrayIterator2[System.String,System.String].
Log.WriteException: At Gauntlet.Utils.TestConstructor.GetTypeForTest(String TestName, IEnumerable1 Namespaces) Locate: D:\Build\++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\AutomationTool\Gauntlet\Framework\Gauntlet.Utils.cs:Line: 540 Log.WriteException: At Gauntlet.Utils.TestConstructor.ConstructTest[TestType,ParamType](String TestName, ParamType Arg, IEnumerable1 Namespaces) Locate: D:\Build++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\AutomationTool\Gauntlet\Framework\Gauntlet.Utils.cs:Line: 575
Log.WriteException: At Gauntlet.RunUnreal.CreateTestList(UnrealTestContext Context, Params DefaultParams, ArgumentWithParams PlatformParams) Locate: D:\Build++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\RunUnreal.cs:Line: 366
Log.WriteException: At Gauntlet.RunUnreal.RunTests(UnrealTestOptions ContextOptions) Locate: D:\Build++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\RunUnreal.cs:Line: 237
Log.WriteException: At Gauntlet.RunUnreal.Execute() Locate: D:\Build++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\RunUnreal.cs:Line: 56
Log.WriteException: At AutomationTool.Automation.Execute(List1 CommandsToExecute, Dictionary2 Commands) Locate: D:\Build++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\AutomationTool\AutomationUtils\Automation.cs:Line: 551
Log.WriteException: At AutomationTool.Automation.Process(String[] Arguments, StartupTraceListener StartupListener) Locate: D:\Build++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\AutomationTool\AutomationUtils\Automation.cs:Line: 521
Log.WriteException: At AutomationTool.Program.MainProc(String[] Arguments, StartupTraceListener StartupListener) Locate: D:\Build++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\AutomationTool\Program.cs:Line: 176
Log.WriteException: At AutomationTool.Program.<>c__DisplayClass1_0.

b__2() Locate: D:\Build++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\AutomationTool\Program.cs:Line: 88
Log.WriteException: At AutomationTool.InternalUtils.RunSingleInstance(Func`1 Main) Locate: D:\Build++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\AutomationTool\AutomationUtils\Utils.cs:Line: 725
Log.WriteException: At AutomationTool.Program.Main(String[] Arguments) Locate: D:\Build++UE4\Sync\Engine\Saved\CsTools\Engine\Source\Programs\AutomationTool\Program.cs:Line: 88
Log.WriteException: ==============================================================================
Program.Main: AutomationTool exiting with ExitCode=1 (Error_Unknown)

Gauntlet seemingly cant find Testclass

Hi, incredible framwrok, thanks a lot. Can anyone help me out with this problem.

Trying to launch tests in UE5 with Gauntlet results using the following command:

"C:\Program Files\Epic Games\UE_5.1\Engine\Build\BatchFiles\RunUAT.bat" RunUnreal -project="E:\daedalustest\daedalustest\daedalustest.uproject" -scriptdir="E:\daedalustest\daedalustest" -platform=Win64 -configuration=Development -build=editor -test="DaedalicTestAutomationPlugin.Automation.DaeGauntletTest"

results in the following error message:

### Role: EditorGame (Win64 Development)
  ##### Failed:  Process encountered fatal error (EncounteredFatalError, ExitCode=-1)
  * CommandLine: E:\daedalustest\daedalustest\daedalustest.uproject   -gauntlet=DaeGauntletTestController  -ResX=1280  -ResY=720  -windowed  -unattended  -nosplash  -stdout  -AllowStdOutLogVerbosity  -gauntlet.heartbeatperiod=30  -multihome=10.96.212.47  -multihomehttp=10.96.212.47  -nomcp  -notimeouts  -noepicportal  -game  -userdir="C:\Program Files\Epic Games\UE_5.1\GauntletTemp\DeviceCache\Win64\LocalDevice0\UserDir"
  * Log: C:\Program Files\Epic Games\UE_5.1\GauntletTemp\Logs\DaedalicTestAutomationPlugin.Automation.DaeGauntletTest_(Win64_Development_EditorGame)\EditorGame\EditorGameOutput.log
  * SavedDir: C:\Program Files\Epic Games\UE_5.1\GauntletTemp\Logs\DaedalicTestAutomationPlugin.Automation.DaeGauntletTest_(Win64_Development_EditorGame)\EditorGame
  * Fatal Errors: 1
  * Log Errors: 20
  * Log Warnings: 15

  ##### Fatal Error: Assertion failed: TestClass [File:D:\build\++UE5\Sync\Engine\Plugins\Experimental\Gauntlet\Source\Gauntlet\Private\GauntletModule.cpp] [Line: 222]
  Could not find class for controller DaeGauntletTestController
  * 0x00007ff88efd7358 UnrealEditor-Gauntlet.dll!UnknownFunction [Unknown File]
  * 0x00007ff88efd7ac0 UnrealEditor-Gauntlet.dll!UnknownFunction [Unknown File]
  * 0x00007ff88efd599f UnrealEditor-Gauntlet.dll!UnknownFunction [Unknown File]
  * 0x00007ff6d8c585d9 UnknownFunction [Unknown File]
  * 0x00007ff6d8c4291a UnknownFunction [Unknown File]
  * 0x00007ff6d8c62783 UnknownFunction [Unknown File]
  * 0x00007ff6d8c6296a UnknownFunction [Unknown File]
  * 0x00007ff6d8c65680 UnknownFunction [Unknown File]
  * 0x00007ff6d8c771b4 UnknownFunction [Unknown File]
  * 0x00007ff6d8c7a516 UnknownFunction [Unknown File]
  * 0x00007ff955c17604 KERNEL32.DLL!UnknownFunction [Unknown File]
  * 0x00007ff956d026a1 ntdll.dll!UnknownFunction [Unknown File]

  # Gauntlet summary events:
  --------------------------------------------------------------------------------
  ##### Fatal Error: Fatal Error Encountered
  Could not parse callstack. See log for full callstack

The EditorGameOutpuy.Log contains the following:

Script Stack (0 frames) :

[2023.05.15-09.20.12:295][  0]LogWindows: Error: appError called: Assertion failed: TestClass [File:D:\build\++UE5\Sync\Engine\Plugins\Experimental\Gauntlet\Source\Gauntlet\Private\GauntletModule.cpp] [Line: 222] 
Could not find class for controller DaeGauntletTestController

[2023.05.15-09.20.12:295][  0]LogWindows: Windows GetLastError: Операция успешно завершена. (0)
[2023.05.15-09.20.13:027][  0]LogWindows: Error: === Critical error: ===
[2023.05.15-09.20.13:027][  0]LogWindows: Error: 
[2023.05.15-09.20.13:027][  0]LogWindows: Error: Assertion failed: TestClass [File:D:\build\++UE5\Sync\Engine\Plugins\Experimental\Gauntlet\Source\Gauntlet\Private\GauntletModule.cpp] [Line: 222] 
[2023.05.15-09.20.13:027][  0]LogWindows: Error: Could not find class for controller DaeGauntletTestController
[2023.05.15-09.20.13:027][  0]LogWindows: Error: 
[2023.05.15-09.20.13:027][  0]LogWindows: Error: 
[2023.05.15-09.20.13:027][  0]LogWindows: Error: [Callstack] 0x00007ff88efd7358 UnrealEditor-Gauntlet.dll!UnknownFunction []
[2023.05.15-09.20.13:027][  0]LogWindows: Error: [Callstack] 0x00007ff88efd7ac0 UnrealEditor-Gauntlet.dll!UnknownFunction []
[2023.05.15-09.20.13:027][  0]LogWindows: Error: [Callstack] 0x00007ff88efd599f UnrealEditor-Gauntlet.dll!UnknownFunction []
[2023.05.15-09.20.13:027][  0]LogWindows: Error: [Callstack] 0x00007ff6d8c585d9 UnrealEditor.exe!UnknownFunction []
[2023.05.15-09.20.13:027][  0]LogWindows: Error: [Callstack] 0x00007ff6d8c4291a UnrealEditor.exe!UnknownFunction []
[2023.05.15-09.20.13:027][  0]LogWindows: Error: [Callstack] 0x00007ff6d8c62783 UnrealEditor.exe!UnknownFunction []
[2023.05.15-09.20.13:027][  0]LogWindows: Error: [Callstack] 0x00007ff6d8c6296a UnrealEditor.exe!UnknownFunction []
[2023.05.15-09.20.13:027][  0]LogWindows: Error: [Callstack] 0x00007ff6d8c65680 UnrealEditor.exe!UnknownFunction []
[2023.05.15-09.20.13:027][  0]LogWindows: Error: [Callstack] 0x00007ff6d8c771b4 UnrealEditor.exe!UnknownFunction []
[2023.05.15-09.20.13:027][  0]LogWindows: Error: [Callstack] 0x00007ff6d8c7a516 UnrealEditor.exe!UnknownFunction []
[2023.05.15-09.20.13:027][  0]LogWindows: Error: [Callstack] 0x00007ff955c17604 KERNEL32.DLL!UnknownFunction []
[2023.05.15-09.20.13:027][  0]LogWindows: Error: [Callstack] 0x00007ff956d026a1 ntdll.dll!UnknownFunction []
[2023.05.15-09.20.13:027][  0]LogWindows: Error: 
[2023.05.15-09.20.13:034][  0]LogExit: Executing StaticShutdownAfterError
[2023.05.15-09.20.13:035][  0]LogWindows: FPlatformMisc::RequestExit(1)
[2023.05.15-09.20.13:035][  0]LogWindows: FPlatformMisc::RequestExitWithStatus(1, 3)
[2023.05.15-09.20.13:035][  0]LogCore: Engine exit requested (reason: Win RequestExit)

Could anyone tell me why could that be? I honestly fail to see a reason why gauntlet fails to find the test class.

Could not resolve white space in test path

Description

The test failed to run in a project that has white space in its path.

Steps to reproduce

Create a project in a directory that has any white space
Install plugin as decribed in the README
Open TestAlwaysSucceed level and save as local Content map with a new name
Include the map in Daedalic Test Automation Plugin settings
Run test from Session Frontend > Automation

Expected result

All test succeded with no warning/error.

Actual result

Test worker is running, but produced an error output.

LogAutomationTest: Error: Test C:/Users/User/Documents/Unreal Projects/TestAutomation/Content/TestMap/SomeTestMap.umap does not exist and could not be run.

Automation window not displaying tests

Hello,

I can't get my different DaeTestActor to appear in Session Frontend Automation window, only the name of my Level appears there.

Do you have an idea?

Thanks!

Not able to compile with target framework != .NET Core 3.1

I downloaded the UE5 branch of your project,imported all,but I have problems with your .Automation Class Library.

I can see it in VS2019,but when I add the references to Gauntlet.Automation, it wont compile (says that Gauntlet.Automation tagets .NET core 3.1 framework)

When I imported your Library,it automatically imported it targeting 4.6.2 .NET framework (the newest installed on my machine).
Am I missing something?

Any plan to support 4.26?

Hi, thanks to share this fancy automation framework. I am trying to use this in the new version of unreal by source code (4.26), but meets some problems, any plans to support?

Has a compiler error at:
FActorSpawnParameters SpawnInfo;
SpawnInfo.Instigator = Instigator;
2.When run by Automation window, the cases never stopped after success..

other: The GPU version becomes to a array of float[4] in 4.26, I just make simple change to only read the first one...(runs ok..)

Fails to compile for Linux & Android platforms

Fixed in PR #23

Build error reported when addign the plug in to a project with a Linux build will be:


[11:02:08] :	 [Step 1/1] Building 7 actions with 64 processes...
[11:02:09] :	 [Step 1/1]   ** For VrProject4_26-Linux-Development
[11:02:09] :	 [Step 1/1]   BuildSettings.cpp
[11:02:18] :	 [Step 1/1]   Module.Gauntlet.gen.cpp
[11:02:19] :	 [Step 1/1]   Module.DaedalicTestAutomationPlugin.gen.cpp
[11:02:19] :	 [Step 1/1]   Module.Gauntlet.cpp
[11:02:21] :	 [Step 1/1]   Module.DaedalicTestAutomationPlugin.cpp
[11:02:21] :	 [Step 1/1]   In file included from D:/BuildAgent/work/85a46caaa9dd1588/VrProject4_26/Intermediate/Build/Linux/B4D820EA/VrProject4_26/Development/DaedalicTestAutomationPlugin/Module.DaedalicTestAutomationPlugin.cpp:1:
[11:02:21] :	 [Step 1/1]   In file included from D:/BuildAgent\work\85a46caaa9dd1588\VrProject4_26\Intermediate\Build\Linux\B4D820EA\VrProject4_26\Development\Engine\SharedPCH.Engine.ShadowErrors.h:168:
[11:02:21] :	 [Step 1/1]   In file included from D:/BuildAgent\work\85a46caaa9dd1588\Engine\Source\Runtime\Engine\Public\EngineSharedPCH.h:86:
[11:02:21] :	 [Step 1/1]   In file included from D:\BuildAgent\work\85a46caaa9dd1588\Engine\Source\Runtime\Core\Public\Templates\SharedPointer.h:113:
[11:02:21] :	 [Step 1/1]   D:\BuildAgent\work\85a46caaa9dd1588\Engine\Source\Runtime\Core\Public\Templates\SharedPointerInternals.h(164,4): error: delete called on non-final 'FDaeTestReportWriterJUnit' that has virtual functions but non-virtual destructor [-Werror,-Wdelete-non-abstract-non-virtual-dtor]
[11:02:21] :	 [Step 1/1]                           delete Object;
[11:02:21] :	 [Step 1/1]                           ^
[11:02:21] :	 [Step 1/1]   D:\BuildAgent\work\85a46caaa9dd1588\Engine\Source\Runtime\Core\Public\Templates\SharedPointerInternals.h(116,4): note: in instantiation of member function 'SharedPointerInternals::DefaultDeleter<FDaeTestReportWriterJUnit>::operator()' requested here
[11:02:21] :	 [Step 1/1]                           (*static_cast<DeleterType*>(this))(Object);
[11:02:21] :	 [Step 1/1]                           ^
[11:02:21] :	 [Step 1/1]   D:\BuildAgent\work\85a46caaa9dd1588\Engine\Source\Runtime\Core\Public\Templates\SharedPointerInternals.h(108,12): note: in instantiation of member function 'SharedPointerInternals::TReferenceControllerWithDeleter<FDaeTestReportWriterJUnit, SharedPointerInternals::DefaultDeleter<FDaeTestReportWriterJUnit> >::DestroyObject' requested here
[11:02:21] :	 [Step 1/1]                   explicit TReferenceControllerWithDeleter(ObjectType* InObject, DeleterType&& Deleter)
[11:02:21] :	 [Step 1/1]                            ^
[11:02:21] :	 [Step 1/1]   D:\BuildAgent\work\85a46caaa9dd1588\Engine\Source\Runtime\Core\Public\Templates\SharedPointerInternals.h(172,14): note: in instantiation of member function 'SharedPointerInternals::TReferenceControllerWithDeleter<FDaeTestReportWriterJUnit, SharedPointerInternals::DefaultDeleter<FDaeTestReportWriterJUnit> >::TReferenceControllerWithDeleter' requested here
[11:02:21] :	 [Step 1/1]                   return new TReferenceControllerWithDeleter<ObjectType, DefaultDeleter<ObjectType>>(Object, DefaultDeleter<ObjectType>());
[11:02:21] :	 [Step 1/1]                              ^
[11:02:21] :	 [Step 1/1]   D:\BuildAgent\work\85a46caaa9dd1588\Engine\Source\Runtime\Core\Public\Templates\SharedPointerInternals.h(204,27): note: in instantiation of function template specialization 'SharedPointerInternals::NewDefaultReferenceController<FDaeTestReportWriterJUnit>' requested here
[11:02:21] :	 [Step 1/1]                           , ReferenceController( NewDefaultReferenceController( InObject ) )
[11:02:21] :	 [Step 1/1]                                                  ^
[11:02:21] :	 [Step 1/1]   D:\BuildAgent\work\85a46caaa9dd1588\Engine\Source\Runtime\Core\Public\Templates\SharedPointer.h(1817,9): note: in instantiation of member function 'SharedPointerInternals::FRawPtrProxy<FDaeTestReportWriterJUnit>::FRawPtrProxy' requested here
[11:02:21] :	 [Step 1/1]           return SharedPointerInternals::FRawPtrProxy< ObjectType >( InObject );
[11:02:21] :	 [Step 1/1]                  ^
[11:02:21] :	 [Step 1/1]   D:\BuildAgent\work\85a46caaa9dd1588\VrProject4_26\Plugins\DaedalicTestAutomationPlugin\Source\DaedalicTestAutomationPlugin\Private\DaeTestActor.cpp(107,23): note: in instantiation of function template specialization 'MakeShareable<FDaeTestReportWriterJUnit>' requested here
[11:02:21] :	 [Step 1/1]       ReportWriters.Add(MakeShareable(new FDaeTestReportWriterJUnit()));
[11:02:21] :	 [Step 1/1]                         ^
[11:02:21] :	 [Step 1/1]   In file included from D:/BuildAgent/work/85a46caaa9dd1588/VrProject4_26/Intermediate/Build/Linux/B4D820EA/VrProject4_26/Development/DaedalicTestAutomationPlugin/Module.DaedalicTestAutomationPlugin.cpp:1:
[11:02:21] :	 [Step 1/1]   In file included from D:/BuildAgent\work\85a46caaa9dd1588\VrProject4_26\Intermediate\Build\Linux\B4D820EA\VrProject4_26\Development\Engine\SharedPCH.Engine.ShadowErrors.h:168:
[11:02:21] :	 [Step 1/1]   In file included from D:/BuildAgent\work\85a46caaa9dd1588\Engine\Source\Runtime\Engine\Public\EngineSharedPCH.h:86:
[11:02:21] :	 [Step 1/1]   In file included from D:\BuildAgent\work\85a46caaa9dd1588\Engine\Source\Runtime\Core\Public\Templates\SharedPointer.h:113:
[11:02:21] :	 [Step 1/1]   D:\BuildAgent\work\85a46caaa9dd1588\Engine\Source\Runtime\Core\Public\Templates\SharedPointerInternals.h(164,4): error: delete called on non-final 'FDaeTestReportWriterPerformance' that has virtual functions but non-virtual destructor [-Werror,-Wdelete-non-abstract-non-virtual-dtor]
[11:02:21] :	 [Step 1/1]                           delete Object;
[11:02:21] :	 [Step 1/1]                           ^
[11:02:21] :	 [Step 1/1]   D:\BuildAgent\work\85a46caaa9dd1588\Engine\Source\Runtime\Core\Public\Templates\SharedPointerInternals.h(116,4): note: in instantiation of member function 'SharedPointerInternals::DefaultDeleter<FDaeTestReportWriterPerformance>::operator()' requested here
[11:02:21] :	 [Step 1/1]                           (*static_cast<DeleterType*>(this))(Object);
[11:02:21] :	 [Step 1/1]                           ^
[11:02:21] :	 [Step 1/1]   D:\BuildAgent\work\85a46caaa9dd1588\Engine\Source\Runtime\Core\Public\Templates\SharedPointerInternals.h(108,12): note: in instantiation of member function 'SharedPointerInternals::TReferenceControllerWithDeleter<FDaeTestReportWriterPerformance, SharedPointerInternals::DefaultDeleter<FDaeTestReportWriterPerformance> >::DestroyObject' requested here
[11:02:21] :	 [Step 1/1]                   explicit TReferenceControllerWithDeleter(ObjectType* InObject, DeleterType&& Deleter)
[11:02:21] :	 [Step 1/1]                            ^
[11:02:21] :	 [Step 1/1]   D:\BuildAgent\work\85a46caaa9dd1588\Engine\Source\Runtime\Core\Public\Templates\SharedPointerInternals.h(172,14): note: in instantiation of member function 'SharedPointerInternals::TReferenceControllerWithDeleter<FDaeTestReportWriterPerformance, SharedPointerInternals::DefaultDeleter<FDaeTestReportWriterPerformance> >::TReferenceControllerWithDeleter' requested here
[11:02:21] :	 [Step 1/1]                   return new TReferenceControllerWithDeleter<ObjectType, DefaultDeleter<ObjectType>>(Object, DefaultDeleter<ObjectType>());
[11:02:21] :	 [Step 1/1]                              ^
[11:02:21] :	 [Step 1/1]   D:\BuildAgent\work\85a46caaa9dd1588\Engine\Source\Runtime\Core\Public\Templates\SharedPointerInternals.h(204,27): note: in instantiation of function template specialization 'SharedPointerInternals::NewDefaultReferenceController<FDaeTestReportWriterPerformance>' requested here
[11:02:21] :	 [Step 1/1]                           , ReferenceController( NewDefaultReferenceController( InObject ) )
[11:02:21] :	 [Step 1/1]                                                  ^
[11:02:21] :	 [Step 1/1]   D:\BuildAgent\work\85a46caaa9dd1588\Engine\Source\Runtime\Core\Public\Templates\SharedPointer.h(1817,9): note: in instantiation of member function 'SharedPointerInternals::FRawPtrProxy<FDaeTestReportWriterPerformance>::FRawPtrProxy' requested here
[11:02:21] :	 [Step 1/1]           return SharedPointerInternals::FRawPtrProxy< ObjectType >( InObject );
[11:02:21] :	 [Step 1/1]                  ^
[11:02:21] :	 [Step 1/1]   D:\BuildAgent\work\85a46caaa9dd1588\VrProject4_26\Plugins\DaedalicTestAutomationPlugin\Source\DaedalicTestAutomationPlugin\Private\DaeTestPerformanceBudgetActor.cpp(280,23): note: in instantiation of function template specialization 'MakeShareable<FDaeTestReportWriterPerformance>' requested here
[11:02:21] :	 [Step 1/1]       ReportWriters.Add(MakeShareable(new FDaeTestReportWriterPerformance()));
[11:02:21] :	 [Step 1/1]                         ^
[11:02:21] :	 [Step 1/1]   2 errors generated.
[11:02:21] :	 [Step 1/1] BUILD FAILED:  failed, retries not enabled:
[11:02:21] :	 [Step 1/1] AutomationTool exiting with ExitCode=1 (Error_Unknown)
[11:02:21] :	 [Step 1/1] BUILD FAILED
[11:02:21] :	 [Step 1/1]

JUnitReportPath and Multi target builds

Apologies if these come across as silly issues. I couldn't figure out how to get a JUnit report to output while attempting to use a multiclient config. When I use 1 client, I get a report. If I use 1 client and 1 server or n clients where n > 1, no report outputs. Was wondering if anyone found a solution for this.

Also the second issue I'm having is being able to spin up a client in Win64 and a server in Linux. The RequireRole function has a value for global::UnrealBuildTool.UnrealTargetPlatform PlatformOverride but I can't get it to work for the life of me. I also tried seeing if I can import the UnrealBuildTools dll's from the Engine source and had no luck. I imagine this might just be me not knowing how to use c# correctly. I spent a fair deal of time on it so I imagine I'm just completely missing something.

Here's my current config

using Gauntlet;

namespace DaedalicTestAutomationPlugin.Automation
{
    public class DaeGauntletTest : UnrealTestNode<DaeTestConfig>
    {
        public DaeGauntletTest(UnrealTestContext InContext) : base(InContext)
        {
        }

        public override DaeTestConfig GetConfiguration()
        {
            DaeTestConfig Config = base.GetConfiguration();
            Config.PreAssignAccount = false;
            Config.NoMCP = true;

            UnrealTestRole Client = Config.RequireRole(UnrealTargetRole.Client);
            UnrealTestRole Server = Config.RequireRole(UnrealTargetRole.Server);

            Client.Controllers.Add("DaeGauntletTestController");

            return Config;
        }
    }
}

Capitalization of <HAL/PlatformFileManager.h> is incorrect in DaeTestReportWriterPerformance.cpp

ue4-test-automation/DaedalicTestAutomationPlugin/Source/DaedalicTestAutomationPlugin/Private/DaeTestReportWriterPerformance.cpp

Line 5 in d8bfd91

#include <HAL/PlatformFilemanager.h>

When building UE5 for Linux, this causes a compile error.

Tests fail when run from Jenkins

Hello!
I successfully managed to integrate the plugin in my 5.0 project and now i'm looking into running the tests through Jenkins.

Running from the command line works fine but I get an error related with the 3d device when trying with Jenkins, I'm assuming it's because the Jenkins process runs without an active window:

LogD3D12RHI: Error: hr failed 
 at D:\Projects\MyProject\Engine\Source\Runtime\D3D12RHI\Private\Windows\WindowsD3D12Viewport.cpp:224 
 with error DXGI_ERROR_NOT_CURRENTLY_AVAILABLE

Is there anything i'm missing or it's just not possible to do this with the plugin at the time?

Cheers!

Filter out streaming levels of test maps

Help with parameterized tests

Hello, my team is looking to use this plugin for automated tests. I've been looking at the parameterized tests and may be misunderstanding how to use them.

I'll use a movement test as an example of how I picture them working:
BP_MoveTest: The test blueprint will apply InputAxis based on parameters provided and use a trigger box to confirm.
BP_MoveParams: Base class for parameters inherited from Object.
BP_MoveForwardParams: Inherits from BP_MoveParams providing correct parameters to test moving forward. This would be a parameter added to an instance of BP_MoveTest
BP_MoveRightParams: Similar to BP_MoveForwardParams but for moving right

After trying it this way it appears to not be correct as the parameters passed to the test won't be valid. Your examples use 2 different blueprint classes both inherited from Object, but only use display name as a difference between the tests. I am struggling to figure out how you would retrieve data from the parameter object.

It's been a while since I've used Unreal so probably something obvious that I'm missing. Thanks for any help.

4.24 -> 4.25 Error Fixes

Please change: Developer to DeveloperTool in your .uplugin file.

Debug GauntletTestController from VisualStudio

Hi great framework, many thanks.
Can you explain how to debug a GauntletTestController from VisualStudio? I want to do some modification and don't know how to attach when running from console.

Something like that, maybe?
https://docs.unrealengine.com/4.26/en-US/ProductionPipelines/BuildTools/AutomationTool/Overview/

Many thanks!

Can use this plugin to add unit test and run the unit tests in the Unreal Engine?

This is a great tool!

I understand to use a blueprint is a medium test, and currently Unreal Engine have lots of unit tests like TimespanTest.cpp. The plugin can run such unt tests? And how to add the unt tests via this tool?

Where defines the testsuite and testcase name?

Hey,
The testcase and testsuite are named as "TestSuite1_2" and "TestCase1_6" automatically, where the codes set suite and case "_2" and "_6"? Can I customize those names?

heres the log:

Empty DaeTestConfig fields

I don't know why, but if I don't force it to read DaeTestConfig fields (JUnitReportPath, ReportPath, TestName), it leaves them empty and it doesn't generate the reports or choose the map to test. (DaeGauntletTest.cs)

`public override DaeTestConfig GetConfiguration()
{
DaeTestConfig Config = base.GetConfiguration();
// force read commandline field
Config.JUnitReportPath = Context.TestParams.ParseValue("JUnitReportPath", "");
Config.ReportPath = Context.TestParams.ParseValue("ReportPath", "");
Config.TestName = Context.TestParams.ParseValue("TestName", "");

        Log.Warning("DaeTestConfig Config: {0}, {1}, {2}", Config.JUnitReportPath, Config.ReportPath, Config.TestName);

        // Start a single instance of the game.
        UnrealTestRole ClientRole = Config.RequireRole(UnrealTargetRole.Client);
        ClientRole.Controllers.Add("DaeGauntletTestController");
        // force write client command fields
        ClientRole.CommandLine += string.Format(" -JUnitReportPath=\"{0}\"", Config.JUnitReportPath);
        ClientRole.CommandLine += string.Format(" -ReportPath=\"{0}\"", Config.ReportPath);
        ClientRole.CommandLine += string.Format(" -TestName=\"{0}\"", Config.TestName);

        Log.Warning("DaeTestConfig ClientRole.CommandLine: {0}", ClientRole.CommandLine);

        // Ignore user account management.
        Config.NoMCP = true;

        return Config;

Support for UE5

In UE5 Preview 1, Gauntlet targets "netcoreapp3.1" and DaedalicTestAutomation targets .NetFramework4.6.2, which makes the sln fail to compile.

Are there plans to add support for UE5?

daedalicentertainment / ue4-test-automation Goto Github PK

ue4-test-automation's People

Contributors

Stargazers

Watchers

Forkers

ue4-test-automation's Issues

Description

Steps to reproduce

Expected result

Actual result

Recommend Projects

Recommend Topics

Recommend Org