tinkoff / allure.xunit Goto Github PK

If we use xunit's TestOutputHelper to log something, it also appear in allure, but only for success tests. For failed tests exceptions are logged but test output is missing in allure.

So if we have some debug output like "Sample test userId = 123", test start failing and we compare test results, we don't see useful debug info in comparison.

e.g. for imaginary test GetUserNameByUserId() with some logging inside:

Success test sample output :

"Sample test userId = 123"
"Received username is Oleg"

Current failed test output:
"Assert.Equal() Failure (pos 0) Expected: "Oleg" Actual: "MissingName"

Fixed new failed test output:

"Assert.Equal() Failure        (pos 0) Expected: "Oleg" Actual:   "MissingName"
"Sample test userId = 123"

Steps to reproduce:
[AllureXunitTheory]
[MemberData(nameof(GetData), parameters: "testParam")]
public void InformationApiTests(string scenario, Test test) {}

public static TheoryData GetData(string testParam)
{
//return TheoryData object
}

Expected result
Tests are discovered and can be run

Actual result
Tests are not discovered and running hangs without error. From cmd, the message is: "No test is available in C:\Repos\test\bin\Debug\net5.0.\test.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again."

Hello Team,
I have cloned the Allure.Xunit repository to my local host and opened the project in Visual Studio.
When I navigate to Allure.XUnit.Examples with git Bash and run dotnet test I do not see any allure-results being generated. Any advice ???

Thank you!!!

Hi,
Nice work on creating this library.

I've worked a lot with NUnit and NUnit.Allure and think their implementation is pretty nice to use. Most notably is the use of attributes at the class level.

Is there any technical reason you did not cater for class-level attributes and is this something you are looking to add?

Cheers
Craig

When trying to run a parameterized test using [AllureXunitTheory] attribute along with [MemberData] pointing to a static class member, tests cannot be found and executed.
As static class I use the TheoryData class instead of using IEnumerable for various reasons, but it seems it's not supported yet.

Please note that in my current solution I am using:
Allure version: 2.19.0
Allure.XUnit: 2.1.3
xunit: 2.4.1
xunit.runner.visualstudio: 2.4.5
Microsoft.AspNetCore.Mvc.Testing: 6.0.8
Microsoft.NET.Test.Sdk: 17.3.1
Visual Studio version: 4.8.04084
.NET 6

If add Xunit.dependencyInjection nugget package and Startup class to .NET project then allure reports stop being created.
I suppose it's a matter of changing the entry point to the application

Current state and problem
When allure_result/*.json are uploaded to Allure TestOps, it performs automatic mapping between test case and result, based on <package>.<class name>.<method name>. When test launch is closed, automatic test cases are created, and all next results are linked to it.

It's recommended to specify the id of test case to avoid wrong automapping after test code changes, using allureId in terminology, don't confuse with Allure.Commons "testCaseId" property, which is id from external TMS.

More details at https://docs.qameta.io/allure-testops/ecosystem/intellij-plugin/#link-tests-in-ide-to-tests-cases-in-allure-testops-via-allureid

Proposed solution
[AllureAllureId("1")] attribute is created, internally it adds label with name "as_id", same as java version do.

https://javadoc.io/static/io.qameta.allure/allure-java-commons/2.13.9/io/qameta/allure/AllureId.html https://github.com/allure-framework/allure-java/blob/708a1f113842fadfc6d0cb092bb084b319525a49/allure-java-commons/src/main/java/io/qameta/allure/util/ResultsUtils.java#L81

Testing
Before fix: get result*.json, upload to Allure TestOps test launch and finish it, it creates automatic test case. Upload another test run results again, it's automapped to same test case. Try to change test code namespace for example, upload results, then new automaped test case is created, no connection to a previous one.

After fix:
get result*.json, upload to Allure TestOps, it will always be mapped to specified test case id.

Unit tests:
One extra tests is created, in result*.json we have label with name="as_id" and value.

Trying to use your code example from README https://github.com/TinkoffCreditSystems/Allure.XUnit/blob/master/src/Allure.Xunit.StepExtensions.Examples/ExampleSteps.cs

Not possible because AllureStep, AllureBefore, AllureAfter attributes are missing in the package

Exception is thrown
System.InvalidOperationException: The test method expected 1 parameter value, but 0 parameter values were provided.

System.InvalidOperationException
The test method expected 1 parameter value, but 0 parameter values were provided.
Exception doesn't have a stacktrace

Note: It works fine with InlineData

If Custom type was declared and passed throw MemberData Attribute to Test InvalidOperationException is got.

Precondition:

class MyClass
{
}

public static IEnumerable<object[]> TestData = new[]
{
    new object[] {new MyClass()}
};

[AllureXunitTheory]
[MemberData(nameof(TestData))]
public async Task GetCallInformation_ReturnsDuration(MyClass input)
{
}

Result:

System.InvalidOperationException: The test method expected 1 parameter values, but 0 parameter values were provided.

Expected Result:

Test Passed

subj.
XUnit theory works correctly.
AllureXUnitTheory does not work.
public class TestClassData : IEnumerable<object[]> { public IEnumerator<object[]> GetEnumerator() { yield return new object[] {DateOnly.MaxValue}; } IEnumerator IEnumerable.GetEnumerator() => GetEnumerator(); }

[AllureXunitTheory] [AllureParentSuite("AllTests")] [AllureSuite("Test AllureXunitTheory")] [AllureSubSuite("Test ClassData")] [ClassData(typeof(TestClassData))] public void TestTheoryWithClassData(DateOnly expected) { Assert.Equal(expected, DateOnly.MaxValue); }

Returns: System.InvalidOperationException: The test method expected 1 parameter value, but 0 parameter values were provided.