marathonlabs / marathon Goto Github PK

Fork provides very simple strategy: totalAllowedRetryQuota and retryPerTestCaseQuota.
Should we implement same strategy? @Malinskiy @dsvoronin

TODO @Malinskiy

linkedin/dex-test-parser#13

We already implemented analytics integration(with influxdb) and we can start implementing custom sorting strategies based on this analytics.

Disconnected device error hidden under unhelpful Cannot round NaN value in ProgressReporter:27
val percent = (float * 100.0).roundToInt()

For example:

    @Test
    fun someTest() {
        assumeTrue(false)
    }

This test reported as ignored when using default junit runner, but it's marked as failed when running using marathon. I think it will be better to mark it as "ignored" in marathon as well or add an option to configure this behaviour.

See https://junit.org/junit4/javadoc/4.12/org/junit/Assume.html

• Kotlin script
• Groovy
• Marathonfile

Why it is separated and not included in multiproject build? It would be handy(if possible) to check integration in single gradle run

Something like marathon --dump option to compress all the reports tabs for easily sending them

Some extra metrics required for "smart" batching and sorting.

• Save execution results to InfluxDb
• Query test success rate
• Query test execution time(percentile)

Steps

1. Run sample project https://github.com/xanderblinov/KakaoX (for correct running comment influx db)
2. Some test will pass and some has errors
3. Report contains not all test, but you may find logs for all tests

I was testing the last snapshot and when I have a failed test I can't see it in the report.
When tests are correct no problem.

Example repo:
https://github.com/cdsap/ExampleAppMarathon

Thanks

Whitelist + blacklist of tests by

• test class simple or FQN name
• test package
• FQN test annotation

The logic should get all the available tests first, then it should apply all the whitelists from different groups (if available), then apply all blacklists to everything that is still left (if available)

Due some infrastructure problems(adb/ddms) sometimes we are getting TestResults like:
'{"testName":"Test])","status":"INCOMPLETE","startDate":0,"endDate":0,"expectedValue":0.0,"variance":0}'
Causing problems in the distribution report:

Currently, marathon doesn't capture screenshots if video recording is not available on the device.

Time always 0.0
Example
<?xml version="1.0" encoding="UTF-8"?><testsuite name="MainActivityFlakyTest.testTextFlaky2" tests="1" failures="0" errors="0" skipped="0" time="0.0" timestamp="2018-07-24T10:36:58"><properties></properties><testcase classname="com.example.MainActivityFlakyTest" name="testTextFlaky2" time="0.0"></testcase></testsuite>

We need ci to validate changes (develop is broken atm >.>)

I can help with circle ci setup if you want.

We need define more pooling strategies. Right now we have only one strategy - common pool.
Probably we need strategy which will separate tablets and phones, run all tests on each device.
@dsvoronin @Malinskiy what do you think?

We need to research if our current vendor abstraction is enough for iOS testing

Detekt integrated, but I set maxIssues: 150.

TODO:

• Change config
• Fix warning

Currently testApplicationOutput param is required and marathon-gradle-plugin throws an exception if you try to apply it to a library project:

A problem occurred configuring project ':components:SomeUiPart'.
> com.android.build.gradle.internal.api.LibraryVariantOutputImpl_Decorated cannot be cast to com.android.build.gradle.api.ApkVariantOutput

Implement unit tests for:

• DeviceActor
• PoolProgressTracker
• TestResultReporter

• Remove mutable job field from DeviceActor.
• Extract StateMachine. Add tests

  private val context = newSingleThreadContext(device.toString())

    private var job by Delegates.observable<Job?>(null) { _, _, newValue ->
        newValue?.invokeOnCompletion {
            if (it == null) {
                state.transition(DeviceEvent.Complete)
            } else {
                it.printStackTrace()
                logger.error(it) { "Error ${it.message}" }
                state.transition(DeviceEvent.Terminate)
                terminate()
            }
        }
    }

Add Dynamic reporter loading (ServiceLoader).

missing dependencies when apply plugin: 'marathon'
Caused by: java.lang.NoClassDefFoundError: com/malinskiy/marathon/execution/Configuration
unless add dependencies by adding "com.malinskiy.marathon:core"

TODO:

• InfluxDB (#103)
• Android device testing

Useful links:

1. https://github.com/testcontainers/testcontainers-java

parameter exists in Configuration but not implement yet

It's feature request: consider to add the possibility to launch single UI test class (with several tests) from terminal/console

It's important for development/debugging

P.S. IDE plugin also could be good option in the future.

We need CLI/YML configuration, because with current implementation we cannot use marathon w/o gradle

I could've add something like that: https://raw.githubusercontent.com/avito-tech/android-ui-testing/master/build.gradle.kts
but not necessarily with bintray
it will be handy to ./gradlew publishToMavenLocal / localArtifactory for developement cycles

When Analytics enabled and db is empty we can't execute tests

TODO:

• Extract analytics core to module
• Extract influx
• Provide generic test cases for analytics backends

We still don't have implementation of batching strategy, except default strategy 1 test = 1 batch.

• Fixed size batches #29
• Batches based on data from analytics for example batch size up to 5 minutes #32

We want to have anonymised usage data from users of marathon test runner to better understand most relevant use-cases for the runner. For example we want to understand how long is the average queue during the test run, how many execution units are available and so on

This should be enable only by user accepting to send this data. So either a configuration option in Marathonfile/CLI/gradle-plugin or some interactive question during the run, but we need to think about CI use-cases where interactive questions are not an option and we shouldn't hang the build because of this.

• Remove blocking IO from Queue. (TestRusultsTracker.trackTestResult)

2.0.0 dex parser lib support annotation values.
We should add support for values too, because it can be useful for some cases.

We need tooling to understand if the devices are working as expected. It would be nice if timeline had visual information of time it takes to initialise device and also if device was disconnected - visual time point of disconnect and connect

marathon 0.1.1
seems like a valid case (it's always Unknown for emulator?)

Exception in thread "AndroidDevice(model=Unknown, serial=openstf-emulator.dev:1001)" java.lang.NumberFormatException: For input string: "unknown"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
        at java.lang.Integer.parseInt(Integer.java:580)
        at java.lang.Integer.parseInt(Integer.java:615)
        at com.malinskiy.marathon.android.AndroidDevice.getDeviceFeatures(AndroidDevice.kt:43)
        at com.malinskiy.marathon.device.DeviceInfoKt.toDeviceInfo(DeviceInfo.kt:17)
        at com.malinskiy.marathon.android.executor.listeners.AnalyticsListenerKt.toTestResult(AnalyticsListener.kt:26)
        at com.malinskiy.marathon.android.executor.listeners.AnalyticsListener.handleTestRunResults(AnalyticsListener.kt:19)
        at com.malinskiy.marathon.android.executor.listeners.AbstractTestRunResultListener.testRunEnded(AbstractTestRunResultListener.kt:44)
        at com.malinskiy.marathon.android.executor.listeners.CompositeTestRunListener.testRunEnded(CompositeTestRunListener.kt:44)
        at com.android.ddmlib.testrunner.InstrumentationResultParser.handleTestRunFailed(InstrumentationResultParser.java:618)
        at com.android.ddmlib.testrunner.InstrumentationResultParser.submitCurrentKeyValue(InstrumentationResultParser.java:339)
        at com.android.ddmlib.testrunner.InstrumentationResultParser.parse(InstrumentationResultParser.java:310)
        at com.android.ddmlib.testrunner.InstrumentationResultParser.processNewLines(InstrumentationResultParser.java:267)
        at com.android.ddmlib.MultiLineReceiver.addOutput(MultiLineReceiver.java:103)
        at com.android.ddmlib.AdbHelper.executeRemoteCommand(AdbHelper.java:572)
        at com.android.ddmlib.AdbHelper.executeRemoteCommand(AdbHelper.java:383)
        at com.android.ddmlib.Device.executeShellCommand(Device.java:635)
        at com.android.ddmlib.testrunner.RemoteAndroidTestRunner.run(RemoteAndroidTestRunner.java:270)
        at com.android.ddmlib.testrunner.RemoteAndroidTestRunner.run(RemoteAndroidTestRunner.java:255)
        at com.malinskiy.marathon.android.executor.AndroidDeviceTestRunner.execute(AndroidDeviceTestRunner.kt:47)
        at com.malinskiy.marathon.android.AndroidDevice$execute$2.doResume(AndroidDevice.kt:89)
        at kotlin.coroutines.experimental.jvm.internal.CoroutineImpl.resume(CoroutineImpl.kt:54)
        at kotlinx.coroutines.experimental.DispatchTask.run(CoroutineDispatcher.kt:123)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)