alshammaria / flakeflagger Goto Github PK

License: BSD 3-Clause "New" or "Revised" License

Java 99.37% Shell 0.02% PHP 0.05% Python 0.56%

flakeflagger's Introduction

FlakeFlagger

This repository is linked to our paper "FlakeFlagger: Predicting Flakiness Without Rerunning Tests". The preprint of the paper is avaiable in this link. Authors: Abdulrahman Alshammari, Christopher Morris, Michael Hilton, Jonathan Bell

Step 0: Identifying Flaky Tests through Reruns

The first part of experimental process is to generate a dataset of flaky tests by rerunning the test suite of each project in our study set 10,000 times. We ran this experiment set up in a simple cluster, where one machine was designated as the leader, which shared a mountpoint using NFS with the rest of the cluster (coded as /experiment/ in the attached scripts), and also ran a Redis queue. All other machines in the cluster were configured to execute the script runOneBuild.php upon boot. This script polls the Redis queue for a build job to perform, runs the build, saves the results, and reboots the machine when done. The generateCMDsByArgs.php script populates that build queue with projects to build. These scripts are in the test-rerun-scripts directory of this repository. The results of our rerun experiment are in test-rerun-scripts/test_results.csv.

Running this experiment requires non-trivial resources and general cloud computing expertise. We expect that what might be most useful for other researchers is the rich dataset that we collected from this experiment - which includes not only the outcomes of each test run (pass/fail), but also the console output and execution time of each test.

All results from this experiment are permanently archived in Zenodo. Once this artifact completes the artifact review process, we will add the entire contents of this GitHub repository to that Zenodo archive, creating a final version of our artifact that will be permanently archived and available.

Step 1: Feature Collection

We created a Maven build extension that automates the process of collecting the various static and dynamic features of each test case that we use in our predictive model. This tooling, including a detailed readme are located in the test-feature-collector directory.

Step 2: Flakiness Prediction

We used the machine learning concepts to learn and predict flaky tests. This part is a classification problem and it is well explained with a readme file in the flakiness-predicter directory.

Dataset notes for static analysis

As described in our ICSE 2021 paper, the tests in our dataset were gathered by inspecting the logs reported by maven of which tests were executed. Static approaches that aim to analyze the method bodies of these tests may fail to locate tests, for example because they are written in another language (like Groovy) or are dynamically instantiated (using a parameterized or custom runner, or through inheritance). As others have reported, there are 453 tests in our dataset that can not be easily extracted statically.

flakeflagger's People

Contributors

Stargazers

Watchers

Forkers

mostafij-rahman hayasam taher-ghaleb sakinafatima shanto-rahman nashid binaysahoo ti-tanium harel-coffee thhuestc rsd6170 zhangl64

flakeflagger's Issues

Trouble with running FlakeFlagger

Hello,

I am trying to use test-feature-collector to detect test flakiness on a maven project.

I have cloned your repository to my machine and ran "mvn clean install" on test-feature-collector, then copied test-smells-maven-extension-1.0-SNAPSHOT.jar to /Users/alperenyildiz/.m2/wrapper/dists/apache-maven-3.6.3-bin/1iopthnavndlasol9gbrbg6bf2/apache-maven-3.6.3/lib/ext/

Then, went to the maven repository I am trying to analyze and ran "mvn verify" on it couple times; then I investigated target/test-classes and target/classes folders and subfolders beneath them but I could not find smells.csv file. Thus, I suspected that something is wrong with my setup or I am using your tool incorrectly and wanted to ask for your assistance. Any help would be much appreciated.

Thank you,

Unable to Execute detect-smells-plugin: Invalid Environment Settings

I encountered an issue while trying to execute the detect-smells-plugin for the dbdeploy-core project. The build fails with the error invalid environment settings. Below are the details of the error and steps I have taken to try to resolve it.

Environment:

Operating System: Windows 10
JDK Version: 1.8.0_181
Maven Version: Apache Maven 3.3.9
Project: dbdeploy
Plugin: detect-smells-plugin 1.0-SNAPSHOT

Error Log:

[ERROR] Failed to execute goal edu.gmu.swe.smells:detect-smells-plugin:1.0-SNAPSHOT:detectSmells (test-smells-report-tests) on project dbdeploy-core: Execution test-smells-report-tests of goal edu.gmu.swe.smells:detect-smells-plugin:1.0-SNAPSHOT:detectSmells failed: invalid environment settings -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal edu.gmu.swe.smells:detect-smells-plugin:1.0-SNAPSHOT:detectSmells (test-smells-report-tests) on project dbdeploy-core: Execution test-smells-report-tests of goal edu.gmu.swe.smells:detect-smells-plugin:1.0-SNAPSHOT:detectSmells failed: invalid environment settings
...
Caused by: java.lang.IllegalStateException: invalid environment settings
at org.eclipse.jdt.core.dom.ASTParser.getClasspath(ASTParser.java:261)
at org.eclipse.jdt.core.dom.ASTParser.createASTs(ASTParser.java:993)
at edu.gmu.swe.smells.detector.internal.JDTContext.collectAllTypes(JDTContext.java:119)
at edu.gmu.swe.smells.detector.internal.JDTContext.(JDTContext.java:30)
at edu.gmu.swe.smells.detector.internal.DetectorEntryPoint.buildTestContextMap(DetectorEntryPoint.java:166)
at edu.gmu.swe.smells.detector.maven.TestSmellDetectorMojo.execute(TestSmellDetectorMojo.java:306)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
... 22 more

I need assistance to resolve the invalid environment settings error encountered while executing the detect-smells-plugin. Any insights or suggestions on how to fix this issue would be greatly appreciated.

Thank you.

maven-extension/target/test-smells-maven-extension-1.0-SNAPSHOT.jar file missing

Hello, I am trying to execute these instruction" mvn install at the top-level of this project (this directory), then copy maven-extension/target/test-smells-maven-extension-1.0-SNAPSHOT.jar to your $MAVEN_DIRECTORY/lib/ext directory" but this file is missing from the repository. Can you please provide a path to jar file? Thanks

Cannot build `test-feature-collector`

Here is my maven version:

Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Maven home: /opt/homebrew/Cellar/maven/3.8.6/libexec
Java version: 1.8.0_292, vendor: AdoptOpenJDK, runtime: /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.16", arch: "x86_64", family: "mac"

When I run mvn install under FlakeFlagger/test-feature-collector/, I see the following error:

testResourceOptimismDirectly(edu.gmu.swe.smells.test.ResourceOptimismSmellDetectorTest)  Time elapsed: 0.001 sec  <<< ERROR!
java.lang.IllegalStateException: invalid environment settings
        at org.eclipse.jdt.core.dom.ASTParser.getClasspath(ASTParser.java:261)
        at org.eclipse.jdt.core.dom.ASTParser.createASTs(ASTParser.java:993)
        at edu.gmu.swe.smells.detector.internal.JDTContext.collectAllTypes(JDTContext.java:119)
        at edu.gmu.swe.smells.detector.internal.JDTContext.<init>(JDTContext.java:30)
        at edu.gmu.swe.smells.detector.internal.DetectorEntryPoint.buildTestContextMap(DetectorEntryPoint.java:166)
        at edu.gmu.swe.smells.test.BaseSmellDetectorTest.doTestsHaveSmells(BaseSmellDetectorTest.java:113)
        at edu.gmu.swe.smells.test.ResourceOptimismSmellDetectorTest.testResourceOptimismDirectly(ResourceOptimismSmellDetectorTest.java:22)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
        at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
        at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
        at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
        at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)


Results :

Tests in error: 
  testNoRouletteTest(edu.gmu.swe.smells.test.AssertionRouletteDetectorTest): invalid environment settings
  testRouletteTest(edu.gmu.swe.smells.test.AssertionRouletteDetectorTest): invalid environment settings
  testDirectTest(edu.gmu.swe.smells.test.IndirectTestingDetectorTest): invalid environment settings
  testIndirectOnlyConsidersTestClassCode(edu.gmu.swe.smells.test.IndirectTestingDetectorTest): invalid environment settings
  testIndirectTest2(edu.gmu.swe.smells.test.IndirectTestingDetectorTest): invalid environment settings
  testIndirectTest(edu.gmu.swe.smells.test.IndirectTestingDetectorTest): invalid environment settings
  testIndirectIndirectTest(edu.gmu.swe.smells.test.IndirectTestingDetectorTest): invalid environment settings
  testIndirectDirectTest(edu.gmu.swe.smells.test.IndirectTestingDetectorTest): invalid environment settings
  testStaticFieldTest(edu.gmu.swe.smells.test.RunWarDetectorTest): invalid environment settings
  testStaticFinalFieldTest(edu.gmu.swe.smells.test.RunWarDetectorTest): invalid environment settings
  testNoRunWarTest(edu.gmu.swe.smells.test.RunWarDetectorTest): invalid environment settings
  testSameFileTest(edu.gmu.swe.smells.test.RunWarDetectorTest): invalid environment settings
  testCallsSleepIndirectly(edu.gmu.swe.smells.test.CallsSleepSmellDetectorTest): invalid environment settings
  testCallsSleepDirectly(edu.gmu.swe.smells.test.CallsSleepSmellDetectorTest): invalid environment settings
  testNonStaticNonEagerTest(edu.gmu.swe.smells.test.EagerDetectorTest): invalid environment settings
  testEagerTest(edu.gmu.swe.smells.test.EagerDetectorTest): invalid environment settings
  testNotEagerTest(edu.gmu.swe.smells.test.EagerDetectorTest): invalid environment settings
  testNonStaticEagerTest(edu.gmu.swe.smells.test.EagerDetectorTest): invalid environment settings
  testNoConditionalTest(edu.gmu.swe.smells.test.ConditionalTestDetectorTest): invalid environment settings
  testConditionalTest(edu.gmu.swe.smells.test.ConditionalTestDetectorTest): invalid environment settings
  testFireAndForgetDirectly(edu.gmu.swe.smells.test.FireAndForgetSmellDetectorTest): invalid environment settings
  testFireAndForgetIndirectly(edu.gmu.swe.smells.test.FireAndForgetSmellDetectorTest): invalid environment settings
  testMysteryGuestDirectly(edu.gmu.swe.smells.test.MysteryGuestSmellDetectorTest): invalid environment settings
  testMysteryGuestDatabaseDirectly(edu.gmu.swe.smells.test.MysteryGuestSmellDetectorTest): invalid environment settings
  testMysteryGuestIndirectly(edu.gmu.swe.smells.test.MysteryGuestSmellDetectorTest): invalid environment settings
  testMysteryGuestOnConstructors(edu.gmu.swe.smells.test.MysteryGuestSmellDetectorTest): invalid environment settings
  testMysteryGuestDatabaseIndirectly(edu.gmu.swe.smells.test.MysteryGuestSmellDetectorTest): invalid environment settings
  testResourceOptimismIndirectly(edu.gmu.swe.smells.test.ResourceOptimismSmellDetectorTest): invalid environment settings
  testResourceOptimismDirectly(edu.gmu.swe.smells.test.ResourceOptimismSmellDetectorTest): invalid environment settings

Tests run: 29, Failures: 0, Errors: 29, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for test-smells-parent 1.0-SNAPSHOT:
[INFO] 
[INFO] test-smells-parent ................................. SUCCESS [  2.118 s]
[INFO] test-smells-maven-extension ........................ SUCCESS [  5.602 s]
[INFO] detect-smells-plugin ............................... FAILURE [ 11.772 s]
[INFO] kp-test-listener ................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  20.669 s
[INFO] Finished at: 2022-10-04T22:13:11-05:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test (default-test) on project detect-smells-plugin: There are test failures.
[ERROR] 
[ERROR] Please refer to /Users/xiaoxiongli/Documents/UT Austin/Courses/Software Test/FlakeFlagger/test-feature-collector/reporting-plugin/target/surefire-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :detect-smells-plugin

It seems that some of the test cases failed. Is there something wrong with my environment?

Any suggestion would be greatly appreciated.

Thanks!

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.