JSR - The Java Test Suite Reduction Framework
MIT License
JSR is a test suite reduction framework for Java. It consists of the following modules:
Please note that JSR is currently still undergoing active Development. Features are subject to change and the installation process will change in the future. Our long-term goal is to create a test suite reduction framework for practical use in the software industry. If you want to contribute to JSR, please feel free to submit an issue or a PR. I would be happy if you want to get in touch with me.
The core framework module, containing the functionality and implementation of algorithms, interactions with programs, libraries, etc.
More specifically, the Core modules provides APIs to clients for the following features:
A CLI frontend of the JSR framework, supporting the following features:
The IntelliJ IDE Plugin serves as the front-end GUI. It directly integrates with IntelliJ so that developers can interact with JSR in a convenient and already established way. The Plugin provides JSR-Core functionality with a few additional features that only make sense in the IDE/GUI context:
This chapter outlines how to install JSR. All steps are required to set it up properly.
Please note that the installation process is subject to change as the project progresses.
JAVA_HOME
points to your default JDKgit clone [email protected]:Lms24/JSR.git
cd jsr
sh install.sh
Before building the actual project though, we need to initialize our testing environment so that the project build can properly run the unit tests:
cd jsr/JSR-Core/src/test/resources/smallProject
gradle
gradle build -x test
gradle testJar
Then, we can finally build the JSR framework
cd ../../.././../.. #so that we're in the 'jsr' root directory
gradle
gradle build
Congratulations, JSR was built successfully.
Note: The commands in step 3 can also be started by opening the respective projects in IntelliJ and running the gradle tasks from the gradle plugin.
The JSR IntelliJ plugin is not yet published. Thus, it cannot simply be installed into your IntelliJ IDE instance.
Currently, one has to start IntelliJ in sandbox mode to work with the JSR plugin. The easiest way is to run:
gradle JSR-IntelliJ-Plugin:runIde
Alternatively, when the JSR project is already opened in IntelliJ, one can run the same gradle task from the gradle plugin.
The following rundown is based on the smallProject
that is used for integration tests
in the core library.
To try out the plugin, run IntelliJ as described above and open the smallProject
under jsr/JSR-Core/src/test/resources/smallProject
.
This project includes a test suite with around 15 tests of which many are purposefully redundant.
Once the project is loaded and IntelliJ finished building its indexes, open the JSR plugin. It should be located either at the left or right hand side bars ("JSR").
JSR needs to know where some parts the project it should be ran on are located. The following
list explains the settings items (with example values for smallProject
):
Test Directory
is the root directory of your test suite.
For default Java projects this is usually similar to project/src/test/java
Example: smallProject/src/test/java
Source Directory
is the root source code directory of your project.
For default Java projects this is usually similar to project/src/main/java
Example: smallProject/src/main/java
Jar File
is the path to your fat jar. This jar must contain the compiled source and test classes.
Example: smallProject/build/libs/testJar.jar
Classes Directory
is the path to your compiled java class files (required by JaCoCo)
Example: smallProject/build/classes/java/main
Base Source Package
is the base package name of your project classes for which
line and method coverage should be calculated. You could use this as a filter if
you want to perform TSR based on only a part of your project's sources (i.e. a sub package).
This is an optional setting.
Example: at.tugraz
Slicer Directory
is the path to Slicer4J. This helps JSR to find the
correct path to the slicer which is necessary to calculate checked coverage.
Example: <jsrProjectRoot>/jsr/slicer/Slicer4J
Output Directory
is the path where all output files generated by JSR should be
written to. This directory will contain reports (coverage, Tsr), Exports (SFL) and
temporary files.
Example: smallProject/build/jsr/pluginTest01
Code Output Directory
is the path to the directory where the generated test
source code is written to. The generated sources are equal to the orginal test
suite sources with the exception of identified redundant test cases. Those
are deactivated via a JUnit annotation.
Example: smallProject/build/jsr/pluginTest01/gen
Under the TSR Options
tab, you can specify your TSR parameters.
They should be pretty self expanatory. A short description can be found in the following list:
Coverage Report Selection
determines which coverage report should be used for reduction:
New Report
will create a new coverage report with the selected coverage metricPrevious Report
will use a report that was generated in previous TSR runs.Reduction Algorithm
determines the employed algorithm in the reduction step
Greedy HGS
and a Genetic
algorithm.Deactivate redundant test cases
determines if test suite source code with redundant test casesAfter specifying the required settings and options parameters, you can start a TSR run by
clicking the Start TSR
button. Depending on whether a coverage report should be created,
this might take a while.
Once the TSR procedure is finished, two lists appear, one containing all relevant and thus retained test cases and the other one, containing all identified redundant test cases. Clicking on a test case in either list will trigger a navigation to the test case source code in the IntelliJ editor window(s).
The JSR command line interface can be used to leverage TSR core functionality via a command line tool. It supports the same settings, parameters and core functionality as the IntelliJ IDE plugin.
Additionally, the CLI supports Spectrum-based Fault Localization matrix export as a bonus feature for research purposes.
Usage instructions can be found by running
java -jar JSR-CLI-1.0-SNAPSHOT.jar jsr --help
This produces the following output:
Usage: jsr [-hV] [COMMAND]
-h, --help Show this help message and exit.
-V, --version Print version information and exit.
Commands:
reduce Reduces a test suite based on the given options and parameters.
sfl Creates Spectrum-based Fault Localization matrices.
By invoking the reduce
subcommand, the CLI can perform TSR:
java -jar JSR-CLI-1.0-SNAPSHOT.jar jsr reduce --help
This produces the following output:
Usage: jsr reduce [-hV] [--algorithm=<algorithm>] -c=<pathClasses> [--coverage=<coverageMetric>]
[--gen=<pathGenOut>] -j=<pathJar> -l=<pathSlicer> -o=<pathOut>
[--package=<basePackage>] [--report=<pathCoverageReport>] -s=<pathSources>
<testSourceDir>
Reduces a test suite based on the given options and parameters.
<testSourceDir> The root directory of the test suite sources
-h, --help Show this help message and exit.
-V, --version Print version information and exit.
Required Parameters:
-c, --classes=<pathClasses>
The path to the root directory containing the compiled source code
classes
-j, --jar=<pathJar> The path to the jar file containing the source and test classes
-l, --slicer=<pathSlicer>
The path to the Slicer4J directory
-o, --out=<pathOut> The path to the directory where all reports and output files are saved
-s, --sources=<pathSources>
The root directory of the main source code
Optional Parameters:
--algorithm=<algorithm>
The reduction algorithm used to reduce the test suite.
Available options: greedyHGS, genetic
Default: greedyHGS
--coverage=<coverageMetric>
The coverage metric calculated before the reduction when no coverage
report is specified. This option only has an effect if --report is not
specified
Available options: checked, line, method
Default: checked
--gen=<pathGenOut> The path to the directory where the modified test classes are generated
and saved
--package=<basePackage>
When specified, only classes under this package are instrumented for
line and method coverage calculation
--report=<pathCoverageReport>
Path to the coverage report which is used for the reduction
A sample call to the CLI for reducing the smallProject
test suite is provided below. The working
directory of the command line in this example is <jsrProjectRoot>/jsr
.
This example performs TSR with optional parameters: It creates a new line coverage report (--coverage
)
and employs the genetic reduction algorithm (--algorithm
) that ships with JSR-Core.
Additionally, test suite source code is generated (--gen
).
The --package
option is set to the base package of the source code classes
to slightly enhance JaCoCo instrumentation and test case execution performance
java -jar JSR-CLI-1.0-SNAPSHOT.jar jsr reduce
-s JSR-Core/src/test/resources/smallProject/src/main/java
-j JSR-Core/src/test/resources/smallProject/build/libs/testJar.jar
-l ../slicer/Slicer4J
-o JSR-CLI/build/jsr/cliTest01
-c JSR-Core/src/test/resources/smallProject/build/classes/java/main
--gen JSR-CLI/build/jsr/cliTest01/gen
--package at.tugraz.ist.stracke.jsr
--coverage line
--algorithm genetic
JSR-Core/src/test/resources/smallProject/src/test/java
The CLI can generate Spectrum-based Fault Localization matrices by invoking the sfl
subcommand:
java -jar JSR-CLI-1.0-SNAPSHOT.jar jsr sfl --help
This produces the following output:
Usage: jsr sfl [-hV] -c=<pathClasses> -j=<pathJar> -l=<pathSlicer> -o=<pathOut>
[--package=<basePackage>] -s=<pathSources> <testSourceDir>
Creates Spectrum-based Fault Localization matrices.
<testSourceDir> The root directory of the test suite sources
-h, --help Show this help message and exit.
-V, --version Print version information and exit.
Required Parameters:
-c, --classes=<pathClasses>
The path to the root directory containing the compiled source code classes
-j, --jar=<pathJar> The path to the jar file containing the source and test classes
-l, --slicer=<pathSlicer>
The path to the Slicer4J directory
-o, --out=<pathOut> The path to the directory where the SFL matrices are written to
-s, --sources=<pathSources>
The root directory of the main source code
Optional Parameters:
--package=<basePackage>
When specified, only classes under this package are instrumented for line
and method coverage calculation
A sample SFL call is provided below. The sfl
commands generates two CSV files:
outcomeMatrix.csv
containing information about the outcome (pass/fail) of each executed test case.coverageMatrix.csv
containing the coverage data of each test case:
java -jar JSR-CLI-1.0-SNAPSHOT.jar jsr sfl
-s JSR-Core/src/test/resources/smallProject/src/main/java
-j JSR-Core/src/test/resources/smallProject/build/libs/testJar.jar
-l ../slicer/Slicer4J
-o JSR-CLI/build/jsr/cliTest01
-c JSR-Core/src/test/resources/smallProject/build/classes/java/main
--package at.tugraz
JSR-Core/src/test/resources/smallProject/src/test/java
The JSR project uses the following libraries and tools: