Multiprocess | Parallel Cucumber-JVM | Parallelize your Java Cucumber tests on a feature level or on a scenario level.
MIT License
Courgette-JVM is an extension of Cucumber-JVM with added capabilities to run cucumber tests in parallel on a feature level or on a scenario level. It also provides an option to automatically re-run failed scenarios.
<dependency>
<groupId>io.github.prashant-ramcharan</groupId>
<artifactId>courgette-jvm</artifactId>
<version>6.17.0</version>
</dependency>
implementation group: 'io.github.prashant-ramcharan', name: 'courgette-jvm', version: '6.17.0'
Example projects:
Courgette-JVM supports JUnit and TestNG to run cucumber features and scenarios in parallel. A JUnit runner class must be annotated with @RunWith(Courgette.class) and a TestNG runner class must extend TestNGCourgette.
threads : The number of concurrent threads to run cucumber features where each thread is started in an independant JVM process.
runLevel : Options are CourgetteRunLevel.FEATURE or CourgetteRunLevel.SCENARIO
rerunFailedScenarios : If set to true, any failed scenario will be immediately re-run in the same thread. If the re-run succeeds, the initial failure will be ignored and not cause the build to fail.
excludeFeatureFromRerun : If set, Courgette will prevent features from re-running when it fails.
excludeFeatureFromRerun = {Feature1.feature}
and Feature1.feature
and Feature2.feature
both fail, Courgette will not re-run Feature1.feature
but will re-run Feature2.feature
. This is useful when you need to prevent a specific feature or scenario from re-running.excludeTagFromRerun : If set, Courgette will prevent features and scenarios from re-running on failure when it finds a matching Cucumber tag.
excludeTagFromRerun = {@stable}
and a test tagged with @stable
fails, Courgette will not re-run that specific test but will continue to re-run other failing tests.rerunAttempts : The number of re-run attempts for a failed scenario. (rerunFailedScenarios must be set to true)
testOutput : Redirects the output for each parallel test run.
${reportTargetDir}/courgette-test-ouput
reportTargetDir : Target directory where courgette-report is generated. Set to target by default.
generateCourgetteRunLog : If set to true, Courgette will generate a run log and save it to ${reportTargetDir}/courgette-run-{courgette_session_id}.json
at the end of the test run.
plugin : Courgette supported plugins
environmentInfo : Additional environment information that is displayed in the Courgette html report.
;
character and adhere to the following format: key1=value1; key2=value2
.disableHtmlReport: If set, the Courgette and Cucumber html reports will not be generated at the end of the test run.
HtmlReport.COURGETTE_HTML
, HtmlReport.CUCUMBER_HTML
and HtmlReport.COURGETTE_AND_CUCUMBER_HTML
persistParallelCucumberJsonReports: If set to true, Courgette will save the Cucumber json and ndjson reports for each parallel test to ${reportTargetDir}/session-reports/{session}
classPath: Allows a custom class path to be used when running tests.
{ "path-to-project-jars", "path-to-test-classes" }
slackWebhookUrl: The incoming webhook URL that Courgette uses to send messages to Slack.
slackChannel: The Slack channels that Courgette will post messages to.
slackTestId: A custom Slack identifier that will be sent with each message.
slackEventSubscription: The Courgette events to subscribe to that gets posted to Slack.
mobileDevice: The devices that Courgette will use to track and allocate for parallel mobile tests.
CourgettePlugin.MOBILE_DEVICE_ALLOCATOR
plugin.mobileDeviceType: The mobile device types used for device allocation. This can be one of:
deviceName:deviceUUID
realMobileDeviceTag: If set, Courgette will allocate a real mobile device for tests matching any one of the provided tags. To use this option, you must also specify mobileDeviceType as MobileDeviceType.REAL_DEVICE
or MobileDeviceType.SIMULATOR_AND_REAL_DEVICE
fixedThreadDelay: A fixed time in milliseconds that Courgette will pause before the start of each feature or scenario.
randomThreadDelay: A random time in milliseconds that Courgette will pause before the start of each feature or scenario. Courgette will automatically set a random time between 0 and this value.
cucumberOptions : The standard cucumber options for specifying feature paths, glue, tags etc..
publish
cucumber option (supported from version 5.1.0) will publish a single cucumber report after parallel execution.
${reportTargetDir}/cucumber-report-link.txt
.CUCUMBER_PUBLISH_TOKEN
via a system property or an environment variable. You can get your token from https://reports.cucumber.io/profile
At the end of the test run, a single report ( if included in the cucumberOptions ) listing all executed features and scenarios will be created in the specified report path.
A courgette-rerun.txt file listing all failed scenarios will be created in the specified rerun plugin path or the target folder ( default )
A Courgette-JVM html report will be created in the reportTargetDir (defaulted to the target directory).
@RunWith(Courgette.class)
@CourgetteOptions(
threads = 10,
runLevel = CourgetteRunLevel.SCENARIO,
rerunFailedScenarios = true,
rerunAttempts = 1,
testOutput = CourgetteTestOutput.CONSOLE,
reportTitle = "Courgette-JVM Example",
reportTargetDir = "build",
environmentInfo = "browser=chrome; git_branch=master",
cucumberOptions = @CucumberOptions(
features = "src/test/resources/features",
glue = "steps",
tags = "@regression and not @bug",
publish = true,
plugin = {
"pretty",
"json:build/cucumber-report/cucumber.json",
"html:build/cucumber-report/cucumber.html",
"junit:build/cucumber-report/cucumber.xml"}
))
public class RegressionTestSuite {
}
@Test
@CourgetteOptions(
threads = 10,
runLevel = CourgetteRunLevel.SCENARIO,
rerunFailedScenarios = true,
rerunAttempts = 1,
testOutput = CourgetteTestOutput.CONSOLE,
reportTitle = "Courgette-JVM Example",
reportTargetDir = "build",
environmentInfo = "browser=chrome; git_branch=master",
cucumberOptions = @CucumberOptions(
features = "src/test/resources/features",
glue = "steps",
tags = "@regression and not @bug",
publish = true,
plugin = {
"pretty",
"json:build/cucumber-report/cucumber.json",
"html:build/cucumber-report/cucumber.html"}
))
public class RegressionTestSuite extends TestNGCourgette {
}
tasks.withType(Test) {
systemProperties = System.getProperties()
}
// JUnit
task regressionSuite(type: Test) {
include '**/RegressionTestSuite.class'
outputs.upToDateWhen { false }
}
// TestNG
task regressionSuite(type: Test) {
useTestNG()
include '**/RegressionTestSuite.class'
outputs.upToDateWhen { false }
}
To override the hard-coded courgette options (threads, runLevel, rerunFailedScenarios, reportTargetDir, environmentInfo) set in the runner class, you can provide system properties to the gradle or maven task.
[gradle | mvn] test -Dcourgette.threads=2 -Dcourgette.runLevel=FEATURE -Dcourgette.rerunFailedScenarios=false -Dcourgette.reportTargetDir=build -Dcourgette.environmentInfo="git_branch=master; platform=ci"
To override the hard-coded cucumber options (tags, glue, plugin, name, junit) set in the runner class, you can provide comma separated system properties to the gradle task.
[gradle | mvn] test -Dcucumber.tags="@regression and not @bug" -Dcucumber.glue="steps, hooks"
To specify non standard VM options (-X options)
[gradle | mvn] test -Dcourgette.vmoptions="-Xms256m -Xmx512m"
You can add global setup and tear-down code to your Courgette test runner using the @CourgetteBeforeAll
and @CourgetteAfterAll
annotations. For example:
@RunWith(Courgette.class)
@CourgetteOptions(/* Your Courgette options here... */)
public class RegressionTestSuite {
@CourgetteBeforeAll
public static void setUp() {
System.out.println("I will run before any tests execute");
}
@CourgetteAfterAll
public static void tearDown() {
System.out.println("I will run after all of the tests execute");
}
}
You can add any number of annotated methods to your test suite class.
If you need your callbacks to run in a specific order, pass order
to the annotation: @CourgetteBeforeAll(order = 2)
.
You can access test statistics and additional run information if you need to analyze or perform extra tasks before or after the parallel test run.
Note: CourgetteRunInfo
can only be accessed from a Courgette runner class.
JUnit Runner
@RunWith(Courgette.class)
@CourgetteOptions(/* Your Courgette options here... */)
public class RegressionTestSuite {
@CourgetteBeforeAll
public static void beforeRun() {
System.out.println("Starting Courgette parallel run: " + CourgetteRunInfo.sessionId());
}
@CourgetteAfterAll
public static void afterRun() {
if (CourgetteRunInfo.testStatistics().hasFailures()) {
// do something extra here
}
}
}
TestNG Runner
@Test
@CourgetteOptions(/* Your Courgette options here... */)
public class RegressionTestSuite extends TestNGCourgette {
@BeforeTest
public static void beforeRun() {
System.out.println("Starting Courgette parallel run: " + CourgetteRunInfo.sessionId());
}
@AfterTest
public static void afterRun() {
if (CourgetteRunInfo.testStatistics().hasFailures()) {
// do something extra here
}
}
}
Retrieve the Courgette Thread ID and Name
System.getProperty("courgette.threadId")
System.getProperty("courgette.threadName")
Courgette allows real time test results and events to be posted to Slack as tests are run.
To enable this feature, add the following Courgette options to the Courgette runner:
@CourgetteOptions(
...
slackWebhookUrl = "https://hooks.slack.com/services/your-slack-url",
slackChannel = {"channel1", "channel2"},
slackTestId = "Production test - Build 1.0.0",
slackEventSubscription = {CourgetteEvent.ALL},
cucumberOptions = @CucumberOptions(
// cucumber options here
)
)
You need to create an incoming webhook URL to allow Courgette to post messages to your Slack application.
https://api.slack.com/messaging/webhooks#create_a_webhook
You can subscribe to single or multiple Courgette events. When events are triggered as the tests run, Courgette will post a message to the Slack channels defined in the runner.
Courgette allows test results to be published in real time to the Report Portal server as tests run.
To enable this feature, add the following Courgette option to the Courgette runner:
@CourgetteOptions(
...
plugin = { CourgettePlugin.REPORT_PORTAL },
cucumberOptions = @CucumberOptions(
// cucumber options here
)
)
You must have the reportportal.properties file in your classpath and the following properties must be defined:
# Report Portal server (mandatory)
rp.endpoint = http://localhost:8080
# Report Portal project (mandatory)
rp.project = courgette_example
# Report Portal API access token (mandatory)
rp.apitoken = a1e5ee78-317c-477d-b27e-f174c562aedc
# Report Portal launch name (optional)
rp.launch = My Demo Project
# Report Portal test suite (optional)
rp.testsuite = Regression Test Suite
# Report Portal launch attributes (optional)
# Each attribute must be separated by ';'
rp.attributes = suite:regression;build:12345
Note: Any property other than those defined above will be ignored by Courgette.
An API access token is required to allow Courgette to publish the report. To obtain an API access token, log in to Report Portal UI and navigate to http://localhost:8080/ui/#api -> UAT -> sso-endpoint -> Get api token
After the test run is complete, the test results will be published to the Report Portal server.
Courgette allows the creation of interactive reports using the Extent Reports Courgette plugin.
To enable this feature, add the following Courgette option to the Courgette runner:
@CourgetteOptions(
...
plugin = { CourgettePlugin.EXTENT_REPORTS },
cucumberOptions = @CucumberOptions(
// cucumber options here
)
)
At the end of the test run the report will be saved to ${reportTargetDir}/courgette-extentreports
To configure custom reports (i.e. change the report name or theme) you should create the extent-config.xml
file in the classpath. Courgette will load this XML config when it builds the report. If this file is not provided then default values will be used. View an example here
Courgette allows the generation of Allure reports using the Allure Cucumber plugin.
@CourgetteOptions(
...
cucumberOptions = @CucumberOptions(
plugin = {
"io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm"
}
)
)
Courgette provides a mobile device allocator to allocate and keep track of devices for parallel mobile testing. Courgette keeps track of devices that are currently in use and automatically allocates a free device for each test.
@CourgetteOptions(
...
plugin = { CourgettePlugin.MOBILE_DEVICE_ALLOCATOR },
mobileDeviceType = MobileDeviceType.SIMULATOR,
mobileDevice = {
"iPhone 8",
"iPhone 12",
"iPhone 13"
},
cucumberOptions = @CucumberOptions(
// cucumber options here
)
)
The Courgette mobile device allocator plugin will:
mobileDevice
and will automatically allocate a randomly selected available device for each parallel test.mobileDevice
. The sum of mobileDevice
will take precedence over threads
defined in the Courgette runner.SIMULATOR_AND_REAL_DEVICE
then Courgette will allocate a real device for tests tagged with any matching tag defined in Courgette option realMobileDeviceTag
and allocate a simulator for all other tests.Notes:
mobileDevice
must be unique unless using real devices where a UUID
is also required.CourgetteMobileDeviceAllocator.DEVICE_NAME
returns one of the available devices from the mobileDevice
list.CourgetteMobileDeviceAllocator.UDID
returns the UUID for the device (only required for real devices).CourgetteMobileDeviceAllocator.PARALLEL_PORT
returns a free local port (required for parallel device testing).The above properties are only available when running tests using a Courgette runner with the CourgettePlugin.MOBILE_DEVICE_ALLOCATOR
plugin.
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability("deviceName", CourgetteMobileDeviceAllocator.DEVICE_NAME);
capabilities.setCapability("udid", CourgetteMobileDeviceAllocator.UDID);
capabilities.setCapability("wdaLocalPort", CourgetteMobileDeviceAllocator.PARALLEL_PORT);
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability("avd", CourgetteMobileDeviceAllocator.DEVICE_NAME);
capabilities.setCapability("udid", CourgetteMobileDeviceAllocator.UDID);
capabilities.setCapability("systemPort", CourgetteMobileDeviceAllocator.PARALLEL_PORT);
MutableCapabilities capabilities = new MutableCapabilities();
capabilities.setCapability("appium:deviceName", CourgetteMobileDeviceAllocator.DEVICE_NAME);
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability("device", CourgetteMobileDeviceAllocator.DEVICE_NAME);
[mvn | gradle] test -Dcourgette.mobileDevice="iPhone X, iPhone 12, iPhone 13"
Unlike simulators where the device name must be unique, on real devices you can specify the same device name as long as the UUIDs are different for each device.
Format: deviceName:deviceUUID
mobileDevice = {
"iPhone 8:00000000-000-0000-0000-000000000001",
"iPhone 8:00000000-000-0000-0000-000000000002",
"iPhone 8:00000000-000-0000-0000-000000000003"
}
For any issues or requests, please submit here