What's Changed

New features

• AWS Lambda - third-party plugins and engines are now supported when running tests on AWS Lambda (#1812)
• New notStatusCode check in expect plugin (#1793)
• Datadog integration: includeOnly and excluded options to configure the list of Artillery metrics which are sent to Datadog (https://github.com/artilleryio/artillery/commit/a879b4fdba86f5e8dd28821d9cf3bed438a98272)

Fixes & improvements

• When artillery run is used with the --output option, Artillery will check if destination exists before running the test
• Miscellaneous dependency updates to address deprecation warnings and security advisories

New Contributors

• @InesNi made their first contribution in https://github.com/artilleryio/artillery/pull/1793
• @anoopknayak made their first contribution in https://github.com/artilleryio/artillery/pull/1801

as well as contributions by @juanig1 @hassy

Fixes & Improvements

• Fix installation issue due to mis-specified dependency version (#1777)

New features

• Add support for publishing test metrics to AWS CloudWatch
• HTTP - compressed responses (e.g. gzip) will be decompressed by default now. (Set gzip: false to disable automatic decompression if required to improve performance.)
• CLI - suggest closest match if a command name is mistyped

AWS Lambda

• Print custom role ARN if provided
• Add versioning to default Lambda role and policy created for running tests

Fixes & improvements

• Fix arrival phases with arrivalRate and rampTo set to 0
• Improve rampTo logic to result in more predictable and deterministic ramps
• Fix issue with maxVusers setting that would lead to unexpected number of VUs to be generated
• Fix wildcard message handling support in Socket.io
• Fix issue with setting some properties on the global artillery object

AWS Lambda

• Improve test startup logic: workers will wait for each other to be ready before kicking off the test run
• Fix some intermediate metrics not being flushed & reported
• Fix handling of memory-size and security-group-ids options

Breaking changes

• AWS Lambda: the default IAM role and policy are now versioned. If you attached any custom permissions to the artilleryio-default-lambda-role role, you will need to re-attach it to the new artilleryio-default-lambda-role-20230116 role
• CLI: The probe command has been removed from the CLI. The functionality has been moved to the new Skytrace CLI

New features

• AWS Graviton is now used by default when running tests on AWS Lambda. Using Graviton offers both cost savings and performance boost for load tests running on AWS Lambda. Legacy x86 architecture may be enabled by setting --platform-opt architecture=x86_64 (#1662)

Fixes and improvements

• Fixed an issue where AWS Lambda bootstrap time would be counted and reported as the time VUs ran for (https://github.com/artilleryio/artillery/commit/07e0e06214eb7ba74dc859ff1a589b6031fc0c02)
• Fix an issue where tests where reporting for test with many workers (50+) would be delayed (https://github.com/artilleryio/artillery/commit/06aa19c840f40781899f93bdd05786ea6faa6bca)

New features

• Artillery will now print the contents of the test bundle created for tests running on AWS Lambda
  

Fixes and improvements

• Fix issue with installing dependencies when running on AWS Lambda
• Fix issue with printing error logs from AWS Lambda workers

New features

AWS Lambda support

• Test scripts using custom JS code (in config.processor), CSV files, and external config files (used with the --config flag) can now run on Lambda
• Cost estimations for each test are now reported at the end of each test run
  
• nodejs16.x runtime will be used by the Lambdas

Fixes and improvements

• Always resolve path to processor relative to the config file when a separate config file is used via the --config flag
• Better handling of errors in worker Lambdas

New features

• Cookie parsing behavior may be customized via config.http.cookieJarOptions by passing options to the underlying ToughCookie instance (docs)

Fixes & Improvements

• More predictable rampTo behavior when using smaller values of arrivalRate/rampTo with a large duration (#1621)
• maxVusers setting is applied correctly when running on multiple CPU cores
• AWS Lambda functions will always use the latest release of Artillery

Contributors: @juanig1 @hassy

New features

• Add cdnHit expectation to expect plugin to check that the response was served from cache by a CDN
• Add support for providing a CA certificate to use with Prometheus Pushgateway in publish-metrics plugin

Fixes & Improvements

• Handle malformed cookies from HTTP services (#1540)
• Fix crashes that could be caused by redirect chains in combination with capture/afterResponse hooks (#1544 #1568)
• Make sure response body is available to afterResponse hooks when the response is compressed (#1563)
• Add support for JSON-LD in the expect plugin (https://github.com/artilleryio/artillery-plugin-expect/pull/34)
• Make sure metrics sent to Prometheus Pushgateway with publish-metrics are flushed before Artillery exits
• Routine dependency upgrades to address security advisories

Contributors: @juanig1, @luneo7, @maximetaqt, @hassy

Fixes & Improvements

• Improve error handling on AWS Lambda
• Fix issue which could cause errors when handling HTTP redirects
• Fix issue with cookies not being saved and reused under some conditions
• Various maintenance & security dependency updates

New features

This release adds support for distributed cloud-native testing with AWS Lambda out of the box.

This feature is to be considered experimental at the moment. Please report bugs in Issues.

The --platform and --platform-opt flags have been added to the run command.

To run a test on AWS Lambda in the us-east-1 region, with 100 parallel Lambda workers, run:

artillery run --platform aws:lambda --platform-opt region=us-east-1 my-script.yml

This requires an AWS profile configured locally.

Fixes & Improvements

• Allow think, count and timeout properties to be numbers or strings
• Allow zero duration think steps
• Dependency updates (async, got, socket.io-client, oclif, etc)

New

• Add support for loading environment variables from a dotenv file with --dotenv (#1478)
• Add loadAll option to make all CSV data available to each VU (#1485)
• New & improved HTML reports

• Fix: retain histograms field in JSON reports for backwards-compatibility
• Replace deep-equal with fast-deep-equal (ty @toastal)

• Node.js v16+ is now recommended
• Upgrade Docker image to Node v16 LTS
• Fix a bug with rampTo under certain conditions (#1443) (@GeekyDeaks)
• Include artillery-plugin-ensure (#1457)

artillery probe

New

• Add support for querying HTML/XML with Cheerio

Fixes

• Always pretty-print JSON if -q / --jmespath

• Add experimental probe/http command for sending HTTP requests from the command-line
• Add wait action to WebSocket engine (@GeekyDeaks)
• Add basic validation for test scripts in run (#1410, @flentini)
• Fix issue with aggregating summary metrics (#1425, @juanig1)
• Improve test suite (#1391, ty @chringwer)

This is a maintenance release with dependency upgrades and minor bug fixes.

• Fix issue with intermediate metrics not being aggregated for short test runs (#1385)
• Upgrade dependencies to address security advisories

This is a maintenance release with dependency upgrades and minor bug fixes.

• Fix issue with phase name sometimes not being printed (#1313)
• Add support for expectations on multiple headers with the same name (#1354)
• publish-metrics, expect and metrics-by-endpoint are now included as dependencies of Artillery itself

This is a maintenance release with dependency upgrades and minor bug fixes.

The Docker image now bundles the following official plugins:

• publish-metrics
• expect
• metrics-by-endpoint

New:

• Display metrics in alphabetical order in console output
• Display environment details in artillery version output
• Track and display vusers.failed through the entire test (report zero when no VUs have failed)

Maintenance:

• Update Docker image to use Node.js 14

Fixes:

• Fix issue with arrivalCount arrivals to not being spaced out as expected
• Fix issue with --output in run command when destination is a directory
• Update HTML report to remove distinction between core and custom metrics