Artillery v2.0.14

This is a hotfix release to fix an issue introduced in Artillery v2.0.13 which would lead to excessive logging when running Playwright tests.

Artillery v2.0.13

AWS Lambda

Lambda Container Images

Artillery will now always use Lambda Container Images to run tests on AWS Lambda. The --container flag introduced in v2.0.12 is no longer required and will be removed in the next release. Creating Lambda functions from .zip archives is no longer supported.

All existing tests should continue running normally without needing any changes. If you run into any issues please open an issue.

This change brings improved startup times for tests running on AWS Lambda and brings AWS Lambda support to parity with AWS Fargate.

Fixes

• Increase startup timeout time for Lambdas running in a VPC (#2778)
• Exit with non-zero code if a Lambda worker fails (#2766)
• Fix issue that could lead tests to crash when triggered from a Windows machine (#2767)
• Expose current worker ID to Lambda workers as WORKED_ID environment variable (#2754)

Playwright

• Accept 0 values for defaultNavigationTimeout and defaultTimeout settings (#2786)
• A warning will be printed if a Playwright testFunction cannot be found (#2782)

Monitoring & observability (publish-metrics plugin)

• Add url attribute to page spans when tracing is enabled for Playwright tests (#2779)
• Fix issue which could cause Core Web Vitals to not be set as span attributes (#2779)
• A test_id attribute is no longer set on all metrics. The built-in $testId variable may be used to set that attribute when needed (#2779)
• Request timing phases such as DNS lookups and SSL handshake timings are now recorded as attributes instead of spans for HTTP tests (#2779)

Slack integration

Fix a number of issues that prevented the Slack plugin from working with tests running on AWS Fargate and AWS Lambda (#2754)

WebSocket

• Errors thrown by hook functions will now be tracked and reported (#2783)

---

With contributions from: @bbeesley @HugoImaios @InesNi @bernardobridge @hassy

Artillery v2.0.12

AWS Lambda

Artillery can now use Lambda Container Images to run tests on AWS Lambda. Pass the --container flag to run-lambda command to use a container instead of a zip file. No further configuration is required.

This change brings two main benefits:

1. Much faster startup times for tests
2. Ability to use certain features that were previously only available in tests running on AWS Fargate:

• Run test scripts that TypeScript hooks
• Use third-party dependencies
• Use the built-in publish-metrics plugin

Container-based Lambda functions will become the default option in a future release of Artillery.

• #2674 • #2724

Playwright

• Update Playwright to v1.44.0
• Fix tests failing on Fargate with ENOSPC error with Playwright tracing enabled #2717
• Improve reliability of trace collection:
  • Reduce trace collection cooldown period to 1-5m
  • Increase the number of concurrent trace recordings to 3 up from 1
• Track the number of collected/discarded recordings with browser.traces_collected and browser.traces_discarded metrics

CLI & Core

• Make path to the config directory available to test scripts via the special $dirname variable #2614 docs
• The --scenario-name flag will now default to using an exact string match, and fall back to using a regex #2709
• Clarify the meaning of --tags flag #2699
• Remove an unnecessary dependency #2703

Slack integration

• Artillery now includes an official Slack integration, which can be used to post test results to a Slack channel automatically #2688 docs

---

With contributions from: @bernardobridge @InesNi @hassy @43081j @MIJOTHY-V2

Artillery v2.0.11

Playwright

• Display errors from the Playwright engine in Artillery reports and the console (#2677) - docs
• Report errors from failed Playwright assertions (#2677) - docs

Fargate

• Fix flag allowing custom IAM role in Fargate tasks (--task-role-name ) (#2669) - docs

CLI & Core

• Log an error message when using an unsupported expectation with the expect plugin (#2667)

---

This release brought to you by @InesNi

Artillery v2.0.10

CLI & Core

• Add parallel action for making parallel requests in HTTP tests

Fargate

• Add ability to suppress metrics output with --quiet flag (#2639) - docs
• Add eu-north-1 to supported regions (#2646)

Lambda

• Add ability to suppress metrics output with --quiet flag (#2639) - docs

Playwright

• Fix issue preventing stack traces for errors from being displayed (#2643)
• Update Playwright version to 1.43.1 (#2644) - docs

Publish Metrics

• Make reporters using OpenTelemetry compatible with parallel requests (#2628)

---

This release brought to you by @hassy @InesNi @bernardobridge

Artillery v2.0.9

Publish Metrics

• Sending traces to AWS X-Ray is now available through the CloudWatch reporter (#2555) - Docs
• Allow replacing of all tracing span names with replaceSpanNameRegex config option (#2607) - Docs
• Fix bug in Datadog reporter causing last batches of traces to occasionally drop (#2610)
• Fix errors happening when sampleRate tracing option set (#2601)
• Fix errors in tracing happening when using HTTP streaming (#2601)
• Display number of spans exported in the Artillery report (plugins.publish-metrics.spans.exported ) when tracing is enabled. (#2597)

CLI & Core

• Fix incorrectly declared dependencies in Artillery CLI (#2549)

---

This release brought to you by @InesNi @95th

Artillery v2.0.8

Artillery Cloud

• Fix issue preventing ARTILLERY_CLOUD_API_KEY environment variable from being picked up by tests on Fargate when using --dotenv (#2585)

Metrics by Endpoint

• Report request URLs for requests that resulted in a network error such as ETIMEOUT or ECONNRESET (#2574) - docs

CLI and Core

• Fix issue causing some intermediate reports to not be reported (#2565)
• Fix issue with incorrect formatting of time units in “phase started” messages (#2581)

---

This release brought to you by @hassy @bernardobridge

Artillery v2.0.7

CLI and Core

• Add support for loading custom code from ES modules (#2522) - docs
• Add support for using async functions as hooks (#2522) - docs

Playwright

• Update Playwright version to 1.42.1
• Fix bug that prevented traces from Playwright tests from being sent to Datadog and Honeycomb (#2533)

AWS Fargate

• Tests running in Fargate will now use AWS Distro for OpenTelemetry instead of Datadog Agent for sending traces to Datadog (#2481) - docs
• Fix bug that prevented setting CPU and memory allocation with the --launch-config flag (#2526)

---

This release brought to you by @hassy @bernardobridge @InesNi

Artillery v2.0.6

Playwright

• Make expect from @playwright/test work with Typescript support (#2488) - Documentation
• Playwright’s baseURL is now set from target (#2493) - Documentation
• Update Playwright version to 1.41.2 (#2492) - Documentation

Artillery Cloud

• Load phases are now shown in Artillery Dashboard for Fargate tests (#2485)

CLI & Core

• Improve Typescript support to work with more packages, by exposing config.bundling.external to set packages that can’t be bundled as external (#2488) - Documentation
• Make test run id available to test runs via {{ $testId }} in scenarios/config and vuContext.vars.$testId in custom code (#2499)
• Routine dependency updates (#2500)

Publish Metrics

• Add test_id attribute to metrics and traces sent via the OpenTelemetry reporter (#2505) - Documentation

Fargate

• Improve error message and handling when setting CloudWatch retention policy (#2489)

---

This release brought to you by @bernardobridge, @InesNi, @hassy

Artillery v2.0.5

New release cadence

A new stable version of the Artillery CLI will now be released twice a month - on the first and third Monday of the month. Canary releases will continue being released from the main branch.

AWS Fargate

• Add support for running tests with Fargate Spot (#2464) - Documentation
• Set default bucket lifecycle policy for the S3 bucket containing artifacts from test runs (#2434) - Documentation
• Fix issue with traces sent via the OpenTelemetry reporter causing spans to go missing or be formatted incorrectly (#2439)
• Set default retention policy on CloudWatch logs (#2446) - Documentation
• Force headless mode for Playwright tests running on Fargate (#2455)
• Fix bug causing Fargate tests triggered on Windows to crash when running with custom JS code (#2431)
• Fix bug that would cause Fargate workers to crash on startup (#2428, #2459)

Playwright

• Track and report the INP Core Web Vitals metrics (#2461) - Documentation
• Add support for configuring Playwright’s testIdAttribute setting (#2441) - Documentation
• Update Playwright version used by Artillery to v1.41.0 (#2425) - Documentation

publish-metrics plugin

• Add tracing support for the Datadog, New Relic and Dynatrace reporters using OpenTelemetry (#2418, #2466)
• Improve traces sent to Honeycomb to include full metadata, exceptions, and events (#2418, #2466)

AWS Lambda

• Disable artillery-plugin-publish-metrics on Lambda (#2457). This is a temporary workaround to get tests to run on Lambda reliably again
• Add default bucket lifecycle policy for S3 bucket containing artifacts from test runs (#2434) - Documentation

CLI & Core

• Fix $randomString(length) utility to return correct number of characters and allow length larger than 10 (#2342)

Artillery Cloud

• Fix bug that prevented CloudWatch logs for all workers from being shown on test reports (#2421)

---

This release brought to you by @InesNi, @hassy, @bernardobridge, @Georgegriff