Bug Fixes

• Docker: fix base image dependency inference for parametrized targets. (Cherry-pick of #20633) (#20657)

New Features

• Allow fromfile config value to be optional (#20571)

User API Changes

• Upgrade Ruff to v0.3.0 (#20647)

Bug Fixes

• Docker: fix base image dependency inference for parametrized targets. (#20633)

No changes since 2.18.3rc0.

Changes since 2.18.2:

Bug Fixes

• python-infer: Avoid false positive strings (Cherry-pick of #20472) (#20599)

• python-infer: respect ignore pragma w/ strings (Cherry-pick of #20477) (#20601)

• Fix direct_url in python PEP660 editable wheels (Cherry-pick of #20486) (#20602)

• Remove asm -compiling-runtime flag for go 1.22+ compatible versions (Cherry-pick of #20554) (#20563)

Documentation

• Document the need for migrating tool to user lockfiles for export (Cherry-pick of #20604) (#20615)

• Fix capitalization of Terraform in left nav (#20489)

No changes since 2.19.1rc1.

Changes since 2.19.0:

Bug Fixes

• Fix formatting of the "pex.platforms is deprecated" message (Cherry-pick of #20514) (#20527)

• Bump PyO3 to 0.20 to fix backtraces (Cherry-pick of #20517) (#20532)

• Fix direct_url in python PEP660 editable wheels (Cherry-pick of #20486) (#20494)

• python-infer: respect ignore pragma w/ strings (Cherry-pick of #20477) (#20484)

• python-infer: Avoid false positive strings (Cherry-pick of #20472) (#20483)

• fix: in-repo plugin requirements.txt not loading (Cherry-pick of #20355) (#20381)

• Remove asm -compiling-runtime flag for go 1.22+ compatible versions (Cherry-pick of #20554) (#20561)

Documentation

• Fix capitalization of Terraform in left nav (#20490)

• Actually call bin_name in help string (Cherry-pick of #20434) (#20437)

• Document the need for migrating tool to user lockfiles for export (Cherry-pick of #20604) (#20614)

New Features

• allow passing extra docker run args to docker_image (#20607)

• Add support for Scalafix (#20394)

• run terraform plan when setting --dry-run flag on experimental-deploy goal (#20488)

User API Changes

• Remove now-expired deprecated python_awslambda alias (#20619)

Bug Fixes

• Resolve adhoc_tool, code_quality_tool execution dependencies relative to target location (#20581)

• Silence warnings from Pex by default, for now (#20590)

• javascript: package yarn v2 (#20540)

• Fix update build files formatter selection (#20580)

• Remove asm -compiling-runtime flag for go 1.22+ compatible versions (#20554)

• Add all new Ruff backends to list of plugins (#20555)

Documentation

• Document the need for migrating tool to user lockfiles for export (#20604)

• Update doc_url calls for new website (#20583)

• docs: export dependency graph as adjacency list (#20566)

• Restructure JVM docs. (#20544)

Bug Fixes

• Remove asm -compiling-runtime flag for go 1.22+ compatible versions (Cherry-pick of #20554) (#20561)

Documentation

• Document the need for migrating tool to user lockfiles for export (Cherry-pick of #20604) (#20614)

Bug Fixes

• Resolve adhoc_tool, code_quality_tool execution dependencies relative to target location (Cherry-pick of #20581) (#20608)

• Silence warnings from Pex by default, for now (Cherry-pick of #20590) (#20593)

• Fix update build files formatter selection (Cherry-pick of #20580) (#20582)

• Remove asm -compiling-runtime flag for go 1.22+ compatible versions (Cherry-pick of #20554) (#20562)

• Add all new Ruff backends to list of plugins (Cherry-pick of #20555) (#20556)

Documentation

• Document the need for migrating tool to user lockfiles for export (Cherry-pick of #20604) (#20613)

• Update doc_url calls for new website (Cherry-pick of #20583) (#20588)

• docs: export dependency graph as adjacency list (Cherry-pick of #20566) (#20585)

• Restructure JVM docs. (Cherry-pick of #20544) (#20564)

Bug Fixes

• python-infer: Avoid false positive strings (Cherry-pick of #20472) (#20599)

• python-infer: respect ignore pragma w/ strings (Cherry-pick of #20477) (#20601)

• Fix direct_url in python PEP660 editable wheels (Cherry-pick of #20486) (#20602)

• Remove asm -compiling-runtime flag for go 1.22+ compatible versions (Cherry-pick of #20554) (#20563)

Documentation

• Document the need for migrating tool to user lockfiles for export (Cherry-pick of #20604) (#20615)

• Fix capitalization of Terraform in left nav (#20489)

Bug Fixes

• Fix formatting of the "pex.platforms is deprecated" message (Cherry-pick of #20514) (#20527)

• Bump PyO3 to 0.20 to fix backtraces (Cherry-pick of #20517) (#20532)

• Fix direct_url in python PEP660 editable wheels (Cherry-pick of #20486) (#20494)

• python-infer: respect ignore pragma w/ strings (Cherry-pick of #20477) (#20484)

• python-infer: Avoid false positive strings (Cherry-pick of #20472) (#20483)

• fix: in-repo plugin requirements.txt not loading (Cherry-pick of #20355) (#20381)

Documentation

• Fix capitalization of Terraform in left nav (#20490)

• Actually call bin_name in help string (Cherry-pick of #20434) (#20437)

New Features

• add more module mappings for popular packages (#20551)

• Docker: Add full_directory interpolation value for repository configuration. (#20530)

• stats: add output_file option to output the stats to a file (#20512)

• python: respect closed option when exporting dependency graph as JSON (#20523)

• Generate jvm_artifact targets from pom.xml (#20336)

• Add support for the pex --executable argument (#20497)

• python: improve error message when parsing Python interpreter constraints (#20297)

• upgrade known terraform versions (#20469)

• Allow using Ruff to format BUILD files (#20411)

• options: do not ignore .github directory with pants_ignore (#20471)

• Add (optional) support for podman. (#20470)

• python: add _typeshed module to the list of unowned dependencies (#20468)

• Terraform lockfiles (take 2) (#20303)

• Extend dependents goal with output format to support JSON (#20453)

• Extend dependencies goal with output format to support JSON (#20443)

User API Changes

• Update built-in ruff 0.1.6 -> 0.2.1 (#20499)

Plugin API Changes

• Allow AbstractLintRequest subclasses to disable lint rules (#20407)

Bug Fixes

• stop swallowing warnings from Pex by default (#20480)

• javascript: fix running scripts with yarn (#20543)

• docker: fix missing image_id when using containerd-snapshotter (#20533)

• Bump PyO3 to 0.20 to fix backtraces (#20517)

• Fix formatting of the "pex.platforms is deprecated" message (#20514)

• Handle unresolved ambiguous entrypoint dependency for PEX as unowned dependency (#20390)

• plumb through Pex's --check zipapp validation (#20481)

• Allow unmatching "changed" globs (#20505)

• add module mapping overrides for some django-* modules (#20504)

• upgrade Pex to 2.1.163 (#20502)

• upgrade Pex to 2.1.162 (#20496)

• Fix direct_url in python PEP660 editable wheels (#20486)

• python-infer: respect ignore pragma w/ strings (#20477)

• Introduce ruff-check and ruff-format tool ids (#20358)

• python-infer: Avoid false positive strings (#20472)

• Don't eagerly merge configs. (#20459)

• Docker: Update hadolint version to fix segmentation fault issue (#20456)

Documentation

• Fix reference to default_run_goal_use_sandbox option in docs. (#20546)

• fix pants.log location in docs (#20547)

• docs: mention .pants.bootstrap file (#20538)

• docs: provide example of having a --option in [cli.alias] (#20539)

• docs: provide options to refer to a group of targets (#20522)

• docs: mention that when piping xargs may end up invoking Pants goal more than once (#20521)

• docs: provide example how to convert target addresses to source files in rules API (#20524)

• Fix Pex references to point to new home. (#20519)

• Update broken link to default versioning scheme (#20482)

• docs: fix slack channel references (#20475)

• Add a test case for Pants integration testing docs (#20451)

• Add a test case for Pants unit testing docs (#20452)

• Rename tutorials section in the docs (#20449)

2.19.x Release Series

Pants 2 is a fast, scalable, user-friendly build system for codebases of all sizes. It's currently focused on Python, Go, Java, Scala, Kotlin, Shell, and Docker, with support for other languages and frameworks coming soon.

Individuals and companies can now sponsor Pants financially.

Pants is an open-source project that is not owned or controlled by any one company or organization, and does incur some expenses. These expenses are managed by Pants Build, a non-profit that was established for this purpose. This non-profit's only source of revenue is sponsorship by individuals and companies that use Pants.

We offer formal sponsorship tiers for companies, as well as individual sponsorships via GitHub.

What's New

Highlights

• Test retries for flaky Python tests.
• Using the parametrize to set multiple fields at once when generating new groups of targets.
• buildx support in the Docker backend.
• New backends for running semgrep and openapi-format.

Keep reading to see the details and what's also included.

Overall

The .pids/ internal directory is now under .pants.d/ by default, rather that adjacent to it (this can be controlled via the [GLOBAL].pants_subprocessdir option).

The new [system-binaries] subsystem allows controlling where Pants searches for binaries like bash, zip, tar that is uses as part of normal operation. Set the system_binaries_paths as appropriate, potentially using the <PATH> special value. This can also be set on specific environments using the system_binaries_system_binary_paths field.

Additional features and fixes:

• New versions of the get-pants.sh script install to ~/.local/bin, not ~/bin.
• Environment variables that aren't valid UTF-8 no longer cause Pants to crash (note, PANTS_... environment variables that Pants needs to read but aren't UTF-8 will be ignored).
• Reference documentation for subsystem options includes how to set them in pants.toml, in addition to the environment variable and CLI arguments.
• Environments are now more resilient to being (partially) defined via macros, although this is not recommended.
• Docker environments can now pull public images without credentials, and can now handle docker images without Python.
• The fix goal now partitions files across multiple processes in a way that's more similar to the partitioning of the lint goal, which can work around some spurious linting failures.

BUILD files

The parametrize helper now supports parametrizing multiple fields together, in lock step. For instance:

# Creates two targets:
#
#    example:tests@parametrize=py2
#    example:tests@parametrize=py3

python_test(
    name="tests",
    source="tests.py",
    **parametrize("py2", interpreter_constraints=["==2.7.*"], resolve="lock-a"),
    **parametrize("py3", interpreter_constraints=[">=3.6,<3.7"], resolve="lock-b"),
)

In addition, parametrize can now be used with a single value (field=parametrize("for-consistency")), and now works on target generations in more cases (particularly those with "plugin" fields).

Performance

Some processes are run with more consistent "color" configuration which may result in more cache hits: Pytest, Helm unittest and shunit2 processes now always run with colored output enabled. If [GLOBAL].colors is not enabled, Pants removes the coloring before displaying.

Remote caching/execution

Large files that need to be stored to a remote cache are now streamed to it directly from disk, in a way that should reduce overhead and increase concurrency (#19711).

The deprecation has expired for the [GLOBAL].remote_store_chunk_upload_timeout_seconds and [GLOBAL].remote_cache_read_timeout_millis options. Use [GLOBAL].remote_store_rpc_timeout_millis and [GLOBAL].remote_cache_rpc_timeout_millis instead.

Backends

Docker

The docker backend now has explicit support for using buildx (and thus BuildKit), via the new use_buildx option. This also allows adding few additional fields for docker_image targets, for buildx-specific functionality:

• cache_from and cache_to for using external cache storage backends
• build_platform for configuring multi-platform builds
• output to control how/where buildx outputs an image when running pants package ::

JVM

JVM lock-files are more reliable, handling some cases of Coursier not including dependencies when expected.

Using shading_rules is now supported on deploy_jar targets defined in directories of the build root.

Kotlin

The Kotlin analyzer now runs using the Zulu JDK, to be able to run natively on Apple Silicon (arm64 macOS).

Java

Dependency inference for Java now supports code using recently added syntax like sealed class & permits, switch expressions and yield.

Scala

Scala 3 is now supported when using the BSP in an IDE.

OpenAPI

The new pants.backend.experimental.openapi.lint.openapi_format backend supports running openapi-format to format openapi_... targets.

Protobuf

The pants.backend.codegen.protobuf.python backend now supports:

• protobuf_sources targets that are part of multiple Python resolves, like python_resolve=parametrize("first-resolve", "second-resolve").
• [python-infer].ambiguity_resolution = "by_source_root" to infer dependencies without warnings or errors in more cases.

Version 24.4 of the protoc compiler is included as a known version (but is not the default). It can be selected by setting [protoc].version = "24.4". This version has native support for Apple Silicon (arm64 macOS).

Buf (pants.backend.codegen.protobuf.lint.buf) now correctly ignores buf.work.yaml, buf.lock, buf.gen.lock when running as a linter.

Python

Deprecation: Using the platforms field on pex_binary is now deprecated. It is not a recommended way to build for platforms other than the local one, as it is a lossy abbreviation of the target platform and often leads to binaries that don't work. To replace this, either use complete_platforms to continue cross-compiling or use an environment matching the target platform.

Python tests can now be attempted multiple times, if they fail. This can help with limiting the impact of flaky tests. For instance, to require tests to fail three times before the overall pants test goal fails, set [test].attempts_default = 3. In this case, a very flaky test might fail on the first and second runs but finally pass on the third run, and thus the overall Pants invocation would pass.

Pants now uses version 2.1.148 of the PEX CLI by default, which includes support for Python 3.12. To use Python 3.12, you will need to set [python].pip_version to 23.2 or newer (or latest), and ensure all of your tools use resolves generated with this new version (Pants' built-in ones are not, yet).

Python interpreters provided by ASDF are now searched by default, in addition those provided by pyenv and on the system PATH, by being included in the [python-bootstrap].search_path option.

Many more "groups" of PyPI dependencies have default module mappings that guide how an import statement in code matches particular python_requirements. All packages from the azure-..., django-..., google-cloud-..., opentelemetry-instrumentation-..., oslo-... ecosystems are now matched to modules matching their usual conventions. In addition, packages starting with python-... now have that python- prefix removed by default: for instance, a package like python-example-name will be implicitly assumed to be imported like import example_name. The value of any module_mapping field can thus be tweaked (hopefully simplified!).

The [generate-lockfiles].diff option is now True by default, so running pants generate-lockfiles now pretty-prints any differences in the dependencies and their versions.

Additional features and fixes:

• Additional arguments can be passed to the IPython REPL, like pants repl --shell=ipython :: -- -i helloworld/main.py.
• Additional arguments can be passed to the underlying PEX invocations when building FaaS artifacts (AWS Lambda, Google Cloud Functions) via the new pex3_venv_create_extra_args field. For instance, if dependencies have packaged files in unexpected locations, passing pex3_venv_create_extra_args=["--collisions-ok"] can side-step collision errors.
• The pip_version field no longer restricts the values, and so allows more flexibility when upgrading the PEX CLI if it supports newer Pip versions.
• Running tests or pants run a source file now make binaries provided by python_requirement available for execution as normal processes (for instance, using subprocess.run(["name-of-binary", ...])).
• The PATH environment variable can now be set using [subprocess-environment].env_vars
• Import statements within a with contextlib.suppress(ImportError) context are now weak, and so no longer warn if not provided by a dependency.
• Computing an aggregate coverage sandbox report is now more reliable in the presence of name collisions in files created via code-generation (including relocated_files).
• Pyright (pants.backend.experimental.python.typecheck.pyright) now runs with an appropriately initialised environment and so runs more reliably (especially in transient CI environments, where "named caches" might not be preserved).
• Ruff (pants.backend.experimental.python.lint.ruff) now properly respects any exclude configuration from ruff.toml or pyproject.toml.

The following code has been removed, because deprecations have expired:

• Support for building FaaS artifacts (AWS Lambda, Google Cloud Functions) via Lambdex has been fully replaced with the new zip layout: remove the [lambdex] section and its layout setting from pants.toml.
• Passing both runtime and complete_platforms to a FaaS target (python_aws_lambda_function, python_google_cloud_function) is now longer supported, as complete_platforms fully overrides runtime.
• Dependency inference now only supports running with the new Rust-based parser. That is, the [python-infer].use_rust_parser option cannot be disabled and is now unnecessary.

NEW: Semgrep

The new pants.backend.experimental.tools.semgrep backend supports running Semgrep as a local linter on all files that Pants knows about (any target with a source or sources field). Configuration files are automatically discovered.

Shell

The experimental_test_shell_command target now obeys its environment field, and thus runs the command within the specified environment instead of the host.

Terraform

The check goal runs terraform validate on terraform_modules targets by default, again. For modules for which this doesn't work and/or isn't desirable, use the new skip_terraform_validate field. (This restores the behaviour from Pants 2.17 and earlier, with the addition of the skip field, based on feedback about Pants 2.18.0 which switched to only validating terraform_deployments.)

terraform_deployment's var_files field now supports files in a different directory to the root_module.

Transitive dependencies are now automatically included, in addition to direct dependencies.

Plugin API changes

Processes, especially tests, can now be run with retries on failure using ProcessWithRetries (#20378). In addition, to support tests that have been retried multiple times, TestResult.from_fallible_process_result now accepts a tuple of multiple FallibleProcessResults, instead of just one. (#19760).

Some @rules may now need to pass canonical_name_suffix in preparation for "call by name". (#19755)

The new TransitivelyExcludeDependenciesRequest can be used to emulate transitive excludes (!!<address>). (#20080)

The TestResult.stdout and TestResult.stderr fields are now removed, in favour of stdout_bytes and stderr_bytes. (#19768)

Full Changelog

For the full changelog, see the individual GitHub Releases for this series: https://github.com/pantsbuild/pants/releases

New Features

• Add version/local scheme fields to vcs_version (#20446)

• upgrade PEX to 2.1.159 (#20416)

• Add jvm_index Option to Coursier Subsystem (#20271)

• python-protobuf backend now support multiple protoc plugins. (#20387)

User API Changes

• Add remote_provider = "..." option, replacing scheme-look-ups (#20240)

Bug Fixes

• Fix issue with grouping parametrizations on target generators. (#20429)

Documentation

• Mention the Scala repl and its current gotchas in the docs (#20445)

• Fix missing registration for environments rules in pants.core. (#20444)

• Actually call bin_name in help string (#20434)

• Apply latest hand-edits to pantsbuild/pantsbuild.org repo (#20419)

• Fix the weird headings of initial-configuration.mdx (#20422)

• Update docker docs for buildx (#20413)

• What's new in 2.19 (#20310)

• Rename 'environments' docs file/URL to be simpler (#20404)

Bug Fixes

• Fix issue with grouping parametrizations on target generators. (Cherry-pick of #20429) (#20433)

Documentation

• Apply latest hand-edits to pantsbuild/pantsbuild.org repo (#20420)

• Fix the weird headings of initial-configuration.mdx (Cherry-pick of #20422) (#20426)

• Update docker docs for buildx (Cherry-pick of #20413) (#20424)

• What's new in 2.19 (Cherry-pick of #20310) (#20409)

• Rename 'environments' docs file/URL to be simpler (cherry-pick of #20404) (#20406)

• Replace this repo's docs content with the update docusaurus content (#20400)

Documentation

• Rename 'environments' docs file/URL to be simpler (cherry-pick of #20404) (#20405)

• Replace this repo's docs content with the update docusaurus content (#20399)

New Features

• plumb sh-boot through to pex_binary (#19925)

Plugin API Changes

• Add support for positional arguments in by-name @rule calls (#20366)

Performance

• Upgrade to PEX 2.1.156 (#20391)

• Update to Pex 2.1.155 (#20347)

Documentation

• Collapse "overview" pages (#20401)

• Replace this repo's docs content with the update docusaurus content (#20395)

• docs: update python test coverage docs (#20335)

• Update the docs on how to profile Pants. (#20334)

• Improve the cache nuking recommendation. (#20385)

Bug Fixes

• Fix: Include transitive dependencies for Terraform (Cherry-pick of #20241) (#20371)

Bug Fixes

• fix: in-repo plugin requirements.txt not loading (#20355)

• Fix: Include transitive dependencies for Terraform (#20241)

• Allow using scalac plugins that emit additional compilation results (#20357)

Bug Fixes

• Fix: Include transitive dependencies for Terraform (Cherry-pick of #20241) (#20372)

Bug Fixes

• Make docker_environment's Pants-provided-PBS an absolute path (Cherry-pick of #20314) (#20349)

• Ensure experimental_test_shell_command runs in relevant environment (Cherry-pick of #20319) (#20320)