rebar3

Security release. See https://ferd.ca/you-ve-got-to-upgrade-rebar3.html

This release drops multiple OTP versions as officially supported. We're aligning rebar3 development with the OTP team's compatibility guarantees, as some deprecations from OTP-21 were removed in OTP-24 and we just could not maintain compatible code bases. We're using the breakage opportunity to change just how much we support to something more sustainable.

The release also contains extra fixes that should prove useful.

• Only support 3 newest OTP versions
• add support to configure application resource naming
• Support ERL_LIBS libraries in escript building
• Fix CLI arg order in makefile template
• Increase OTP 24 readiness + move to OTP 20 min.
• Bump erlware_commons and certifi
• Treat applications and included_applications the same way when builing Dialyzer PLTs
• Strip CLI args from provider hooks
• Fix plugins upgrade for non-tuple specification

Here's all the changes that may impact some behaviour:

• Lots of Relx-related fixes as 4.x stabilizes, including:
  • fixes in handling of system_libs
  • bringing back start and attach commands (with warnings)
  • additional bugfixes for embedded, NixOS, and windows systems
• Rework DEBUG=1 mode to be user-centric output and introduce separate DIAGNOSTIC=1 mode for implementer-level logging (see also: Reworking more of the DEBUG output to be friendly)
• Trigger rebuilds when OTP compiler version changes (also includes fixes to plugins using checkout directories)
• Fix endless recursion in del_dep overrides
• Fix lock file not being generated when no deps are present in the config
• Change xref paths to only those of runtime deps
• Fix compile issue on windows when OS language is not english
• fix logger reset for the shell provider
• Re-vendor hex_core in at 0.7.x
• Fix relup command arguments and add validation
• Add checkouts location back to rebar3 path
• Support pre-fetched deps for nix
• set explicit encoding when reading and writing hex config
• Update references to changed API elements
• Remove verbosity option that was taken out in relx 4.x
• Run pkg updates on plugin upgrades (see also: Fixup plugin upgrade autoupdate)
• Make rebar3 templates check for name clashes
• Patch rebar_base_compiler to handle extensions with no leading dot (legacy compiler interface)
• Handle unicode in author's name in templates
• Specify dependencies for MIB files
• Ignore global and default in empty profiles warning for as
• Use infinity timeout for rebar_compiler_epp resolve call
• Add alias to example rebar.config.sample
• Fix dialyzer warning output for option unknown
• Add xref_ignores to sample config file
• Prevent xref crash when function deemed undefined has no local context
• Output relative paths in Dialyzer output (now controlled by the compiler_source_format option)

And some Internal Changes:

• Prevent dialyzer warnings on OTP 23
• Add a docker recipe for rebar3 itself
• Fix spec for download callback
• Add rebar_hex_repos:remove_from_auth_config/2
• Bumping cth_readable version and other small cleanups
• Add regression test for dep override infinite loop
• Fix README (see also: Update README.md and Update README with supported Erlang/OTP versions)
• Update link for Rebar 2.x to Rebar3
• use 'aws cp' when copying escript to s3

Contributions from 17 people are part of this release, and the project health is starting to look better; thanks to everyone involved!

• git lock: abort if version is not equal to or greater than 1.8.5
• require the app name as the last part of git_subdir path
• Fix Recursive Behaviour Lookup Regression
• bump relx to 4.0.2

New Features (since 3.13.2):

• git_subdir dependencies (and #2290) allow to only get a subdirectory of a given git repository. This change will not work with older rebar3 versions and may error out their build if used, although an identical plugin could be written to patch up for that use case. Declare the dependency as:
  • {git_subdir, Url, {branch, Name}, Dir}
  • {git_subdir, Url, {tag, Ref}, Dir}
  • {git_subdir, Url, {ref, Ref}, Dir}
• Add powershell script generation option for escriptize on windows; use for rebar3 itself
• dialyzer support: run single-app success typing and store project apps in PLT
• Add symlink to the ct_run of the last common test run
• ipv6 support for proxy configuration
• Source file encoding specifier support for templates
• Allow passing arguments in r3:do and r3:async_do
• Add templating feature for "from" file/template

Relx Changes (including more new features):

• relx 4.0 support, see the full release notes for the lot of new features and changes it contains
• add --all option for release building in parallel

Compiler Changes:

• Update compiler hooks order and internal compiler phases
• Split up the compiler DAG
• Switch to EPP analysis covering all apps at once when compiling
• Track build artifacts in DAG
• Another Compiler Refactor: Performance & Cleanup
• Name the app being escriptized to help readability when escriptize is run as a hook
• Additional performance improvements for large projects
• Fix directory recursion in compiler
• prv_plugins_upgrade: fix compilation of upgraded plugins

Other Changes:

• OTP-23 readiness
• Update hex core to v0.6.8, which includes some security work, and creates a new lockfile version. Related: eliminate lock file warnings when upgrading from 1.1.0 locks
• Add vendor_hex_core.sh script
• Allow users to override default hex auth config
• Fix tests in Windows; adjust absolute paths behaviour for OTP-22 and later
• output checkouts compilation to _build/<profile>/checkouts/, which helps prevent long-standing issues with the ebin/ directory being left around in the source location
• Support ENV var for mix in the bare compiler (related to - Allow specifying the outdir in bare compiler added in 3.13.1)
• checking epmd is started without a crash for features requiring named nodes
• Fix typo in abort message
• Newer version of the certifi certs bundle; a good thing to keep up
• Annotate generated escript with last release version for quicker version checks without calling the tool
• Simplify windows script
• Return an error if no apps are found when using bare compiler
• xref now outputs relative file paths
• Move absolute_path/2 and normalized_path to rebar_file_utils
• rebar_git_resource: fix {vsn, git} parser
• Fix race condition in rebar_prv_shell boot
• re-enable use of rebar_packages_cdn in rebar_pkg_resource
• only write lock file if locks have changed
• rebar_git_resource: allow non-single-digit git versions
• rebar_dir: return _checkouts path in a canonical form
• Bump cth_readable (1.4.8) to fix assertion macros
• Proper Unicode handling in port process output
• get a sorted list of modules for coverage output (screen and HTML)
• Run dialyzer for extra_src_dirs
• remove warning when booting applications in the shell
• Fix bootstrap crashes when compilers get modified
• Remove erl_interface from makefile template which is deprecated.
• Improve warnings about checkout dependencies when locking and upgrading
• Bump a bunch of dependencies
• Extract rebar3 libs in a version dependent directory when using rebar3 local commands
• Feature shell.script_file in rebar.config.sample
• Add extra pattern matching clause to not crash handling with keep_logs in rebar3 ct
• Fix symlink to last CT logs directory with logdir option present
• Fix keep_logs option crashing when having more than N directories
• shell: provide error message when applications list is bad
• Match on dep name in overrides
• Fix use of ct on projects without CT suites and configuration
• properly track app vsn that is the evaluated application vsn rather than just the source one
• Escape more characters for shell stuff

New Features (since 3.13.2):

• git_subdir dependencies (and #2290) allow to only get a subdirectory of a given git repository. This change will not work with older rebar3 versions and may error out their build if used, although an identical plugin could be written to patch up for that use case. Declare the dependency as:
  • {git_subdir, Url, Dir}
  • {git_subdir, Url, {tag, Ref}, Dir}
  • {git_subdir, Url, {ref, Ref}, Dir}
• Add powershell script generation option for escriptize on windows; use for rebar3 itself
• dialyzer support: run single-app success typing and store project apps in PLT
• Add symlink to the ct_run of the last common test run
• ipv6 support for proxy configuration
• Source file encoding specifier support for templates
• Allow passing arguments in r3:do and r3:async_do

Relx Changes (including more new features):

• relx 4.0 support, see the full release notes from rc-1 and the changes in rc-2 for the lot of new features and changes it contains
• add --all option for release building in parallel

Compiler Changes:

• Update compiler hooks order and internal compiler phases
• Split up the compiler DAG
• Switch to EPP analysis covering all apps at once when compiling
• Track build artifacts in DAG
• Another Compiler Refactor: Performance & Cleanup
• Name the app being escriptized to help readability when escriptize is run as a hook

Other Changes:

• OTP-23 readiness
• Update hex core to v0.6.8, which includes some security work, and creates a new lockfile version. Related: eliminate lock file warnings when upgrading from 1.1.0 locks
• Add vendor_hex_core.sh script
• Allow users to override default hex auth config
• Fix tests in Windows; adjust absolute paths behaviour for OTP-22 and later
• output checkouts compilation to _build/<profile>/checkouts/, which helps prevent long-standing issues with the ebin/ directory being left around in the source location
• Support ENV var for mix in the bare compiler (related to - Allow specifying the outdir in bare compiler added in 3.13.1)
• checking epmd is started without a crash for features requiring named nodes
• Fix typo in abort message
• Newer version of the certifi certs bundle; a good thing to keep up
• Annotate generated escript with last release version for quicker version checks without calling the tool
• Simplify windows script
• Return an error if no apps are found when using bare compiler
• xref now outputs relative file paths
• Move absolute_path/2 and normalized_path to rebar_file_utils
• rebar_git_resource: fix {vsn, git} parser
• Fix race condition in rebar_prv_shell boot
• re-enable use of rebar_packages_cdn in rebar_pkg_resource
• only write lock file if locks have changed
• rebar_git_resource: allow non-single-digit git versions
• rebar_dir: return _checkouts path in a canonical form
• Bump cth_readable (1.4.8) to fix assertion macros
• Proper Unicode handling in port process output
• get a sorted list of modules for coverage output (screen and HTML)
• Run dialyzer for extra_src_dirs
• remove warning when booting applications in the shell
• Fix bootstrap crashes when compilers get modified

This is a small patch release that aims to backport a fix from master to help interoperability of rebar3 3.13.x with external tooling by annotating the built scripts with versions, so that said tools can find the version supported without needing to invoke rebar3 itself.

• Initial relx 4.0 support
  • See relx 4.0.0-rc1 release for more details
• Track build artifacts in DAG
• Return an error if no apps are found when using bare compiler
• xref improvements: now outputs relative file paths
• Add symlink to the ct_run of the last common test run
• Move absolute_path/2 and normalized_path to rebar_file_utils
• rebar_git_resource: fix {vsn, git} parser
• rebar3: ipv6 support for proxy
• Switch to EPP analysis covering all apps at once when compiling
• Fix race condition in rebar_prv_shell boot
• rebar_compiler_dag: implement single-pass pruning
• Force use of tls1.2 for http fetching to work on OTP-23-rc1
• Re-enable use of rebar_packages_cdn in rebar_pkg_resource
• Source file encoding specifier support for template
• Only write lock file if locks have changed
• rebar_git_resource: allow non-single-digit git versions
• rebar_dir: return _checkouts path in a canonical form
• Bump cth_readable (1.4.8) to fix assertion macros
• Update hex core to v0.6.8
• Split up the compiler DAG
• Add vendor_hex_core.sh
• Proper Unicode handling in port process output
• Update compiler hooks order and internal compiler phases
• Get a sorted list of modules for coverage output (screen and HTML)
• Run dialyzer for extra_src_dirs
• Remove warning when booting applications in the shell
• Allow passing arguments in r3:do and r3:async_do
• Allow users to override default hex auth config

• OTP-23 compatibility and warning removal
• Allow specifying the outdir in bare compiler
• force use of tls1.2 for http fetching to work on OTP-23-rc1

• Silence http_uri deprecation warnings on OTP-23
• add rebar_string:trim function for removing all leading/trailing whitespace
• Add --statistics option to dialyzer command
• Fix epmd start on windows
• Allow cleaning specific apps or deps only
• Disable bbmustache escaping
• Handle missing OTP apps better
• Support profile_string overlay var in releases
• Reload logger config in CT suites
• Disable the default logger handler in shell if required
• Fix case clause which used wrong element order
• Fix a couple typos in readme

New features:

• upgrade relx 3.33.0, which contains a lot of useful fixes regarding booting of epmd, arbitrary overlay variables now being supported from rebar3, fixes around templates, paths on Windows, and other path handling issues.
• support REBAR_BASE_DIR env variable to optionally choose another target than _build. This value was already settable from a rebar.config file, but not from the environment. This can prove useful for Docker images, for example.
• Support rebar config dir env var, allowing to use REBAR_GLOBAL_CONFIG_DIR to change the global directory used by rebar3 for a specific run only
• Warn when there's mismatch between rebar.lock & original package checksum. The feature was already safe, but would not communicate the failure clearly.

And still a lot of fixes:

• format error from provider init and print as warning, which fixes issues with some plugins, particularly rebar_mix, used for elixir dependencies
• Fixing duplicate macro definition in umbrella edoc. This used to cause crashes in specific project structures.
• Ensure EDoc opts in umbrella apps are respected
• Repair edoc functionality on rebar3 itself
• Update vendored in hex_core to version 0.5.1
• de-duplicate dependency list in rebar_compiler_erl.erl, which could be costly for third-party languages when it came to module-level dependency analysis
• Fix reloading logger_level in the shell or tests with custom sys.config files
• rebar3 dialyzer: Warn when debug_info is disabled
• keep resources in new state used in plugins upgrade
• Fix crash when a dependency is missing app.src file
• Fix custom compiler mods typespecs, add edoc
• Bump cth_readable to fix lager support on OTP-21+
• Make missing file report more comprehensive during compilation errors
• Gracious loading of unloaded but blacklisted mods in rebar3 shell.
• Only lint app files when compiling, to prevent too frequent warnings

• bump relx to 3.32.1 and providers to 1.8.1
• add support for usage message after template is done
• fix install of s3cmd for publishing releases in cirrus
• upgrade ssl_verify_fun to 1.1.5

The added support for OTP-22 means a tacit deprecation of OTP-17 support. The build is likely to keep working, but maintainers will not make an explicit effort to keep it compatible in the future.

• Bug when reread_config running on OTP 17
• Add support for reading in environment variables from a file
• Upgrade relx to 3.32.0
• Do not apply overrides to a root application.
• OTP-22.0 readiness
• Add support for git ref and file content as app version
• Supervisor template: add missing comma
• Remove large comment blocks from templates and use maps in sup
• Fix cth_readable crash with bad lager interaction
• Format hex packages in a nice/human way
• Fix duplicate module naming in edoc test suite data
• Add basic linting for .app file

• upgrade relx 3.31.0
• Update certifi, ssl_verify_fun, bbmustache
• support dialyzer plt location and prefix in cli args
• rebar3 eunit: Hide slowest tests profile output by default
• Enable parallel build
• Catch errors in writing the package cache
• Fix rebar3 deps command
• Attempt at support sys_config_src in shell
• Fix provider hook handling for escriptize
• Remove ignored hex_core artifact from repo
• Fix handling of transient locks during upgrade

• upgrade relx 3.29.0, containing various fixes to the extended start script and nodetool
• Fix fetching private packages from Hex orgs
• Abort when file:consult/1 returns an error in rebar3_hex_repos:auth_config/1
• switch documentation links to https where supported
• Display plugin versions in plugin list command

This is a release that both adds a few shell-related features, but more importantly fixes some significant bugs.
First the bugs:

• Fix performance regression in compiler around behaviours, which sometimes resulted in massive slowdowns
• Fix common test logging output with OTP-21.2 and logger when multiple crashes happen
• Fix handling of updated hrl files not triggering a recompile in tests when they are changed
• Unhang the logger when swapping the TTY in shell in OTP-21.1 and later
• Use double quotes in git shell command on Windows to prevent failures
• fix case clause handling when fetching hg dep
• Reorder rebar3 tree output on umbrella apps to be predictable
• update templates to remove deprecated maintainers field

The shell features are some workarounds and niceties related to known limitations:

• Support setting shell hot code loading blacklists to prevent frequently crashing applications from being killed
• Add --relvsn and --relname to rebar3 shell to allow selecting which release definition to use

A bunch of fixes, with some new minor features! The new features are:

• Add --generator option for eunit, which lets you run individual test cases
• implement mib_first_files support in rebar_compiler_mib
• Support ignoring entire modules in xref results
• Add a --fail_fast switch to common test which aborts the whole test run as soon as one of them fails -- this feature is experimental

And for the bug fixes:

• update relx to 3.28.0
• Update hex_core and add mirror_of repo config
• bump erlware_commons to 1.3.1
• re-expose download/3 for backwards compatibility in git deps-related plugins
• Fix pkg name during update by using pkg_name understood by hex
• Fix load order for custom project builders and parse transforms

• search subdirectories as well for imports/transforms/behaviours
• Fix path coverage loading artifacts
• Add hint for rebar3 new help on rebar3 help new

• revert ebin_modules to include all beams minus extra_src (#1955) -- fix for adding modules built from non-erl files, like beams built from erlydtl templates

• Fix a compiler issue that could cause problem in some plugins