rules_ll

Based on https://github.com/llvm/llvm-project/commit/5e57f1bf52834cd5133011624c9bbc8cbf58c272.

🥳 Hello AMDGPU

The ROCm/HIP overlays are finally in a usable state. You can use compilation_mode = "hip_amdgpu" and rules_ll will build all dependencies from the upstream ROCm sources. We've also added OFFLOAD_ALL_NVPTX and OFFLOAD_ALL_AMDGPU shortcuts that let you target all supported NVPTX or AMDGPU architectures in the compile_flags attribute. Check out the updated CUDA/HIP Guide for more details.

It's now possible to use C++ modules and heterogeneous code together.

✨ Better code quality

We've added a new ll_test target that works just like ll_binary but is usable via the bazel test infrastructure. The examples act as tests which will be added to CI in future releases. The test suite is not complete yet, but the basic toolchains should be fairly robust now.

The pre-commit hooks now run in CI to ensure consistent code quality for all PRs.

This release is the first release that satisfies the badge.

As part of our improvements to the rules_ll infrastructure we now follow the LLVM community code of conduct.

😅 Better onboarding... again

The ll init command wasn't ergonomic and has been removed. Create new projects via our new Nix templates:

nix flake init -t github:eomii/rules_ll/20230411.0

🗜️ Faster compression

rules_ll now supports zstd compression. You can now compress your debug sections with at least 3x the speed of old zlib compression. We also swapped out zlib to zlib-ng though, so whatever compression you choose, it will be faster.

🤖 More bots

To keep up to date with all the dependencies that rules_ll uses at this point, we started tracking dependencies with bots. Check out the dependency dashboard to view all tracked dependencies.

Bugfix release.

Fix module builds, depends_on_llvm, and use the correct dynamic linker in the nix environment.

Based on https://github.com/llvm/llvm-project/commit/7a282bd2aaa5c1337023578426b15c294eb274bc.

A lot has changed.

❄️ Enter Nix

rules_ll is now a nix flake. Previously, if your host toolchain wasn't properly configured, rules_ll could fail during the build of Clang/LLVM and other external rules_cc dependencies. Since Bazel would use your regular host toolchain to build the rules_ll toolchain, builds were not easily reproducible across machines.

To eliminate these issues, we now ship an entire reproducible host toolchain as part of rules_ll. Every dependency is now traceable and the rules_ll toolchain is guaranteed to build fully reproducible and hermetic. Nix environment variables are piped into action environments through a custom bazel wrapper. This way if you change an external dependency in a way that Bazel would usually be unable to detect rules_ll will still notice it and trigger a rebuild.

This approach makes cache poisoning virtually impossible and makes rules_ll highly portable, even to older systems that never would be able to build upstream Clang/LLVM otherwise. It also makes rules_ll immune to misconfigured (or nonexistent) host toolchains.

🚢 Improved onboarding

If you have an existing rules_ll project you can install the nix package manager, enable flakes and run nix develop github:eomii/rules_ll to get dropped into a rules_ll development shell. Invoking bazel from within that shell will use the new fully hermetic bazel wrapper.

If you want to start a new rules_ll project, you can now enter the new ll init command from within a rules_ll shell to automatically create the files you previously had to set up manually.

🧪 Experimental new rules

Added a new ll_coverage_test rule that automatically builds a target with coverage instrumentation, runs it, and creates coverage reports, either output to the terminal or as Browser-viewable web files.

💔 Breaking changes

The relative_* include attributes have been renamed to just includes, angled_includes etc. The old includes attributes have been removed as they didn't provide any additional functionality that one couldn't already achieve by adding includes to compile_flags.

Based on https://github.com/llvm/llvm-project/commit/108b08f2a91272f82d524616a337a8ce52edeed5

CUDA has not been updated to 12.0. The required drivers won't work with Linux 6.1. We'll have to wait until fixed Nvidia drivers are available as we don't want to downgrade the kernels on our development machines.

📜 New docs

The documentation has been completely reworked. We now use the MkDocs engine with the Material for MkDocs theme and Vale linting. The existing guides have been reworked and new guides for sanitizers and heterogeneous compilation were added. We hope that this makes rules_ll a lot more accessible.

Check out the new docs at ll.eomii.org.

✨ New expectations

As part of the LLVM commit version bump you can now use the C++23 feature std::expected.

💔 Breaking changes

The cuda_nvidia and hip_nvidia settings for compilation_mode were renamed to cuda_nvptx and hip_nvptx. This sets us up for more consistent naming with future compilation modes.

Additionally, rules_ll is now distributed under the Apache 2.0 license with LLVM exceptions. This should not have much of an impact (we used Apache 2.0 without LLVM exceptions before) but is more consistent with the LLVM project.

💮 Internal changes

Reworked the internal build directory structure. Artifacts are no longer loosely in bazel-bin. Instead, every target now creates a directory of the same name and artifacts are in these directories. This enables us to build the same file in multiple configurations in the same build.

❤️‍🩹 Fixes and known issues

This release is based on https://github.com/llvm/llvm-project/commit/3661458b4cb3b4af2211a7e9889768cace9c710f, which includes an important fix for using <format> in modules. Libraries like spdlog can now be #included in the GMFs of module interface and implementation units without crashing clang.

If you get weird errors regarding symbol visibility in C++ modules, check if the library that breaks uses #pragma once before include guards. This is currently a bug in clang. A workaround is to either remove #pragma once in the problematic headers or swap the include guards to be defined before the pragma.

⚗️ Experimental OpenMP CPU support

Added a new compilation_mode = "omp_cpu" flag which enables #include "omp.h" and use of OpenMP pragmas. This is very experimental right now. Our goal with this will be to enable OpenMP GPU offloading in the future.

🎄 Multi-target compilation databases

The target attribute has been removed from ll_compilation_database. Instead it now expects a list of targets in the targets (with s) attribute. To migrate, change your buildfiles from

ll_compilation_database(
    ...
    target = ":mytarget",
)

to

ll_compilation_database(
    ...
    targets = [":mytarget"],
)

Keep in mind that if you have a directory structure like

BUILD.bazel
- pkg1
  BUILD.bazel  # contains "target1"
- pkg2
  BUILD.bazel  # contains "target2"

and you want to use a single ll_compilation_database at the root build file for the entire project, the visibilities for target1 and target2 need to be set to something like visibility = ["@//:__pkg__"] (or visibility = ["@myproject//:__pkg__"], if your project specifies a module name).

Based on LLVM commit 0ec197a30dadd1f639fe73786fa2da96af97affd.

• Enabled basic compatibility with remote caching. Link actions are explicitly marked to run locally until we have a way to distribute the local dependencies we currently require as part of the toolchain.
• Added CUPTI to the CUDA and HIP toolchains.
• Bumped HIP and hipamd to rocm-5.3.3.
• Fixed compatibility with new Bazel prereleases officially using 7.0.0-pre.20221102.3 now.
• Fixed some issues with C++ modules when imported from external targets.
• Added .cu to compilable extensions.

The eomii-bazel-registry has been rebased against upstream.

This release reworks the bundling and linking steps. We now use the new clang offload driver and invoke the linkers via the clang-linker-wrapper. This allows us to build relocatable device code in targets using the hip_nvidia and cuda_nvidia compilation modes while still using clang as device compiler.

To enable relocatable device code, add -fgpu-rdc to compile_flags.

The infrastructure has been mostly updated to the new externalized llvm repo now.

Based on LLVM commit 7ded25c0cb298f474ffc1d9691a7d7c57c08c9f6.
Updates CUDA to 11.8.

Major changes:

• Docs for //ll:defs.bzl should now display correctly.
• Removed all transitive_* attributes in favor of new exposed_* attributes. These new attributes only expose to direct dependents, not to transitive dependents. This simplifies dependency resolution and will likely lead to BUILD files that are easier to reason about.
• A lot of the logic around CUDA has changed. We now have more fine-grained control over the toolkit and are able to link shared CUDA libraries in build targets.
• Some reworking around C++ module logic. The internal code generating precompilation paths etc should now be easier to understand.

Initial release using the new externalized llvm-project-overlay module. This release is mainly for testing and adjusting the new infrastructure.

Adds support for C++ modules.

See new docs at https://ll.eomii.org/guides/modules.html

Most things seem to work. Heterogeneous programming doesn't (yet).

We applied a bunch of patches and workarounds which will be cleaned up when we find better alternatives 😇

Fixes missing format and ranges.

Introduce support for C++ modules.

Fix centos detection.

Fix faulty OS ID detection.

Obsoletes manual setting of symbolic links for *crt*.o detection. Also adds automatic OS detection to determine whether the local library path is /usr/lib64 or /usr/lib/x86_64-linux-gnu.

Add confusables support for clang-tidy, add externally fetched zlib, remove terminfo.

Also contains some code required for the not yet functional sycl_cpu toolchains.

Adds support for sanitizers.

Major toolchain refactoring.

Add workspace-relative inclusions.

Rewrite overlay logic and fetch patches remotely.