Support dual execution modes in rbe_config.bzl with automatic host mode (#2747)
This change introduces automatic support for --exec_mode=host in
rbe_config.bzl when generating RBE toolchain configurations:
- By default (when RBE_CONFIG_CONTAINER is unset), rbe_config continues
to use docker mode (--exec_mode=docker), compiling rbe_configs_gen via a
golang:1.21 container (DooD) and running detection inside Docker.
- When RBE_CONFIG_CONTAINER is exported in the environment (e.g. CI
workers), rbe_config automatically switches to host mode
(--exec_mode=host), compiling rbe_configs_gen natively on the host using
go build and running detection against the host filesystem without
Docker.
- Enforces Host Container Authority in host mode: the target container
image string in @rbe_ubuntu//config:platform is authoritatively
determined by RBE_CONFIG_CONTAINER to prevent toolchain and ABI
discrepancies.
- Updates rules/README.md to document execution modes and environment
variables.
diff --git a/rules/README.md b/rules/README.md
index d001840..35ef4af 100644
--- a/rules/README.md
+++ b/rules/README.md
@@ -8,8 +8,8 @@
## 1. Design & Architectural Highlights
-- **Self-Compiling Go (DooD)**: The repository rule downloads the upstream `bazel-toolchains` source repository and dynamically compiles `rbe_configs_gen` inside a sibling `golang:1.21` Docker container via the host's Docker socket (Docker-out-of-Docker). This eliminates any local Go compiler installation requirements.
-- **On-Demand Auto-Detection**: The compiled Go binary launches your target toolchain container, mounts your running host Bazel binary, auto-detects the compiler and JDK runtimes inside the container sandbox, and extracts the generated C++ and Java toolchain configurations directly into Bazel's `output_base`.
+- **Dual Execution Modes (`docker` vs `host`)**: Supports both standard Docker-out-of-Docker generation (default when `RBE_CONFIG_CONTAINER` is unset) and direct host environment generation (automatically enabled when `RBE_CONFIG_CONTAINER` is set).
+- **Direct Host Toolchain Detection**: When `RBE_CONFIG_CONTAINER` is exported, `rbe_configs_gen` compiles natively on the host using `go build` and runs with `--exec_mode=host`, executing C++ and Java auto-detection (`@@rules_cc...`, `java -version`) against the host filesystem without Docker. The target container image is authoritative from the `RBE_CONFIG_CONTAINER` environment variable.
- **Decoupled Presets Rollout**: Standard environment configurations are stored in a public JSON file (`rules/rbe_presets.json`) on the `master` branch of this repository. At runtime, the repository rule dynamically downloads this file. If the CI maintainers update a container image tag or environment parameter on `master`, **all projects immediately receive the update without modifying their pinned ruleset commit hashes!**
---
@@ -131,7 +131,25 @@
---
-## 5. Advanced: Bazel Version & Binary Resolution Strategy
+## 5. Execution Modes (`docker` vs `host`)
+
+`rbe_config` supports two execution modes for toolchain auto-detection, automatically selected based on your environment:
+
+### 5.1 Docker Mode (`--exec_mode=docker`, Default)
+When `RBE_CONFIG_CONTAINER` is **unset** (e.g., on developer workstations running macOS or Linux):
+- **Go Compilation**: `rbe_configs_gen` is compiled inside a sibling `golang:1.21` Docker container using Docker-out-of-Docker (`/var/run/docker.sock`). No local Go compiler is required.
+- **Toolchain Detection**: `rbe_configs_gen` pulls and runs the target toolchain container image (from `preset` or `spec["container"]`), mounts host Bazel, and runs C++ and Java auto-detection inside the sandboxed container.
+
+### 5.2 Host Mode (`--exec_mode=host`, Automatic in Container CI)
+When the **`RBE_CONFIG_CONTAINER`** environment variable is exported (e.g., in containerized CI workers or Buildkite pipelines):
+- **Automatic Activation**: `rbe_config` automatically switches to `host` mode.
+- **Go Compilation**: `rbe_configs_gen` is compiled natively on the host using `go build` (requires Go installed in `PATH`; no Docker required).
+- **Direct Host Detection**: `rbe_configs_gen` runs with `--exec_mode=host`, discovering C++ compilers (`gcc`/`clang`) and JDK runtimes directly from the host filesystem without invoking Docker.
+- **Host Container Authority**: In `host` mode, because detection happens against the host filesystem, the RBE execution platform image in `@rbe_ubuntu//config:platform` is **authoritatively determined by `RBE_CONFIG_CONTAINER`** (overriding any requested preset container image to prevent ABI or toolchain discrepancies).
+
+---
+
+## 6. Advanced: Bazel Version & Binary Resolution Strategy
To generate toolchain configurations, `rbe_config` must determine which Bazel version to target and potentially mount a host Bazel binary inside the compiler container. It resolves this using the following **four-tier precedence lookup strategy**:
diff --git a/rules/rbe_config.bzl b/rules/rbe_config.bzl
index 0489ea4..24935ad 100644
--- a/rules/rbe_config.bzl
+++ b/rules/rbe_config.bzl
@@ -38,29 +38,36 @@
return repository_ctx.path("bazel-toolchains-src")
-def _compile_generator(repository_ctx, src_dir):
- """Compiles rbe_configs_gen inside a sibling Go container.
+def _compile_generator(repository_ctx, src_dir, exec_mode):
+ """Compiles rbe_configs_gen using Docker Go container (docker mode) or host go compiler (host mode).
Returns:
Path to the compiled generator executable.
"""
rbe_gen_path = repository_ctx.path("bazel-toolchains-src/rbe_configs_gen")
- print("rbe_config: Compiling rbe_configs_gen via Go container...")
- compile_res = repository_ctx.execute([
- "docker", "run", "--rm",
- "-v", "{}:/srcdir".format(src_dir),
- "-w", "/srcdir",
- "golang:1.21",
- "go", "build", "-o", "/srcdir/rbe_configs_gen", "./cmd/rbe_configs_gen"
- ])
+ if exec_mode == "docker":
+ print("rbe_config: Compiling rbe_configs_gen via Go container (docker mode)...")
+ compile_res = repository_ctx.execute([
+ "docker", "run", "--rm",
+ "-v", "{}:/srcdir".format(src_dir),
+ "-w", "/srcdir",
+ "golang:1.21",
+ "go", "build", "-o", "/srcdir/rbe_configs_gen", "./cmd/rbe_configs_gen"
+ ])
+ else:
+ print("rbe_config: Compiling rbe_configs_gen on host environment (host mode)...")
+ compile_res = repository_ctx.execute(
+ ["go", "build", "-o", str(rbe_gen_path), "./cmd/rbe_configs_gen"],
+ working_directory = str(src_dir),
+ )
if compile_res.return_code != 0:
- fail("rbe_config: compilation of rbe_configs_gen failed:\nStdout: {}\nStderr: {}".format(compile_res.stdout, compile_res.stderr))
+ fail("rbe_config: compilation of rbe_configs_gen failed (exec_mode={}):\nStdout: {}\nStderr: {}".format(exec_mode, compile_res.stdout, compile_res.stderr))
return rbe_gen_path
-def _generate_toolchains(repository_ctx, rbe_gen_path, bazel_version, bazel_path, container_image, cpp_env):
- """Runs the generator inside the sandboxed container and extracts RBE files."""
+def _generate_toolchains(repository_ctx, rbe_gen_path, bazel_version, bazel_path, container_image, cpp_env, exec_mode):
+ """Runs the generator in docker or host mode and extracts RBE files."""
# Serialize C++ environment dict to JSON inside sandbox
cpp_env_json = "cpp_env.json"
repository_ctx.file(cpp_env_json, json.encode(cpp_env))
@@ -70,6 +77,7 @@
args = [
rbe_gen_path,
+ "--exec_mode=" + exec_mode,
"--toolchain_container=" + container_image,
"--cpp_env_json=" + str(repository_ctx.path(cpp_env_json)),
"--output_tarball=" + str(repository_ctx.path(output_tarball)),
@@ -85,11 +93,11 @@
else:
fail("rbe_config: Neither bazel_version nor bazel_path is available.")
- print("rbe_config: Executing generator to detect toolchains inside {}...".format(container_image))
- exec_res = repository_ctx.execute(args)
+ print("rbe_config: Executing generator (exec_mode={}) to detect toolchains (target container: {})...".format(exec_mode, container_image))
+ exec_res = repository_ctx.execute(args, environment = cpp_env if cpp_env else {})
if exec_res.return_code != 0:
- fail("rbe_config: Dynamic generation failed:\nStdout: {}\nStderr: {}".format(exec_res.stdout, exec_res.stderr))
+ fail("rbe_config: Dynamic generation failed (exec_mode={}):\nStdout: {}\nStderr: {}".format(exec_mode, exec_res.stdout, exec_res.stderr))
# Extract the generated configs directly into the repository's directory
repository_ctx.extract(archive = output_tarball)
@@ -98,20 +106,33 @@
# --- Private Repository Rule Entrypoint ---
def _rbe_config_impl(repository_ctx):
# 1. Resolve presets/custom image container and environment
- container_image, cpp_env = _resolve_preset(
+ preset_container, cpp_env = _resolve_preset(
repository_ctx,
repository_ctx.attr.preset_name,
repository_ctx.attr.container,
repository_ctx.attr.cpp_env
)
- # 2. Download bazel-toolchains source code
+ # 2. Determine execution mode ("docker" vs "host") and target container image
+ host_container = repository_ctx.os.environ.get("RBE_CONFIG_CONTAINER")
+ if host_container:
+ exec_mode = "host"
+ container_image = host_container
+ if preset_container and preset_container != host_container:
+ print("rbe_config: RBE_CONFIG_CONTAINER ('{}') overrides requested preset container ('{}') in host mode.".format(host_container, preset_container))
+ else:
+ exec_mode = "docker"
+ container_image = preset_container
+
+ print("rbe_config: Using exec_mode='{}' for toolchain generation.".format(exec_mode))
+
+ # 3. Download bazel-toolchains source code
src_dir = _download_bazel_toolchains(repository_ctx)
- # 3. Compile rbe_configs_gen
- rbe_gen_path = _compile_generator(repository_ctx, src_dir)
+ # 4. Compile rbe_configs_gen in docker or host mode
+ rbe_gen_path = _compile_generator(repository_ctx, src_dir, exec_mode)
- # 4. Resolve Bazel version or locate host Bazel
+ # 5. Resolve Bazel version or locate host Bazel
bazel_version = None
bazel_path = repository_ctx.os.environ.get("RBE_CONFIG_BAZEL_PATH")
@@ -133,8 +154,8 @@
else:
fail("rbe_config: Bazel executable not found in PATH. If you are using a custom-named binary or non-standard layout, please export RBE_CONFIG_BAZEL_PATH=/path/to/your/binary or use a tools/bazel wrapper script.")
- # 5. Run the generator and extract configurations
- _generate_toolchains(repository_ctx, rbe_gen_path, bazel_version, bazel_path, container_image, cpp_env)
+ # 6. Run the generator and extract configurations
+ _generate_toolchains(repository_ctx, rbe_gen_path, bazel_version, bazel_path, container_image, cpp_env, exec_mode)
# Private repository rule
@@ -149,7 +170,8 @@
environ = [
"RBE_CONFIG_BAZEL_PATH",
"BAZEL_REAL",
- ], # Propagate custom Bazel path environment variables
+ "RBE_CONFIG_CONTAINER",
+ ], # Propagate custom Bazel path and container environment variables
)
# --- Public WORKSPACE wrapper macro ---
