Support remote execution in repository_rule

Motivation:

This change adds support for running a command on a remote machine from a
repository_rule and getting its stdout, stderr and exit code back. The
initial use case of this feature is to allow a repository_rule to inspect
a remote machine, discover the installed tools, libraries and environment
in order to generate Bazel platforms and toolchains for the remote
environment.

The upload of files as part of the command and download of its outputs is
not supported. The use cases looked at so far don't have a need for it.
For example, Bazel's own cc_configure rule only uses one file as part of
ctx.execute(). The contents of the file are "int main() {}". There the
lack of file inputs can simply be worked around with
"echo 'int main() {}' > empty.cc && gcc empty.cc". I am not aware of a
use case that would require downloading output files.

How to Use:

This feature is not on by default. It can be enabled with
--experimental_repository_remote_exec.

repository_rule gains an additional attribute remotable. If true any
repository_ctx.execute() call may be executed on a remote worker. That
is, if Bazel is configured build/test with remote execution so is
repository_ctx.execute():

def _cc_configure_impl(ctx):
  result = ctx.execute(["/bin/bash", "-c", "export"])
  print("stdout: " + result.stdout)
  print("stderr: " + result.stderr)
  print("exit code: " + result.return_code)

cc_configure = repository_rule(
  implementation = _cc_configure_impl,
  remotable = True,
)

Note: The 'remotable' attribute is set by rule author as any such rule needs
to be carefully crafted to work with both local and remote execution.

The remote platform of repository_ctx.execute() is specified via the newly
introduced exec_properties attribute on the repository_rule target:

load(..., "cc_configure")

cc_configure(
  name = "remote_cc_configure",
  exec_properties = {
    "OSFamily" : "Linux",
    "ISA" : "x86-64"
  }
)

The intended usage is for the repository_rule to generate the Bazel
platform and toolchains for the remote platform that the build/test
actions will be run on:

bazel build \
  --experimental_repository_remote_exec \
  --platforms=@remote_cc_configure//:platform \
  --remote_executor=remote.example.com \
  ...

If --remote_executor is not specified then repository_ctx.execute()
is executed on the local/host machine (just like before this change).

RELNOTES: None
PiperOrigin-RevId: 281476515
9 files changed
tree: 3818f6d3e89ffff5981ac05e0d06097aac635e24
  1. .bazelci/
  2. examples/
  3. scripts/
  4. site/
  5. src/
  6. third_party/
  7. tools/
  8. .bazelrc
  9. .gitattributes
  10. .gitignore
  11. AUTHORS
  12. BUILD
  13. CHANGELOG.md
  14. CODEOWNERS
  15. combine_distfiles.py
  16. combine_distfiles_to_tar.sh
  17. compile.sh
  18. CONTRIBUTING.md
  19. CONTRIBUTORS
  20. distdir.bzl
  21. ISSUE_TEMPLATE.md
  22. LICENSE
  23. README.md
  24. WORKSPACE
README.md

Bazel

{Fast, Correct} - Choose two

Build and test software of any size, quickly and reliably.

  • Speed up your builds and tests: Bazel rebuilds only what is necessary. With advanced local and distributed caching, optimized dependency analysis and parallel execution, you get fast and incremental builds.

  • One tool, multiple languages: Build and test Java, C++, Android, iOS, Go, and a wide variety of other language platforms. Bazel runs on Windows, macOS, and Linux.

  • Scalable: Bazel helps you scale your organization, codebase, and continuous integration solution. It handles codebases of any size, in multiple repositories or a huge monorepo.

  • Extensible to your needs: Easily add support for new languages and platforms with Bazel's familiar extension language. Share and re-use language rules written by the growing Bazel community.

Getting Started

Documentation

Contributing to Bazel

See CONTRIBUTING.md

Build status