Used checked in configs with rbe_autoconf when possible (#276)
For users of the rbe ubuntu 16_04 containers, attempt to first find if a published config will work for their chosen Bazel version. If such a config exists, rule only creates BUILD files with aliases pointing to the published configs.
Pulling container / generating configs will only happen if the above step does not find a published config.
Behavior can be overriden with use_checked_in_confs = False,
diff --git a/WORKSPACE b/WORKSPACE
index b0fb8a5..225f6e9 100644
--- a/WORKSPACE
+++ b/WORKSPACE
@@ -106,6 +106,11 @@
rbe_autoconfig(name = "rbe_default")
rbe_autoconfig(
+ name = "rbe_default_no_checked_in_confs",
+ use_checked_in_confs = False,
+)
+
+rbe_autoconfig(
name = "rbe_default_with_output_base",
config_dir = "default",
output_base = "configs/ubuntu16_04_clang/1.1",
diff --git a/configs/ubuntu16_04_clang/versions.bzl b/configs/ubuntu16_04_clang/versions.bzl
new file mode 100644
index 0000000..0dbf8b8
--- /dev/null
+++ b/configs/ubuntu16_04_clang/versions.bzl
@@ -0,0 +1,20 @@
+# Copyright 2016 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License
+
+# Returns a dict with major container versions mapped to supported bazel versions.
+def config_to_bazel_versions():
+ return {
+ "1.0": ["0.13.0", "0.14.1", "0.15.0", "0.16.1"],
+ "1.1": ["0.16.1", "0.17.1", "0.18.0", "0.19.0", "0.19.2", "0.20.0", "0.21.0"],
+ }
diff --git a/rules/BUILD.std_container.tpl b/rules/BUILD.std_container.tpl
new file mode 100644
index 0000000..4470fa3
--- /dev/null
+++ b/rules/BUILD.std_container.tpl
@@ -0,0 +1,34 @@
+# Copyright 2016 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This file is auto-generated by an rbe_autoconfig repository rule
+# and should not be modified directly.
+# See @bazel_toolchains//rules:rbe_repo.bzl
+
+package(default_visibility = ["//visibility:public"])
+
+alias(
+ name = "jdk",
+ actual = "%{jdk}",
+)
+
+alias(
+ name = "cc-toolchain",
+ actual = "%{cc-toolchain}",
+)
+
+alias(
+ name = "platform",
+ actual = "%{platform}",
+)
diff --git a/rules/BUILD.std_toolchain.tpl b/rules/BUILD.std_toolchain.tpl
new file mode 100644
index 0000000..5319850
--- /dev/null
+++ b/rules/BUILD.std_toolchain.tpl
@@ -0,0 +1,25 @@
+# Copyright 2016 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This file is auto-generated by an rbe_autoconfig repository rule
+# and should not be modified directly.
+# See @bazel_toolchains//rules:rbe_repo.bzl
+
+package(default_visibility = ["//visibility:public"])
+
+alias(
+ name = "toolchain",
+ actual = "%{toolchain}",
+)
+
diff --git a/rules/rbe_repo.bzl b/rules/rbe_repo.bzl
index ce153cc..98f2ce6 100644
--- a/rules/rbe_repo.bzl
+++ b/rules/rbe_repo.bzl
@@ -12,13 +12,16 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-"""Repository Rules to generate toolchain configs for a container image.
+"""Repository Rules to pick/generate toolchain configs for a container image.
-The toolchain configs (+ platform) produced by this rule can be used to, e.g.,
-run a remote build in which remote actions will run inside a container image.
+The toolchain configs (+ platform) produced/selected by this rule can be used
+to, e.g., run a remote build in which remote actions will run inside a
+container image.
Exposes the rbe_autoconfig macro that does the following:
-- Pulls an rbe-ubuntu 16_04 image (using 'docker pull'). Image to pull can be overriden.
+- If users selects the standard rbe-ubuntu 16_04 image, create aliases to
+ the appropriate toolchain / platform targets for the current version of Bazel
+- Otherwise, pull the selected toolchain container image (using 'docker pull').
- Starts up a container using the pulled image, mounting either a small sample
project or the current project (if output_base is set).
- Installs the current version of Bazel (one currently running) on the container
@@ -89,9 +92,10 @@
other special chars).
There are two modes of using this repo rules:
- 1 - When output_base set (recommended; env var "RBE_AUTOCONF_ROOT" is required),
- running the repo rule target will copy the toolchain config files to the
- output_base folder in the project sources.
+ 1 - When output_base set (recommended if using a custom toolchain container
+ image; env var "RBE_AUTOCONF_ROOT" is required), running the repo rule
+ target will copy the toolchain config files to the output_base folder in
+ the project sources.
After that, you can run an RBE build pointing your crosstool_top flag to the
produced files. If output_base is set to "rbe-configs" (recommended):
@@ -113,7 +117,8 @@
you do not need to run this rule until there is a new version of Bazel
you want to support running with, or you need to update your container).
- 2 - When output_base is not set (env var "RBE_AUTOCONF_ROOT" is not required),
+ 2 - When output_base is not set (recommended for users of the
+ rbe-ubuntu 16_04 images - env var "RBE_AUTOCONF_ROOT" is not required),
running this rule will create targets in the
external repository (e.g., rbe_default) which can be used to point your
flags to:
@@ -131,16 +136,16 @@
Note running bazel clean --expunge_async, or otherwise modifying attrs or
env variables used by this rule will trigger it to re-execute. Running this
- repo rule takes some time as it needs to pull a container, run it, and then
- run some commands inside. We recommend you use output_base and check in the produced
+ repo rule can take some time (if you are not using the rbe-ubuntu 16_04 container)
+ as it needs to pull a container, run it, and then run some commands inside.
+ We recommend you use output_base and check in the produced
files so you dont need to run this rule with every clean build.
The {bazel_version} above corresponds to the version of bazel installed locally.
-Note you can override this version and pass an optional rc# if desired.
-Running this rule with a non release version (e.g., built from source) will not work.
-If running with bazel built from source you must pass a bazel_version and bazel_rc
-to rbe_autoconfig. Also, note the bazel_version bazel_rc must be published in
-https://releases.bazel.build/...
+Note you can override this version and pass an optional rc # if desired.
+Running this rule with a non release version (e.g., built from source) will result in
+picking as bazel version <bazel_version_fallback> defined in the attrs of this rule.
+Note the bazel_version / bazel_rc must be published in https://releases.bazel.build/...
Note this is a very not hermetic repository rule that can actually change the
contents of your project sources. While this is generally not recommended by
@@ -149,14 +154,14 @@
of any build actions.
Note: this rule expects the following utilities to be installed and available on
-the PATH:
+the PATH (if any container other than rbe-ubuntu 16_04 is used):
- docker
- tar
- bash utilities (e.g., cp, mv, rm, etc)
- docker authentication to pull the desired container should be set up
(rbe-ubuntu16-04 does not require any auth setup currently).
-Known limitations:
+Known limitations (if any container other than rbe-ubuntu 16_04 is used):
- This rule cannot be executed inside a docker container.
- This rule can only run in Linux.
"""
@@ -169,8 +174,13 @@
load(
"//rules:toolchain_containers.bzl",
"RBE_UBUNTU16_04_LATEST",
+ "public_rbe_ubuntu16_04_config_version",
"public_rbe_ubuntu16_04_sha256s",
)
+load(
+ "//configs/ubuntu16_04_clang:versions.bzl",
+ "config_to_bazel_versions",
+)
load("//rules:environments.bzl", "clang_env")
# External folder is set to be deprecated, lets keep it here for easy
@@ -199,11 +209,31 @@
"@bazel_tools//platforms:linux",
"@bazel_tools//platforms:x86_64",
]
-_VERBOSE = True
+_VERBOSE = False
def _impl(ctx):
"""Core implementation of _rbe_autoconfig repository rule."""
+ # Resolve the Bazel version to use
+ bazel_version = None
+ bazel_rc_version = None
+ if ctx.attr.bazel_version == "local":
+ bazel_version = str(extract_version_number(ctx.attr.bazel_version_fallback))
+ rc = parse_rc(native.bazel_version)
+ bazel_rc_version = rc if rc != -1 else None
+ if ctx.attr.bazel_version != "local":
+ bazel_version = ctx.attr.bazel_version
+ bazel_rc_version = ctx.attr.bazel_rc_version
+
+ # Deal with the simple case first: if user picks rbe-ubuntu 16_04 container and
+ # a config exists for the current version of Bazel, just create aliases
+ if (ctx.attr.use_checked_in_confs and
+ ctx.attr.registry == _RBE_UBUNTU_REGISTRY and
+ ctx.attr.repository == _RBE_UBUNTU_REPO):
+ if _use_standard_config(ctx, bazel_version, bazel_rc_version, ctx.attr.revision):
+ # If a standard config was found we are done and can just return.
+ return
+
# Perform some safety checks
_validate_host(ctx)
project_root = ctx.os.environ.get(_RBE_AUTOCONF_ROOT, None)
@@ -227,21 +257,10 @@
name = ctx.attr.name
outputs_tar = ctx.attr.name + "_out.tar"
- image_name = ctx.attr.registry + "/" + ctx.attr.repository + "@" + ctx.attr.digest
-
# Pull the image using 'docker pull'
+ image_name = ctx.attr.registry + "/" + ctx.attr.repository + "@" + ctx.attr.digest
_pull_image(ctx, image_name)
- bazel_version = None
- bazel_rc_version = None
- if ctx.attr.bazel_version == "local":
- bazel_version = str(extract_version_number(ctx.attr.bazel_version_fallback))
- rc = parse_rc(native.bazel_version)
- bazel_rc_version = rc if rc != -1 else None
- if ctx.attr.bazel_version != "local":
- bazel_version = ctx.attr.bazel_version
- bazel_rc_version = ctx.attr.bazel_rc_version
-
# Get the value of JAVA_HOME to set in the produced
# java_runtime
java_home = _get_java_home(ctx, image_name)
@@ -298,6 +317,53 @@
if not ctx.which("tar"):
fail("Cannot run rbe_autoconfig as 'tar' was not found on the path.")
+# Checks if a standard published config can be used for the given revision
+# of the rbe ubuntu 16_04 container. If so, produces BUILD files with aliases
+# for all the required toolchain / platform targets. Otherwise returns false.
+def _use_standard_config(ctx, bazel_version, bazel_rc_version, revision):
+ # If rc version is required, simply return
+ if bazel_rc_version:
+ return False
+
+ # Verify a toolchain config exists for the given version of Bazel and the
+ # given revision of the container
+ config_version = public_rbe_ubuntu16_04_config_version().get(revision, None)
+ config_found = False
+ for b_v in config_to_bazel_versions().get(config_version):
+ if b_v == bazel_version:
+ config_found = True
+ break
+ if not config_found:
+ return False
+
+ # If a config is found, create the BUILD files with the aliases
+ template = ctx.path(Label("@bazel_toolchains//rules:BUILD.std_container.tpl"))
+ jdk = "@bazel_toolchains//configs/ubuntu16_04_clang/%s:jdk8" % config_version
+ cc_toolchain = "@bazel_toolchains//configs/ubuntu16_04_clang/%s/bazel_%s/cpp:cc-toolchain-clang-x86_64-default" % (config_version, bazel_version)
+ platform = "@bazel_toolchains//configs/ubuntu16_04_clang/%s:rbe_ubuntu1604" % config_version
+ ctx.template(
+ _PLATFORM_DIR + "/BUILD",
+ template,
+ {
+ "%{jdk}": jdk,
+ "%{cc-toolchain}": cc_toolchain,
+ "%{platform}": platform,
+ },
+ False,
+ )
+
+ template = ctx.path(Label("@bazel_toolchains//rules:BUILD.std_toolchain.tpl"))
+ toolchain = "@bazel_toolchains//configs/ubuntu16_04_clang/%s/bazel_%s/default:toolchain" % (config_version, bazel_version)
+ ctx.template(
+ _RBE_CONFIG_DIR + "/BUILD",
+ template,
+ {
+ "%{toolchain}": toolchain,
+ },
+ False,
+ )
+ return True
+
# Pulls an image using 'docker pull'.
def _pull_image(ctx, image_name):
print("Pulling image.")
@@ -551,9 +617,9 @@
"must be available in https://releases.bazel.build."),
),
"bazel_version_fallback": attr.string(
- default = "0.20.0",
+ default = "0.21.0",
doc = ("Version to fallback to if not provided explicitly and local " +
- "is non release version."),
+ "is non-release version."),
),
"digest": attr.string(
mandatory = True,
@@ -625,6 +691,11 @@
"example, [\"@bazel_tools//platforms:linux\"]. Default " +
" is set to values for rbe-ubuntu16-04 container."),
),
+ "use_checked_in_confs": attr.bool(
+ default = True,
+ doc = ("If set to False the checked-in configs in bazel-toolchains" +
+ "repo are ignored and a container is allways pulled."),
+ ),
},
environ = [
_RBE_AUTOCONF_ROOT,
@@ -645,7 +716,8 @@
revision = None,
registry = None,
repository = None,
- target_compatible_with = None):
+ target_compatible_with = None,
+ use_checked_in_confs = True):
""" Creates a repository with toolchain configs generated for a container image.
This macro wraps (and simplifies) invocation of _rbe_autoconfig rule.
@@ -686,6 +758,10 @@
target_compatible_with: List of constraints to add to the produced
toolchain target (e.g., ["@bazel_tools//platforms:linux"]) in the
target_compatible_with attr.
+ use_checked_in_confs: Default: True. Try to look for checked in configs
+ before generating them. If set to false the rule will allways attempt
+ to generate the configs by pulling a toolchain container and running
+ Bazel inside.
env: dict. Optional. Additional env variables that will be set when
running the Bazel command to generate the toolchain configs.
Set to values for rbe-ubuntu16-04 container.
@@ -729,4 +805,5 @@
repository = repository,
revision = revision,
target_compatible_with = target_compatible_with,
+ use_checked_in_confs = use_checked_in_confs,
)
diff --git a/rules/toolchain_containers.bzl b/rules/toolchain_containers.bzl
index 2a0cc14..c3c312f 100644
--- a/rules/toolchain_containers.bzl
+++ b/rules/toolchain_containers.bzl
@@ -40,3 +40,15 @@
"r328903": "sha256:59bf0e191a6b5cc1ab62c2224c810681d1326bad5a27b1d36c9f40113e79da7f",
"r327695": "sha256:b940d4f08ea79ce9a07220754052da2ac4a4316e035d8799769cea3c24d10c66",
}
+
+# Map from revisions of rbe ubuntu16_04 to corresponding major container versions.
+# Kept here as it needs to be updated along with the def above.
+def public_rbe_ubuntu16_04_config_version():
+ return {
+ "r346485": "1.1",
+ "r342117": "1.1",
+ "r340178": "1.1",
+ "r337145": "1.0",
+ "r328903": "1.0",
+ "r327695": "1.0",
+ }
