Enable to use configuration files from the repository
The configuration files are currently checked into the CI system
repository. However, they really belongs to the repository, they
are the "blueprint" of what should be build and tested on the CI system.
The file can be checked in either at .ci/${name}.json or
at scripts/ci/${name}.json.
Change-Id: I2f0dedf7378ccb85605d0aa22d354e74b5a06ff7
diff --git a/README.md b/README.md
index 706eee0..4c7230f 100644
--- a/README.md
+++ b/README.md
@@ -3,6 +3,19 @@
This workspace contains the setup for the continuous integration
system of Bazel. This setup is based on docker images built by bazel.
+## For users of Bazel continuous integration system
+
+If you are a user of the CI system, you might be interested in the
+following document:
+
+* [owners](docs/owners.md): explains how to add a job for a repository
+ owner.
+* [user](docs/user.md): explains how to use the CI system for a Bazel
+ contributor.
+
+
+## For maintainers of Bazel continuous integration system
+
Make sure you have a Bazel installed with a recent enough version of
it. Also make sure [gcloud](https://cloud.google.com/sdk/) and
[docker](https://www.docker.com) are correctly configured on your
@@ -19,5 +32,3 @@
* [workflow](docs/workflow.md): explains the CI workflow, and
how you can test local changes with jenkins-staging
* [jobs](docs/jobs.md): explains what jobs are running on the CI
-* [user](docs/user.md): explains how to use the CI system for a Bazel
- contributor.
diff --git a/docs/owner.md b/docs/owner.md
new file mode 100644
index 0000000..1908d9e
--- /dev/null
+++ b/docs/owner.md
@@ -0,0 +1,222 @@
+# Bazel continuous integration for project owners
+
+## Adding a project
+
+To add a project to Bazel CI that is on the `bazelbuild` organization
+on GitHub, add a `bazel_github_job` in the `jenkins/jobs/BUILD`
+file and send a code review to a CI admin (see
+`jenkins/config.bzl`). In the permission of your GitHub repository, you
+want to add the team [`robots`](https://github.com/orgs/bazelbuild/teams/robot)
+(or [`bazel-io`](https://github.com/bazel-io) for repository outside of
+the `bazelbuild` organization) to have write access to the repository.
+
+By default, if the project is in the bazelbuild org and does not need
+special tweaks you should add the project to the comprehension
+list in the BUILD file. Otherwise you can add a `bazel_github_job` that
+takes the following parameters:
+
+* `name` is the name of the job as it will appear in Jenkins.
+* `branch` is the branch to build and test, `master` by default.
+* `project` is the name of the project, by default it is set to the
+ value of `name`. Usefull when renaming a project but keeping the job
+ history.
+* `org` is the name of the organization on github.
+* `git_url` is the URL to the git repository, default to `https://github.com/<org>/<project>`.
+* `project_url` is the URL to the project, default to the value of `git_url`.
+* `workspace` is the directory where the workspace is relative to the
+ root of the git repository, by default `.`
+* `config` can be used to specify a different default configuration
+ file. By default it is `//jenkins/build_defs:default.json`. Normally,
+ it is no longer needed since the configuration can be changed using a
+ file in the repository (see next section).
+* `enable_trigger` enable postsubmit test (enabled by default).
+* `gerrit_project` specifies a project on the
+ [Bazel Gerrit server](https://bazel-review.googlesource.com) that
+ mirrors the GitHub project and will be used to trigger presubmit from
+ Gerrit.
+* `enabled` is used to deactivate a project, `True` by default.
+* `pr_enabled` can be set to `False` to turn off presubmit from Pull Requests.
+* `run_sequential` can be used to make the job non concurrent, meaning
+ that each configuration will run in sequence from the other
+ one. Useful if the job use some exclusive resource like [Sauce
+ Labs](https://wiki.saucelabs.com/).
+* `sauce_enabled` activate [Sauce Labs](https://wiki.saucelabs.com/) support.
+
+A variation of `bazel_github_job`, `bazel_git_job` can be used to
+specify a job that runs from a random Git repository instead of a
+GitHub repository. It supports the same non GitHub specific argument
+but need either `git_url` or `project_url` to be specified.
+
+## Customizing a project
+
+By default, the CI system tries to build `//...` then tests `//...` on Darwin and Linuxes.
+To change how the project must be built, a JSON description file can be added to the
+repository under `.ci/<name>.json` or `scripts/ci/<name>.json` (where `<name>` is
+the name of the project declared in `jobs.bzl`).
+
+This file contains a list of configurations to build and test, each
+configuration is a dictionary containing platform description (as
+key-value pairs, e.g. `node: "windows-x86\_64"`), a list of parameters
+under the key `parameters` and a list of sub-configurations under the
+key `configurations`.
+
+A simple configuration with one platform would then look like:
+
+```javascript
+[
+ {"node": "linux-x86_64"}
+]
+```
+
+This configuration would build and test on a node that has the label
+`linux-x86\_64` with the default set of parameters (i.e. build `//...`
+then tests `//...`). To change the target to build and test, this can
+be set through `parameters`:
+
+```javascript
+[
+ {
+ "node": "linux-x86_64",
+ "parameters": {
+ "targets": ["//my:target"],
+ "tests": ["//my:test"],
+ }
+ }
+]
+```
+
+This example used `targets` and `tests` parameters to set the targets
+to respectively build and tests.
+
+Adding a platform can be done by simply adding another configuration:
+
+```javascript
+[
+ {
+ "node": "linux-x86_64",
+ "parameters": {
+ "targets": ["//my:target"],
+ "tests": ["//my:test"],
+ }
+ },
+ {
+ "node": "darwin-x86_64",
+ "parameters": {
+ "targets": ["//my:target"],
+ "tests": ["//my:test"],
+ }
+ }
+]
+```
+
+But that's a lot of duplication, so we can use sub-configurations instead:
+
+```javascript
+[
+ {
+ "configurations": [
+ {"node": "linux-x86_64"},
+ {"node": "darwin-x86_64"},
+ ],
+ "parameters": {
+ "targets": ["//my:target"],
+ "tests": ["//my:test"],
+ }
+ }
+]
+```
+
+Each child configuration inherits parent configuration description. The
+child configurations get factored with the parent configuration to create N
+configurations that inherit the parameters and descriptor of the parent
+configuration. If a child configuration specifies a value already present in the
+parent configuration, the parent configuration value will be ignored and the child
+configuration value will be used.
+
+For example:
+
+```javascript
+[
+ {
+ "descriptor": "yeah",
+ "parameters": ["targets": ["//:target1"]],
+ "configurations": [
+ {
+ "descriptor2": "a",
+ "parameters": ["tests": ["//:test"]]
+ },
+ {
+ "descriptor2": "b",
+ "parameters": ["targets": ["//:target2"], "tests": ["//:test"]]
+ }
+ ]
+ }
+]
+```
+
+would expand to the following configurations:
+
+```javascript
+[
+ {
+ "descriptor": "yeah",
+ "descriptor2": "a",
+ "parameters": ["targets": ["//:target1"], "tests": ["//:test"]]
+ },
+ {
+ "descriptor": "yeah",
+ "descriptor2": "b",
+ "parameters": ["targets": ["//:target2"], "tests": ["//:test"]]
+ }
+]
+```
+
+## Reference
+
+### Configuration descriptor keys
+
+Descriptor keys that have special meaning:
+
+* `node` is a label that describe the platform to run on, e.g.:
+ `linux-x86_64`, `windows-x86_64`, `freebsd-11`, ... The complete
+ list of currently connected nodes is available on
+ http://ci.bazel.io/computer/ and each nodes can be selected either
+ by name or by label (to see list of labels of a specific node, click
+ on it on the Jenkins UI).
+* `variation` is a variation of the Bazel binary, e.g. `-jdk7`.
+
+More descriptor keys can be used to specify more
+configuration combination but they won't have any special effects.
+
+
+### Parameter keys
+
+Possible parameters:
+
+* `configure`: a list of shell (batch on Windows) comnands to execute
+ before the build.
+* `targets`: the list of targets to build.
+* `tests`: the list of targets to test, can be expressed as bazel query expression.
+* `build_tag_filters`: list of tags to filter build on.
+* `test_tag_filters`: list of tags to filter test on (`-noci,-manual`
+ are automatically added).
+* `build_opts`: list of options to add to the bazelrc as `build`
+ options, note that they impact testing too.
+* `test_opts`: list of options to add to the bazelrc as `test` options.
+
+
+## Bazel bootstrap
+
+__For Bazel developers.__
+
+The Bazel project itself has a separate configuration file for
+creating release artifacts. It is stored under
+`scripts/ci/bootstrap.json`.
+
+This file follows the same syntax but accepts different parameters:
+
+* `archive`: list of files to archive as a map of target, new name. `%{release_name}` string will
+ be replaced by the release name. An empty list means we do not
+ archive anything (for non release build).
+* `stash`: list of artifacts to stash (to be released / push but no need to keep it forever)
+* `targets`: list of targets to build, in addition to //src:bazel.
diff --git a/docs/user.md b/docs/user.md
index 2c61a96..0574096 100644
--- a/docs/user.md
+++ b/docs/user.md
@@ -8,6 +8,10 @@
postsubmit (on the master branch). It also handles the release project
of Bazel and the performance benchmarks.
+If you wish to add or modify a configuration for one of the project
+tested on Bazel CI, go see the
+[project owner documentation](owner.md).
+
## Postsubmit
Every project that runs on Bazel CI is run on postsubmit. It is done
diff --git a/jenkins/lib/vars/bazelCiConfiguredJob.groovy b/jenkins/lib/vars/bazelCiConfiguredJob.groovy
index 4c10dff..bb01fb1 100644
--- a/jenkins/lib/vars/bazelCiConfiguredJob.groovy
+++ b/jenkins/lib/vars/bazelCiConfiguredJob.groovy
@@ -11,7 +11,6 @@
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
-import build.bazel.ci.BazelConfiguration
// Work around https://issues.jenkins-ci.org/browse/JENKINS-27421
@NonCPS
@@ -25,10 +24,15 @@
def createJobsFromConfiguration(config, configNames, script) {
def cfgs = []
- def flattenConfigurations = BazelConfiguration.flattenConfigurations(
- BazelConfiguration.parse(config.configuration), config.restrict_configuration)
- def entrySet = flattenConfigurations.entrySet().toArray();
- flattenConfiguration = null
+ def name = currentBuild.projectName
+ // Convert to an array to avoid serialization issue with Jenkins
+ def entrySet = readConfiguration(files: [".ci/${name}.json", "scripts/ci/${name}.json"],
+ repository: config.repository,
+ branch: config.branch,
+ refspec: config.refspec,
+ default_configuration: config.configuration,
+ restrict_configuration: config.restrict_configuration
+ ).entrySet().toArray()
for (int k = 0; k < entrySet.length; k++) {
def params = entrySet[k].value
def conf = entrySet[k].key
diff --git a/jenkins/lib/vars/bootstrapBazelAll.groovy b/jenkins/lib/vars/bootstrapBazelAll.groovy
index 64135c3..daa57c4 100644
--- a/jenkins/lib/vars/bootstrapBazelAll.groovy
+++ b/jenkins/lib/vars/bootstrapBazelAll.groovy
@@ -10,8 +10,6 @@
// See the License for the specific language governing permissions and
// limitations under the License.
-import build.bazel.ci.BazelConfiguration
-
// A step to bootstrap bazel on several platforms
def call(config = [:]) {
def variation = config.get("variation", "")
@@ -22,11 +20,14 @@
def restrict_configuration = config.get("restrict_configuration", [:])
def jobs = [:]
- def flattenConfigurations = BazelConfiguration.flattenConfigurations(
- BazelConfiguration.parse(config.configuration), config.restrict_configuration)
-
- // Avoid serialization
- def entrySet = flattenConfigurations.entrySet().toArray()
+ // Convert to an array to avoid serialization issue with Jenkins
+ def entrySet = readConfiguration(files: ["scripts/ci/bootstrap.json"],
+ repository: config.repository,
+ branch: config.branch,
+ refspec: config.refspec,
+ default_configuration: config.get("configuration", null),
+ restrict_configuration: config.restrict_configuration
+ ).entrySet().toArray()
def values = []
def keys = []
for (int k = 0; k < entrySet.length; k++) {
diff --git a/jenkins/lib/vars/readConfiguration.groovy b/jenkins/lib/vars/readConfiguration.groovy
new file mode 100644
index 0000000..1318af5
--- /dev/null
+++ b/jenkins/lib/vars/readConfiguration.groovy
@@ -0,0 +1,57 @@
+// Copyright (C) 2017 The Bazel Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+import build.bazel.ci.BazelConfiguration
+
+// Step to read a configuration from the repository
+// Parameters:
+// - repository, branch, refspec: which repository to fetch, see recursiveGit
+// - files: the list of files to read from the repository (if the first one
+// does not exists, try the second one and so on...), default to .ci/bazel.json.
+// - default_configuration: default json content to use if the file cannot be found
+// - restrict_configuration: restriction to the returned configuration, see
+// BazelConfiguration.flattenConfiguration
+def call(config = [:]) {
+ def conf = null
+ def files = config.get("files", [".ci/bazel.json"])
+ def filename = null
+ if ("repository" in config) {
+ node {
+ recursiveGit(repository: config.repository,
+ branch: config.get("branch", "master"),
+ refspec: config.get("refspec", "+refs/heads/*:refs/remotes/origin/*"))
+ for(int k = 0; k < files.size() && conf == null; k++) {
+ if (fileExists(files[k])) {
+ filename = files[k]
+ conf = readFile(filename)
+ }
+ }
+ }
+ }
+ if (conf == null) {
+ conf = config.get("default_configuration")
+ if (conf == null) {
+ error(
+ """Cannot read configuration file from the repository and no fallback was provided.
+Please check a configuration file under one of: ${files.join ', '}.""")
+ }
+ }
+ try {
+ return BazelConfiguration.flattenConfigurations(
+ BazelConfiguration.parse(conf), config.get("restrict_configuration", [:]))
+ } catch(Exception ex) {
+ error(filename != null ? "Failed to validate configuration (file was ${filename}): ${ex.message}"
+ : "Failed to validate default configuration: ${ex.message}")
+ }
+}
