Using ci.bazel.io

Bazel CI tests Bazel and a variety of open-source project that use Bazel. The projects using Bazel are also used to validate Bazel changes.

Bazel CI is testing both on presubmit (on a pending change) or on postsubmit (on the master branch). It also handles the release project of Bazel and the performance benchmarks.

If you wish to add or modify a configuration for one of the project tested on Bazel CI, go see the project owner documentation.

Every project that runs on Bazel CI is run on postsubmit. It is done using the GitHub API and handled automatically if bazel-io has write access to the repository.

The result of a build can be either:

  • Sucesss (job is green).
  • Unstable (job is yellow). Some tests failed. Blue Ocean View1 will show the failing platforms in Pipeline view, and list of failing tests in Tests view.
  • Failed (job is red). Compilation failed, or configuration files broken. Blue Ocean View1 will show the build breakage. If it does not fall back to the full console log.

1 Open Blue Ocean view with the button on the left of the job view.

Tips:

  • Tests logs are available under the artifacts list (<joburl>/artifact, e.g. http://ci.bazel.io/job/bazel-tests/lastCompletedBuild/artifact/).
  • Flaky tests can be analyzed with the Test Results Analyzer (available in the normal job view on the side menu) which show history per tests.
  • The “Pipeline Steps” button on the side menu on a job view let you examine each step of the Jenkins pipeline one by one. Looking for the enclosing workspace or node start step of another step give you access to the workspace of that step.

Current limitations:

  • Jenkins Blue Ocean UI has no good way to mark an unstable step so if a platform stage fails without clear sub-step failing look for the last shell step in the platform stage view.
  • Tests are not ordered by platforms in the test view.

Presubmit

The Bazel CI is able to run presubmit tests of changes from GitHub and from Gerrit.

A GitHub pull request is tested when an pull request admin comments test this please or retest this please. This presubmit job is also automatically triggered when an admin pushes a pull request.

  • Admins for pull request are the people from either the bazelbuild organization or the google organization.
  • Jenkins identifies those people as admins whose visibility is public. To make someone public, go to the Bazel organization page (or the Google organization page respectively), look for the person's name, and change their visibility to public.

To vet pull request, bazel-io must have write access to the repository (automatic if in the bazelbuild organization).

Tests on a Gerrit code review are triggered when someone marks the code review as Presubmit-Ready+1. It will update the review thread with the link to the test results and mark the code review as Verified+1 or Verified-1 depending on the result of the test. To retrigger a test, simply reset the Presubmit-Ready label.

The output should be read the same way as the output of the postsubmit.

In addition to pre- and postsubmit tests for an individual change, the Bazel CI performs a “global test” which builds Bazel from a branch, and uses that build of Bazel to run all the other jobs on the Bazel CI.

If it succeed to build Bazel (if it is not red), it produces a report comparing the global test results of this build of Bazel with the global test results from the latest release of Bazel.

This report can be found at http://ci.bazel.io/job/bazel/job/<nightly|release|presubmit>/<buildNumber>/Downstream_projects/, for instance for the last nigthly run it will be at http://ci.bazel.io/job/bazel/job/nightly/lastBuild/Downstream_projects/.

The way to read that report is:

  • Every newly failing jobs are problematic and likely to indicate a failure due to a Bazel change. It cause the build to be unstable (yellow).
  • Every already failing jobs means that the job result is no worse, it is generally safe to ignore those failure but we should aim at having 0 of them to make sure we do not hide problem (a build that breaks because of Bazel whereas it was broken before of a project issue).
  • Every passing job can be safely ignored.

Release process

In addition to testing all the jobs, the global test handles pushing the artifacts that are created by a release branch to GCS and to the GitHub release system.

Performance benchmarks

Performance benchmarks run once a day. This job triggers a run of the performance tests on each new commit. The performance for each commit is reported at perf.bazel.build.