This is the guide to conducting a Bazel release. This is especially relevant for release managers, but will be of interest to anyone who is curious about the release process.
For future reference and release managers - the release manager playbook should be treated like an IKEA manual. That means: Do not try to be smart, optimize / skip / reorder steps, otherwise chaos will ensue. Just follow it and the end result will be.. well, a usable piece of furniture, or a Bazel release (depending on the manual).
Like aviation and workplace safety regulations, the playbook is written in the tears and blood of broken Bazelisks, pipelines, releases and Git branches. Assume that every step is exactly there for a reason, even if it might not be obvious. If you follow them to the letter, they are not error prone. Errors have only happened in the past, when a release manager thought it's ok to follow them by spirit instead. ;)
-- @philwo
Do this once.
Do these steps once per release.
Each release has a tracking bug (see the list). The last and the next releases should be pinned. The bug includes a “Target RC date”. On that day, create a new release candidate.
A milestone should also be created for the release to keep track of any release blocking issues. (Example)
The release manager adds a comment to the issue with the following content:
# Status of Bazel X.Y - Target baseline: [date] - Expected release date: [date] - [List of release blockers](link-to-milestone) To report a release-blocking bug, please add it to the release blocker milestone above, and cc me. Task list: - [ ] Pick release baseline: - [ ] Create release candidate: - [ ] Check downstream projects: - [ ] [Create draft release announcement](https://docs.google.com/document/d/1wDvulLlj4NAlPZamdlEVFORks3YXJonCjyuQMUQEmB0/edit) - [ ] Send for review the release announcement PR: - [ ] Push the release, notify package maintainers: - [ ] Update the documentation - [ ] Push the blog post - [ ] Update the [release page](https://github.com/bazelbuild/bazel/releases/)
Keep the task list updated and check boxes as you follow the release process. Example.
Create candidates with the release.sh
script. See above how to pick a good baseline commit.
If it's the first candidate for this version, run:
RELEASE_NUMBER=<CURRENT RELEASE NUMBER x.yy.z> BASELINE_COMMIT=01234567890abcdef # From the Setup phase git clone https://github.com/bazelbuild/bazel.git ~/bazel-release-$RELEASE_NUMBER cd ~/bazel-release-$RELEASE_NUMBER scripts/release/release.sh create $RELEASE_NUMBER $BASELINE_COMMIT [CHERRY_PICKS...]
Note that the three-digit version is important: “0.19.0”. not “0.19”.
For cherry-picks, you need --force_rc=N
where N
is the number of the release candidate of $RELEASE_NUMBER
. For example, the first time you do a cherry-pick (after the initial candidate), N will be 2.
scripts/release/release.sh create --force_rc=2 $RELEASE_NUMBER $BASELINE_COMMIT [CHERRY_PICKS...]
If you already did some cherry-picks and you want to add more, use “git log” to find the latest commit (this corresponds to the last cherry-pick commit). Use that as the new baseline and list the new cherry-picks to add on top. Or simply re-use the same baseline and cherrypicks from the previous candidate, and add the new cherrypicks.
scripts/release/release.sh create --force_rc=3 $RELEASE_NUMBER NEW_BASELINE_COMMIT [NEW_CHERRY_PICKS...]
Resolve conflicts if there are any, type exit
when you are done, then the script will continue. WARNING: release.sh create
handles conflicts in a subshell (which is why you need to type exit
).
Editing the release notes is not needed (it will be done later).
Run release.sh push
. This uploads the candidate and starts the release process on BuildKite.
scripts/release/release.sh push
Update GitHub issue with the command that was run and the new candidate name (ie, 0.19.1rc3). Update the issue with the estimated release date, assuming no regression is found.
Check BuildKite results at https://buildkite.com/bazel-trusted/bazel-release. You should see the release-$RELEASE_NUMBER
branch here and a new build running for your release.
Check the postsubmit test run for the release branch to ensure that all tests on all platforms pass with the version you're about to release.
release-$RELEASE_NUMBER
branch in the list. A build should automatically run. Make sure that it passes.When it all looks good, go back to the job in the release pipeline, click “Deploy release artifacts” for the deployment step.
If that worked, click on the “Generate announcement mail text” step to unblock it. If it's the first release candidate, prepare the release announcement (see next section).
Copy & paste the generated text into a new e-mail and send it. If you're creating a new release candidate, reply to the previous e-mail to keep all the information in one thread.
https://docs.google.com/document/d/1wDvulLlj4NAlPZamdlEVFORks3YXJonCjyuQMUQEmB0/view
Trigger a new pipeline in BuildKite to test the release candidate with all the downstream projects.
Look for failing projects in red.
Compare the results with the latest Bazel release:
If a project is failing with release candidate but not with the latest Bazel release, then there's probably a regression in the candidate. Ask the Bazel sheriff if the problem is already noticed. If not, find out the culprit, file an issue at https://github.com/bazelbuild/bazel/issues and mark it as release blocker.
If a project is failing with both release candidate and the latest Bazel release, it could be a breakage from the project itself. Go through the build history (eg. TensorFlow_serving) to confirm this, then file an issue to their owners.
File bugs (TODO: how to find the owner/project link?)
Once issues are fixed, create a new candidate with the relevant cherry-picks.
Once the first candidate is available:
The release manager is responsible for the draft release announcement.
Once the first candidate is available:
After a few days of iteration:
Generate a new identifier: https://bazel.googlesource.com/new-password (and paste the code in your shell). This is only necessary the first time you handle a release.
Push the final release (do not cancel midway):
scripts/release/release.sh release
A CI job is uploading the release artifacts to GitHub. Look for the release workflow on https://buildkite.com/bazel-trusted/bazel-release/. Unblock the steps.
Ensure all binaries were uploaded to GitHub properly.
https://storage.googleapis.com/bazel/$RELEASE_NUMBER/release/index.html
.Update the release bug:
Publish versioned documentation
Fetch the git tag for the release: git fetch --tags
Do a checkout to that tag: git checkout $RELEASE_NUMBER
$ git checkout 0.21.0 Note: checking out '0.21.0'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b <new-branch-name> HEAD is now at defd737761 Release 0.21.0 (2018-12-19)
Install gsutil
and ensure you have access to the bazel-public
GCP project.
Run scripts/docs/generate_versioned_docs.sh
. If you get interrupted, it is safe to re-run the script. This script will build the web assets for the documentation, generate a tarball from them, and push the tarball to Google Cloud Storage.
Update site/_config.yml
and scripts/docs/doc_versions.bzl
with $RELEASE_NUMBER
.
site/_config.yml
: set version:
to $RELEASE_NUMBER
and add $RELEASE_NUMBER
to doc_versions:
.scripts/docs/doc_versions.bzl
: add the following list item to DOC_VERSIONS
Merge the blog post pull request.
Update the release page to replace the generated notes with a link to the blog post.
Close the release-tracking bug. If you need to do a patch release, create a new tracking bug.
Please ping @philwo and @meteorcloudy to copy the release binary to their internal mirror.
Homebrew is a package manager for OS X. This section assumes that you are on a Mac OS machine with homebrew installed.
To update the bazel
recipe on Homebrew, you can send a pull request to https://github.com/Homebrew/homebrew-core/blob/master/Formula/bazel.rb.
Example: https://github.com/Homebrew/homebrew-core/pull/57966
However, usually the Homebrew community takes care of this reasonably quickly, so feel free to skip this step, if you aren't familiar with it.
As of November 2016, this is done by an external contributor, @petemounce on GitHub. Ping him when there's a new release coming out.
As of February 2019, this is done by an external contributor, @excitoon on GitHub. Ping him when there's a new release coming out.
This is done by an external contributor, @vbatts on GitHub. Ping him when there's a new release coming out.