Add remaining changes for #246
See https://github.com/bazelbuild/continuous-integration/blob/master/buildkite/README.md