[9.2.0] remote: trust in-memory outputs across incremental builds (#30005)

Modified from the original via:
* using a distinct version for the action cache that can't collide with
future versions at HEAD
* including
https://github.com/bazelbuild/bazel/commit/741ee016bd45aa3df7ea8dde109e2fe3d0b5e972
(see https://github.com/bazelbuild/bazel/issues/29976 for context)

Original commit message follows:

In-memory outputs (--experimental_inmemory_{dotd,jdeps}_files) are kept
only in memory and never written to disk. shouldTrustMetadata() gates
trust of a remote output's metadata on shouldDownloadOutput(), which
under --remote_download_outputs=all is unconditionally true, so an
in-memory output's still-alive remote metadata is distrusted and its
action is re-executed on every incremental build (e.g. every CppCompile,
which emits a .d, and every Java compile, which emits a .jdeps): the
action re-executes, hits the remote cache, and re-downloads its outputs,
turning a no-op build into a full re-validation pass.

This is a regression from #27291, which moved these outputs from on-disk
(local, always trusted) to in-memory (remote) metadata.

An in-memory output is indistinguishable from an ordinary remote output
that an earlier build left unmaterialized (e.g. under
--remote_download_outputs=minimal): both are remote with no contents
proxy. So mark in-memory outputs when their metadata is injected, carry
the bit through the action cache (bumping its version), and trust a
marked output via its TTL in shouldTrustMetadata(). This is read from
the metadata alone, so it holds in both ActionCacheChecker and
FilesystemValueChecker, and on a cold server. Ordinary unmaterialized
remote outputs stay unmarked and are still re-fetched, so switching to
--remote_download_outputs=all correctly materializes them.

The regression test covers a warm rebuild, a rebuild after a server
restart, and the minimal -> all transition. Also manually tested with a
full Chromium build - the second build with a warm server went from
re-executing 72k compile actions in ~216s down to ~0.9s with 0 actions.
Verified again with a cold server, which also re-ran 0 actions.

Fixes #29313

RELNOTES: None

Closes #29993.

PiperOrigin-RevId: 937976000
Change-Id: I1fb24ae4db407101ebb4f4c203f258bd7c15f51f
(cherry picked from commit 4c929580dc50a02398fde9a0c9046bd8bf982960)

Fixes #29995

Co-authored-by: Philipp Wollermann <philwo@google.com>
15 files changed
tree: f2991ccdcc3092cfc3684547879fa7c79bedf0ae
  1. .bazelci/
  2. .github/
  3. docs/
  4. examples/
  5. scripts/
  6. site/
  7. src/
  8. third_party/
  9. tools/
  10. .bazelrc
  11. .bazelversion
  12. .gitattributes
  13. .gitignore
  14. AUTHORS
  15. bazel_downloader.cfg
  16. BUILD
  17. CHANGELOG.md
  18. CODE_OF_CONDUCT.md
  19. CODEOWNERS
  20. combine_distfiles.py
  21. combine_distfiles_to_tar.sh
  22. compile.sh
  23. CONTRIBUTING.md
  24. CONTRIBUTORS
  25. distdir.bzl
  26. extensions.bzl
  27. LICENSE
  28. maven_install.json
  29. MODULE.bazel
  30. MODULE.bazel.lock
  31. README.md
  32. repositories.bzl
  33. requirements.txt
  34. SECURITY.md
README.md

Bazel

{Fast, Correct} - Choose two

Build and test software of any size, quickly and reliably.

  • Speed up your builds and tests: Bazel rebuilds only what is necessary. With advanced local and distributed caching, optimized dependency analysis and parallel execution, you get fast and incremental builds.

  • One tool, multiple languages: Build and test Java, C++, Android, iOS, Go, and a wide variety of other language platforms. Bazel runs on Windows, macOS, and Linux.

  • Scalable: Bazel helps you scale your organization, codebase, and continuous integration solution. It handles codebases of any size, in multiple repositories or a huge monorepo.

  • Extensible to your needs: Easily add support for new languages and platforms with Bazel's familiar extension language. Share and re-use language rules written by the growing Bazel community.

Getting Started

Documentation

Reporting a Vulnerability

To report a security issue, please email security@bazel.build with a description of the issue, the steps you took to create the issue, affected versions, and, if known, mitigations for the issue. Our vulnerability management team will respond within 3 working days of your email. If the issue is confirmed as a vulnerability, we will open a Security Advisory. This project follows a 90 day disclosure timeline.

Contributing to Bazel

See CONTRIBUTING.md

Build status