commit 1de1e649eea0a5893786b8de678c1a2cdaa81908
author ulfjack <ulfjack@google.com> Tue Feb 19 03:23:11 2019 -0800
committer Copybara-Service <copybara-worker@google.com> Tue Feb 19 03:24:47 2019 -0800
tree 7ddbf714248c97780554406ee9e80998b9461212
parent 2d1186b48c6c7d6bad1d30727b8dd4d05c84688c

Experimental: spawn actions can execute asynchronously

The new code path is only enabled if experimental_async_execution is set
to true. It is also not particularly useful at this time, as it only
covers a subset of spawn actions. However, I think having this checked
in (even without tests) makes it clearer how the subsequent changes need
to work.

My plan for test coverage is to incrementally refactor the code such
that we only have a single code path rather than two, for which the
existing integration tests should provide plenty of coverage.

There are two parallel threads for refactoring:

1. Extend the concept of action continuations to the Action interface
(thereby generalizing the rudimentary interface added here). Unfortunately,
doing that naively will add a third level of continuations (there's one level
in ActionExecutionFunction, and one in SkyframeActionExecutor), which I'm
reluctant to do. I think the ideal solution is to unify the two existing
continuation levels into one and then find a way to extend it to Action.

I am concerned about the current complexity of exception handling in this area,
which probably needs to be simplified in order to be able to straighten the
continuations story.

On the plus side, moving more code into continuations should reduce the
amount of duplicate work we're doing on function restarts, although that
may not be a significant performance win.

2. Start integrating between the new ListenableFuture support in Skyframe and
ActionExecutionFunction. That means we'll start relying on the newly-added
Skyframe support for _all_ builds, not just when the flag is set. All
existing test coverage for shared actions implicitly applies here.

The first step is to do this for shared actions, which seems fine since it
neither affects profiling nor maximum execution load. While builds with a lot
of shared actions may start to run more actions in parallel, they're still
limited by --jobs.

Progress on #6394.

PiperOrigin-RevId: 234572533