Add Bazel docs on dynamic execution.
RELNOTES: None.
PiperOrigin-RevId: 388423466
diff --git a/site/_includes/documentation-sidebar/master/sidebar.html b/site/_includes/documentation-sidebar/master/sidebar.html
index e439d19..98e0995 100644
--- a/site/_includes/documentation-sidebar/master/sidebar.html
+++ b/site/_includes/documentation-sidebar/master/sidebar.html
@@ -33,6 +33,7 @@
</ul>
</li>
<li><a href="/versions/{{ current_version }}/remote-execution-ci.html">Remote Execution and CI</a></li>
+ <li><a href="/versions/{{ current_version }}/dynamic-execution.html">Dynamic Execution</a></li>
<li>
<a class="sidebar-nav-heading" data-toggle="collapse"
href="#remote-caching-menu" aria-expanded="false"
diff --git a/site/assets/dyn-trace-alldynamic.png b/site/assets/dyn-trace-alldynamic.png
new file mode 100644
index 0000000..fe36b25
--- /dev/null
+++ b/site/assets/dyn-trace-alldynamic.png
Binary files differ
diff --git a/site/assets/dyn-trace-javaconly.png b/site/assets/dyn-trace-javaconly.png
new file mode 100644
index 0000000..2ae41e5
--- /dev/null
+++ b/site/assets/dyn-trace-javaconly.png
Binary files differ
diff --git a/site/docs/creating-workers.md b/site/docs/creating-workers.md
index fab339d..e7eec53 100644
--- a/site/docs/creating-workers.md
+++ b/site/docs/creating-workers.md
@@ -35,8 +35,6 @@
If your program upholds these requirements, it can be used as a persistent worker!
-
-
### Work requests
A `WorkRequest` contains a list of arguments to the worker, a list of
@@ -55,6 +53,12 @@
}
```
+The optional `verbosity` field can be used to request extra debugging output
+from the worker. It is entirely up to the worker what and how to output. Higher
+values indicate more verbose output. Passing the `--worker_verbose` flag to
+Bazel sets the `verbosity` field to 10, but smaller or larger values can be
+used manually for different amounts of output.
+
### Work responses
A `WorkResponse` contains a request id, a zero or nonzero exit
@@ -73,7 +77,7 @@
}
```
-As per the norm for protobufs, the fields are optional. However, Bazel requires
+As per the norm for protobufs, all fields are optional. However, Bazel requires
the `WorkRequest` and the corresponding `WorkResponse`, to have the same request
id, so the request id must be specified if it is nonzero. This is a valid
`WorkResponse`.
@@ -176,7 +180,6 @@
The rule that uses the worker creates actions for the worker to perform. These
actions have a couple of requirements.
-
* The _“arguments”_ field. This takes a list of strings, all but the last
of which are arguments passed to the worker upon startup. The last element in
the “arguments” list is a `flag-file` (@-preceded) argument. Workers read
@@ -200,7 +203,6 @@
* Temporary files generated in the course of the action should be saved to the
worker's directory. This enables sandboxing.
-
**Note**: To pass an argument starting with a literal `@`, start the argument
with `@@` instead. If an argument is also an external repository label, it will
not be considered a flagfile argument.
diff --git a/site/docs/dynamic-execution.md b/site/docs/dynamic-execution.md
new file mode 100644
index 0000000..40d370d
--- /dev/null
+++ b/site/docs/dynamic-execution.md
@@ -0,0 +1,136 @@
+---
+layout: documentation
+title: Dynamic Execution
+---
+
+# Dynamic Execution
+
+__Dynamic execution__ is a feature in Bazel
+[since version 0.21](https://blog.bazel.build/2019/02/01/dynamic-spawn-scheduler.html),
+where local and remote execution of the same action are started in parallel,
+using the output from the first branch that finishes, cancelling the other
+branch. It combines the execution power and/or large shared cache of a remote
+build system with the low latency of local execution, providing the best of both
+worlds for clean and incremental builds alike.
+
+This page describes how to enable, tune, and debug dynamic execution. If you
+have both local and remote execution set up and are trying to adjust Bazel
+settings for better performance, this page is for you. If you don't already have
+remote execution set up, go to the Bazel
+[Remote Execution Overview](remote-execution.html) first.
+
+## Enabling dynamic execution?
+
+The dynamic execution module is part of Bazel, but to make use of dynamic
+execution, you must already be able to compile both locally and remotely from
+the same Bazel setup.
+
+To enable the dynamic execution module, pass the `--internal_spawn_scheduler`
+flag to Bazel. This adds a new execution strategy called `dynamic`. You can now
+use this as your strategy for the mnemonics you want to run dynamically, e.g.
+`--strategy=Javac=dynamic`. See the next section for how to pick which mnemonics
+to enable dynamic execution for.
+
+For any mnemonic using the dynamic strategy, the remote execution strategies are
+taken from the `--dynamic_remote_strategy` flag, and local strategies from the
+`--dynamic_local_strategy` flag. Passing
+`--dynamic_local_strategy=worker,sandboxed` sets the default for the local
+branch of dynamic execution to try with workers or sandboxed execution in that
+order. Passing `--dynamic_local_strategy=Javac=worker` overrides the default for
+the Javac mnemonic only. The remote version works the same way. Both flags can
+be specified multiple times. If an action cannot be executed locally, it is
+executed remotely as normal, and vice-versa.
+
+If your remote system has a cache, the `--experimental_local_execution_delay`
+flag adds a delay in milliseconds to the local execution after the remote system
+has indicated a cache hit. This avoids running local execution when more cache
+hits are likely. The default value is 1000ms, but should be tuned to being just
+a bit longer than cache hits usually take. The actual time depends both on the
+remote system and on how long a round-trip takes. Usually, the value will be the
+same for all users of a given remote system, unless some of them are far enough
+away to add roundtrip latency. You can use the
+[Bazel profiling features](skylark/performance.html#performance-profiling)
+to look at how long typical cache hits take.
+
+Dynamic execution can be used with local sandboxed strategy as well as with
+[persistent workers](/persistent-workers.html). Persistent workers will
+automatically run with sandboxing when used with dynamic execution, and cannot
+use [multiplex workers](/multiplex-worker.html). On Darwin and Windows systems,
+the sandboxed strategy can be slow, you can pass
+`--experimental_reuse_sandbox_directories` to try a new approach speeding up
+sandboxes on these systems.
+
+Dynamic execution can also run with the `standalone` strategy, though since the
+`standalone` strategy must take the output lock when it starts executing, it
+effectively blocks the remote strategy from finishing first. The
+`--experimental_local_lockfree_output` flag enables a way around this problem by
+allowing the local execution to write directly to the output, but be aborted by
+the remote execution, should that finish first.
+
+If one of the branches of dynamic execution finishes first but is a failure, the
+entire action fails. This is an intentional choice to prevent differences
+between local and remote execution from going unnoticed.
+
+For more background on how dynamic execution and its locking works, see Julio
+Merino's excellent
+[blog posts](https://jmmv.dev/series/bazel-dynamic-execution/)
+
+## When should I use dynamic execution?
+
+Dynamic execution obviously requires some form of
+[remote execution system](/remote-execution-services.html). It is not currently
+possible to use a cache-only remote system, as a cache miss would be considered
+a failed action.
+
+Not all types of actions are well suited for remote execution. The best
+candidates are those that are inherently faster locally, for instance through
+the use of [persistent workers](/persistent-workers.html), or those that run
+fast enough that the overhead of remote execution dominates execution time.
+Since each locally executed action locks some amount of CPU and memory
+resources, running actions that don't fall into those categories merely delays
+execution for those that do.
+
+As of release
+[5.0.0-pre.20210708.4](https://github.com/bazelbuild/bazel/releases/tag/5.0.0-pre.20210708.4),
+[performance profiling](/skylark/performance.html#performance-profiling)
+contains data about worker execution, including time spent finishing a work
+request after losing a dynamic execution race. If you see dynamic execution
+worker threads spending significant time acquiring resources, or a lot of time
+in the `async-worker-finish`, you may have some slow local actions delaying the
+worker threads.
+
+<p align="center">
+<img width="596px" alt="Profiling data with poor dynamic execution performance"
+ src="/assets/dyn-trace-alldynamic.png">
+</p>
+
+In the profile above, which uses 8 Javac workers, we see many Javac workers
+having lost the races and finishing their work on the `async-worker-finish`
+threads. This was caused by a non-worker mnemonic taking enough resources to
+delay the workers.
+
+<p align="center">
+<img width="596px" alt="Profiling data with better dynamic execution performance"
+ src="/assets/dyn-trace-javaconly.png">
+</p>
+
+When only Javac is run with dynamic execution, only about half of the started
+workers end up losing the race after starting their work.
+
+The previously recommended `--experimental_spawn_scheduler` flag is deprecated.
+It turns on dynamic execution and sets `dynamic` as the default strategy for all
+mnemonics, which would often lead to these kinds of problems.
+
+## Troubleshooting
+
+Problems with dynamic execution can be subtle and hard to debug, as they can
+manifest only under some specific combinations of local and remote execution.
+The `--experimental_debug_spawn_scheduler` adds extra output from the dynamic
+execution system that can help debug these problems. You can also adjust the
+`--experimental_local_execution_delay` flag and number of remote vs. local jobs
+to make it easier to reproduce the problems.
+
+If you are experiencing problems with dynamic execution using the `standalone`
+strategy, try running without `--experimental_local_lockfree_output`, or run
+your local actions sandboxed. This may slow down your build a bit (see above if
+you're on Mac or Windows), but removes some possible causes for failures.
diff --git a/site/docs/multiplex-worker.md b/site/docs/multiplex-worker.md
index 04a1f0a..34bfb15 100644
--- a/site/docs/multiplex-worker.md
+++ b/site/docs/multiplex-worker.md
@@ -60,14 +60,17 @@
`--strategy=[some_mnemonic]=worker`) or generally at the strategy level (for
example, `--dynamic_local_strategy=worker,standalone`.) No additional flags are
necessary, and `supports-multiplex-workers` takes precedence over
-`supports-workers`, if both are set. A ruleset is encouraged to use multiplex
-workers if possible, to improve performance.
+`supports-workers`, if both are set.
+
+A ruleset is encouraged to use multiplex workers if possible, to reduce memory
+pressure and improve performance. However, multiplex workers are not currently
+compatible with [dynamic execution](/dynamic-execution.html).
### Warning about rare bug
-Due to a rare bug, multiplex workers are currently unstable. Occasionally,
-Bazel hangs indefinitely at the execution phase. If you see this behavior,
-stop the Bazel server and rerun. This delay is probably caused by
+Due to a rare bug, multiplex workers are currently not enabled by default.
+Occasionally, Bazel hangs indefinitely at the execution phase. If you see this
+behavior, stop the Bazel server and rerun. This delay is probably caused by
* Multiplex workers waiting for responses from the worker process that never
comes.
