--------------------------------------------------------------------------------

layout: documentation

## title: Persistent workers

# Persistent workers

_Persistent workers_ (often called _workers_ for short) is an
[execution strategy](https://docs.bazel.build/versions/master/user-manual.html#strategy-options)
that decreases start-up overhead, allows more JIT compilation, and enables
caching of for example the abstract syntax trees in the action execution. This
strategy achieves these improvements by sending multiple requests to a
long-running process. Persistent workers are available for
[Java (built-in)](https://cs.opensource.google/bazel/bazel/+/master:src/java_tools/buildjar/java/com/google/devtools/build/buildjar/BazelJavaBuilder.java),
[TypeScript](https://bazelbuild.github.io/rules_nodejs/TypeScript.html),
[Scala](https://github.com/bazelbuild/rules_scala), and more.

## Using persistent workers <a name="usage"></a>

[Bazel 0.27 and higher](https://blog.bazel.build/2019/06/19/list-strategy.html)
by default uses persistent workers when executing builds, though remote
execution takes precedence. For actions that do not support persistent workers,
Bazel falls back to regular workers instead. You can explicitly set your build
to use persistent workers by setting the `worker`
[strategy](user-manual.html#strategy-options) for the applicable tool mnemonics.
As a best practice, this example includes specifying `local` as a fallback to
the `worker` strategy:

```
bazel build //my:target --strategy=Javac=worker,local
```

Using the workers strategy instead of the local strategy can boost compilation
speed significantly, depending on implementation. For Java, we typically see 2–4
times faster builds, sometimes more for incremental compilation. Compiling Bazel
is about 2.5 times as fast with workers. For more details, see the
"[Choosing number of workers](#number-of-workers)" section.

If you also have a remote build environment that matches your local build
environment, you can use the experimental
[_dynamic_ strategy](https://blog.bazel.build/2019/02/01/dynamic-spawn-scheduler.html),
which races a remote execution and a worker execution. To enable the dynamic
strategy, pass the
[--experimental_spawn_scheduler](command-line-reference.html#flag--experimental_spawn_scheduler)
flag. This strategy automatically enables workers, so there is no need to
specify the `worker` strategy, but you can still use `local` or `sandboxed` as
fallbacks.

## Choosing number of workers <a name="number-of-workers"></a>

The default number of worker instances per mnemonic is 4, but can be adjusted
with the
[`worker_max_instances`](command-line-reference.html#flag--worker_max_instances)
flag. There is a trade-off between making good use of the available CPUs and the
amount of JIT compilation and cache hits you get. With more workers, more
targets will pay start-up costs of running non-JITted code and hitting cold
caches. If you have a small number of targets to build, a single worker may give
the best trade-off between compilation speed and resource usage (for example,
see [issue #8586](https://github.com/bazelbuild/bazel/issues/8586). The
`worker_max_instances` flag sets the maximum number of worker instances per
mnemonic and flag set (see below), so in a mixed system you could end up using
quite a lot of memory if you keep the default value. For incremental builds the
benefit of multiple worker instances is even smaller.

This graph shows the from-scratch compilation times for Bazel (target
`//src:bazel`) on a 6-core hyper-threaded Intel Xeon 3.5 GHz Linux workstation
with 64 GB of RAM. For each worker configuration, we ran 5 clean builds and took
the average of the last 4.

<p align="center">
<img width="596px" alt="Graph of performance improvements of clean builds" src="/assets/workers-clean-chart.png">
</p>

For this configuration, two workers give the fastest compile, though at only 14%
improvement compared to one worker. One worker is a good option if you want to
use less memory.

Incremental compilation typically benefits even more. Clean builds are
relatively rare, but changing a single file between compiles is common, in
particular in test-driven development. The above example also has some non-Java
packaging actions to it that can overshadow the incremental compile time.
Recompiling the Java sources only
(`//src/main/java/com/google/devtools/build/lib/bazel:BazelServer_deploy.jar`)
after changing an internal string constant in
[AbstractContainerizingSandboxedSpawn.java](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/sandbox/AbstractContainerizingSandboxedSpawn.java)
gives a 3x speed-up (average of 20 incremental builds with one warmup build
discarded):

<p align="center">
<img width="592px" alt="Graph of performance improvements of incremental builds" src="/assets/workers-incremental-chart.png">
</p>

The speed-up depends on the change being made. We have measured a speed-up of a
factor 6 in the above situation when a commonly used constant was changed.

## Modifying persistent workers<a name="options"></a>

You can pass the
[`--worker_extra_flag`](command-line-reference.html#flag--worker_extra_flag)
flag to specify start-up flags to workers, keyed by mnemonic. For instance,
passing `--worker_extra_flag=javac=--debug` turns on debugging for Javac only.
Only one worker flag can be set per use of this flag, and only for one mnemonic.
Workers are not just created separately for each mnemonic, but also for
variations in their start-up flags. Each combination of mnemonic and start-up
flags is combined into a `WorkerKey`, and for each `WorkerKey` up to
`worker_max_instances` workers may be created. See the next section for how the
action configuration can also specify set-up flags.

You can use the
[`--high_priority_workers`](command-line-reference.html#flag--high_priority_workers)
flag to specify a mnemonic that should be run in preference to normal-priority
mnemonics. This can help prioritize actions that are always in the critical
path. If there are two or more high priority workers executing requests, all
other workers are prevented from running. This flag can be used multiple times.

Passing the
[`--worker_sandboxing`](command-line-reference.html#flag--worker_sandboxing)
flag makes each worker request use a separate sandbox directory for all its
inputs. Setting up the sandbox takes some extra time, especially on MacOS, but
gives a better correctness guarantee.

The
[`--worker_quit_after_build`](command-line-reference.html#flag--worker_quit_after_build)
flag is mainly useful for debugging and profiling. This flag forces all workers
to quit once a build is done. You can also pass
[`--worker_verbose`](command-line-reference.html#flag--worker_verbose) to get
more output about what the workers are doing. Workers store their logs in the
`<outputBase>/bazel-workers` directory.

For Android builds, see details at the
[Android Build Performance page](android-build-performance.html).

## Implementing persistent workers<a name="implementation"></a>

Persistent workers are implemented for multiple languages, including Java,
[TypeScript](https://bazelbuild.github.io/rules_nodejs/TypeScript.html),
[Scala](https://github.com/bazelbuild/rules_scala),
[Kotlin](https://github.com/bazelbuild/rules_kotlin), and more. You can
implement persistent workers in other languages and for other tools, as well.

Each worker is a long-running process started by the Bazel server, which
functions as a _wrapper_ around the actual _tool_ (typically a compiler). In
order to benefit from persistent workers, the tool must support doing a sequence
of compilations, and the wrapper needs to translate between the tool's API and
the request/response format described below. The wrapper must recognize the
`--persistent_worker` command-line flag and only make itself persistent if that
flag is passed, otherwise it must do a one-shot compilation and exit. The same
worker program might be called with and without the `--persistent_worker` flag
in the same build, and is responsible for appropriately spawning and talking to
the tool, as well as shutting down workers on exit. Each worker instance is
assigned (but not chrooted to) a separate working directory under
`<outputBase>/bazel-workers`.

The Bazel server communicates with the worker using stdin/stdout. The
compilation requests are sent as using
[`WorkerRequest`](https://github.com/bazelbuild/bazel/blob/master/src/main/protobuf/worker_protocol.proto)
protocol buffers in standard binary format, and responses are similarly returned
as
[`WorkerResponse`](https://github.com/bazelbuild/bazel/blob/master/src/main/protobuf/worker_protocol.proto)
protocol buffers. Each protocol buffer is preceded by its length in varint
format (see
[`MessageLite.writeDelimitedTo()`](https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/MessageLite.html#writeDelimitedTo-java.io.OutputStream-).
The request proto's `args` field should contain a list of strings that describe
the action to be done. The `inputs` field may contain input file names and their
hash digests, allowing the caching of intermediate results without having to
recompute the digest.

<p class="warning">Because responses are sent on stdout, neither the worker nor the underlying tool should write other messages into that stream.</p>

Writing other things to stdout crashes the worker. Any output that should be
shown to the user can be put in the `output` field of the response, and output
that should be logged should go to stderr. The wrapper should make sure that
what the tool writes on stdout is appropriately redirected.

To enable the `worker` strategy for an action, the `execution_requirements` for
that action must include `{"supports-workers": "1"}`. You can also add a
`worker-key-mnemonic` to the `execution_requirements` section, allowing the
mnemonic for workers to be separate from the mnemonic for the action. This can
be useful when the same executable is used for several mnemonics, though it
limits how much the user can control when to use workers.

The action definition must also contain an `arguments` definition with a
flag-file (`@`-preceded) argument at the end. Any non-flag-file arguments are
_startup flags_ that will be passed to the worker on startup, allowing
configuration common to all requests. The flag-file argument is used to read
arguments for each request, including possible _non-startup flags_. To pass an
argument starting with a literal `@`, start the argument with `@@` instead. If
an argument is also an external repository label, it will not be considered a
flag-file argument.

This example shows a Starlark configuration:

```python
args_file = ctx.actions.declare_file(ctx.label.name + "_args_file")
ctx.actions.write(
    output = args_file,
    content = "\n".join(["-g", "-source", "1.5"] + ctx.files.srcs),
)
ctx.actions.run(
    mnemonic = "SomeCompiler",
    executable = "bin/some_compiler_wrapper",
    inputs = inputs,
    outputs = outputs,
    arguments = [ "-max_mem=4G",  "@%s" % args_file.path],
    execution_requirements = { "supports-workers" : "1" }
)
```

With this definition, the first use of this action would start with executing
the command line `/bin/some_compiler -max_mem=4G --persistent_worker`. A request
to compile `Foo.java` would then look like:

```prototext
arguments: [ "-g", "-source", "1.5", "Foo.java" ]
inputs: [
  {path: "symlinkfarm/input1" digest: "d49a..." },
  {path: "symlinkfarm/input2", digest: "093d..."},
]
```

The worker receives this in binary-encoded protobuffer format on `stdin`. The
`WorkerKey` is derived from the mnemonic and the shared flags, so if this
configuration allowed changing the `max_mem` parameter, a separate worker would
be spawned for each value used. This can lead to excessive memory consumption if
too many variations are used.

Each worker can currently only process one request at a time. The experimental
[multiplex workers](multiplex-worker.html) feature allows using multiple
threads, if the underlying tool is multithreaded and the wrapper is set up to
understand this.

In [this GitHub repo](https://github.com/Ubehebe/bazel-worker-examples), you can
see example worker wrappers written in Java as well as in Python.

## How do workers affect sandboxing? <a name="sandboxing"></a>

Using the `worker` strategy by default does not run the action in a sandbox,
similar to the `local` strategy. You can set the `--worker_sandboxing` flag to
run all workers inside sandboxes, making sure each execution of the tool only
sees the input files it's supposed to have. The tool may still leak information
between requests internally, for instance through a cache. Using the
experimental `dynamic` strategy
[requires workers to be sandboxed](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/exec/SpawnStrategyRegistry.java).

To allow correct use of compiler caches with workers, a digest is passed along
with each input file. Thus the compiler or the wrapper can check if the input is
still valid without having to read the file.

Even when using the input digests to guard against unwanted caching, sandboxed
workers offer less strict sandboxing than a pure sandbox, because the tool may
keep other internal state that has been affected by previous requests.

## Further reading <a name="further-reading"></a>

For more information on persistent workers, see:

*   [Original persistent workers blog post](https://blog.bazel.build/2015/12/10/java-workers.html)
*   [Haskell implementation description](https://www.tweag.io/blog/2019-09-25-bazel-ghc-persistent-worker-internship/)
*   [Blog post by Mike Morearty](https://medium.com/@mmorearty/how-to-create-a-persistent-worker-for-bazel-7738bba2cabb)
*   [Front End Development with Bazel: Angular/TypeScript and Persistent Workers
    w/ Asana](https://www.youtube.com/watch?v=0pgERydGyqo)
*   [Bazel strategies explained](https://jmmv.dev/2019/12/bazel-strategies.html)
*   [Informative worker strategy discussion on the bazel-discuss mailing list](https://groups.google.com/forum/#!msg/bazel-discuss/oAEnuhYOPm8/ol7hf4KWJgAJ)