Re-organizing persistent worker pages PiperOrigin-RevId: 343195325
diff --git a/site/docs/creating-workers.md b/site/docs/creating-workers.md index 952ddaa..49b98d1 100644 --- a/site/docs/creating-workers.md +++ b/site/docs/creating-workers.md
@@ -10,29 +10,38 @@ would benefit from cross-action caching, you may want to implement your own persistent worker to perform these actions. +The Bazel server communicates with the worker using `stdin`/`stdout`. It +supports the use of protocol buffers or JSON strings. Support for JSON is +experimental and thus subject to change. It is guarded behind the +`--experimental_worker_allow_json_protocol` flag. + The worker implementation has two parts: -* The [worker](#making-the-worker), +* The [worker](#making-the-worker). * The [rule that uses the worker](#making-the-rule-that-uses-the-worker). ## Making the worker -A worker upholds a few requirements: +A persistent worker upholds a few requirements: * It reads [WorkRequests](https://github.com/bazelbuild/bazel/blob/6d1b9725b1e201ca3f25d8ec2a730a20aab62c6e/src/main/protobuf/worker_protocol.proto#L35) -from its `stdin`. + from its `stdin`. * It writes [WorkResponses](https://github.com/bazelbuild/bazel/blob/6d1b9725b1e201ca3f25d8ec2a730a20aab62c6e/src/main/protobuf/worker_protocol.proto#L49) -(and only `WorkResponse`s) to its `stdout`. -* It accepts the `--persistent_worker` flag. + (and only `WorkResponse`s) to its `stdout`. +* It accepts the `--persistent_worker` flag. The wrapper must recognize the + `--persistent_worker` command-line flag and only make itself persistent if + that flag is passed, otherwise it must do a one-shot compilation and exit. -If your program upholds these requirements, it can be used as a worker! +If your program upholds these requirements, it can be used as a persistent worker! + + ### Work requests -A `WorkRequest` contains a list of arguments to the worker, a list of path-digest -pairs representing the inputs the worker can access (this isn’t enforced, but -you can use this info for caching), and a request id, which is 0 for singleplex -workers. +A `WorkRequest` contains a list of arguments to the worker, a list of +path-digest pairs representing the inputs the worker can access (this isn’t +enforced, but you can use this info for caching), and a request id, which is 0 +for singleplex workers. ```json { @@ -47,10 +56,12 @@ ### Work responses -A `WorkResponse` should contain the same request id, a zero or nonzero exit -code, and an output string that contains any errors encountered in processing -or executing the request. Workers may write additional output to `stderr`, but -they must only write `WorkResponse`s to `stdout`. +A `WorkResponse` contains a request id, a zero or nonzero exit +code, and an output string that describes any errors encountered in processing +or executing the request. The `output` field contains a short +description; complete logs may be written to the worker's `stderr`. Because +workers may only write `WorkResponses` to `stdout`, it's common for the worker +to redirect the `stdout` of any tools it uses to `stderr`. ```json { @@ -72,6 +83,16 @@ } ``` +**Notes** + +* Each protocol buffer is preceded by its length in `varint` format (see +[`MessageLite.writeDelimitedTo()`](https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/MessageLite.html#writeDelimitedTo-java.io.OutputStream-). +* JSON requests and responses are not preceded by a size indicator. +* JSON requests uphold the same structure as the protobuf, but use standard + JSON. +* Bazel stores requests as protobufs and converts them to JSON using +[protobuf's JSON format](https://cs.opensource.google/protobuf/protobuf/+/master:java/util/src/main/java/com/google/protobuf/util/JsonFormat.java) + ## Making the rule that uses the worker You'll also need to create a rule that generates actions to be performed by the @@ -81,10 +102,10 @@ there are some requirements for the actions it produces. ### Referring to the worker -The rule that uses the worker needs to contain a field that refers to the worker itself, -so you'll need to create an instance of a `\*\_binary` rule to define your -worker. If your worker is called `MyWorker.Java`, this might be the associated -rule: +The rule that uses the worker needs to contain a field that refers to the worker +itself, so you'll need to create an instance of a `\*\_binary` rule to define +your worker. If your worker is called `MyWorker.Java`, this might be the +associated rule: ```python java_binary( @@ -133,13 +154,18 @@ a valid execution requirement, though it’s not required for proto workers, since they are the default. - You can also set a "worker-key-mnemonic" in the execution requirements. This + You can also set a `worker-key-mnemonic` in the execution requirements. This may be useful if you're reusing the executable for multiple action types and want to distinguish actions by this worker. * Temporary files generated in the course of the action should be saved to the worker's directory. This enables sandboxing. + +**Note**: To pass an argument starting with a literal `@`, start the argument +with `@@` instead. If an argument is also an external repository label, it will +not be considered a flagfile argument. + Assuming a rule definition with "worker" attribute described above, in addition to a "srcs" attribute representing the inputs, an "output" attribute representing the outputs, and an "args" attribute representing the worker @@ -157,6 +183,9 @@ arguments=ctx.attr.args + [“@flagfile”] ) ``` + +For another example, see [Implementing persistent workers](persistent-workers.html#implementation). + ## Examples The Bazel code base uses [Java compiler workers](https://github.com/bazelbuild/bazel/blob/a4251eab6988d6cf4f5e35681fbe2c1b0abe48ef/src/java_tools/buildjar/java/com/google/devtools/build/buildjar/BazelJavaBuilder.java), @@ -168,5 +197,6 @@ For an example of a rule that uses a worker, take a look at Bazel's [worker integration test](https://github.com/bazelbuild/bazel/blob/22b4dbcaf05756d506de346728db3846da56b775/src/test/shell/integration/bazel_worker_test.sh#L106). -External contributors have implemented workers in a variety of languages; you -can [find many more examples on GitHub](https://github.com/search?q=bazel+workrequest&type=Code)! +External contributors have implemented workers in a variety of languages; take a +look at [Polyglot implementations of Bazel persistent workers](https://github.com/Ubehebe/bazel-worker-examples). +You can [find many more examples on GitHub](https://github.com/search?q=bazel+workrequest&type=Code)!
diff --git a/site/docs/persistent-workers.md b/site/docs/persistent-workers.md index 27a3fc8..cb8e4c2 100644 --- a/site/docs/persistent-workers.md +++ b/site/docs/persistent-workers.md
@@ -3,25 +3,38 @@ title: Persistent workers --- -# Persistent workers +# Overview -_Persistent workers_ (often called _workers_ for short) is an +A persistent worker is a long-running process started by the Bazel server, which +functions as a _wrapper_ around the actual _tool_ (typically a compiler), or is +the _tool_ itself. In order to benefit from persistent workers, the tool must +support doing a sequence of compilations, and the wrapper needs to translate +between the tool's API and the request/response format described below. The same +worker might be called with and without the `--persistent_worker` flag +in the same build, and is responsible for appropriately starting and talking to +the tool, as well as shutting down workers on exit. Each worker instance is +assigned (but not chrooted to) a separate working directory under +`<outputBase>/bazel-workers`. + +Using persistent workers is an [execution strategy](https://docs.bazel.build/versions/master/user-manual.html#strategy-options) that decreases start-up overhead, allows more JIT compilation, and enables caching of for example the abstract syntax trees in the action execution. This strategy achieves these improvements by sending multiple requests to a -long-running process. Persistent workers are available for -[Java (built-in)](https://cs.opensource.google/bazel/bazel/+/master:src/java_tools/buildjar/java/com/google/devtools/build/buildjar/BazelJavaBuilder.java), +long-running process. + +Persistent workers are implemented for multiple languages, including Java, [TypeScript](https://bazelbuild.github.io/rules_nodejs/TypeScript.html), -[Scala](https://github.com/bazelbuild/rules_scala), and more. +[Scala](https://github.com/bazelbuild/rules_scala), +[Kotlin](https://github.com/bazelbuild/rules_kotlin), and more. ## Using persistent workers <a name="usage"></a> [Bazel 0.27 and higher](https://blog.bazel.build/2019/06/19/list-strategy.html) -by default uses persistent workers when executing builds, though remote +uses persistent workers by default when executing builds, though remote execution takes precedence. For actions that do not support persistent workers, -Bazel falls back to regular workers instead. You can explicitly set your build -to use persistent workers by setting the `worker` +Bazel falls back to starting a tool instance for each action. You can explicitly +set your build to use persistent workers by setting the `worker` [strategy](user-manual.html#strategy-options) for the applicable tool mnemonics. As a best practice, this example includes specifying `local` as a fallback to the `worker` strategy: @@ -143,80 +156,9 @@ ## Implementing persistent workers<a name="implementation"></a> -Persistent workers are implemented for multiple languages, including Java, -[TypeScript](https://bazelbuild.github.io/rules_nodejs/TypeScript.html), -[Scala](https://github.com/bazelbuild/rules_scala), -[Kotlin](https://github.com/bazelbuild/rules_kotlin), and more. You can -implement persistent workers in other languages and for other tools, as well. - See the [creating persistent workers](creating-workers.html) page for information on how to make a worker. -Each worker is a long-running process started by the Bazel server, which -functions as a _wrapper_ around the actual _tool_ (typically a compiler). In -order to benefit from persistent workers, the tool must support doing a sequence -of compilations, and the wrapper needs to translate between the tool's API and -the request/response format described below. The wrapper must recognize the -`--persistent_worker` command-line flag and only make itself persistent if that -flag is passed, otherwise it must do a one-shot compilation and exit. The same -worker program might be called with and without the `--persistent_worker` flag -in the same build, and is responsible for appropriately spawning and talking to -the tool, as well as shutting down workers on exit. Each worker instance is -assigned (but not chrooted to) a separate working directory under -`<outputBase>/bazel-workers`. - -The Bazel server communicates with the worker using stdin/stdout. It supports -the use of protocol buffers or JSON strings. Support for JSON is experimental -and thus subject to change. It is guarded behind the -`--experimental_worker_allow_json_protocol` flag. - -When using protobuf, the compilation requests are sent as -[`WorkerRequest`](https://github.com/bazelbuild/bazel/blob/master/src/main/protobuf/worker_protocol.proto) -protocol buffers in standard binary format, and responses are similarly returned -as -[`WorkerResponse`](https://github.com/bazelbuild/bazel/blob/master/src/main/protobuf/worker_protocol.proto) -protocol buffers. Each protocol buffer is preceded by its length in varint -format (see -[`MessageLite.writeDelimitedTo()`](https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/MessageLite.html#writeDelimitedTo-java.io.OutputStream-). - -JSON requests uphold the same structure as the protobuf, but uses standard JSON. -Bazel stores the requests as protobufs and converts them to JSON using -[protobuf's JSON format](https://source.corp.google.com/piper///depot/google3/java/com/google/protobuf/util/JsonFormat.java) -Responses are parsed by a JSON parser into the same structure as the -WorkResponse protobuf, then converted to proto manually. -JSON requests and responses are not preceded by a size indicator. - -The request's `args` field should contain a list of strings that describe -the action to be done. The `inputs` field may contain input file names and their -hash digests, allowing the caching of intermediate results without having to -recompute the digest. - -<p class="warning">Because responses are sent on stdout, neither the worker nor the underlying tool should write other messages into that stream.</p> - -Writing other things to stdout crashes the worker. Any output that should be -shown to the user can be put in the `output` field of the response, and output -that should be logged should go to stderr. The wrapper should make sure that -what the tool writes on stdout is appropriately redirected. - -To enable the `worker` strategy for an action, the `execution_requirements` for -that action must include `{"supports-workers": "1"}`. It can also include a -`requires-worker-protocol` requirement specifying whether Bazel should -communicate with that worker using `json` or `proto`. This is required for JSON -but is optional for proto since proto is the default. You can also add a -`worker-key-mnemonic` to the `execution_requirements` section, allowing the -mnemonic for workers to be separate from the mnemonic for the action. This can -be useful when the same executable is used for several mnemonics, though it -limits how much the user can control when to use workers. - -The action definition must also contain an `arguments` definition with a -flag-file (`@`-preceded) argument at the end. Any non-flag-file arguments are -_startup flags_ that will be passed to the worker on startup, allowing -configuration common to all requests. The flag-file argument is used to read -arguments for each request, including possible _non-startup flags_. To pass an -argument starting with a literal `@`, start the argument with `@@` instead. If -an argument is also an external repository label, it will not be considered a -flag-file argument. - This example shows a Starlark configuration for a worker that uses JSON: ```python @@ -247,12 +189,15 @@ ] ``` -The worker receives this on stdin in JSON format (because -"requires-worker-protocol" is set to JSON, and +The worker receives this on `stdin` in JSON format (because +`requires-worker-protocol` is set to JSON, and `--experimental_worker_allow_json_protocol` is passed to the build to enable -this option). To communicate with the associated worker using binary-encoded -protobuf instead of json, `requires-worker-protocol` would be set to `proto`, -like this: +this option). The worker then performs the action, and sends a JSON-formatted +`WorkResponse` to Bazel on its stdout. Bazel then parses this response and +manually converts it to a `WorkResponse` proto. To communicate +with the associated worker using binary-encoded protobuf instead of JSON, +`requires-worker-protocol` would be set to `proto`, like this: + ``` execution_requirements = { "supports-workers" : "1" ,