This directory includes the implementation for distributed caching support in Bazel and support for remote execution.
The detailed design document and discussion can be found in this forum thread.
https://groups.google.com/forum/#!msg/bazel-discuss/7JSbF6DT6OU/ewuXO6ydBAAJ
This section summarizes the options available in .bazelrc for distributed caching and remote execution support.
startup --host_jvm_args=-Dbazel.DigestFunction=SHA1This option is always needed to support distributed caching and remote execution.
build --spawn_strategy=remote --rest_cache_url=http://remote-cache:8080/cacheThis option enables distributed caching with a REST endpoint that supports GET, HEAD and PUT.
build --spawn_strategy=remote --remote_cache=remote-cache:8081This option enables distributed caching using a gRPC content-addressable storage (CAS) service.
build --spawn_strategy=remote --hazelcast_node=remote-cache:5701This option enables distributed caching using Hazelcast memory cluster as a content-addressable storage (CAS). Please watch for future announcement as this might be removed in favor of the REST endpoint.
build --spawn_strategy=remote --remote_worker=grpc-builder:5000 --remote_cache=grpc-builder:5000This option enables remote execution with a gRPC service at grpc-builder:5000. Remote execution requires a distributed caching service, which is also at grpc-builder:5000.
Distributed caching support in Bazel depends heavily on content-addressable storage.
A Bazel build consists of many actions. An action is defined by the command to execute, the arguments and a list of input files. Before executing an action Bazel computes a hash code from an action. This hash code will be used to lookup and index the output from executing the action. Bazel will lookup the hash code in the content-addressable storage (CAS) backend. If there is a match then the output files are downloaded. If there is no match the action will be executed and the output files will be uploaded to the CAS backend.
There are 2 kinds of CAS backend support implemented in Bazel.
If all you need is just distributed caching this is probably the most reliable path as the REST APIs are simple and will remain stable.
For quick setup you can use Hazelcast's REST interface. Alternatively you can also use NGINX with WebDav module or Apache HTTP Server with WebDav enabled. This enables simple remote caching for sharing between users.
You should enable SHA1 digest for Bazel with distributed caching. Edit ~/.bazelrc and put the following line:
startup --host_jvm_args=-Dbazel.DigestFunction=SHA1
We made it easy by providing Hazelcast in Bazel and it provides a REST interface for caching purposes.
Run using the following command and it will listen to port 5701. The REST endpoint will be http://localhost:5701/hazelcast/rest/maps/cache.
bazel build //third_party:hazelcast_rest_server bazel-bin/third_party/hazelcast_rest_server
You can customize the Hazelcast server by providing a configuration file during startup.
bazel-bin/third_party/hazelcast_rest_server --jvm_flags=-Dhazelcast.config=/path/to/hz.xml
You can copy and edit the default Hazelcast configuration. Refer to Hazelcast manual for more details.
First you need to set up NGINX with WebDav support. On Debian or Ubuntu Linux you can install nginx-extras package. On OSX you can install the nginx-full package from homebrew with brew install nginx-full --with-webdav.
Once installed, edit nginx.conf with a section for uploading and serving cache objects.
location /cache/ {
root /some/document/root;
dav_methods PUT;
autoindex on;
allow all;
client_max_body_size 256M;
}
You will need to change /some/document/root to a valid directory where NGINX can write to and read from. You may need to change client_max_body_size option to a larger value in case the cache object is too large.
Assuming Apache HTTP Server is installed with Dav modules installed. You need to edit httpd.conf to enable the following modules:
LoadModule dav_module libexec/apache2/mod_dav.so LoadModule dav_fs_module libexec/apache2/mod_dav_fs.so
Edit httpd.conf to use a directory for uploading and serving cache objects. You may want to edit this directory to include security control.
<Directory "/some/directory/for/cache">
AllowOverride None
Require all granted
Options +Indexes
Dav on
<Limit HEAD OPTIONS GET POST PUT DELETE>
Order Allow,Deny
Allow from all
</Limit>
<LimitExcept HEAD OPTIONS GET POST PUT DELETE>
Order Deny,Allow
Deny from all
</LimitExcept>
</Directory>
Any REST endpoint with GET, PUT and HEAD support will be sufficient. GET is used to fetch a cache object. PUT is used to upload a cache object and HEAD is used to check the existence of a cache object.
Once you have a REST endpoint that supports GET, PUT and HEAD then you can run Bazel with the following options to enable distributed caching. Change http://server-address:port/cache to the one that you provide. You may also put the options in ~/.bazelrc.
build --spawn_strategy=remote --rest_cache_url=http://server-address:port/cache
A gRPC CAS endpoint that implements the distributed caching and remote execution protocol will give the best performance and is the most actively developed distributed caching solution.
You should enable SHA1 digest for Bazel with distributed caching. Edit ~/.bazelrc and put the following line:
startup --host_jvm_args=-Dbazel.DigestFunction=SHA1
Bazel currently provides a sample gRPC CAS implementation with a SimpleBlobStore or Hazelcast as caching backend. To use it you need to clone from Bazel and then build it.
bazel build //src/tools/remote_worker:all
The following command will then start the cache server listening on port 8080 using a local in-memory cache.
bazel-bin/src/tools/remote_worker/remote_worker --listen_port=8080
To connect to a running instance of Hazelcast instead, use.
bazel-bin/src/tools/remote_worker/remote_worker --listen_port=8080 --hazelcast_node=address:port
If you want to change Hazelcast settings to enable distributed memory cache you can provide your own hazelcast.xml with the following command.
bazel-bin/src/tools/remote_worker/remote_worker --jvm_flags=-Dhazelcast.config=/path/to/hz.xml --listen_port 8080
You can copy and edit the default Hazelcast configuration. Refer to Hazelcast manual for more details.
Use the following build options to use the gRPC CAS endpoint for sharing build artifacts. Change address:8080 to the correct server address and port number.
build --spawn_strategy=remote --remote_cache=address:8080
Bazel can connect to a Hazelcast distributed memory cluster directly for sharing build artifacts. This feature will be removed in the future in favor of the gRPC protocol for distributed caching. Hazelcast may still be used as a distributed caching backend but Bazel will connect to it through a gRPC CAS endpoint.
If you do not already have a Hazelcast memory cluster you can clone Bazel and run this command:
java -cp third_party/hazelcast/hazelcast-3.6.4.jar com.hazelcast.core.server.StartServer
You will need to put the following line in ~/.bazelrc.
startup --host_jvm_args=-Dbazel.DigestFunction=SHA1
The following build options will use Hazelcast as a distributed cache during build. Change address:5701 to the actual server address assuming Hazelcast listens to port 5701.
build --hazelcast_node=address:5701 --spawn_strategy=remote
The implementation of remote execution worker in Bazel can only serve as a demonstration. The client-side implementation is being actively developed in Bazel. However there is no fully functional implementation of remote worker yet.
You should enable SHA1 digest for Bazel with distributed caching. Edit ~/.bazelrc and put the following line:
startup --host_jvm_args=-Dbazel.DigestFunction=SHA1
bazel build //src/tools/remote_worker:remote_worker bazel-bin/src/tools/remote_worker/remote_worker --work_path=/tmp --listen_port 8080
The sample gRPC cache server and gRPC remote worker share the same distributed memory cluster for storing and accessing CAS objects. It is important the CAS objects are shared between the two server processes.
You can modify hazelcast configuration by providing a hazelcast.xml. Please refer to Hazelcast manual for details. Make sure the cache server and the remote worker server shares the same memory cluster.
Use the following build options.
build --spawn_strategy=remote --remote_worker=localhost:8080 --remote_cache=localhost:8080