tree 903d21622b3d7666800f85b629c4dd0a3ef606c2
parent 9083bbe645818a25894b23e70d723f7c7dad36e4
author ulfjack <ulfjack@google.com> 1559743744 -0700
committer Copybara-Service <copybara-worker@google.com> 1559743800 -0700

Query performance: Support length-delimited protos

Add support for --output=streamed_proto, which outputs the targets as
a stream of length-delimited plain protos rather than a single
monolithic proto.

For queries with a lot of results, this is significantly faster (up to
~30% in my tests, though note that my baseline test contained a series
of as-yet unsubmitted improvements).

The primary driver of the performance improvement on the Bazel side
appears to be the decreased memory pressure resulting in lower gc
times.

In any case, the requirement to retain the full result on the Bazel
side as well as the receiver side is a showstopper for large queries.

We might want to consider wrapping the resulting targets in another
proto that allows us to add some per-package information in the future.
Another performance issue is the duplicate location field on every
single target, which constitutes up to ~30% of the output size. Even
with relative paths, we see around ~14% of the output being location
data.

PiperOrigin-RevId: 251636100