)]}'
{
  "commit": "3d8f9fed2b8052193a92ee08db40e33c86c2491d",
  "tree": "a5d69eea8d611b15c64e0b48d7630dce33365142",
  "parents": [
    "c28d26767ef0d265dcbb2190ece7e768211232e1"
  ],
  "author": {
    "name": "Son Luong Ngoc",
    "email": "sluongng@gmail.com",
    "time": "Mon Oct 26 09:58:53 2026 -0700"
  },
  "committer": {
    "name": "Copybara-Service",
    "email": "copybara-worker@google.com",
    "time": "Mon Oct 26 10:00:17 2026 -0700"
  },
  "message": "remote: avoid tiny zstd upload blocks (#31528)\n\n### Description\n\nBazel\u0027s zstd upload adapter uses a 512-byte pipe and flushes after\nat most 491 source bytes. This forces short compressed blocks and\nadds encoding and pipe overhead.\n\nReplace the pipe with a reusable 16 KiB input buffer and a bounded\noutput collector. Let zstd choose when to emit blocks, drain that\noutput before reading more input, and finish the frame at EOF. Keep\nsource cleanup and retries from an uncompressed resume offset unchanged.\n\nLimit each upload chunk to eight compressed reads, plus EOF lookahead.\nThis avoids reading a large, highly compressible blob just to fill\none chunk before checking upload readiness. Returning partly filled\nchunks can add WriteRequest messages for such inputs.\n\nTwo matched benchmarks compare this change with the old implementation.\nThe tables show medians; change is (after / before - 1) * 100.\n\nThe compression-and-chunking benchmark uses 1,348 historical source\ninputs totaling 35,742,901 raw bytes, with zstd-jni 1.5.6-9 at level 3:\n\n    Metric                         Before       After Change %\n    ------------------------- ----------- ----------- --------\n    Encoded bytes              10,429,640   7,893,477   -24.3%\n    Elapsed time, ms              170.257      74.355   -56.3%\n    Thread CPU, ms                169.217      73.350   -56.7%\n    Java allocation, MB           250.704     254.813    +1.6%\n\nEach arm uses three JVMs, with seven measured rounds after two warmups\nper JVM. Times cover resident inputs, normalized to one encoding per\ninput, and include compressor creation, encoding, chunking and cleanup.\nThey exclude loading and verification. This uses JDK 27 on a shared AMD\nRyzen AI MAX+ 395 host; MB is decimal. One-shot compression produces\n7,899,530 bytes as a control, not as a bounded streaming fix.\n\nThe end-to-end benchmark builds //src:bazel from the same fixed\nsource with the before and after clients. Each run starts with a fresh\noutput base, empty remote blob/action caches and fresh executor file\ncache. Disk cache is disabled; repository download/content caches\nand image layers are warm.  A local BuildBuddy server and executor\n(cbc28103a282) run on the same shared host, using a pinned Ubuntu\n24.04/GCC 13.3 image. Both clients embed JDK 26.0.1. Builds use 32\njobs; the executor advertises 16 CPUs.\n\n    Build metric                   Before       After Change %\n    ------------------------- ----------- ----------- --------\n    Command wall, s               517.537     516.352    -0.2%\n    Client CPU, s                 224.269     205.670    -8.3%\n    Client peak memory, MiB     3,437.117   3,378.918    -1.7%\n    Upload payload, bytes     571,625,818 535,215,489    -6.4%\n    Same-blob payload, bytes  568,918,992 528,362,873    -7.1%\n    WriteRequest messages         117,089     115,263    -1.6%\n\nThere are three runs per arm, ordered before/after, after/before, then\nbefore/after. All six request the same 4,699 remote action digests\nand produce identical binaries, with no remote/disk cache hits,\nunexpected RPC failures or retries. Paired wall changes are -4.6%,\n-3.5% and +3.7%.  Median wall time is essentially flat; these runs\ndo not establish a build-speed improvement.\n\nUpload bytes count client Write payload, excluding network framing\nand executor transfers. Uploaded sets differ slightly, so the\nsame-blob row compares only the 51,539 compressed blobs uploaded\nin every run (918,011,952 raw bytes). Most small blobs still fit in\none message, so message count falls less than payload bytes in this\nbuild. Client CPU includes the batch JVM and waited local children,\nnot server/executor CPU or encode-only work. Peak memory is the largest\nprocess resident-memory high-water mark, not the sum across processes.\n\nThe tradeoff is more retained memory per compressor. Including the\nunchanged JNI destination array, Java buffer payload rises from about\n129.0 KiB to 160.5 KiB initially, and to 272.5 KiB at the collector\ncapacity observed for incompressible inputs: 31.5-143.5 KiB extra per\nstream. These observed capacities are not strict maxima. They exclude\nobjects, native zstd memory, upload messages and transient growth.\nBuffering remains independent of blob size; generated 64 and 512\nMiB inputs complete under a 32 MiB Java heap. Lower measured process\nmemory does not negate the higher per-stream cost.\n\nAt 10,000 initialized compressors, the observed larger-capacity case\nadds about 1.37 GiB of Java-array payload, not a total-memory bound.\nJobs limit actions, not blob fanout; asynchronous uploads can outlive\njob slots. RPC slots do not cap compressors retained during retry\nbackoff, so those limits alone do not bound total compressor memory.\n\n### Motivation\n\nReduce compressed upload size and client encoding work without buffering whole blobs.\n\n### Build API Changes\n\nNo.\n\n### Checklist\n\n- [x] I have added tests for the new use cases.\n- [x] Documentation is not applicable (no setup or API changes).\n\n### Release Notes\n\nRELNOTES: Reduce zstd-compressed remote upload size and encoding overhead.\n\nCloses #31528\n\nCOPYBARA_INTEGRATE_REVIEW\u003dhttps://github.com/bazelbuild/bazel/pull/31528 from sluongng:sluongng/zstd-streaming-upload-perf 87009db24809f51f9c12b53a555dfa96751a594e\nPiperOrigin-RevId: 1007356225\nChange-Id: I695120fa944ea95d83ae25ac7e1da07557fbf8b0\n",
  "tree_diff": [
    {
      "type": "modify",
      "old_id": "60b04aef7281fff9f1c8ed484e344448014111eb",
      "old_mode": 33188,
      "old_path": "src/main/java/com/google/devtools/build/lib/remote/Chunker.java",
      "new_id": "ee38f24878977ec1711f9ca05289f1813fbc823d",
      "new_mode": 33188,
      "new_path": "src/main/java/com/google/devtools/build/lib/remote/Chunker.java"
    },
    {
      "type": "modify",
      "old_id": "e5dd1903dfd3091b5bde4a0f6f350926c73a3767",
      "old_mode": 33188,
      "old_path": "src/main/java/com/google/devtools/build/lib/remote/zstd/ZstdCompressingInputStream.java",
      "new_id": "dfe6e6ca1b2d831f41ffd667b56869348e8b4b68",
      "new_mode": 33188,
      "new_path": "src/main/java/com/google/devtools/build/lib/remote/zstd/ZstdCompressingInputStream.java"
    },
    {
      "type": "modify",
      "old_id": "a43659458a14f86d1066ec07cb4a896d2acb9d2a",
      "old_mode": 33188,
      "old_path": "src/test/java/com/google/devtools/build/lib/remote/ByteStreamUploaderTest.java",
      "new_id": "31ac6d9dbb668e3d840fd7e51cd40f604970aae4",
      "new_mode": 33188,
      "new_path": "src/test/java/com/google/devtools/build/lib/remote/ByteStreamUploaderTest.java"
    },
    {
      "type": "modify",
      "old_id": "42c159cb562df3f500610c584e9cd9c0d08dc624",
      "old_mode": 33188,
      "old_path": "src/test/java/com/google/devtools/build/lib/remote/ChunkerTest.java",
      "new_id": "9c2f4c0d80f3e7f704f361d95abc0bcaf7e5fc09",
      "new_mode": 33188,
      "new_path": "src/test/java/com/google/devtools/build/lib/remote/ChunkerTest.java"
    },
    {
      "type": "modify",
      "old_id": "1008f6f65077cf2fdeef651a7e5f00819eb95b44",
      "old_mode": 33188,
      "old_path": "src/test/java/com/google/devtools/build/lib/remote/zstd/ZstdCompressingInputStreamTest.java",
      "new_id": "6d5eb9944d5c3bb3c05bf231f8e79a207950fe31",
      "new_mode": 33188,
      "new_path": "src/test/java/com/google/devtools/build/lib/remote/zstd/ZstdCompressingInputStreamTest.java"
    }
  ]
}