)]}' { "commit": "8541f6d8ed512b6d64456a5d700761b78329f0b3", "tree": "fe0a7e9623d6bcc264eb017ed5f769e94d71f33c", "parents": [ "3a4be3c93813987a27a97dade3f9ebbc5770e349" ], "author": { "name": "janakr", "email": "janakr@google.com", "time": "Tue Jun 11 14:40:21 2019 -0700" }, "committer": { "name": "Copybara-Service", "email": "copybara-worker@google.com", "time": "Tue Jun 11 14:41:21 2019 -0700" }, "message": "Use Artifact#getGeneratingActionKey to avoid creating Artifact nodes in the graph for all \"normal\" generated artifacts.\n\nAny generated artifact that does not represent multiple other artifacts (aggregating middleman and tree artifacts are the exceptions) can be looked up directly from the generating action\u0027s ActionExecutionValue. The artifact\u0027s type gives a conservative decision procedure for this: non-middleman, non-tree generated artifacts are definitely ok. Non-aggregating middlemen and tree artifacts that are not produced by template expansion are also ok, but knowing that requires checking more data, so we continue to call into to ArtifactFunction for those artifacts.\n\nSaves 60/4100 post-execution heap on a medium-sized build, or 1.5% of memory:\n\n$ diffheap.pl ~/blaze/histo{_before,}.txt\n objsize chg instances space KB class name\n------------------------------------------------------\n 77 +0 -711 -735 KB [B\n 32 +0 -43594 -1362 KB com.google.devtools.build.lib.actions.FileArtifactValue$RegularFileArtifactValue\n 24 +0 -90829 -2128 KB java.util.ArrayList\n 746 -173 -5 -9230 KB [Ljava.util.concurrent.ConcurrentHashMap$Node;\n 32 +0 -309720 -9678 KB java.util.concurrent.ConcurrentHashMap$Node\n 40 +0 -309641 -12095 KB com.google.devtools.build.skyframe.InMemoryNodeEntry\n 90 +0 -400463 -26764 KB [Ljava.lang.Object;\n------------------------------------------------------\n total change: -62092 KB\n\nThere are three main external changes in this CL:\n\n(1) Every consumer of Artifact metadata must pass all artifacts through ArtifactSkyKey#key and friends, since generated artifacts\u0027 keys are no longer necessarily themselves. This is something of a partial rollback of https://github.com/bazelbuild/bazel/commit/bf4123df23b5f93e572cd920f15afba340f92391 (originally unknown commit): see, for example, https://github.com/bazelbuild/bazel/commit/bf4123df23b5f93e572cd920f15afba340f92391#diff-619984696e738a6f3ccb9b3802ab7d90.\n\n(2) Similarly every such consumer must be prepared for the returned SkyValue for an Artifact to be an ActionExecutionValue, not a FileArtifactValue or TreeArtifactValue. This means that the consumer must iterate over the original list of Artifacts, constructing each key on the fly, not the returned map, since the returned map may not have any indication of the original Artifacts. The construction of some FileArtifactValues on the fly here may slightly increase garbage (although not in the Google-internal or common Bazel case, since we store FileArtifactValues directly in ActionExecutionValue there), but it should be dominated by the memory savings. I\u0027m hoping to clean up ActionExecutionValues in a separate change, so that we don\u0027t have this overlapping data, although the garbage issue may still remain.\n\nThe main complication in (2) is in ActionExecutionFunction, where we need to construct keys for non-mandatory source artifacts.\n\n(3) Action rewinding no longer needs to invalidate ordinary generated Artifact nodes in the graph. This simplifies the resulting graphs, but can complicate the rewinding logic, since some Artifacts will still have nodes while others won\u0027t. Instead of tracking on the basis of artifacts, we now track on the basis of actions and artifacts.\n\nWhile modifying ActionRewindStrategy, I took the liberty of doing some clean-ups: for instance, after https://github.com/bazelbuild/bazel/commit/efb3f1595ee897484c477168b8da42b67602e10e, the HashMultimap\u003cDerivedArtifact, ActionInput\u003e lostInputsByDepOwners was only using its values for a check that was guaranteed to succeed, so I just made it a set.\n\nPiperOrigin-RevId: 252701678\n", "tree_diff": [ { "type": "modify", "old_id": "9f8680ac4993faa7d3096492cf9cd41ac98aacb4", "old_mode": 33188, "old_path": "src/main/java/com/google/devtools/build/lib/actions/Artifact.java", "new_id": "1a8b54f0a88bb2aba397b91da8c9d593d0c69f5e", "new_mode": 33188, "new_path": "src/main/java/com/google/devtools/build/lib/actions/Artifact.java" }, { "type": "modify", "old_id": "ef66e8305a6a850432b519d58209746c14e1675a", "old_mode": 33188, "old_path": "src/main/java/com/google/devtools/build/lib/actions/ArtifactSkyKey.java", "new_id": "c3b3080ca6411aec8692e2f6796af7e468f0adbd", "new_mode": 33188, "new_path": "src/main/java/com/google/devtools/build/lib/actions/ArtifactSkyKey.java" }, { "type": "modify", "old_id": "d77ef7f51483449de0dcfa71a387941a6e5c0068", "old_mode": 33188, "old_path": "src/main/java/com/google/devtools/build/lib/buildtool/SkyframeBuilder.java", "new_id": "0a8f4cec83e27492721fe54ff925a5ff55e53e7a", "new_mode": 33188, "new_path": "src/main/java/com/google/devtools/build/lib/buildtool/SkyframeBuilder.java" }, { "type": "modify", "old_id": "f0ed9d2189d5204360eb44bb75e61f3e52d2be7c", "old_mode": 33188, "old_path": "src/main/java/com/google/devtools/build/lib/skyframe/ActionExecutionFunction.java", "new_id": "2d5c9bad0da6a4add135edc4fd752a54f90eb353", "new_mode": 33188, "new_path": "src/main/java/com/google/devtools/build/lib/skyframe/ActionExecutionFunction.java" }, { "type": "modify", "old_id": "d5771fc285b62eb788717fe77244ca7327c07da5", "old_mode": 33188, "old_path": "src/main/java/com/google/devtools/build/lib/skyframe/ActionInputMapHelper.java", "new_id": "be4cc991bf4f649085fa3d9ebc7ba1bf5c14678b", "new_mode": 33188, "new_path": "src/main/java/com/google/devtools/build/lib/skyframe/ActionInputMapHelper.java" }, { "type": "modify", "old_id": "caa8388906903c3b67cdaef58b734cf3498c3ae9", "old_mode": 33188, "old_path": "src/main/java/com/google/devtools/build/lib/skyframe/ActionRewindStrategy.java", "new_id": "ba82e9051906fc248e06c81ee65a6740171c908f", "new_mode": 33188, "new_path": "src/main/java/com/google/devtools/build/lib/skyframe/ActionRewindStrategy.java" }, { "type": "modify", "old_id": "41bed76bc58aed672e073abd5371de8e13b36bf3", "old_mode": 33188, "old_path": "src/main/java/com/google/devtools/build/lib/skyframe/ActionTemplateExpansionFunction.java", "new_id": "51fa410e1af8f9820fd3511270895835ce29717e", "new_mode": 33188, "new_path": "src/main/java/com/google/devtools/build/lib/skyframe/ActionTemplateExpansionFunction.java" }, { "type": "modify", "old_id": "b2b4d96d3a60242fa4918eb729090281d9ca296b", "old_mode": 33188, "old_path": "src/main/java/com/google/devtools/build/lib/skyframe/ArtifactFunction.java", "new_id": "0fad11b9645bd507249ad167eeb8cb47dfeb65f5", "new_mode": 33188, "new_path": "src/main/java/com/google/devtools/build/lib/skyframe/ArtifactFunction.java" }, { "type": "modify", "old_id": "7f4e8b4e1e7cc5f4cca0f5ad1a728cbd5153a7b2", "old_mode": 33188, "old_path": "src/main/java/com/google/devtools/build/lib/skyframe/CompletionFunction.java", "new_id": "2d5ab17142720f882e892e626b11f0f263c82f4d", "new_mode": 33188, "new_path": "src/main/java/com/google/devtools/build/lib/skyframe/CompletionFunction.java" }, { "type": "modify", "old_id": "49f687014e85419f27a64fb43aa5bee7625c3830", "old_mode": 33188, "old_path": "src/main/java/com/google/devtools/build/lib/skyframe/RecursiveFilesystemTraversalFunction.java", "new_id": "62f894b6ac355657970da474c6253c165624ce02", "new_mode": 33188, "new_path": "src/main/java/com/google/devtools/build/lib/skyframe/RecursiveFilesystemTraversalFunction.java" }, { "type": "modify", "old_id": "f24127ae2cff95813e8b7a5adf52a6266920fe08", "old_mode": 33188, "old_path": "src/main/java/com/google/devtools/build/lib/skyframe/SkyframeActionExecutor.java", "new_id": "8f4f07e754a20f6e31be21462ab6040199bb2f75", "new_mode": 33188, "new_path": "src/main/java/com/google/devtools/build/lib/skyframe/SkyframeActionExecutor.java" }, { "type": "modify", "old_id": "5b07b3d9f2803b27469fa27d4fd424734ae837c7", "old_mode": 33188, "old_path": "src/main/java/com/google/devtools/build/lib/skyframe/SkyframeExecutor.java", "new_id": "72383813e4368c2f535bd7a93c74090ae99508b0", "new_mode": 33188, "new_path": "src/main/java/com/google/devtools/build/lib/skyframe/SkyframeExecutor.java" }, { "type": "modify", "old_id": "4ba23c3278d980c7cc999fe7225a69411f1db0e3", "old_mode": 33188, "old_path": "src/test/java/com/google/devtools/build/lib/skyframe/ArtifactFunctionTest.java", "new_id": "7d4f3c76cdd7e35dd1813fde06a6f2e2dc9e84b8", "new_mode": 33188, "new_path": "src/test/java/com/google/devtools/build/lib/skyframe/ArtifactFunctionTest.java" }, { "type": "modify", "old_id": "1fc649245eb61779888202a69afb8883584d0d6c", "old_mode": 33188, "old_path": "src/test/java/com/google/devtools/build/lib/skyframe/RecursiveFilesystemTraversalFunctionTest.java", "new_id": "87f81ac082d4098aa691104a43a73130eabca9ec", "new_mode": 33188, "new_path": "src/test/java/com/google/devtools/build/lib/skyframe/RecursiveFilesystemTraversalFunctionTest.java" }, { "type": "modify", "old_id": "e128babc5e617c4c5a919b434716517dbbbc8497", "old_mode": 33188, "old_path": "src/test/java/com/google/devtools/build/lib/skyframe/TimestampBuilderTestCase.java", "new_id": "63054cb2f799cc28017d9a646daa90e57ac4756d", "new_mode": 33188, "new_path": "src/test/java/com/google/devtools/build/lib/skyframe/TimestampBuilderTestCase.java" }, { "type": "modify", "old_id": "98c06472525f081fa4d721c17eab8e8ac4429243", "old_mode": 33188, "old_path": "src/test/java/com/google/devtools/build/lib/skyframe/TreeArtifactBuildTest.java", "new_id": "8df59da0866e62ba5b8f7143952bf56e805f336a", "new_mode": 33188, "new_path": "src/test/java/com/google/devtools/build/lib/skyframe/TreeArtifactBuildTest.java" } ] }