Add `bazel mod show_repo --output=streamed_proto` and `--output=streamed_jsonproto`
Serialize repo definitions to the same `Target` proto that `bazel query` uses.
A few more notable details:
* Add back `_original_name`, even to the Starlark output. This was removed in https://github.com/bazelbuild/bazel/pull/26493, but I believe it's still useful for debugging.
* The output protos may contain a `$apparent_repo_name` or `$module_key` pseudo-attribute, which is the equivalent of the `## @repo_name` / `## module@version` line in the Starlark output.
* Similar to the original Starlark output, the same (canonical) repo can be shown multiple times if the user explicitly specified the same repo in different ways:
```sh
❯ bazel-bin/src/bazel_nojdk mod show_repo @@rules_cc+ @rules_cc rules_cc
```
```starlark
## @@rules_cc+:
http_archive(
name = "rules_cc+",
...
## @rules_cc:
http_archive(
name = "rules_cc+",
...
## rules_cc@0.2.9:
http_archive(
name = "rules_cc+",
...
```
```sh
❯ bazel-bin/src/bazel_nojdk mod show_repo --output=streamed_jsonproto @@rules_cc+ @rules_cc rules_cc
```
```js
{"canonicalName":"rules_cc+", …
{"canonicalName":"rules_cc+","apparentName":"@rules_cc", …
{"canonicalName":"rules_cc+","moduleKey":"rules_cc@0.2.14", …
```
* Tighten up command argument validation so that `bazel mod show_{repo,extension} --output={graph,json}` now exits with an error, addressing a common source of user confusion. I decided not to add the same validation to all `bazel mod` subcommands since no one can possibly expect `bazel mod tidy --output=graph` to do something.
Fixes #21617.
Works towards #24692.
Closes #27242.
PiperOrigin-RevId: 843878072
Change-Id: I275ad2d3d24472a85d875176374d29bf42397d16
diff --git a/docs/external/mod-command.mdx b/docs/external/mod-command.mdx
index 84086fb..122cd88 100644
--- a/docs/external/mod-command.mdx
+++ b/docs/external/mod-command.mdx
@@ -78,6 +78,8 @@
The `<label_to_bzl_file>` part must be a repo-relative label (for example,
`//pkg/path:file.bzl`).
+### Graph command options
+
The following options only affect the subcommands that print graphs (`graph`,
`deps`, `all_paths`, `path`, and `explain`):
@@ -138,7 +140,7 @@
legacy platforms which cannot use Unicode.
* `--output <mode>`: Include information about the module extension usages as
- part of the output graph. `<mode`> can be one of:
+ part of the output graph. `<mode>' can be one of:
* `text` *(default)*: A human-readable representation of the output graph
(flattened as a tree).
@@ -155,6 +157,22 @@
bazel mod graph --output graph | dot -Tsvg > /tmp/graph.svg
```
+ ### show_repo options
+`show_repo` supports a different set of output formats:
+* `--output <mode>`: Change how repository definitions are displayed.
+ `<mode>` can be one of:
+ * `text` *(default)*: Display repo definitions in Starlark.
+ * `streamed_proto`: Prints a
+ [length-delimited](https://protobuf.dev/programming-guides/encoding/#size-limit)
+ stream of
+ [`Repository`](https://github.com/bazelbuild/bazel/blob/master/src/main/protobuf/build.proto)
+ protocol buffers.
+ * `streamed_jsonproto`: Similar to `--output streamed_proto`, prints a
+ stream of [`Repository`](https://github.com/bazelbuild/bazel/blob/master/src/main/protobuf/build.proto)
+ protocol buffers but in [NDJSON](https://github.com/ndjson/ndjson-spec) format.
+
+ ### Other options
+
Other options include:
* `--base_module <arg>` *default: `<root>`*: Specify a module relative to
diff --git a/site/en/external/mod-command.md b/site/en/external/mod-command.md
index 1e3556f..e086c46 100644
--- a/site/en/external/mod-command.md
+++ b/site/en/external/mod-command.md
@@ -82,6 +82,8 @@
The `<label_to_bzl_file>` part must be a repo-relative label (for example,
`//pkg/path:file.bzl`).
+### Graph command options
+
The following options only affect the subcommands that print graphs (`graph`,
`deps`, `all_paths`, `path`, and `explain`):
@@ -142,7 +144,7 @@
legacy platforms which cannot use Unicode.
* `--output <mode>`: Include information about the module extension usages as
- part of the output graph. `<mode`> can be one of:
+ part of the output graph. `<mode>` can be one of:
* `text` *(default)*: A human-readable representation of the output graph
(flattened as a tree).
@@ -159,6 +161,31 @@
bazel mod graph --output graph | dot -Tsvg > /tmp/graph.svg
```
+### show_repo options
+
+`show_repo` supports a different set of output formats:
+
+* `--output <mode>`: Change how repository definitions are displayed.
+ `<mode>` can be one of:
+
+ * `text` *(default)*: Display repository definitions in Starlark.
+
+ * `streamed_proto`: Prints a
+ [length-delimited](https://protobuf.dev/programming-guides/encoding/#siz
+ e-limit)
+ stream of
+ [`Repository`](https://github.com/bazelbuild/bazel/blob/master/src/main/
+ protobuf/build.proto)
+ protocol buffers.
+
+ * `streamed_jsonproto`: Similar to `--output streamed_proto`, prints a
+ stream of [`Repository`](https://github.com/bazelbuild/bazel/blob/master
+ /src/main/protobuf/build.proto)
+ protocol buffers but in [NDJSON](https://github.com/ndjson/ndjson-spec)
+ format.
+
+### Other options
+
Other options include:
* `--base_module <arg>` *default: `<root>`*: Specify a module relative to
diff --git a/src/main/java/com/google/devtools/build/lib/bazel/bzlmod/modcommand/BUILD b/src/main/java/com/google/devtools/build/lib/bazel/bzlmod/modcommand/BUILD
index 8c000b2..caa869c 100644
--- a/src/main/java/com/google/devtools/build/lib/bazel/bzlmod/modcommand/BUILD
+++ b/src/main/java/com/google/devtools/build/lib/bazel/bzlmod/modcommand/BUILD
@@ -21,14 +21,20 @@
"//src/main/java/com/google/devtools/build/lib/bazel/repository:repo_definition",
"//src/main/java/com/google/devtools/build/lib/bazel/repository:repo_definition_value",
"//src/main/java/com/google/devtools/build/lib/cmdline",
+ "//src/main/java/com/google/devtools/build/lib/packages",
+ "//src/main/java/com/google/devtools/build/lib/packages:label_printer",
"//src/main/java/com/google/devtools/build/lib/util:maybe_complete_set",
+ "//src/main/java/com/google/devtools/build/lib/util:string_encoding",
"//src/main/java/com/google/devtools/common/options",
"//src/main/java/net/starlark/java/eval",
+ "//src/main/protobuf:build_java_proto",
"//src/main/protobuf:failure_details_java_proto",
"//third_party:auto_value",
"//third_party:error_prone_annotations",
"//third_party:gson",
"//third_party:guava",
"//third_party:jsr305",
+ "@com_google_protobuf//:protobuf_java",
+ "@com_google_protobuf//:protobuf_java_util",
],
)
diff --git a/src/main/java/com/google/devtools/build/lib/bazel/bzlmod/modcommand/ModExecutor.java b/src/main/java/com/google/devtools/build/lib/bazel/bzlmod/modcommand/ModExecutor.java
index b1125b5..dd0d108 100644
--- a/src/main/java/com/google/devtools/build/lib/bazel/bzlmod/modcommand/ModExecutor.java
+++ b/src/main/java/com/google/devtools/build/lib/bazel/bzlmod/modcommand/ModExecutor.java
@@ -17,6 +17,8 @@
import static com.google.common.collect.ImmutableSet.toImmutableSet;
import static com.google.common.collect.ImmutableSortedMap.toImmutableSortedMap;
import static com.google.common.collect.ImmutableSortedSet.toImmutableSortedSet;
+import static java.nio.charset.StandardCharsets.US_ASCII;
+import static java.nio.charset.StandardCharsets.UTF_8;
import static java.util.Comparator.reverseOrder;
import static java.util.Objects.requireNonNull;
import static java.util.stream.Collectors.joining;
@@ -39,13 +41,13 @@
import com.google.devtools.build.lib.bazel.bzlmod.modcommand.ModExecutor.ResultNode.IsExpanded;
import com.google.devtools.build.lib.bazel.bzlmod.modcommand.ModExecutor.ResultNode.IsIndirect;
import com.google.devtools.build.lib.bazel.bzlmod.modcommand.ModExecutor.ResultNode.NodeMetadata;
-import com.google.devtools.build.lib.bazel.repository.RepoDefinition;
import com.google.devtools.build.lib.bazel.repository.RepoDefinitionValue;
-import com.google.devtools.build.lib.bazel.repository.RepoRule;
import com.google.devtools.build.lib.util.MaybeCompleteSet;
import com.google.errorprone.annotations.CanIgnoreReturnValue;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
import java.io.PrintWriter;
-import java.io.Writer;
import java.util.ArrayDeque;
import java.util.Collections;
import java.util.Comparator;
@@ -71,19 +73,22 @@
private final ImmutableSetMultimap<ModuleExtensionId, String> extensionRepos;
private final Optional<MaybeCompleteSet<ModuleExtensionId>> extensionFilter;
private final ModOptions options;
+ private final OutputStream outputStream;
private final PrintWriter printer;
private ImmutableMap<ModuleExtensionId, ImmutableSetMultimap<String, ModuleKey>>
extensionRepoImports;
public ModExecutor(
- ImmutableMap<ModuleKey, AugmentedModule> depGraph, ModOptions options, Writer writer) {
+ ImmutableMap<ModuleKey, AugmentedModule> depGraph,
+ ModOptions options,
+ OutputStream outputStream) {
this(
depGraph,
ImmutableTable.of(),
ImmutableSetMultimap.of(),
Optional.of(MaybeCompleteSet.completeSet()),
options,
- writer);
+ outputStream);
}
public ModExecutor(
@@ -92,13 +97,17 @@
ImmutableSetMultimap<ModuleExtensionId, String> extensionRepos,
Optional<MaybeCompleteSet<ModuleExtensionId>> extensionFilter,
ModOptions options,
- Writer writer) {
+ OutputStream outputStream) {
this.depGraph = depGraph;
this.extensionUsages = extensionUsages;
this.extensionRepos = extensionRepos;
this.extensionFilter = extensionFilter;
this.options = options;
- this.printer = new PrintWriter(writer);
+ this.outputStream = outputStream;
+ this.printer =
+ new PrintWriter(
+ new OutputStreamWriter(
+ outputStream, options.charset == ModOptions.Charset.UTF8 ? UTF_8 : US_ASCII));
// Easier lookup table for repo imports by module.
// It is updated after pruneByDepthAndLink to filter out pruned modules.
this.extensionRepoImports = computeRepoImportsTable(depGraph.keySet());
@@ -163,16 +172,15 @@
}
public void showRepo(ImmutableMap<String, RepoDefinitionValue> targetRepoDefinitions) {
+ var formatter = new RepoOutputFormatter(printer, outputStream, options.outputFormat);
for (Map.Entry<String, RepoDefinitionValue> e : targetRepoDefinitions.entrySet()) {
- if (e.getValue() instanceof RepoDefinitionValue.Found repoDefValue) {
- printer.printf("## %s:\n", e.getKey());
- printRepoDefinition(repoDefValue.repoDefinition());
- }
- if (e.getValue() instanceof RepoDefinitionValue.RepoOverride repoOverrideValue) {
- printer.printf(
- "## %s:\nBuiltin or overridden repo located at: %s\n\n",
- e.getKey(), repoOverrideValue.repoPath());
- }
+ formatter.print(e.getKey(), e.getValue());
+ }
+
+ try {
+ outputStream.flush();
+ } catch (IOException ex) {
+ // Ignore IOException like PrintWriter.
}
printer.flush();
}
@@ -781,27 +789,4 @@
abstract ResultNode build();
}
}
-
- private void printRepoDefinition(RepoDefinition repoDefinition) {
- RepoRule repoRule = repoDefinition.repoRule();
- printer
- .append("load(\"")
- .append(repoRule.id().bzlFileLabel().getUnambiguousCanonicalForm())
- .append("\", \"")
- .append(repoRule.id().ruleName())
- .append("\")\n");
- printer.append(repoRule.id().ruleName()).append("(\n");
- printer.append(" name = \"").append(repoDefinition.name()).append("\",\n");
- for (Map.Entry<String, Object> attr : repoDefinition.attrValues().attributes().entrySet()) {
- printer
- .append(" ")
- .append(attr.getKey())
- .append(" = ")
- .append(Starlark.repr(attr.getValue()))
- .append(",\n");
- }
- printer.append(")\n");
- // TODO: record and print the call stack for the repo definition itself?
- printer.append("\n");
- }
}
diff --git a/src/main/java/com/google/devtools/build/lib/bazel/bzlmod/modcommand/ModOptions.java b/src/main/java/com/google/devtools/build/lib/bazel/bzlmod/modcommand/ModOptions.java
index c7f9757..b7be841 100644
--- a/src/main/java/com/google/devtools/build/lib/bazel/bzlmod/modcommand/ModOptions.java
+++ b/src/main/java/com/google/devtools/build/lib/bazel/bzlmod/modcommand/ModOptions.java
@@ -266,9 +266,21 @@
/** Possible formats of the `mod` command result. */
public enum OutputFormat {
+ // Default
TEXT,
+
+ // For graph commands:
JSON,
- GRAPH
+ GRAPH,
+
+ // For show_repo:
+ STREAMED_PROTO,
+ STREAMED_JSONPROTO;
+
+ @Override
+ public String toString() {
+ return Ascii.toLowerCase(this.name());
+ }
}
/** Converts an output format option string to a properly typed {@link OutputFormat} */
diff --git a/src/main/java/com/google/devtools/build/lib/bazel/bzlmod/modcommand/OutputFormatters.java b/src/main/java/com/google/devtools/build/lib/bazel/bzlmod/modcommand/OutputFormatters.java
index 4f8170b..160fbaf 100644
--- a/src/main/java/com/google/devtools/build/lib/bazel/bzlmod/modcommand/OutputFormatters.java
+++ b/src/main/java/com/google/devtools/build/lib/bazel/bzlmod/modcommand/OutputFormatters.java
@@ -49,6 +49,7 @@
case JSON -> jsonFormatter;
case GRAPH -> graphvizFormatter;
case null -> throw new IllegalArgumentException("Output format cannot be null.");
+ default -> throw new IllegalArgumentException("Unsupported output format: " + format);
};
}
diff --git a/src/main/java/com/google/devtools/build/lib/bazel/bzlmod/modcommand/RepoOutputFormatter.java b/src/main/java/com/google/devtools/build/lib/bazel/bzlmod/modcommand/RepoOutputFormatter.java
new file mode 100644
index 0000000..c71a90e
--- /dev/null
+++ b/src/main/java/com/google/devtools/build/lib/bazel/bzlmod/modcommand/RepoOutputFormatter.java
@@ -0,0 +1,170 @@
+// Copyright 2025 The Bazel Authors. All rights reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package com.google.devtools.build.lib.bazel.bzlmod.modcommand;
+
+import static com.google.devtools.build.lib.util.StringEncoding.internalToUnicode;
+
+import com.google.devtools.build.lib.bazel.bzlmod.modcommand.ModOptions.OutputFormat;
+import com.google.devtools.build.lib.bazel.repository.RepoDefinition;
+import com.google.devtools.build.lib.bazel.repository.RepoDefinitionValue;
+import com.google.devtools.build.lib.bazel.repository.RepoRule;
+import com.google.devtools.build.lib.packages.Attribute;
+import com.google.devtools.build.lib.packages.AttributeFormatter;
+import com.google.devtools.build.lib.packages.LabelPrinter;
+import com.google.devtools.build.lib.query2.proto.proto2api.Build;
+import com.google.protobuf.InvalidProtocolBufferException;
+import com.google.protobuf.util.JsonFormat;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.Map;
+import net.starlark.java.eval.Starlark;
+
+/** Outputs repository definitions for {@code mod show_repo}. */
+public class RepoOutputFormatter {
+ private static final JsonFormat.Printer jsonPrinter =
+ JsonFormat.printer().omittingInsignificantWhitespace();
+
+ private final PrintWriter printer;
+ private final OutputStream outputStream;
+ private final OutputFormat outputFormat;
+
+ public RepoOutputFormatter(
+ PrintWriter printer, OutputStream outputStream, OutputFormat outputFormat) {
+ this.printer = printer;
+ this.outputStream = outputStream;
+ this.outputFormat = outputFormat;
+ }
+
+ public void print(String key, RepoDefinitionValue repoDefinition) {
+ switch (outputFormat) {
+ case TEXT -> printStarlark(key, repoDefinition);
+ case STREAMED_JSONPROTO, STREAMED_PROTO -> {
+ // In proto output formats, we only print repo definitions, not overrides.
+ if (repoDefinition instanceof RepoDefinitionValue.Found repoDefValue) {
+ if (outputFormat == OutputFormat.STREAMED_JSONPROTO) {
+ printProtoJson(key, repoDefValue.repoDefinition());
+ } else {
+ printStreamedProto(key, repoDefValue.repoDefinition());
+ }
+ }
+ }
+ default -> throw new IllegalArgumentException("Unknown output format: " + outputFormat);
+ }
+ }
+
+ private void printStarlark(String key, RepoDefinitionValue repoDefinition) {
+ if (repoDefinition instanceof RepoDefinitionValue.Found repoDefValue) {
+ printer.printf("## %s:\n", key);
+ printStarlark(repoDefValue.repoDefinition());
+ }
+ if (repoDefinition instanceof RepoDefinitionValue.RepoOverride repoOverrideValue) {
+ printer.printf(
+ "## %s:\nBuiltin or overridden repo located at: %s\n\n",
+ key, repoOverrideValue.repoPath());
+ }
+ }
+
+ private void printStarlark(RepoDefinition repoDefinition) {
+ RepoRule repoRule = repoDefinition.repoRule();
+ printer
+ .append("load(\"")
+ .append(repoRule.id().bzlFileLabel().getUnambiguousCanonicalForm())
+ .append("\", \"")
+ .append(repoRule.id().ruleName())
+ .append("\")\n");
+ printer.append(repoRule.id().ruleName()).append("(\n");
+ printer.append(" name = \"").append(repoDefinition.name()).append("\",\n");
+ if (repoDefinition.originalName() != null) {
+ printer.append(" _original_name = \"").append(repoDefinition.originalName()).append("\",\n");
+ }
+ for (Map.Entry<String, Object> attr : repoDefinition.attrValues().attributes().entrySet()) {
+ printer
+ .append(" ")
+ .append(attr.getKey())
+ .append(" = ")
+ .append(Starlark.repr(attr.getValue()))
+ .append(",\n");
+ }
+ printer.append(")\n");
+ // TODO: record and print the call stack for the repo definition itself?
+ printer.append("\n");
+ }
+
+ private void printStreamedProto(String key, RepoDefinition repoDefinition) {
+ Build.Repository serialized = serializeRepoDefinitionAsProto(key, repoDefinition);
+ try {
+ serialized.writeDelimitedTo(outputStream);
+ } catch (IOException e) {
+ // Ignore IOException like PrintWriter.
+ }
+ }
+
+ private void printProtoJson(String key, RepoDefinition repoDefinition) {
+ Build.Repository serialized = serializeRepoDefinitionAsProto(key, repoDefinition);
+ try {
+ printer.println(jsonPrinter.print(serialized));
+ } catch (InvalidProtocolBufferException e) {
+ throw new IllegalArgumentException(e);
+ }
+ }
+
+ private Build.Repository serializeRepoDefinitionAsProto(
+ String key, RepoDefinition repoDefinition) {
+ RepoRule repoRule = repoDefinition.repoRule();
+
+ Build.Repository.Builder pbBuilder = Build.Repository.newBuilder();
+ pbBuilder.setCanonicalName(internalToUnicode(repoDefinition.name()));
+ pbBuilder.setRepoRuleName(internalToUnicode(repoRule.id().ruleName()));
+ pbBuilder.setRepoRuleBzlLabel(
+ internalToUnicode(repoRule.id().bzlFileLabel().getUnambiguousCanonicalForm()));
+
+ // TODO: record and print the call stack for the repo definition itself?
+
+ if (key.startsWith("@")) {
+ if (!key.startsWith("@@")) {
+ pbBuilder.setApparentName(internalToUnicode(key));
+ }
+ } else {
+ pbBuilder.setModuleKey(internalToUnicode(key));
+ }
+ if (repoDefinition.originalName() != null) {
+ pbBuilder.setOriginalName(internalToUnicode(repoDefinition.originalName()));
+ }
+
+ for (Map.Entry<String, Integer> attr : repoRule.attributeIndices().entrySet()) {
+ String attrName = attr.getKey();
+ Attribute attrDefinition = repoRule.attributes().get(attr.getValue());
+
+ boolean explicitlySpecified = repoDefinition.attrValues().attributes().containsKey(attrName);
+ Object attrValue = repoDefinition.attrValues().attributes().get(attrName);
+ if (attrValue == null) {
+ attrValue = attrDefinition.getDefaultValueUnchecked();
+ }
+ Build.Attribute serializedAttribute =
+ AttributeFormatter.getAttributeProto(
+ attrDefinition,
+ attrValue,
+ explicitlySpecified,
+ /* encodeBooleanAndTriStateAsIntegerAndString= */ true,
+ /* sourceAspect= */ null,
+ /* includeAttributeSourceAspects= */ false,
+ LabelPrinter.legacy());
+ pbBuilder.addAttribute(serializedAttribute);
+ }
+
+ return pbBuilder.build();
+ }
+}
diff --git a/src/main/java/com/google/devtools/build/lib/bazel/commands/ModCommand.java b/src/main/java/com/google/devtools/build/lib/bazel/commands/ModCommand.java
index 2558e9a..90e6f12 100644
--- a/src/main/java/com/google/devtools/build/lib/bazel/commands/ModCommand.java
+++ b/src/main/java/com/google/devtools/build/lib/bazel/commands/ModCommand.java
@@ -141,7 +141,45 @@
private void validateArgs(ModSubcommand subcommand, ModOptions modOptions, List<String> args)
throws InvalidArgumentException {
- // More validations can be added here in the future...
+ // Validate output format.
+ switch (subcommand) {
+ case SHOW_REPO -> {
+ switch (modOptions.outputFormat) {
+ case TEXT, STREAMED_JSONPROTO, STREAMED_PROTO -> {} // supported
+ default ->
+ throw new InvalidArgumentException(
+ String.format(
+ "Invalid --output '%s' for the 'show_repo' subcommand. Only 'text',"
+ + " 'streamed_jsonproto', and 'streamed_proto' are supported.",
+ modOptions.outputFormat),
+ Code.INVALID_ARGUMENTS);
+ }
+ }
+ case SHOW_EXTENSION -> {
+ if (modOptions.outputFormat != ModOptions.OutputFormat.TEXT) {
+ throw new InvalidArgumentException(
+ String.format(
+ "Invalid --output '%s' for the 'show_extension' subcommand. Only 'text' is"
+ + " supported.",
+ modOptions.outputFormat),
+ Code.INVALID_ARGUMENTS);
+ }
+ }
+ case ModSubcommand sub when sub.isGraph() -> {
+ switch (modOptions.outputFormat) {
+ case TEXT, JSON, GRAPH -> {} // supported
+ default ->
+ throw new InvalidArgumentException(
+ String.format(
+ "Invalid --output '%s' for the '%s' subcommand. "
+ + "Only 'text', 'json', and 'graph' are supported.",
+ modOptions.outputFormat, sub),
+ Code.INVALID_ARGUMENTS);
+ }
+ }
+ // We don't validate other subcommands yet since they are less confusing.
+ default -> {}
+ }
if (subcommand == ModSubcommand.SHOW_REPO) {
int selectedModes = 0;
@@ -554,9 +592,7 @@
moduleInspector.extensionToRepoInternalNames(),
filterExtensions,
modOptions,
- new OutputStreamWriter(
- env.getReporter().getOutErr().getOutputStream(),
- modOptions.charset == UTF8 ? UTF_8 : US_ASCII));
+ env.getReporter().getOutErr().getOutputStream());
try (SilentCloseable c =
Profiler.instance().profile(ProfilerTask.BZLMOD, "execute mod " + subcommand)) {
diff --git a/src/main/protobuf/build.proto b/src/main/protobuf/build.proto
index 737375d..e665f5e 100644
--- a/src/main/protobuf/build.proto
+++ b/src/main/protobuf/build.proto
@@ -361,6 +361,32 @@
optional stardoc_output.RuleInfo rule_class_info = 17;
}
+// A Bazel repository, either representing a module, or created by a module
+// extension.
+message Repository {
+ // The canonical name of the repository.
+ optional string canonical_name = 1;
+
+ // The name of the repository rule (e.g., http_archive).
+ optional string repo_rule_name = 2;
+
+ // The canonical label of the bzl file that defined the repository rule.
+ optional string repo_rule_bzl_label = 3;
+
+ // The apparent name of the repository, as visible to --base_module.
+ optional string apparent_name = 4;
+
+ // If this repository is a module (not created by a module extension),
+ // this is the module key.
+ optional string module_key = 5;
+
+ // Original name in the repo as created by a module extension.
+ optional string original_name = 6;
+
+ // All of the attributes that describe the repository rule.
+ repeated Attribute attribute = 7;
+}
+
// Direct dependencies of a rule in <depLabel, depConfiguration> form.
message ConfiguredRuleInput {
// Dep's target label.
diff --git a/src/test/java/com/google/devtools/build/lib/bazel/bzlmod/modcommand/ModExecutorTest.java b/src/test/java/com/google/devtools/build/lib/bazel/bzlmod/modcommand/ModExecutorTest.java
index 5a7d16d..9192340 100644
--- a/src/test/java/com/google/devtools/build/lib/bazel/bzlmod/modcommand/ModExecutorTest.java
+++ b/src/test/java/com/google/devtools/build/lib/bazel/bzlmod/modcommand/ModExecutorTest.java
@@ -18,7 +18,6 @@
import static com.google.devtools.build.lib.bazel.bzlmod.BzlmodTestUtil.AugmentedModuleBuilder.buildAugmentedModule;
import static com.google.devtools.build.lib.bazel.bzlmod.BzlmodTestUtil.buildTag;
import static com.google.devtools.build.lib.bazel.bzlmod.BzlmodTestUtil.createModuleKey;
-import static java.nio.charset.StandardCharsets.UTF_8;
import com.google.common.collect.ImmutableBiMap;
import com.google.common.collect.ImmutableMap;
@@ -44,12 +43,11 @@
import com.google.devtools.build.lib.cmdline.LabelSyntaxException;
import com.google.devtools.build.lib.cmdline.PackageIdentifier;
import com.google.devtools.build.lib.util.MaybeCompleteSet;
+import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
-import java.io.OutputStreamWriter;
-import java.io.StringWriter;
-import java.io.Writer;
+import java.io.OutputStream;
import java.nio.file.Files;
import java.util.List;
import java.util.Optional;
@@ -65,7 +63,7 @@
// TODO(andreisolo): Add a Json output test
// TODO(andreisolo): Add a PATH query test
- private final Writer writer = new StringWriter();
+ private final OutputStream outputStream = new ByteArrayOutputStream();
// Tests for the ModExecutor::expandAndPrune core function.
//
@@ -95,7 +93,7 @@
.buildOrThrow();
ModOptions options = ModOptions.getDefaultOptions();
- ModExecutor executor = new ModExecutor(depGraph, options, writer);
+ ModExecutor executor = new ModExecutor(depGraph, options, outputStream);
// RESULT:
// <root> ...> ccc -> ddd
@@ -170,7 +168,7 @@
ModOptions options = ModOptions.getDefaultOptions();
options.cycles = true;
options.depth = 1;
- ModExecutor executor = new ModExecutor(depGraph, options, writer);
+ ModExecutor executor = new ModExecutor(depGraph, options, outputStream);
ImmutableSet<ModuleKey> targets =
ImmutableSet.of(createModuleKey("eee", "1.0"), createModuleKey("hhh", "1.0"));
@@ -246,7 +244,7 @@
ModOptions options = ModOptions.getDefaultOptions();
options.cycles = true;
options.depth = 1;
- ModExecutor executor = new ModExecutor(depGraph, options, writer);
+ ModExecutor executor = new ModExecutor(depGraph, options, outputStream);
ImmutableSet<ModuleKey> targets = ImmutableSet.of(createModuleKey("eee", "1.0"));
// RESULT:
@@ -464,9 +462,7 @@
File file = File.createTempFile("output_text", "txt");
file.deleteOnExit();
- Writer writer = new OutputStreamWriter(new FileOutputStream(file), UTF_8);
- ModExecutor executor = new ModExecutor(depGraph, options, writer);
ImmutableSet<ModuleKey> targets =
ImmutableSet.of(
createModuleKey("C", "0.1"),
@@ -476,11 +472,16 @@
createModuleKey("E", "1.0"),
createModuleKey("H", "1.0"));
- // Double check for human error
- assertThat(executor.expandPathsToTargets(ImmutableSet.of(ModuleKey.ROOT), targets, false))
- .isEqualTo(result);
+ try (var outputStream = new FileOutputStream(file)) {
+ ModExecutor executor = new ModExecutor(depGraph, options, outputStream);
- executor.allPaths(ImmutableSet.of(ModuleKey.ROOT), targets);
+ // Double check for human error
+ assertThat(executor.expandPathsToTargets(ImmutableSet.of(ModuleKey.ROOT), targets, false))
+ .isEqualTo(result);
+
+ executor.allPaths(ImmutableSet.of(ModuleKey.ROOT), targets);
+ }
+
List<String> textOutput = Files.readAllLines(file.toPath());
assertThat(textOutput)
@@ -502,10 +503,10 @@
options.outputFormat = OutputFormat.GRAPH;
File fileGraph = File.createTempFile("output_graph", "txt");
fileGraph.deleteOnExit();
- writer = new OutputStreamWriter(new FileOutputStream(fileGraph), UTF_8);
- executor = new ModExecutor(depGraph, options, writer);
-
- executor.allPaths(ImmutableSet.of(ModuleKey.ROOT), targets);
+ try (var outputStream = new FileOutputStream(fileGraph)) {
+ var executor = new ModExecutor(depGraph, options, outputStream);
+ executor.allPaths(ImmutableSet.of(ModuleKey.ROOT), targets);
+ }
List<String> graphOutput = Files.readAllLines(fileGraph.toPath());
assertThat(graphOutput)
@@ -662,7 +663,6 @@
File file = File.createTempFile("output_text", "txt");
file.deleteOnExit();
- Writer writer = new OutputStreamWriter(new FileOutputStream(file), UTF_8);
// Contains the already-filtered map of target extensions along with their full list of repos
ImmutableSetMultimap<ModuleExtensionId, String> extensionRepos =
@@ -675,11 +675,12 @@
options.outputFormat = OutputFormat.TEXT;
options.extensionInfo = ExtensionShow.ALL;
- ModExecutor executor =
- new ModExecutor(
- depGraph, extensionUsages, extensionRepos, Optional.empty(), options, writer);
-
- executor.graph(ImmutableSet.of(ModuleKey.ROOT));
+ try (var outputStream = new FileOutputStream(file)) {
+ ModExecutor executor =
+ new ModExecutor(
+ depGraph, extensionUsages, extensionRepos, Optional.empty(), options, outputStream);
+ executor.graph(ImmutableSet.of(ModuleKey.ROOT));
+ }
List<String> textOutput = Files.readAllLines(file.toPath());
@@ -714,12 +715,13 @@
options.outputFormat = OutputFormat.GRAPH;
File fileGraph = File.createTempFile("output_graph", "txt");
fileGraph.deleteOnExit();
- writer = new OutputStreamWriter(new FileOutputStream(fileGraph), UTF_8);
- executor =
- new ModExecutor(
- depGraph, extensionUsages, extensionRepos, Optional.empty(), options, writer);
- executor.graph(ImmutableSet.of(ModuleKey.ROOT));
+ try (var outputStream = new FileOutputStream(fileGraph)) {
+ var executor =
+ new ModExecutor(
+ depGraph, extensionUsages, extensionRepos, Optional.empty(), options, outputStream);
+ executor.graph(ImmutableSet.of(ModuleKey.ROOT));
+ }
List<String> graphOutput = Files.readAllLines(fileGraph.toPath());
assertThat(graphOutput)
@@ -764,18 +766,19 @@
options.depth = 1;
File fileText2 = File.createTempFile("output_text2", "txt");
fileText2.deleteOnExit();
- writer = new OutputStreamWriter(new FileOutputStream(fileText2), UTF_8);
- executor =
- new ModExecutor(
- depGraph,
- extensionUsages,
- extensionRepos,
- Optional.of(MaybeCompleteSet.copyOf(ImmutableSet.of(mavenId))),
- options,
- writer);
+ try (var outputStream = new FileOutputStream(fileText2)) {
+ var executor =
+ new ModExecutor(
+ depGraph,
+ extensionUsages,
+ extensionRepos,
+ Optional.of(MaybeCompleteSet.copyOf(ImmutableSet.of(mavenId))),
+ options,
+ outputStream);
+ executor.allPaths(
+ ImmutableSet.of(ModuleKey.ROOT), ImmutableSet.of(createModuleKey("Y", "2.0")));
+ }
- executor.allPaths(
- ImmutableSet.of(ModuleKey.ROOT), ImmutableSet.of(createModuleKey("Y", "2.0")));
List<String> textOutput2 = Files.readAllLines(fileText2.toPath());
assertThat(textOutput2)
@@ -887,13 +890,13 @@
File file = File.createTempFile("output_text", "txt");
file.deleteOnExit();
- Writer writer = new OutputStreamWriter(new FileOutputStream(file), UTF_8);
- ModExecutor executor = new ModExecutor(depGraph, options, writer);
- // Test `executor.allPaths`, it should output all "interesting" paths to the target modules.
- executor.allPaths(
- ImmutableSet.of(ModuleKey.ROOT), ImmutableSet.of(createModuleKey("D", "1.0")));
- writer.close();
+ try (var outputStream = new FileOutputStream(file)) {
+ ModExecutor executor = new ModExecutor(depGraph, options, outputStream);
+ // Test `executor.allPaths`, it should output all "interesting" paths to the target modules.
+ executor.allPaths(
+ ImmutableSet.of(ModuleKey.ROOT), ImmutableSet.of(createModuleKey("D", "1.0")));
+ }
List<String> textOutput = Files.readAllLines(file.toPath());
@@ -913,8 +916,8 @@
// be the shortest one
File file2 = File.createTempFile("output_text", "txt");
file2.deleteOnExit();
- try (Writer writer2 = new OutputStreamWriter(new FileOutputStream(file2), UTF_8)) {
- ModExecutor executor2 = new ModExecutor(depGraph, options, writer2);
+ try (var outputStream2 = new FileOutputStream(file2)) {
+ ModExecutor executor2 = new ModExecutor(depGraph, options, outputStream2);
executor2.path(ImmutableSet.of(ModuleKey.ROOT), ImmutableSet.of(createModuleKey("D", "1.0")));
}
@@ -926,8 +929,8 @@
// Test multiple targets D and G for allPaths
File file3 = File.createTempFile("output_text_multi_all", "txt");
file3.deleteOnExit();
- try (Writer writer3 = new OutputStreamWriter(new FileOutputStream(file3), UTF_8)) {
- ModExecutor executor3 = new ModExecutor(depGraph, options, writer3);
+ try (var outputStream3 = new FileOutputStream(file3)) {
+ ModExecutor executor3 = new ModExecutor(depGraph, options, outputStream3);
executor3.allPaths(
ImmutableSet.of(ModuleKey.ROOT),
ImmutableSet.of(createModuleKey("D", "1.0"), createModuleKey("G", "1.0")));
@@ -950,8 +953,8 @@
// Test multiple targets D and G for path (shortest path)
File file4 = File.createTempFile("output_text_multi_path", "txt");
file4.deleteOnExit();
- try (Writer writer4 = new OutputStreamWriter(new FileOutputStream(file4), UTF_8)) {
- ModExecutor executor4 = new ModExecutor(depGraph, options, writer4);
+ try (var outputStream4 = new FileOutputStream(file4)) {
+ ModExecutor executor4 = new ModExecutor(depGraph, options, outputStream4);
executor4.path(
ImmutableSet.of(ModuleKey.ROOT),
ImmutableSet.of(createModuleKey("D", "1.0"), createModuleKey("G", "1.0")));
@@ -965,8 +968,8 @@
// Test starting from E to D for allPaths
File file5 = File.createTempFile("output_text_E_to_D_all", "txt");
file5.deleteOnExit();
- try (Writer writer5 = new OutputStreamWriter(new FileOutputStream(file5), UTF_8)) {
- ModExecutor executor5 = new ModExecutor(depGraph, options, writer5);
+ try (var outputStream5 = new FileOutputStream(file5)) {
+ ModExecutor executor5 = new ModExecutor(depGraph, options, outputStream5);
executor5.allPaths(
ImmutableSet.of(createModuleKey("E", "1.0")),
ImmutableSet.of(createModuleKey("D", "1.0")));
@@ -1012,8 +1015,8 @@
File file = File.createTempFile("output_text_cycle", "txt");
file.deleteOnExit();
- try (Writer writer = new OutputStreamWriter(new FileOutputStream(file), UTF_8)) {
- ModExecutor executor = new ModExecutor(depGraph, options, writer);
+ try (var outputStream = new FileOutputStream(file)) {
+ ModExecutor executor = new ModExecutor(depGraph, options, outputStream);
executor.graph(ImmutableSet.of(ModuleKey.ROOT));
}
@@ -1065,20 +1068,18 @@
File file = File.createTempFile("output_text_repro", "txt");
file.deleteOnExit();
- Writer writer = new OutputStreamWriter(new FileOutputStream(file), UTF_8);
-
- ModExecutor executor =
- new ModExecutor(
- depGraph,
- extensionUsages,
- extensionRepos,
- Optional.of(MaybeCompleteSet.copyOf(ImmutableSet.of(mavenId))),
- options,
- writer);
-
- // This should not throw NPE
- executor.graph(ImmutableSet.of(ModuleKey.ROOT));
- writer.close();
+ try (var outputStream = new FileOutputStream(file)) {
+ ModExecutor executor =
+ new ModExecutor(
+ depGraph,
+ extensionUsages,
+ extensionRepos,
+ Optional.of(MaybeCompleteSet.copyOf(ImmutableSet.of(mavenId))),
+ options,
+ outputStream);
+ // This should not throw NPE
+ executor.graph(ImmutableSet.of(ModuleKey.ROOT));
+ }
List<String> textOutput = Files.readAllLines(file.toPath());
assertThat(textOutput)
@@ -1159,7 +1160,7 @@
File file = File.createTempFile("output_text_cycle_ext", "txt");
file.deleteOnExit();
- try (Writer writer = new OutputStreamWriter(new FileOutputStream(file), UTF_8)) {
+ try (var outputStream = new FileOutputStream(file)) {
ModExecutor executor =
new ModExecutor(
depGraph,
@@ -1167,7 +1168,7 @@
extensionRepos,
Optional.of(MaybeCompleteSet.copyOf(ImmutableSet.of(extensionId))),
options,
- writer);
+ outputStream);
executor.graph(ImmutableSet.of(ModuleKey.ROOT));
}
diff --git a/src/test/py/bazel/bzlmod/mod_command_test.py b/src/test/py/bazel/bzlmod/mod_command_test.py
index 42c11cd..b10d7ce 100644
--- a/src/test/py/bazel/bzlmod/mod_command_test.py
+++ b/src/test/py/bazel/bzlmod/mod_command_test.py
@@ -504,6 +504,148 @@
],
)
+ def testShowModuleAndExtensionReposFromBaseModuleJson(self):
+ _, stdout, _ = self.RunBazel(
+ [
+ 'mod',
+ 'show_repo',
+ '--base_module=foo@2.0',
+ '--output=streamed_jsonproto',
+ '@bar_from_foo2',
+ 'ext@1.0',
+ '@my_repo3',
+ 'bar',
+ ],
+ rstrip=True,
+ )
+ repos = [json.loads(line) for line in stdout]
+
+ ignored_attrs = {
+ 'integrity',
+ 'path',
+ 'remote_module_file_urls',
+ 'remote_module_file_integrity',
+ 'urls',
+ }
+ for repo in repos:
+ attrs = repo.get('attribute')
+ if attrs:
+ repo['attribute'] = [
+ attr
+ for attr in attrs
+ if attr.get('explicitlySpecified', False)
+ and attr['name'] not in ignored_attrs
+ ]
+
+ self.assertListEqual(
+ repos,
+ [
+ {
+ 'canonicalName': 'bar+',
+ 'repoRuleName': 'http_archive',
+ 'repoRuleBzlLabel': (
+ '@@bazel_tools//tools/build_defs/repo:http.bzl'
+ ),
+ 'apparentName': '@bar_from_foo2',
+ 'attribute': [
+ {
+ 'name': 'strip_prefix',
+ 'type': 'STRING',
+ 'stringValue': '',
+ 'explicitlySpecified': True,
+ 'nodep': False,
+ },
+ {
+ 'name': 'remote_file_urls',
+ 'type': 'STRING_LIST_DICT',
+ 'explicitlySpecified': True,
+ },
+ {
+ 'name': 'remote_file_integrity',
+ 'type': 'STRING_DICT',
+ 'explicitlySpecified': True,
+ },
+ {
+ 'name': 'remote_patches',
+ 'type': 'STRING_DICT',
+ 'explicitlySpecified': True,
+ },
+ {
+ 'name': 'remote_patch_strip',
+ 'type': 'INTEGER',
+ 'intValue': 0,
+ 'explicitlySpecified': True,
+ },
+ ],
+ },
+ {
+ 'canonicalName': 'ext+',
+ 'repoRuleName': 'local_repository',
+ 'repoRuleBzlLabel': (
+ '@@bazel_tools//tools/build_defs/repo:local.bzl'
+ ),
+ 'moduleKey': 'ext@1.0',
+ 'attribute': [],
+ },
+ {
+ 'canonicalName': 'ext++ext+repo3',
+ 'repoRuleName': 'data_repo',
+ 'repoRuleBzlLabel': '@@ext+//:ext.bzl',
+ 'apparentName': '@my_repo3',
+ 'originalName': 'repo3',
+ 'attribute': [
+ {
+ 'name': 'data',
+ 'type': 'STRING',
+ 'stringValue': 'requested repo',
+ 'nodep': False,
+ 'explicitlySpecified': True,
+ },
+ ],
+ },
+ {
+ 'canonicalName': 'bar+',
+ 'repoRuleName': 'http_archive',
+ 'repoRuleBzlLabel': (
+ '@@bazel_tools//tools/build_defs/repo:http.bzl'
+ ),
+ 'moduleKey': 'bar@2.0',
+ 'attribute': [
+ {
+ 'name': 'strip_prefix',
+ 'type': 'STRING',
+ 'stringValue': '',
+ 'explicitlySpecified': True,
+ 'nodep': False,
+ },
+ {
+ 'name': 'remote_file_urls',
+ 'type': 'STRING_LIST_DICT',
+ 'explicitlySpecified': True,
+ },
+ {
+ 'name': 'remote_file_integrity',
+ 'type': 'STRING_DICT',
+ 'explicitlySpecified': True,
+ },
+ {
+ 'name': 'remote_patches',
+ 'type': 'STRING_DICT',
+ 'explicitlySpecified': True,
+ },
+ {
+ 'name': 'remote_patch_strip',
+ 'type': 'INTEGER',
+ 'intValue': 0,
+ 'explicitlySpecified': True,
+ },
+ ],
+ },
+ ],
+ 'wrong output in the show query for module and extension-generated'
+ ' repos',
+ )
+
def testShowModuleAndExtensionReposFromBaseModule(self):
_, stdout, _ = self.RunBazel(
[
@@ -526,11 +668,11 @@
)
self.assertRegex(stdout.pop(8), r'^ remote_module_file_integrity = ".*",$')
self.assertRegex(stdout.pop(15), r'^ path = ".*",$')
- self.assertRegex(stdout.pop(35), r'^ urls = \[".*"\],$')
- self.assertRegex(stdout.pop(35), r'^ integrity = ".*",$')
- self.assertRegex(stdout.pop(39), r'^ remote_module_file_urls = \[".*"\],$')
+ self.assertRegex(stdout.pop(37), r'^ urls = \[".*"\],$')
+ self.assertRegex(stdout.pop(37), r'^ integrity = ".*",$')
+ self.assertRegex(stdout.pop(41), r'^ remote_module_file_urls = \[".*"\],$')
self.assertRegex(
- stdout.pop(39), r'^ remote_module_file_integrity = ".*",$'
+ stdout.pop(41), r'^ remote_module_file_integrity = ".*",$'
)
self.assertListEqual(
stdout,
@@ -567,6 +709,7 @@
'load("@@ext+//:ext.bzl", "data_repo")',
'data_repo(',
' name = "ext++ext+repo3",',
+ ' _original_name = "repo3",',
' data = "requested repo",',
')',
'',
@@ -574,6 +717,7 @@
'load("@@ext+//:ext.bzl", "data_repo")',
'data_repo(',
' name = "ext++ext+repo4",',
+ ' _original_name = "repo4",',
' data = "requested repo",',
')',
'',
@@ -584,14 +728,14 @@
),
'http_archive(',
' name = "bar+",',
- # pop(35) -- urls=[...]
- # pop(35) -- integrity=...
+ # pop(37) -- urls=[...]
+ # pop(37) -- integrity=...
' strip_prefix = "",',
' remote_patches = {},',
' remote_file_urls = {},',
' remote_file_integrity = {},',
- # pop(39) -- remote_module_file_urls=[...]
- # pop(39) -- remote_module_file_integrity=...
+ # pop(41) -- remote_module_file_urls=[...]
+ # pop(41) -- remote_module_file_integrity=...
' remote_patch_strip = 0,',
')',
'',
@@ -792,6 +936,16 @@
self.assertIn('Builtin or overridden repo located at: ', stdout)
self.assertIn('/embedded_tools', stdout)
+ def testShowRepoBazelToolsJson(self):
+ # @bazel_tools should be omitted from proto outputs
+ exit_code, stdout, stderr = self.RunBazel(
+ ['mod', 'show_repo', '--output=streamed_jsonproto', '@bazel_tools'],
+ rstrip=True,
+ )
+ self.AssertExitCode(exit_code, 0, stderr)
+ stdout = '\n'.join(stdout)
+ self.assertEqual('', stdout)
+
def testDumpRepoMapping(self):
_, stdout, _ = self.RunBazel(
[
