[8.8.0] Accept and ignore Bazel 10 Starlark type syntax (#30373)
### Description
Backports the Bazel 10 Starlark type syntax parser to the 8.x line in
"accept but ignore" mode: `.bzl` files using type annotations, type
aliases, generic `def` parameters, `...`, `cast()`, and `isinstance()`
now load successfully, with all type information ignored. This matches
master's default behavior (`--experimental_starlark_type_syntax`
defaults to true, type checking off ⇒ annotations are parsed tolerantly
as arbitrary expressions and never resolved).
Without this PR, 8.8.0 fails to load even the most basic annotated file
with a syntax error:
```starlark
def f(x: list[str]): # syntax error at ':': type annotations are disallowed
pass
```
The following commits are cherry-picked, in order:
* 299e903710 Extend syntax with parameter type annotations
* 2dbda274cd Use type specific productions
* d1c3727795 Extend syntax with return types
* ab034aea73 Implement --experimental_starlark_types* flags
* fb7ca12439 Implement TypeApplication and fix NodePrinter
* aa61fc8a8d Allow empty type arguments for type applications (parser
support only)
* 0b615ff55e Introduce Expression.parseTypeExpression() (parser support
only)
* f60a22e8ef Allow parsing arbitrary uninterpreted Starlark type
expressions
* f3653ca8ae Allow parsing type alias statements
* 88678c00c4 Allow parsing generic type parameters in `def` statements
* 96d2f60fb9 Allow parsing `cast` expressions (and evaluating them as
the value argument)
* 664e87b594 Evaluate type alias statements as a no-op
* 5c8655ca1f Add variable type annotation syntax and resolver behavior
* 2390ce6760 Add Ellipsis token and node
* 515083c24a Add `isinstance` keyword, and allow isinstance(x,t) to be
parsed - but not resolved
* 44665ecd71 Properly gate dynamic type checking with flag (syntax
tolerance only)
* f38f0e448a Disallow type syntax in .scl files
* f1b3273f9f, 2004bad12f, f0011d8f29 (the three parser-only slices from
#30092: `tuple[T, ...]`, `tuple[()]`, struct type)
### How to review
Every commit keeps its original `Change-Id`, so the deviation of each
cherry-pick from its master original can be computed mechanically with
`git range-diff` (checking out this PR as `<pr-head>`):
```sh
git log --reverse --format=%H origin/release-8.8.0..<pr-head> | while read pick; do
cid=$(git log -1 --format=%B $pick | sed -n 's/^Change-Id: //p' | head -1)
orig=$(git log --format=%H --grep="Change-Id: $cid" origin/master | tail -1)
git range-diff --creation-factor=100 $orig^! $pick^!
done
```
Only the lines range-diff marks as changed between the two patches need
human review; everything else is upstream code verbatim. Running this
today gives the following review-effort map (changed diff lines per
commit): all commits are at or below 22 except 0b615ff55e (99),
f60a22e8ef (51), 664e87b594 (44), 5c8655ca1f (267), 44665ecd71 (133),
and f38f0e448a (91) — which are exactly the sliced/adapted commits
explained below. For the last three commits, substitute the
already-sliced commits from #30092 (1681dfb115, b172b104c3, 745ec85067)
as the range-diff baseline instead of the master originals; against that
baseline they deviate by 2, 12, and 2 lines respectively.
Each commit is best reviewed against its master original (referenced by
`PiperOrigin-RevId`/`Change-Id` in the message). Everything not listed
below is verbatim upstream code; in particular, the type grammar in
`Parser.java` (all `parseType*` productions, type aliases, generics,
`cast`/`isinstance`) and the new AST node files `TypeAliasStatement`,
`TypeApplication`, `Ellipsis`, and `IsInstanceExpression` are
byte-identical to current master.
Non-trivial deviations from master, grouped by cause:
**1. The type checking machinery is intentionally not backported**
(~4,800 lines on master: `StarlarkType`, `Types`, `TypeChecker`,
`TypeTagger`, `TypeResolver`, and the dynamic checks in
`Eval`/`StarlarkFunction`):
* 0b615ff55e is taken without the static `Resolver.resolveType()` (it
needs the types package).
* 44665ecd71 is reduced to its
`FileOptions`/`Parser`/flag/`BzlCompileFunction` hunks; the
`Eval`/`StarlarkFunction` dynamic-check gating hunks are dropped. The
`StarlarkSemantics.EXPERIMENTAL_STARLARK_TYPE_CHECKING` key constant is
added by hand since `BzlCompileFunction` still needs it.
* `CastExpression` lacks master's `starlarkType` field/getter/setter
(the type-tagging slot, only written by `TypeTagger`).
* Hunks touching checker test files
(`TypeCheckTest`/`DynamicTypeCheckTest`, the syntax-package
`StarlarkTypesTest`) are dropped wherever a commit touched them.
* Net effect: `--experimental_starlark_type_checking` (default off) on
8.x only switches annotations from tolerant to structured parsing; it
performs no type checking.
**2. The doc-comments feature (03921eded1) is not backported**, so its
threading is stripped from 5c8655ca1f's hunks:
* `AssignmentStatement` and `VarStatement` have no `docComments`
field/getter; their constructors take one fewer parameter.
* `Parser#parseAssignment` does not call `maybeParseTrailingDocComment`
(method not introduced), and `Expression#parseExpression` does not skip
`DOC_COMMENT_*` tokens.
* `Resolver#bind` is `bind(id, isLoad, hasType)` instead of master's
four-parameter form; `createBindingsForLHS` takes no doc-comments
parameter.
**3. `Binding.isSyntactic` does not exist on 8.x.** In the
one-annotation-per-declaration check inside `Resolver#bind`, master's
`bind.isSyntactic` is replaced by `bind.first != null` (on 8.x,
`Binding#first` is documented as "first binding use, if syntactic", so
the two are equivalent).
**4. Older test scaffolding on 8.x:**
* `EvaluationTestCase` gains `setFileOptions`/`getFileOptions` (ported
from master, folded into the 664e87b594 pick) because the cherry-picked
`EvaluationTest` cases need them.
* In `ParserTest`/`NodePrinterTest`/`ResolverTest`, only each commit's
own payload tests are taken where the surrounding context consists of
master-only tests that don't exist on 8.x; two `NodePrinterTest`
assertions stay in 8.x's `join(...)` style.
**5. The `parseExpression` → `parseExpr` rename** from f60a22e8ef is
applied in full so the parser stays textually close to master.
### Verification
`//src/test/java/net/starlark/java/...` and
`//src/test/java/com/google/devtools/build/lib/starlark:StarlarkTypesTest`
pass. Additionally verified end-to-end: a workspace exercising unknown
type names, providers as types, `list[str]`, `tuple[int, ...]`,
`tuple[()]`, `struct[{"a": int}]`, parameterized type aliases, variable
annotations, string-literal types, and runtime `cast()` passthrough
loads identically (default flags) with a Bazel built from this branch
and one built from master, and annotated module-level assignments
execute correctly.
### Motivation
Forward compatibility for the 8.x LTS line: once rulesets start adopting
Bazel 10 type annotations, users on Bazel 8 must still be able to load
those rulesets. This is the 8.x counterpart of the tolerance work that
already landed for 9.0.0 (#27760, #27838, #28069) and is pending for
9.3.0 (#30092).
### Build API Changes
Yes: this extends the accepted `.bzl` grammar and adds the experimental
flags `--experimental_starlark_type_syntax` (default true),
`--experimental_starlark_types_allowed_paths`, and
`--experimental_starlark_type_checking` (default false) to 8.x. Part of
the Starlark types effort (#27370). The change is purely additive and
backward compatible: previously invalid syntax now parses and is
ignored; `.scl` files continue to reject type syntax.
### Checklist
- [x] I have added tests for the new use cases (if any).
- [ ] I have updated the documentation (if applicable).
### Release Notes
RELNOTES: Bazel now parses and ignores the Starlark type annotation
syntax introduced by newer Bazel versions (gated by
`--experimental_starlark_type_syntax`, enabled by default).
---------
Co-authored-by: Googler <ilist@google.com>
Co-authored-by: arostovtsev <arostovtsev@google.com>
Co-authored-by: brandjon <brandjon@google.com>
Co-authored-by: Ian (Hee) Cha <heec@google.com>diff --git a/src/main/java/com/google/devtools/build/lib/packages/semantics/BuildLanguageOptions.java b/src/main/java/com/google/devtools/build/lib/packages/semantics/BuildLanguageOptions.java
index 6f41b54..9699437 100644
--- a/src/main/java/com/google/devtools/build/lib/packages/semantics/BuildLanguageOptions.java
+++ b/src/main/java/com/google/devtools/build/lib/packages/semantics/BuildLanguageOptions.java
@@ -777,6 +777,30 @@
public boolean experimentalDormantDeps;
@Option(
+ name = "experimental_starlark_type_syntax",
+ defaultValue = FlagConstants.DEFAULT_EXPERIMENTAL_STARLARK_TYPE_SYNTAX,
+ documentationCategory = OptionDocumentationCategory.STARLARK_SEMANTICS,
+ effectTags = {OptionEffectTag.LOADING_AND_ANALYSIS},
+ metadataTags = {OptionMetadataTag.EXPERIMENTAL},
+ help =
+ """
+ Enables type annotations and related syntax in .bzl files. Locations of files where \
+ these are allowed is further restricted by `--experimental_starlark_types_allowed_paths`.
+ Type syntax is never permitted in .scl files regardless of this flag.
+ """)
+ public boolean experimentalStarlarkTypeSyntax;
+
+ @Option(
+ name = "experimental_starlark_types_allowed_paths",
+ converter = CommaSeparatedOptionListConverter.class,
+ defaultValue = FlagConstants.DEFAULT_EXPERIMENTAL_STARLARK_TYPES_ALLOWED_PATHS,
+ documentationCategory = OptionDocumentationCategory.STARLARK_SEMANTICS,
+ effectTags = {OptionEffectTag.LOADING_AND_ANALYSIS},
+ metadataTags = {OptionMetadataTag.EXPERIMENTAL},
+ help = "List of canonical Label prefixes under which Starlark type annotations are allowed.")
+ public List<String> experimentalStarlarkTypesAllowedPaths;
+
+ @Option(
name = "incompatible_enable_deprecated_label_apis",
defaultValue = "true",
documentationCategory = OptionDocumentationCategory.STARLARK_SEMANTICS,
@@ -1007,6 +1031,8 @@
incompatibleDisableTargetDefaultProviderFields)
.setBool(EXPERIMENTAL_RULE_EXTENSION_API, experimentalRuleExtensionApi)
.setBool(EXPERIMENTAL_DORMANT_DEPS, experimentalDormantDeps)
+ .setBool(EXPERIMENTAL_STARLARK_TYPE_SYNTAX, experimentalStarlarkTypeSyntax)
+ .set(EXPERIMENTAL_STARLARK_TYPES_ALLOWED_PATHS, experimentalStarlarkTypesAllowedPaths)
.setBool(INCOMPATIBLE_ENABLE_DEPRECATED_LABEL_APIS, enableDeprecatedLabelApis)
.setBool(
INCOMPATIBLE_STOP_EXPORTING_BUILD_FILE_PATH, incompatibleStopExportingBuildFilePath)
@@ -1193,6 +1219,9 @@
public static final String EXPERIMENTAL_RULE_EXTENSION_API =
FlagConstants.DEFAULT_EXPERIMENTAL_RULE_EXTENSION_API_NAME;
public static final String EXPERIMENTAL_DORMANT_DEPS = "-experimental_dormant_deps";
+
+ public static final String EXPERIMENTAL_STARLARK_TYPE_SYNTAX =
+ FlagConstants.EXPERIMENTAL_STARLARK_TYPE_SYNTAX_FLAG_NAME;
public static final String INCOMPATIBLE_ENABLE_DEPRECATED_LABEL_APIS =
"+incompatible_enable_deprecated_label_apis";
public static final String INCOMPATIBLE_STOP_EXPORTING_BUILD_FILE_PATH =
@@ -1229,6 +1258,15 @@
INCOMPATIBLE_ENFORCE_STARLARK_UTF8 =
new StarlarkSemantics.Key<>(
"incompatible_enforce_starlark_utf8", Utf8EnforcementMode.WARNING);
+ public static final StarlarkSemantics.Key<List<String>>
+ EXPERIMENTAL_STARLARK_TYPES_ALLOWED_PATHS =
+ new StarlarkSemantics.Key<>(
+ "experimental_starlark_types",
+ FlagConstants.DEFAULT_EXPERIMENTAL_STARLARK_TYPES_ALLOWED_PATHS.isEmpty()
+ ? ImmutableList.of()
+ : ImmutableList.copyOf(
+ FlagConstants.DEFAULT_EXPERIMENTAL_STARLARK_TYPES_ALLOWED_PATHS.split(",")));
+
public static final StarlarkSemantics.Key<Long> MAX_COMPUTATION_STEPS =
new StarlarkSemantics.Key<>("max_computation_steps", 0L);
public static final StarlarkSemantics.Key<Integer> NESTED_SET_DEPTH_LIMIT =
diff --git a/src/main/java/com/google/devtools/build/lib/packages/semantics/FlagConstants.java b/src/main/java/com/google/devtools/build/lib/packages/semantics/FlagConstants.java
index f550e1c..bdaa781 100644
--- a/src/main/java/com/google/devtools/build/lib/packages/semantics/FlagConstants.java
+++ b/src/main/java/com/google/devtools/build/lib/packages/semantics/FlagConstants.java
@@ -22,9 +22,11 @@
private FlagConstants() {}
public static final String DEFAULT_EXPERIMENTAL_RULE_EXTENSION_API = "true";
- public static final String DEFAULT_EXPERIMENTAL_RULE_EXTENSION_API_NAME = "+experimental_rule_extension_api";
+ public static final String DEFAULT_EXPERIMENTAL_RULE_EXTENSION_API_NAME =
+ "+experimental_rule_extension_api";
- // TODO - ilist@: once Java providers are removed, the whole line can be compressed to "@rules_java"
+ // TODO - ilist@: once Java providers are removed, the whole line can be compressed to
+ // "@rules_java"
public static final String DEFAULT_INCOMPATIBLE_AUTOLOAD_EXTERNALLY =
"+@rules_python," +
"+java_common,+JavaInfo,+JavaPluginInfo,ProguardSpecProvider," +
@@ -34,6 +36,13 @@
"@rules_shell," +
"+@rules_android";
+ // Enable annotations, but not actual type checking, with the effect that the parser tolerates
+ // arbitrary expressions in annotations for now.
+ public static final String EXPERIMENTAL_STARLARK_TYPE_SYNTAX_FLAG_NAME =
+ "+experimental_starlark_type_syntax";
+ public static final String DEFAULT_EXPERIMENTAL_STARLARK_TYPE_SYNTAX = "true";
+ public static final String DEFAULT_EXPERIMENTAL_STARLARK_TYPES_ALLOWED_PATHS = "";
+
public static final String DEFAULT_INCOMPATIBLE_PACKAGE_GROUP_HAS_PUBLIC_SYNTAX = "true";
public static final String DEFAULT_INCOMPATIBLE_FIX_PACKAGE_GROUP_REPOROOT_SYNTAX = "true";
diff --git a/src/main/java/com/google/devtools/build/lib/skyframe/BzlCompileFunction.java b/src/main/java/com/google/devtools/build/lib/skyframe/BzlCompileFunction.java
index c65f773..184a5d4 100644
--- a/src/main/java/com/google/devtools/build/lib/skyframe/BzlCompileFunction.java
+++ b/src/main/java/com/google/devtools/build/lib/skyframe/BzlCompileFunction.java
@@ -20,6 +20,7 @@
import com.google.devtools.build.lib.cmdline.BazelCompileContext;
import com.google.devtools.build.lib.cmdline.Label;
import com.google.devtools.build.lib.events.Event;
+import com.google.devtools.build.lib.events.EventHandler;
import com.google.devtools.build.lib.packages.AutoloadSymbols;
import com.google.devtools.build.lib.packages.BazelStarlarkEnvironment;
import com.google.devtools.build.lib.packages.semantics.BuildLanguageOptions;
@@ -32,10 +33,12 @@
import com.google.devtools.build.skyframe.SkyKey;
import com.google.devtools.build.skyframe.SkyValue;
import java.io.IOException;
+import java.util.List;
import javax.annotation.Nullable;
import net.starlark.java.eval.Module;
import net.starlark.java.eval.StarlarkSemantics;
import net.starlark.java.syntax.FileOptions;
+import net.starlark.java.syntax.Location;
import net.starlark.java.syntax.ParserInput;
import net.starlark.java.syntax.Program;
import net.starlark.java.syntax.StarlarkFile;
@@ -185,6 +188,9 @@
StarlarkUtil.InvalidUtf8Exception e) {
return BzlCompileValue.noFile("compilation of '%s' failed", inputName);
}
+
+ boolean useTypeSyntax = shouldUseTypeSyntax(semantics, key);
+
FileOptions options =
FileOptions.builder()
// By default, Starlark load statements create file-local bindings.
@@ -199,6 +205,10 @@
// matching the error message or reworking the interpreter API to put more structured
// detail in errors (i.e. new fields or error subclasses).
.stringLiteralsAreAsciiOnly(key.isSclDialect())
+ .allowTypeSyntax(useTypeSyntax)
+ // Bazel 8 only tolerates type syntax, it never type checks, so arbitrary expressions
+ // are accepted in type positions.
+ .tolerateInvalidTypeExpressions(true)
.build();
StarlarkFile file = StarlarkFile.parse(input, options);
@@ -221,7 +231,7 @@
Program prog = Program.compileFile(file, module);
return BzlCompileValue.withProgram(prog, digest);
} catch (SyntaxError.Exception ex) {
- Event.replayEventsOn(env.getListener(), ex.errors());
+ addSyntaxErrorsToListener(env.getListener(), ex.errors(), key);
return BzlCompileValue.noFile(
"compilation of module '%s'%s failed",
key.label.toPathFragment(),
@@ -229,6 +239,54 @@
}
}
+ /**
+ * Whether the file should permit type syntax (annotations, etc.) based on flags and the type of
+ * file.
+ */
+ private static boolean shouldUseTypeSyntax(StarlarkSemantics semantics, BzlCompileValue.Key key) {
+ boolean typeSyntaxFlag =
+ semantics.getBool(BuildLanguageOptions.EXPERIMENTAL_STARLARK_TYPE_SYNTAX);
+ List<String> allowlist =
+ semantics.get(BuildLanguageOptions.EXPERIMENTAL_STARLARK_TYPES_ALLOWED_PATHS);
+
+ boolean okFiletype =
+ // annotations in prelude not allowed (it has null key.label)
+ !key.isBuildPrelude()
+ // annotations in SCL now allowed (not yet compatible with Go-Starlark interpreter)
+ && !key.isSclDialect();
+
+ if (typeSyntaxFlag && okFiletype) {
+ if (allowlist.isEmpty()
+ || allowlist.stream().anyMatch(s -> key.label.getCanonicalForm().startsWith(s))) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Replays the syntax errors from a file onto an event handler, adding more context if necessary.
+ */
+ private static void addSyntaxErrorsToListener(
+ EventHandler handler, List<SyntaxError> errors, BzlCompileValue.Key key) {
+ Event.replayEventsOn(handler, errors);
+ // If type annotations are disallowed, it could either be because the required flags aren't
+ // enabled or because the filetype disallows it.
+ for (var err : errors) {
+ if (err.message().contains(": type annotations are disallowed")) {
+ Location fileLoc = Location.fromFile(err.location().file());
+ String explanation =
+ key.isSclDialect()
+ ? "Type annotations are not permitted in .scl files."
+ : """
+ Type annotations syntax can be enabled with --experimental_starlark_type_syntax \
+ and/or --experimental_starlark_types_allowed_paths.\
+ """;
+ handler.handle(Event.error(fileLoc, explanation));
+ }
+ }
+ }
+
static final class FailedIOException extends Exception {
private final Transience transience;
diff --git a/src/main/java/net/starlark/java/eval/Eval.java b/src/main/java/net/starlark/java/eval/Eval.java
index cbe576d..2a0fcac 100644
--- a/src/main/java/net/starlark/java/eval/Eval.java
+++ b/src/main/java/net/starlark/java/eval/Eval.java
@@ -28,6 +28,7 @@
import net.starlark.java.syntax.AssignmentStatement;
import net.starlark.java.syntax.BinaryOperatorExpression;
import net.starlark.java.syntax.CallExpression;
+import net.starlark.java.syntax.CastExpression;
import net.starlark.java.syntax.Comprehension;
import net.starlark.java.syntax.ConditionalExpression;
import net.starlark.java.syntax.DefStatement;
@@ -304,6 +305,10 @@
return TokenKind.PASS;
case RETURN:
return execReturn(fr, (ReturnStatement) st);
+ case TYPE_ALIAS:
+ return TokenKind.PASS;
+ case VAR:
+ return TokenKind.PASS;
}
throw new IllegalArgumentException("unexpected statement: " + st.kind());
}
@@ -559,6 +564,11 @@
return evalDot(fr, (DotExpression) expr);
case CALL:
return evalCall(fr, (CallExpression) expr);
+ case CAST:
+ return eval(fr, ((CastExpression) expr).getValue());
+ case ISINSTANCE:
+ fr.setErrorLocation(expr.getStartLocation());
+ throw new EvalException("isinstance() is not yet supported");
case IDENTIFIER:
return evalIdentifier(fr, (Identifier) expr);
case INDEX:
@@ -587,6 +597,10 @@
return ((StringLiteral) expr).getValue();
case UNARY_OPERATOR:
return evalUnaryOperator(fr, (UnaryOperatorExpression) expr);
+ case ELLIPSIS:
+ case TYPE_APPLICATION:
+ // fall through, these only appear in type expressions and should be unreachable from
+ // evaluated code.
}
throw new IllegalArgumentException("unexpected expression: " + expr.kind());
}
diff --git a/src/main/java/net/starlark/java/syntax/AssignmentStatement.java b/src/main/java/net/starlark/java/syntax/AssignmentStatement.java
index 0145e67..d0d00c2 100644
--- a/src/main/java/net/starlark/java/syntax/AssignmentStatement.java
+++ b/src/main/java/net/starlark/java/syntax/AssignmentStatement.java
@@ -14,6 +14,7 @@
package net.starlark.java.syntax;
+import com.google.common.base.Preconditions;
import javax.annotation.Nullable;
/**
@@ -23,8 +24,13 @@
public final class AssignmentStatement extends Statement {
private final Expression lhs; // = IDENTIFIER | DOT | INDEX | LIST_EXPR
+
+ // non-null only when lhs is an identifier and we're not augmented
+ @Nullable private final Expression type;
+
@Nullable private final TokenKind op; // TODO(adonovan): make this mandatory even when '='.
private final int opOffset;
+
private final Expression rhs;
/**
@@ -32,14 +38,28 @@
* expression must be of the form {@code id}, {@code x.y}, {@code x[i]}, {@code [e, ...]}, or
* {@code (e, ...)}, where x, i, and e are arbitrary expressions. For an augmented assignment, the
* list and tuple forms are disallowed.
+ *
+ * <p>If a type annotation is present ({@code x : T = ...}), the LHS expression must be an
+ * identifier, and the assignment must not be augmented.
*/
AssignmentStatement(
- FileLocations locs, Expression lhs, @Nullable TokenKind op, int opOffset, Expression rhs) {
+ FileLocations locs,
+ Expression lhs,
+ @Nullable Expression type,
+ @Nullable TokenKind op,
+ int opOffset,
+ Expression rhs) {
super(locs, Kind.ASSIGNMENT);
this.lhs = lhs;
+ this.type = type;
this.op = op;
this.opOffset = opOffset;
this.rhs = rhs;
+ if (type != null) {
+ Preconditions.checkState(
+ lhs.kind() == Expression.Kind.IDENTIFIER, "Can't have type annotation on complex LHS");
+ Preconditions.checkState(op == null, "Can't have augmented assignment with type annotation");
+ }
}
/** Returns the LHS of the assignment. */
@@ -47,6 +67,12 @@
return lhs;
}
+ /** Returns the type expression (if present) of the variable on the LHS. */
+ @Nullable
+ public Expression getType() {
+ return type;
+ }
+
/** Returns the operator of an augmented assignment, or null for an ordinary assignment. */
@Nullable
public TokenKind getOperator() {
diff --git a/src/main/java/net/starlark/java/syntax/BUILD b/src/main/java/net/starlark/java/syntax/BUILD
index 9b13f78..1797511 100644
--- a/src/main/java/net/starlark/java/syntax/BUILD
+++ b/src/main/java/net/starlark/java/syntax/BUILD
@@ -21,12 +21,14 @@
"AssignmentStatement.java",
"BinaryOperatorExpression.java",
"CallExpression.java",
+ "CastExpression.java",
"Comment.java",
"Comprehension.java",
"ConditionalExpression.java",
"DefStatement.java",
"DictExpression.java",
"DotExpression.java",
+ "Ellipsis.java",
"Expression.java",
"ExpressionStatement.java",
"FileLocations.java",
@@ -38,6 +40,7 @@
"IfStatement.java",
"IndexExpression.java",
"IntLiteral.java",
+ "IsInstanceExpression.java",
"LambdaExpression.java",
"Lexer.java",
"ListExpression.java",
@@ -58,7 +61,10 @@
"StringLiteral.java",
"SyntaxError.java",
"TokenKind.java",
+ "TypeAliasStatement.java",
+ "TypeApplication.java",
"UnaryOperatorExpression.java",
+ "VarStatement.java",
],
visibility = ["//src/main/java/net/starlark/java:clients"],
# Do not add Bazel or Google dependencies here!
diff --git a/src/main/java/net/starlark/java/syntax/CastExpression.java b/src/main/java/net/starlark/java/syntax/CastExpression.java
new file mode 100644
index 0000000..3b8732c
--- /dev/null
+++ b/src/main/java/net/starlark/java/syntax/CastExpression.java
@@ -0,0 +1,66 @@
+// Copyright 2025 The Bazel Authors. All rights reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package net.starlark.java.syntax;
+
+/** Syntax node for cast() expressions. */
+public final class CastExpression extends Expression {
+ private final int startOffset;
+ private final Expression type;
+ private final Expression value;
+ private final int rparenOffset;
+
+ CastExpression(
+ FileLocations locs, int startOffset, Expression type, Expression value, int rparenOffset) {
+ super(locs, Kind.CAST);
+ this.startOffset = startOffset;
+ this.type = type;
+ this.value = value;
+ this.rparenOffset = rparenOffset;
+ }
+
+ @Override
+ public int getStartOffset() {
+ return startOffset;
+ }
+
+ @Override
+ public int getEndOffset() {
+ return rparenOffset + 1;
+ }
+
+ public Expression getType() {
+ return type;
+ }
+
+ public Expression getValue() {
+ return value;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder buf = new StringBuilder();
+ buf.append("cast(");
+ buf.append(type);
+ buf.append(", ");
+ buf.append(value);
+ buf.append(')');
+ return buf.toString();
+ }
+
+ @Override
+ public void accept(NodeVisitor visitor) {
+ visitor.visit(this);
+ }
+}
diff --git a/src/main/java/net/starlark/java/syntax/DefStatement.java b/src/main/java/net/starlark/java/syntax/DefStatement.java
index 884010d..920e8b8 100644
--- a/src/main/java/net/starlark/java/syntax/DefStatement.java
+++ b/src/main/java/net/starlark/java/syntax/DefStatement.java
@@ -22,8 +22,10 @@
private final int defOffset;
private final Identifier identifier;
+ private final ImmutableList<Identifier> typeParameters;
private final ImmutableList<Statement> body; // non-empty if well formed
private final ImmutableList<Parameter> parameters;
+ @Nullable private final Expression returnType;
// set by resolver
@Nullable private Resolver.Function resolved;
@@ -32,12 +34,16 @@
FileLocations locs,
int defOffset,
Identifier identifier,
+ ImmutableList<Identifier> typeParameters,
ImmutableList<Parameter> parameters,
+ @Nullable Expression returnType,
ImmutableList<Statement> body) {
super(locs, Kind.DEF);
this.defOffset = defOffset;
this.identifier = identifier;
+ this.typeParameters = typeParameters;
this.parameters = Preconditions.checkNotNull(parameters);
+ this.returnType = returnType;
this.body = Preconditions.checkNotNull(body);
}
@@ -58,10 +64,19 @@
return body;
}
+ public ImmutableList<Identifier> getTypeParameters() {
+ return typeParameters;
+ }
+
public ImmutableList<Parameter> getParameters() {
return parameters;
}
+ @Nullable
+ public Expression getReturnType() {
+ return returnType;
+ }
+
void setResolvedFunction(Resolver.Function resolved) {
this.resolved = resolved;
}
diff --git a/src/main/java/net/starlark/java/syntax/Ellipsis.java b/src/main/java/net/starlark/java/syntax/Ellipsis.java
new file mode 100644
index 0000000..d20206a
--- /dev/null
+++ b/src/main/java/net/starlark/java/syntax/Ellipsis.java
@@ -0,0 +1,46 @@
+// Copyright 2025 The Bazel Authors. All rights reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package net.starlark.java.syntax;
+
+/** Syntax node for the singleton ellipsis expression. */
+public final class Ellipsis extends Expression {
+
+ private final int startOffset;
+
+ Ellipsis(FileLocations locs, int startOffset) {
+ super(locs, Kind.ELLIPSIS);
+ this.startOffset = startOffset;
+ }
+
+ @Override
+ public int getStartOffset() {
+ return startOffset;
+ }
+
+ @Override
+ public int getEndOffset() {
+ return startOffset + 3;
+ }
+
+ @Override
+ public String toString() {
+ return "...";
+ }
+
+ @Override
+ public void accept(NodeVisitor visitor) {
+ visitor.visit(this);
+ }
+}
diff --git a/src/main/java/net/starlark/java/syntax/Expression.java b/src/main/java/net/starlark/java/syntax/Expression.java
index fa66a18..92f6c8f 100644
--- a/src/main/java/net/starlark/java/syntax/Expression.java
+++ b/src/main/java/net/starlark/java/syntax/Expression.java
@@ -35,15 +35,19 @@
DICT_EXPR,
DOT,
CALL,
+ CAST,
+ ELLIPSIS,
FLOAT_LITERAL,
IDENTIFIER,
INDEX,
INT_LITERAL,
+ ISINSTANCE,
LAMBDA,
LIST_EXPR,
SLICE,
STRING_LITERAL,
UNARY_OPERATOR,
+ TYPE_APPLICATION,
}
// Materialize kind as a field so its accessor can be non-virtual.
@@ -73,4 +77,20 @@
public static Expression parse(ParserInput input) throws SyntaxError.Exception {
return parse(input, FileOptions.DEFAULT);
}
+
+ /**
+ * Parses a type expression.
+ *
+ * @param options parsing options; note that {@link FileOptions#allowStarlarkTypeSyntax} doesn't
+ * need to be set - this method supports Starlark types implicitly.
+ */
+ public static Expression parseTypeExpression(ParserInput input, FileOptions options)
+ throws SyntaxError.Exception {
+ return Parser.parseTypeExpression(input, options);
+ }
+
+ /** Parses a type expression with default options. */
+ public static Expression parseTypeExpression(ParserInput input) throws SyntaxError.Exception {
+ return parseTypeExpression(input, FileOptions.DEFAULT);
+ }
}
diff --git a/src/main/java/net/starlark/java/syntax/FileOptions.java b/src/main/java/net/starlark/java/syntax/FileOptions.java
index a7111dc..b3106cb 100644
--- a/src/main/java/net/starlark/java/syntax/FileOptions.java
+++ b/src/main/java/net/starlark/java/syntax/FileOptions.java
@@ -78,6 +78,21 @@
*/
public abstract boolean stringLiteralsAreAsciiOnly();
+ /** Whether type annotations and related syntax are allowed in the source code. */
+ public abstract boolean allowTypeSyntax();
+
+ /**
+ * If true, type expressions in annotations and {@code type} declarations may be any valid
+ * expression (except for unparenthesized tuples, which are grammatically ambiguous). Otherwise
+ * type expressions must represent a valid type.
+ *
+ * <p>Enabling this boolean is helpful for backwards compatibility, but results in an AST that is
+ * not usable for type checking.
+ *
+ * <p>This has no effect if {@link #allowTypeSyntax} is false.
+ */
+ public abstract boolean tolerateInvalidTypeExpressions();
+
public static Builder builder() {
// These are the DEFAULT values.
return new AutoValue_FileOptions.Builder()
@@ -85,7 +100,9 @@
.allowToplevelRebinding(false)
.loadBindsGlobally(false)
.requireLoadStatementsFirst(true)
- .stringLiteralsAreAsciiOnly(false);
+ .stringLiteralsAreAsciiOnly(false)
+ .allowTypeSyntax(false)
+ .tolerateInvalidTypeExpressions(false);
}
public abstract Builder toBuilder();
@@ -104,6 +121,10 @@
public abstract Builder stringLiteralsAreAsciiOnly(boolean value);
+ public abstract Builder allowTypeSyntax(boolean value);
+
+ public abstract Builder tolerateInvalidTypeExpressions(boolean value);
+
public abstract FileOptions build();
}
}
diff --git a/src/main/java/net/starlark/java/syntax/IsInstanceExpression.java b/src/main/java/net/starlark/java/syntax/IsInstanceExpression.java
new file mode 100644
index 0000000..469f889
--- /dev/null
+++ b/src/main/java/net/starlark/java/syntax/IsInstanceExpression.java
@@ -0,0 +1,66 @@
+// Copyright 2025 The Bazel Authors. All rights reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package net.starlark.java.syntax;
+
+/** Syntax node for isinstance() expressions. */
+public final class IsInstanceExpression extends Expression {
+ private final int startOffset;
+ private final Expression value;
+ private final Expression type;
+ private final int rparenOffset;
+
+ IsInstanceExpression(
+ FileLocations locs, int startOffset, Expression value, Expression type, int rparenOffset) {
+ super(locs, Kind.ISINSTANCE);
+ this.startOffset = startOffset;
+ this.value = value;
+ this.type = type;
+ this.rparenOffset = rparenOffset;
+ }
+
+ @Override
+ public int getStartOffset() {
+ return startOffset;
+ }
+
+ @Override
+ public int getEndOffset() {
+ return rparenOffset + 1;
+ }
+
+ public Expression getValue() {
+ return value;
+ }
+
+ public Expression getType() {
+ return type;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder buf = new StringBuilder();
+ buf.append("isinstance(");
+ buf.append(value);
+ buf.append(", ");
+ buf.append(type);
+ buf.append(')');
+ return buf.toString();
+ }
+
+ @Override
+ public void accept(NodeVisitor visitor) {
+ visitor.visit(this);
+ }
+}
diff --git a/src/main/java/net/starlark/java/syntax/Lexer.java b/src/main/java/net/starlark/java/syntax/Lexer.java
index fcfa1d0..3d9e28e 100644
--- a/src/main/java/net/starlark/java/syntax/Lexer.java
+++ b/src/main/java/net/starlark/java/syntax/Lexer.java
@@ -487,6 +487,9 @@
private static final Map<String, TokenKind> keywordMap = new HashMap<>();
+ /** Additional keywords that are only recognized if --experimental_starlark_type_syntax is set. */
+ private static final Map<String, TokenKind> typeSyntaxExtraKeywordMap = new HashMap<>();
+
static {
keywordMap.put("and", TokenKind.AND);
keywordMap.put("as", TokenKind.AS);
@@ -519,6 +522,9 @@
keywordMap.put("while", TokenKind.WHILE);
keywordMap.put("with", TokenKind.WITH);
keywordMap.put("yield", TokenKind.YIELD);
+
+ typeSyntaxExtraKeywordMap.put("cast", TokenKind.CAST);
+ typeSyntaxExtraKeywordMap.put("isinstance", TokenKind.ISINSTANCE);
}
/**
@@ -531,10 +537,13 @@
int oldPos = pos - 1;
String id = identInterner.intern(scanIdentifier());
TokenKind kind = keywordMap.get(id);
+ if (kind == null && options.allowTypeSyntax()) {
+ kind = typeSyntaxExtraKeywordMap.get(id);
+ }
if (kind == null) {
setToken(TokenKind.IDENTIFIER, oldPos, pos);
- // setValue allocates a new String for the raw text, but it's not retained so we don't bother
- // interning it.
+ // setValue allocates a new String for the raw text, but it's not retained so we don't
+ // bother interning it.
setValue(id);
} else {
setToken(kind, oldPos, pos);
@@ -693,7 +702,12 @@
setToken(TokenKind.PLUS, pos - 1, pos);
break;
case '-':
- setToken(TokenKind.MINUS, pos - 1, pos);
+ if (peek(0) == '>') {
+ setToken(TokenKind.RARROW, pos - 1, pos + 1);
+ pos += 1;
+ } else {
+ setToken(TokenKind.MINUS, pos - 1, pos);
+ }
break;
case '|':
setToken(TokenKind.PIPE, pos - 1, pos);
@@ -777,10 +791,10 @@
}
}
- // int or float literal, or dot
+ // int or float literal, or dot, or ellipsis
if (c == '.' || isdigit(c)) {
pos--; // unconsume
- scanNumberOrDot(c);
+ scanNumberOrDotOrEllipsis(c);
break;
}
@@ -808,22 +822,28 @@
setToken(TokenKind.EOF, pos, pos);
}
- // Scans a number (INT or FLOAT) or DOT.
+ // Scans a number (INT or FLOAT) or DOT or ELLIPSIS.
// Precondition: c == peek(0) (a dot or digit)
//
// TODO(adonovan): make this the precondition for all scan functions;
- // currenly most assume their argument c has been consumed already.
- private void scanNumberOrDot(int c) {
+ // currently most assume their argument c has been consumed already.
+ private void scanNumberOrDotOrEllipsis(int c) {
int start = this.pos;
boolean fraction = false;
boolean exponent = false;
if (c == '.') {
- // dot or start of fraction
+ // dot or ellipsis or start of fraction
if (!isdigit(peek(1))) {
- pos++; // consume '.'
- setToken(TokenKind.DOT, start, pos);
- return;
+ if (peek(1) == '.' && peek(2) == '.') {
+ pos += 3; // consume '...'
+ setToken(TokenKind.ELLIPSIS, start, pos);
+ return;
+ } else {
+ pos++; // consume '.'
+ setToken(TokenKind.DOT, start, pos);
+ return;
+ }
}
fraction = true;
diff --git a/src/main/java/net/starlark/java/syntax/NodePrinter.java b/src/main/java/net/starlark/java/syntax/NodePrinter.java
index 16ca6e2..f21a596 100644
--- a/src/main/java/net/starlark/java/syntax/NodePrinter.java
+++ b/src/main/java/net/starlark/java/syntax/NodePrinter.java
@@ -97,7 +97,7 @@
} else if (arg instanceof Argument.StarStar) {
buf.append("**");
}
- printExpr(arg.getValue());
+ printExpr(arg.getValue(), true);
}
private void printParameter(Parameter param) {
@@ -129,14 +129,33 @@
void printDefSignature(DefStatement def) {
buf.append("def ");
printExpr(def.getIdentifier());
+ if (!def.getTypeParameters().isEmpty()) {
+ buf.append("[");
+ String sep = "";
+ for (Identifier typeParam : def.getTypeParameters()) {
+ buf.append(sep);
+ printExpr(typeParam);
+ sep = ", ";
+ }
+ buf.append("]");
+ }
buf.append('(');
String sep = "";
for (Parameter param : def.getParameters()) {
buf.append(sep);
printParameter(param);
+ if (param.getType() != null) {
+ buf.append(": ");
+ printExpr(param.getType(), true);
+ }
sep = ", ";
}
- buf.append("):");
+ buf.append(")");
+ if (def.getReturnType() != null) {
+ buf.append(" -> ");
+ printExpr(def.getReturnType(), true);
+ }
+ buf.append(":");
}
private void printStmt(Statement s) {
@@ -147,6 +166,11 @@
{
AssignmentStatement stmt = (AssignmentStatement) s;
printExpr(stmt.getLHS());
+ Expression type = stmt.getType();
+ if (type != null) {
+ buf.append(" : ");
+ printExpr(type);
+ }
buf.append(' ');
if (stmt.isAugmented()) {
buf.append(stmt.getOperator());
@@ -250,23 +274,61 @@
buf.append('\n');
break;
}
+
+ case TYPE_ALIAS:
+ {
+ TypeAliasStatement stmt = (TypeAliasStatement) s;
+ buf.append("type ");
+ printExpr(stmt.getIdentifier());
+ if (!stmt.getParameters().isEmpty()) {
+ buf.append('[');
+ String sep = "";
+ for (Identifier param : stmt.getParameters()) {
+ buf.append(sep);
+ printExpr(param);
+ sep = ", ";
+ }
+ buf.append(']');
+ }
+ buf.append(" = ");
+ printExpr(stmt.getDefinition(), /* canSkipParenthesis= */ true);
+ buf.append('\n');
+ break;
+ }
+
+ case VAR:
+ {
+ VarStatement stmt = (VarStatement) s;
+ printExpr(stmt.getIdentifier());
+ buf.append(" : ");
+ printExpr(stmt.getType());
+ buf.append('\n');
+ break;
+ }
}
}
private void printExpr(Expression expr) {
+ printExpr(expr, false);
+ }
+
+ private void printExpr(Expression expr, boolean canSkipParenthesis) {
switch (expr.kind()) {
case BINARY_OPERATOR:
{
BinaryOperatorExpression binop = (BinaryOperatorExpression) expr;
- // TODO(bazel-team): retain parentheses in the syntax tree so we needn't
- // conservatively emit them here.
- buf.append('(');
+ // TODO(bazel-team): print minimal number of parentheses
+ if (!canSkipParenthesis) {
+ buf.append('(');
+ }
printExpr(binop.getX());
buf.append(' ');
buf.append(binop.getOperator());
buf.append(' ');
printExpr(binop.getY());
- buf.append(')');
+ if (!canSkipParenthesis) {
+ buf.append(')');
+ }
break;
}
@@ -342,6 +404,23 @@
break;
}
+ case CAST:
+ {
+ CastExpression cast = (CastExpression) expr;
+ buf.append("cast(");
+ printExpr(cast.getType(), /* canSkipParenthesis= */ true);
+ buf.append(", ");
+ printExpr(cast.getValue(), /* canSkipParenthesis= */ true);
+ buf.append(')');
+ break;
+ }
+
+ case ELLIPSIS:
+ {
+ buf.append("...");
+ break;
+ }
+
case IDENTIFIER:
buf.append(((Identifier) expr).getName());
break;
@@ -362,6 +441,17 @@
break;
}
+ case ISINSTANCE:
+ {
+ IsInstanceExpression isinstance = (IsInstanceExpression) expr;
+ buf.append("isinstance(");
+ printExpr(isinstance.getValue(), /* canSkipParenthesis= */ true);
+ buf.append(", ");
+ printExpr(isinstance.getType(), /* canSkipParenthesis= */ true);
+ buf.append(')');
+ break;
+ }
+
case FLOAT_LITERAL:
{
buf.append(((FloatLiteral) expr).getValue());
@@ -390,7 +480,7 @@
String sep = "";
for (Expression e : list.getElements()) {
buf.append(sep);
- printExpr(e);
+ printExpr(e, true);
sep = ", ";
}
if (list.isTuple() && list.getElements().size() == 1) {
@@ -472,12 +562,30 @@
case UNARY_OPERATOR:
{
UnaryOperatorExpression unop = (UnaryOperatorExpression) expr;
- // TODO(bazel-team): retain parentheses in the syntax tree so we needn't
- // conservatively emit them here.
+ // TODO(bazel-team): print minimal number of parentheses
buf.append(unop.getOperator() == TokenKind.NOT ? "not " : unop.getOperator().toString());
- buf.append('(');
+ if (!canSkipParenthesis) {
+ buf.append('(');
+ }
printExpr(unop.getX());
- buf.append(')');
+ if (!canSkipParenthesis) {
+ buf.append(')');
+ }
+ break;
+ }
+
+ case TYPE_APPLICATION:
+ {
+ TypeApplication typeApplication = (TypeApplication) expr;
+ printExpr(typeApplication.getConstructor());
+ buf.append('[');
+ String sep = "";
+ for (Expression arg : typeApplication.getArguments()) {
+ buf.append(sep);
+ printExpr(arg, true);
+ sep = ", ";
+ }
+ buf.append(']');
break;
}
}
diff --git a/src/main/java/net/starlark/java/syntax/NodeVisitor.java b/src/main/java/net/starlark/java/syntax/NodeVisitor.java
index 096f950..d399842 100644
--- a/src/main/java/net/starlark/java/syntax/NodeVisitor.java
+++ b/src/main/java/net/starlark/java/syntax/NodeVisitor.java
@@ -32,6 +32,7 @@
// All four subclasses of Parameter are handled together.
public void visit(Parameter node) {
+ // TODO(brandjon): visit type annotation
visit(node.getIdentifier());
if (node.getDefaultValue() != null) {
visit(node.getDefaultValue());
@@ -53,6 +54,16 @@
visitAll(node.getArguments());
}
+ public void visit(CastExpression node) {
+ visit(node.getValue());
+ }
+
+ public void visit(IsInstanceExpression node) {
+ visit(node.getValue());
+ }
+
+ public void visit(Ellipsis node) {}
+
public void visit(Identifier node) {}
public void visit(Comprehension node) {
@@ -98,6 +109,7 @@
public void visit(@SuppressWarnings("unused") StringLiteral node) {}
public void visit(AssignmentStatement node) {
+ // TODO(brandjon): Visit type annotation if present
visit(node.getRHS());
visit(node.getLHS());
}
@@ -115,6 +127,7 @@
}
public void visit(DefStatement node) {
+ // TODO(brandjon): Visit return type annotation
visit(node.getIdentifier());
visitAll(node.getParameters());
visitBlock(node.getBody());
@@ -128,6 +141,14 @@
public void visit(FlowStatement node) {}
+ public void visit(TypeAliasStatement node) {
+ // TODO(brandjon): resolve type alias
+ }
+
+ public void visit(VarStatement node) {
+ // TODO(brandjon): resolve var statement
+ }
+
public void visit(DictExpression node) {
visitAll(node.getEntries());
}
diff --git a/src/main/java/net/starlark/java/syntax/Parameter.java b/src/main/java/net/starlark/java/syntax/Parameter.java
index 8b7da9b..836a469 100644
--- a/src/main/java/net/starlark/java/syntax/Parameter.java
+++ b/src/main/java/net/starlark/java/syntax/Parameter.java
@@ -13,6 +13,7 @@
// limitations under the License.
package net.starlark.java.syntax;
+import com.google.common.base.Preconditions;
import javax.annotation.Nullable;
/**
@@ -20,14 +21,19 @@
*
* <p>Parameters may be of four forms, as in {@code def f(a, b=c, *args, **kwargs)}. They are
* represented by the subclasses Mandatory, Optional, Star, and StarStar.
+ *
+ * <p>Each parameter may have a type annotation. Star parameter without id/name, `(..., *, ...)`,
+ * cannot be annotated.
*/
public abstract class Parameter extends Node {
@Nullable private final Identifier id;
+ @Nullable private final Expression type;
- private Parameter(FileLocations locs, @Nullable Identifier id) {
+ private Parameter(FileLocations locs, @Nullable Identifier id, @Nullable Expression type) {
super(locs);
this.id = id;
+ this.type = type;
}
@Nullable
@@ -45,13 +51,18 @@
return null;
}
+ @Nullable
+ public Expression getType() {
+ return type;
+ }
+
/**
* Syntax node for a mandatory parameter, {@code f(id)}. It may be positional or keyword-only
* depending on its position.
*/
public static final class Mandatory extends Parameter {
- Mandatory(FileLocations locs, Identifier id) {
- super(locs, id);
+ Mandatory(FileLocations locs, Identifier id, @Nullable Expression type) {
+ super(locs, id, type);
}
@Override
@@ -61,7 +72,7 @@
@Override
public int getEndOffset() {
- return getIdentifier().getEndOffset();
+ return getType() != null ? getType().getEndOffset() : getIdentifier().getEndOffset();
}
}
@@ -73,8 +84,9 @@
public final Expression defaultValue;
- Optional(FileLocations locs, Identifier id, @Nullable Expression defaultValue) {
- super(locs, id);
+ Optional(
+ FileLocations locs, Identifier id, @Nullable Expression type, Expression defaultValue) {
+ super(locs, id, type);
this.defaultValue = defaultValue;
}
@@ -100,12 +112,14 @@
}
}
- /** Syntax node for a star parameter, {@code f(*id)} or or {@code f(..., *, ...)}. */
+ /** Syntax node for a star parameter, {@code f(*id)} or {@code f(..., *, ...)}. */
public static final class Star extends Parameter {
private final int starOffset;
- Star(FileLocations locs, int starOffset, @Nullable Identifier id) {
- super(locs, id);
+ Star(FileLocations locs, int starOffset, @Nullable Identifier id, @Nullable Expression type) {
+ super(locs, id, type);
+ Preconditions.checkArgument(
+ id != null || type == null, "Star parameter without id cannot have a type");
this.starOffset = starOffset;
}
@@ -116,7 +130,7 @@
@Override
public int getEndOffset() {
- return getIdentifier().getEndOffset();
+ return getType() != null ? getType().getEndOffset() : getIdentifier().getEndOffset();
}
}
@@ -124,8 +138,8 @@
public static final class StarStar extends Parameter {
private final int starStarOffset;
- StarStar(FileLocations locs, int starStarOffset, Identifier id) {
- super(locs, id);
+ StarStar(FileLocations locs, int starStarOffset, Identifier id, @Nullable Expression type) {
+ super(locs, id, type);
this.starStarOffset = starStarOffset;
}
@@ -136,7 +150,7 @@
@Override
public int getEndOffset() {
- return getIdentifier().getEndOffset();
+ return getType() != null ? getType().getEndOffset() : getIdentifier().getEndOffset();
}
}
diff --git a/src/main/java/net/starlark/java/syntax/Parser.java b/src/main/java/net/starlark/java/syntax/Parser.java
index 6f65fdc..5ddb304 100644
--- a/src/main/java/net/starlark/java/syntax/Parser.java
+++ b/src/main/java/net/starlark/java/syntax/Parser.java
@@ -18,12 +18,15 @@
import com.google.common.base.Throwables;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
+import com.google.errorprone.annotations.CanIgnoreReturnValue;
import com.google.errorprone.annotations.FormatMethod;
import java.util.ArrayList;
import java.util.EnumSet;
import java.util.HashMap;
+import java.util.HashSet;
import java.util.List;
import java.util.Map;
+import java.util.Set;
import javax.annotation.Nullable;
/** Parser is a recursive-descent parser for Starlark. */
@@ -90,15 +93,25 @@
TokenKind.RPAREN,
TokenKind.SLASH);
+ /** "type" is a keyword iff it precedes an identifier (such as in a type alias expression). */
+ private static final String TYPE_SOFT_KEYWORD = "type";
+
+ private static final boolean DEBUGGING = false;
+
/** Current lookahead token. May be mutated by the parser. */
private final Lexer token; // token.kind is a prettier alias for lexer.kind
- private static final boolean DEBUGGING = false;
+ private final FileOptions options;
private final Lexer lexer;
private final FileLocations locs;
private final List<SyntaxError> errors;
+ // State tracking whether we're currently parsing a type expression.
+ // Used for conditionally allowing the Ellipsis token.
+ private boolean insideTypeExpr = false;
+
+
// TODO(adonovan): opt: compute this by subtraction.
private static final Map<TokenKind, TokenKind> augmentedAssignments =
new ImmutableMap.Builder<TokenKind, TokenKind>()
@@ -150,11 +163,12 @@
// lexer can't handle.
private final Map<String, String> stringInterner = new HashMap<>();
- private Parser(Lexer lexer, List<SyntaxError> errors) {
+ private Parser(Lexer lexer, List<SyntaxError> errors, FileOptions options) {
this.lexer = lexer;
this.locs = lexer.locs;
this.errors = errors;
this.token = lexer;
+ this.options = options;
nextToken();
}
@@ -174,7 +188,7 @@
static ParseResult parseFile(ParserInput input, FileOptions options) {
List<SyntaxError> errors = new ArrayList<>();
Lexer lexer = new Lexer(input, errors, options);
- Parser parser = new Parser(lexer, errors);
+ Parser parser = new Parser(lexer, errors, options);
StarlarkFile.ParseProfiler profiler = Parser.profiler;
long profileStartNanos = profiler != null ? profiler.start() : -1;
@@ -209,12 +223,23 @@
/** Parses an expression, possibly followed by newline tokens. */
static Expression parseExpression(ParserInput input, FileOptions options)
throws SyntaxError.Exception {
+ return parseValueOrTypeExpr(input, options, /* isTypeExpr= */ false);
+ }
+
+ /** Parses a type expression, possibly preceded or followed by comments or whitespace. */
+ static Expression parseTypeExpression(ParserInput input, FileOptions options)
+ throws SyntaxError.Exception {
+ return parseValueOrTypeExpr(input, options, /* isTypeExpr= */ true);
+ }
+
+ private static Expression parseValueOrTypeExpr(
+ ParserInput input, FileOptions options, boolean isTypeExpr) throws SyntaxError.Exception {
List<SyntaxError> errors = new ArrayList<>();
Lexer lexer = new Lexer(input, errors, options);
- Parser parser = new Parser(lexer, errors);
+ Parser parser = new Parser(lexer, errors, options);
Expression result = null;
try {
- result = parser.parseExpression();
+ result = isTypeExpr ? parser.parseTypeExprWithFallback() : parser.parseExpr();
while (parser.token.kind == TokenKind.NEWLINE) {
parser.nextToken();
}
@@ -235,14 +260,20 @@
return result;
}
- // Equivalent to 'testlist' rule in Python grammar. It can parse every kind of
- // expression. In many cases, we need to use parseTest to avoid ambiguity:
- // e.g. fct(x, y) vs fct((x, y))
+ // Parses every kind of expression, including unparenthesized tuples.
//
- // A trailing comma is disallowed in an unparenthesized tuple.
- // This prevents bugs where a one-element tuple is surprisingly created:
- // e.g. foo = f(x),
- private Expression parseExpression() {
+ // In Python the corresponding grammar production is called `expressions` (or previously, in
+ // Python 3.8 and older, `testlist`).
+ //
+ // In many cases we need to use parseTest() in place of parseExpr() to avoid ambiguity, e.g.:
+ //
+ // f(x, y) vs f((x, y))
+ //
+ // Unlike Python, a trailing comma is disallowed in an unparenthesized tuple.
+ // This prevents bugs where a one-element tuple is surprisingly created, e.g.:
+ //
+ // foo = f(x),
+ private Expression parseExpr() {
Expression e = parseTest();
if (token.kind != TokenKind.COMMA) {
return e;
@@ -266,12 +297,16 @@
}
private void syntaxError(String message) {
+ syntaxError(token.start, token.kind, token.value, message);
+ }
+
+ private void syntaxError(int offset, TokenKind tokenKind, Object tokenValue, String message) {
if (!recoveryMode) {
- if (token.kind == TokenKind.INDENT) {
- reportError(token.start, "indentation error");
+ if (tokenKind == TokenKind.INDENT) {
+ reportError(offset, "indentation error");
} else {
reportError(
- token.start, "syntax error at '%s': %s", tokenString(token.kind, token.value), message);
+ offset, "syntax error at '%s': %s", tokenString(tokenKind, tokenValue), message);
}
recoveryMode = true;
}
@@ -432,14 +467,21 @@
return new Argument.Positional(locs, expr);
}
- // arg = IDENTIFIER '=' test
- // | IDENTIFIER
- private Parameter parseParameter() {
+ // arg = IDENTIFIER [':' TypeExpr] [ '=' test ]
+ // | * [IDENTIFIER [':' TypeExpr]]
+ // | ** IDENTIFIER [':' TypeExpr]
+ // Type annotations are only available on def statements (not lambdas)
+ private Parameter parseParameter(boolean defStatement) {
+ Expression type = null;
+
// **kwargs
if (token.kind == TokenKind.STAR_STAR) {
int starStarOffset = nextToken();
Identifier id = parseIdent();
- return new Parameter.StarStar(locs, starStarOffset, id);
+ if (defStatement) {
+ type = maybeParseTypeAnnotationAfter(TokenKind.COLON);
+ }
+ return new Parameter.StarStar(locs, starStarOffset, id, type);
}
// * or *args
@@ -447,21 +489,30 @@
int starOffset = nextToken();
if (token.kind == TokenKind.IDENTIFIER) {
Identifier id = parseIdent();
- return new Parameter.Star(locs, starOffset, id);
+ if (defStatement) {
+ type = maybeParseTypeAnnotationAfter(TokenKind.COLON);
+ }
+ return new Parameter.Star(locs, starOffset, id, type);
}
- return new Parameter.Star(locs, starOffset, null);
- }
-
- // name=default
- Identifier id = parseIdent();
- if (token.kind == TokenKind.EQUALS) {
- nextToken(); // TODO: save token pos?
- Expression expr = parseTest();
- return new Parameter.Optional(locs, id, expr);
+ return new Parameter.Star(locs, starOffset, null, null);
}
// name
- return new Parameter.Mandatory(locs, id);
+ Identifier id = parseIdent();
+
+ // name: type
+ if (defStatement) {
+ type = maybeParseTypeAnnotationAfter(TokenKind.COLON);
+ }
+
+ // name=default
+ if (token.kind == TokenKind.EQUALS) {
+ nextToken(); // TODO: save token pos?
+ Expression expr = parseTest();
+ return new Parameter.Optional(locs, id, type, expr);
+ }
+
+ return new Parameter.Mandatory(locs, id, type);
}
// call_suffix = '(' arg_list? ')'
@@ -475,6 +526,36 @@
return new CallExpression(locs, fn, locs.getLocation(lparenOffset), args, rparenOffset);
}
+ // cast_expression = 'cast' '(' TypeExpr ',' expr [','] ')'
+ private Expression parseCastExpression() {
+ checkAllowTypeSyntax(token.start, token.kind, token.value);
+ int startOffset = expect(TokenKind.CAST);
+ expect(TokenKind.LPAREN);
+ Expression typeExpr = parseTypeExprWithFallback();
+ expect(TokenKind.COMMA);
+ Expression valueExpr = parseTest();
+ if (token.kind == TokenKind.COMMA) {
+ expect(TokenKind.COMMA);
+ }
+ int rparenOffset = expect(TokenKind.RPAREN);
+ return new CastExpression(locs, startOffset, typeExpr, valueExpr, rparenOffset);
+ }
+
+ // isinstance_expression = 'isinstance' '(' expr ',' TypeExpr [','] ')'
+ private Expression parseIsInstanceExpression() {
+ checkAllowTypeSyntax(token.start, token.kind, token.value);
+ int startOffset = expect(TokenKind.ISINSTANCE);
+ expect(TokenKind.LPAREN);
+ Expression valueExpr = parseTest();
+ expect(TokenKind.COMMA);
+ Expression typeExpr = parseTypeExprWithFallback();
+ if (token.kind == TokenKind.COMMA) {
+ expect(TokenKind.COMMA);
+ }
+ int rparenOffset = expect(TokenKind.RPAREN);
+ return new IsInstanceExpression(locs, startOffset, valueExpr, typeExpr, rparenOffset);
+ }
+
// Parse a list of call arguments.
//
// arg_list = ( (arg ',')* arg ','? )?
@@ -575,6 +656,8 @@
// | '(' expr ')' // a parenthesized expression
// | dict_expression
// | '-' primary_with_suffix
+ // | cast_expression
+ // | ellipsis // if in type expression
private Expression parsePrimary() {
switch (token.kind) {
case INT:
@@ -655,6 +738,20 @@
return new UnaryOperatorExpression(locs, op, offset, x);
}
+ case CAST:
+ return parseCastExpression();
+
+ case ISINSTANCE:
+ return parseIsInstanceExpression();
+
+ case ELLIPSIS:
+ if (!insideTypeExpr) {
+ syntaxError("ellipsis ('...') is not allowed outside type expressions");
+ // Fall-through, may as well emit this instead of makeErrorExpression().
+ }
+ int offset = nextToken();
+ return new Ellipsis(locs, offset);
+
default:
{
int start = token.start;
@@ -691,7 +788,7 @@
Expression step = null;
if (token.kind != TokenKind.COLON) {
- start = parseExpression();
+ start = parseExpr();
// index x[i]
if (token.kind == TokenKind.RBRACKET) {
@@ -718,7 +815,7 @@
// Equivalent to 'exprlist' rule in Python grammar.
// loop_variables = primary_with_suffix ( ',' primary_with_suffix )* ','?
private Expression parseForLoopVariables() {
- // We cannot reuse parseExpression because it would parse the 'in' operator.
+ // We cannot reuse parseExpr because it would parse the 'in' operator.
// e.g. "for i in e: pass" -> we want to parse only "i" here.
Expression e1 = parsePrimaryWithSuffix();
if (token.kind != TokenKind.COMMA) {
@@ -929,11 +1026,244 @@
return new BinaryOperatorExpression(locs, x, op, opOffset, y);
}
- // Parses a non-tuple expression ("test" in Python terminology).
+ /**
+ * Returns true if type syntax is allowed. Otherwise, reports a syntax error for the given offset
+ * and token kind and value, and returns false.
+ */
+ @CanIgnoreReturnValue
+ private boolean checkAllowTypeSyntax(int offset, TokenKind tokenKind, Object tokenValue) {
+ if (options.allowTypeSyntax()) {
+ return true;
+ } else {
+ syntaxError(offset, tokenKind, tokenValue, "type annotations are disallowed");
+ return false;
+ }
+ }
+
+ @Nullable
+ private Expression maybeParseTypeAnnotationAfter(TokenKind expectedToken) {
+ if (token.kind == expectedToken && checkAllowTypeSyntax(token.start, token.kind, token.value)) {
+ nextToken();
+ return parseTypeExprWithFallback();
+ }
+ return null;
+ }
+
+ // Hook for parsing either a structured type expression, or an unstructured arbitrary expression
+ // (except for unparenthesized tuples). The latter is useless for type checking but allows the
+ // parser to never fail on parsing a type annotation it doesn't recognize (e.g. supported by a
+ // future version of Bazel), so long as it's valid expression syntax.
+ private Expression parseTypeExprWithFallback() {
+ Expression result;
+ this.insideTypeExpr = true;
+ if (options.tolerateInvalidTypeExpressions()) {
+ // parseTest, because allowing unparenthesized tuples here would consume subsequent params in
+ // function signatures.
+ result = parseTest();
+ } else {
+ result = parseTypeExpr();
+ }
+ this.insideTypeExpr = false;
+ return result;
+ }
+
+ // TypeExpr = TypeAtom {'|' TypeAtom}.
+ // TypeAtom = identifier [TypeArguments].
+ private Expression parseTypeExpr() {
+ if (token.kind != TokenKind.IDENTIFIER) {
+ int start = token.start;
+ syntaxError("expected a type");
+ int end = syncTo(EXPR_TERMINATOR_SET);
+ return makeErrorExpression(start, end);
+ }
+ Identifier typeOrConstructor = parseIdent();
+ Expression expr;
+ if (token.kind == TokenKind.LBRACKET) {
+ expr = parseTypeApplication(typeOrConstructor);
+ } else {
+ expr = typeOrConstructor;
+ }
+ while (token.kind == TokenKind.PIPE) {
+ int opOffset = nextToken();
+ Identifier secondTypeOrConstructor = parseIdent();
+ Expression y;
+ if (token.kind == TokenKind.LBRACKET) {
+ y = parseTypeApplication(secondTypeOrConstructor);
+ } else {
+ y = secondTypeOrConstructor;
+ }
+ expr = new BinaryOperatorExpression(locs, expr, TokenKind.PIPE, opOffset, y);
+ }
+ return expr;
+ }
+
+ // TypeArgument = TypeExpr | ListOfTypes | DictOfTypes | '(' ')' | string | ellipsis
+ private Expression parseTypeArgument() {
+ switch (token.kind) {
+ case LBRACKET: // [...]
+ return parseTypeList();
+ case LBRACE: // {...}
+ return parseTypeDict();
+ case LPAREN: // the empty tuple ()
+ {
+ int lparenOffset = expect(TokenKind.LPAREN);
+ int rparenOffset = expect(TokenKind.RPAREN);
+ return new ListExpression(
+ locs, /* isTuple= */ true, lparenOffset, ImmutableList.of(), rparenOffset);
+ }
+ case STRING:
+ return parseStringLiteral();
+ case ELLIPSIS:
+ return parsePrimary();
+ default:
+ }
+ if (token.kind != TokenKind.IDENTIFIER) {
+ int start = token.start;
+ syntaxError("expected a type argument");
+ int end = syncTo(EXPR_TERMINATOR_SET);
+ return makeErrorExpression(start, end);
+ }
+ return parseTypeExpr();
+ }
+
+ // ListOfTypes = '[' [TypeArgument {',' TypeArgument} [',']] ']'.
+ private Expression parseTypeList() {
+ int lbracketOffset = expect(TokenKind.LBRACKET);
+ ImmutableList.Builder<Expression> elems = ImmutableList.builder();
+ if (token.kind != TokenKind.RBRACKET) {
+ elems.add(parseTypeArgument());
+ }
+ while (token.kind != TokenKind.RBRACKET && token.kind != TokenKind.EOF) {
+ expect(TokenKind.COMMA);
+ if (token.kind == TokenKind.RBRACKET) {
+ break;
+ }
+ elems.add(parseTypeArgument());
+ }
+ int rbracketOffset = nextToken();
+ return new ListExpression(
+ locs, /* isTuple= */ false, lbracketOffset, elems.build(), rbracketOffset);
+ }
+
+ // TypeEntry = string ':' TypeArgument .
+ private DictExpression.Entry parseTypeDictEntry() {
+ Expression key;
+ if (token.kind == TokenKind.STRING) {
+ key = parseStringLiteral();
+ } else {
+ int start = token.start;
+ syntaxError(String.format("expected %s", TokenKind.STRING));
+ int end = syncTo(EXPR_TERMINATOR_SET);
+ key = makeErrorExpression(start, end);
+ }
+ int colonOffset = expect(TokenKind.COLON);
+ Expression value = parseTypeArgument();
+ return new DictExpression.Entry(locs, key, colonOffset, value);
+ }
+
+ // DictOfTypes = '{' [TypeEntry {',' TypeEntry} [',']] '}' .
+ private Expression parseTypeDict() {
+ int lbraceOffset = expect(TokenKind.LBRACE);
+
+ ImmutableList.Builder<DictExpression.Entry> entries = ImmutableList.builder();
+ if (token.kind != TokenKind.RBRACE) {
+ entries.add(parseTypeDictEntry());
+ }
+ while (token.kind != TokenKind.RBRACE && token.kind != TokenKind.EOF) {
+ expect(TokenKind.COMMA);
+ if (token.kind == TokenKind.RBRACE) {
+ break;
+ }
+ entries.add(parseTypeDictEntry());
+ }
+
+ int rbraceOffset = nextToken();
+ return new DictExpression(locs, lbraceOffset, entries.build(), rbraceOffset);
+ }
+
+ // TypeArguments = '[' TypeArgument {',' TypeArgument} ']'.
+ private Expression parseTypeApplication(Identifier constructor) {
+ expect(TokenKind.LBRACKET);
+ ImmutableList.Builder<Expression> args = ImmutableList.builder();
+ args.add(parseTypeArgument());
+ while (token.kind != TokenKind.RBRACKET && token.kind != TokenKind.EOF) {
+ expect(TokenKind.COMMA);
+ args.add(parseTypeArgument());
+ }
+ int rbracketOffset = expect(TokenKind.RBRACKET);
+ return new TypeApplication(locs, constructor, args.build(), rbracketOffset);
+ }
+
+ private static boolean isTypeSoftKeyword(Node node) {
+ return node instanceof Identifier id && id.getName().equals(TYPE_SOFT_KEYWORD);
+ }
+
+ // type_alias_stmt = 'type' type_alias_stmt_tail
+ // type_alias_stmt_tail = identifier optional_type_params '=' TypeExpr
+ //
+ // This method assumes that 'type' has already been consumed to produce typeSoftKeywordNode.
+ private Statement parseTypeAliasStatementTail(Node typeSoftKeywordNode) {
+ Preconditions.checkArgument(isTypeSoftKeyword(typeSoftKeywordNode));
+ int startOffset = typeSoftKeywordNode.getStartOffset();
+ // For user-friendliness, mark the error as if it was detected at 'type'
+ checkAllowTypeSyntax(startOffset, TokenKind.IDENTIFIER, TYPE_SOFT_KEYWORD);
+ Identifier identifier = parseIdent();
+ ImmutableList<Identifier> parameters = parseOptionalTypeParameters();
+ expect(TokenKind.EQUALS);
+ Expression definition = parseTypeExprWithFallback();
+ return new TypeAliasStatement(locs, startOffset, identifier, parameters, definition);
+ }
+
+ // optional_type_params = ['[' identifier {',' identifier} [','] ']']
+ //
+ // For syntactic compatibility with Python, the list of identifiers in optional_type_params cannot
+ // contain duplicates; duplicate identifiers are treated as a syntax error.
+ //
+ // If the optional_type_params is absent (in other words, if the initial token is not '['), this
+ // method returns an empty list. (Note that if optional_type_params is present, it must contain at
+ // least one identifier.)
+ private ImmutableList<Identifier> parseOptionalTypeParameters() {
+ if (token.kind == TokenKind.LBRACKET) {
+ checkAllowTypeSyntax(token.start, token.kind, token.value);
+ nextToken();
+ ImmutableList.Builder<Identifier> parameters = ImmutableList.builder();
+ Set<String> uniqueParameterNames = new HashSet<>();
+ parameters.add(parseTypeParameter(uniqueParameterNames));
+ while (token.kind != TokenKind.RBRACKET && token.kind != TokenKind.EOF) {
+ expect(TokenKind.COMMA);
+ if (token.kind == TokenKind.RBRACKET) {
+ break;
+ }
+ parameters.add(parseTypeParameter(uniqueParameterNames));
+ }
+ expect(TokenKind.RBRACKET);
+ return parameters.build();
+ } else {
+ return ImmutableList.of();
+ }
+ }
+
+ private Identifier parseTypeParameter(Set<String> uniqueParameterNames) {
+ int tokenStart = token.start;
+ TokenKind tokenKind = token.kind;
+ Object tokenValue = token.value;
+ Identifier ident = parseIdent();
+ // If parseIdent() encountered a syntax error, Identifier.isValid(param.getName()) would be
+ // false, and in that case, there's no need to check for the param's uniqueness.
+ if (Identifier.isValid(ident.getName()) && !uniqueParameterNames.add(ident.getName())) {
+ syntaxError(tokenStart, tokenKind, tokenValue, "duplicate type parameter");
+ }
+ return ident;
+ }
+
+ // Parses any expression except for an unparenthesized tuple.
+ //
+ // In Python the corresponding grammar production is called `expression` (or previously, in
+ // Python 3.8 and older, `test`).
private Expression parseTest() {
int start = token.start;
if (token.kind == TokenKind.LAMBDA) {
- return parseLambda(/*allowCond=*/ true);
+ return parseLambda(/* allowCond= */ true);
}
Expression expr = parseTest(0);
@@ -966,7 +1296,7 @@
// The allowCond flag allows the body to be an 'a if b else c' conditional.
private LambdaExpression parseLambda(boolean allowCond) {
int lambdaOffset = expect(TokenKind.LAMBDA);
- ImmutableList<Parameter> params = parseParameters();
+ ImmutableList<Parameter> params = parseParameters(/* defStatement= */ false);
expect(TokenKind.COLON);
Expression body = allowCond ? parseTest() : parseTestNoCond();
return new LambdaExpression(locs, lambdaOffset, params, body);
@@ -1114,14 +1444,18 @@
}
// small_stmt = assign_stmt
+ // | type_alias_stmt
// | expr
// | load_stmt
// | return_stmt
+ // | var_stmt
// | BREAK | CONTINUE | PASS
//
- // assign_stmt = expr ('=' | augassign) expr
+ // assign_stmt = expr (':' expr)? ('=' | augassign) expr
//
// augassign = '+=' | '-=' | '*=' | '/=' | '%=' | '//=' | '&=' | '|=' | '^=' |'<<=' | '>>='
+ //
+ // var_stmt = IDENTIFIER ':' expr DOC_COMMENT_TRAILING?
private Statement parseSmallStatement() {
// return
if (token.kind == TokenKind.RETURN) {
@@ -1142,16 +1476,61 @@
return parseLoadStatement();
}
- Expression lhs = parseExpression();
+ // All other cases require an expression. Parse it now.
+ Expression lhs = parseExpr();
- // lhs = rhs or lhs += rhs
+ // Type alias. This is the only context in which an identifier can be immediately followed by
+ // another identifier; the first identifier is the soft keyword `type`.
+ if (token.kind == TokenKind.IDENTIFIER && isTypeSoftKeyword(lhs)) {
+ return parseTypeAliasStatementTail(lhs);
+ }
+
+ // Possible type expression for var statement or assignment.
+ int colonOffset = token.start; // valid only if type != null below.
+ @Nullable Expression type = maybeParseTypeAnnotationAfter(TokenKind.COLON);
+
+ // If it's an assignment, the equals or augmented-equals operator will be next.
+ // op == null for ordinary assignment. TODO(adonovan): represent as EQUALS.
TokenKind op = augmentedAssignments.get(token.kind);
if (token.kind == TokenKind.EQUALS || op != null) {
+ // Assignment.
int opOffset = nextToken();
- Expression rhs = parseExpression();
+ Expression rhs = parseExpr();
+ // Validate usage of type annotation if present.
+ if (type != null) {
+ if (!(lhs instanceof Identifier)) {
+ syntaxError(
+ colonOffset,
+ TokenKind.COLON,
+ null,
+ "type annotations must have a single identifier on the left-hand side");
+ type = null;
+ }
+ if (op != null) {
+ syntaxError(
+ colonOffset,
+ TokenKind.COLON,
+ null,
+ "type annotations not allowed on augmented assignment statements");
+ type = null;
+ }
+ }
// op == null for ordinary assignment. TODO(adonovan): represent as EQUALS.
- return new AssignmentStatement(locs, lhs, op, opOffset, rhs);
+ return new AssignmentStatement(locs, lhs, type, op, opOffset, rhs);
+ } else if (type != null) {
+ // Var statement.
+ if (!(lhs instanceof Identifier id)) {
+ syntaxError(
+ colonOffset,
+ TokenKind.COLON,
+ null,
+ "type annotations must have a single identifier on the left-hand side");
+ return new ExpressionStatement(
+ locs, makeErrorExpression(lhs.getStartOffset(), type.getEndOffset()));
+ }
+ return new VarStatement(locs, id, type);
} else {
+ // Not an assignment or var statement, so must be an expression.
return new ExpressionStatement(locs, lhs);
}
}
@@ -1187,27 +1566,29 @@
int forOffset = expect(TokenKind.FOR);
Expression vars = parseForLoopVariables();
expect(TokenKind.IN);
- Expression collection = parseExpression();
+ Expression collection = parseExpr();
expect(TokenKind.COLON);
ImmutableList<Statement> body = parseSuite();
return new ForStatement(locs, forOffset, vars, collection, body);
}
- // def_stmt = DEF IDENTIFIER '(' arguments ')' ':' suite
+ // def_stmt = DEF IDENTIFIER optional_type_parameters '(' arguments ')' ['->' TypeExpr] ':' suite
private DefStatement parseDefStatement() {
int defOffset = expect(TokenKind.DEF);
Identifier ident = parseIdent();
+ ImmutableList<Identifier> typeParams = parseOptionalTypeParameters();
expect(TokenKind.LPAREN);
- ImmutableList<Parameter> params = parseParameters();
+ ImmutableList<Parameter> params = parseParameters(/* defStatement= */ true);
expect(TokenKind.RPAREN);
+ Expression returnType = maybeParseTypeAnnotationAfter(TokenKind.RARROW);
expect(TokenKind.COLON);
ImmutableList<Statement> block = parseSuite();
- return new DefStatement(locs, defOffset, ident, params, block);
+ return new DefStatement(locs, defOffset, ident, typeParams, params, returnType, block);
}
// Parse a list of function parameters.
// Validation of parameter ordering and uniqueness is the job of the Resolver.
- private ImmutableList<Parameter> parseParameters() {
+ private ImmutableList<Parameter> parseParameters(boolean defStatement) {
boolean hasParam = false;
ImmutableList.Builder<Parameter> list = ImmutableList.builder();
@@ -1221,7 +1602,7 @@
break;
}
}
- Parameter param = parseParameter();
+ Parameter param = parseParameter(defStatement);
hasParam = true;
list.add(param);
}
@@ -1256,7 +1637,7 @@
Expression result = null;
if (!STATEMENT_TERMINATOR_SET.contains(token.kind)) {
- result = parseExpression();
+ result = parseExpr();
}
return new ReturnStatement(locs, returnOffset, result);
}
diff --git a/src/main/java/net/starlark/java/syntax/Resolver.java b/src/main/java/net/starlark/java/syntax/Resolver.java
index a9fe1c0..3495d9c 100644
--- a/src/main/java/net/starlark/java/syntax/Resolver.java
+++ b/src/main/java/net/starlark/java/syntax/Resolver.java
@@ -154,6 +154,7 @@
private final String name;
private final Location location;
private final ImmutableList<Parameter> params;
+ @Nullable private final Expression returnType;
private final ImmutableList<Statement> body;
private final boolean hasVarargs;
private final boolean hasKwargs;
@@ -169,6 +170,7 @@
String name,
Location loc,
ImmutableList<Parameter> params,
+ @Nullable Expression returnType,
ImmutableList<Statement> body,
boolean hasVarargs,
boolean hasKwargs,
@@ -179,6 +181,7 @@
this.name = name;
this.location = loc;
this.params = params;
+ this.returnType = returnType;
this.body = body;
this.hasVarargs = hasVarargs;
this.hasKwargs = hasKwargs;
@@ -283,6 +286,11 @@
return params;
}
+ @Nullable
+ public Expression getReturnType() {
+ return returnType;
+ }
+
/**
* Returns the effective statements of the function's body. (For the implicit function created
* to evaluate a single standalone expression, this may contain a synthesized Return statement.)
@@ -456,7 +464,16 @@
private void createBindings(Statement stmt) {
switch (stmt.kind()) {
case ASSIGNMENT:
- createBindingsForLHS(((AssignmentStatement) stmt).getLHS());
+ AssignmentStatement assignStmt = (AssignmentStatement) stmt;
+ if (assignStmt.getType() != null) {
+ bind((Identifier) assignStmt.getLHS(), /* isLoad= */ false, /* hasType= */ true);
+ } else {
+ createBindingsForLHS(assignStmt.getLHS());
+ }
+ break;
+ case VAR:
+ VarStatement varStmt = (VarStatement) stmt;
+ bind(varStmt.getIdentifier(), /* isLoad= */ false, /* hasType= */ true);
break;
case IF:
IfStatement ifStmt = (IfStatement) stmt;
@@ -472,7 +489,15 @@
break;
case DEF:
DefStatement def = (DefStatement) stmt;
- bind(def.getIdentifier(), /*isLoad=*/ false);
+ // Def statements are considered to supply a type annotation on the function identifier
+ // iff they have at least one piece of type syntax in their signature -- a type annotation
+ // on a parameter or return value, or a list of generic type variables.
+ // .
+ boolean hasType =
+ def.getParameters().stream().anyMatch(p -> p.getType() != null)
+ || def.getReturnType() != null
+ || !def.getTypeParameters().isEmpty();
+ bind(def.getIdentifier(), /* isLoad= */ false, /* hasType= */ hasType);
break;
case LOAD:
LoadStatement load = (LoadStatement) stmt;
@@ -488,12 +513,14 @@
// even if options.allowToplevelRebinding.
Identifier local = b.getLocalName();
if (names.add(local.getName())) {
- bind(local, /*isLoad=*/ true);
+ bind(local, /* isLoad= */ true, /* hasType= */ false);
} else {
errorf(local, "load statement defines '%s' more than once", local.getName());
}
}
break;
+ case TYPE_ALIAS:
+ // TODO(brandjon): create a type-valence binding for the alias
case EXPRESSION:
case FLOW:
case RETURN:
@@ -501,9 +528,14 @@
}
}
+ /**
+ * Calls {@link #bind} for appropriate identifiers of the LHS of an assignment.
+ *
+ * <p>This is only appropriate when no type annotation applies.
+ */
private void createBindingsForLHS(Expression lhs) {
for (Identifier id : Identifier.boundIdentifiers(lhs)) {
- bind(id, /*isLoad=*/ false);
+ bind(id, /* isLoad= */ false, /* hasType= */ false);
}
}
@@ -644,8 +676,7 @@
pushLocalBlock(node, this.locals.frame, this.locals.freevars);
for (Comprehension.Clause clause : clauses) {
- if (clause instanceof Comprehension.For) {
- Comprehension.For forClause = (Comprehension.For) clause;
+ if (clause instanceof Comprehension.For forClause) {
createBindingsForLHS(forClause.getVars());
}
}
@@ -714,6 +745,29 @@
assign(node.getLHS());
}
+ @Override
+ public void visit(VarStatement node) {
+ assign(node.getIdentifier());
+ }
+
+ @Override
+ public void visit(IsInstanceExpression node) {
+ // TODO(b/350661266): restrict the types that can be used on the RHS of isinstance(); e.g.
+ // `list` or `list | tuple` (or aliases resolving to those!) are allowed, but `list[int]` isn't,
+ // since a list can subsequently be mutated to add a non-int element.
+ errorf(node, "isinstance() is not yet supported");
+ }
+
+ @Override
+ public void visit(TypeAliasStatement node) {
+ if (!(locals.syntax instanceof StarlarkFile)) {
+ errorf(node, "type alias statement not at top level");
+ }
+
+ // TODO(brandjon): resolve type alias
+ super.visit(node);
+ }
+
// Resolves a non-binding identifier to an existing binding, or null.
@Nullable
private Binding use(Identifier id) {
@@ -909,6 +963,7 @@
name,
loc,
params.build(),
+ syntax instanceof DefStatement def ? def.getReturnType() : null,
body,
star != null && star.getIdentifier() != null,
starStar != null,
@@ -919,7 +974,13 @@
}
private void bindParam(ImmutableList.Builder<Parameter> params, Parameter param) {
- if (bind(param.getIdentifier(), /*isLoad=*/ false)) {
+ if (!bind(
+ param.getIdentifier(),
+ /* isLoad= */ false,
+ // We set hasType to false, even if there is a param annotation. This is to avoid
+ // complaining that an erroneous duplicated parameter has a type annotation, when we should
+ // really just be complaining about the fact the param was duplicated at all.
+ /* hasType= */ false)) {
errorf(param, "duplicate parameter: %s", param.getName());
}
params.add(param);
@@ -927,9 +988,14 @@
/**
* Process a binding use of a name by adding a binding to the current block if not already bound,
- * and associate the identifier with it. Reports whether the name was already bound in this block.
+ * and associate the identifier with it.
+ *
+ * @param hasType true if this binding use has a type annotation associated with it and is not a
+ * function parameter; an error is reported when hasType is true but the binding already
+ * exists in this block.
+ * @return true if the name was newly bound in this block, or false if it already existed
*/
- private boolean bind(Identifier id, boolean isLoad) {
+ private boolean bind(Identifier id, boolean isLoad, boolean hasType) {
String name = id.getName();
boolean isNew = false;
Binding bind;
@@ -990,9 +1056,24 @@
}
id.setBinding(bind);
- return !isNew;
+
+ if (hasType && !isNew) {
+ if (bind.first != null) {
+ errorf(
+ id, "type annotation on '%s' may only appear at its first declaration", id.getName());
+ errorf(bind.first, "'%s' first declared here", id.getName());
+ } else {
+ // The binding already exists and yet had no definition in this syntax tree. This shouldn't
+ // really be possible -- any binding that we should be shadowing would've be introduced by
+ // `use()` in the `visit()` traversal, which hasn't run yet.
+ errorf(id, "symbol '%s' cannot be annotated with a type", id.getName());
+ }
+ }
+
+ return isNew;
}
+
// Report conflicting top-level bindings of same scope, unless options.allowToplevelRebinding.
private void toplevelRebinding(Identifier id, Binding prev) {
if (!options.allowToplevelRebinding()) {
@@ -1090,13 +1171,14 @@
new Function(
"<toplevel>",
file.getStartLocation(),
- /*params=*/ ImmutableList.of(),
- /*body=*/ stmts,
- /*hasVarargs=*/ false,
- /*hasKwargs=*/ false,
- /*numKeywordOnlyParams=*/ 0,
+ /* params= */ ImmutableList.of(),
+ /* returnType= */ null,
+ /* body= */ stmts,
+ /* hasVarargs= */ false,
+ /* hasKwargs= */ false,
+ /* numKeywordOnlyParams= */ 0,
frame,
- /*freevars=*/ ImmutableList.of(),
+ /* freevars= */ ImmutableList.of(),
r.globals));
}
@@ -1123,13 +1205,14 @@
return new Function(
"<expr>",
expr.getStartLocation(),
- /*params=*/ ImmutableList.of(),
+ /* params= */ ImmutableList.of(),
+ /* returnType= */ null,
ImmutableList.of(ReturnStatement.make(expr)),
- /*hasVarargs=*/ false,
- /*hasKwargs=*/ false,
- /*numKeywordOnlyParams=*/ 0,
+ /* hasVarargs= */ false,
+ /* hasKwargs= */ false,
+ /* numKeywordOnlyParams= */ 0,
frame,
- /*freevars=*/ ImmutableList.of(),
+ /* freevars= */ ImmutableList.of(),
r.globals);
}
diff --git a/src/main/java/net/starlark/java/syntax/Statement.java b/src/main/java/net/starlark/java/syntax/Statement.java
index 717abe6..860ed4c 100644
--- a/src/main/java/net/starlark/java/syntax/Statement.java
+++ b/src/main/java/net/starlark/java/syntax/Statement.java
@@ -29,6 +29,8 @@
IF,
LOAD,
RETURN,
+ TYPE_ALIAS,
+ VAR,
}
// Materialize kind as a field so its accessor can be non-virtual.
diff --git a/src/main/java/net/starlark/java/syntax/TokenKind.java b/src/main/java/net/starlark/java/syntax/TokenKind.java
index 0b6ac6e..dab4cc8 100644
--- a/src/main/java/net/starlark/java/syntax/TokenKind.java
+++ b/src/main/java/net/starlark/java/syntax/TokenKind.java
@@ -24,6 +24,8 @@
BREAK("break"),
CARET("^"),
CARET_EQUALS("^="),
+ /** Emitted only if --experimental_starlark_type_syntax is enabled. */
+ CAST("cast"),
CLASS("class"),
COLON(":"),
COMMA(","),
@@ -32,6 +34,8 @@
DEL("del"),
DOT("."),
ELIF("elif"),
+ /** Valid only in type expressions. */
+ ELLIPSIS("..."),
ELSE("else"),
EOF("EOF"),
EQUALS("="),
@@ -54,6 +58,8 @@
INDENT("indent"),
INT("integer literal"),
IS("is"),
+ /** Emitted only if --experimental_starlark_type_syntax is enabled. */
+ ISINSTANCE("isinstance"),
LAMBDA("lambda"),
LBRACE("{"),
LBRACKET("["),
@@ -80,6 +86,7 @@
PLUS("+"),
PLUS_EQUALS("+="),
RAISE("raise"),
+ RARROW("->"),
RBRACE("}"),
RBRACKET("]"),
RETURN("return"),
diff --git a/src/main/java/net/starlark/java/syntax/TypeAliasStatement.java b/src/main/java/net/starlark/java/syntax/TypeAliasStatement.java
new file mode 100644
index 0000000..f17e894
--- /dev/null
+++ b/src/main/java/net/starlark/java/syntax/TypeAliasStatement.java
@@ -0,0 +1,83 @@
+// Copyright 2025 The Bazel Authors. All rights reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+package net.starlark.java.syntax;
+
+import com.google.common.collect.ImmutableList;
+
+/** Represents a type alias statement in the Starlark AST. */
+public final class TypeAliasStatement extends Statement {
+ private final int startOffset;
+ private final Identifier identifier;
+ private final ImmutableList<Identifier> parameters;
+ private final Expression definition;
+
+ TypeAliasStatement(
+ FileLocations locs,
+ int startOffset,
+ Identifier identifier,
+ ImmutableList<Identifier> parameters,
+ Expression definition) {
+ super(locs, Kind.TYPE_ALIAS);
+ this.startOffset = startOffset;
+ this.identifier = identifier;
+ this.parameters = parameters;
+ this.definition = definition;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder buf = new StringBuilder();
+ buf.append("type ");
+ buf.append(identifier.getName());
+ if (!parameters.isEmpty()) {
+ buf.append('[');
+ ListExpression.appendNodes(buf, parameters);
+ buf.append(']');
+ }
+ buf.append(" = ...\n");
+ return buf.toString();
+ }
+
+ public Identifier getIdentifier() {
+ return identifier;
+ }
+
+ public ImmutableList<Identifier> getParameters() {
+ return parameters;
+ }
+
+ public Expression getDefinition() {
+ return definition;
+ }
+
+ /**
+ * {@inheritDoc}
+ *
+ * <p>Note that this is the start offset of the statement's {@code type} keyword.
+ */
+ @Override
+ public int getStartOffset() {
+ return startOffset;
+ }
+
+ @Override
+ public int getEndOffset() {
+ return definition.getEndOffset();
+ }
+
+ @Override
+ public void accept(NodeVisitor visitor) {
+ visitor.visit(this);
+ }
+}
diff --git a/src/main/java/net/starlark/java/syntax/TypeApplication.java b/src/main/java/net/starlark/java/syntax/TypeApplication.java
new file mode 100644
index 0000000..4cbfa5f
--- /dev/null
+++ b/src/main/java/net/starlark/java/syntax/TypeApplication.java
@@ -0,0 +1,72 @@
+// Copyright 2025 The Bazel Authors. All rights reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package net.starlark.java.syntax;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+/** Syntax node for a type application expression. */
+public final class TypeApplication extends Expression {
+
+ private final Identifier constructor;
+ private final ImmutableList<Expression> arguments;
+ private final int rbracketOffset;
+
+ TypeApplication(
+ FileLocations locs,
+ Identifier constructor,
+ ImmutableList<Expression> arguments,
+ int rbracketOffset) {
+ super(locs, Kind.TYPE_APPLICATION);
+ this.constructor = Preconditions.checkNotNull(constructor);
+ this.arguments = arguments;
+ this.rbracketOffset = rbracketOffset;
+ }
+
+ /** Returns the type constructor. */
+ public Identifier getConstructor() {
+ return this.constructor;
+ }
+
+ /** Returns the type arguments. */
+ public ImmutableList<Expression> getArguments() {
+ return arguments;
+ }
+
+ @Override
+ public int getStartOffset() {
+ return constructor.getStartOffset();
+ }
+
+ @Override
+ public int getEndOffset() {
+ return rbracketOffset + 1;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder buf = new StringBuilder();
+ buf.append(constructor);
+ buf.append('[');
+ ListExpression.appendNodes(buf, arguments);
+ buf.append(']');
+ return buf.toString();
+ }
+
+ @Override
+ public void accept(NodeVisitor visitor) {
+ visitor.visit(this);
+ }
+}
diff --git a/src/main/java/net/starlark/java/syntax/VarStatement.java b/src/main/java/net/starlark/java/syntax/VarStatement.java
new file mode 100644
index 0000000..5d65449
--- /dev/null
+++ b/src/main/java/net/starlark/java/syntax/VarStatement.java
@@ -0,0 +1,67 @@
+// Copyright 2025 The Bazel Authors. All rights reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package net.starlark.java.syntax;
+
+import javax.annotation.Nullable;
+
+/**
+ * Syntax node for a variable type annotation appearing as its own statement ({@code foo : int}), as
+ * opposed to in an assignment statement where there's an initializer on the right-hand side.
+ *
+ * <p>(The name of this class is meant to be reminiscent of the `var` keyword that some languages
+ * use, although Python and Starlark have no special keyword for variable declarations.)
+ */
+public final class VarStatement extends Statement {
+
+ private final Identifier identifier;
+
+ private final Expression type;
+
+
+ /** Constructs a {@code VarStatement}. */
+ VarStatement(
+ FileLocations locs,
+ Identifier identifier,
+ Expression type) {
+ super(locs, Kind.VAR);
+ this.identifier = identifier;
+ this.type = type;
+ }
+
+ @Override
+ public int getStartOffset() {
+ return identifier.getStartOffset();
+ }
+
+ @Override
+ public int getEndOffset() {
+ return type.getEndOffset();
+ }
+
+ /** Returns the variable being declared and annotated. */
+ public Identifier getIdentifier() {
+ return identifier;
+ }
+
+ /** Returns the type expression associated with the variable. */
+ public Expression getType() {
+ return type;
+ }
+
+ @Override
+ public void accept(NodeVisitor visitor) {
+ visitor.visit(this);
+ }
+}
diff --git a/src/test/java/com/google/devtools/build/lib/packages/semantics/ConsistencyTest.java b/src/test/java/com/google/devtools/build/lib/packages/semantics/ConsistencyTest.java
index 2ab8688..7e7f8c1 100644
--- a/src/test/java/com/google/devtools/build/lib/packages/semantics/ConsistencyTest.java
+++ b/src/test/java/com/google/devtools/build/lib/packages/semantics/ConsistencyTest.java
@@ -30,7 +30,9 @@
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
-// TODO(b/173631499): We really should just delete this test entirely.
+// TODO(b/173631499): We really should just delete this test entirely. However, it does catch the
+// case of flipping a flag default but forgetting to update its string from "-foo" to "+foo", so
+// make sure we have coverage for that.
/**
* Tests for the flow of flags from {@link BuildLanguageOptions} to {@link StarlarkSemantics}, and
diff --git a/src/test/java/com/google/devtools/build/lib/starlark/BUILD b/src/test/java/com/google/devtools/build/lib/starlark/BUILD
index 7db7312..4af9504 100644
--- a/src/test/java/com/google/devtools/build/lib/starlark/BUILD
+++ b/src/test/java/com/google/devtools/build/lib/starlark/BUILD
@@ -119,6 +119,17 @@
)
java_test(
+ name = "StarlarkTypesTest",
+ srcs = ["StarlarkTypesTest.java"],
+ shard_count = 2,
+ deps = [
+ "//src/test/java/com/google/devtools/build/lib/analysis/util",
+ "//third_party:junit4",
+ "//third_party:truth",
+ ],
+)
+
+java_test(
name = "StarlarkRuleClassFunctionsTest",
srcs = ["StarlarkRuleClassFunctionsTest.java"],
shard_count = 5,
diff --git a/src/test/java/com/google/devtools/build/lib/starlark/StarlarkTypesTest.java b/src/test/java/com/google/devtools/build/lib/starlark/StarlarkTypesTest.java
new file mode 100644
index 0000000..f445c80
--- /dev/null
+++ b/src/test/java/com/google/devtools/build/lib/starlark/StarlarkTypesTest.java
@@ -0,0 +1,120 @@
+// Copyright 2025 The Bazel Authors. All rights reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+package com.google.devtools.build.lib.starlark;
+
+import static com.google.common.truth.Truth.assertThat;
+import static org.junit.Assert.assertThrows;
+
+import com.google.devtools.build.lib.analysis.util.BuildViewTestCase;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for Starlark types. */
+@RunWith(JUnit4.class)
+public class StarlarkTypesTest extends BuildViewTestCase {
+ @Test
+ public void experimentalStarlarkTypes_on_allowsTypeAnnotations() throws Exception {
+ setBuildLanguageOptions(
+ "--experimental_starlark_type_syntax",
+ "--experimental_starlark_types_allowed_paths=//test");
+ scratch.file(
+ "test/foo.bzl",
+ """
+ def f(a: int):
+ pass\
+ """);
+ scratch.file("test/BUILD", "load(':foo.bzl', 'f')");
+
+ getTarget("//test:BUILD");
+
+ assertNoEvents();
+ }
+
+ @Test
+ public void experimentalStarlarkTypes_off_disallowsTypeAnnotations() throws Exception {
+ setBuildLanguageOptions(
+ "--noexperimental_starlark_type_syntax",
+ "--experimental_starlark_types_allowed_paths=//test");
+ scratch.file(
+ "test/foo.bzl",
+ """
+ def f(a: int):
+ pass\
+ """);
+ scratch.file("test/BUILD", "load(':foo.bzl', 'f')");
+
+ checkLoadingPhaseError("//test:BUILD", "syntax error at ':': type annotations are disallowed");
+ assertContainsEvent(
+ "Type annotations syntax can be enabled with --experimental_starlark_type_syntax and/or"
+ + " --experimental_starlark_types_allowed_paths.");
+ }
+
+ @Test
+ public void experimentalStarlarkTypes_prohibitedInSclRegardlessOfFlag() throws Exception {
+ setBuildLanguageOptions("--experimental_starlark_type_syntax");
+ scratch.file(
+ "test/foo.scl",
+ """
+ def f(a: int):
+ pass\
+ """);
+ scratch.file("test/BUILD", "load(':foo.scl', 'f')");
+
+ checkLoadingPhaseError("//test:BUILD", "syntax error at ':': type annotations are disallowed");
+ assertContainsEvent("Type annotations are not permitted in .scl files.");
+ }
+
+ @Test
+ public void starlarkTypesAllowedPath_notOnPath_disallowsTypeAnnotations() throws Exception {
+ setBuildLanguageOptions(
+ "--experimental_starlark_type_syntax",
+ "--experimental_starlark_types_allowed_paths=//main");
+ scratch.file(
+ "test/foo.bzl",
+ """
+ def f(a: int):
+ pass\
+ """);
+ scratch.file("test/BUILD", "load(':foo.bzl', 'f')");
+
+ checkLoadingPhaseError("//test:BUILD", "syntax error at ':': type annotations are disallowed");
+ assertContainsEvent(
+ "Type annotations syntax can be enabled with --experimental_starlark_type_syntax and/or"
+ + " --experimental_starlark_types_allowed_paths.");
+ }
+
+ @Test
+ public void starlarkTypesAllowedPath_externalPath_allowsTypeAnnotations() throws Exception {
+ setBuildLanguageOptions(
+ "--experimental_starlark_type_syntax",
+ "--experimental_starlark_types_allowed_paths=@@r+//test");
+ scratch.overwriteFile(
+ "MODULE.bazel", "bazel_dep(name='r')", "local_path_override(module_name='r', path='/r')");
+ scratch.file("/r/MODULE.bazel", "module(name='r')");
+ scratch.file(
+ "/r/test/foo.bzl",
+ """
+ def f(a: int):
+ pass\
+ """);
+ scratch.file("/r/test/BUILD", "load(':foo.bzl', 'f')");
+
+ // Required since we have a new MODULE.bazel file.
+ invalidatePackages(true);
+ getTarget("@@r+//test:BUILD");
+
+ assertNoEvents();
+ }
+}
diff --git a/src/test/java/net/starlark/java/eval/EvaluationTest.java b/src/test/java/net/starlark/java/eval/EvaluationTest.java
index 23446bd..47c7dec 100644
--- a/src/test/java/net/starlark/java/eval/EvaluationTest.java
+++ b/src/test/java/net/starlark/java/eval/EvaluationTest.java
@@ -865,4 +865,53 @@
}
assertThat(module.getDocumentation()).isEqualTo("preset docstring");
}
+
+ @Test
+ public void typeAliasStatement_evalsAsNoop() throws Exception {
+ ev.setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ ev.new Scenario().setUp("type X = int").testLookup("X", null);
+ ev.new Scenario().setUp("Y = 'foo'; type Y = bool").testLookup("Y", "foo");
+ }
+
+ @Test
+ public void varStatement_evalsAsNoop() throws Exception {
+ ev.setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ ev.new Scenario().setUp("X : int").testLookup("X", null);
+ }
+
+ @Test
+ public void varStatement_canLeaveToplevelSymbolcUninitialized() throws Exception {
+ ev.setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ ev.new Scenario()
+ .setUp(
+ """
+ X : int
+ def f():
+ print(X)
+ """)
+ .testIfErrorContains("global variable 'X' is referenced before assignment", "f()");
+ }
+
+ @Test
+ public void castExpression_evalsAsIdentity() throws Exception {
+ // The dynamic behavior of `cast` (disregarding type checking) is to return its value unchanged.
+ ev.setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ ev.new Scenario()
+ .setUp(
+ """
+ x = cast(list, [1])
+ y = cast(int, "this is not an int")
+ z = cast(dict[str, str], 42)
+ """)
+ .testEval("x", "[1]")
+ .testEval("y", "\"this is not an int\"")
+ .testEval("z", "42");
+ }
+
+ // TODO(b/350661266): resolve types in isinstance().
+ @Test
+ public void isinstanceExpression_notYetSupported() throws Exception {
+ ev.setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ ev.new Scenario().testIfExactError("isinstance() is not yet supported", "isinstance(x, list)");
+ }
}
diff --git a/src/test/java/net/starlark/java/eval/EvaluationTestCase.java b/src/test/java/net/starlark/java/eval/EvaluationTestCase.java
index 272101b..e96d271 100644
--- a/src/test/java/net/starlark/java/eval/EvaluationTestCase.java
+++ b/src/test/java/net/starlark/java/eval/EvaluationTestCase.java
@@ -33,6 +33,7 @@
class EvaluationTestCase {
private StarlarkSemantics semantics = StarlarkSemantics.DEFAULT;
+ private FileOptions fileOptions = FileOptions.DEFAULT;
private StarlarkThread thread = null; // created lazily by getStarlarkThread
private Module module = null; // created lazily by getModule
@@ -48,6 +49,14 @@
this.module = null;
}
+ public FileOptions getFileOptions() {
+ return fileOptions;
+ }
+
+ public void setFileOptions(FileOptions fileOptions) {
+ this.fileOptions = fileOptions;
+ }
+
// TODO(adonovan): don't let subclasses inherit vaguely specified "helpers".
// Separate all the tests clearly into tests of the scanner, parser, resolver,
// and evaluation.
@@ -69,14 +78,14 @@
/** Joins the lines, parses them as an expression, and evaluates it. */
final Object eval(String... lines) throws Exception {
ParserInput input = ParserInput.fromLines(lines);
- return Starlark.eval(input, FileOptions.DEFAULT, getModule(), getStarlarkThread());
+ return Starlark.eval(input, fileOptions, getModule(), getStarlarkThread());
}
/** Joins the lines, parses them as a file, and executes it. */
final void exec(String... lines)
throws SyntaxError.Exception, EvalException, InterruptedException {
ParserInput input = ParserInput.fromLines(lines);
- Starlark.execFile(input, FileOptions.DEFAULT, getModule(), getStarlarkThread());
+ Starlark.execFile(input, fileOptions, getModule(), getStarlarkThread());
}
// A hook for subclasses to alter the created module.
diff --git a/src/test/java/net/starlark/java/syntax/BUILD b/src/test/java/net/starlark/java/syntax/BUILD
index cc7d468..c952f26 100644
--- a/src/test/java/net/starlark/java/syntax/BUILD
+++ b/src/test/java/net/starlark/java/syntax/BUILD
@@ -34,5 +34,6 @@
"//third_party:guava",
"//third_party:junit4",
"//third_party:truth",
+ "@maven//:com_google_testparameterinjector_test_parameter_injector",
],
)
diff --git a/src/test/java/net/starlark/java/syntax/LexerTest.java b/src/test/java/net/starlark/java/syntax/LexerTest.java
index 1346919..e3e1790 100644
--- a/src/test/java/net/starlark/java/syntax/LexerTest.java
+++ b/src/test/java/net/starlark/java/syntax/LexerTest.java
@@ -339,6 +339,17 @@
check("foo.123", "IDENTIFIER(foo) FLOAT(0.123) NEWLINE EOF");
check("foo.bcd", "IDENTIFIER(foo) DOT IDENTIFIER(bcd) NEWLINE EOF"); // 'b' are hex chars
check("foo.xyz", "IDENTIFIER(foo) DOT IDENTIFIER(xyz) NEWLINE EOF");
+
+ check("..", "DOT DOT NEWLINE EOF");
+ check("...", "ELLIPSIS NEWLINE EOF");
+ check("....", "ELLIPSIS DOT NEWLINE EOF"); // ellipsis is consumed greedily before dot
+ check(".......", "ELLIPSIS ELLIPSIS DOT NEWLINE EOF");
+ check(". . . ", "DOT DOT DOT NEWLINE EOF");
+
+ check("1...", "FLOAT(1.0) DOT DOT NEWLINE EOF");
+ check("1...1", "FLOAT(1.0) DOT FLOAT(0.1) NEWLINE EOF");
+ check("1....1", "FLOAT(1.0) ELLIPSIS INT(1) NEWLINE EOF");
+ check("foo...bcd", "IDENTIFIER(foo) ELLIPSIS IDENTIFIER(bcd) NEWLINE EOF");
}
@Test
diff --git a/src/test/java/net/starlark/java/syntax/NodePrinterTest.java b/src/test/java/net/starlark/java/syntax/NodePrinterTest.java
index ac3bae0..da135ab 100644
--- a/src/test/java/net/starlark/java/syntax/NodePrinterTest.java
+++ b/src/test/java/net/starlark/java/syntax/NodePrinterTest.java
@@ -24,22 +24,27 @@
/** Tests {@link Node#toString} and {@code NodePrinter}. */
@RunWith(JUnit4.class)
public final class NodePrinterTest {
+ private FileOptions fileOptions = FileOptions.DEFAULT;
- private static StarlarkFile parseFile(String... lines) throws SyntaxError.Exception {
+ private void setFileOptions(FileOptions fileOptions) {
+ this.fileOptions = fileOptions;
+ }
+
+ private StarlarkFile parseFile(String... lines) throws SyntaxError.Exception {
ParserInput input = ParserInput.fromLines(lines);
- StarlarkFile file = StarlarkFile.parse(input);
+ StarlarkFile file = StarlarkFile.parse(input, fileOptions);
if (!file.ok()) {
throw new SyntaxError.Exception(file.errors());
}
return file;
}
- private static Statement parseStatement(String... lines) throws SyntaxError.Exception {
+ private Statement parseStatement(String... lines) throws SyntaxError.Exception {
return parseFile(lines).getStatements().get(0);
}
- private static Expression parseExpression(String... lines) throws SyntaxError.Exception {
- return Expression.parse(ParserInput.fromLines(lines));
+ private Expression parseExpression(String... lines) throws SyntaxError.Exception {
+ return Expression.parse(ParserInput.fromLines(lines), fileOptions);
}
private static String join(String... lines) {
@@ -74,27 +79,27 @@
* Parses the given string as an expression, and asserts that its pretty print matches the given
* string.
*/
- private static void assertExprPrettyMatches(String source, String expected)
+ private void assertExprPrettyMatches(String source, String expected)
throws SyntaxError.Exception {
- Expression node = parseExpression(source);
- assertPrettyMatches(node, expected);
+ Expression node = parseExpression(source);
+ assertPrettyMatches(node, expected);
}
/**
* Parses the given string as an expression, and asserts that its {@code toString} matches the
* given string.
*/
- private static void assertExprTostringMatches(String source, String expected)
+ private void assertExprTostringMatches(String source, String expected)
throws SyntaxError.Exception {
- Expression node = parseExpression(source);
- assertThat(node.toString()).isEqualTo(expected);
+ Expression node = parseExpression(source);
+ assertThat(node.toString()).isEqualTo(expected);
}
/**
* Parses the given string as an expression, and asserts that both its pretty print and {@code
* toString} return the original string.
*/
- private static void assertExprBothRoundTrip(String source) throws SyntaxError.Exception {
+ private void assertExprBothRoundTrip(String source) throws SyntaxError.Exception {
assertExprPrettyMatches(source, source);
assertExprTostringMatches(source, source);
}
@@ -103,7 +108,7 @@
* Parses the given string as a statement, and asserts that its pretty print with one indent
* matches the given string.
*/
- private static void assertStmtIndentedPrettyMatches(String source, String expected)
+ private void assertStmtIndentedPrettyMatches(String source, String expected)
throws SyntaxError.Exception {
Statement node = parseStatement(source);
assertIndentedPrettyMatches(node, expected);
@@ -113,7 +118,7 @@
* Parses the given string as an statement, and asserts that its {@code toString} matches the
* given string.
*/
- private static void assertStmtTostringMatches(String source, String expected)
+ private void assertStmtTostringMatches(String source, String expected)
throws SyntaxError.Exception {
Statement node = parseStatement(source);
assertThat(node.toString()).isEqualTo(expected);
@@ -158,6 +163,7 @@
assertExprBothRoundTrip("f(a)");
assertExprBothRoundTrip("f(a, b = B, c = C, *d, **e)");
assertExprBothRoundTrip("o.f()");
+ assertExprBothRoundTrip("f(1 + 1)");
}
@Test
@@ -168,6 +174,9 @@
@Test
public void indexExpression() throws SyntaxError.Exception {
assertExprBothRoundTrip("a[i]");
+ assertExprBothRoundTrip("a[(1,)]");
+ assertExprPrettyMatches("a[1,2]", "a[(1, 2)]");
+ assertExprTostringMatches("a[1,2]", "a[(1, 2)]");
}
@Test
@@ -246,6 +255,13 @@
}
@Test
+ public void assignmentStatementWithTypeAnnotation() throws SyntaxError.Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ assertStmtIndentedPrettyMatches("x : T = y", " x : T = y\n");
+ assertStmtTostringMatches("x : T = y", "x : T = y\n");
+ }
+
+ @Test
public void expressionStatement() throws SyntaxError.Exception {
assertStmtIndentedPrettyMatches("5", " 5\n");
assertStmtTostringMatches("5", "5\n");
@@ -276,16 +292,87 @@
" print(x)"),
"def f(a, b=B, *c, d=D, **e): ...\n");
+ assertStmtIndentedPrettyMatches(join("def f():", " pass"), join(" def f():", " pass", ""));
+ assertStmtTostringMatches(join("def f():", " pass"), "def f(): ...\n");
+ }
+
+ @Test
+ public void defStatementWithTypeAnnotations() throws SyntaxError.Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
assertStmtIndentedPrettyMatches(
- join("def f():",
- " pass"),
- join(" def f():",
- " pass",
- ""));
+ join("def f(x:int):", " print(x)"), join(" def f(x: int):", " print(x)", ""));
+ assertStmtTostringMatches(join("def f(x:bool):", " print(x)"), "def f(x: bool): ...\n");
+
+ assertStmtIndentedPrettyMatches(
+ join("def f()->int:", " print(x)"), join(" def f() -> int:", " print(x)", ""));
+ assertStmtTostringMatches(join("def f() -> bool:", " print(x)"), "def f() -> bool: ...\n");
+ assertStmtIndentedPrettyMatches(
+ """
+ def f[T,U,](x:dict[T,U])->list[U]:
+ print(x)\
+ """,
+ """
+ def f[T, U](x: dict[T, U]) -> list[U]:
+ print(x)
+ """);
assertStmtTostringMatches(
- join("def f():",
- " pass"),
- "def f(): ...\n");
+ """
+ def f[T,U,](x:dict[T,U]|set[U]) -> bool:
+ print(x)
+ """,
+ "def f[T, U](x: dict[T, U] | set[U]) -> bool: ...\n");
+ }
+
+ @Test
+ public void typeAnnotations() throws SyntaxError.Exception {
+ // TODO(ilist@): replace with parsing type annotations directly (remove `def` from this test)
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ assertStmtTostringMatches("def f(x:bool): pass", "def f(x: bool): ...\n");
+ assertStmtTostringMatches("def f(x:None | bool): pass", "def f(x: None | bool): ...\n");
+ assertStmtTostringMatches("def f(x:list[str]): pass", "def f(x: list[str]): ...\n");
+ assertStmtTostringMatches("def f(x:dict[str,int]): pass", "def f(x: dict[str, int]): ...\n");
+ assertStmtTostringMatches(
+ "def f(x:Callable[[str],int]): pass", "def f(x: Callable[[str], int]): ...\n");
+ assertStmtTostringMatches(
+ "def f(x:Callable[[str|int],int]): pass", "def f(x: Callable[[str | int], int]): ...\n");
+ assertStmtTostringMatches(
+ "def f(x:TypedDict[{'field1': int}]): pass",
+ "def f(x: TypedDict[{\"field1\": int}]): ...\n");
+ }
+
+ @Test
+ public void typeAliasStatement() throws SyntaxError.Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ assertStmtTostringMatches("type my_int=int", "type my_int = ...\n");
+ assertStmtIndentedPrettyMatches("type my_int=int", " type my_int = int\n");
+ assertStmtTostringMatches("type X[T,U]=dict[T,U]|list[U]", "type X[T, U] = ...\n");
+ assertStmtIndentedPrettyMatches(
+ "type X[T,U]=dict[T,U]|list[U]", " type X[T, U] = dict[T, U] | list[U]\n");
+ }
+
+ @Test
+ public void ellipsisExpression() throws SyntaxError.Exception {
+ setFileOptions(
+ FileOptions.builder().allowTypeSyntax(true).tolerateInvalidTypeExpressions(true).build());
+ // Use `def` rather than `type` to wrap the type expression, because `type`'s toString()
+ // introduces its own metasyntactic "..." placeholder.
+ assertStmtTostringMatches(
+ "def f(x:Callable[...,int]): pass", "def f(x: Callable[(..., int)]): ...\n");
+ assertStmtIndentedPrettyMatches("type x=...", " type x = ...\n");
+ }
+
+ @Test
+ public void castExpression() throws SyntaxError.Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ assertExprPrettyMatches("cast(list[int]|str,x+y)", "cast(list[int] | str, x + y)");
+ assertExprTostringMatches("cast(set|None,bar(),)", "cast(set | None, bar())");
+ }
+
+ @Test
+ public void isinstanceExpression() throws SyntaxError.Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ assertExprPrettyMatches("isinstance(x+y, list|tuple)", "isinstance(x + y, list | tuple)");
+ assertExprTostringMatches("isinstance(foo(), T[U],)", "isinstance(foo(), T[U])");
}
@Test
@@ -366,6 +453,13 @@
}
@Test
+ public void varStatement() throws SyntaxError.Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ assertStmtIndentedPrettyMatches("x : T", " x : T\n");
+ assertStmtTostringMatches("x : T\n", "x : T\n");
+ }
+
+ @Test
public void returnStatement() throws SyntaxError.Exception {
assertStmtIndentedPrettyMatches("return \"foo\"", " return \"foo\"\n");
assertStmtTostringMatches("return \"foo\"", "return \"foo\"\n");
diff --git a/src/test/java/net/starlark/java/syntax/ParserTest.java b/src/test/java/net/starlark/java/syntax/ParserTest.java
index 5abf798..7d849d5 100644
--- a/src/test/java/net/starlark/java/syntax/ParserTest.java
+++ b/src/test/java/net/starlark/java/syntax/ParserTest.java
@@ -21,19 +21,21 @@
import com.google.common.collect.ImmutableList;
import com.google.common.collect.Iterables;
import com.google.common.collect.Sets;
+import com.google.testing.junit.testparameterinjector.TestParameter;
+import com.google.testing.junit.testparameterinjector.TestParameterInjector;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import org.junit.Test;
import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
/** Tests of parser. */
-@RunWith(JUnit4.class)
+@RunWith(TestParameterInjector.class)
public final class ParserTest {
private final List<SyntaxError> events = new ArrayList<>();
private boolean failFast = true;
+ private FileOptions fileOptions = FileOptions.DEFAULT;
private SyntaxError assertContainsError(String expectedMessage) {
return LexerTest.assertContainsError(events, expectedMessage);
@@ -43,21 +45,52 @@
this.failFast = failFast;
}
+ private void setFileOptions(FileOptions fileOptions) {
+ this.fileOptions = fileOptions;
+ }
+
// Joins the lines, parse, and returns an expression.
- private static Expression parseExpression(String... lines) throws SyntaxError.Exception {
+ private Expression parseExpression(String... lines) throws SyntaxError.Exception {
ParserInput input = ParserInput.fromLines(lines);
- return Expression.parse(input);
+ return Expression.parse(input, fileOptions);
}
// Parses the expression, asserts that parsing fails,
// and returns the first error message.
- private static String parseExpressionError(String src) {
+ private String parseExpressionError(String src) {
ParserInput input = ParserInput.fromLines(src);
try {
- Expression.parse(input);
- throw new AssertionError("parseExpression(%s) succeeded unexpectedly: " + src);
+ Expression.parse(input, fileOptions);
+ throw new AssertionError("parseExpressionError() succeeded unexpectedly: " + src);
} catch (SyntaxError.Exception ex) {
- return ex.errors().get(0).message();
+ return ex.errors().get(0).toString();
+ }
+ }
+
+ // Parses the statement, asserts that parsing fails, and returns the first error message.
+ private String parseStatementError(String src) throws SyntaxError.Exception {
+ ParserInput input = ParserInput.fromLines(src);
+ StarlarkFile file = StarlarkFile.parse(input, fileOptions);
+ if (file.ok()) {
+ throw new AssertionError("parseStatementError() succeeded unexpectedly: " + src);
+ }
+ return file.errors().get(0).toString();
+ }
+
+ // Joins the lines, parse, and returns a type expression.
+ private Expression parseTypeExpression(String... lines) throws SyntaxError.Exception {
+ ParserInput input = ParserInput.fromLines(lines);
+ return Expression.parseTypeExpression(input, fileOptions);
+ }
+
+ // Parses the type expression, asserts that parsing fails, and returns the first error message.
+ private String parseTypeExpressionError(String src) {
+ ParserInput input = ParserInput.fromLines(src);
+ try {
+ Expression.parseTypeExpression(input, fileOptions);
+ throw new AssertionError("parseTypeExpressionError() succeeded unexpectedly: " + src);
+ } catch (SyntaxError.Exception ex) {
+ return ex.errors().get(0).toString();
}
}
@@ -65,7 +98,7 @@
// Errors are added to this.events, or thrown if this.failFast;
private StarlarkFile parseFile(String... lines) throws SyntaxError.Exception {
ParserInput input = ParserInput.fromLines(lines);
- StarlarkFile file = StarlarkFile.parse(input);
+ StarlarkFile file = StarlarkFile.parse(input, fileOptions);
if (!file.ok()) {
if (failFast) {
throw new SyntaxError.Exception(file.errors());
@@ -339,8 +372,7 @@
assertLocation(0, 14, slice);
}
- private static void evalSlice(String statement, Object... expectedArgs)
- throws SyntaxError.Exception {
+ private void evalSlice(String statement, Object... expectedArgs) throws SyntaxError.Exception {
SliceExpression e = (SliceExpression) parseExpression(statement);
// There is no way to evaluate the expression here, so we rely on string comparison.
@@ -474,6 +506,89 @@
}
@Test
+ public void testVarAnnotation_basic() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+
+ Statement stmt = parseStatement("x : T");
+ assertThat(stmt).isInstanceOf(VarStatement.class);
+ assertThat(((VarStatement) stmt).getType().toString()).isEqualTo("T");
+
+ stmt = parseStatement("x : T = 123");
+ assertThat(stmt).isInstanceOf(AssignmentStatement.class);
+ assertThat(((AssignmentStatement) stmt).getType().toString()).isEqualTo("T");
+ }
+
+ @Test
+ public void testVarAnnotation_requiresTypeSyntax() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(false).build());
+ assertThat(parseStatementError("x : T"))
+ .contains("syntax error at ':': type annotations are disallowed");
+ assertThat(parseStatementError("x : T = 123"))
+ .contains("syntax error at ':': type annotations are disallowed");
+ }
+
+ @Test
+ public void testVarAnnotation_takesOneIdentifier() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+
+ // Complaint located at colon at column 6.
+ String errMessage =
+ ":1:6: syntax error at ':': type annotations must have a single identifier on the"
+ + " left-hand side";
+
+ assertThat(parseStatementError("x, y : T")).contains(errMessage);
+ assertThat(parseStatementError("x, y : T = 123")).contains(errMessage);
+
+ // This is *not* parsed as `x : (T, y)`, even though it might look unambiguous. Doing so would
+ // require knowing what the comma is before we know whether we're in a VarStatement or
+ // assignment statement. It's also not allowed by Python.
+ assertThat(parseStatementError("x : T, y"))
+ .contains(":1:6: syntax error at ',': expected newline");
+ assertThat(parseStatementError("x : T, y = 123"))
+ .contains(":1:6: syntax error at ',': expected newline");
+
+ assertThat(parseStatementError("x[0] : T")).contains(errMessage);
+ assertThat(parseStatementError("x[0] : T = 123")).contains(errMessage);
+
+ // Only applicable to assignment, not VarStatement.
+ assertThat(parseStatementError("(x : T, y) = 123"))
+ // TODO: #27370 - Is there a reasonable way to produce a more informative error message
+ // here, e.g. "type annotations are only allowed in assignment statements or variable
+ // declarations"?
+ .contains(":1:4: syntax error at ':': expected )");
+ }
+
+ @Test
+ public void testVarAnnotation_notAllowedOnAugmentedAssignment() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ assertThat(parseStatementError("x : T += 123"))
+ .contains(
+ ":1:3: syntax error at ':': type annotations not allowed on augmented assignment"
+ + " statements");
+ }
+
+ @Test
+ public void testVarAnnotation_illegalTypeExpression_allowedWithFlag() throws Exception {
+ setFileOptions(
+ FileOptions.builder().allowTypeSyntax(true).tolerateInvalidTypeExpressions(true).build());
+ assertThat(((VarStatement) parseStatement("x : (lambda x: x)")).getType())
+ .isInstanceOf(LambdaExpression.class);
+ assertThat(((AssignmentStatement) parseStatement("x : (lambda x: x) = 123")).getType())
+ .isInstanceOf(LambdaExpression.class);
+ }
+
+ @Test
+ public void testAssignWithAnnotation_illegalTypeExpression_disallowedWithoutFlag()
+ throws Exception {
+ setFileOptions(
+ FileOptions.builder().allowTypeSyntax(true).tolerateInvalidTypeExpressions(false).build());
+ assertThat(parseStatementError("x : (lambda x: x)"))
+ .contains(":1:5: syntax error at '(': expected a type");
+ assertThat(parseStatementError("x : (lambda x: x) = 123"))
+ .contains(":1:5: syntax error at '(': expected a type");
+ }
+
+ @Test
public void testPrettyPrintFunctions() throws Exception {
assertThat(parseStatements("x[1:3]").toString()).isEqualTo("[x[1:3]\n]");
assertThat(parseStatements("x[1:3:1]").toString()).isEqualTo("[x[1:3:1]\n]");
@@ -624,7 +739,7 @@
assertThat(getText(stmtStr, stmt)).isEqualTo(stmtStr);
}
- private static void assertExpressionLocationCorrect(String exprStr) throws SyntaxError.Exception {
+ private void assertExpressionLocationCorrect(String exprStr) throws SyntaxError.Exception {
Expression expr = parseExpression(exprStr);
assertThat(getText(exprStr, expr)).isEqualTo(exprStr);
// Also try it with another token at the end (newline), which broke the location in the past.
@@ -1014,6 +1129,435 @@
}
@Test
+ public void testTypeExpression() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ // basic examples
+ assertThat(parseTypeExpression("int")).isInstanceOf(Identifier.class);
+ assertThat(parseTypeExpression("list[str]")).isInstanceOf(TypeApplication.class);
+ assertThat(parseTypeExpression("dict[str, int]")).isInstanceOf(TypeApplication.class);
+ // type applications must have at least one argument
+ assertThat(assertThrows(SyntaxError.Exception.class, () -> parseTypeExpression("tuple[]")))
+ .hasMessageThat()
+ .contains("syntax error at ']': expected a type argument");
+ // type expressions can use list literals
+ assertThat(parseTypeExpression("Callable[[int, str], int]"))
+ .isInstanceOf(TypeApplication.class);
+ // type expressions can use dict literals with string keys and type expression values
+ assertThat(parseTypeExpression("TypedDict[{'a': int, 'b': bool}]"))
+ .isInstanceOf(TypeApplication.class);
+ // (non-string keys, or non-type-expression values, are a parse-time error)
+ assertThat(
+ assertThrows(
+ SyntaxError.Exception.class, () -> parseTypeExpression("TypedDict[{x: y}]")))
+ .hasMessageThat()
+ .contains("syntax error at 'x': expected string literal");
+ assertThat(
+ assertThrows(
+ SyntaxError.Exception.class, () -> parseTypeExpression("TypedDict[{'x': foo()}]")))
+ .hasMessageThat()
+ .contains("syntax error at '(': expected ,");
+ // type expressions can use empty tuple literals
+ assertThat(parseTypeExpression("tuple[()]")).isInstanceOf(TypeApplication.class);
+ // ...but not non-empty tuples
+ assertThat(
+ assertThrows(
+ SyntaxError.Exception.class, () -> parseTypeExpression("tuple[(int, str)]")))
+ .hasMessageThat()
+ .contains("syntax error at 'int': expected )");
+ // type expressions can use string literals
+ assertThat(parseTypeExpression("Literal['abc']")).isInstanceOf(TypeApplication.class);
+ // composition
+ assertThat(parseTypeExpression("list[str, dict[str, bool]]"))
+ .isInstanceOf(TypeApplication.class);
+ // type unions
+ assertThat(parseTypeExpression("str | int")).isInstanceOf(BinaryOperatorExpression.class);
+ assertThat(parseTypeExpression("str | int | bool"))
+ .isInstanceOf(BinaryOperatorExpression.class);
+ // empty dict and list literals
+ assertThat(parseTypeExpression("Callable[[], TypeDict[{}]]"))
+ .isInstanceOf(TypeApplication.class);
+ // trailing commas in dict and list arguments
+ assertThat(parseTypeExpression("Callable[[int,],bool]")).isInstanceOf(TypeApplication.class);
+ assertThat(parseTypeExpression("TypeDict[{'foo': int, }]")).isInstanceOf(TypeApplication.class);
+ }
+
+ @Test
+ public void testIllegalTypeExpression_disallowed() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ setFailFast(false);
+ parseStatement("def f(a : (lambda x: x)): pass");
+ assertContainsError("syntax error at '(': expected a type");
+ }
+
+ @Test
+ public void testIllegalTypeExpression_allowedWithFlag() throws Exception {
+ setFileOptions(
+ FileOptions.builder().allowTypeSyntax(true).tolerateInvalidTypeExpressions(true).build());
+
+ parseStatement("def f(a : (lambda x: x)): pass");
+ assertThat(parseTypeExpression("lambda x: x")).isInstanceOf(LambdaExpression.class);
+
+ // Annotations shouldn't consume adjacent params.
+ Statement stmt = parseStatement("def f(p1 : x, p2): pass");
+ assertThat(stmt.kind()).isEqualTo(Statement.Kind.DEF);
+ assertThat(((DefStatement) stmt).getParameters().stream().map(p -> p.getName()))
+ .containsExactly("p1", "p2")
+ .inOrder();
+ }
+
+ @Test
+ public void testDefWithTypeAnnotations() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ parseStatement("def f(a: int): pass");
+ parseStatement("def f(a: tuple[()]): pass");
+ parseStatement("def f(a: list[str]): pass");
+ parseStatement("def f(a: dict[str, int]): pass");
+
+ // test with default values
+ parseStatement("def f(a: int, *, b: bool = True, c): pass");
+
+ // test args and kwargs
+ parseStatement("def f(*args: list[int]): pass");
+ parseStatement("def f(**kwargs: dict[str, Any]): pass");
+
+ // Return type
+ parseStatement("def f() -> int: pass");
+
+ // Type parameters
+ parseStatement("def f[T, U](x: dict[T, U]) -> dict[U, T]: pass");
+ }
+
+ @Test
+ public void testDefBareStarCannotHaveTypeAnnotation() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ setFailFast(false);
+ parseStatement("def f(a, *: int, b: bool): pass");
+ assertContainsError("syntax error at ':': expected )");
+ }
+
+ // TODO(ilist): Python allows trailing commas in type arguments - we probably should too.
+ @Test
+ public void testTrailingCommaNotAllowedInTypeArgumentList() throws Exception {
+ assertThat(parseTypeExpressionError("list[int,]"))
+ .contains("syntax error at ']': expected a type argument");
+ }
+
+ @Test
+ public void testDefWithDisallowedTypeAnnotations() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(false).build());
+ setFailFast(false);
+ parseStatement("def f(a: int): pass");
+ assertContainsError("syntax error at ':': type annotations are disallowed");
+ events.clear();
+ parseStatement("def f[T](): pass");
+ assertContainsError("syntax error at '[': type annotations are disallowed");
+ }
+
+ @Test
+ public void testTypeApplicationsRequireConstructor() throws Exception {
+ assertThat(parseTypeExpressionError("[int]")).contains("syntax error at '[': expected a type");
+ }
+
+ @Test
+ public void testFunctionCallsNotAllowedInTypeExpressions() throws Exception {
+ assertThat(parseTypeExpressionError("int[f(1)]")).contains("syntax error at '(': expected ,");
+ }
+
+ @Test
+ public void testOnlyPipeOperatorsAllowedInTypeExpressions() throws Exception {
+ ImmutableList<TokenKind> badOperators =
+ ImmutableList.of(
+ TokenKind.AMPERSAND,
+ TokenKind.EQUALS,
+ TokenKind.GREATER,
+ TokenKind.LESS,
+ TokenKind.MINUS,
+ TokenKind.PLUS,
+ TokenKind.SLASH,
+ TokenKind.STAR);
+ for (TokenKind op : badOperators) {
+ assertThat(parseTypeExpressionError("int " + op + " str"))
+ .contains("syntax error at '" + op + "'");
+ }
+ }
+
+ @Test
+ public void testTypeAliasStatement_basicFunctionality() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ TypeAliasStatement stmt = (TypeAliasStatement) parseStatement("type X = list");
+ assertThat(stmt.getIdentifier().getName()).isEqualTo("X");
+ assertThat(stmt.getParameters()).isEmpty();
+ assertThat(stmt.getDefinition()).isInstanceOf(Identifier.class);
+ assertThat(((Identifier) stmt.getDefinition()).getName()).isEqualTo("list");
+ }
+
+ @Test
+ public void testTypeAliasStatement_typeParams() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ TypeAliasStatement stmt =
+ (TypeAliasStatement) parseStatement("type my_nullable_dict[T, U] = dict[T, U] | None");
+ assertThat(stmt.getIdentifier().getName()).isEqualTo("my_nullable_dict");
+ assertThat(stmt.getParameters().stream().map(p -> p.getName()))
+ .containsExactly("T", "U")
+ .inOrder();
+ assertThat(stmt.getDefinition()).isInstanceOf(BinaryOperatorExpression.class);
+ }
+
+ @Test
+ public void testTypeAliasStatement_requiresTypeSyntax() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(false).build());
+ setFailFast(false);
+ parseStatement("type X = list");
+ assertContainsError("syntax error at 'type': type annotations are disallowed");
+ }
+
+ @Test
+ public void testTypeAliasStatement_requiresExactlyOneName() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ setFailFast(false);
+ parseStatement("type X, Y = list, int");
+ assertContainsError("syntax error at ',': expected =");
+ }
+
+ @Test
+ public void testTypeAliasStatement_requiresDefinition() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ setFailFast(false);
+ parseStatement("type X # = define_later");
+ assertContainsError("syntax error at 'newline': expected =");
+ }
+
+ @Test
+ public void testTypeAliasStatement_allowsParsingWithUnresolvableDefinition() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ parseStatement("type x = no_such_type");
+ }
+
+ @Test
+ public void testTypeAliasStatement_disallowsIllegalDefinition() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ setFailFast(false);
+ parseStatement("type X = lambda x: x");
+ assertContainsError("syntax error at 'lambda': expected a type");
+ }
+
+ @Test
+ public void testTypeAliasStatement_allowsIllegalDefinition_withFlag() throws Exception {
+ setFileOptions(
+ FileOptions.builder().allowTypeSyntax(true).tolerateInvalidTypeExpressions(true).build());
+ TypeAliasStatement stmt = (TypeAliasStatement) parseStatement("type X = lambda x: x");
+ assertThat(stmt.getDefinition()).isInstanceOf(LambdaExpression.class);
+ }
+
+ @Test
+ public void testTypeIsSoftKeyword() throws Exception {
+ // Test that `type` may be used as an identifier in any context where identifiers are allowed.
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ parseStatement("type = type.type(type)");
+ parseStatement("type type = type");
+ }
+
+ private static enum TypeParamTestKind {
+ DEF_STATEMENT,
+ TYPE_ALIAS_STATEMENT
+ }
+
+ private Statement parseTypeParamTestCaseStatement(TypeParamTestKind testKind, String typeParams)
+ throws Exception {
+ return switch (testKind) {
+ // Extra space in DEF_STATEMENT case to place typeParams at offset 6 in both cases
+ case DEF_STATEMENT -> parseStatement(String.format("def f%s(): pass", typeParams));
+ case TYPE_ALIAS_STATEMENT -> parseStatement(String.format("type X%s = int", typeParams));
+ };
+ }
+
+ @Test
+ public void testTypeParams_mayBeUnused(@TestParameter TypeParamTestKind testKind)
+ throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ parseTypeParamTestCaseStatement(testKind, "[T, U]");
+ }
+
+ @Test
+ public void testTypeParams_allowOnlyIdentifiers(@TestParameter TypeParamTestKind testKind)
+ throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ setFailFast(false);
+ parseTypeParamTestCaseStatement(testKind, "[1]");
+ assertContainsError("syntax error at '1': expected identifier");
+ events.clear();
+ parseTypeParamTestCaseStatement(testKind, "['two']");
+ assertContainsError("syntax error at '\"two\"': expected identifier");
+ events.clear();
+ parseTypeParamTestCaseStatement(testKind, "[(THREE)]");
+ assertContainsError("syntax error at '(': expected identifier");
+ }
+
+ @Test
+ public void testTypeParams_disallowDuplicates(@TestParameter TypeParamTestKind testKind)
+ throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ setFailFast(false);
+ parseTypeParamTestCaseStatement(testKind, "[T, U, T]");
+ assertContainsError("1:14: syntax error at 'T': duplicate type parameter");
+ }
+
+ @Test
+ public void testTypeParams_allowsTrailingCommas(@TestParameter TypeParamTestKind testKind)
+ throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ parseTypeParamTestCaseStatement(testKind, "[T, U,]");
+ }
+
+ @Test
+ public void testTypeParams_cannotBeEmptyIfPresent(@TestParameter TypeParamTestKind testKind)
+ throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ setFailFast(false);
+ parseTypeParamTestCaseStatement(testKind, "[]");
+ assertContainsError("syntax error at ']': expected identifier");
+ events.clear();
+ parseTypeParamTestCaseStatement(testKind, "[,]");
+ assertContainsError("syntax error at ',': expected identifier");
+ }
+
+ @Test
+ public void testEllipsisNotAllowedInValueExpressions() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ assertThat(parseExpressionError("print(...)"))
+ .contains("ellipsis ('...') is not allowed outside type expressions");
+ }
+
+ @Test
+ public void testEllipsisAllowedInTypeExpressionArgumentsOnly() throws Exception {
+ setFileOptions(
+ FileOptions.builder().allowTypeSyntax(true).tolerateInvalidTypeExpressions(false).build());
+ parseStatement("x : tuple[int, ...]");
+ assertThat(parseStatementError("x : ...")).contains("syntax error at '...': expected a type");
+ assertThat(parseStatementError("x : int | ..."))
+ .contains("syntax error at '...': expected identifier");
+ assertThat(parseStatementError("x : tuple[int | ...]"))
+ .contains("syntax error at '...': expected identifier");
+ }
+
+ @Test
+ public void testCastExpression_basicFunctionality() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ CastExpression cast = (CastExpression) parseExpression("cast(list[int], foo())");
+ assertThat(cast.getType()).isInstanceOf(TypeApplication.class);
+ assertThat(cast.getValue()).isInstanceOf(CallExpression.class);
+ }
+
+ @Test
+ public void testCastExpression_isExpression() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ parseStatement("cast(list, x) += cast(list[list], (cast(struct, y)).foo())[cast(int, z)]");
+ }
+
+ @Test
+ public void testCast_isKeyword() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ setFailFast(false);
+ assertThat(parseExpressionError("something.cast(list, x)"))
+ .contains("syntax error at 'cast': expected identifier after dot");
+ assertThat(parseExpressionError("(cast)(list, x)")).contains("expected (");
+ }
+
+ @Test
+ public void testCastExpression_requiresTypeSyntax() throws Exception {
+ // If type syntax is disabled, `cast` is treated as an ordinary identifier.
+ setFileOptions(FileOptions.builder().allowTypeSyntax(false).build());
+ assertThat(parseExpression("cast(list[str], foo())")).isInstanceOf(CallExpression.class);
+ }
+
+ @Test
+ public void testCastExpression_goodSyntax() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ parseExpression(
+ """
+ cast(
+ int,
+ n
+ )\
+ """);
+ parseExpression("cast(int, x,)");
+ }
+
+ @Test
+ public void testCastExpression_badSyntax() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ setFailFast(false);
+ assertThat(parseExpressionError("cast int, x")).contains("syntax error at 'int': expected (");
+ assertThat(parseExpressionError("cast(int, x, y)")).contains("syntax error at 'y': expected )");
+ assertThat(parseExpressionError("cast(int, x"))
+ .contains("syntax error at 'newline': expected )");
+ assertThat(parseExpressionError("cast(*args)"))
+ .contains("syntax error at '*': expected a type");
+ assertThat(parseExpressionError("cast(type=int, value=x"))
+ .contains("syntax error at '=': expected ,");
+ }
+
+ @Test
+ public void testIsInstanceExpression_basicFunctionality() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ IsInstanceExpression cast =
+ (IsInstanceExpression) parseExpression("isinstance(foo(), list | tuple)");
+ assertThat(cast.getType()).isInstanceOf(BinaryOperatorExpression.class);
+ assertThat(cast.getValue()).isInstanceOf(CallExpression.class);
+ }
+
+ @Test
+ public void testIsInstanceExpression_isExpression() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ parseStatement("if isinstance(isinstance(y, list), bool): isinstance(z, str)");
+ }
+
+ @Test
+ public void testIsInstance_isKeyword() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ setFailFast(false);
+ assertThat(parseExpressionError("something.isinstance(x, list)"))
+ .contains("syntax error at 'isinstance': expected identifier after dot");
+ assertThat(parseExpressionError("(isinstance)(x, list)")).contains("expected (");
+ }
+
+ @Test
+ public void testIsInstanceExpression_requiresTypeSyntax() throws Exception {
+ // If type syntax is disabled, `isinstance` is treated as an ordinary identifier.
+ setFileOptions(FileOptions.builder().allowTypeSyntax(false).build());
+ assertThat(parseExpression("isinstance(x, T)")).isInstanceOf(CallExpression.class);
+ }
+
+ @Test
+ public void testIsInstanceExpression_goodSyntax() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ parseExpression(
+ """
+ isinstance(
+ x,
+ T | U[V]
+ )\
+ """);
+ parseExpression("isinstance(x, tuple,)");
+ }
+
+ @Test
+ public void testIsInstanceExpression_badSyntax() throws Exception {
+ setFileOptions(FileOptions.builder().allowTypeSyntax(true).build());
+ setFailFast(false);
+ assertThat(parseExpressionError("isinstance x, int"))
+ .contains("syntax error at 'x': expected (");
+ assertThat(parseExpressionError("isinstance(x, y, int)"))
+ .contains("syntax error at 'int': expected )");
+ assertThat(parseExpressionError("isinstance(x, int"))
+ .contains("syntax error at 'newline': expected )");
+ assertThat(parseExpressionError("isinstance(*args)"))
+ .contains("syntax error at '*': expected expression");
+ assertThat(parseExpressionError("isinstance(value=x, type=int)"))
+ .contains("syntax error at '=': expected ,");
+ }
+
+ @Test
public void testLambda() throws Exception {
parseExpression("lambda a, b=1, *args, **kwargs: a+b");
parseExpression("lambda *, a, *b: 0");
diff --git a/src/test/java/net/starlark/java/syntax/ResolverTest.java b/src/test/java/net/starlark/java/syntax/ResolverTest.java
index 847340c..378b9b0 100644
--- a/src/test/java/net/starlark/java/syntax/ResolverTest.java
+++ b/src/test/java/net/starlark/java/syntax/ResolverTest.java
@@ -40,10 +40,7 @@
// Assertions that parsing and resolution succeeds.
private void assertValid(String... lines) throws SyntaxError.Exception {
- StarlarkFile file = resolveFile(lines);
- if (!file.ok()) {
- throw new SyntaxError.Exception(file.errors());
- }
+ getValidFile(lines);
}
// Asserts that parsing of the program succeeds but resolution fails
@@ -53,6 +50,14 @@
assertContainsError(errors, expectedError);
}
+ private StarlarkFile getValidFile(String... lines) throws SyntaxError.Exception {
+ StarlarkFile file = resolveFile(lines);
+ if (!file.ok()) {
+ throw new SyntaxError.Exception(file.errors());
+ }
+ return file;
+ }
+
// Returns the non-empty list of resolution errors of the program.
private List<SyntaxError> getResolutionErrors(String... lines) throws SyntaxError.Exception {
StarlarkFile file = resolveFile(lines);
@@ -456,6 +461,8 @@
assertThat(errors.get(0).message()).isEqualTo("name 'undef' is not defined");
}
+ // TODO: #27370 - Add resolver behavior for type expressions, add bindingScopeAndIndex tests here.
+
@Test
public void testBindingScopeAndIndex() throws Exception {
checkBindings(
@@ -505,6 +512,236 @@
" aᶠ₀, bᴳ₀, cᶠ₁, dᶠ₂, eᴸ₀, fᴳ₁, gᶠ₃, hᶠ₄");
}
+ @Test
+ public void testBindingScopeAndIndex_varStatement() throws Exception {
+ options.allowTypeSyntax(true);
+ checkBindings(
+ // Var statement creates a binding, even in the absence of assignment.
+ "xᴳ₀ : T",
+ // Var statement can shadow predeclared.
+ "preᴳ₁ : T",
+ "def fᴳ₂():",
+ " xᴳ₀",
+ " preᴳ₁");
+ }
+
+ @Test
+ public void testTypeAliasStatement_mustBeAtTopLevel() throws Exception {
+ options.allowTypeSyntax(true);
+ assertInvalid(
+ ":2:3: type alias statement not at top level",
+ """
+ def f():
+ type X = int
+ """);
+ }
+
+ @Test
+ public void testMultipleTypeAnnotationsDisallowed_topLevel() throws Exception {
+ options.allowTypeSyntax(true);
+ List<SyntaxError> errors =
+ getResolutionErrors(
+ // All four permutations of VarStatement vs annotated assignment statement.
+ """
+ a : int
+ a : str
+
+ b : int = 123
+ b : str
+
+ c : int
+ c : str = "abc"
+
+ d : int = 123
+ d : str = "abc"
+ """);
+ assertContainsError(errors, ":2:1: 'a' redeclared at top level");
+ assertContainsError(errors, ":5:1: 'b' redeclared at top level");
+ assertContainsError(errors, ":8:1: 'c' redeclared at top level");
+ assertContainsError(errors, ":11:1: 'd' redeclared at top level");
+ }
+
+ @Test
+ public void testMultipleTypeAnnotationsDisallowed_localLevel() throws Exception {
+ // Same as testMultipleTypeAnnotationsDisallowed_topLevel but inside a function, where
+ // reassignment is always allowed.
+ options.allowTypeSyntax(true);
+ List<SyntaxError> errors =
+ getResolutionErrors(
+ // All four permutations of VarStatement vs annotated assignment statement.
+ """
+ def f():
+ a : int
+ a : str
+
+ b : int = 123
+ b : str
+
+ c : int
+ c : str = "abc"
+
+ d : int = 123
+ d : str = "abc"
+ """);
+ assertContainsError(errors, "type annotation on 'a' may only appear at its first declaration");
+ assertContainsError(errors, "type annotation on 'b' may only appear at its first declaration");
+ assertContainsError(errors, "type annotation on 'c' may only appear at its first declaration");
+ assertContainsError(errors, "type annotation on 'd' may only appear at its first declaration");
+ }
+
+ @Test
+ public void testMultipleTypeAnnotationsDisallowed_defStatement() throws Exception {
+ options.allowTypeSyntax(true);
+
+ assertValid(
+ """
+ def f():
+ # Redefinition is allowed (but bad style) if second definition has no type
+ # annotation.
+ def a(x : int):
+ pass
+ def a(x):
+ pass
+ """);
+
+ List<SyntaxError> errors =
+ getResolutionErrors(
+ """
+ def f():
+ # Second definition may not have a type annotation, even if first definition has
+ # none.
+ def b(x):
+ pass
+ def b(x : int):
+ pass
+
+ # Return type annotation counts too.
+ def c(x):
+ pass
+ def c(x) -> int:
+ pass
+
+ # Even generic type vars count.
+ def d(x):
+ pass
+ def d[T](x):
+ pass
+ """);
+ // TODO: #27371 - For the case of redefining a function, the error message is a little
+ // confusing. But this is also a pretty rare case.
+ assertContainsError(errors, "type annotation on 'b' may only appear at its first declaration");
+ assertContainsError(errors, "type annotation on 'c' may only appear at its first declaration");
+ assertContainsError(errors, "type annotation on 'd' may only appear at its first declaration");
+ }
+
+ @Test
+ public void testSingleAnnotationWithReassignmentIsAllowed() throws Exception {
+ options.allowTypeSyntax(true);
+ assertValid(
+ """
+ def f():
+ a : int
+ a = 123
+ """);
+ }
+
+ @Test
+ public void testAnnotationFollowedByAssignmentStillCountsAsRedeclaration() throws Exception {
+ options.allowTypeSyntax(true);
+ assertInvalid(
+ "'a' redeclared at top level",
+ """
+ a : int
+ a = 123
+ """);
+ }
+
+ @Test
+ public void testVarStatementMustPreceedAssignment() throws Exception {
+ options.allowTypeSyntax(true);
+ assertInvalid(
+ "type annotation on 'x' may only appear at its first declaration",
+ """
+ def f():
+ x = 123
+ x : int
+ """);
+ }
+
+ @Test
+ public void onlyFirstAssignmentMayBeAnnotated() throws Exception {
+ options.allowTypeSyntax(true);
+ assertInvalid(
+ "type annotation on 'x' may only appear at its first declaration",
+ """
+ def f():
+ x = 123
+ x : int = 123
+ """);
+ }
+
+ @Test
+ public void cannotAnnotateParamInBody() throws Exception {
+ options.allowTypeSyntax(true);
+ assertInvalid(
+ "type annotation on 'x' may only appear at its first declaration",
+ """
+ def f(x):
+ # Invalid even though x has no type annotation above.
+ x : int
+ """);
+ }
+
+ @Test
+ public void testCastExpression_cannotBeLhsOfAssignment() throws Exception {
+ options.allowTypeSyntax(true);
+ StarlarkFile file =
+ resolveFile(
+ """
+ cast(int, x) = 42
+ cast(int, y[0]) = 42
+ cast(list[int], z) += [42]
+ """);
+ assertThat(file.ok()).isFalse();
+ assertContainsError(file.errors(), "cannot assign to 'cast(int, x)'");
+ assertContainsError(file.errors(), "cannot assign to 'cast(int, y[0])'");
+ assertContainsError(file.errors(), "cannot assign to 'cast(list[int], z)'");
+ }
+
+ @Test
+ public void testCastExpression_value_isResolved() throws Exception {
+ options.allowTypeSyntax(true);
+ StarlarkFile badFile = resolveFile("cast(int, f())");
+ assertThat(badFile.ok()).isFalse();
+ assertContainsError(badFile.errors(), "name 'f' is not defined");
+
+ StarlarkFile goodFile =
+ resolveFile(
+ """
+ def f():
+ return 1
+ cast(int, f())
+ """);
+ assertThat(goodFile.ok()).isTrue();
+ }
+
+ @Test
+ public void testCastExpression_type_notResolved() throws Exception {
+ // TODO(brandjon): resolve the cast's type once we have type checking.
+ options.allowTypeSyntax(true);
+ StarlarkFile badFile = resolveFile("cast(NoSuchType[int], 42)");
+ assertThat(badFile.ok()).isTrue();
+ }
+
+ // TODO(b/350661266): resolve types in isinstance().
+ @Test
+ public void testIsInstanceExpression_notYetSupported() throws Exception {
+ options.allowTypeSyntax(true);
+ StarlarkFile badFile = resolveFile("isinstance(x, list)");
+ assertThat(badFile.ok()).isFalse();
+ assertContainsError(badFile.errors(), "isinstance() is not yet supported");
+ }
+
// checkBindings verifies the binding (scope and index) of each identifier.
// Every variable must be followed by a superscript letter (its scope)
// and a subscript numeral (its index). They are replaced by spaces, the
@@ -527,6 +764,13 @@
+ "₀₁₂₃₄₅₆₇₈₉".charAt(id.getBinding().getIndex()) // 10 is plenty
+ out[0].substring(id.getEndOffset() + 2);
}
+
+ @Override
+ public void visit(VarStatement varStatement) {
+ visit(varStatement.getIdentifier());
+ // Don't visit type expression, it isn't processed.
+ // TODO: #27370 - Include the type expression in these tests.
+ }
}.visit(file);
assertThat(out[0]).isEqualTo(src);
}
