[8.8.0] Accept and ignore Bazel 10 Starlark type syntax (#30373)
### Description
Backports the Bazel 10 Starlark type syntax parser to the 8.x line in
"accept but ignore" mode: `.bzl` files using type annotations, type
aliases, generic `def` parameters, `...`, `cast()`, and `isinstance()`
now load successfully, with all type information ignored. This matches
master's default behavior (`--experimental_starlark_type_syntax`
defaults to true, type checking off ⇒ annotations are parsed tolerantly
as arbitrary expressions and never resolved).
Without this PR, 8.8.0 fails to load even the most basic annotated file
with a syntax error:
```starlark
def f(x: list[str]): # syntax error at ':': type annotations are disallowed
pass
```
The following commits are cherry-picked, in order:
* 299e903710 Extend syntax with parameter type annotations
* 2dbda274cd Use type specific productions
* d1c3727795 Extend syntax with return types
* ab034aea73 Implement --experimental_starlark_types* flags
* fb7ca12439 Implement TypeApplication and fix NodePrinter
* aa61fc8a8d Allow empty type arguments for type applications (parser
support only)
* 0b615ff55e Introduce Expression.parseTypeExpression() (parser support
only)
* f60a22e8ef Allow parsing arbitrary uninterpreted Starlark type
expressions
* f3653ca8ae Allow parsing type alias statements
* 88678c00c4 Allow parsing generic type parameters in `def` statements
* 96d2f60fb9 Allow parsing `cast` expressions (and evaluating them as
the value argument)
* 664e87b594 Evaluate type alias statements as a no-op
* 5c8655ca1f Add variable type annotation syntax and resolver behavior
* 2390ce6760 Add Ellipsis token and node
* 515083c24a Add `isinstance` keyword, and allow isinstance(x,t) to be
parsed - but not resolved
* 44665ecd71 Properly gate dynamic type checking with flag (syntax
tolerance only)
* f38f0e448a Disallow type syntax in .scl files
* f1b3273f9f, 2004bad12f, f0011d8f29 (the three parser-only slices from
#30092: `tuple[T, ...]`, `tuple[()]`, struct type)
### How to review
Every commit keeps its original `Change-Id`, so the deviation of each
cherry-pick from its master original can be computed mechanically with
`git range-diff` (checking out this PR as `<pr-head>`):
```sh
git log --reverse --format=%H origin/release-8.8.0..<pr-head> | while read pick; do
cid=$(git log -1 --format=%B $pick | sed -n 's/^Change-Id: //p' | head -1)
orig=$(git log --format=%H --grep="Change-Id: $cid" origin/master | tail -1)
git range-diff --creation-factor=100 $orig^! $pick^!
done
```
Only the lines range-diff marks as changed between the two patches need
human review; everything else is upstream code verbatim. Running this
today gives the following review-effort map (changed diff lines per
commit): all commits are at or below 22 except 0b615ff55e (99),
f60a22e8ef (51), 664e87b594 (44), 5c8655ca1f (267), 44665ecd71 (133),
and f38f0e448a (91) — which are exactly the sliced/adapted commits
explained below. For the last three commits, substitute the
already-sliced commits from #30092 (1681dfb115, b172b104c3, 745ec85067)
as the range-diff baseline instead of the master originals; against that
baseline they deviate by 2, 12, and 2 lines respectively.
Each commit is best reviewed against its master original (referenced by
`PiperOrigin-RevId`/`Change-Id` in the message). Everything not listed
below is verbatim upstream code; in particular, the type grammar in
`Parser.java` (all `parseType*` productions, type aliases, generics,
`cast`/`isinstance`) and the new AST node files `TypeAliasStatement`,
`TypeApplication`, `Ellipsis`, and `IsInstanceExpression` are
byte-identical to current master.
Non-trivial deviations from master, grouped by cause:
**1. The type checking machinery is intentionally not backported**
(~4,800 lines on master: `StarlarkType`, `Types`, `TypeChecker`,
`TypeTagger`, `TypeResolver`, and the dynamic checks in
`Eval`/`StarlarkFunction`):
* 0b615ff55e is taken without the static `Resolver.resolveType()` (it
needs the types package).
* 44665ecd71 is reduced to its
`FileOptions`/`Parser`/flag/`BzlCompileFunction` hunks; the
`Eval`/`StarlarkFunction` dynamic-check gating hunks are dropped. The
`StarlarkSemantics.EXPERIMENTAL_STARLARK_TYPE_CHECKING` key constant is
added by hand since `BzlCompileFunction` still needs it.
* `CastExpression` lacks master's `starlarkType` field/getter/setter
(the type-tagging slot, only written by `TypeTagger`).
* Hunks touching checker test files
(`TypeCheckTest`/`DynamicTypeCheckTest`, the syntax-package
`StarlarkTypesTest`) are dropped wherever a commit touched them.
* Net effect: `--experimental_starlark_type_checking` (default off) on
8.x only switches annotations from tolerant to structured parsing; it
performs no type checking.
**2. The doc-comments feature (03921eded1) is not backported**, so its
threading is stripped from 5c8655ca1f's hunks:
* `AssignmentStatement` and `VarStatement` have no `docComments`
field/getter; their constructors take one fewer parameter.
* `Parser#parseAssignment` does not call `maybeParseTrailingDocComment`
(method not introduced), and `Expression#parseExpression` does not skip
`DOC_COMMENT_*` tokens.
* `Resolver#bind` is `bind(id, isLoad, hasType)` instead of master's
four-parameter form; `createBindingsForLHS` takes no doc-comments
parameter.
**3. `Binding.isSyntactic` does not exist on 8.x.** In the
one-annotation-per-declaration check inside `Resolver#bind`, master's
`bind.isSyntactic` is replaced by `bind.first != null` (on 8.x,
`Binding#first` is documented as "first binding use, if syntactic", so
the two are equivalent).
**4. Older test scaffolding on 8.x:**
* `EvaluationTestCase` gains `setFileOptions`/`getFileOptions` (ported
from master, folded into the 664e87b594 pick) because the cherry-picked
`EvaluationTest` cases need them.
* In `ParserTest`/`NodePrinterTest`/`ResolverTest`, only each commit's
own payload tests are taken where the surrounding context consists of
master-only tests that don't exist on 8.x; two `NodePrinterTest`
assertions stay in 8.x's `join(...)` style.
**5. The `parseExpression` → `parseExpr` rename** from f60a22e8ef is
applied in full so the parser stays textually close to master.
### Verification
`//src/test/java/net/starlark/java/...` and
`//src/test/java/com/google/devtools/build/lib/starlark:StarlarkTypesTest`
pass. Additionally verified end-to-end: a workspace exercising unknown
type names, providers as types, `list[str]`, `tuple[int, ...]`,
`tuple[()]`, `struct[{"a": int}]`, parameterized type aliases, variable
annotations, string-literal types, and runtime `cast()` passthrough
loads identically (default flags) with a Bazel built from this branch
and one built from master, and annotated module-level assignments
execute correctly.
### Motivation
Forward compatibility for the 8.x LTS line: once rulesets start adopting
Bazel 10 type annotations, users on Bazel 8 must still be able to load
those rulesets. This is the 8.x counterpart of the tolerance work that
already landed for 9.0.0 (#27760, #27838, #28069) and is pending for
9.3.0 (#30092).
### Build API Changes
Yes: this extends the accepted `.bzl` grammar and adds the experimental
flags `--experimental_starlark_type_syntax` (default true),
`--experimental_starlark_types_allowed_paths`, and
`--experimental_starlark_type_checking` (default false) to 8.x. Part of
the Starlark types effort (#27370). The change is purely additive and
backward compatible: previously invalid syntax now parses and is
ignored; `.scl` files continue to reject type syntax.
### Checklist
- [x] I have added tests for the new use cases (if any).
- [ ] I have updated the documentation (if applicable).
### Release Notes
RELNOTES: Bazel now parses and ignores the Starlark type annotation
syntax introduced by newer Bazel versions (gated by
`--experimental_starlark_type_syntax`, enabled by default).
---------
Co-authored-by: Googler <ilist@google.com>
Co-authored-by: arostovtsev <arostovtsev@google.com>
Co-authored-by: brandjon <brandjon@google.com>
Co-authored-by: Ian (Hee) Cha <heec@google.com>{Fast, Correct} - Choose two
Build and test software of any size, quickly and reliably.
Speed up your builds and tests: Bazel rebuilds only what is necessary. With advanced local and distributed caching, optimized dependency analysis and parallel execution, you get fast and incremental builds.
One tool, multiple languages: Build and test Java, C++, Android, iOS, Go, and a wide variety of other language platforms. Bazel runs on Windows, macOS, and Linux.
Scalable: Bazel helps you scale your organization, codebase, and continuous integration solution. It handles codebases of any size, in multiple repositories or a huge monorepo.
Extensible to your needs: Easily add support for new languages and platforms with Bazel's familiar extension language. Share and re-use language rules written by the growing Bazel community.
Follow our tutorials:
To report a security issue, please email security@bazel.build with a description of the issue, the steps you took to create the issue, affected versions, and, if known, mitigations for the issue. Our vulnerability management team will respond within 3 working days of your email. If the issue is confirmed as a vulnerability, we will open a Security Advisory. This project follows a 90 day disclosure timeline.
See CONTRIBUTING.md