bazel syntax: fine-grained syntax locations
This change improves the precision with which the locations
of source tokens are recorded in the syntax tree. Prior to
this change, every Node held a single LexerLocation object
that recorded the start and end offsets of the node, plus
a reference to the shared LineNumberTable (LNT), that maps
these offsets to Locations. This had a cost of one reference
and one LexerLocation object per node.
This change causes every Node to record the offsets only of
its salient tokens, plus a reference to the LNT. For example,
in the expression "1 + 2", the only salient token is the plus
operator; the start and end offsets can be computed inductively
by delegating to x.getStartLocation and y.getEndLocation.
Similarly, in f(x), the salient tokens are '(' and ')'.
This has a cost of 1 word plus approximately 1 int per Node.
Consequently, we can record the exact position of operators
that fail, and do so using less memory than before.
Now, when an expression such as 'f().g() + 1' fails,
the location in the error message will refer to the '+'
operator or one of the two '(' tokens. Before, all
three errors would be wrongly reported at the same place:
f, since it is the start of all three subexpressions.
Overview:
- Every Node has a reference to the LNT, set immediately
after construction. (Morally it is part of the constructor
but it's fussy to set it that way.)
- Every node defines getStartOffset and getEndOffset,
typically by delegating to its left and right subtrees.
- Node end offsets are exclusive again. CL 170723732 was a mistake:
half-open intervals are mathematically simpler.
A client that wants to subtract one may do that.
But there are none.
- Comprehension.{For,If} are now true Nodes.
- StarlarkFile's extent is now (correctly) the entire file,
not just the range from the first statement to the last.
- The parser provides offsets of salient tokens to the Node constructors.
- IntegerLiteral now retains the raw token text in addition to the value.
- Token is gone. Its four fields are now embedded in the Lexer.
- Eval uses the following token positions in run-time error messages:
x+y f(x) x[i] x.y x[i:j] k: v
^ ^ ^ ^ ^ ^
- Location is final. LexerLocation and LineAndColumn are gone.
- Misparsed source represented as an Identifier now has the text of the
source instead of "$error$". This is more faithful and causes
the offsets to be correct.
- The offsets of the orig Identifier in load("module", local="orig")
coincide with the text 'orig', sans quotation marks.
Benchmark: saves about 65MB (1% of live RAM) retained by the
Usual Benchmark, a deps query.
RELNOTES: N/A
PiperOrigin-RevId: 305803031
diff --git a/src/main/java/com/google/devtools/build/lib/syntax/Argument.java b/src/main/java/com/google/devtools/build/lib/syntax/Argument.java
index cc54b58..37d2738 100644
--- a/src/main/java/com/google/devtools/build/lib/syntax/Argument.java
+++ b/src/main/java/com/google/devtools/build/lib/syntax/Argument.java
@@ -24,7 +24,7 @@
*/
public abstract class Argument extends Node {
- private final Expression value;
+ protected final Expression value;
Argument(Expression value) {
this.value = Preconditions.checkNotNull(value);
@@ -34,6 +34,11 @@
return value;
}
+ @Override
+ public int getEndOffset() {
+ return value.getEndOffset();
+ }
+
/** Return the name of this argument's parameter, or null if it is not a Keyword argument. */
@Nullable
public String getName() {
@@ -45,6 +50,11 @@
Positional(Expression value) {
super(value);
}
+
+ @Override
+ public int getStartOffset() {
+ return value.getStartOffset();
+ }
}
/** Syntax node for a keyword argument, {@code f(id=expr)}. */
@@ -53,34 +63,55 @@
// Unlike in Python, keyword arguments in Bazel BUILD files
// are about 10x more numerous than positional arguments.
- final Identifier identifier;
+ final Identifier id;
- Keyword(Identifier identifier, Expression value) {
+ Keyword(Identifier id, Expression value) {
super(value);
- this.identifier = identifier;
+ this.id = id;
}
public Identifier getIdentifier() {
- return identifier;
+ return id;
}
@Override
public String getName() {
- return identifier.getName();
+ return id.getName();
+ }
+
+ @Override
+ public int getStartOffset() {
+ return id.getStartOffset();
}
}
/** Syntax node for an argument of the form {@code f(*expr)}. */
public static final class Star extends Argument {
- Star(Expression value) {
+ private final int starOffset;
+
+ Star(int starOffset, Expression value) {
super(value);
+ this.starOffset = starOffset;
+ }
+
+ @Override
+ public int getStartOffset() {
+ return starOffset;
}
}
/** Syntax node for an argument of the form {@code f(**expr)}. */
public static final class StarStar extends Argument {
- StarStar(Expression value) {
+ private final int starStarOffset;
+
+ StarStar(int starStarOffset, Expression value) {
super(value);
+ this.starStarOffset = starStarOffset;
+ }
+
+ @Override
+ public int getStartOffset() {
+ return starStarOffset;
}
}
