Stop allocating new tokens in the lexer
There's only one Token and it gets reused.
This reduces the memory usage of the lexer. Parsing time seems to be 5%-10%
faster with this change on a large file. This makes little difference on the
overall performance of Bazel though.
RELNOTES: None.
PiperOrigin-RevId: 199310860
diff --git a/src/main/java/com/google/devtools/build/lib/syntax/Token.java b/src/main/java/com/google/devtools/build/lib/syntax/Token.java
index 3c99df5..733b538 100644
--- a/src/main/java/com/google/devtools/build/lib/syntax/Token.java
+++ b/src/main/java/com/google/devtools/build/lib/syntax/Token.java
@@ -22,14 +22,14 @@
class Token {
TokenKind kind;
- final int left;
- final int right;
+ int left;
+ int right;
/**
* value is an Integer if the kind is INT.
* It is a String if the kind is STRING, IDENTIFIER, or COMMENT.
* It is null otherwise.
*/
- @Nullable final Object value;
+ @Nullable Object value;
Token(TokenKind kind, int left, int right) {
this(kind, left, right, null);
@@ -42,6 +42,10 @@
this.value = value;
}
+ Token copy() {
+ return new Token(kind, left, right, value);
+ }
+
/**
* Constructs an easy-to-read string representation of token, suitable for use
* in user error messages.
