tree 99cf873fda5b90e85a62ce16b0f800e6999fafa3
parent c9c28a9b4324f1eebbbe2e23f471e0880fabd559
author Googler <brandjon@google.com> 1684847191 -0700
committer Copybara-Service <copybara-worker@google.com> 1684847294 -0700

Add flag for lexer to reject non-ASCII strings

The toggle is added to FileOptions, where syntax-based switches for the interpreter go. FileOptions is now threaded through to the Lexer from the public entrypoints (StarlarkFile#parse, Expression#parse); previously it only went to the Resolver.

Both raw characters and octal escapes are checked. Raw characters are processed in two separate code paths since there's a fast-path for scanning strings without escapes.

Added additional assertions to existing test cases for octal escapes. Added two new test cases covering non-ascii raw data and non-ascii octal escapes respectively. FileOptions is statefully manipulated in the tests; I consider this less evil than trying to thread another parameter through the existing test helpers.

(When reading the test cases, mind Java escapes vs Starlark ones.)

PiperOrigin-RevId: 534397406
Change-Id: I5a2e57963fdecd606114deff0c19b38ba8fb738c