Noir::JSLiteralScanner
JSLiteralScanner provides utilities for scanning JavaScript source code while properly skipping string literals, comments, template literals, and regex. This ensures parenthesis/brace matching doesn't get confused by literals.
Implementation note: every public entry point takes CHAR indices and
returns CHAR indices (callers slice with char-based String#[]). The
scanners themselves run over an indexed char source instead of probing
the string with String#[](Int) — which is O(index) once a string
contains any multi-byte UTF-8 — and accumulate through String::Builder
instead of per-char String#+, which reallocated the whole prefix on
every append. ASCII content (the overwhelmingly common case) scans its
byte slice with zero allocation; non-ASCII content pays one up-front
chars materialization and stays linear.
Constants
The regex-context checks only ever look at the tail of the scanned
output: the last non-whitespace char, and ends_with? against the
keywords above (longest: "instanceof", 10 chars). A rolling window of
this size replaces re-materializing the whole accumulated prefix on
every '/' encountered.
Characters that can precede a regex literal (operators/punctuation expecting expression).
'<' is deliberately absent: a < /re/ is never written, while JSX
closing tags (</div>) put a '/' after '<' on every other line.
Keywords that can precede a regex literal in JavaScript
Words a string operand may follow with no space between (minified
import{a}from"./x", export*from"./x", await"x"). Not else/do:
JSX prose writes else's and do's far more often than code does.
End (exclusive) of the template literal whose opening backtick is at
pos, or size when it never closes. ${ … } substitutions are
tracked with their own brace depth, so a nested template
(${items.map(i =>
)}), a string or a } inside a
substitution does not end the outer literal early. Each stack entry
is either TEMPLATE_TEXT or the brace depth of an open substitution;
an explicit stack keeps hostile nesting off the call stack.
Schemes whose :// in JSX text is not a comment (see url_slashes?).
Class methods
Extract content between parentheses while skipping literals Returns the content inside the parentheses (not including the parens)
extract_paren_content from a BYTE offset; end_pos is a byte offset
too. The byte walk emits each byte of a multi-byte char as its own
U+0080..U+00FF char, so fold those back into the source bytes.
Find matching closing brace, skipping literals
The same over a string's already materialized chars, for a caller
matching many delimiters in one non-ASCII string: the String forms
above re-materialize it on every call.
The char forms above over a prebuilt CharOffsets: same answers, but a
per-route caller on a non-ASCII file reuses one chars array instead
of rebuilding it per call.
BYTE-offset variants over the raw UTF-8 bytes, for any content. Every
delimiter, quote and comment marker is ASCII and no byte of a
multi-byte char is, so non-ASCII text is just opaque bytes here — and
the walk costs only the distance scanned, where the char variants above
materialize content.chars on every call once a file is non-ASCII.
Find matching closing bracket, skipping literals
Find matching closing paren, skipping literals
Does the quote at pos open a string literal? JavaScript never puts
a string right after a name, so a quote glued to a word is prose: JSX
text such as <p>We don't track you</p>, whose apostrophe would
otherwise open a string that hides every delimiter after it. Keywords
that take an operand are the exception (return'x', minified
case"a":). Non-ASCII counts as a word char, so the byte and char
walks agree on a multi-byte letter before the quote.
The single rule for "can a '/' here start a regex literal?".
last_char is the last non-whitespace character of the code scanned
so far (nil when nothing has been scanned yet) and last_word is the
identifier that ends at last_char, or "" when last_char is
punctuation. Every JavaScript scanner in the tree asks this question —
the literal scanners below, JSRouteExtractor.strip_js_comments and
JSLexer#looks_like_regex? — and they must agree: a '/' misread as
division lets the regex body's quotes and // open a string or a
comment that runs to EOF, which silently drops every route in the file.
before_last is the raw character just before last_char: a '/'
after i++ / i-- is division, not a regex.
Scans the regex literal whose opening '/' is at pos. Returns
{end, true} (end exclusive, flags included) when it closes, or
{line_end, false} when no closing '/' appears before the newline (or
EOF) at line_end. A regex literal cannot span lines, so a '/' the
context rule misjudged (a JSX </tag>, a++ / 2) is division, not a
regex that swallows the rest of the file. Callers treat every '/'
before line_end as division afterwards.
Are the // at pos the slashes of an http:// / https:// URL in
JSX text (<p>Docs at https://hono.dev</p>) rather than a comment
opener? Code never glues a comment to a bare http: and a non-space
char. Not a comment, the // then reads as an empty regex after ':'.