module

Noir::JSLiteralScanner

JSLiteralScanner provides utilities for scanning JavaScript source code while properly skipping string literals, comments, template literals, and regex. This ensures parenthesis/brace matching doesn't get confused by literals.

Implementation note: every public entry point takes CHAR indices and returns CHAR indices (callers slice with char-based String#[]). The scanners themselves run over an indexed char source instead of probing the string with String#[](Int) — which is O(index) once a string contains any multi-byte UTF-8 — and accumulate through String::Builder instead of per-char String#+, which reallocated the whole prefix on every append. ASCII content (the overwhelmingly common case) scans its byte slice with zero allocation; non-ASCII content pays one up-front chars materialization and stays linear.

Constants

KEYWORD_WINDOW = 12

The regex-context checks only ever look at the tail of the scanned output: the last non-whitespace char, and ends_with? against the keywords above (longest: "instanceof", 10 chars). A rolling window of this size replaces re-materializing the whole accumulated prefix on every '/' encountered.

REGEX_PRECEDING_CHARS = Set {'(', '[', '{', ',', ':', ';', '=', '!', '&', '|', '?', '+', '-', '*', '%', '>', '~', '^'}

Characters that can precede a regex literal (operators/punctuation expecting expression). '<' is deliberately absent: a < /re/ is never written, while JSX closing tags (</div>) put a '/' after '<' on every other line.

REGEX_PRECEDING_KEYWORDS = ["return", "case", "throw", "in", "of", "typeof", "instanceof", "void", "delete", "new"]

Keywords that can precede a regex literal in JavaScript

STRING_PRECEDING_KEYWORDS = REGEX_PRECEDING_KEYWORDS + ["from", "import", "export", "yield", "await"]

Words a string operand may follow with no space between (minified import{a}from"./x", export*from"./x", await"x"). Not else/do: JSX prose writes else's and do's far more often than code does.

TEMPLATE_TEXT = -1

End (exclusive) of the template literal whose opening backtick is at pos, or size when it never closes. ${ … } substitutions are tracked with their own brace depth, so a nested template (${items.map(i =>

  • ${i}
  • )}), a string or a } inside a substitution does not end the outer literal early. Each stack entry is either TEMPLATE_TEXT or the brace depth of an open substitution; an explicit stack keeps hostile nesting off the call stack.

    URL_SCHEMES = {"http", "https"}

    Schemes whose :// in JSX text is not a comment (see url_slashes?).

    Class methods

    extract_paren_content(content : String, start_pos : Int32) : ScanResult | Nil

    Extract content between parentheses while skipping literals Returns the content inside the parentheses (not including the parens)

    Source
    extract_paren_content_at_byte(content : String, start_byte : Int32) : ScanResult | Nil

    extract_paren_content from a BYTE offset; end_pos is a byte offset too. The byte walk emits each byte of a multi-byte char as its own U+0080..U+00FF char, so fold those back into the source bytes.

    Source
    find_matching_brace(content : String, open_brace_idx : Int32) : Int32 | Nil

    Find matching closing brace, skipping literals

    Source
    find_matching_brace(chars : Array(Char), open_brace_idx : Int32) : Int32 | Nil

    The same over a string's already materialized chars, for a caller matching many delimiters in one non-ASCII string: the String forms above re-materialize it on every call.

    Source
    find_matching_brace(offsets : CharOffsets, open_brace_idx : Int32) : Int32 | Nil

    The char forms above over a prebuilt CharOffsets: same answers, but a per-route caller on a non-ASCII file reuses one chars array instead of rebuilding it per call.

    Source
    find_matching_brace_at_byte(content : String, open_brace_byte : Int32) : Int32 | Nil

    BYTE-offset variants over the raw UTF-8 bytes, for any content. Every delimiter, quote and comment marker is ASCII and no byte of a multi-byte char is, so non-ASCII text is just opaque bytes here — and the walk costs only the distance scanned, where the char variants above materialize content.chars on every call once a file is non-ASCII.

    Source
    find_matching_bracket(content : String, open_bracket_idx : Int32) : Int32 | Nil

    Find matching closing bracket, skipping literals

    Source
    find_matching_paren(content : String, open_paren_idx : Int32) : Int32 | Nil

    Find matching closing paren, skipping literals

    Source
    find_matching_paren(chars : Array(Char), open_paren_idx : Int32) : Int32 | Nil
    Source
    find_matching_paren(offsets : CharOffsets, open_paren_idx : Int32) : Int32 | Nil
    Source
    find_matching_paren_at_byte(content : String, open_paren_byte : Int32) : Int32 | Nil
    Source
    quote_opens_string?(src, pos : Int32) : Bool

    Does the quote at pos open a string literal? JavaScript never puts a string right after a name, so a quote glued to a word is prose: JSX text such as <p>We don't track you</p>, whose apostrophe would otherwise open a string that hides every delimiter after it. Keywords that take an operand are the exception (return'x', minified case"a":). Non-ASCII counts as a word char, so the byte and char walks agree on a multi-byte letter before the quote.

    Source
    regex_context?(last_char : Char | Nil, last_word : String, before_last : Char | Nil = nil) : Bool

    The single rule for "can a '/' here start a regex literal?".

    last_char is the last non-whitespace character of the code scanned so far (nil when nothing has been scanned yet) and last_word is the identifier that ends at last_char, or "" when last_char is punctuation. Every JavaScript scanner in the tree asks this question — the literal scanners below, JSRouteExtractor.strip_js_comments and JSLexer#looks_like_regex? — and they must agree: a '/' misread as division lets the regex body's quotes and // open a string or a comment that runs to EOF, which silently drops every route in the file.

    before_last is the raw character just before last_char: a '/' after i++ / i-- is division, not a regex.

    Source
    regex_literal_end(src, size : Int32, pos : Int32) : Tuple(Int32, Bool)

    Scans the regex literal whose opening '/' is at pos. Returns {end, true} (end exclusive, flags included) when it closes, or {line_end, false} when no closing '/' appears before the newline (or EOF) at line_end. A regex literal cannot span lines, so a '/' the context rule misjudged (a JSX </tag>, a++ / 2) is division, not a regex that swallows the rest of the file. Callers treat every '/' before line_end as division afterwards.

    Source
    template_literal_end(src, size : Int32, pos : Int32) : Int32
    Source
    url_slashes?(src, pos : Int32) : Bool

    Are the // at pos the slashes of an http:// / https:// URL in JSX text (<p>Docs at https://hono.dev</p>) rather than a comment opener? Code never glues a comment to a bare http: and a non-space char. Not a comment, the // then reads as an empty regex after ':'.

    Source

    Nested types