module

Hwaro::MarkdRegexStackFix

Constants

ANCHORED = {Markd::Rule::ESCAPABLE => anchored(Markd::Rule::ESCAPABLE), Markd::Rule::EMAIL_AUTO_LINK => EMAIL_AUTO_LINK, Markd::Rule::AUTO_LINK => anchored(Markd::Rule::AUTO_LINK), Markd::Rule::NUMERIC_HTML_ENTITY => anchored(Markd::Rule::NUMERIC_HTML_ENTITY)}

Anchored copies of the other ^-rules Inline#match is called with.

COMMENT_STRING = "<!---->|<!--(?:-?[^>-])(?:-?[^-])*+-->"
EMAIL_AUTO_LINK = Regex.new("<([a-zA-Z0-9.!#$%&'*+\\/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*+)>", Regex::Options::ANCHORED)

Rule::EMAIL_AUTO_LINK with its domain-label loop possessive: every iteration starts with ., and giving one back (or shortening its label) leaves a . or an alphanumeric where the closing > must be, so backtracking can never produce a match.

ESCAPABLE_BYTES = Set(UInt8).new("!\"#$%&'()*+,./:;<=>?@[\\]^_`{|}~-".bytes)

CommonMark's escapable ASCII punctuation (Rule::ESCAPABLE_STRING).

HTML_BLOCK_OPEN = Markd::Rule::HTML_BLOCK_OPEN[0...-1] + [Regex.new(((("^(?:" + OPEN_TAG) + "|") + Markd::Rule::CLOSE_TAG) + ")\\s*$", Regex::Options::IGNORE_CASE)]
HTML_TAG = Regex.new(HTML_TAG_STRING, Regex::Options::IGNORE_CASE | Regex::Options::ANCHORED)

Compile-time ANCHORED: matches only at the start offset handed to match_at_byte_index, so the parser never has to copy the rest of the paragraph to anchor a ^ pattern.

HTML_TAG_STRING = "(?:#{OPEN_TAG_STRING}|#{Markd::Rule::CLOSE_TAG_STRING}|#{COMMENT_STRING}|#{Markd::Rule::PROCESSING_INSTRUCTION_STRING}|#{Markd::Rule::DECLARATION_STRING}|#{Markd::Rule::CDATA_STRING})"
LINK_DESTINATION_BRACES = Regex.new(("(?:[<](?:[^<>\\t\\n\\\\\\x00]|" + Markd::Rule::ESCAPED_CHAR_STRING) + ")*+[>])", Regex::Options::ANCHORED)
NON_ASCII_SPACE = /(?=[^\x00-\x7F])\s/

A whitespace character outside ASCII. Crystal compiles regexes with PCRE2 UCP, so markd's \s also matches U+00A0, U+3000, U+2000–U+200A, U+202F, U+205F, U+1680, U+0085, U+2028 and U+2029 — and an unquoted attribute value may contain those characters too.

OPEN_TAG = OPEN_TAG_STRING
OPEN_TAG_STRING = "<#{Markd::Rule::TAG_NAME_STRING}#{Markd::Rule::ATTRIBUTE}*+\\s*/?>"

Rule::ATTRIBUTE repeated possessively (see the file comment).

UPSTREAM_HTML_TAG = anchored(Markd::Rule::HTML_TAG)

Upstream's own tag regexes, for the fallback below.

Class methods

anchored(regex : Regex) : Regex
Source
bracketed_length(text : String, pos : Int32, close : UInt8, stops : Tuple) : Int32 | Nil

Byte length of the bracketed construct starting at pos — exactly the match the upstream open (ESC | body)* close regex produces, including its backtracking — or nil when there is none. stops are the bytes the body cannot contain besides close (NUL for titles, [ for labels).

Source
bracketed_length(table : Array(Int32), pos : Int32) : Int32 | Nil
Source
bracketed_table(text : String, close : UInt8, stops : Tuple) : Array(Int32)

For every position p, where the regex's body loop, entered at p, finally matches the closer (the index just past it), or -1 for no match. The result for a bracket opening at pos is table[pos + 1], and it does not depend on pos itself — so one table answers every attempt in a paragraph, and a line of thousands of unclosed titles costs one pass instead of one pass per title.

The regex's search order at each position is: a \X escape pair (X escapable), then a single body byte (a \ is always one), and only when no further iteration succeeds, the closer. So the result at p depends only on the results at p + 1 and p + 2, and one backward pass computes all of them with no recursion and no stack. A close or stops byte NOT preceded by \ is where every parse stops: the loop cannot step over it, so the result there is the closer or nothing.

Source
html_block_open?(line : String) : Bool

Whether line opens a type-7 HTML block, exactly as the last Rule::HTML_BLOCK_OPEN pattern decides (same fast path and fallback).

Source
html_tag_length(text : String, pos : Int32) : Int32 | Nil

Byte length of the HTML tag at pos, exactly as Rule::HTML_TAG reads it.

The possessive fast path takes the greedy path upstream tries first, so when it matches, upstream matches the same text. It can only miss a tag upstream finds by backtracking an unquoted value to end before a NON-ASCII \s character (the only whitespace a value can contain), so just then upstream's regex decides. ucp is whether the text contains such a character (callers cache it per paragraph). A fallback that still exhausts the JIT stack reads as "no tag", where upstream crashed.

Source
html_tag_length(text : String, pos : Int32, valid : Bool, & : -> Bool) : Int32 | Nil

ucp is asked only when the fast path misses, so callers can compute (or cache) it lazily.

Source
match_at(regex : Regex, text : String, pos : Int32, valid : Bool) : Regex::MatchData | Nil

Match regex at byte pos of text without copying it. PCRE2 checks the WHOLE subject for valid UTF-8 on every call unless told not to, so matching in place once per bracket was still quadratic on a long line; valid (the caller checked text.valid_encoding? once) lets it skip that check. Invalid text keeps the check, and PCRE2's own error for it.

Source
non_ascii_space?(text : String) : Bool
Source