module

Hwaro::Content::Processors::ServerHighlighter

Build-time highlighter: Tartrazine lexers + hljs-compatible classes.

Constants

MAX_CACHE_ENTRIES = 2048
MAX_CACHED_BLOCK_BYTES = 65536

Blocks larger than this are highlighted but not cached, keeping memory bounded; the entry cap guards pathological unique-block counts (clear-on-full is deterministic for output, only a speed hit).

TOKEN_CLASS_PREFIXES = [{"CommentPreproc", "hljs-meta"}, {"Comment", "hljs-comment"}, {"KeywordConstant", "hljs-literal"}, {"KeywordType", "hljs-type"}, {"Keyword", "hljs-keyword"}, {"NameKeyword", "hljs-keyword"}, {"NameAttribute", "hljs-attr"}, {"NameBuiltin", "hljs-built_in"}, {"NameClass", "hljs-title class_"}, {"NameConstant", "hljs-variable constant_"}, {"NameDecorator", "hljs-meta"}, {"NameEntity", "hljs-symbol"}, {"NameException", "hljs-title class_"}, {"NameFunction", "hljs-title function_"}, {"NameLabel", "hljs-symbol"}, {"NameNamespace", "hljs-title class_"}, {"NameProperty", "hljs-property"}, {"NameTag", "hljs-name"}, {"NameVariable", "hljs-variable"}, {"NameOperator", "hljs-operator"}, {"LiteralDate", "hljs-string"}, {"LiteralNumber", "hljs-number"}, {"LiteralStringEscape", "hljs-string"}, {"LiteralStringInterpol", "hljs-subst"}, {"LiteralStringRegex", "hljs-regexp"}, {"LiteralStringSymbol", "hljs-symbol"}, {"LiteralStringDoc", "hljs-doctag"}, {"LiteralString", "hljs-string"}, {"Literal", "hljs-literal"}, {"OperatorWord", "hljs-keyword"}, {"Operator", "hljs-operator"}, {"Punctuation", "hljs-punctuation"}, {"GenericDeleted", "hljs-deletion"}, {"GenericInserted", "hljs-addition"}, {"GenericHeading", "hljs-section"}, {"GenericSubheading", "hljs-section"}, {"GenericEmph", "hljs-emphasis"}, {"GenericStrong", "hljs-strong"}, {"GenericPrompt", "hljs-meta prompt_"}]

Pygments/Chroma token-type prefixes mapped to Highlight.js classes, checked in order — first matching prefix wins, so more specific prefixes must precede their generic parent (KeywordType before Keyword).

TOKEN_CLASSES = begin map = {} of String => String | ::Nil Tartrazine::Abbreviations.each_key do |token_type| map[token_type] = resolve_class(token_type) end map end

Precomputed token-type → hljs class for every known token type. Built once at program start from Tartrazine's own type list, so lookups during parallel rendering are read-only and fiber-safe.

TOKENIZE_SLOTS = 4

Tokenization runs in parallel across -Dpreview_mt workers. It used to be serialized behind a global mutex here: the shard's compiled rules each held a single PCRE2 match_data buffer reused by every match() call and shared across lexer instances via the template cache, so two workers tokenizing the same language corrupted each other's matches (raising intermittently, which the rescue below degraded to nondeterministic plain output). That shared state — and the template cache's unsynchronized fast path, and combined actions mutating the shared states Hash — is fixed at the source in ext/tartrazine_mt_fix.cr, so correctness no longer needs a lock around Tartrazine work.

Concurrency is still BOUNDED, not unlimited: tokenization is allocation-dense (token values, per-rule arrays), and past ~4 simultaneous tokenizers the Boehm GC's global allocation lock becomes a convoy that slows the WHOLE build (measured: a 5k-page site with 10k unique code blocks at 8 workers built ~25% slower fully unbounded than with this cap, while small and mid-size sites kept their full parallel speedup). A buffered channel acts as a counting semaphore; blocked fibers suspend — the worker thread moves on to other pages meanwhile.

Instance methods

highlight(code : String, lang : String) : String | Nil

Highlight code as lang, returning HTML-escaped markup with hljs-class spans — or nil when no lexer exists for the language (callers fall back to plain client-style output).

Source