Hwaro::Utils::TextUtils

Elements the author marked aria-hidden="true" are decoration, not text: a screen reader skips them and so must every plain-text projection of the page. Two live in rendered content and both leaked into the search index and feed summaries — the heading anchor link (insert_anchor_links, a 🔗 after every heading) and the code-block line-number gutter ([highlight] line_numbers, which indexed 1 2 3 4 … alongside the code). Same lazy [\s\S]*? shape as RAW_TEXT_ELEMENT: an unterminated or same-tag-nested element just falls through to the tag stripper, which is harmless here because the result is text either way.

Markup the Markdown extensions emit whose text is not prose either: display math (TeX source until KaTeX/MathJax runs in the browser), mermaid diagrams (a code fence, like <pre>), footnote reference markers ([1]) and the trailing footnotes section. Inline math (math-inline) is part of the sentence and keeps its \(x\) source.

Elements whose CONTENT is not prose and must not seed an automatic summary: code blocks (<pre>), raw-text elements, figures/images (captions and alt text read badly out of context) and headings (a summary that opens with the page's own title is noise). Inline <code> is part of the sentence and stays. Matched with a lazy [\s\S]*? so multi-line blocks are removed whole; the <code> inside <pre> is consumed by the outer match. Unterminated tags fall through to the generic tag stripper.

Trailing characters dropped before the ellipsis so a cut never ends in word,… / word;…. Covers ASCII and the full-width CJK forms.

The attribute run of an opening tag: anything but >, except that a quoted value may contain > (alt="Home > Docs"). Possessive all the way out, so neither a long data: URI value nor a long run of attributes costs a PCRE JIT stack frame per step. A user follows it with the tag's > (or \/?>), which it only reaches at its own end, so giving nothing back changes no match.

Like HTML_TAG_ATTRS, but a quote opens a value only right after =, as in HTML: a stray quote (alt=Bob's, alt="55" TV") is plain text, so the tag still ends at its > exactly as a [^>]* scan did.

Display names for common language codes, used by scaffold configs for --include-multilingual. Unknown codes fall back to upcase.

Longest error message any console emitter prints on one line.

A template error carries Crinja's source excerpt, so a single minified/one-line template turns one failure into a multi-megabyte console line: a 3 MB {% if %} line produced a 6 MB log, and in serve mode every rebuild repeated it. The clamp has to be applied by each emitter (rather than inside Logger) because the machine-readable contracts hwaro prints elsewhere are byte-preserved.

Widest padding run pad_display will emit. Far past any real terminal, so alignment is unaffected; it only bounds a nonsense width.

Longest slug safe_slugify returns, in bytes.

A taxonomy term becomes one path segment (public/tags/<slug>/), and nothing bounded it: pasting a phrase into tags produced a 300-byte directory name and the build died at the first mkdir with ENAMETOOLONG. The two filesystem limits that matter disagree on their UNIT — ext4 caps a name at 255 bytes, APFS at 255 characters — which is why a 264-byte / 88-character Korean tag builds on macOS but not on Linux. A byte bound satisfies both, because a string's character count is never greater than its byte count.

The bound is set as close to 255 as the -N disambiguation suffix disambiguated_slugs may append allows (10 bytes of headroom, enough for -999999999), deliberately NOT lower: a slug of 201..245 bytes publishes fine on every mainstream filesystem, so a tighter cap would silently move the URL of a term that works today — page.url, the sitemap <loc>, the feeds and every get_taxonomy_url link with it — and 404 anything already indexed. Only terms that could not have been written at all are rewritten.

Raw-text HTML elements whose content is code, not display text. <style>/<script> bodies must be dropped along with their tags; otherwise the CSS/JS source survives tag-stripping and pollutes search indexes, feed summaries, and excerpts (a page with an inline <style> gallery block had its whole search entry replaced by CSS). [\s\S] matches across newlines (CSS/JS span multiple lines) without relying on a dotall flag; \1 ties the close tag to the open tag. A self-closing or unterminated tag won't match and is left to the tag stripper below.

Emoji modifiers (Fitzpatrick skin tones). They are category So, not a combining mark, but they recolour the preceding emoji rather than adding a cell: 👍🏽 renders in two columns, not four.

Hex characters of the term digest appended to a truncated slug — enough that two terms sharing a long prefix keep separate URLs.

Codepoint ranges that occupy two terminal columns: East Asian Wide (W) and Fullwidth (F) per UAX #11, plus the emoji blocks terminals render double-width. Ascending and non-overlapping, so wide_char? can stop as soon as a range starts past the codepoint. Anything outside is one column.

Zero-width joiner. 👨‍👩‍👧 is three wide emoji plus two joiners but renders as a single two-column glyph, so a joiner makes the codepoint that follows it contribute nothing.

Other generated file names (OG images) bound theirs with a tighter max_bytes that leaves room for their extension.

Check if a character is in a CJK Unicode range

CJK-dominant: more than half of the non-whitespace characters are CJK (same ranges tokenize_cjk/slugify use).

Count words the way the build does.

Single pass: skip HTML tags, and treat Markdown punctuation as a separator rather than a word so ## Heading counts one word and a table row's pipes count none. This is the single source of truth behind both page.word_count / page.reading_time and hwaro tool stats, which previously split on whitespace alone and reported ~30% more words than the site itself rendered.

Chinese and Japanese are written without spaces, so each Han, Hiragana or Katakana character counts as one word (Hangul keeps space-based counting: Korean separates words with spaces). Any Unicode whitespace (NBSP, U+3000 ideographic space) splits words.

{words, cjk}: the count_words total, and how many of those words are single Han/Kana characters (reading time weighs them apart).

Map a set of terms to UNIQUE slugs. Distinct terms can slugify to the same value ("C++"/"C#" → "c", "Hello World"/"hello-world" → "hello-world"); on a clash the later term (in sorted order) gets a numeric suffix, and the candidate is re-checked so it never collides with another generated slug or a real term whose base slug already ends in "-2". Sorting makes the result deterministic across builds regardless of input order.

This is the single source of truth for taxonomy term slugs: the taxonomy generator (term-page paths + index links) and the get_taxonomy / get_taxonomy_url template helpers must all run terms through here so the links they emit point at the pages that were actually written.

Bases arrive already capped by safe_slugify; the few bytes a -N suffix adds on top are what MAX_SLUG_BYTES' headroom under 255 is for.

How many terminal columns a string occupies.

String#size counts codepoints, which is the wrong unit for padding a column: a Hangul or CJK title renders twice as wide as its codepoint count, a combining mark renders in the previous cell, and a control byte renders as nothing — so ljust left every table row after a non-ASCII cell visibly misaligned.

Printable ASCII returns exactly size, so ASCII-only output — every existing table — is unchanged. A TAB is deliberately counted as one column (what ljust charged it) rather than zero: its real width depends on the terminal's tab stops, and one keeps the "ASCII measures like String#size" invariant total rather than only-for-printable.

Percent-encode the path component of a URL for spec-strict XML outputs (sitemap <loc>, RSS/Atom <link>/<id>): the sitemap protocol and RSS require RFC 3986 URIs, so non-ASCII paths like /posts/한글/ must become /posts/%ED%95%9C%EA%B8%80/.

The scheme/host prefix (if any) is left untouched, and escapes that are already present are stepped over so pre-encoded URLs don't get double-encoded.

encode_url_path for a URL that may carry a ?query or #fragment (an author-written image path such as /img/사진.png?v=2): only the part before the first ?/# is encoded, the rest is kept as written. encode_url_path alone escaped the delimiters themselves (…png%3Fv%3D2, a 404).

Escape XML special characters

Escapes: & < > " '

Example: escape_xml("Tom & Jerry") # => "Tom &amp; Jerry" escape_xml("") # => "<script>"

Plain prose of a rendered HTML body, for the automatic summary: code-block/figure/heading/display-math/footnote blocks dropped, tags stripped, entities decoded (after stripping, so <p> can never be read as markup), and whitespace collapsed to single spaces. The result is TEXT — a consumer that embeds it in HTML must escape it again.

Rendered HTML as plain text: tags stripped, entities decoded, edges trimmed. The one spelling for excerpts and feed descriptions.

A taxonomy term as the build identifies it: stripped and NFC-normalised. macOS filenames, IMEs and the clipboard produce NFD text, and slugify drops the combining marks of an NFD \u00E9 (so /tags/cafe/ next to /tags/caf\u00E9/) while NFD and NFC Hangul share one directory on APFS \u2014 visually identical tags must be one term. NFC is the identity for ASCII.

Left-align s in a field width terminal columns across. The display-width counterpart of String#ljust; identical to it for ASCII-only input.

Like slugify but never returns "" and never exceeds MAX_SLUG_BYTES. An all-symbol/emoji input (e.g. a tag of "!!!" or "🎉") slugifies to "", which would make distinct terms collide onto the same URL/output path and create a // path segment. Falls back to a deterministic, stable token derived from the input's UTF-8 bytes so distinct inputs stay distinct and the slug is identical across builds (unlike String#hash, which is per-process seeded).

Cap slug at MAX_SLUG_BYTES, keeping distinct terms on distinct URLs.

Truncating alone would fold every term sharing a long prefix onto one slug — two tags silently rendering as one page — so the kept head is followed by a digest of the WHOLE source term. SHA-1 is an identifier here, not a security primitive; what it buys is determinism across runs, processes and platforms, which String#hash (per-process seeded) does not give. That matters because this is the single source of truth for term slugs: bounding here means the on-disk path, page.url, the sitemap, the feeds and the get_taxonomy_url helper all shorten together. Bounding at the filesystem layer instead would leave every generated link pointing at the un-truncated path — a site of dead links, worse than the crash.

Slugs at or under the bound — every term on every existing site — are returned untouched, so no published URL moves. True when safe_slugify(term) had to shorten the term to fit MAX_SLUG_BYTES, i.e. its published URL is not derived from the whole term. safe_slugify itself is called many times per term (index links, term pages, template helpers), so it must stay silent; callers that enumerate the term set ONCE per build use this to warn exactly once.

Convert text to a URL-friendly slug

Supports Unicode characters (CJK, Hangul, etc.) in addition to ASCII.

Examples: slugify("Hello World!") # => "hello-world" slugify("My Blog Post") # => "my-blog-post" slugify("한글 제목") # => "한글-제목" slugify("CJK 테스트!") # => "cjk-테스트" slugify("日本語　テスト") # => "日本語-テスト" (U+3000) slugify("CI/CD") # => "ci-cd" slugify("보안—우회") # => "보안-우회" (em dash)

Character ranges of each candidate URL in a srcset value.

Strip a leading UTF-8 byte order mark.

Windows editors (Notepad, PowerShell > redirection, "UTF-8 with BOM" in VS Code) prepend U+FEFF. Every parser hwaro hands file text to anchors on the first character — the front matter fences (\A\+\+\+, \A---, a leading {), TOML's first token, JSON's first token — so a BOM silently turns front matter into body text and makes config.toml and data files unparseable. Strip it once at the point of read.

Remove terminal control characters (ANSI escapes, CR, BEL, \u2026) plus the Unicode line/paragraph separators.

Titles, tags and link targets come from semi-trusted content \u2014 a docs or blog PR \u2014 and end up printed to a maintainer's terminal by the tool reports. Raw escape bytes there can repaint or spoof the console, and they wreck column alignment even when benign.

Scope is deliberately Cc only. Char#control? is ALSO true for the Cf format category, and stripping that mangles real words: U+200C ZWNJ turns Persian \u0645\u06cc\u200c\u0631\u0648\u062f into the different word \u0645\u06cc\u0631\u0648\u062f, U+200D ZWJ splits \ud83d\udc68\u200d\ud83d\udc69\u200d\ud83d\udc67 into three separate people, and dropping U+200F RLM flips where punctuation lands in a Hebrew line. Those codepoints are invisible, not dangerous \u2014 they carry no cursor movement \u2014 so they stay. Apply this at the RENDER layer only; a model or JSON payload must keep the author's bytes intact.

Strip HTML tags from text (single-pass)

Example: strip_html("

Hello World

Tokenize CJK text into overlapping bigrams for search indexing

CJK languages (Chinese, Japanese, Korean) often lack spaces between words. This splits CJK character runs into overlapping 2-character pairs (bigrams) so search libraries can match substrings.

Example: tokenize_cjk("검색엔진") # => "검색 색엔 엔진" tokenize_cjk("hello世界测试") # => "hello世界 界测 测试"

Clamp message to max CHARACTERS (not bytes — the count is what the suffix reports, and slicing on a character boundary can never split a codepoint in the terminal). Returns the message untouched when it fits, which is every ordinary error.

Truncate plain text for an automatic summary. Returns the excerpt and whether anything was cut.

Units: space-delimited text is measured in WORDS. Text whose non-space characters are mostly CJK (Chinese/Japanese have no word boundaries; Korean has them but sentences pack far more meaning per word) is measured in CHARACTERS with the same numeric setting scaled ×2, so length = 70 reads as ~70 words or ~140 CJK characters. The character cut is never placed inside a Latin word embedded in CJK text (it backs up to the preceding boundary), and every index is a Char index, so a multibyte sequence cannot be split. Entities are not a concern: callers pass decoded text. length <= 0 means "no limit" — callers gate the feature on it.

A combining mark that is part of a word (vowel signs, virama, tone and accent marks). Emoji presentation machinery is not: variation selectors and the keycap enclosure stay out of slugs, so "1️⃣ Step" and "ℹ️ Info" slugify as "1-step" and "ℹ-info" as before.