Hwaro::Utils::TextUtils

Raw-text HTML elements whose content is code, not display text. <style>/<script> bodies must be dropped along with their tags; otherwise the CSS/JS source survives tag-stripping and pollutes search indexes, feed summaries, and excerpts (a page with an inline <style> gallery block had its whole search entry replaced by CSS). [\s\S] matches across newlines (CSS/JS span multiple lines) without relying on a dotall flag; \1 ties the close tag to the open tag. A self-closing or unterminated tag won't match and is left to the tag stripper below.

Check if a character is in a CJK Unicode range

Map a set of terms to UNIQUE slugs. Distinct terms can slugify to the same value ("C++"/"C#" → "c", "Hello World"/"hello-world" → "hello-world"); on a clash the later term (in sorted order) gets a numeric suffix, and the candidate is re-checked so it never collides with another generated slug or a real term whose base slug already ends in "-2". Sorting makes the result deterministic across builds regardless of input order.

This is the single source of truth for taxonomy term slugs: the taxonomy generator (term-page paths + index links) and the get_taxonomy / get_taxonomy_url template helpers must all run terms through here so the links they emit point at the pages that were actually written.

Percent-encode the path component of a URL for spec-strict XML outputs (sitemap <loc>, RSS/Atom <link>/<id>): the sitemap protocol and RSS require RFC 3986 URIs, so non-ASCII paths like /posts/한글/ must become /posts/%ED%95%9C%EA%B8%80/.

The scheme/host prefix (if any) is left untouched, and paths that already contain a percent-escape are passed through unchanged so pre-encoded URLs don't get double-encoded.

Escape XML special characters

Escapes: & < > " '

Example: escape_xml("Tom & Jerry") # => "Tom & Jerry" escape_xml("") # => "<script>"

Like slugify but never returns "". An all-symbol/emoji input (e.g. a tag of "!!!" or "🎉") slugifies to "", which would make distinct terms collide onto the same URL/output path and create a // path segment. Falls back to a deterministic, stable token derived from the input's UTF-8 bytes so distinct inputs stay distinct and the slug is identical across builds (unlike String#hash, which is per-process seeded).

Convert text to a URL-friendly slug

Supports Unicode characters (CJK, Hangul, etc.) in addition to ASCII.

Examples: slugify("Hello World!") # => "hello-world" slugify("My Blog Post") # => "my-blog-post" slugify("한글 제목") # => "한글-제목" slugify("CJK 테스트!") # => "cjk-테스트"

Strip HTML tags from text (single-pass)

Example: strip_html("

Hello World

Tokenize CJK text into overlapping bigrams for search indexing

CJK languages (Chinese, Japanese, Korean) often lack spaces between words. This splits CJK character runs into overlapping 2-character pairs (bigrams) so search libraries can match substrings.

Example: tokenize_cjk("검색엔진") # => "검색 색엔 엔진" tokenize_cjk("hello世界测试") # => "hello世界 界测 测试"