Smith::Web::HtmlToMarkdown
Turns a fetched page into something worth putting in a context window.
Deliberately not a parser: Crystal has no established HTML-to-Markdown shard, and a full one is not the goal here. Documentation pages — headings, lists, links, code — come out readable, and that is the bar. Anything relying on precise nesting or on tables will come out flattened.
Constants
DROPPED = ["script", "style", "head", "nav", "footer", "aside", "noscript", "svg", "iframe", "template"] of ::String
Everything inside these carries no content worth reading, so they go before any other processing — script and style especially, whose bodies would otherwise survive as text.
ENTITIES = {" " => " ", "&" => "&", "<" => "<", ">" => ">", """ => "\"", "'" => "'", "'" => "'", "—" => "—", "–" => "–", "…" => "…", "©" => "©", "®" => "®"}
Class methods
convert(html : String) : String
Source