module

LinkFollowability

Classifies links the way search engines read them: every rel token on every anchor, plus the page-level <meta name="robots"> and X-Robots-Tag header.

Google treats nofollow, ugc and sponsored as hints that a link should not be followed, and more than one of them can sit on the same anchor. noopener and noreferrer are browser-security tokens and do not change followability.

Constants

NOT_FOLLOWED_TOKENS = {"nofollow", "ugc", "sponsored"}

rel tokens that mark a link as not followed.

VERSION = "0.1.0"

Class methods

anchors(html : String) : Array(Anchor)

Extracts every <a> start tag with an href, in document order. Handles double-quoted, single-quoted and unquoted attribute values. Anchors without an href are skipped. This is a tokenizer for start tags, not a full HTML parser: comments and script bodies are removed first so that markup inside them is not counted.

Source
followed?(anchor : Anchor, page : PageDirectives) : Bool

Whether a given anchor is followed once page-level directives are applied.

Source
merge(*sets : PageDirectives) : PageDirectives

Merges several directive sets; any noindex/nofollow wins.

Source
meta_robots(html : String) : String | Nil

Returns the content of the first <meta name="robots"> tag, or nil.

Source
parse_directives(value : String | Nil) : PageDirectives

Parses a robots directive list such as "noindex, nofollow" or "none". none is equivalent to noindex, nofollow; all sets neither.

Source
parse_x_robots_tag(value : String | Nil, user_agent : String = "googlebot") : PageDirectives

Parses one X-Robots-Tag header value. A value may be scoped to a crawler ("googlebot: noindex"); scoped values only apply when user_agent matches (case-insensitive). Unscoped values apply to every crawler. Directives that carry their own value after a colon (e.g. "unavailable_after: ...") are not mistaken for a crawler scope.

Source
rel_tokens(rel : String | Nil) : Array(String)

Splits a rel attribute into lower-case tokens. Tokens are separated by ASCII whitespace (HTML spec); duplicates are kept once, order is preserved.

Source

Nested types