module

Deadfinder::HttpClient

Constants

CROSS_ORIGIN_SENSITIVE_HEADERS = ["Authorization", "Cookie", "Proxy-Authorization"]

Headers that must not be replayed to a different origin after a redirect. Mirrors curl's default behaviour: following a redirect off-host with a user-supplied Authorization/Cookie would leak the credential.

HEAD_UNTRUSTED_STATUS = 400

In auto mode a HEAD response at or above this status is not trusted and is confirmed with a GET before it can be reported.

MAX_REDIRECTS = 5

Redirect hops followed when fetching a document (a scan target page or a sitemap). Link status checks deliberately pass 0: there the 30x status is the answer being reported (see --include30x).

METHOD_AUTO = "auto"

--method values. These select the HTTP method used for a link status check; documents (the scan target page, a sitemap) are always fetched with GET because their body is what the caller wants.

auto (default) — send a HEAD first, and re-issue the check as a GET whenever that HEAD answer cannot be trusted. "Cannot be trusted" means any status >= 400 (which covers the 405/501 "method not supported" pair, the CDNs and app servers that answer HEAD with a 403/404/500 while GET is fine, and bot-defense codes such as LinkedIn's 999) as well as any transport-level failure of the HEAD itself. A link is therefore never reported dead on the strength of a HEAD alone — a GET always confirms it. head — HEAD only. Cheapest, but takes the server's HEAD answer at face value, so it can produce false positives. get — GET only. The pre-2.1 behaviour: always downloads the body just to read the status line.

METHOD_GET = "get"
METHOD_HEAD = "head"
METHODS = [METHOD_AUTO, METHOD_HEAD, METHOD_GET]
UNREACHABLE_ERRORS = {Socket::ConnectError, Socket::Addrinfo::Error, IO::TimeoutError}

Transport failures that mean the host was never reached at all. A GET cannot succeed where the TCP connect timed out, was refused, or the name did not resolve, so auto must not spend a second full timeout proving it — that turned one connect timeout per unreachable link into four (HEAD + GET on the first attempt, then one GET per retry). Any other failure (a reset mid-response, a read timeout after the connection was established) still earns a GET: those can be HEAD-specific.

Class methods

absolute_uri(uri : URI) : String

For HTTP proxy, requests need to use absolute URI as path

Source
build_headers(raw : Array(String), user_agent : String) : HTTP::Headers

Parse "Name: value" header strings. Accepts ":" or ": " as the separator and trims both sides — every request the tool makes (target page, sitemap document, link check) builds its headers here so users don't hit depending-on-which-flag surprises.

Source
check(uri : URI, options : Options, headers : HTTP::Headers, force_get : Bool = false) : HTTP::Client::Response

Runs a single link status check against uri, honoring --method. Redirects are never followed here: the 30x status is itself the reported result (--include30x), which is why every call passes 0 hops.

force_get is set by the retry loop once an attempt for this URL has already failed at the transport level: the HEAD probe has nothing left to prove there, and re-running it would make every retry of an unreachable host pay two connect timeouts instead of one.

Source
close_idle_connections

Drops every pooled connection. Only needed when a process performs several independent runs (specs, embedded usage); a one-shot CLI run just exits.

Source
create(uri : URI, options : Options) : HTTP::Client
Source
decompress_if_gzip(body : String) : String

Sitemaps are routinely published gzipped (sitemap.xml.gz). When the file is served as an opaque application/gzip body there is no Content-Encoding for HTTP::Client to transparently undo, so detect the gzip magic bytes and inflate here. Returns the body unchanged when it is not gzip, or when it cannot be inflated.

Source
fetch(uri : URI, options : Options, headers : HTTP::Headers, max_redirects : Int32 = 0, method : String = "GET") : Tuple(HTTP::Client::Response, URI)

Fetches uri with method and returns the response together with the URI it was finally served from. max_redirects hops of 30x Location are followed; with the default of 0 the redirect response itself is returned untouched.

Callers need the final URI because relative links (and relative sitemap <loc> entries) must resolve against the post-redirect location, not against the address the user originally typed.

Source
monotonic_now
Source
origin_key(uri : URI) : String

Origin identity used for both throttling and pooling: two URLs share a connection only when scheme, host and effective port all match.

Source
parse_retry_after(value : String | Nil) : Time::Span | Nil

Parses a Retry-After header in either of its two RFC 9110 forms — delta-seconds ("120") or an HTTP-date ("Wed, 21 Oct 2015 07:28:00 GMT"). Returns nil when the header is absent or unparseable, and clamps a past date to zero so a stale header cannot produce a negative sleep.

Source
pool
Source
proxy_configured?(options : Options) : Bool
Source
request_path(uri : URI, options : Options) : String

Request-line target for uri: an absolute URI when talking plain HTTP to a forward proxy, an origin-form path otherwise.

Source
throttle
Source

Nested types