class

Waypoints::HTTPPageFetcher

Inherits Waypoints::PageFetcher < Reference < Object

Fetches pages with the Crystal stdlib HTTP client: 10s timeouts, an http->https upgrade, a small redirect follow, and <title>/body extraction.

Constants

MAX_REDIRECTS = 5
MAX_TEXT = 4000

Constructors

new(timeout : Time::Span = 10.seconds)

Builds a fetcher with a per-request connect/read timeout.

Source

Instance methods

fetch(url : String) : PageContent

Fetches url, following a few redirects, and extracts its title and text.

Source