class

Robots::Parser

Inherits Reference < Object

Constructors

new(base_url : String, user_agent : String)
Source

Instance methods

allowed?(uri)

Given a URI object, or a string representing one, determine whether this robots.txt would allow access to the path.

Source
base_url
Source
body
Source
rules
Source
sitemaps
Source
user_agent
Source

Nested types