module

DomainDB

Constants

Log = ::Log.for("DomainDB")

SUFFIX_URL = "https://publicsuffix.org/list/public_suffix_list.dat"

the url for the mozilla public suffixes list

TLD_URL = "https://www.iana.org/domains/root/db"

the URL for the IANA TLD extensions list

Class methods

backoff_factor

see #update_tlds or #update_suffixes

backoff_factor=(backoff_factor : Float64)

see #update_tlds or #update_suffixes

backoff_time

see #update_tlds or #update_suffixes

backoff_time=(backoff_time : Time::Span)

see #update_tlds or #update_suffixes

retry_count

see #update_tlds or #update_suffixes

retry_count=(retry_count : Int32)

see #update_tlds or #update_suffixes

strip_subdomains(hostname : String, tld_only = false) : String

extracts the domain name from hostname using the public suffixes database to identify the portion of hostname that is a public suffix. The next token to the left is returned (with its suffix) as the domain name. This effectively strips subdomains from an arbitrary domain name. If tld_only is set to true, only top-level domains according to IANA will be used (meaning "co.uk" would be the detected domain for hostanmes like site.co.uk). By default the mozilla public suffixes database is used.

strip_suffix(hostname : String, tld_only = false) : String

removes the domain extension / suffix from the end of the specified hostname. Follows the same options and semantics as #strip_subdomains.

suffixes

contains the public suffix database from mozilla as a set of registerable domain extensions (com, com.mx, etc.). This set is a super set of #tld_extensions and all registerable domain names.

#update_suffixes must be called before this set will be populated

tld_extensions

contains the TLD database as a set of top level domain extensions (com, net, etc.)

#update_tlds must be called before this set will be populated

update_suffixes(retry_count : Int32 = self.retry_count, backoff_time : Time::Span = self.backoff_time, backoff_factor : Float64 = self.backoff_factor)

updates the mozilla public suffixes database by downloading and parsing data from #SUFFIX_URL. Upon a failure (non-200 status code) exponential backoff will be used until retry_count is reached.

arguments: retry_count (optional): specifies maximum number of retries before raising backoff_time (optional): initial amount of time we should wait before trying again upon a failure backoff_factor (optional): backoff_time is multiplied by this factor on each failure. Should be greater than 1

update_tlds(retry_count : Int32 = self.retry_count, backoff_time : Time::Span = self.backoff_time, backoff_factor : Float64 = self.backoff_factor)

updates the tld extensions database by downloading and parsing html from #TLD_URL. Upon a failure (non-200 status code) exponential backoff will be used until retry_count is reached.

arguments: retry_count (optional): specifies maximum number of retries before raising backoff_time (optional): initial amount of time we should wait before trying again upon a failure backoff_factor (optional): backoff_time is multiplied by this factor on each failure. Should be greater than 1