module

Crawler

Constants

GH_API = "https://api.github.com"
RAW = "https://raw.githubusercontent.com"

Class methods

build_shard(item : JSON::Any, full_name : String, meta) : Shard
Source
collect_repos(gh : Github, limit : Int32) : Array(JSON::Any)

GitHub search caps results at 1000 per query, so enumerate by creation-date windows, splitting any window that overflows. CRAWL_FROM bounds the range (handy for quick test crawls).

Source
crawl_details(gh : Github, repos : Array(JSON::Any), cache : Hash(String, Shard), concurrency : Int32) : Array(Shard)

For every repo: reuse the cached record when pushed_at is unchanged, otherwise refetch shard.yml. Repos without shard.yml are dropped.

Source
fetch_shard_yml(full_name : String) : String | Nil
Source
generate_site(site : SiteData, public_dir : String)
Source
parse_shard_yml(raw : String)
Source

Nested types