vug.cr

Favicon fetching library with pluggable storage callbacks.

Features

• Fetch favicons from direct URLs
• Extract favicon URLs from HTML pages (<link rel="icon">, <link rel="apple-touch-icon">, etc.)
• Web App Manifest parsing for icon extraction
• Multiple favicon collection with intelligent best/largest/by-size selection
• DuckDuckGo and Google favicon service fallbacks
• Inline base64 data URL support (data:image/png;base64,...)
• SVG placeholder generation when no real favicon is found
• In-memory caching with TTL and size limits
• SSRF protection with DNS rebinding detection
• Pluggable storage via callbacks (disk, S3, database, memory — your choice)
• Coordinated caching (in-memory + config-backed) to prefer existing on-disk stores
• Image processing abstraction with a default processor for validation and saving
• Safer redirect handling with stricter URL validation to prevent unsafe redirects

Installation

Add to your shard.yml:

dependencies:
  vug:
    github: kritoke/vug.cr

Quick Start

require "vug"

# Minimal — no config needed
result = Vug.site("https://example.com")
if result.success?
  puts result.local_path
end

# Feed URLs are auto-detected — works with /atom.xml, /rss.xml, /feed/, etc.
result = Vug.site("https://jvns.ca/atom.xml")

With Storage Callbacks

require "vug"

config = Vug::Config.new(
  on_save: ->(url : String, data : Bytes, content_type : String) {
    path = "/tmp/favicons/#{Digest::SHA256.hexdigest(url)}.png"
    File.write(path, data)
    path  # return the saved path
  },
  on_load: ->(url : String) {
    path = "/tmp/favicons/#{Digest::SHA256.hexdigest(url)}.png"
    File.exists?(path) ? path : nil  # return path or nil
  },
  on_debug: ->(msg : String) { Log.debug { msg } },
)

# Fetch favicon for a website (tries HTML, manifest, fallbacks, placeholder)
result = Vug.site("https://example.com", config)
if result.success?
  puts "Saved to: #{result.local_path}"
elsif result.failure?
  puts "Error: #{result.error}"
end

# Fetch from a direct URL
result = Vug.fetch("https://example.com/favicon.ico", config)

# Get all available favicons for custom selection
collection = Vug.favicons("https://example.com", config)
if collection
  best = collection.best       # highest quality available
  largest = collection.largest # biggest pixel area
  puts "Found #{collection.size} favicons"
end

# Generate a placeholder SVG (first letter of domain)
result = Vug.placeholder("https://example.com", config)

Caching

Pass a MemoryCache to share cache across calls:

cache = Vug::MemoryCache.new(size_limit: 5_000_000, entry_ttl: 1.hour)

result1 = Vug.site("https://example.com", config, cache) # fetches
result2 = Vug.site("https://example.com", config, cache) # cache hit

Configuration

All options have sensible defaults. Override only what you need:

config = Vug::Config.new(
  timeout: 15.seconds,
  max_redirects: 5,
  max_size: 200 * 1024,           # 200KB max favicon size
  user_agent: "MyApp/1.0",
)

See API.md for the full configuration reference and advanced usage.

What's new in v0.5.8

• Crystal 1.19.1 required — the library now uses the new Time::Instant API introduced in Crystal 1.19. If you're on 1.18.x, please upgrade Crystal before pulling this version.
• Faster failure on bad redirects — unparseable redirect URLs now return immediately with an invalid_redirect error instead of waiting for the fetch timeout. Useful when scraping sites with broken redirect chains.

What's new in v0.5.1

• Fixed crash on SSL errors (e.g., SSL_read failures) — OpenSSL::SSL::Error is now caught in all HTTP request handlers.

What's new in v0.5.0

• Feed URL auto-detection — pass /atom.xml, /rss.xml, /feed/, etc. directly and vug resolves to the site root for HTML extraction.
• Vug.best now uses the same fallback chain as Vug.site (standard paths, DuckDuckGo, Google).

What's new in v0.4.0

• Coordinated caching that prefers your configured on-disk storage while keeping a small in-memory cache for speed.
• Image processing is pluggable — you can provide a custom processor to validate or transform images before they are saved.
• Improved redirect handling to prevent unsafe redirects.

Examples

Custom ImageProcessor

class MyProcessor < Vug::ImageProcessor
  def initialize(config)
    super(config)
  end

  def process_bytes(url, data, content_type)
    # custom validation or transform
    path = "/tmp/#{Digest::SHA256.hexdigest(url)}.png"
    File.write(path, data)
    Vug.success(url, path, content_type, data)
  end
end

# Use it with the Fetcher
fetcher = Vug::Fetcher.new(Vug::Config.default, nil, nil, nil, nil, nil, MyProcessor.new)
result = fetcher.fetch("https://example.com/favicon.ico")

Custom RedirectHandler

class AllowAllRedirects < Vug::RedirectHandler
  def decide(original, redirect_url, redirect_count)
    Vug::FetchAction::Follow.new(redirect_url)
  end
end

fetcher = Vug::Fetcher.new(Vug::Config.default, nil, nil, nil, AllowAllRedirects.new)

License

MIT