module

Semtrace::Prepare

Downloads GPT-2 weights from HuggingFace and extracts the embedding matrix.

Constants

GPT2_MODELS = {"gpt2" => {repo: "openai-community/gpt2", label: "GPT-2 Small (768d)"}, "gpt2-medium" => {repo: "openai-community/gpt2-medium", label: "GPT-2 Medium (1024d)"}, "gpt2-large" => {repo: "openai-community/gpt2-large", label: "GPT-2 Large (1280d)"}, "gpt2-xl" => {repo: "openai-community/gpt2-xl", label: "GPT-2 XL (1600d)"}}

HF_BASE = "https://huggingface.co/openai-community/gpt2/resolve/main"

MERGES_URL = "#{HF_BASE}/merges.txt"

SAFETENSORS_URL = "#{HF_BASE}/model.safetensors"

VOCAB_URL = "#{HF_BASE}/vocab.json"

Class methods

download(url : String, dest : String)

Downloads a URL to a local file, following redirects. Shows progress for large files.

Source

run(data_dir : String, model : String = "gpt2")

Downloads GPT-2 files and extracts the embedding matrix to our binary format.

Source