class

Semtrace::Tokenizer

Inherits Reference < Object

GPT-2 BPE tokenizer.

Splits text into tokens matching GPT-2's vocabulary using byte-pair encoding.

Constructors

new(vocab_path : String, merges_path : String)

Instance methods

encode(text : String) : Array(Int32)

Tokenizes text into token IDs.

tokenize(text : String) : Array(String)

Tokenizes text into token strings (BPE-encoded form).