class

ML::GGUF::UnigramTokenizer

Inherits Reference / Object

Constructors

new(gguf : GGUFFile)
Source

Instance methods

add_bos
Source
add_eos
Source
add_space_prefix
Source
bos_id
Source
encode(text : String) : Array(Int32)

Tokenize text → array of token IDs

Source
eos_id
Source
pad_id
Source
unk_id
Source
vocab_size
Source