class

Llama::Vocab

Inherits Reference / Object

Wrapper for the llama_vocab structure

Constructors

new(handle : Pointer(LibLlama::LlamaVocab))

Creates a new Vocab instance from a raw pointer

Note: This constructor is intended for internal use. Users should obtain Vocab instances through Model#vocab.

Source

Instance methods

add_bos?

Returns whether the model adds BOS token by default

Source
add_eos?

Returns whether the model adds EOS token by default

Source
add_sep?

Returns whether the model adds SEP token by default

Source
bos

Returns the beginning-of-sentence token ID

Source
control?(token : Int32) : Bool

Checks if a token is a control token

Source
eog?(token : Int32) : Bool

Checks if a token is an end-of-generation token

Source
eos

Returns the end-of-sentence token ID

Source
eot

Returns the end-of-turn token ID

Source
format_token(token : Int32, show_id : Bool = true, show_text : Bool = true) : String

Format a token for display

Parameters:

  • token: The token to format
  • show_id: Whether to show the token ID
  • show_text: Whether to show the token text

Returns:

  • A formatted string representation of the token
Source
mask

Returns the mask token ID (if defined by the tokenizer)

Source
n_tokens

Returns the number of tokens in the vocabulary

Source
nl

Returns the newline token ID

Source
pad

Returns the padding token ID

Source
to_unsafe

Returns the raw pointer to the underlying llama_vocab structure

Source
token_to_piece(token : Int32, lstrip : Int32 = 0, special : Bool = false) : String

Converts a token to a piece of text This is similar to token_to_text but provides more control over the output format

Parameters:

  • token: The token to convert
  • lstrip: Whether to strip leading spaces (0 = no, 1 = yes)
  • special: Whether to render special tokens

Returns:

  • The text representation of the token
Source
token_to_text(token : Int32) : String

Returns the text representation of a token

Source
tokenize(text : String, add_special : Bool = true, parse_special : Bool = true) : Array(Int32)

Tokenizes a string into an array of token IDs

Source