class

ML::LLM::Context

Inherits Reference / Object

Inference context - manages KV cache and generation state

Constructors

new(model : Model, n_ctx : Int32 = 0, n_batch : Int32 = 512, n_ubatch : Int32 = 0, n_threads : Int32 = 0, flash_attn : Bool = true, embeddings : Bool = false)
Source

Instance methods

clear_memory

Clear KV cache + recurrent state (decoder/hybrid models only, NOT BERT)

Source
encode(tokens : Array(Int32)) : Bool

Encode tokens for BERT/encoder models (uses llama_encode instead of llama_decode)

Source
eval(tokens : Array(Int32)) : Bool

Process prompt tokens (prefill)

Source
finalize
Source
free
Source
get_embeddings

Get embeddings (requires embeddings=true in context)

Source
get_logits

Get logits for last token

Source
get_seq_embeddings(seq_id : Int32) : Slice(Float32)

Get sequence embeddings (for BERT/encoder models with pooling)

Source
model
Source
n_batch
Source
n_ctx
Source
perf_stats

Get performance stats

Source
position

Current position in context

Source
remaining_ctx

Remaining context space

Source
reset

Reset position + sampler (safe for all context types including BERT)

Source
reset_perf

Reset performance counters

Source
sample

Sample next token

Source
setup_greedy_sampler

Greedy sampler (deterministic)

Source
setup_sampler(temperature : Float32 = 0.8_f32, top_k : Int32 = 40, top_p : Float32 = 0.95_f32, min_p : Float32 = 0.05_f32, seed : UInt32 = LlamaFFI::LLAMA_DEFAULT_SEED)

Setup sampler chain with default parameters

Source