Llama::Context
Wrapper for the llama_context structure
Constructors
Creates a new Context instance for a model.
Parameters:
- model: The Model to create a context for.
- n_ctx: Text context (default: 0). The maximum context size. If 0, a minimum context size of 512 is used.
- n_batch: Logical maximum batch size that can be submitted to llama_decode (default: 512).
- n_threads: Number of threads to use for generation (default: 0). If 0, uses the number of hardware threads.
- n_threads_batch: Number of threads to use for batch processing (default: 0). If 0, uses the number of hardware threads.
- embeddings: Extract embeddings (together with logits) (default: false). If true, extract embeddings (together with logits).
- offload_kqv: Whether to offload the KQV ops (including the KV cache) to GPU (default: false). Requires a GPU build of llama.cpp.
Raises:
- Llama::Context::Error if the context cannot be created.
Instance methods
Applies a control vector to the LoRA adapter
Parameters:
- data: The control vector data
- n_embd: Embedding dimension per layer
- il_start: Start layer index (inclusive, 1-based)
- il_end: End layer index (inclusive, 1-based)
Returns:
- 0 on success, non-zero on error
Raises:
- Llama::Context::Error if the control vector cannot be applied
Applies the chat template to the given messages and returns the formatted prompt.
Parameters:
- messages: Array of ChatMessage (user/assistant/system)
- add_assistant: Whether to add assistant role (default: true)
- template: Optional template string (default: model's template)
Returns:
- The formatted prompt string.
Attaches a LoRA adapter to this context
Parameters:
- adapter: The LoRA adapter to attach
- scale: Scaling factor for the adapter (default: 1.0)
Returns:
- 0 on success, non-zero on error
Raises:
- Llama::Context::Error if the adapter cannot be attached
Generates a response in a chat conversation
Parameters:
- messages: Array of chat messages
- max_tokens: Maximum number of tokens to generate
- temperature: Sampling temperature
- template: Optional chat template (nil to use model's default)
Returns:
- The generated response text
Raises:
- ArgumentError if parameters are invalid
- Llama::Context::Error if text generation fails
- Llama::TokenizationError if the prompt cannot be tokenized
Processes a batch of tokens with the decoder part of the model
Parameters:
- batch: The batch to process (can be a LibLlama::LlamaBatch or a Batch instance)
Returns:
- 0 on success
- 1 if no KV slot was found for the batch
- < 0 on error
Raises:
- Llama::Batch::Error on error
Detaches a LoRA adapter from this context
Parameters:
- adapter: The LoRA adapter to detach
Returns:
- 0 on success, non-zero on error
Raises:
- Llama::Context::Error if the adapter cannot be detached
Gets embeddings for the latest output token.
Internally this uses the index-based embeddings API.
Returns:
- An array of embeddings, or nil if embeddings are not available
Raises:
- Llama::Context::Error if embeddings mode is not enabled
Sets whether the model is in embeddings mode or not If true, embeddings will be returned but logits will not
Parameters:
- enabled: Whether to enable embeddings mode
Processes a batch of tokens with the encoder part of the model
This function is used for encoder-decoder models to encode the input before generating text with the decoder.
Parameters:
- batch: The batch to process (can be a LibLlama::LlamaBatch or a Batch instance)
Returns:
- 0 on success
- < 0 on error
Raises:
- Llama::Batch::Error on error
Generates text from a prompt
Parameters:
- prompt: The input prompt
- max_tokens: Maximum number of tokens to generate (must be positive)
- temperature: Sampling temperature (0.0 = greedy, 1.0 = more random)
Returns:
- The generated text
Raises:
- ArgumentError if parameters are invalid
- Llama::Context::Error if text generation fails
- Llama::TokenizationError if the prompt cannot be tokenized
Generates text using a sampler chain
Parameters:
- prompt: The input prompt
- sampler: The sampler chain to use
- max_tokens: Maximum number of tokens to generate (must be positive)
Returns:
- The generated text
Raises:
- ArgumentError if parameters are invalid
- Llama::Context::Error if text generation fails
- Llama::TokenizationError if the prompt cannot be tokenized
- Llama::Sampler::Error if sampling fails
Gets the embeddings for a specific token
Parameters:
- i: The token index (negative indices can be used to access in reverse order)
Returns:
- An array of embedding values, or nil if not available
Raises:
- Llama::Context::Error if embeddings mode is not enabled
Gets the embeddings for a specific sequence
Parameters:
- seq_id: The sequence ID
Returns:
- An array of embedding values, or nil if not available
Raises:
- Llama::Context::Error if embeddings mode is not enabled
Gets the logits for a specific output index.
Parameters:
- i: Output index (negative indices access from the end; -1 is latest)
Returns:
- A pointer to the logits array for the specified output, or nil if unavailable
Returns the memory for this context (modern API)
The memory system provides unified access to various memory types:
- Standard KV cache (llama_kv_cache_unified)
- SWA (Sliding Window Attention) cache
- Recurrent layer memory
- Hybrid attention/recurrent models
Returns:
- A Memory instance
Gets the pooling type used for embeddings
Returns:
- The pooling type as a PoolingType enum
Print performance information for this context
This method prints performance statistics about the context to STDERR. It's useful for debugging and performance analysis.
Process embeddings
Parameters:
- embeddings: Array of embedding vectors
- seq_ids: Sequence IDs to use for all embeddings
- n_seq_max: Maximum number of sequence IDs per token (default: 8)
Returns:
- The result of the decode operation (0 on success)
Raises:
- Llama::Batch::Error on error
Process multiple prompts in batch
Parameters:
- prompts: Array of text prompts to process
- compute_logits_for_last: Whether to compute logits only for the last token of each prompt
Returns:
- Array of decode operation results (0 on success)
Raises:
- Llama::Batch::Error on error
- Llama::TokenizationError if a prompt cannot be tokenized
Process a sequence of tokens
Parameters:
- tokens: Array of token IDs to process
- compute_logits_for_last: Whether to compute logits only for the last token
- seq_ids: Sequence IDs to use for all tokens
- n_seq_max: Maximum number of sequence IDs per token (default: 8)
Returns:
- The result of the decode operation (0 on success)
Raises:
- Llama::Batch::Error on error
Reset performance counters for this context
This method resets all performance counters for the context.
Returns the state manager for this context Lazily initializes the state if it doesn't exist yet