class

ML::GGUF::MetalBackend

Inherits ML::GGUF::ComputeBackend / Reference / Object

Constants

MM_BATCH_THRESHOLD = 8
MM_NR0 = 64
MM_NR1 = 32
MM_SHMEM = 16384
SIMD_N_ROWS = 2
SIMD_NR0 = 2

Constructors

Instance methods

dot(a : Array(Float32), a_off : Int32, b : Array(Float32), b_off : Int32, len : Int32) : Float32

Dot product for attention scores

Source
encode_layers(hidden_data : Array(Float32), seq_len : Int32, layers : Array(NomicBertMoE::LayerWeights), dim : Int32, n_heads : Int32, head_dim : Int32, ffn_dim : Int32, n_experts : Int32, n_experts_used : Int32, moe_every_n : Int32, cpu_ref_per_layer : Array(Array(Float32)) | Nil = nil) : Array(Float32)
Source
gelu(x : Float32) : Float32

GELU activation (returns new value)

Source
init_workspace(max_seq : Int32, dim : Int32, ffn_dim : Int32, n_heads : Int32, head_dim : Int32, n_experts_used : Int32, rope_cos : Array(Float32), rope_sin : Array(Float32))
Source
layer_norm!(x : Array(Float32), n_pos : Int32, dim : Int32, w : Array(Float32), b : Array(Float32)) : Nil

In-place layer norm

Source
matmul(x : Array(Float32), rows : Int32, qw : QuantWeight, bias : Array(Float32)) : Array(Float32)

ComputeBackend interface fallbacks

Source
softmax_row!(scores : Array(Float32), offset : Int32, len : Int32) : Nil

In-place softmax over a row

Source
upload_f32(data : Array(Float32)) : ML::MetalBuffer
Source
upload_weight(qw : QuantWeight, bias : Array(Float32)) : Nil
Source