class

MicroGPT::TransformerBlock

Inherits MicroGPT::Layer < Reference < Object

Constructors

new(config : Config, head_dims : Array(Int32))

Heterogeneous heads

Source
new(config : Config)

Uniform heads

Source

Instance methods

attn
Source
backward(grad : Mat) : Mat
Source
backward_accumulate(grad : Mat) : Mat
Source
forward(x : Mat, mask : Mat | Nil = nil) : Mat

Pre-norm: x + attn(norm(x)), then x + ff(norm(x))

Source
update(lr : Float64)
Source