class

ML::NN::TransformerEncoderBlock

Inherits Reference / Object

Transformer Encoder Block LayerNorm -> Self-Attention -> Residual -> LayerNorm -> MLP -> Residual

Constructors

new(embed_dim : Int32, num_heads : Int32, mlp_ratio : Float32 = 4.0_f32, dropout : Float32 = 0.0_f32, device : Tensor::Device = Tensor.default_device)
Source

Instance methods

attention
Source
call(x : Autograd::Variable, attn_mask : Tensor | Nil = nil) : Autograd::Variable
Source
dropout
Source
forward(x : Autograd::Variable, attn_mask : Tensor | Nil = nil) : Autograd::Variable

Forward: Pre-norm architecture (like ViT) x: [batch, seq_len, embed_dim]

Source
norm1
Source
norm2
Source
parameters
Source