class

Num::NN::AdamOptimizer(T)

Inherits Num::NN::Optimizer / Reference / Object

Adam (short for Adaptive Moment Estimation) is an update to the RMSProp optimizer. In this optimization algorithm, running averages of both the gradients and the second moments of the gradients are used.

Constructors

new(learning_rate : Float64 = 0.001, beta1 : Float64 = 0.9, beta2 : Float64 = 0.999, epsilon : Float64 = 1e-8)

Initializes an Adam optimizer, disconnected from a network. In order to link this optimizer to a Num::NN::Network, calling build_params will register each variable in the computational graph with this optimizer.

Arguments

  • learning_rate : Float - Learning rate of the optimizer
  • beta1 : Float - The exponential decay rate for the 1st moment estimates
  • beta2 : Float - The exponential decay rate for the 2nd moment estimates
  • epsilon : Float - A small constant for numerical stability
Source

Instance methods

build_params(l : Array(Layer(T)))

Adds variables from a Num::NN::Network to the optimizer, to be tracked and updated after each forward pass through a network.

Arguments

  • l : Array(Layer(T)) - Array of Layers in the Network
Source
update

Updates all Num::Grad::Variables registered to the optimizer based on weights present in the network and the parameters of the optimizer. Resets all gradients to 0.

Source