Optimizers

Collection of Ivy optimizers.

class ivy.neural_net_stateful.optimizers.Adam(lr=0.0001, beta1=0.9, beta2=0.999, epsilon=1e-07, compile_step=False, inplace=True, stop_gradients=True, dev_str=None)[source]

Bases: ivy.neural_net_stateful.optimizers.Optimizer

__init__(lr=0.0001, beta1=0.9, beta2=0.999, epsilon=1e-07, compile_step=False, inplace=True, stop_gradients=True, dev_str=None)[source]

Construct an ADAM optimizer.

Parameters
  • lr (float, optional) – Learning rate, default is 1e-4.

  • beta1 (float, optional) – gradient forgetting factor, default is 0.9

  • beta2 (float, optional) – second moment of gradient forgetting factor, default is 0.999

  • epsilon (float, optional) – divisor during adam update, preventing division by zero, default is 1e-07

  • compile_step (bool, optional) – Whether to compile the optimizer step, default is False.

  • inplace (bool, optional) – Whether to update the variables in-place, or to create new variable handles. This is only relevant for frameworks with stateful variables such as PyTorch. Default is True.

  • stop_gradients (bool, optional) – Whether to stop the gradients of the variables after each gradient step. Default is True.

  • dev_str (str, optional) – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc.

set_state(state)[source]

Set state of the optimizer.

Parameters

state (Ivy container of state tensors) – Nested state to update.

property state
class ivy.neural_net_stateful.optimizers.LAMB(lr=0.0001, beta1=0.9, beta2=0.999, epsilon=1e-07, max_trust_ratio=10, decay_lambda=0, compile_step=False, inplace=True, stop_gradients=True, dev_str=None)[source]

Bases: ivy.neural_net_stateful.optimizers.Optimizer

__init__(lr=0.0001, beta1=0.9, beta2=0.999, epsilon=1e-07, max_trust_ratio=10, decay_lambda=0, compile_step=False, inplace=True, stop_gradients=True, dev_str=None)[source]

Construct an LAMB optimizer.

Parameters
  • lr (float, optional) – Learning rate, default is 1e-4.

  • beta1 (float, optional) – gradient forgetting factor, default is 0.9

  • beta2 (float, optional) – second moment of gradient forgetting factor, default is 0.999

  • epsilon (float, optional) – divisor during adam update, preventing division by zero, default is 1e-07

  • max_trust_ratio (float, optional) – The max value of the trust ratio; the ratio between the norm of the layer weights and norm of gradients update. Default is 10.

  • decay_lambda (float, optional) – The factor used for weight decay. Default is zero.

  • compile_step (bool, optional) – Whether to compile the optimizer step, default is False.

  • inplace (bool, optional) – Whether to update the variables in-place, or to create new variable handles. This is only relevant for frameworks with stateful variables such as PyTorch. Default is True.

  • stop_gradients (bool, optional) – Whether to stop the gradients of the variables after each gradient step. Default is True.

  • dev_str (str, optional) – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc.

set_state(state)[source]

Set state of the optimizer.

Parameters

state (Ivy container of state tensors) – Nested state to update.

property state
class ivy.neural_net_stateful.optimizers.LARS(lr=<function LARS.<lambda>>, decay_lambda=0, compile_step=False, inplace=True, stop_gradients=True)[source]

Bases: ivy.neural_net_stateful.optimizers.Optimizer

__init__(lr=<function LARS.<lambda>>, decay_lambda=0, compile_step=False, inplace=True, stop_gradients=True)[source]

Construct a Layerwise Adaptive Rate Scaling (LARS) optimizer.

Parameters
  • lr (float, optional) – Learning rate, default is 1e-4.

  • decay_lambda (float, optional) – The factor used for weight decay. Default is zero.

  • compile_step (bool, optional) – Whether to compile the optimizer step, default is False.

  • inplace (bool, optional) – Whether to update the variables in-place, or to create new variable handles. This is only relevant for frameworks with stateful variables such as PyTorch. Default is True.

  • stop_gradients (bool, optional) – Whether to stop the gradients of the variables after each gradient step. Default is True.

set_state(state)[source]

Set state of the optimizer.

Parameters

state (Ivy container of state tensors) – Nested state to update.

property state
class ivy.neural_net_stateful.optimizers.Optimizer(lr, compile_step=False, inplace=True, stop_gradients=True)[source]

Bases: abc.ABC

__init__(lr, compile_step=False, inplace=True, stop_gradients=True)[source]

Construct an general Optimizer. This is an abstract class, and must be derived.

Parameters
  • lr (function or float.) – Learning rate.

  • compile_step (bool, optional) – Whether to compile the optimizer step, default is False.

  • inplace (bool, optional) – Whether to update the variables in-place, or to create new variable handles. This is only relevant for frameworks with stateful variables such as PyTorch. Default is True.

  • stop_gradients (bool, optional) – Whether to stop the gradients of the variables after each gradient step. Default is True.

abstract set_state(state)[source]

Set state of the optimizer.

Parameters

state (Ivy container of state tensors) – Nested state to update.

step(v, grads, ignore_missing=False)[source]

Update nested variables container v from possibly compiled overriden private self._step_fn

Parameters
  • v (Ivy container of variables) – Nested variables to update.

  • grads (sequence of arrays) – Nested gradients to update.

  • ignore_missing (bool, optional) – Whether to ignore keys missing from the gradients which exist in the variables. Default is False.

Returns

The updated variables, following update step.

class ivy.neural_net_stateful.optimizers.SGD(lr=<function SGD.<lambda>>, compile_step=False, inplace=True, stop_gradients=True)[source]

Bases: ivy.neural_net_stateful.optimizers.Optimizer

__init__(lr=<function SGD.<lambda>>, compile_step=False, inplace=True, stop_gradients=True)[source]

Construct a Stochastic-Gradient-Descent (SGD) optimizer.

Parameters
  • lr (float, optional) – Learning rate, default is 1e-4.

  • compile_step (bool, optional) – Whether to compile the optimizer step, default is False.

  • inplace (bool, optional) – Whether to update the variables in-place, or to create new variable handles. This is only relevant for frameworks with stateful variables such as PyTorch. Default is True.

  • stop_gradients (bool, optional) – Whether to stop the gradients of the variables after each gradient step. Default is True.

set_state(state)[source]

Set state of the optimizer.

Parameters

state (Ivy container of state tensors) – Nested state to update.

property state

Optimizers