Layers

Collection of Ivy neural network layers as stateful classes.

class ivy.neural_net_stateful.layers.Conv1D(input_channels, output_channels, filter_size, strides, padding, weight_initializer=<ivy.neural_net_stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.neural_net_stateful.initializers.Zeros object>, data_format='NWC', dilations=1, dev_str='cpu', v=None)[source]

Bases: ivy.neural_net_stateful.module.Module

__init__(input_channels, output_channels, filter_size, strides, padding, weight_initializer=<ivy.neural_net_stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.neural_net_stateful.initializers.Zeros object>, data_format='NWC', dilations=1, dev_str='cpu', v=None)[source]

1D convolutional layer.

Parameters
  • input_channels (int) – Number of input channels for the layer.

  • output_channels (int) – Number of output channels for the layer.

  • filter_size (int) – Size of the convolutional filter.

  • strides (int or sequence of ints) – The stride of the sliding window for each dimension of input.

  • padding (string or sequence of ints) – “SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • weight_initializer (ivy.Initializer, optional) – Initializer for the weights. Default is GlorotUniform.

  • bias_initializer (ivy.Initializer, optional) – Initializer for the bias. Default is Zeros.

  • data_format (string) – “NWC” or “NCW”. Defaults to “NWC”.

  • dilations (int or sequence of ints) – The dilation factor for each dimension of input.

  • dev_str (str, optional) – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v (ivy container of variables, optional) – the variables for each of the linear layer, as a container, constructed internally by default.

class ivy.neural_net_stateful.layers.Conv1DTranspose(input_channels, output_channels, filter_size, strides, padding, weight_initializer=<ivy.neural_net_stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.neural_net_stateful.initializers.Zeros object>, output_shape=None, data_format='NWC', dilations=1, dev_str='cpu', v=None)[source]

Bases: ivy.neural_net_stateful.module.Module

__init__(input_channels, output_channels, filter_size, strides, padding, weight_initializer=<ivy.neural_net_stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.neural_net_stateful.initializers.Zeros object>, output_shape=None, data_format='NWC', dilations=1, dev_str='cpu', v=None)[source]

1D transpose convolutional layer.

Parameters
  • input_channels (int) – Number of input channels for the layer.

  • output_channels (int) – Number of output channels for the layer.

  • filter_size (int) – Size of the convolutional filter.

  • strides (int or sequence of ints) – The stride of the sliding window for each dimension of input.

  • padding (string or sequence of ints) – “SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • weight_initializer (ivy.Initializer, optional) – Initializer for the weights. Default is GlorotUniform.

  • bias_initializer (ivy.Initializer, optional) – Initializer for the bias. Default is Zeros.

  • output_shape (sequence of ints, needed for TensorFlow) – Shape of the output

  • data_format (string) – “NWC” or “NCW”. Defaults to “NWC”.

  • dilations (int or sequence of ints) – The dilation factor for each dimension of input.

  • dev_str (str, optional) – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v (ivy container of variables, optional) – the variables for each of the linear layer, as a container, constructed internally by default.

class ivy.neural_net_stateful.layers.Conv2D(input_channels, output_channels, filter_shape, strides, padding, weight_initializer=<ivy.neural_net_stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.neural_net_stateful.initializers.Zeros object>, data_format='NHWC', dilations=1, dev_str='cpu', v=None)[source]

Bases: ivy.neural_net_stateful.module.Module

__init__(input_channels, output_channels, filter_shape, strides, padding, weight_initializer=<ivy.neural_net_stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.neural_net_stateful.initializers.Zeros object>, data_format='NHWC', dilations=1, dev_str='cpu', v=None)[source]

2D convolutional layer.

Parameters
  • input_channels (int) – Number of input channels for the layer.

  • output_channels (int) – Number of output channels for the layer.

  • filter_shape (sequence of ints) – Shape of the convolutional filter.

  • strides (int or sequence of ints) – The stride of the sliding window for each dimension of input.

  • padding (string or sequence of ints) – “SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • weight_initializer (ivy.Initializer, optional) – Initializer for the weights. Default is GlorotUniform.

  • bias_initializer (ivy.Initializer, optional) – Initializer for the bias. Default is Zeros.

  • data_format (string) – “NHWC” or “NCHW”. Defaults to “NHWC”.

  • dilations (int or sequence of ints) – The dilation factor for each dimension of input.

  • dev_str (str, optional) – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v (ivy container of variables, optional) – the variables for each of the linear layer, as a container, constructed internally by default.

class ivy.neural_net_stateful.layers.Conv2DTranspose(input_channels, output_channels, filter_shape, strides, padding, weight_initializer=<ivy.neural_net_stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.neural_net_stateful.initializers.Zeros object>, output_shape=None, data_format='NHWC', dilations=1, dev_str='cpu', v=None)[source]

Bases: ivy.neural_net_stateful.module.Module

__init__(input_channels, output_channels, filter_shape, strides, padding, weight_initializer=<ivy.neural_net_stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.neural_net_stateful.initializers.Zeros object>, output_shape=None, data_format='NHWC', dilations=1, dev_str='cpu', v=None)[source]

2D convolutional transpose layer.

Parameters
  • input_channels (int) – Number of input channels for the layer.

  • output_channels (int) – Number of output channels for the layer.

  • filter_shape (sequence of ints) – Shape of the convolutional filter.

  • strides (int or sequence of ints) – The stride of the sliding window for each dimension of input.

  • padding (string or sequence of ints) – “SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • weight_initializer (ivy.Initializer, optional) – Initializer for the weights. Default is GlorotUniform.

  • bias_initializer (ivy.Initializer, optional) – Initializer for the bias. Default is Zeros.

  • output_shape (sequence of ints, needed for TensorFlow) – Shape of the output

  • data_format (string) – “NHWC” or “NCHW”. Defaults to “NHWC”.

  • dilations (int or sequence of ints) – The dilation factor for each dimension of input.

  • dev_str (str, optional) – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v (ivy container of variables, optional) – the variables for each of the linear layer, as a container, constructed internally by default.

class ivy.neural_net_stateful.layers.Conv3D(input_channels, output_channels, filter_shape, strides, padding, weight_initializer=<ivy.neural_net_stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.neural_net_stateful.initializers.Zeros object>, data_format='NDHWC', dilations=1, dev_str='cpu', v=None)[source]

Bases: ivy.neural_net_stateful.module.Module

__init__(input_channels, output_channels, filter_shape, strides, padding, weight_initializer=<ivy.neural_net_stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.neural_net_stateful.initializers.Zeros object>, data_format='NDHWC', dilations=1, dev_str='cpu', v=None)[source]

3D convolutional layer.

Parameters
  • input_channels (int) – Number of input channels for the layer.

  • output_channels (int) – Number of output channels for the layer.

  • filter_shape (sequence of ints) – Shape of the convolutional filter.

  • strides (int or sequence of ints) – The stride of the sliding window for each dimension of input.

  • padding (string or sequence of ints) – “SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • weight_initializer (ivy.Initializer, optional) – Initializer for the weights. Default is GlorotUniform.

  • bias_initializer (ivy.Initializer, optional) – Initializer for the bias. Default is Zeros.

  • data_format (string) – “NDHWC” or “NCDHW”. Defaults to “NDHWC”.

  • dilations (int or sequence of ints) – The dilation factor for each dimension of input.

  • dev_str (str, optional) – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v (ivy container of variables, optional) – the variables for each of the linear layer, as a container, constructed internally by default.

class ivy.neural_net_stateful.layers.Conv3DTranspose(input_channels, output_channels, filter_shape, strides, padding, weight_initializer=<ivy.neural_net_stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.neural_net_stateful.initializers.Zeros object>, output_shape=None, data_format='NDHWC', dilations=1, dev_str='cpu', v=None)[source]

Bases: ivy.neural_net_stateful.module.Module

__init__(input_channels, output_channels, filter_shape, strides, padding, weight_initializer=<ivy.neural_net_stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.neural_net_stateful.initializers.Zeros object>, output_shape=None, data_format='NDHWC', dilations=1, dev_str='cpu', v=None)[source]

3D convolutional transpose layer.

Parameters
  • input_channels (int) – Number of input channels for the layer.

  • output_channels (int) – Number of output channels for the layer.

  • filter_shape (sequence of ints) – Shape of the convolutional filter.

  • strides (int or sequence of ints) – The stride of the sliding window for each dimension of input.

  • padding (string or sequence of ints) – “SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • weight_initializer (ivy.Initializer, optional) – Initializer for the weights. Default is GlorotUniform.

  • bias_initializer (ivy.Initializer, optional) – Initializer for the bias. Default is Zeros.

  • output_shape (sequence of ints, needed for TensorFlow) – Shape of the output

  • data_format (string) – “NDHWC” or “NCDHW”. Defaults to “NDHWC”.

  • dilations (int or sequence of ints) – The dilation factor for each dimension of input.

  • dev_str (str, optional) – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v (ivy container of variables, optional) – the variables for each of the linear layer, as a container, constructed internally by default.

class ivy.neural_net_stateful.layers.DepthwiseConv2D(num_channels, filter_shape, strides, padding, weight_initializer=<ivy.neural_net_stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.neural_net_stateful.initializers.Zeros object>, data_format='NHWC', dilations=1, dev_str='cpu', v=None)[source]

Bases: ivy.neural_net_stateful.module.Module

__init__(num_channels, filter_shape, strides, padding, weight_initializer=<ivy.neural_net_stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.neural_net_stateful.initializers.Zeros object>, data_format='NHWC', dilations=1, dev_str='cpu', v=None)[source]

depthwise 2D convolutional layer.

Parameters
  • num_channels (int) – Number of input channels for the layer.

  • filter_shape (sequence of ints) – Shape of the convolutional filter.

  • strides (int or sequence of ints) – The stride of the sliding window for each dimension of input.

  • padding (string or sequence of ints) – “SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • weight_initializer (ivy.Initializer, optional) – Initializer for the weights. Default is GlorotUniform.

  • bias_initializer (ivy.Initializer, optional) – Initializer for the bias. Default is Zeros.

  • data_format (string) – “NHWC” or “NCHW”. Defaults to “NHWC”.

  • dilations (int or sequence of ints) – The dilation factor for each dimension of input.

  • dev_str (str, optional) – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v (ivy container of variables, optional) – the variables for each of the linear layer, as a container, constructed internally by default.

class ivy.neural_net_stateful.layers.Dropout(prob, scale=True)[source]

Bases: ivy.neural_net_stateful.module.Module

__init__(prob, scale=True)[source]

Dropout layer. The layer randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoull distribution.

Parameters
  • prob (float) – The probability of zeroing out each array element.

  • scale (bool, optional) – Whether to scale the output by 1/(1-prob), default is True.

class ivy.neural_net_stateful.layers.LSTM(input_channels, output_channels, weight_initializer=<ivy.neural_net_stateful.initializers.GlorotUniform object>, num_layers=1, return_sequence=True, return_state=True, dev_str='cpu', v=None)[source]

Bases: ivy.neural_net_stateful.module.Module

__init__(input_channels, output_channels, weight_initializer=<ivy.neural_net_stateful.initializers.GlorotUniform object>, num_layers=1, return_sequence=True, return_state=True, dev_str='cpu', v=None)[source]

LSTM layer, which is a set of stacked lstm cells.

Parameters
  • input_channels (int) – Number of input channels for the layer

  • output_channels (int) – Number of output channels for the layer

  • weight_initializer (ivy.Initializer, optional) – Initializer for the weights. Default is GlorotUniform.

  • num_layers (int, optional) – Number of lstm cells in the lstm layer, default is 1.

  • return_sequence (bool, optional) – Whether or not to return the entire output sequence, or just the latest timestep. Default is True.

  • return_state (bool, optional) – Whether or not to return the latest hidden and cell states. Default is True.

  • dev_str (str, optional) – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v (ivy container of parameter arrays, optional) – the variables for each of the lstm cells, as a container, constructed internally by default.

get_initial_state(batch_shape)[source]

Get the initial state of the hidden and cell states, if not provided explicitly

class ivy.neural_net_stateful.layers.Linear(input_channels, output_channels, weight_initializer=<ivy.neural_net_stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.neural_net_stateful.initializers.Zeros object>, with_bias=True, dev_str=None, v=None)[source]

Bases: ivy.neural_net_stateful.module.Module

__init__(input_channels, output_channels, weight_initializer=<ivy.neural_net_stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.neural_net_stateful.initializers.Zeros object>, with_bias=True, dev_str=None, v=None)[source]

Linear layer, also referred to as dense or fully connected. The layer receives tensors with input_channels last dimension and returns a new tensor with output_channels last dimension, following matrix multiplication with the weight matrix and addition with the bias vector.

Parameters
  • input_channels (int) – Number of input channels for the layer.

  • output_channels (int) – Number of output channels for the layer.

  • weight_initializer (ivy.Initializer, optional) – Initializer for the weights. Default is GlorotUniform.

  • bias_initializer (ivy.Initializer, optional) – Initializer for the bias. Default is Zeros.

  • with_bias (bool, optional) – Whether or not to include a bias term, default is True.

  • dev_str (str, optional) – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v (ivy container of variables, optional) – the variables for the linear layer, as a container, constructed internally by default.

class ivy.neural_net_stateful.layers.MultiHeadAttention(query_dim, num_heads=8, head_dim=64, dropout_rate=0.0, context_dim=None, scale=None, dev_str=None, v=None, build_mode='on_init')[source]

Bases: ivy.neural_net_stateful.module.Module

__init__(query_dim, num_heads=8, head_dim=64, dropout_rate=0.0, context_dim=None, scale=None, dev_str=None, v=None, build_mode='on_init')[source]

Multi Head Attention layer.

Parameters
  • query_dim (int) – The dimension of the attention queries.

  • num_heads (int, optional) – Number of attention heads. Default is 8.

  • head_dim (int, optional) – The dimension of each of the heads. Default is 64.

  • dropout_rate (float, optional) – The rate of dropout. Default is 0.

  • context_dim (int, optional.) – The dimension of the context array. Default is None, in which case the query dim is used.

  • scale (float, optional) – The value by which to scale the query-key similarity measure. Default is head_dim^-0.5

  • dev_str (str, optional) – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v (ivy container of variables, optional) – the variables for the attention layer, as a container, constructed internally by default.

  • build_mode (str, optional) – How the Module is built, either on initialization (now), explicitly by the user by calling build(), or the first time the __call__ method is run. Default is on initialization.