Layers

Collection of Ivy neural network layers in functional form.

ivy.neural_net_functional.layers.conv1d(x, filters, strides, padding, data_format='NWC', dilations=1, f=None)[source]

Computes a 1-D convolution given 3-D input x and filters arrays.

Parameters
  • x (array) – Input image [batch_size,w,d_in].

  • filters (array) – Convolution filters [fw,d_in,d_out].

  • strides (int or sequence of ints) – The stride of the sliding window for each dimension of input.

  • padding (string or sequence of ints) – “SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • data_format (string) – “NWC” or “NCW”. Defaults to “NWC”.

  • dilations (int or sequence of ints) – The dilation factor for each dimension of input.

  • f (ml_framework, optional) – Machine learning library. Inferred from Inputs if None.

Returns

The result of the convolution operation.

ivy.neural_net_functional.layers.conv1d_transpose(x, filters, strides, padding, output_shape=None, data_format='NWC', dilations=1, f=None)[source]

Computes a 1-D transpose convolution given 3-D input x and filters arrays.

Parameters
  • x (array) – Input image [batch_size,w,d_in].

  • filters (array) – Convolution filters [fw,d_in,d_out].

  • strides (int or sequence of ints) – The stride of the sliding window for each dimension of input.

  • padding (string or sequence of ints) – “SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • output_shape (sequence of ints, needed for TensorFlow) – Shape of the output

  • data_format (string) – “NWC” or “NCW”. Defaults to “NWC”.

  • dilations (int or sequence of ints) – The dilation factor for each dimension of input.

  • f (ml_framework, optional) – Machine learning library. Inferred from Inputs if None.

Returns

The result of the transpose convolution operation.

ivy.neural_net_functional.layers.conv2d(x, filters, strides, padding, data_format='NHWC', dilations=1, f=None)[source]

Computes a 2-D convolution given 4-D input x and filters arrays.

Parameters
  • x (array) – Input image [batch_size,h,w,d_in].

  • filters (array) – Convolution filters [fh,fw,d_in,d_out].

  • strides (int or sequence of ints) – The stride of the sliding window for each dimension of input.

  • padding (string or sequence of ints) – “SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • data_format (string) – “NHWC” or “NCHW”. Defaults to “NHWC”.

  • dilations (int or sequence of ints) – The dilation factor for each dimension of input.

  • f (ml_framework, optional) – Machine learning library. Inferred from Inputs if None.

Returns

The result of the convolution operation.

ivy.neural_net_functional.layers.conv2d_transpose(x, filters, strides, padding, output_shape=None, data_format='NHWC', dilations=1, f=None)[source]

Computes a 2-D transpose convolution given 4-D input x and filters arrays.

Parameters
  • x (array) – Input image [batch_size,h,w,d_in].

  • filters (array) – Convolution filters [fh,fw,d_in,d_out].

  • strides (int or sequence of ints) – The stride of the sliding window for each dimension of input.

  • padding (string or sequence of ints) – “SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • output_shape (sequence of ints, needed for TensorFlow) – Shape of the output

  • data_format (string) – “NHWC” or “NCHW”. Defaults to “NHWC”.

  • dilations (int or sequence of ints) – The dilation factor for each dimension of input.

  • f (ml_framework, optional) – Machine learning library. Inferred from Inputs if None.

Returns

The result of the transpose convolution operation.

ivy.neural_net_functional.layers.conv3d(x, filters, strides, padding, data_format='NDHWC', dilations=1, f=None)[source]

Computes a 3-D convolution given 5-D input x and filters arrays.

Parameters
  • x (array) – Input volume [batch_size,d,h,w,d_in].

  • filters (array) – Convolution filters [fd,fh,fw,d_in,d_out].

  • strides (sequence of ints) – The stride of the sliding window for each dimension of input.

  • padding (string or sequence of ints) – “SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • data_format (string) – “NDHWC” or “NCDHW”. Defaults to “NDHWC”.

  • dilations (int or sequence of ints) – The dilation factor for each dimension of input.

  • f (ml_framework, optional) – Machine learning library. Inferred from Inputs if None.

Returns

The result of the convolution operation.

ivy.neural_net_functional.layers.conv3d_transpose(x, filters, strides, padding, output_shape=None, data_format='NDHWC', dilations=1, f=None)[source]

Computes a 3-D transpose convolution given 5-D input x and filters arrays.

Parameters
  • x (array) – Input image [batch_size,d,h,w,d_in].

  • filters (array) – Convolution filters [fd,fh,fw,d_in,d_out].

  • strides (int or sequence of ints) – The stride of the sliding window for each dimension of input.

  • padding (string or sequence of ints) – “SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • output_shape (sequence of ints, needed for TensorFlow) – Shape of the output

  • data_format (string) – “NDHWC” or “NCDHW”. Defaults to “NDHWC”.

  • dilations (int or sequence of ints) – The dilation factor for each dimension of input.

  • f (ml_framework, optional) – Machine learning library. Inferred from Inputs if None.

Returns

The result of the transpose convolution operation.

ivy.neural_net_functional.layers.depthwise_conv2d(x, filters, strides, padding, data_format='NHWC', dilations=1, f=None)[source]

Computes a 2-D depthwise convolution given 4-D input x and filters arrays.

Parameters
  • x (array) – Input image [batch_size,h,w,d].

  • filters (array) – Convolution filters [fh,fw,d].

  • strides (int or sequence of ints) – The stride of the sliding window for each dimension of input.

  • padding (string or sequence of ints) – “SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • data_format (string) – “NHWC” or “NCHW”. Defaults to “NHWC”.

  • dilations (int or sequence of ints) – The dilation factor for each dimension of input.

  • f (ml_framework, optional) – Machine learning library. Inferred from Inputs if None.

Returns

The result of the convolution operation.

ivy.neural_net_functional.layers.dropout(x, prob, scale=True)[source]

Randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoull distribution.

Parameters
  • x (array) – The input array x to perform dropout on.

  • prob (float) – The probability of zeroing out each array element.

  • scale (bool, optional) – Whether to scale the output by 1/(1-prob), default is True.

Returns

Result array of the linear transformation. [N,∗,out_features]

ivy.neural_net_functional.layers.linear(x, weight, bias=None)[source]

Applies a linear transformation to the incoming data: y = x * t(weight) + bias. The operation also supports batching of the weight matrices. This is useful if a batch of different network parameters are to be represented.

Parameters
  • x (array) – The input x compute linear transformation on. [outer_batch_shape,inner_batch_shape,in_features]

  • weight (array) – The weight matrix. [outer_batch_shape,out_features,in_features]

  • bias (array, optional) – The bias vector, default is None. [outer_batch_shape,out_features]

Returns

Result array of the linear transformation. [outer_batch_shape,inner_batch_shape,out_features]

ivy.neural_net_functional.layers.lstm_update(x, init_h, init_c, kernel, recurrent_kernel, bias=None, recurrent_bias=None)[source]

Perform long-short term memory update by unrolling time dimension of input array.

Parameters
  • x (array) – input tensor of LSTM layer [batch_shape, t, in].

  • init_h (array) – initial state tensor for the cell output [batch_shape, out].

  • init_c (array) – initial state tensor for the cell hidden state [batch_shape, out].

  • kernel (array) – weights for cell kernel [in, 4 x out].

  • recurrent_kernel (array) – weights for cell recurrent kernel [out, 4 x out].

  • bias (array) – bias for cell kernel [4 x out].

  • recurrent_bias (array) – bias for cell recurrent kernel [4 x out].

Returns

hidden state for all timesteps [batch_shape,t,out] and cell state for last timestep [batch_shape,out]

ivy.neural_net_functional.layers.multi_head_attention(x, to_q_fn, to_kv_fn, to_out_fn, scale, num_heads, context=None, mask=None, to_q_v=None, to_kv_v=None, to_out_v=None)[source]

Applies multi-head attention to inputs x.

Parameters
  • x (array) – The array to determine the queries from [batch_shape,num_queries,x_feats].

  • to_q_fn (callable) – The function to compute queries from input x.

  • to_kv_fn (callable) – The function to compute keys and values from the context.

  • to_out_fn (callable) – The function to compute the output from the scaled dot-product attention.

  • scale (float) – The value by which to scale the query-key similarity measure before softmax.

  • num_heads (int) – The number of attention heads to use.

  • context (array, optional) – The array to determine the keys and values from. Default is None. [batch_shape,num_values,cont_feats].

  • mask (array, optional) – The mask to apply to the query-key values. Default is None. [batch_shape,num_queries,num_values]

  • to_q_v (variables array, optional) – The variables for function to_q_fn. Default is None.

  • to_kv_v (variables array, optional) – The variables for function to_kv_fn. Default is None.

  • to_out_v (variables array, optional) – The variables for function to_out_fn. Default is None.

:return The output following application of multi-head attention. [batch_shape,num_queries,out_feats]

ivy.neural_net_functional.layers.scaled_dot_product_attention(q, k, v, scale, mask=None)[source]

Applies scaled dot product attention to inputs x using optional mask.

Parameters
  • q (array) – The queries [batch_shape,num_queries,feat_dim].

  • k (array) – The keys [batch_shape,num_values,feat_dim].

  • v (array) – The values [batch_shape,num_values,feat_dim].

  • scale (float) – The value by which to scale the query-key pairs before softmax.

  • mask (array, optional) – The mask to apply to the query-key values. Default is None. [batch_shape,num_queries,num_values]

:return The output following application of scaled dot-product attention.