neuralmonkey.nn package

Submodules

neuralmonkey.nn.highway module

This module implements the highway networks.

neuralmonkey.nn.highway.highway(inputs, activation=<function relu>, scope='HighwayNetwork')

Simple highway layer

y = H(x, Wh) * T(x, Wt) + x * C(x, Wc)

where:

C(x, Wc) = 1 - T(x, Wt)

Parameters:
  • inputs – A tensor or list of tensors. It should be 2D tensors with equal length in the first dimension (batch size)
  • activation – Activation function of the linear part of the formula H(x, Wh).
  • scope – The name of the scope used for the variables.
Returns:

A tensor of shape tf.shape(inputs)

neuralmonkey.nn.mlp module

class neuralmonkey.nn.mlp.MultilayerPerceptron(mlp_input: tensorflow.python.framework.ops.Tensor, layer_configuration: typing.List[int], dropout_keep_prob: float, output_size: int, train_mode: tensorflow.python.framework.ops.Tensor, activation_fn: typing.Callable[[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor] = <function relu>, name: str = 'multilayer_perceptron') → None

Bases: object

General implementation of the multilayer perceptron.

classification
softmax

neuralmonkey.nn.noisy_gru_cell module

class neuralmonkey.nn.noisy_gru_cell.NoisyGRUCell(num_units: int, training) → None

Bases: tensorflow.python.ops.rnn_cell_impl._RNNCell

Gated Recurrent Unit cell (cf. http://arxiv.org/abs/1406.1078) with noisy activation functions (http://arxiv.org/abs/1603.00391). The theano code is availble at https://github.com/caglar/noisy_units.

It is based on the TensorFlow implementatin of GRU just the activation function are changed for the noisy ones.

output_size
state_size
neuralmonkey.nn.noisy_gru_cell.noisy_activation(x, generic, linearized, training, alpha: float = 1.1, c: float = 0.5)

Implements the noisy activation with Half-Normal Noise for Hard-Saturation functions. See http://arxiv.org/abs/1603.00391, Algorithm 1.

Parameters:
  • x – Tensor which is an input to the activation function
  • generic – The generic formulation of the activation function. (denoted as h in the paper)
  • linearized – Linearization of the activation based on the first-order Tailor expansion around zero. (denoted as u in the paper)
  • training – A boolean tensor telling whether we are in the training stage (and the noise is sampled) or in runtime when the expactation is used instead.
  • alpha – Mixing hyper-parameter. The leakage rate from the linearized function to the nonlinear one.
  • c – Standard deviation of the sampled noise.
neuralmonkey.nn.noisy_gru_cell.noisy_sigmoid(x, training)
neuralmonkey.nn.noisy_gru_cell.noisy_tanh(x, training)

neuralmonkey.nn.ortho_gru_cell module

class neuralmonkey.nn.ortho_gru_cell.OrthoGRUCell(num_units, input_size=None, activation=<function tanh>)

Bases: tensorflow.contrib.rnn.python.ops.core_rnn_cell_impl.GRUCell

Classic GRU cell but initialized using random orthogonal matrices

neuralmonkey.nn.pervasive_dropout_wrapper module

class neuralmonkey.nn.pervasive_dropout_wrapper.PervasiveDropoutWrapper(cell, mask, scale) → None

Bases: tensorflow.python.ops.rnn_cell_impl._RNNCell

output_size
state_size

neuralmonkey.nn.projection module

This module implements various types of projections.

neuralmonkey.nn.projection.glu(input_, gating_fn=<function sigmoid>)

Gated linear unit - Dauphin et al. (2016) http://arxiv.org/abs/1612.08083

neuralmonkey.nn.projection.linear(inputs: tensorflow.python.framework.ops.Tensor, size: int, scope: str = 'LinearProjection')

Simple linear projection

y = Wx + b

Parameters:
  • inputs – A tensor or list of tensors. It should be 2D tensors with equal length in the first dimension (batch size)
  • size – The size of dimension 1 of the output tensor.
  • scope – The name of the scope used for the variables.
Returns:

A tensor of shape batch x size

neuralmonkey.nn.projection.maxout(inputs: tensorflow.python.framework.ops.Tensor, size: int, scope: str = 'MaxoutProjection')

Implementation of Maxout layer (Goodfellow et al., 2013) http://arxiv.org/pdf/1302.4389.pdf

z = Wx + b y_i = max(z_{2i-1}, z_{2i})

Parameters:
  • inputs – A tensor or list of tensors. It should be 2D tensors with equal length in the first dimension (batch size)
  • size – The size of dimension 1 of the output tensor.
  • scope – The name of the scope used for the variables
Returns:

A tensor of shape batch x size

neuralmonkey.nn.projection.multilayer_projection(input_: tensorflow.python.framework.ops.Tensor, layer_sizes: typing.List[int], train_mode: tensorflow.python.framework.ops.Tensor, activation: typing.Callable[[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor] = <function relu>, dropout_keep_prob: float = 1.0, scope: str = 'mlp')
neuralmonkey.nn.projection.nonlinear(inputs: tensorflow.python.framework.ops.Tensor, size: int, activation: typing.Callable[[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor], scope: str = 'NonlinearProjection')

Linear projection with non-linear activation function

y = activation(Wx + b)

Parameters:
  • inputs – A tensor or list of tensors. It should be 2D tensors with equal length in the first dimension (batch size)
  • size – The size of the second dimension (index 1) of the output tensor
  • scope – The name of the scope used for the variables
Returns:

A tensor of shape batch x size

neuralmonkey.nn.utils module

This module provides utility functions used across the package.

neuralmonkey.nn.utils.dropout(variable: tensorflow.python.framework.ops.Tensor, keep_prob: float, train_mode: tensorflow.python.framework.ops.Tensor) → tensorflow.python.framework.ops.Tensor

Performs dropout on a variable, depending on mode.

Parameters:
  • variable – The variable to be dropped out
  • keep_prob – The probability of keeping a value in the variable
  • train_mode – A bool Tensor specifying whether to dropout or not

Module contents