neuralmonkey.nn package

Submodules

neuralmonkey.nn.bidirectional_rnn_layer module

class neuralmonkey.nn.bidirectional_rnn_layer.BidirectionalRNNLayer(forward_cell, backward_cell, inputs, sentence_lengths_placeholder)

Bases: object

Bidirectional RNN Layer class - forward and backward RNN layers in one.

encoded

Last state of the bidirectional layer

outputs_bidi

Outputs of the bidirectional layer

neuralmonkey.nn.init_ops module

neuralmonkey.nn.init_ops.orthogonal_initializer(gain=1.0, dtype=tf.float32, seed=None)

Returns an initializer that generates an orthogonal matrix or a reshaped orthogonal matrix.

If the shape of the tensor to initialize is two-dimensional, i is initialized with an orthogonal matrix obtained from the singular value decomposition of a matrix of uniform random numbers.

If the shape of the tensor to initialize is more than two-dimensional, a matrix of shape (shape[0] * ... * shape[n - 2], shape[n - 1]) is initialized, where n is the length of the shape vector. The matrix is subsequently reshaped to give a tensor of the desired shape.

Parameters:
  • gain – multiplicative factor to apply to the orthogonal matrix
  • dtype – The type of the output.
  • seed – A Python integer. Used to create random seeds.
Returns:

An initializer that generates orthogonal tensors
Raises:
  • ValueError – if dtype is not a floating point type or if shape has
  • fewer than two entries.

neuralmonkey.nn.mlp module

class neuralmonkey.nn.mlp.MultilayerPerceptron(mlp_input, layer_configuration, dropout_plc, output_size, name='multilayer_perceptron', activation_fn=<function relu>)

Bases: object

General implementation of the multilayer perceptron.

classification
softmax

neuralmonkey.nn.noisy_gru_cell module

class neuralmonkey.nn.noisy_gru_cell.NoisyGRUCell(num_units, training)

Bases: tensorflow.python.ops.rnn_cell.RNNCell

Gated Recurrent Unit cell (cf. http://arxiv.org/abs/1406.1078) with noisy activation functions (http://arxiv.org/abs/1603.00391). The theano code is availble at https://github.com/caglar/noisy_units.

It is based on the TensorFlow implementatin of GRU just the activation function are changed for the noisy ones.

output_size
state_size
neuralmonkey.nn.noisy_gru_cell.noisy_activation(x, generic, linearized, training, alpha=1.1, c=0.5)

Implements the noisy activation with Half-Normal Noise for Hard-Saturation functions. See http://arxiv.org/abs/1603.00391, Algorithm 1.

Parameters:
  • x – Tensor which is an input to the activation function
  • generic – The generic formulation of the activation function. (denoted as h in the paper)
  • linearized – Linearization of the activation based on the first-order Tailor expansion around zero. (denoted as u in the paper)
  • training – A boolean tensor telling whether we are in the training stage (and the noise is sampled) or in runtime when the expactation is used instead.
  • alpha – Mixing hyper-parameter. The leakage rate from the linearized function to the nonlinear one.
  • c – Standard deviation of the sampled noise.
neuralmonkey.nn.noisy_gru_cell.noisy_sigmoid(x, training)
neuralmonkey.nn.noisy_gru_cell.noisy_tanh(x, training)

neuralmonkey.nn.ortho_gru_cell module

class neuralmonkey.nn.ortho_gru_cell.OrthoGRUCell(num_units, input_size=None, activation=<function tanh>)

Bases: tensorflow.python.ops.rnn_cell.GRUCell

Classic GRU cell but initialized using random orthogonal matrices

neuralmonkey.nn.pervasive_dropout_wrapper module

class neuralmonkey.nn.pervasive_dropout_wrapper.PervasiveDropoutWrapper(cell, mask, scale)

Bases: tensorflow.python.ops.rnn_cell.RNNCell

output_size
state_size

neuralmonkey.nn.projection module

This module implements various types of projections.

neuralmonkey.nn.projection.linear(inputs, size, scope='LinearProjection')

Simple linear projection

y = Wx + b

Parameters:
  • inputs – A tensor or list of tensors. It should be 2D tensors with equal length in the first dimension (batch size)
  • size – The size of dimension 1 of the output tensor.
  • scope – The name of the scope used for the variables.
Returns:

A tensor of shape batch x size
neuralmonkey.nn.projection.maxout(inputs, size, scope='MaxoutProjection')

Implementation of Maxout layer (Goodfellow et al., 2013) http://arxiv.org/pdf/1302.4389.pdf

z = Wx + b y_i = max(z_{2i-1}, z_{2i})

Parameters:
  • inputs – A tensor or list of tensors. It should be 2D tensors with equal length in the first dimension (batch size)
  • size – The size of dimension 1 of the output tensor.
  • scope – The name of the scope used for the variables
Returns:

A tensor of shape batch x size
neuralmonkey.nn.projection.multilayer_projection(input_, layer_sizes, activation=<function relu>, dropout_plc=None, scope='mlp')
neuralmonkey.nn.projection.nonlinear(inputs, size, activation, scope='NonlinearProjection')

Linear projection with non-linear activation function

y = activation(Wx + b)

Parameters:
  • inputs – A tensor or list of tensors. It should be 2D tensors with equal length in the first dimension (batch size)
  • size – The size of the second dimension (index 1) of the output tensor
  • scope – The name of the scope used for the variables
Returns:

A tensor of shape batch x size

neuralmonkey.nn.utils module

This module provides utility functions used across the package.

neuralmonkey.nn.utils.dropout(variable: tensorflow.python.framework.ops.Tensor, keep_prob: float, train_mode: tensorflow.python.framework.ops.Tensor) → tensorflow.python.framework.ops.Tensor

Performs dropout on a variable, depending on mode.

Parameters:
  • variable – The variable to be dropped out
  • keep_prob – The probability of keeping a value in the variable
  • train_mode – A bool Tensor specifying whether to dropout or not

Module contents