neuralmonkey.nn package¶
Submodules¶
neuralmonkey.nn.mlp module¶
neuralmonkey.nn.noisy_gru_cell module¶
-
class
neuralmonkey.nn.noisy_gru_cell.
NoisyGRUCell
(num_units, training)¶ Bases:
tensorflow.python.ops.rnn_cell_impl._RNNCell
Gated Recurrent Unit cell (cf. http://arxiv.org/abs/1406.1078) with noisy activation functions (http://arxiv.org/abs/1603.00391). The theano code is availble at https://github.com/caglar/noisy_units.
It is based on the TensorFlow implementatin of GRU just the activation function are changed for the noisy ones.
-
output_size
¶
-
state_size
¶
-
-
neuralmonkey.nn.noisy_gru_cell.
noisy_activation
(x, generic, linearized, training, alpha=1.1, c=0.5)¶ Implements the noisy activation with Half-Normal Noise for Hard-Saturation functions. See http://arxiv.org/abs/1603.00391, Algorithm 1.
Parameters: - x – Tensor which is an input to the activation function
- generic – The generic formulation of the activation function. (denoted as h in the paper)
- linearized – Linearization of the activation based on the first-order Tailor expansion around zero. (denoted as u in the paper)
- training – A boolean tensor telling whether we are in the training stage (and the noise is sampled) or in runtime when the expactation is used instead.
- alpha – Mixing hyper-parameter. The leakage rate from the linearized function to the nonlinear one.
- c – Standard deviation of the sampled noise.
-
neuralmonkey.nn.noisy_gru_cell.
noisy_sigmoid
(x, training)¶
-
neuralmonkey.nn.noisy_gru_cell.
noisy_tanh
(x, training)¶
neuralmonkey.nn.ortho_gru_cell module¶
-
class
neuralmonkey.nn.ortho_gru_cell.
OrthoGRUCell
(num_units, input_size=None, activation=<function tanh>)¶ Bases:
tensorflow.contrib.rnn.python.ops.core_rnn_cell_impl.GRUCell
Classic GRU cell but initialized using random orthogonal matrices
neuralmonkey.nn.pervasive_dropout_wrapper module¶
neuralmonkey.nn.projection module¶
This module implements various types of projections.
-
neuralmonkey.nn.projection.
linear
(inputs, size, scope='LinearProjection')¶ Simple linear projection
y = Wx + b
Parameters: - inputs – A tensor or list of tensors. It should be 2D tensors with equal length in the first dimension (batch size)
- size – The size of dimension 1 of the output tensor.
- scope – The name of the scope used for the variables.
Returns: A tensor of shape batch x size
-
neuralmonkey.nn.projection.
maxout
(inputs, size, scope='MaxoutProjection')¶ Implementation of Maxout layer (Goodfellow et al., 2013) http://arxiv.org/pdf/1302.4389.pdf
z = Wx + b y_i = max(z_{2i-1}, z_{2i})
Parameters: - inputs – A tensor or list of tensors. It should be 2D tensors with equal length in the first dimension (batch size)
- size – The size of dimension 1 of the output tensor.
- scope – The name of the scope used for the variables
Returns: A tensor of shape batch x size
-
neuralmonkey.nn.projection.
multilayer_projection
(input_, layer_sizes, activation=<function relu>, dropout_plc=None, scope='mlp')¶
-
neuralmonkey.nn.projection.
nonlinear
(inputs, size, activation, scope='NonlinearProjection')¶ Linear projection with non-linear activation function
y = activation(Wx + b)
Parameters: - inputs – A tensor or list of tensors. It should be 2D tensors with equal length in the first dimension (batch size)
- size – The size of the second dimension (index 1) of the output tensor
- scope – The name of the scope used for the variables
Returns: A tensor of shape batch x size
neuralmonkey.nn.utils module¶
This module provides utility functions used across the package.
-
neuralmonkey.nn.utils.
dropout
(variable: tensorflow.python.framework.ops.Tensor, keep_prob: float, train_mode: tensorflow.python.framework.ops.Tensor) → tensorflow.python.framework.ops.Tensor¶ Performs dropout on a variable, depending on mode.
Parameters: - variable – The variable to be dropped out
- keep_prob – The probability of keeping a value in the variable
- train_mode – A bool Tensor specifying whether to dropout or not