neuralmonkey.nn package¶
Submodules¶
neuralmonkey.nn.highway module¶
This module implements the highway networks.
-
neuralmonkey.nn.highway.
highway
(inputs, activation=<function relu>, scope='HighwayNetwork')¶ Simple highway layer
y = H(x, Wh) * T(x, Wt) + x * C(x, Wc)
where:
C(x, Wc) = 1 - T(x, Wt)
Parameters: - inputs – A tensor or list of tensors. It should be 2D tensors with equal length in the first dimension (batch size)
- activation – Activation function of the linear part of the formula H(x, Wh).
- scope – The name of the scope used for the variables.
Returns: A tensor of shape tf.shape(inputs)
neuralmonkey.nn.mlp module¶
-
class
neuralmonkey.nn.mlp.
MultilayerPerceptron
(mlp_input: tensorflow.python.framework.ops.Tensor, layer_configuration: typing.List[int], dropout_keep_prob: float, output_size: int, train_mode: tensorflow.python.framework.ops.Tensor, activation_fn: typing.Callable[[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor] = <function relu>, name: str = 'multilayer_perceptron') → None¶ Bases:
object
General implementation of the multilayer perceptron.
-
classification
¶
-
softmax
¶
-
neuralmonkey.nn.noisy_gru_cell module¶
-
class
neuralmonkey.nn.noisy_gru_cell.
NoisyGRUCell
(num_units: int, training) → None¶ Bases:
tensorflow.python.ops.rnn_cell_impl._RNNCell
Gated Recurrent Unit cell (cf. http://arxiv.org/abs/1406.1078) with noisy activation functions (http://arxiv.org/abs/1603.00391). The theano code is availble at https://github.com/caglar/noisy_units.
It is based on the TensorFlow implementatin of GRU just the activation function are changed for the noisy ones.
-
output_size
¶
-
state_size
¶
-
-
neuralmonkey.nn.noisy_gru_cell.
noisy_activation
(x, generic, linearized, training, alpha: float = 1.1, c: float = 0.5)¶ Implements the noisy activation with Half-Normal Noise for Hard-Saturation functions. See http://arxiv.org/abs/1603.00391, Algorithm 1.
Parameters: - x – Tensor which is an input to the activation function
- generic – The generic formulation of the activation function. (denoted as h in the paper)
- linearized – Linearization of the activation based on the first-order Tailor expansion around zero. (denoted as u in the paper)
- training – A boolean tensor telling whether we are in the training stage (and the noise is sampled) or in runtime when the expactation is used instead.
- alpha – Mixing hyper-parameter. The leakage rate from the linearized function to the nonlinear one.
- c – Standard deviation of the sampled noise.
-
neuralmonkey.nn.noisy_gru_cell.
noisy_sigmoid
(x, training)¶
-
neuralmonkey.nn.noisy_gru_cell.
noisy_tanh
(x, training)¶
neuralmonkey.nn.ortho_gru_cell module¶
-
class
neuralmonkey.nn.ortho_gru_cell.
OrthoGRUCell
(num_units, input_size=None, activation=<function tanh>, reuse=None)¶ Bases:
tensorflow.contrib.rnn.python.ops.core_rnn_cell_impl.GRUCell
Classic GRU cell but initialized using random orthogonal matrices
neuralmonkey.nn.pervasive_dropout_wrapper module¶
neuralmonkey.nn.projection module¶
This module implements various types of projections.
-
neuralmonkey.nn.projection.
linear
(inputs: tensorflow.python.framework.ops.Tensor, size: int, scope: str = 'LinearProjection')¶ Simple linear projection
y = Wx + b
Parameters: - inputs – A tensor or list of tensors. It should be 2D tensors with equal length in the first dimension (batch size)
- size – The size of dimension 1 of the output tensor.
- scope – The name of the scope used for the variables.
Returns: A tensor of shape batch x size
-
neuralmonkey.nn.projection.
maxout
(inputs: tensorflow.python.framework.ops.Tensor, size: int, scope: str = 'MaxoutProjection')¶ Implementation of Maxout layer (Goodfellow et al., 2013) http://arxiv.org/pdf/1302.4389.pdf
z = Wx + b y_i = max(z_{2i-1}, z_{2i})
Parameters: - inputs – A tensor or list of tensors. It should be 2D tensors with equal length in the first dimension (batch size)
- size – The size of dimension 1 of the output tensor.
- scope – The name of the scope used for the variables
Returns: A tensor of shape batch x size
-
neuralmonkey.nn.projection.
multilayer_projection
(input_: tensorflow.python.framework.ops.Tensor, layer_sizes: typing.List[int], train_mode: tensorflow.python.framework.ops.Tensor, activation: typing.Callable[[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor] = <function relu>, dropout_keep_prob: float = 1.0, scope: str = 'mlp')¶
-
neuralmonkey.nn.projection.
nonlinear
(inputs: tensorflow.python.framework.ops.Tensor, size: int, activation: typing.Callable[[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor], scope: str = 'NonlinearProjection')¶ Linear projection with non-linear activation function
y = activation(Wx + b)
Parameters: - inputs – A tensor or list of tensors. It should be 2D tensors with equal length in the first dimension (batch size)
- size – The size of the second dimension (index 1) of the output tensor
- scope – The name of the scope used for the variables
Returns: A tensor of shape batch x size
neuralmonkey.nn.utils module¶
This module provides utility functions used across the package.
-
neuralmonkey.nn.utils.
dropout
(variable: tensorflow.python.framework.ops.Tensor, keep_prob: float, train_mode: tensorflow.python.framework.ops.Tensor) → tensorflow.python.framework.ops.Tensor¶ Performs dropout on a variable, depending on mode.
Parameters: - variable – The variable to be dropped out
- keep_prob – The probability of keeping a value in the variable
- train_mode – A bool Tensor specifying whether to dropout or not