neuralmonkey.decoders package

Submodules

neuralmonkey.decoders.decoder module

class neuralmonkey.decoders.decoder.Decoder(encoders: typing.List[typing.Any], vocabulary: neuralmonkey.vocabulary.Vocabulary, data_id: str, name: str, max_output_len: int, dropout_keep_prob: float = 1.0, rnn_size: typing.Union[int, NoneType] = None, embedding_size: typing.Union[int, NoneType] = None, output_projection: typing.Union[typing.Callable[[tensorflow.python.framework.ops.Tensor, tensorflow.python.framework.ops.Tensor, typing.List[tensorflow.python.framework.ops.Tensor]], tensorflow.python.framework.ops.Tensor], NoneType] = None, encoder_projection: typing.Union[typing.Callable[[tensorflow.python.framework.ops.Tensor, typing.Union[int, NoneType], typing.Union[typing.List[typing.Any], NoneType]], tensorflow.python.framework.ops.Tensor], NoneType] = None, use_attention: bool = False, embeddings_encoder: typing.Any = None, attention_on_input: bool = True, rnn_cell: str = 'GRU', conditional_gru: bool = False, save_checkpoint: typing.Union[str, NoneType] = None, load_checkpoint: typing.Union[str, NoneType] = None) → None

Bases: neuralmonkey.model.model_part.ModelPart

A class that manages parts of the computation graph that are used for the decoding.

feed_dict(dataset: neuralmonkey.dataset.Dataset, train: bool = False) → typing.Dict[tensorflow.python.framework.ops.Tensor, typing.Any]

Populate the feed dictionary for the decoder object

Parameters:
  • dataset – The dataset to use for the decoder.
  • train – Boolean flag, telling whether this is a training run
get_attention_object(encoder, train_mode: bool)

neuralmonkey.decoders.encoder_projection module

This module contains different variants of projection of encoders into the initial state of the decoder.

neuralmonkey.decoders.encoder_projection.concat_encoder_projection(train_mode: tensorflow.python.framework.ops.Tensor, rnn_size: typing.Union[int, NoneType] = None, encoders: typing.Union[typing.List[typing.Any], NoneType] = None) → tensorflow.python.framework.ops.Tensor

Create the initial state by concatenating the encoders’ encoded values

Parameters:
  • train_mode – tf 0-D bool Tensor specifying the training mode (not used)
  • rnn_size – The size of the resulting vector (not used)
  • encoders – The list of encoders
neuralmonkey.decoders.encoder_projection.empty_initial_state(train_mode: tensorflow.python.framework.ops.Tensor, rnn_size: typing.Union[int, NoneType], encoders: typing.Union[typing.List[typing.Any], NoneType] = None) → tensorflow.python.framework.ops.Tensor

Return an empty vector

Parameters:
  • train_mode – tf 0-D bool Tensor specifying the training mode (not used)
  • rnn_size – The size of the resulting vector
  • encoders – The list of encoders (not used)
neuralmonkey.decoders.encoder_projection.linear_encoder_projection(dropout_keep_prob: float) → typing.Callable[[tensorflow.python.framework.ops.Tensor, typing.Union[int, NoneType], typing.Union[typing.List[typing.Any], NoneType]], tensorflow.python.framework.ops.Tensor]

Return a projection function which applies dropout on concatenated encoder final states and returns a linear projection to a rnn_size-sized tensor.

Parameters:dropout_keep_prob – The dropout keep probability

neuralmonkey.decoders.multi_decoder module

class neuralmonkey.decoders.multi_decoder.MultiDecoder(main_decoder, regularization_decoders)

Bases: object

The MultiDecoder class wraps a several child decoders into one parent encoder. The Neural Monkey architecture requires the model to have only one decoder, so this class can be used when more than one output sequence should be generated (i.e. multi-task learning).

The multi decoder object composes of one main decoder and an arbitrary number of additional decoders, called ‘regularization’ decoders.

The reason for this division is that during validation, we need to report a single score of the model as a whole, and based on this score, the training process decides whether to save the model variables or not.

So if the task is translation with POS tagging of the source sentence, the main decoder should be the decoder that generates the target sentence, whereas the sequence labeler used for POS tagging should be included in the regularization decoders list.

During training, the multi decoder works in the following way: According to the value of the _input_selector placeholder, the loss corresponds to one of the child decoders (so in multi-task learning, the weights in each batch are updated with respect only to one sub-task). It is therefore a good practice to alternate between batches of different task. This is because we often do not have the training data that cover all tasks in one corpus.

all_decoded()
cost
data_id
decoded
feed_dict(dataset, train=False)

Populate the feed dictionary for the decoder object

Decoder placeholders:
input_selector: the index of the child decoder used for
computing the loss
learning_step
runtime_loss
train_loss
vocabulary
vocabulary_size

neuralmonkey.decoders.output_projection module

This module contains different variants of projection functions for RNN outputs.

neuralmonkey.decoders.output_projection.maxout_output(maxout_size)

Compute RNN output out of the previous state and output, and the context tensors returned from attention mechanisms, as described in the article

This function corresponds to the equations for computation the t_tilde in the Bahdanau et al. (2015) paper, on page 14, with the maxout projection, before the last linear projection.

Parameters:maxout_size – The size of the hidden maxout layer in the deep output
Returns:Returns the maxout projection of the concatenated inputs
neuralmonkey.decoders.output_projection.mlp_output(layer_sizes, dropout_plc=None, activation=<function tanh>)

Compute RNN deep output using the multilayer perceptron with a specified activation function. (Pascanu et al., 2013 [https://arxiv.org/pdf/1312.6026v5.pdf])

Parameters:
  • layer_sizes – A list of sizes of the hiddel layers of the MLP
  • dropout_plc – Dropout placeholder. TODO this is not going to work with current configuration
  • activation – The activation function to use in each layer.
neuralmonkey.decoders.output_projection.no_deep_output(prev_state, prev_output, ctx_tensors)

Compute RNN output out of the previous state and output, and the context tensors returned from attention mechanisms.

This function corresponds to the equations for computation the t_tilde in the Bahdanau et al. (2015) paper, on page 14, before the linear projection.

Parameters:
  • prev_state – Previous decoder RNN state. (Denoted s_i-1)
  • prev_output – Embedded output of the previous step. (y_i-1)
  • ctx_tensors – Context tensors computed by the attentions. (c_i)
Returns:

This function returns the concatenation of all its inputs.

neuralmonkey.decoders.sequence_classifier module

class neuralmonkey.decoders.sequence_classifier.SequenceClassifier(name: str, encoders: typing.List[typing.Any], vocabulary: neuralmonkey.vocabulary.Vocabulary, data_id: str, layers: typing.List[int], activation_fn: typing.Callable[[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor] = <function relu>, dropout_keep_prob: float = 0.5, save_checkpoint: typing.Union[str, NoneType] = None, load_checkpoint: typing.Union[str, NoneType] = None) → None

Bases: neuralmonkey.model.model_part.ModelPart

A simple MLP classifier over encoders.

The API pretends it is an RNN decoder which always generates a sequence of length exactly one.

decoded
feed_dict(dataset: neuralmonkey.dataset.Dataset, train: bool = False) → typing.Dict[tensorflow.python.framework.ops.Tensor, typing.Any]
runtime_loss
train_loss

neuralmonkey.decoders.sequence_labeler module

class neuralmonkey.decoders.sequence_labeler.SequenceLabeler(name: str, encoder: neuralmonkey.encoders.sentence_encoder.SentenceEncoder, vocabulary: neuralmonkey.vocabulary.Vocabulary, data_id: str, dropout_keep_prob: float = 1.0, save_checkpoint: typing.Union[str, NoneType] = None, load_checkpoint: typing.Union[str, NoneType] = None) → None

Bases: neuralmonkey.model.model_part.ModelPart

Classifier assing a label to each encoder’s state.

cost
decoded
feed_dict(dataset: neuralmonkey.dataset.Dataset, train: bool = False) → typing.Dict[tensorflow.python.framework.ops.Tensor, typing.Any]
logits
logprobs
runtime_loss
train_loss

neuralmonkey.decoders.word_alignment_decoder module

class neuralmonkey.decoders.word_alignment_decoder.WordAlignmentDecoder(encoder: neuralmonkey.encoders.sentence_encoder.SentenceEncoder, decoder: neuralmonkey.decoders.decoder.Decoder, data_id: str, name: str) → None

Bases: neuralmonkey.model.model_part.ModelPart

A decoder that computes soft alignment from an attentive encoder. Loss is computed as cross-entropy against a reference alignment.

cost
feed_dict(dataset: neuralmonkey.dataset.Dataset, train: bool = False) → typing.Dict[tensorflow.python.framework.ops.Tensor, typing.Any]

Module contents