neuralmonkey.decoders package

Submodules

neuralmonkey.decoders.beam_search_decoder module

Beam search decoder.

This module implements the beam search algorithm for the recurrent decoder.

As well as the recurrent decoder, this decoder works dynamically, which means it uses the tf.while_loop function conditioned on both maximum output length and list of finished hypotheses.

The beam search decoder works by appending data from SearchStepOutput objects to a SearchStepOutputTA object. The SearchStepOutput object stores information about the hypotheses in the beam. Each hypothesis keeps its score, its final token, and a pointer to a “parent” hypothesis, which is a one-token-shorter hypothesis which shares the tokens with the child hypothesis.

For the beam search decoder to work, it must keep an inner state which stores information about hypotheses in the beam. It is an object of type SearchState which stores, for each hypothesis, its sum of log probabilities of the tokens, its length, finished flag, ID of the last token, and the last decoder and attention states.

There is another inner state object here, the BeamSearchLoopState. It is a technical structure used with the tf.while_loop function. It stores all the previously mentioned information, plus the decoder LoopState, which is used in the decoder when its own tf.while_loop function is used - this is not the case when using beam search because we want to run the decoder’s steps manually.

class neuralmonkey.decoders.beam_search_decoder.BeamSearchDecoder(name: str, parent_decoder: neuralmonkey.decoders.decoder.Decoder, beam_size: int, length_normalization: float, max_steps: int = None, save_checkpoint: str = None, load_checkpoint: str = None) → None

Bases: neuralmonkey.model.model_part.ModelPart

In-graph beam search for batch size 1.

The hypothesis scoring algorithm is taken from https://arxiv.org/pdf/1609.08144.pdf. Length normalization is parameter alpha from equation 14.

beam_size
feed_dict(dataset: neuralmonkey.dataset.Dataset, train: bool = False) → typing.Dict[tensorflow.python.framework.ops.Tensor, typing.Any]

Populate the feed dictionary for the decoder object.

Parameters:
  • dataset – The dataset to use for the decoder.
  • train – Boolean flag, telling whether this is a training run
get_body() → typing.Callable

Return a body function for tf.while_loop.

get_initial_loop_state() → neuralmonkey.decoders.beam_search_decoder.BeamSearchLoopState
vocabulary
class neuralmonkey.decoders.beam_search_decoder.BeamSearchLoopState(bs_state, bs_output, decoder_loop_state)

Bases: tuple

bs_output

Alias for field number 1

bs_state

Alias for field number 0

decoder_loop_state

Alias for field number 2

class neuralmonkey.decoders.beam_search_decoder.SearchState(logprob_sum, lengths, finished, last_word_ids, last_state, last_attns)

Bases: tuple

finished

Alias for field number 2

last_attns

Alias for field number 5

last_state

Alias for field number 4

last_word_ids

Alias for field number 3

lengths

Alias for field number 1

logprob_sum

Alias for field number 0

class neuralmonkey.decoders.beam_search_decoder.SearchStepOutput(scores, parent_ids, token_ids)

Bases: tuple

parent_ids

Alias for field number 1

scores

Alias for field number 0

token_ids

Alias for field number 2

class neuralmonkey.decoders.beam_search_decoder.SearchStepOutputTA(scores, parent_ids, token_ids)

Bases: tuple

parent_ids

Alias for field number 1

scores

Alias for field number 0

token_ids

Alias for field number 2

neuralmonkey.decoders.classifier module

class neuralmonkey.decoders.classifier.Classifier(name: str, encoders: typing.List[neuralmonkey.model.stateful.Stateful], vocabulary: neuralmonkey.vocabulary.Vocabulary, data_id: str, layers: typing.List[int], activation_fn: typing.Callable[[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor] = <function relu>, dropout_keep_prob: float = 0.5, save_checkpoint: typing.Union[str, NoneType] = None, load_checkpoint: typing.Union[str, NoneType] = None) → None

Bases: neuralmonkey.model.model_part.ModelPart

A simple MLP classifier over encoders.

The API pretends it is an RNN decoder which always generates a sequence of length exactly one.

cost
decoded
decoded_logits
decoded_seq
feed_dict(dataset: neuralmonkey.dataset.Dataset, train: bool = False) → typing.Dict[tensorflow.python.framework.ops.Tensor, typing.Any]
gt_inputs
loss_with_decoded_ins
loss_with_gt_ins
runtime_logprobs
runtime_loss
train_loss
train_mode

neuralmonkey.decoders.ctc_decoder module

class neuralmonkey.decoders.ctc_decoder.CTCDecoder(name: str, encoder: typing.Any, vocabulary: neuralmonkey.vocabulary.Vocabulary, data_id: str, merge_repeated_targets: bool = False, merge_repeated_outputs: bool = True, beam_width: int = 1, save_checkpoint: typing.Union[str, NoneType] = None, load_checkpoint: typing.Union[str, NoneType] = None) → None

Bases: neuralmonkey.model.model_part.ModelPart

Connectionist Temporal Classification.

See tf.nn.ctc_loss, tf.nn.ctc_greedy_decoder etc.

cost
decoded
feed_dict(dataset: neuralmonkey.dataset.Dataset, train: bool = False) → typing.Dict[tensorflow.python.framework.ops.Tensor, typing.Any]
input_lengths
logits
runtime_loss
train_loss
train_mode
train_targets

neuralmonkey.decoders.decoder module

class neuralmonkey.decoders.decoder.Decoder(encoders: typing.List[neuralmonkey.model.stateful.Stateful], vocabulary: neuralmonkey.vocabulary.Vocabulary, data_id: str, name: str, max_output_len: int, dropout_keep_prob: float = 1.0, rnn_size: int = None, embedding_size: int = None, output_projection: typing.Union[typing.Tuple[typing.Callable[[tensorflow.python.framework.ops.Tensor, tensorflow.python.framework.ops.Tensor, typing.List[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor], int], typing.Callable[[tensorflow.python.framework.ops.Tensor, tensorflow.python.framework.ops.Tensor, typing.List[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor]] = None, encoder_projection: typing.Callable[[tensorflow.python.framework.ops.Tensor, int, typing.List[neuralmonkey.model.stateful.Stateful]], tensorflow.python.framework.ops.Tensor] = None, attentions: typing.List[neuralmonkey.attention.base_attention.BaseAttention] = None, embeddings_source: neuralmonkey.model.sequence.EmbeddedSequence = None, attention_on_input: bool = True, rnn_cell: str = 'GRU', conditional_gru: bool = False, save_checkpoint: str = None, load_checkpoint: str = None) → None

Bases: neuralmonkey.model.model_part.ModelPart

Decoder class.

A class that manages parts of the computation graph that are used for the decoding.

batch_size
cost
decoded
decoding_b
decoding_w
embed_input_symbol(*args) → tensorflow.python.framework.ops.Tensor
embedding_matrix

Variables and operations for embedding of input words.

If we are reusing word embeddings, this function takes the embedding matrix from the first encoder

feed_dict(dataset: neuralmonkey.dataset.Dataset, train: bool = False) → typing.Dict[tensorflow.python.framework.ops.Tensor, typing.Any]

Populate the feed dictionary for the decoder object.

Parameters:
  • dataset – The dataset to use for the decoder.
  • train – Boolean flag, telling whether this is a training run
get_body(train_mode: bool, sample: bool = False) → typing.Callable
get_initial_loop_state() → neuralmonkey.decoders.decoder.LoopState
go_symbols
initial_state

Compute initial decoder state.

The part of the computation graph that computes the initial state of the decoder.

input_plus_attention(*args) → tensorflow.python.framework.ops.Tensor

Merge input and previous attentions.

Input and previous attentions are merged into a single vector of the size fo embedding.

loop_continue_criterion(*args) → tensorflow.python.framework.ops.Tensor
runtime_logits
runtime_logprobs
runtime_loop_result
runtime_loss
runtime_mask
runtime_rnn_states
train_inputs
train_logits
train_logprobs
train_loss
train_mode
train_padding
train_xents
class neuralmonkey.decoders.decoder.LoopState(step, input_symbol, train_inputs, prev_rnn_state, prev_rnn_output, rnn_outputs, prev_logits, logits, prev_contexts, mask, finished, attention_loop_states)

Bases: tuple

attention_loop_states

Alias for field number 11

finished

Alias for field number 10

input_symbol

Alias for field number 1

logits

Alias for field number 7

mask

Alias for field number 9

prev_contexts

Alias for field number 8

prev_logits

Alias for field number 6

prev_rnn_output

Alias for field number 4

prev_rnn_state

Alias for field number 3

rnn_outputs

Alias for field number 5

step

Alias for field number 0

train_inputs

Alias for field number 2

neuralmonkey.decoders.encoder_projection module

Encoder Projection Module.

This module contains different variants of projection of encoders into the initial state of the decoder.

Encoder projections are specified in the configuration file. Each encoder projection function has a unified type EncoderProjection, which is a callable that takes three arguments:

  1. train_mode – boolean tensor specifying whether the train mode is on
  2. rnn_size – the size of the resulting initial state
  3. encoders – a list of Stateful objects used as the encoders.

To enable further parameterization of encoder projection functions, one can use higher-order functions.

neuralmonkey.decoders.encoder_projection.concat_encoder_projection(train_mode: tensorflow.python.framework.ops.Tensor, rnn_size: int = None, encoders: typing.List[neuralmonkey.model.stateful.Stateful] = None) → tensorflow.python.framework.ops.Tensor

Concatenate the encoded values of the encoders.

neuralmonkey.decoders.encoder_projection.empty_initial_state(train_mode: tensorflow.python.framework.ops.Tensor, rnn_size: int, encoders: typing.List[neuralmonkey.model.stateful.Stateful] = None) → tensorflow.python.framework.ops.Tensor

Return an empty vector.

neuralmonkey.decoders.encoder_projection.linear_encoder_projection(dropout_keep_prob: float) → typing.Callable[[tensorflow.python.framework.ops.Tensor, int, typing.List[neuralmonkey.model.stateful.Stateful]], tensorflow.python.framework.ops.Tensor]

Return a linear encoder projection.

Return a projection function which applies dropout on concatenated encoder final states and returns a linear projection to a rnn_size-sized tensor.

Parameters:dropout_keep_prob – The dropout keep probability
neuralmonkey.decoders.encoder_projection.nematus_projection(dropout_keep_prob: float = 1.0) → typing.Callable[[tensorflow.python.framework.ops.Tensor, int, typing.List[neuralmonkey.model.stateful.Stateful]], tensorflow.python.framework.ops.Tensor]

Return encoder projection used in Nematus.

The initial state is a dense projection with tanh activation computed on the averaged states of the encoders. Dropout is applied to the means (before the projection).

Parameters:dropout_keep_prob – The dropout keep probability.

neuralmonkey.decoders.output_projection module

Output Projection Module.

This module contains different variants of projection functions of decoder outputs into the logit function inputs.

Output projections are specified in the configuration file. Each output projection function has a unified type OutputProjection, which is a callable that takes four arguments and returns a tensor:

  1. prev_state – the hidden state of the decoder.
  2. prev_output – embedding of the previously decoded word (or train input)
  3. ctx_tensots – a list of context vectors (for each attention object)

To enable further parameterization of output projection functions, one can use higher-order functions.

neuralmonkey.decoders.output_projection.maxout_output(maxout_size: int) → typing.Tuple[typing.Callable[[tensorflow.python.framework.ops.Tensor, tensorflow.python.framework.ops.Tensor, typing.List[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor], int]

Apply maxout.

Compute RNN output out of the previous state and output, and the context tensors returned from attention mechanisms, as described in the article

This function corresponds to the equations for computation the t_tilde in the Bahdanau et al. (2015) paper, on page 14, with the maxout projection, before the last linear projection.

Parameters:maxout_size – The size of the hidden maxout layer in the deep output
Returns:Returns the maxout projection of the concatenated inputs
neuralmonkey.decoders.output_projection.mlp_output(layer_sizes: typing.List[int], activation: typing.Callable[[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor] = <function tanh>, dropout_keep_prob: float = 1.0) → typing.Tuple[typing.Callable[[tensorflow.python.framework.ops.Tensor, tensorflow.python.framework.ops.Tensor, typing.List[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor], int]

Apply a multilayer perceptron.

Compute RNN deep output using the multilayer perceptron with a specified activation function. (Pascanu et al., 2013 [https://arxiv.org/pdf/1312.6026v5.pdf])

Parameters:
  • layer_sizes – A list of sizes of the hiddel layers of the MLP
  • dropout_keep_prob – the dropout keep probability
  • activation – The activation function to use in each layer.
neuralmonkey.decoders.output_projection.nematus_output(output_size: int, dropout_keep_prob: float = 1.0) → typing.Tuple[typing.Callable[[tensorflow.python.framework.ops.Tensor, tensorflow.python.framework.ops.Tensor, typing.List[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor], int]

Apply nonlinear one-hidden-layer deep output.

Implementation consistent with Nematus. Can be used instead of (and is in theory equivalent to) nonlinear_output.

Projects the RNN state, embedding of the previously outputted word, and concatenation of all context vectors into a shared vector space, sums them up and apply a hyperbolic tangent activation function.

neuralmonkey.decoders.output_projection.nonlinear_output(output_size: int, activation_fn: typing.Callable[[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor] = <function tanh>) → typing.Tuple[typing.Callable[[tensorflow.python.framework.ops.Tensor, tensorflow.python.framework.ops.Tensor, typing.List[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor], int]

neuralmonkey.decoders.sequence_labeler module

class neuralmonkey.decoders.sequence_labeler.SequenceLabeler(name: str, encoder: typing.Union[neuralmonkey.encoders.recurrent.RecurrentEncoder, neuralmonkey.encoders.facebook_conv.SentenceEncoder], vocabulary: neuralmonkey.vocabulary.Vocabulary, data_id: str, dropout_keep_prob: float = 1.0, save_checkpoint: typing.Union[str, NoneType] = None, load_checkpoint: typing.Union[str, NoneType] = None) → None

Bases: neuralmonkey.model.model_part.ModelPart

Classifier assing a label to each encoder’s state.

cost
decoded
feed_dict(dataset: neuralmonkey.dataset.Dataset, train: bool = False) → typing.Dict[tensorflow.python.framework.ops.Tensor, typing.Any]
logits
logprobs
runtime_loss
train_loss
train_mode
train_targets
train_weights

neuralmonkey.decoders.sequence_regressor module

class neuralmonkey.decoders.sequence_regressor.SequenceRegressor(name: str, encoders: typing.List[neuralmonkey.model.stateful.Stateful], data_id: str, layers: typing.List[int] = None, activation_fn: typing.Callable[[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor] = <function relu>, dropout_keep_prob: float = 1.0, dimension: int = 1, save_checkpoint: str = None, load_checkpoint: str = None) → None

Bases: neuralmonkey.model.model_part.ModelPart

A simple MLP regression over encoders.

The API pretends it is an RNN decoder which always generates a sequence of length exactly one.

cost
decoded
feed_dict(dataset: neuralmonkey.dataset.Dataset, train: bool = False) → typing.Dict[tensorflow.python.framework.ops.Tensor, typing.Any]
predictions
runtime_loss
train_inputs
train_loss
train_mode

neuralmonkey.decoders.word_alignment_decoder module

class neuralmonkey.decoders.word_alignment_decoder.WordAlignmentDecoder(encoder: neuralmonkey.encoders.recurrent.RecurrentEncoder, decoder: neuralmonkey.decoders.decoder.Decoder, data_id: str, name: str) → None

Bases: neuralmonkey.model.model_part.ModelPart

A decoder that computes soft alignment from an attentive encoder.

Loss is computed as cross-entropy against a reference alignment.

alignment_target
cost
feed_dict(dataset: neuralmonkey.dataset.Dataset, train: bool = False) → typing.Dict[tensorflow.python.framework.ops.Tensor, typing.Any]
ref_alignment

Module contents