neuralmonkey.decoders package¶
Submodules¶
neuralmonkey.decoders.beam_search_decoder module¶
Beam search decoder.
This module implements the beam search algorithm for the recurrent decoder.
As well as the recurrent decoder, this decoder works dynamically, which means
it uses the tf.while_loop
function conditioned on both maximum output
length and list of finished hypotheses.
The beam search decoder works by appending data from SearchStepOutput
objects to a SearchStepOutputTA
object. The SearchStepOutput
object
stores information about the hypotheses in the beam. Each hypothesis keeps its
score, its final token, and a pointer to a “parent” hypothesis, which is a
one-token-shorter hypothesis which shares the tokens with the child hypothesis.
For the beam search decoder to work, it must keep an inner state which stores
information about hypotheses in the beam. It is an object of type
SearchState
which stores, for each hypothesis, its sum of log
probabilities of the tokens, its length, finished flag, ID of the last token,
and the last decoder and attention states.
There is another inner state object here, the BeamSearchLoopState
. It is a
technical structure used with the tf.while_loop
function. It stores all the
previously mentioned information, plus the decoder LoopState
, which is used
in the decoder when its own tf.while_loop
function is used - this is not
the case when using beam search because we want to run the decoder’s steps
manually.
-
class
neuralmonkey.decoders.beam_search_decoder.
BeamSearchDecoder
(name: str, parent_decoder: neuralmonkey.decoders.decoder.Decoder, beam_size: int, length_normalization: float, max_steps: int = None, save_checkpoint: str = None, load_checkpoint: str = None) → None¶ Bases:
neuralmonkey.model.model_part.ModelPart
In-graph beam search for batch size 1.
The hypothesis scoring algorithm is taken from https://arxiv.org/pdf/1609.08144.pdf. Length normalization is parameter alpha from equation 14.
-
beam_size
¶
-
feed_dict
(dataset: neuralmonkey.dataset.Dataset, train: bool = False) → typing.Dict[tensorflow.python.framework.ops.Tensor, typing.Any]¶ Populate the feed dictionary for the decoder object.
Parameters: - dataset – The dataset to use for the decoder.
- train – Boolean flag, telling whether this is a training run
-
get_body
() → typing.Callable¶ Return a body function for
tf.while_loop
.
-
get_initial_loop_state
() → neuralmonkey.decoders.beam_search_decoder.BeamSearchLoopState¶
-
vocabulary
¶
-
-
class
neuralmonkey.decoders.beam_search_decoder.
BeamSearchLoopState
(bs_state, bs_output, decoder_loop_state)¶ Bases:
tuple
-
bs_output
¶ Alias for field number 1
-
bs_state
¶ Alias for field number 0
-
decoder_loop_state
¶ Alias for field number 2
-
-
class
neuralmonkey.decoders.beam_search_decoder.
SearchState
(logprob_sum, lengths, finished, last_word_ids, last_state, last_attns)¶ Bases:
tuple
-
finished
¶ Alias for field number 2
-
last_attns
¶ Alias for field number 5
-
last_state
¶ Alias for field number 4
-
last_word_ids
¶ Alias for field number 3
-
lengths
¶ Alias for field number 1
-
logprob_sum
¶ Alias for field number 0
-
neuralmonkey.decoders.classifier module¶
-
class
neuralmonkey.decoders.classifier.
Classifier
(name: str, encoders: typing.List[neuralmonkey.model.stateful.Stateful], vocabulary: neuralmonkey.vocabulary.Vocabulary, data_id: str, layers: typing.List[int], activation_fn: typing.Callable[[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor] = <function relu>, dropout_keep_prob: float = 0.5, save_checkpoint: typing.Union[str, NoneType] = None, load_checkpoint: typing.Union[str, NoneType] = None) → None¶ Bases:
neuralmonkey.model.model_part.ModelPart
A simple MLP classifier over encoders.
The API pretends it is an RNN decoder which always generates a sequence of length exactly one.
-
cost
¶
-
decoded
¶
-
decoded_logits
¶
-
decoded_seq
¶
-
feed_dict
(dataset: neuralmonkey.dataset.Dataset, train: bool = False) → typing.Dict[tensorflow.python.framework.ops.Tensor, typing.Any]¶
-
gt_inputs
¶
-
loss_with_decoded_ins
¶
-
loss_with_gt_ins
¶
-
runtime_logprobs
¶
-
runtime_loss
¶
-
train_loss
¶
-
train_mode
¶
-
neuralmonkey.decoders.ctc_decoder module¶
-
class
neuralmonkey.decoders.ctc_decoder.
CTCDecoder
(name: str, encoder: typing.Any, vocabulary: neuralmonkey.vocabulary.Vocabulary, data_id: str, merge_repeated_targets: bool = False, merge_repeated_outputs: bool = True, beam_width: int = 1, save_checkpoint: typing.Union[str, NoneType] = None, load_checkpoint: typing.Union[str, NoneType] = None) → None¶ Bases:
neuralmonkey.model.model_part.ModelPart
Connectionist Temporal Classification.
See tf.nn.ctc_loss, tf.nn.ctc_greedy_decoder etc.
-
cost
¶
-
decoded
¶
-
feed_dict
(dataset: neuralmonkey.dataset.Dataset, train: bool = False) → typing.Dict[tensorflow.python.framework.ops.Tensor, typing.Any]¶
-
input_lengths
¶
-
logits
¶
-
runtime_loss
¶
-
train_loss
¶
-
train_mode
¶
-
train_targets
¶
-
neuralmonkey.decoders.decoder module¶
-
class
neuralmonkey.decoders.decoder.
Decoder
(encoders: typing.List[neuralmonkey.model.stateful.Stateful], vocabulary: neuralmonkey.vocabulary.Vocabulary, data_id: str, name: str, max_output_len: int, dropout_keep_prob: float = 1.0, rnn_size: int = None, embedding_size: int = None, output_projection: typing.Union[typing.Tuple[typing.Callable[[tensorflow.python.framework.ops.Tensor, tensorflow.python.framework.ops.Tensor, typing.List[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor], int], typing.Callable[[tensorflow.python.framework.ops.Tensor, tensorflow.python.framework.ops.Tensor, typing.List[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor]] = None, encoder_projection: typing.Callable[[tensorflow.python.framework.ops.Tensor, int, typing.List[neuralmonkey.model.stateful.Stateful]], tensorflow.python.framework.ops.Tensor] = None, attentions: typing.List[neuralmonkey.attention.base_attention.BaseAttention] = None, embeddings_source: neuralmonkey.model.sequence.EmbeddedSequence = None, attention_on_input: bool = True, rnn_cell: str = 'GRU', conditional_gru: bool = False, save_checkpoint: str = None, load_checkpoint: str = None) → None¶ Bases:
neuralmonkey.model.model_part.ModelPart
Decoder class.
A class that manages parts of the computation graph that are used for the decoding.
-
batch_size
¶
-
cost
¶
-
decoded
¶
-
decoding_b
¶
-
decoding_w
¶
-
embed_input_symbol
(*args) → tensorflow.python.framework.ops.Tensor¶
-
embedding_matrix
¶ Variables and operations for embedding of input words.
If we are reusing word embeddings, this function takes the embedding matrix from the first encoder
-
feed_dict
(dataset: neuralmonkey.dataset.Dataset, train: bool = False) → typing.Dict[tensorflow.python.framework.ops.Tensor, typing.Any]¶ Populate the feed dictionary for the decoder object.
Parameters: - dataset – The dataset to use for the decoder.
- train – Boolean flag, telling whether this is a training run
-
get_body
(train_mode: bool, sample: bool = False) → typing.Callable¶
-
get_initial_loop_state
() → neuralmonkey.decoders.decoder.LoopState¶
-
go_symbols
¶
-
initial_state
¶ Compute initial decoder state.
The part of the computation graph that computes the initial state of the decoder.
-
input_plus_attention
(*args) → tensorflow.python.framework.ops.Tensor¶ Merge input and previous attentions.
Input and previous attentions are merged into a single vector of the size fo embedding.
-
loop_continue_criterion
(*args) → tensorflow.python.framework.ops.Tensor¶
-
runtime_logits
¶
-
runtime_logprobs
¶
-
runtime_loop_result
¶
-
runtime_loss
¶
-
runtime_mask
¶
-
runtime_rnn_states
¶
-
train_inputs
¶
-
train_logits
¶
-
train_logprobs
¶
-
train_loss
¶
-
train_mode
¶
-
train_padding
¶
-
train_xents
¶
-
-
class
neuralmonkey.decoders.decoder.
LoopState
(step, input_symbol, train_inputs, prev_rnn_state, prev_rnn_output, rnn_outputs, prev_logits, logits, prev_contexts, mask, finished, attention_loop_states)¶ Bases:
tuple
-
attention_loop_states
¶ Alias for field number 11
-
finished
¶ Alias for field number 10
-
input_symbol
¶ Alias for field number 1
-
logits
¶ Alias for field number 7
-
mask
¶ Alias for field number 9
-
prev_contexts
¶ Alias for field number 8
-
prev_logits
¶ Alias for field number 6
-
prev_rnn_output
¶ Alias for field number 4
-
prev_rnn_state
¶ Alias for field number 3
-
rnn_outputs
¶ Alias for field number 5
-
step
¶ Alias for field number 0
-
train_inputs
¶ Alias for field number 2
-
neuralmonkey.decoders.encoder_projection module¶
Encoder Projection Module.
This module contains different variants of projection of encoders into the initial state of the decoder.
Encoder projections are specified in the configuration file. Each encoder
projection function has a unified type EncoderProjection
, which is a
callable that takes three arguments:
train_mode
– boolean tensor specifying whether the train mode is onrnn_size
– the size of the resulting initial stateencoders
– a list ofStateful
objects used as the encoders.
To enable further parameterization of encoder projection functions, one can use higher-order functions.
-
neuralmonkey.decoders.encoder_projection.
concat_encoder_projection
(train_mode: tensorflow.python.framework.ops.Tensor, rnn_size: int = None, encoders: typing.List[neuralmonkey.model.stateful.Stateful] = None) → tensorflow.python.framework.ops.Tensor¶ Concatenate the encoded values of the encoders.
-
neuralmonkey.decoders.encoder_projection.
empty_initial_state
(train_mode: tensorflow.python.framework.ops.Tensor, rnn_size: int, encoders: typing.List[neuralmonkey.model.stateful.Stateful] = None) → tensorflow.python.framework.ops.Tensor¶ Return an empty vector.
-
neuralmonkey.decoders.encoder_projection.
linear_encoder_projection
(dropout_keep_prob: float) → typing.Callable[[tensorflow.python.framework.ops.Tensor, int, typing.List[neuralmonkey.model.stateful.Stateful]], tensorflow.python.framework.ops.Tensor]¶ Return a linear encoder projection.
Return a projection function which applies dropout on concatenated encoder final states and returns a linear projection to a rnn_size-sized tensor.
Parameters: dropout_keep_prob – The dropout keep probability
-
neuralmonkey.decoders.encoder_projection.
nematus_projection
(dropout_keep_prob: float = 1.0) → typing.Callable[[tensorflow.python.framework.ops.Tensor, int, typing.List[neuralmonkey.model.stateful.Stateful]], tensorflow.python.framework.ops.Tensor]¶ Return encoder projection used in Nematus.
The initial state is a dense projection with tanh activation computed on the averaged states of the encoders. Dropout is applied to the means (before the projection).
Parameters: dropout_keep_prob – The dropout keep probability.
neuralmonkey.decoders.output_projection module¶
Output Projection Module.
This module contains different variants of projection functions of decoder outputs into the logit function inputs.
Output projections are specified in the configuration file. Each output
projection function has a unified type OutputProjection
, which is a
callable that takes four arguments and returns a tensor:
prev_state
– the hidden state of the decoder.prev_output
– embedding of the previously decoded word (or train input)ctx_tensots
– a list of context vectors (for each attention object)
To enable further parameterization of output projection functions, one can use higher-order functions.
-
neuralmonkey.decoders.output_projection.
maxout_output
(maxout_size: int) → typing.Tuple[typing.Callable[[tensorflow.python.framework.ops.Tensor, tensorflow.python.framework.ops.Tensor, typing.List[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor], int]¶ Apply maxout.
Compute RNN output out of the previous state and output, and the context tensors returned from attention mechanisms, as described in the article
This function corresponds to the equations for computation the t_tilde in the Bahdanau et al. (2015) paper, on page 14, with the maxout projection, before the last linear projection.
Parameters: maxout_size – The size of the hidden maxout layer in the deep output Returns: Returns the maxout projection of the concatenated inputs
-
neuralmonkey.decoders.output_projection.
mlp_output
(layer_sizes: typing.List[int], activation: typing.Callable[[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor] = <function tanh>, dropout_keep_prob: float = 1.0) → typing.Tuple[typing.Callable[[tensorflow.python.framework.ops.Tensor, tensorflow.python.framework.ops.Tensor, typing.List[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor], int]¶ Apply a multilayer perceptron.
Compute RNN deep output using the multilayer perceptron with a specified activation function. (Pascanu et al., 2013 [https://arxiv.org/pdf/1312.6026v5.pdf])
Parameters: - layer_sizes – A list of sizes of the hiddel layers of the MLP
- dropout_keep_prob – the dropout keep probability
- activation – The activation function to use in each layer.
-
neuralmonkey.decoders.output_projection.
nematus_output
(output_size: int, dropout_keep_prob: float = 1.0) → typing.Tuple[typing.Callable[[tensorflow.python.framework.ops.Tensor, tensorflow.python.framework.ops.Tensor, typing.List[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor], int]¶ Apply nonlinear one-hidden-layer deep output.
Implementation consistent with Nematus. Can be used instead of (and is in theory equivalent to) nonlinear_output.
Projects the RNN state, embedding of the previously outputted word, and concatenation of all context vectors into a shared vector space, sums them up and apply a hyperbolic tangent activation function.
-
neuralmonkey.decoders.output_projection.
nonlinear_output
(output_size: int, activation_fn: typing.Callable[[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor] = <function tanh>) → typing.Tuple[typing.Callable[[tensorflow.python.framework.ops.Tensor, tensorflow.python.framework.ops.Tensor, typing.List[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor], int]¶
neuralmonkey.decoders.sequence_labeler module¶
-
class
neuralmonkey.decoders.sequence_labeler.
SequenceLabeler
(name: str, encoder: typing.Union[neuralmonkey.encoders.recurrent.RecurrentEncoder, neuralmonkey.encoders.facebook_conv.SentenceEncoder], vocabulary: neuralmonkey.vocabulary.Vocabulary, data_id: str, dropout_keep_prob: float = 1.0, save_checkpoint: typing.Union[str, NoneType] = None, load_checkpoint: typing.Union[str, NoneType] = None) → None¶ Bases:
neuralmonkey.model.model_part.ModelPart
Classifier assing a label to each encoder’s state.
-
cost
¶
-
decoded
¶
-
feed_dict
(dataset: neuralmonkey.dataset.Dataset, train: bool = False) → typing.Dict[tensorflow.python.framework.ops.Tensor, typing.Any]¶
-
logits
¶
-
logprobs
¶
-
runtime_loss
¶
-
train_loss
¶
-
train_mode
¶
-
train_targets
¶
-
train_weights
¶
-
neuralmonkey.decoders.sequence_regressor module¶
-
class
neuralmonkey.decoders.sequence_regressor.
SequenceRegressor
(name: str, encoders: typing.List[neuralmonkey.model.stateful.Stateful], data_id: str, layers: typing.List[int] = None, activation_fn: typing.Callable[[tensorflow.python.framework.ops.Tensor], tensorflow.python.framework.ops.Tensor] = <function relu>, dropout_keep_prob: float = 1.0, dimension: int = 1, save_checkpoint: str = None, load_checkpoint: str = None) → None¶ Bases:
neuralmonkey.model.model_part.ModelPart
A simple MLP regression over encoders.
The API pretends it is an RNN decoder which always generates a sequence of length exactly one.
-
cost
¶
-
decoded
¶
-
feed_dict
(dataset: neuralmonkey.dataset.Dataset, train: bool = False) → typing.Dict[tensorflow.python.framework.ops.Tensor, typing.Any]¶
-
predictions
¶
-
runtime_loss
¶
-
train_inputs
¶
-
train_loss
¶
-
train_mode
¶
-
neuralmonkey.decoders.word_alignment_decoder module¶
-
class
neuralmonkey.decoders.word_alignment_decoder.
WordAlignmentDecoder
(encoder: neuralmonkey.encoders.recurrent.RecurrentEncoder, decoder: neuralmonkey.decoders.decoder.Decoder, data_id: str, name: str) → None¶ Bases:
neuralmonkey.model.model_part.ModelPart
A decoder that computes soft alignment from an attentive encoder.
Loss is computed as cross-entropy against a reference alignment.
-
alignment_target
¶
-
cost
¶
-
feed_dict
(dataset: neuralmonkey.dataset.Dataset, train: bool = False) → typing.Dict[tensorflow.python.framework.ops.Tensor, typing.Any]¶
-
ref_alignment
¶
-