neuralmonkey.attention.namedtuples module

class neuralmonkey.attention.namedtuples.AttentionLoopState

Bases: neuralmonkey.attention.namedtuples.AttentionLoopState

Basic loop state of an attention mechanism.

contexts

A tensor of shape (query_time, batch, context_dim) which stores the context vectors for every decoder time step.

weights

A tensor of shape (query_time, batch, keys_len) which stores the attention distribution over the keys given the query in each decoder time step.

class neuralmonkey.attention.namedtuples.HierarchicalLoopState

Bases: neuralmonkey.attention.namedtuples.HierarchicalLoopState

Loop state of the hierarchical attention mechanism.

The input to the hierarchical attetnion is the output of a set of underlying (child) attentions. To record the inner states of the underlying attentions, we use the HierarchicalLoopState, which holds information about both the underlying attentions, and the top-level attention itself.

child_loop_states

A list of attention loop states of the underlying attention mechanisms.

loop_state

The attention loop state of the top-level attention.

class neuralmonkey.attention.namedtuples.MultiHeadLoopState

Bases: neuralmonkey.attention.namedtuples.MultiHeadLoopState

Loop state of a multi-head attention.

contexts

A tensor of shape (query_time, batch, context_dim) which stores the context vectors for every decoder time step.

head_weights

A tensor of shape (query_time, n_heads, batch, keys_len) which stores the attention distribution over the keys given the query in each decoder time step for each attention head.