neuralmonkey.nn.ortho_gru_cell module

class neuralmonkey.nn.ortho_gru_cell.NematusGRUCell(rnn_size, use_state_bias=False, use_input_bias=True)

Bases: tensorflow.python.ops.rnn_cell_impl.GRUCell

Nematus implementation of gated recurrent unit cell.

The main difference is the order in which the gating functions and linear projections are applied to the hidden state.

The math is equivalent, in practice there are differences due to float precision errors.

__init__(rnn_size, use_state_bias=False, use_input_bias=True)

call(inputs, state)

Gated recurrent unit (GRU) with nunits cells.

class neuralmonkey.nn.ortho_gru_cell.OrthoGRUCell(num_units, activation=None, reuse=None)

Bases: tensorflow.python.ops.rnn_cell_impl.GRUCell

Classic GRU cell but initialized using random orthogonal matrices.

__init__(num_units, activation=None, reuse=None)

neuralmonkey.nn.ortho_gru_cell.orthogonal_initializer()

Return an orthogonal initializer.

Random orthogonal matrix is byproduct of singular value decomposition applied on a matrix initialized with normal distribution.

The initializer works with 2D square matrices and matrices that can be splitted along axis 1 to several 2D matrices. In the latter case, each submatrix is initialized independently and the resulting orthogonal matrices are concatenated along axis 1.

Note this is a higher order function in order to mimic the tensorflow initializer API.