neuralmonkey.nn.highway module

Module implementing the highway networks.

neuralmonkey.nn.highway.highway(inputs, activation=<function relu>, scope='HighwayNetwork')

Create a single highway layer.

y = H(x, Wh) * T(x, Wt) + x * C(x, Wc)

where:

C(x, Wc) = 1 - T(x, Wt)

Parameters:

• inputs – A tensor or list of tensors. It should be 2D tensors with equal length in the first dimension (batch size)
• activation – Activation function of the linear part of the formula H(x, Wh).
• scope – The name of the scope used for the variables.

Returns:

A tensor of shape tf.shape(inputs)