Skip to content

Instantly share code, notes, and snippets.

@skeeet
Forked from bzamecnik/model_summary.txt
Created May 5, 2017 05:46
Show Gist options
  • Save skeeet/b639eea7e3fc51dd03e9b69c06b2fdf1 to your computer and use it in GitHub Desktop.
Save skeeet/b639eea7e3fc51dd03e9b69c06b2fdf1 to your computer and use it in GitHub Desktop.
Residual LSTM in Keras
def make_residual_lstm_layers(input, rnn_depth, rnn_dropout):
"""
The intermediate LSTM layers return sequences, while the last returns a single element.
The input is also a sequence. In order to match the shape of input and output of the LSTM
to sum them we can do it only for all layers but the last.
"""
for i in range(rnn_depth):
return_sequences = i < rnn_depth - 1
x_rnn = LSTM(rnn_width, dropout_W=rnn_dropout, dropout_U=rnn_dropout, return_sequences=return_sequences)(input)
if return_sequences:
# residual block
x = merge([x, x_rnn], mode='sum')
else:
# last layer does not return sequences and cannot be residual
x = x_rnn
return x
@PrithivirajDamodaran
Copy link

Can you help me understand how this is a residual block ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment