-
-
Save skeeet/b639eea7e3fc51dd03e9b69c06b2fdf1 to your computer and use it in GitHub Desktop.
Residual LSTM in Keras
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| def make_residual_lstm_layers(input, rnn_depth, rnn_dropout): | |
| """ | |
| The intermediate LSTM layers return sequences, while the last returns a single element. | |
| The input is also a sequence. In order to match the shape of input and output of the LSTM | |
| to sum them we can do it only for all layers but the last. | |
| """ | |
| for i in range(rnn_depth): | |
| return_sequences = i < rnn_depth - 1 | |
| x_rnn = LSTM(rnn_width, dropout_W=rnn_dropout, dropout_U=rnn_dropout, return_sequences=return_sequences)(input) | |
| if return_sequences: | |
| # residual block | |
| x = merge([x, x_rnn], mode='sum') | |
| else: | |
| # last layer does not return sequences and cannot be residual | |
| x = x_rnn | |
| return x |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Can you help me understand how this is a residual block ?