Skip to content

Instantly share code, notes, and snippets.

@isauravmanitripathi
Created October 10, 2022 15:57
Show Gist options
  • Save isauravmanitripathi/40a6babc56d8ba245cd8f84db76d0f87 to your computer and use it in GitHub Desktop.
Save isauravmanitripathi/40a6babc56d8ba245cd8f84db76d0f87 to your computer and use it in GitHub Desktop.
...
def call(self, x, encoder_output, lookahead_mask, padding_mask, training):
# Multi-head attention layer
multihead_output1 = self.multihead_attention1(x, x, x, lookahead_mask)
# Expected output shape = (batch_size, sequence_length, d_model)
# Add in a dropout layer
multihead_output1 = self.dropout1(multihead_output1, training=training)
# Followed by an Add & Norm layer
addnorm_output1 = self.add_norm1(x, multihead_output1)
# Expected output shape = (batch_size, sequence_length, d_model)
# Followed by another multi-head attention layer
multihead_output2 = self.multihead_attention2(addnorm_output1, encoder_output, encoder_output, padding_mask)
# Add in another dropout layer
multihead_output2 = self.dropout2(multihead_output2, training=training)
# Followed by another Add & Norm layer
addnorm_output2 = self.add_norm1(addnorm_output1, multihead_output2)
# Followed by a fully connected layer
feedforward_output = self.feed_forward(addnorm_output2)
# Expected output shape = (batch_size, sequence_length, d_model)
# Add in another dropout layer
feedforward_output = self.dropout3(feedforward_output, training=training)
# Followed by another Add & Norm layer
return self.add_norm3(addnorm_output2, feedforward_output)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment