Skip to content

Instantly share code, notes, and snippets.

@radi-cho

radi-cho/gmlp_define_norm_proj.py

Created September 10, 2022 20:02
Show Gist options
  • Save radi-cho/b5a90bb7b9ba726707f8b9b3f61d4ac4 to your computer and use it in GitHub Desktop.
Save radi-cho/b5a90bb7b9ba726707f8b9b3f61d4ac4 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gmlp_define_norm_proj.py
self.normalize1 = layers.LayerNormalization(epsilon=1e-6)
self.normalize2 = layers.LayerNormalization(epsilon=1e-6)
self.channel_projection1 = keras.Sequential(
[
layers.Dense(units=embedding_dim * 2),
layers.ReLU(),
layers.Dropout(rate=dropout_rate),
]
)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment