AttentionBlock.py

from tensorflow_addons.layers import MultiHeadAttention

class AttentionBlock(keras.Model):
    def __init__(self, name='AttentionBlock', num_heads=2, head_size=128, ff_dim=None, dropout=0, **kwargs):
        super().__init__(name=name, **kwargs)

        if ff_dim is None:
            ff_dim = head_size

        self.attention = MultiHeadAttention(num_heads=num_heads, head_size=head_size, dropout=dropout)
        self.attention_dropout = keras.layers.Dropout(dropout)
        self.attention_norm = keras.layers.LayerNormalization(epsilon=1e-6)

        self.ff_conv1 = keras.layers.Conv1D(filters=ff_dim, kernel_size=1, activation='relu')
        # self.ff_conv2 at build()
        self.ff_dropout = keras.layers.Dropout(dropout)
        self.ff_norm = keras.layers.LayerNormalization(epsilon=1e-6)

    def build(self, input_shape):
        self.ff_conv2 = keras.layers.Conv1D(filters=input_shape[-1], kernel_size=1) 

    def call(self, inputs):
        x = self.attention([inputs, inputs])
        x = self.attention_dropout(x)
        x = self.attention_norm(inputs + x)

        x = self.ff_conv1(x)
        x = self.ff_conv2(x)
        x = self.ff_dropout(x)

        x = self.ff_norm(inputs + x)
        return x

Thanks so much for posting this work, it's a big help in understanding Attention and Time2Vec. I was experiencing errors after the first training epoch when model weights were being saved. This turned out to be due to duplicate layer names. To fix this, I just added name="conv1" and "conv2" to the Conv1D layers in the AttentionBlock code, then everything worked as expected. Thanks again!

hello author,

i have seen your post at
https://towardsdatascience.com/the-time-series-transformer-2a521a0efad3

I have some doubts. I am new to transformers. I was using rnn/lstm models data with shape (1000, 50) and reshape to time series data as
(window_or_timestep=10, feature_dim=50).
Which data shape should i use for transformers? or time2vec function?
please give me your feedback.

thank you