For the modulator and cross-modulator

[jm20041050]

Thanks for the new code update.
I have a question about the modulator and cross-modulator in decoder.

In the latest code (2022-07-04), the modulator layer is an Embedding layer.

        if modulator:
            self.modulator = nn.Embedding(win_size*win_size, dim) # modulator
        else:
            self.modulator = None

        if cross_modulator:
            self.cross_modulator = nn.Embedding(win_size*win_size, dim) # cross_modulator
            self.cross_attn = Attention(dim,num_heads,qkv_bias=qkv_bias, qk_scale=qk_scale, attn_drop=attn_drop, proj_drop=drop,
                    token_projection=token_projection,)
            self.norm_cross = norm_layer(dim)
        else:
            self.cross_modulator = None

In the forward function, you just use the weights of modulator to add it with x_windows

        if self.cross_modulator is not None:
            shortcut = x
            x_cross = self.norm_cross(x)
            x_cross = self.cross_attn(x, self.cross_modulator.weight)
            x = shortcut + x_cross

        # with_modulator
        if self.modulator is not None:
            wmsa_in = self.with_pos_embed(x_windows,self.modulator.weight)
        else:
            wmsa_in = x_windows

For the modulator, your code just add the x_windows tensor with the weight of Embedding layer.
I wonder that layers can be learned in this case.
Is this your intended implementation?