transformations in MiniViT paper

[gudrb]

Hello, I have a question about the transformations in the MiniViT paper.

I could find the first transformation (implemented in the MiniAttention class) in the code:

Cream/MiniViT/Mini-DeiT/mini_vision_transformer.py

Line 104 in 4a13c40

attn = self.conv_l(attn)

However, i couldn't find the second transformation in the code (which should be before or inside the MLP in the MiniBlock class)

Cream/MiniViT/Mini-DeiT/mini_vision_transformer.py

Line 137 in 4a13c40

class MiniBlock(nn.Module):

Could you please let me know where the second transformation is?

[wkcn]

Hi @gudrb , thanks for your attention to our work!

In Mini-DeiT, the transformation for MLP is the relative position encoding

Cream/MiniViT/Mini-DeiT/mini_vision_transformer.py

Line 117 in 4a13c40

out += self.rpe_v(attn)

In Mini-Swin, the transformation for MLP is the depth-wise convolution layer

Cream/MiniViT/Mini-Swin/models/swin_transformer_minivit.py

Line 275 in 4a13c40

self.local_conv_list = nn.ModuleList()

[gudrb]

On the MiniViT paper,

We make several modifi�cations on DeiT: First, we remove the [class] token. The
model is attached with a global average pooling layer and a
fully-connected layer for image classification. We also utilize relative position encoding to introduce inductive bias to
boost the model convergence [52,59]. Finally, based on our
observation that transformation for FFN only brings limited
performance gains in DeiT, we remove the block to speed up
both training and inference.

-> Does this mean that in MiniDeiT model, IRPE is utilized (for the value), and the MLP transformation is removed, leaving only the attention transformation?

[wkcn]

On the MiniViT paper,

We make several modifi�cations on DeiT: First, we remove the [class] token. The model is attached with a global average pooling layer and a fully-connected layer for image classification. We also utilize relative position encoding to introduce inductive bias to boost the model convergence [52,59]. Finally, based on our observation that transformation for FFN only brings limited performance gains in DeiT, we remove the block to speed up both training and inference.

-> Does this mean that in MiniDeiT model, IRPE is utilized (for the value), and the MLP transformation is removed, leaving only the attention transformation?

Yes. I correct my statement. There is no transformation for FFN in Mini-DeiT. iRPE is utilized for only the key.

Cream/MiniViT/Mini-DeiT/mini_vision_transformer.py

Line 97 in 4a13c40

attn += self.rpe_k(q)

Cream/MiniViT/Mini-DeiT/mini_deit_models.py

Line 17 in 4a13c40

rpe_on='k',