replace transformerencoder with mamba

[sunxin010205]

Hi！
After replacing an eight-layer Transformer encoder with Mamba, the training loss fails to decrease. Could it be that Mamba doesn't perform as effectively as the Transformer in the diffusion model? Looking forward to your response。
Here are my codes.

mamba.txt
mdm.txt
minimamba.txt
loss_log.txt

[GuyTevet]

Hi @sunxin010205 , what is Mamba?

[sunxin010205]

嗨，曼巴是什么？

Mamba: Linear-Time Sequence Modeling with Selective State Spaces，Mamba is a new architecture proposed for the linear complexity of transformer。
When I use mamba instead of transformer encoder, other loss is normal, only loss_q3 cannot be reduced. Do you know what the situation is? Looking forward to your early reply!