mmsegmentation Siamese network 孪生网络

[yao-zq]

请问mmsegmentation如何实现孪生网络，我想做遥感变化检测，计划将图像分别以单独的分支输入网络，而不是合并成一个文件再去输入。谢谢。
Please ask mmsegmentation how to implement siamese networks. I want to do remote sensing change detection and plan to feed the images into the network as separate branches instead of merging them into a single file. thank you.

[Zoulinx]

Currently, mmseg does not include the siamese network. However, it is worth noting that mmseg does not concatenate the change detection images into a "single file", but stacks them up in Transform. This means that to implement a simple twin network, for example, with a batch size of 4, you can modify the stacked tensor size from 4x6x512x512 to 8x3x512x512 to achieve dual-stream input. Finally, connect the feature maps before the decoder_head.

[yao-zq]

Currently, mmseg does not include the siamese network. However, it is worth noting that mmseg does not concatenate the change detection images into a "single file", but stacks them up in Transform. This means that to implement a simple twin network, for example, with a batch size of 4, you can modify the stacked tensor size from 4x6x512x512 to 8x3x512x512 to achieve dual-stream input. Finally, connect the feature maps before the decoder_head.

谢谢您的回答，我的任务是做双时相遥感图像的变化检测，需要比较相同位置的图像对。如果首先输入8x3x512x512，然后拆分为两个batchsize为4的特征组，由于数据是随机抽取的，如何确保两个特征组的数据是来自相同的图像对呢？如果可能的话，我还是想直接使用孪生的骨干网络，但是不知道如何实现，想请教一下如何修改代码？谢谢您。

[Zoulinx]

Let me elaborate further.

1.Currently, in mmseg's strategy for change detection tasks, data is randomly sampled from "two temporal phases" simultaneously, meaning that each sampling results in an image pair.

2.In #3597, mmseg fixed a bug where the same data augmentation is now applied to both temporal phases, yielding the same effect as dual-stream input.

3.The input for change detection data is not 8x3x512x512 but 4x6x512x512(#2903). The input for twin networks is 8x3x512x512. The transformation involved requires reallocating inputs in the backbone network, the specific approach of which depends on the backbone network you're using.

Additionally, I conducted extensive testing and comparison between the twin network strategy and the direct stacking image strategy in terms of accuracy differences. In fact, the direct stacking strategy achieved the highest accuracy across all networks I tested, which is also why I haven't uploaded the twin network code.

[yao-zq]

Let me elaborate further.

1.Currently, in mmseg's strategy for change detection tasks, data is randomly sampled from "two temporal phases" simultaneously, meaning that each sampling results in an image pair.

2.In #3597, mmseg fixed a bug where the same data augmentation is now applied to both temporal phases, yielding the same effect as dual-stream input.

3.The input for change detection data is not 8x3x512x512 but 4x6x512x512(#2903). The input for twin networks is 8x3x512x512. The transformation involved requires reallocating inputs in the backbone network, the specific approach of which depends on the backbone network you're using.

Additionally, I conducted extensive testing and comparison between the twin network strategy and the direct stacking image strategy in terms of accuracy differences. In fact, the direct stacking strategy achieved the highest accuracy across all networks I tested, which is also why I haven't uploaded the twin network code.

谢谢您的回答和指导，我接下来仔细学习mmseg的代码。不过关于最后一点，我还是存在一些疑问，就是最近几年的遥感变化检测论文都表明孪生网络效果优于双时相数据合并成一个作为输入的方式，并且大家都试图构建更好的孪生结构网络。期待您的回复。

[Zoulinx]

@yao-zq
The answer to this question may require conducting comparative experiments with different algorithms in mmseg. For specific steps, refer to mmseg's support for the levir-cd dataset. Wishing you a pleasant experience with it.