Reshaping Vit256-16 output to be consistent with ViT4096-256 and following ViT-WSI

[FabianHoerst]

Hello,

I have two questions regarding your implementation for using it with MIL.

1.) My first question is about connecting Vit256-16, Vit4096-256 and ViT-WSI to build a complete network

As you state in the model walkthrough notebook for weakly supervised subtyping, input shape with pre-extracted ViT256-16 tokens must be

Input: $[M \times L \times D]$ Tensor, where:

• M: Number of (non-overlapping) $[4096 \times 4096]$ Image regions in a WSI (On Average: 38)
• L: Number of (non-overlapping) $[256 \times 256]$ Image Patches in a $[4096 \times 4096]$ Image Region (Default: 256)
• D: Embedding Dimension (Default: 384)

Now I am a little bit confused about the output shape of ViT256-16. If I take the given implementation, the output shape of ViT256-16 is $[(M*L) \times D]$. My first thought to match the new input dimension for ViT4096-256 is just to use x = x.reshape(M, L, D) , respectively x = einops.rearrange(x, '(M L) D -> M L D', M=4, L=16), for a given tensor with shape [M*L, D], but I am not sure if this correctly maintains the spatial dimensions and axis?

2.) How do you handle Multiple WSI per patient in your Framework?

Do you simply use the patient label as the label for every WSI? I have not found information about merging multiple WSI for one patient in your code (e.g. the train loop in core utils simply iterates over WSI with given labels).