Geometric interpolation ViT

[hemangchawla]

Hi,

In the interpolation for ViT, you have:

SimMIM/utils.py

Line 226 in bec329f

dst_num_pos, _ = model.state_dict()[key].size()

However, the keys are from:

SimMIM/utils.py

Line 218 in bec329f

all_keys = list(checkpoint_model.keys())

which includes:

SimMIM/utils.py

Lines 213 to 215 in bec329f

	for i in range(num_layers):
	checkpoint_model["blocks.%d.attn.relative_position_bias_table" % i] = rel_pos_bias.clone()
	checkpoint_model.pop("rel_pos_bias.relative_position_bias_table")

However, these keys are not present in the model, and give an error.

I want to use the ViT base models for downstream tasks.

Could you please tell me if I am missing something in this?

Thanks.

[hemangchawla]

I figured that the function has been written keeping the classification finetuning in mind.
Hence you have:

use_abs_pos_emb=False, use_rel_pos_bias=True use_shared_rel_pos_bias=False

as settings from the simmim_finetune__vit_base__img224__800ep.yaml.

However for a dataset where the image is not a square, it doesn't work:

SimMIM/utils.py

Lines 228 to 229 in bec329f

	if dst_patch_shape[0] != dst_patch_shape[1]:
	raise NotImplementedError()

Would you have any idea on how to go about this?