KD for foundation models like OWL-ViT and Grounding Dino foundation models

[solomonmanuelraj]

Hi team,

I am working on the vision foundation model for object detection. Need your help in KD for vision foundation models of OWL-ViT and Grounding Dino models.

Like to know if there is any existing demo/example codes / links are available for the reference.

thanks for your help

[yoshitomo-matsubara]

Hi @solomonmanuelraj

The following script may be a good starting point.
https://github.com/yoshitomo-matsubara/torchdistill/tree/main/examples/torchvision

Most importantly, what dataset and KD method are in your mind for the project?
Note that it is difficult in general to apply knowledge distillation to object detection tasks, especially if it's a end-to-end manner, as teacher and student models produce different number of bounding boxes and it's difficult to have one's produced bounding boxes match those from the other, which would be required for computing loss values to be minimized in that case.

[yoshitomo-matsubara]

What KD methods are you considering? As I described above, it is very difficult to apply end-to-end KD methods to object detection model training

[solomonmanuelraj]

Hi,
I am new to KD. You can suggest me a KD method for this problem statement to start with.

my first goal is how to apply KD for the foundation model?

the initial results i can keep it as a base results. continuously i can try different options based on your suggestion.

idea is starting with a method and getting the initial results.

[yoshitomo-matsubara]

I suggest you start with forgetting about "foundation models" and seeing them as just transformer-based models because applying KD to the transformer-based models is basically the same as applying KD to non-transformer-based models (let's say CNN-based models), unless you want to take some advantage of pretrained transformer-based models.

Then, explore and implement existing KD methods for object detection models yourself. I repeat that it is very challenging to apply end-to-end KD methods to object detection model training, and I am not aware of any such solid KD methods for object detection models.

[solomonmanuelraj]

Hi,
thanks for your quick response.

I am starting with your example code for object detection with Resnet 50 to Resnet 18 model with COCO dataset.

Is your doc contains the following information / any reference paper / link ? KD_Method used, base model name, based model accuracy, base model response time, base model memory size, target model name, target model accuracy, target model response time & target model memory size details. It will help to measure the impact of KD. Unfortunately combined details are not available in a single table to compare the impact of KD.

[yoshitomo-matsubara]

Check papers in README