SAHI Grounding DINO index out of range

[VitalyVaryvdin]

Search before asking

• I have searched the Autodistill issues and found no similar bug report.

Bug

Labeling large dataset with SAHI enabled GroundingDINO results in the following:

Traceback (most recent call last):
  File "/home/vitalyvaryvdin/projects/ml/autodistill/distill.py", line 7, in <module>
    base_model.label(
  File "/home/vitalyvaryvdin/Python/autodistill/lib64/python3.12/site-packages/autodistill/detection/detection_base_model.py", line 97, in label
    detections = slicer(image)
                 ^^^^^^^^^^^^^
  File "/home/vitalyvaryvdin/Python/autodistill/lib64/python3.12/site-packages/supervision/detection/tools/inference_slicer.py", line 109, in __call__
    detections_list.append(future.result())
                           ^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/lib64/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vitalyvaryvdin/Python/autodistill/lib64/python3.12/site-packages/supervision/detection/tools/inference_slicer.py", line 128, in _run_callback
    detections = self.callback(image_slice)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vitalyvaryvdin/Python/autodistill/lib64/python3.12/site-packages/autodistill_grounding_dino/grounding_dino_model.py", line 44, in predict
    detections = self.grounding_dino_model.predict_with_classes(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vitalyvaryvdin/Python/autodistill/lib64/python3.12/site-packages/groundingdino/util/inference.py", line 194, in predict_with_classes
    boxes, logits, phrases = predict(
                             ^^^^^^^^
  File "/home/vitalyvaryvdin/Python/autodistill/lib64/python3.12/site-packages/groundingdino/util/inference.py", line 66, in predict
    outputs = model(image[None], captions=[caption])
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vitalyvaryvdin/Python/autodistill/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vitalyvaryvdin/Python/autodistill/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vitalyvaryvdin/Python/autodistill/lib64/python3.12/site-packages/groundingdino/models/GroundingDINO/groundingdino.py", line 313, in forward
    hs, reference, hs_enc, ref_enc, init_box_proposal = self.transformer(
                                                        ^^^^^^^^^^^^^^^^^
  File "/home/vitalyvaryvdin/Python/autodistill/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vitalyvaryvdin/Python/autodistill/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vitalyvaryvdin/Python/autodistill/lib64/python3.12/site-packages/groundingdino/models/GroundingDINO/transformer.py", line 301, in forward
    topk_proposals = torch.topk(topk_logits, topk, dim=1)[1]  # bs, nq
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: selected index k out of range

Happens after about 20 images

Environment

• autodistill: 0.1.26
• autodistill-grounding-dino: 0.1.4
• torch: 2.3.0
• supervision: 0.20.0
• Python: 3.12.3
• OS: Fedora 40

Minimal Reproducible Example

from autodistill.detection import CaptionOntology
from autodistill_grounding_dino import GroundingDINO
from autodistill_yolov8 import YOLOv8

base_model = GroundingDINO(ontology=CaptionOntology({"license plate": "license-plate"}))

base_model.label(
	input_folder="../datasets/processed",
	output_folder="../datasets/processed_distilled",
	extension=".jpg"
)

The dataset is quite big to share, but if needed, I can do that

Additional

Size of the image it happens on is 2005x1504 if that matters

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[capjamesg]

It looks like this has been reported as an issue in Grounding DINO: IDEA-Research/GroundingDINO#181. Can you try to run inference on the single image where labeling stops? You can do this using base_model.predict("image.jpeg") and see if you can replicate the bug on a single image?

[VitalyVaryvdin]

The issue appears only once SAHI is enabled, whole folder labelling completed successfully without it enabled