Autodistill quits after 2457 images. Memory leak?

[tfriedel]

I run autodistill like this:

autodistill  --base "grounded_sam" --model_type detection --epochs 50 --upload-to-roboflow False --target "yolov8" --ontology '{"caption": "label"}' --output ./dataset -y True ./images/

The images folder contains about 10.000 images.
It stops with this message, after processing 2457 images:
fish: Job 1, 'autodistill --base "grounded_s…' terminated by signal SIGTERM (Polite quit request)

This is repeatable. I observe it consumed a lot of memory, possibly about 32 GB. My machine has 64 GB and two RTX 2080, so I suspect it may be a memory leak that stopped it, but I wasn't at the machine to see this.

Ideally it wouldn't do this and it would have a resume capability that allows you to restart it where it left off.

Edit:
I think it must be a memory leak based on this code in classification_base_model.py:

        for f_path in progress_bar:
            progress_bar.set_description(desc=f"Labeling {f_path}", refresh=True)
            image = cv2.imread(f_path)

            f_path_short = os.path.basename(f_path)
            images_map[f_path_short] = image.copy()
            detections = self.predict(f_path)
            detections_map[f_path_short] = detections

So it keeps ALL the images in a dict, even after they were processed. This is unneccesary and will cause lots of memory consumption.

[AChangXD]

I OOM'd off of a 7GB dataset (sv.DetectionDataset.from_yolo), it used all 64GB of my RAM and committed another 250GB