Multiclass MOT #927
Labels
question
Further information is requested
Comments
|
It would be great if somebody could share a multi-class dataset or at least a very small subset of it. Like 5 images in each sequence would be enough. So that I can fix this. Otherwise I have no way of moving forward with this issue. |
|
Added a section for multi-class evaluation here: https://github.com/mikel-brostrom/yolo_tracking/wiki/How-to-evaluate-on-custom-tracking-dataset, that was missing. Adapt accordingly. Let me know if it works for you 😄 |
|
The section works for me. But I have one question and one confusion. In val.py where --classes arguments is passet it is "--classes", str(0) . How to pass class ids into in. I have used GT file in format |
|
While using it for Multiclass I didn't changed val.py line 204. I passed classes in format '--classes 0 1 2 3 4 5 ' in line 203 of track.py |
On it |
|
Enabled multi-class passing in val here: e3c242d |
|
|
Hi one question again what is exactly the format to be passed in --classes is it should be "--classes 0 1 2 3 " or "--classes 0,1,2" |
|
And another regarding IDs. I am using VisDrone dataset which has IDs But GT has the original IDs so will it work okay or I have to adjust Gt |
@mikel-brostrom I built a custom dataset VEHICLE.zip on coco classes format using CVAT tool and saving annotations on MOT 1.1 CVAT format. The labeled video contains 27 frames with 4 classes: car, person, motorcycle, and truck. I ran all benchmark pipelines on it and I got the following results: MOT_results.txt. I ran val.py using the following CLI: python examples/val.py --yolo-model yolov8x --benchmark VEHICLE --split test --tracking-method strongsort --classes 0 2 3 5 7after extract VEHICLE.zip dataset to MOT_results.txt shows a strange metrics results: all classes have the same HOTA, MOTA, and IDF1 values when using a multiclass dataset as follows below (to car and truck classes): |
Wow! Thanks so much for this @Aandre99. Can finally debug custom dataset evaluation 🚀. I am traveling today but will try to find time tomorrow to have a look at it. |
|
Yup, can reproduce after some changes in Eval Config:
USE_PARALLEL : True
NUM_PARALLEL_CORES : 4
BREAK_ON_ERROR : True
RETURN_ON_ERROR : False
LOG_ON_ERROR : /home/mikel.brostrom/yolov8_tracking/examples/val_utils/error_log.txt
PRINT_RESULTS : True
PRINT_ONLY_COMBINED : False
PRINT_CONFIG : True
TIME_PROGRESS : True
DISPLAY_LESS_PROGRESS : False
OUTPUT_SUMMARY : True
OUTPUT_EMPTY_CLASSES : True
OUTPUT_DETAILED : True
PLOT_CURVES : True
MotChallenge2DBox Config:
PRINT_CONFIG : True
GT_FOLDER : /home/mikel.brostrom/yolov8_tracking/examples/val_utils/data/VEHICLE/test
TRACKERS_FOLDER : /home/mikel.brostrom/yolov8_tracking/examples/runs/val/exp63
OUTPUT_FOLDER : None
TRACKERS_TO_EVAL : ['labels']
CLASSES_TO_EVAL : ['pedestrian', 'car', 'motorcycle', 'bus', 'truck']
BENCHMARK :
SPLIT_TO_EVAL : train
INPUT_AS_ZIP : False
DO_PREPROC : False
TRACKER_SUB_FOLDER :
OUTPUT_SUB_FOLDER :
TRACKER_DISPLAY_NAMES : None
SEQMAP_FOLDER : None
SEQMAP_FILE : None
SEQ_INFO : {'VEHICLE-01': None}
GT_LOC_FORMAT : {gt_folder}/{seq}/gt/gt.txt
SKIP_SPLIT_FOL : True
CLEAR Config:
METRICS : ['HOTA', 'CLEAR', 'Identity']
THRESHOLD : 0.5
PRINT_CONFIG : True
Identity Config:
METRICS : ['HOTA', 'CLEAR', 'Identity']
THRESHOLD : 0.5
PRINT_CONFIG : True
Evaluating 1 tracker(s) on 1 sequence(s) for 5 class(es) on MotChallenge2DBox dataset using the following metrics: HOTA, CLEAR, Identity, Count
Evaluating labels
All sequences for labels finished in 0.07 seconds
HOTA: labels-pedestrian HOTA DetA AssA DetRe DetPr AssRe AssPr LocA OWTA HOTA(0) LocA(0) HOTALocA(0)
VEHICLE-01 54.538 42.196 70.535 43.594 81.806 72.981 84.335 83.877 55.446 66.792 81.102 54.17
COMBINED 54.538 42.196 70.535 43.594 81.806 72.981 84.335 83.877 55.446 66.792 81.102 54.17
CLEAR: labels-pedestrian MOTA MOTP MODA CLR_Re CLR_Pr MTR PTR MLR sMOTA CLR_TP CLR_FN CLR_FP IDSW MT PT ML Frag
VEHICLE-01 50 81.539 50.658 51.974 97.531 33.333 33.333 33.333 40.405 79 73 2 1 2 2 2 6
COMBINED 50 81.539 50.658 51.974 97.531 33.333 33.333 33.333 40.405 79 73 2 1 2 2 2 6
Identity: labels-pedestrian IDF1 IDR IDP IDTP IDFN IDFP
VEHICLE-01 66.953 51.316 96.296 78 74 3
COMBINED 66.953 51.316 96.296 78 74 3
Count: labels-pedestrian Dets GT_Dets IDs GT_IDs
VEHICLE-01 81 152 6 6
COMBINED 81 152 6 6
HOTA: labels-car HOTA DetA AssA DetRe DetPr AssRe AssPr LocA OWTA HOTA(0) LocA(0) HOTALocA(0)
VEHICLE-01 54.538 42.196 70.535 43.594 81.806 72.981 84.335 83.877 55.446 66.792 81.102 54.17
COMBINED 54.538 42.196 70.535 43.594 81.806 72.981 84.335 83.877 55.446 66.792 81.102 54.17
CLEAR: labels-car MOTA MOTP MODA CLR_Re CLR_Pr MTR PTR MLR sMOTA CLR_TP CLR_FN CLR_FP IDSW MT PT ML Frag
VEHICLE-01 50 81.539 50.658 51.974 97.531 33.333 33.333 33.333 40.405 79 73 2 1 2 2 2 6
COMBINED 50 81.539 50.658 51.974 97.531 33.333 33.333 33.333 40.405 79 73 2 1 2 2 2 6
Identity: labels-car IDF1 IDR IDP IDTP IDFN IDFP
VEHICLE-01 66.953 51.316 96.296 78 74 3
COMBINED 66.953 51.316 96.296 78 74 3
Count: labels-car Dets GT_Dets IDs GT_IDs
VEHICLE-01 81 152 6 6
COMBINED 81 152 6 6
HOTA: labels-motorcycle HOTA DetA AssA DetRe DetPr AssRe AssPr LocA OWTA HOTA(0) LocA(0) HOTALocA(0)
VEHICLE-01 54.538 42.196 70.535 43.594 81.806 72.981 84.335 83.877 55.446 66.792 81.102 54.17
COMBINED 54.538 42.196 70.535 43.594 81.806 72.981 84.335 83.877 55.446 66.792 81.102 54.17
CLEAR: labels-motorcycle MOTA MOTP MODA CLR_Re CLR_Pr MTR PTR MLR sMOTA CLR_TP CLR_FN CLR_FP IDSW MT PT ML Frag
VEHICLE-01 50 81.539 50.658 51.974 97.531 33.333 33.333 33.333 40.405 79 73 2 1 2 2 2 6
COMBINED 50 81.539 50.658 51.974 97.531 33.333 33.333 33.333 40.405 79 73 2 1 2 2 2 6
Identity: labels-motorcycle IDF1 IDR IDP IDTP IDFN IDFP
VEHICLE-01 66.953 51.316 96.296 78 74 3
COMBINED 66.953 51.316 96.296 78 74 3
Count: labels-motorcycle Dets GT_Dets IDs GT_IDs
VEHICLE-01 81 152 6 6
COMBINED 81 152 6 6
HOTA: labels-bus HOTA DetA AssA DetRe DetPr AssRe AssPr LocA OWTA HOTA(0) LocA(0) HOTALocA(0)
VEHICLE-01 54.538 42.196 70.535 43.594 81.806 72.981 84.335 83.877 55.446 66.792 81.102 54.17
COMBINED 54.538 42.196 70.535 43.594 81.806 72.981 84.335 83.877 55.446 66.792 81.102 54.17
CLEAR: labels-bus MOTA MOTP MODA CLR_Re CLR_Pr MTR PTR MLR sMOTA CLR_TP CLR_FN CLR_FP IDSW MT PT ML Frag
VEHICLE-01 50 81.539 50.658 51.974 97.531 33.333 33.333 33.333 40.405 79 73 2 1 2 2 2 6
COMBINED 50 81.539 50.658 51.974 97.531 33.333 33.333 33.333 40.405 79 73 2 1 2 2 2 6
Identity: labels-bus IDF1 IDR IDP IDTP IDFN IDFP
VEHICLE-01 66.953 51.316 96.296 78 74 3
COMBINED 66.953 51.316 96.296 78 74 3
Count: labels-bus Dets GT_Dets IDs GT_IDs
VEHICLE-01 81 152 6 6
COMBINED 81 152 6 6
HOTA: labels-truck HOTA DetA AssA DetRe DetPr AssRe AssPr LocA OWTA HOTA(0) LocA(0) HOTALocA(0)
VEHICLE-01 54.538 42.196 70.535 43.594 81.806 72.981 84.335 83.877 55.446 66.792 81.102 54.17
COMBINED 54.538 42.196 70.535 43.594 81.806 72.981 84.335 83.877 55.446 66.792 81.102 54.17
CLEAR: labels-truck MOTA MOTP MODA CLR_Re CLR_Pr MTR PTR MLR sMOTA CLR_TP CLR_FN CLR_FP IDSW MT PT ML Frag
VEHICLE-01 50 81.539 50.658 51.974 97.531 33.333 33.333 33.333 40.405 79 73 2 1 2 2 2 6
COMBINED 50 81.539 50.658 51.974 97.531 33.333 33.333 33.333 40.405 79 73 2 1 2 2 2 6
Identity: labels-truck IDF1 IDR IDP IDTP IDFN IDFP
VEHICLE-01 66.953 51.316 96.296 78 74 3
COMBINED 66.953 51.316 96.296 78 74 3
Count: labels-truck Dets GT_Dets IDs GT_IDs
VEHICLE-01 81 152 6 6
COMBINED 81 152 6 6 |
|
My import os
import csv
import configparser
import numpy as np
from scipy.optimize import linear_sum_assignment
from ._base_dataset import _BaseDataset
from .. import utils
from .. import _timing
from ..utils import TrackEvalException
class MotChallenge2DBox(_BaseDataset):
"""Dataset class for MOT Challenge 2D bounding box tracking"""
@staticmethod
def get_default_dataset_config():
"""Default class config values"""
code_path = utils.get_code_path()
default_config = {
'GT_FOLDER': os.path.join(code_path, 'data/gt/mot_challenge/'), # Location of GT data
'TRACKERS_FOLDER': os.path.join(code_path, 'data/trackers/mot_challenge/'), # Trackers location
'OUTPUT_FOLDER': None, # Where to save eval results (if None, same as TRACKERS_FOLDER)
'TRACKERS_TO_EVAL': None, # Filenames of trackers to eval (if None, all in folder)
'CLASSES_TO_EVAL': ['pedestrian', 'car', 'motorcycle', 'bus', 'truck'], # Valid: ['pedestrian']
'BENCHMARK': 'MOT17', # Valid: 'MOT17', 'MOT16', 'MOT20', 'MOT15'
'SPLIT_TO_EVAL': 'train', # Valid: 'train', 'test', 'all'
'INPUT_AS_ZIP': False, # Whether tracker input files are zipped
'PRINT_CONFIG': True, # Whether to print current config
'DO_PREPROC': False, # Whether to perform preprocessing (never done for MOT15)
'TRACKER_SUB_FOLDER': 'data', # Tracker files are in TRACKER_FOLDER/tracker_name/TRACKER_SUB_FOLDER
'OUTPUT_SUB_FOLDER': '', # Output files are saved in OUTPUT_FOLDER/tracker_name/OUTPUT_SUB_FOLDER
'TRACKER_DISPLAY_NAMES': None, # Names of trackers to display, if None: TRACKERS_TO_EVAL
'SEQMAP_FOLDER': None, # Where seqmaps are found (if None, GT_FOLDER/seqmaps)
'SEQMAP_FILE': None, # Directly specify seqmap file (if none use seqmap_folder/benchmark-split_to_eval)
'SEQ_INFO': None, # If not None, directly specify sequences to eval and their number of timesteps
'GT_LOC_FORMAT': '{gt_folder}/{seq}/gt/gt.txt', # '{gt_folder}/{seq}/gt/gt.txt'
'SKIP_SPLIT_FOL': False, # If False, data is in GT_FOLDER/BENCHMARK-SPLIT_TO_EVAL/ and in
# TRACKERS_FOLDER/BENCHMARK-SPLIT_TO_EVAL/tracker/
# If True, then the middle 'benchmark-split' folder is skipped for both.
}
return default_config
def __init__(self, config=None):
"""Initialise dataset, checking that all required files are present"""
super().__init__()
# Fill non-given config values with defaults
self.config = utils.init_config(config, self.get_default_dataset_config(), self.get_name())
self.benchmark = self.config['BENCHMARK']
gt_set = self.config['BENCHMARK'] + '-' + self.config['SPLIT_TO_EVAL']
self.gt_set = gt_set
if not self.config['SKIP_SPLIT_FOL']:
split_fol = gt_set
else:
split_fol = ''
self.gt_fol = os.path.join(self.config['GT_FOLDER'], split_fol)
self.tracker_fol = os.path.join(self.config['TRACKERS_FOLDER'], split_fol)
self.should_classes_combine = False
self.use_super_categories = False
self.data_is_zipped = self.config['INPUT_AS_ZIP']
self.do_preproc = self.config['DO_PREPROC']
self.output_fol = self.config['OUTPUT_FOLDER']
if self.output_fol is None:
self.output_fol = self.tracker_fol
self.tracker_sub_fol = self.config['TRACKER_SUB_FOLDER']
self.output_sub_fol = self.config['OUTPUT_SUB_FOLDER']
# Get classes to eval
self.valid_classes = ['pedestrian', 'car', 'motorcycle', 'bus', 'truck']
self.class_list = [cls.lower() if cls.lower() in self.valid_classes else None
for cls in self.config['CLASSES_TO_EVAL']]
if not all(self.class_list):
raise TrackEvalException('Attempted to evaluate an invalid class. Only pedestrian class is valid.')
self.class_name_to_class_id = {'pedestrian': 0, 'car': 2, 'motorcycle': 3, 'bus': 5, 'truck': 7}
self.valid_class_numbers = list(self.class_name_to_class_id.values())
# Get sequences to eval and check gt files exist
self.seq_list, self.seq_lengths = self._get_seq_info()
if len(self.seq_list) < 1:
raise TrackEvalException('No sequences are selected to be evaluated.')
# Check gt files exist
for seq in self.seq_list:
if not self.data_is_zipped:
curr_file = self.config["GT_LOC_FORMAT"].format(gt_folder=self.gt_fol, seq=seq)
if not os.path.isfile(curr_file):
print('GT file not found ' + curr_file)
raise TrackEvalException('GT file not found for sequence: ' + seq)
if self.data_is_zipped:
curr_file = os.path.join(self.gt_fol, 'data.zip')
if not os.path.isfile(curr_file):
print('GT file not found ' + curr_file)
raise TrackEvalException('GT file not found: ' + os.path.basename(curr_file))
# Get trackers to eval
if self.config['TRACKERS_TO_EVAL'] is None:
self.tracker_list = os.listdir(self.tracker_fol)
else:
self.tracker_list = self.config['TRACKERS_TO_EVAL']
if self.config['TRACKER_DISPLAY_NAMES'] is None:
self.tracker_to_disp = dict(zip(self.tracker_list, self.tracker_list))
elif (self.config['TRACKERS_TO_EVAL'] is not None) and (
len(self.config['TRACKER_DISPLAY_NAMES']) == len(self.tracker_list)):
self.tracker_to_disp = dict(zip(self.tracker_list, self.config['TRACKER_DISPLAY_NAMES']))
else:
raise TrackEvalException('List of tracker files and tracker display names do not match.')
for tracker in self.tracker_list:
if self.data_is_zipped:
curr_file = os.path.join(self.tracker_fol, tracker, self.tracker_sub_fol + '.zip')
if not os.path.isfile(curr_file):
print('Tracker file not found: ' + curr_file)
raise TrackEvalException('Tracker file not found: ' + tracker + '/' + os.path.basename(curr_file))
else:
for seq in self.seq_list:
curr_file = os.path.join(self.tracker_fol, tracker, self.tracker_sub_fol, seq + '.txt')
if not os.path.isfile(curr_file):
print('Tracker file not found: ' + curr_file)
raise TrackEvalException(
'Tracker file not found: ' + tracker + '/' + self.tracker_sub_fol + '/' + os.path.basename(
curr_file))
def get_display_name(self, tracker):
return self.tracker_to_disp[tracker]
def _get_seq_info(self):
seq_list = []
seq_lengths = {}
if self.config["SEQ_INFO"]:
seq_list = list(self.config["SEQ_INFO"].keys())
seq_lengths = self.config["SEQ_INFO"]
# If sequence length is 'None' tries to read sequence length from .ini files.
for seq, seq_length in seq_lengths.items():
if seq_length is None:
ini_file = os.path.join(self.gt_fol, seq, 'seqinfo.ini')
if not os.path.isfile(ini_file):
raise TrackEvalException('ini file does not exist: ' + seq + '/' + os.path.basename(ini_file))
ini_data = configparser.ConfigParser()
ini_data.read(ini_file)
seq_lengths[seq] = int(ini_data['Sequence']['seqLength'])
else:
if self.config["SEQMAP_FILE"]:
seqmap_file = self.config["SEQMAP_FILE"]
else:
if self.config["SEQMAP_FOLDER"] is None:
seqmap_file = os.path.join(self.config['GT_FOLDER'], 'seqmaps', self.gt_set + '.txt')
else:
seqmap_file = os.path.join(self.config["SEQMAP_FOLDER"], self.gt_set + '.txt')
if not os.path.isfile(seqmap_file):
print('no seqmap found: ' + seqmap_file)
raise TrackEvalException('no seqmap found: ' + os.path.basename(seqmap_file))
with open(seqmap_file) as fp:
reader = csv.reader(fp)
for i, row in enumerate(reader):
if i == 0 or row[0] == '':
continue
seq = row[0]
seq_list.append(seq)
ini_file = os.path.join(self.gt_fol, seq, 'seqinfo.ini')
if not os.path.isfile(ini_file):
raise TrackEvalException('ini file does not exist: ' + seq + '/' + os.path.basename(ini_file))
ini_data = configparser.ConfigParser()
ini_data.read(ini_file)
seq_lengths[seq] = int(ini_data['Sequence']['seqLength'])
return seq_list, seq_lengths
def _load_raw_file(self, tracker, seq, is_gt):
"""Load a file (gt or tracker) in the MOT Challenge 2D box format
If is_gt, this returns a dict which contains the fields:
[gt_ids, gt_classes] : list (for each timestep) of 1D NDArrays (for each det).
[gt_dets, gt_crowd_ignore_regions]: list (for each timestep) of lists of detections.
[gt_extras] : list (for each timestep) of dicts (for each extra) of 1D NDArrays (for each det).
if not is_gt, this returns a dict which contains the fields:
[tracker_ids, tracker_classes, tracker_confidences] : list (for each timestep) of 1D NDArrays (for each det).
[tracker_dets]: list (for each timestep) of lists of detections.
"""
# File location
if self.data_is_zipped:
if is_gt:
zip_file = os.path.join(self.gt_fol, 'data.zip')
else:
zip_file = os.path.join(self.tracker_fol, tracker, self.tracker_sub_fol + '.zip')
file = seq + '.txt'
else:
zip_file = None
if is_gt:
file = self.config["GT_LOC_FORMAT"].format(gt_folder=self.gt_fol, seq=seq)
else:
file = os.path.join(self.tracker_fol, tracker, self.tracker_sub_fol, seq + '.txt')
# Load raw data from text file
read_data, ignore_data = self._load_simple_text_file(file, is_zipped=self.data_is_zipped, zip_file=zip_file)
# Convert data to required format
num_timesteps = self.seq_lengths[seq]
data_keys = ['ids', 'classes', 'dets']
if is_gt:
data_keys += ['gt_crowd_ignore_regions', 'gt_extras']
else:
data_keys += ['tracker_confidences']
raw_data = {key: [None] * num_timesteps for key in data_keys}
# Check for any extra time keys
current_time_keys = [str( t+ 1) for t in range(num_timesteps)]
extra_time_keys = [x for x in read_data.keys() if x not in current_time_keys]
if len(extra_time_keys) > 0:
if is_gt:
text = 'Ground-truth'
else:
text = 'Tracking'
raise TrackEvalException(
text + ' data contains the following invalid timesteps in seq %s: ' % seq + ', '.join(
[str(x) + ', ' for x in extra_time_keys]))
for t in range(num_timesteps):
time_key = str(t+1)
if time_key in read_data.keys():
try:
time_data = np.asarray(read_data[time_key], dtype=np.float)
except ValueError:
if is_gt:
raise TrackEvalException(
'Cannot convert gt data for sequence %s to float. Is data corrupted?' % seq)
else:
raise TrackEvalException(
'Cannot convert tracking data from tracker %s, sequence %s to float. Is data corrupted?' % (
tracker, seq))
try:
raw_data['dets'][t] = np.atleast_2d(time_data[:, 2:6])
raw_data['ids'][t] = np.atleast_1d(time_data[:, 1]).astype(int)
except IndexError:
if is_gt:
err = 'Cannot load gt data from sequence %s, because there is not enough ' \
'columns in the data.' % seq
raise TrackEvalException(err)
else:
err = 'Cannot load tracker data from tracker %s, sequence %s, because there is not enough ' \
'columns in the data.' % (tracker, seq)
raise TrackEvalException(err)
if time_data.shape[1] >= 8:
raw_data['classes'][t] = np.atleast_1d(time_data[:, 7]).astype(int)
else:
if not is_gt:
raw_data['classes'][t] = np.ones_like(raw_data['ids'][t])
else:
raise TrackEvalException(
'GT data is not in a valid format, there is not enough rows in seq %s, timestep %i.' % (
seq, t))
if is_gt:
gt_extras_dict = {'zero_marked': np.atleast_1d(time_data[:, 6].astype(int))}
raw_data['gt_extras'][t] = gt_extras_dict
else:
raw_data['tracker_confidences'][t] = np.atleast_1d(time_data[:, 6])
else:
raw_data['dets'][t] = np.empty((0, 4))
raw_data['ids'][t] = np.empty(0).astype(int)
raw_data['classes'][t] = np.empty(0).astype(int)
if is_gt:
gt_extras_dict = {'zero_marked': np.empty(0)}
raw_data['gt_extras'][t] = gt_extras_dict
else:
raw_data['tracker_confidences'][t] = np.empty(0)
if is_gt:
raw_data['gt_crowd_ignore_regions'][t] = np.empty((0, 4))
if is_gt:
key_map = {'ids': 'gt_ids',
'classes': 'gt_classes',
'dets': 'gt_dets'}
else:
key_map = {'ids': 'tracker_ids',
'classes': 'tracker_classes',
'dets': 'tracker_dets'}
for k, v in key_map.items():
raw_data[v] = raw_data.pop(k)
raw_data['num_timesteps'] = num_timesteps
raw_data['seq'] = seq
return raw_data
@_timing.time
def get_preprocessed_seq_data(self, raw_data, cls):
""" Preprocess data for a single sequence for a single class ready for evaluation.
Inputs:
- raw_data is a dict containing the data for the sequence already read in by get_raw_seq_data().
- cls is the class to be evaluated.
Outputs:
- data is a dict containing all of the information that metrics need to perform evaluation.
It contains the following fields:
[num_timesteps, num_gt_ids, num_tracker_ids, num_gt_dets, num_tracker_dets] : integers.
[gt_ids, tracker_ids, tracker_confidences]: list (for each timestep) of 1D NDArrays (for each det).
[gt_dets, tracker_dets]: list (for each timestep) of lists of detections.
[similarity_scores]: list (for each timestep) of 2D NDArrays.
Notes:
General preprocessing (preproc) occurs in 4 steps. Some datasets may not use all of these steps.
1) Extract only detections relevant for the class to be evaluated (including distractor detections).
2) Match gt dets and tracker dets. Remove tracker dets that are matched to a gt det that is of a
distractor class, or otherwise marked as to be removed.
3) Remove unmatched tracker dets if they fall within a crowd ignore region or don't meet a certain
other criteria (e.g. are too small).
4) Remove gt dets that were only useful for preprocessing and not for actual evaluation.
After the above preprocessing steps, this function also calculates the number of gt and tracker detections
and unique track ids. It also relabels gt and tracker ids to be contiguous and checks that ids are
unique within each timestep.
MOT Challenge:
In MOT Challenge, the 4 preproc steps are as follow:
1) There is only one class (pedestrian) to be evaluated, but all other classes are used for preproc.
2) Predictions are matched against all gt boxes (regardless of class), those matching with distractor
objects are removed.
3) There is no crowd ignore regions.
4) All gt dets except pedestrian are removed, also removes pedestrian gt dets marked with zero_marked.
"""
# Check that input data has unique ids
self._check_unique_ids(raw_data)
distractor_class_names = []
if self.benchmark == 'MOT20':
distractor_class_names.append('non_mot_vehicle')
distractor_classes = [self.class_name_to_class_id[x] for x in distractor_class_names]
cls_id = self.class_name_to_class_id[cls]
data_keys = ['gt_ids', 'tracker_ids', 'gt_dets', 'tracker_dets', 'tracker_confidences', 'similarity_scores']
data = {key: [None] * raw_data['num_timesteps'] for key in data_keys}
unique_gt_ids = []
unique_tracker_ids = []
num_gt_dets = 0
num_tracker_dets = 0
for t in range(raw_data['num_timesteps']):
# Get all data
gt_ids = raw_data['gt_ids'][t]
gt_dets = raw_data['gt_dets'][t]
gt_classes = raw_data['gt_classes'][t]
gt_zero_marked = raw_data['gt_extras'][t]['zero_marked']
tracker_ids = raw_data['tracker_ids'][t]
tracker_dets = raw_data['tracker_dets'][t]
tracker_classes = raw_data['tracker_classes'][t]
tracker_confidences = raw_data['tracker_confidences'][t]
similarity_scores = raw_data['similarity_scores'][t]
tracker_classes = [0, 2, 3, 5, 7]
# Evaluation is ONLY valid for pedestrian class
# if len(tracker_classes) > 0 and np.max(tracker_classes) > 1:
# raise TrackEvalException(
# 'Evaluation is only valid for pedestrian class. Non pedestrian class (%i) found in sequence %s at '
# 'timestep %i.' % (np.max(tracker_classes), raw_data['seq'], t))
# Match tracker and gt dets (with hungarian algorithm) and remove tracker dets which match with gt dets
# which are labeled as belonging to a distractor class.
to_remove_tracker = np.array([], np.int)
if self.do_preproc and self.benchmark != 'MOT15' and gt_ids.shape[0] > 0 and tracker_ids.shape[0] > 0:
# Check all classes are valid:
invalid_classes = np.setdiff1d(np.unique(gt_classes), self.valid_class_numbers)
if len(invalid_classes) > 0:
print(' '.join([str(x) for x in invalid_classes]))
raise(TrackEvalException('Attempting to evaluate using invalid gt classes. '
'This warning only triggers if preprocessing is performed, '
'e.g. not for MOT15 or where prepropressing is explicitly disabled. '
'Please either check your gt data, or disable preprocessing. '
'The following invalid classes were found in timestep ' + str(t) + ': ' +
' '.join([str(x) for x in invalid_classes])))
matching_scores = similarity_scores.copy()
matching_scores[matching_scores < 0.5 - np.finfo('float').eps] = 0
match_rows, match_cols = linear_sum_assignment(-matching_scores)
actually_matched_mask = matching_scores[match_rows, match_cols] > 0 + np.finfo('float').eps
match_rows = match_rows[actually_matched_mask]
match_cols = match_cols[actually_matched_mask]
is_distractor_class = np.isin(gt_classes[match_rows], distractor_classes)
to_remove_tracker = match_cols[is_distractor_class]
# Apply preprocessing to remove all unwanted tracker dets.
data['tracker_ids'][t] = np.delete(tracker_ids, to_remove_tracker, axis=0)
data['tracker_dets'][t] = np.delete(tracker_dets, to_remove_tracker, axis=0)
data['tracker_confidences'][t] = np.delete(tracker_confidences, to_remove_tracker, axis=0)
similarity_scores = np.delete(similarity_scores, to_remove_tracker, axis=1)
# Remove gt detections marked as to remove (zero marked), and also remove gt detections not in pedestrian
# class (not applicable for MOT15)
if self.do_preproc and self.benchmark != 'MOT15':
gt_to_keep_mask = (np.not_equal(gt_zero_marked, 0)) & \
(np.equal(gt_classes, cls_id))
else:
# There are no classes for MOT15
gt_to_keep_mask = np.not_equal(gt_zero_marked, 0)
data['gt_ids'][t] = gt_ids[gt_to_keep_mask]
data['gt_dets'][t] = gt_dets[gt_to_keep_mask, :]
data['similarity_scores'][t] = similarity_scores[gt_to_keep_mask]
unique_gt_ids += list(np.unique(data['gt_ids'][t]))
unique_tracker_ids += list(np.unique(data['tracker_ids'][t]))
num_tracker_dets += len(data['tracker_ids'][t])
num_gt_dets += len(data['gt_ids'][t])
# Re-label IDs such that there are no empty IDs
if len(unique_gt_ids) > 0:
unique_gt_ids = np.unique(unique_gt_ids)
gt_id_map = np.nan * np.ones((np.max(unique_gt_ids) + 1))
gt_id_map[unique_gt_ids] = np.arange(len(unique_gt_ids))
for t in range(raw_data['num_timesteps']):
if len(data['gt_ids'][t]) > 0:
data['gt_ids'][t] = gt_id_map[data['gt_ids'][t]].astype(np.int)
if len(unique_tracker_ids) > 0:
unique_tracker_ids = np.unique(unique_tracker_ids)
tracker_id_map = np.nan * np.ones((np.max(unique_tracker_ids) + 1))
tracker_id_map[unique_tracker_ids] = np.arange(len(unique_tracker_ids))
for t in range(raw_data['num_timesteps']):
if len(data['tracker_ids'][t]) > 0:
data['tracker_ids'][t] = tracker_id_map[data['tracker_ids'][t]].astype(np.int)
# Record overview statistics.
data['num_tracker_dets'] = num_tracker_dets
data['num_gt_dets'] = num_gt_dets
data['num_tracker_ids'] = len(unique_tracker_ids)
data['num_gt_ids'] = len(unique_gt_ids)
data['num_timesteps'] = raw_data['num_timesteps']
data['seq'] = raw_data['seq']
# Ensure again that ids are unique per timestep after preproc.
self._check_unique_ids(data, after_preproc=True)
return data
def _calculate_similarities(self, gt_dets_t, tracker_dets_t):
similarity_scores = self._calculate_box_ious(gt_dets_t, tracker_dets_t, box_format='xywh')
return similarity_scores
|
@mikel-brostrom Yes, my mot_challenge_2d_box.py looks like this. I have used Wiki end instructions page to change this file. |
Excellent! Do you have any idea what these equal metric values for all classes could be? |
|
It seems to me that it evaluates the same class over and over again or displays some kind of average instead of each class separatly. Will look closer at this tomorrow. |
|
Because I can run HOTA: labels-car HOTA DetA AssA DetRe DetPr AssRe AssPr LocA OWTA HOTA(0) LocA(0) HOTALocA(0)
VEHICLE-01 53.511 39.441 72.903 40.547 81.094 75.61 85.141 82.7 54.335 65.954 79.513 52.442
COMBINED 53.511 39.441 72.903 40.547 81.094 75.61 85.141 82.7 54.335 65.954 79.513 52.442
CLEAR: labels-car MOTA MOTP MODA CLR_Re CLR_Pr MTR PTR MLR sMOTA CLR_TP CLR_FN CLR_FP IDSW MT PT ML Frag
VEHICLE-01 50 79.513 50 50 100 33.333 33.333 33.333 39.756 76 76 0 0 2 2 2 4
COMBINED 50 79.513 50 50 100 33.333 33.333 33.333 39.756 76 76 0 0 2 2 2 4
Identity: labels-car IDF1 IDR IDP IDTP IDFN IDFP
VEHICLE-01 66.667 50 100 76 76 0
COMBINED 66.667 50 100 76 76 0
Count: labels-car Dets GT_Dets IDs GT_IDs
VEHICLE-01 76 152 4 6
COMBINED 76 152 4 6... |
|
Had to switch to IDF1 IDP IDR Rcll Prcn GT MT PT ML FP FN IDs FM MOTA MOTP IDt IDa IDm
VEHICLE-01-0 11.1% 25.0% 7.1% 7.1% 25.0% 1 0 0 1 6 26 0 0 -14.3% 0.461 0 0 0
VEHICLE-01-2 85.7% 79.7% 92.6% 94.1% 81.0% 3 2 1 0 15 4 1 2 70.6% 0.178 0 1 0
VEHICLE-01-3 6.9% 100.0% 3.6% 3.6% 100.0% 1 0 0 1 0 27 0 0 3.6% 0.211 0 0 0
VEHICLE-01-7 51.3% 90.9% 35.7% 35.7% 90.9% 1 0 1 0 1 18 0 4 32.1% 0.129 0 0 0
OVERALL 70.9% 89.9% 58.6% 60.5% 92.9% 6 3 1 2 7 60 4 7 53.3% 0.207 0 4 0So |
|
Multiclass MOT is working here: I ran And got the following results 2023-06-18 16:13:29.600 | INFO | __main__:evaluate:192 - Running metrics on: ['VEHICLE-01'] for class 0
2023-06-18 16:13:29.636 | SUCCESS | __main__:evaluate:199 -
IDF1 IDP IDR Rcll Prcn GT MT PT ML FP FN IDs FM MOTA MOTP IDt IDa IDm
VEHICLE-01 11.1% 25.0% 7.1% 7.1% 25.0% 1 0 0 1 6 26 0 0 -14.3% 0.461 0 0 0
OVERALL 11.1% 25.0% 7.1% 7.1% 25.0% 1 0 0 1 6 26 0 0 -14.3% 0.461 0 0 0
2023-06-18 16:13:29.637 | INFO | __main__:evaluate:192 - Running metrics on: ['VEHICLE-01'] for class 2
2023-06-18 16:13:29.678 | SUCCESS | __main__:evaluate:199 -
IDF1 IDP IDR Rcll Prcn GT MT PT ML FP FN IDs FM MOTA MOTP IDt IDa IDm
VEHICLE-01 85.7% 79.7% 92.6% 94.1% 81.0% 3 2 1 0 15 4 1 2 70.6% 0.178 0 1 0
OVERALL 85.7% 79.7% 92.6% 94.1% 81.0% 3 2 1 0 15 4 1 2 70.6% 0.178 0 1 0
2023-06-18 16:13:29.679 | INFO | __main__:evaluate:192 - Running metrics on: ['VEHICLE-01'] for class 3
2023-06-18 16:13:29.707 | SUCCESS | __main__:evaluate:199 -
IDF1 IDP IDR Rcll Prcn GT MT PT ML FP FN IDs FM MOTA MOTP IDt IDa IDm
VEHICLE-01 6.9% 100.0% 3.6% 3.6% 100.0% 1 0 0 1 0 27 0 0 3.6% 0.211 0 0 0
OVERALL 6.9% 100.0% 3.6% 3.6% 100.0% 1 0 0 1 0 27 0 0 3.6% 0.211 0 0 0
2023-06-18 16:13:29.708 | INFO | __main__:evaluate:192 - Running metrics on: ['VEHICLE-01'] for class 7
2023-06-18 16:13:29.739 | SUCCESS | __main__:evaluate:199 -
IDF1 IDP IDR Rcll Prcn GT MT PT ML FP FN IDs FM MOTA MOTP IDt IDa IDm
VEHICLE-01 51.3% 90.9% 35.7% 35.7% 90.9% 1 0 1 0 1 18 0 4 32.1% 0.129 0 0 0
OVERALL 51.3% 90.9% 35.7% 35.7% 90.9% 1 0 1 0 1 18 0 4 32.1% 0.129 0 0 0
2023-06-18 16:17:21.433 | INFO | __main__:evaluate:201 - Running metrics on: ['VEHICLE-01'] for ALL classes
2023-06-18 16:17:21.463 | SUCCESS | __main__:evaluate:208 -
MOTA IDF1
VEHICLE-01 53.3% 70.9%
OVERALL 53.3% 70.9%I get the exact same results as when using trackeval |
|
Solution here. Not merged as HOTA is not calculatable with the motmetrics pip package |
|
Thanks for this great job! Have you considered implementing HOTA metrics after that? I think HOTA is a important metric too. Some utils HOTA information: |
|
Yup, I am aware that HOTA is not present in |
|
Have you modified VEHICLE ground truth dataset format? I ran python examples/val.py --benchmark VEHICLE --split test --conf 0.3 --classes 0 2 3 7 --tracking-method strongsortyield the following output error: val: yolo_model=examples/weights/yolov8n.pt, reid_model=examples/weights/osnet_x0_25_msmt17.pt, tracking_method=strongsort, name=exp, classes=['0', '2', '3', '7'], project=examples/runs/val, exist_ok=False, benchmark=VEHICLE, split=test, eval_existing=False, conf=0.3, imgsz=[1280], device=[''], processes_per_device=2
2023-06-21 15:48:07.854 | INFO | __main__:generate_tracks:261 - Staring evaluation process on examples/val_utils/data/VEHICLE/test/VEHICLE-01/VEHICLE-01
2023-06-21 15:48:16.786 | SUCCESS | __main__:generate_tracks:293 - examples/val_utils/data/VEHICLE/test/VEHICLE-01/VEHICLE-01 evaluation succeeded
val: yolo_model=examples/weights/yolov8n.pt, reid_model=examples/weights/osnet_x0_25_msmt17.pt, tracking_method=strongsort, name=exp, classes=['0', '2', '3', '7'], project=examples/runs/val, exist_ok=False, benchmark=VEHICLE, split=test, eval_existing=False, conf=0.3, imgsz=[1280], device=[''], processes_per_device=2
2023-06-21 15:48:16.787 | INFO | __main__:evaluate:161 - Found 1 groundtruths and 29 test files.
2023-06-21 15:48:16.787 | WARNING | __main__:evaluate:163 - The number of gt files and tracking results files differ.
2023-06-21 15:48:16.787 | WARNING | __main__:evaluate:164 - Proceeding with the calculation of partial results
2023-06-21 15:48:16.788 | INFO | __main__:evaluate:165 - Available LAP solvers ['lap', 'scipy']
2023-06-21 15:48:16.788 | INFO | __main__:evaluate:166 - Default LAP solver 'lap'
2023-06-21 15:48:16.788 | INFO | __main__:evaluate:167 - Loading files.
2023-06-21 15:48:16.888 | INFO | __main__:evaluate:195 - Running metrics on: ['VEHICLE-01'] for class 0
2023-06-21 15:48:16.914 | SUCCESS | __main__:evaluate:202 -
IDF1 IDP IDR Rcll Prcn GT MT PT ML FP FN IDs FM MOTA MOTP IDt IDa IDm
VEHICLE-01 0.0% 0.0% NaN NaN 0.0% 0 0 0 0 2 0 0 0 -inf% NaN 0 0 0
OVERALL 0.0% 0.0% NaN NaN 0.0% 0 0 0 0 2 0 0 0 -inf% NaN 0 0 0
2023-06-21 15:48:16.914 | INFO | __main__:evaluate:195 - Running metrics on: ['VEHICLE-01'] for class 2
2023-06-21 15:48:16.941 | SUCCESS | __main__:evaluate:202 -
IDF1 IDP IDR Rcll Prcn GT MT PT ML FP FN IDs FM MOTA MOTP IDt IDa IDm
VEHICLE-01 0.0% 0.0% NaN NaN 0.0% 0 0 0 0 70 0 0 0 -inf% NaN 0 0 0
OVERALL 0.0% 0.0% NaN NaN 0.0% 0 0 0 0 70 0 0 0 -inf% NaN 0 0 0
2023-06-21 15:48:16.942 | INFO | __main__:evaluate:195 - Running metrics on: ['VEHICLE-01'] for class 3
2023-06-21 15:48:16.971 | SUCCESS | __main__:evaluate:202 -
IDF1 IDP IDR Rcll Prcn GT MT PT ML FP FN IDs FM MOTA MOTP IDt IDa IDm
VEHICLE-01 0.0% NaN 0.0% 0.0% NaN 3 0 0 3 0 68 0 0 0.0% NaN 0 0 0
OVERALL 0.0% NaN 0.0% 0.0% NaN 3 0 0 3 0 68 0 0 0.0% NaN 0 0 0
2023-06-21 15:48:16.972 | INFO | __main__:evaluate:195 - Running metrics on: ['VEHICLE-01'] for class 7
2023-06-21 15:48:17.006 | SUCCESS | __main__:evaluate:202 -
IDF1 IDP IDR Rcll Prcn GT MT PT ML FP FN IDs FM MOTA MOTP IDt IDa IDm
VEHICLE-01 0.0% 0.0% NaN NaN 0.0% 0 0 0 0 8 0 0 0 -inf% NaN 0 0 0
OVERALL 0.0% 0.0% NaN NaN 0.0% 0 0 0 0 8 0 0 0 -inf% NaN 0 0 0
2023-06-21 15:48:17.006 | INFO | __main__:evaluate:204 - Running metrics on: ['VEHICLE-01'] for ALL classes
WARNING:root:No ground truth for 000026, skipping.
WARNING:root:No ground truth for 000008, skipping.
WARNING:root:No ground truth for 000009, skipping.
WARNING:root:No ground truth for 000022, skipping.
WARNING:root:No ground truth for 000013, skipping.
WARNING:root:No ground truth for 000028, skipping.
WARNING:root:No ground truth for 000023, skipping.
WARNING:root:No ground truth for 000027, skipping.
WARNING:root:No ground truth for 000014, skipping.
WARNING:root:No ground truth for 000005, skipping.
WARNING:root:No ground truth for 000016, skipping.
WARNING:root:No ground truth for 000007, skipping.
WARNING:root:No ground truth for 000011, skipping.
WARNING:root:No ground truth for 000020, skipping.
WARNING:root:No ground truth for 000012, skipping.
WARNING:root:No ground truth for 000015, skipping.
WARNING:root:No ground truth for 000006, skipping.
WARNING:root:No ground truth for 000017, skipping.
WARNING:root:No ground truth for 000004, skipping.
WARNING:root:No ground truth for 000002, skipping.
WARNING:root:No ground truth for 000010, skipping.
WARNING:root:No ground truth for 000025, skipping.
WARNING:root:No ground truth for 000003, skipping.
WARNING:root:No ground truth for 000024, skipping.
WARNING:root:No ground truth for 000018, skipping.
WARNING:root:No ground truth for 000019, skipping.
WARNING:root:No ground truth for 000001, skipping.
WARNING:root:No ground truth for 000021, skipping.
2023-06-21 15:48:17.137 | SUCCESS | __main__:evaluate:211 -
MOTA IDF1
VEHICLE-01 52.6% 69.0%
OVERALL 52.6% 69.0%
Would be an error in the implementation of evaluate() function on the val.py script? def evaluate(self):
gttxtfiles = list(self.gt_folder.glob('*/gt/gt.txt'))
# get sequences in the right order; strip letters, only sort by numbers
gttxtfiles.sort(key=lambda x: re.sub(r'[^0-9]*', "", str(x)))
tstxtfiles = [f for f in (self.save_dir / 'labels').glob('*.txt')]
LOGGER.info(f"Found {len(gttxtfiles)} groundtruths and {len(tstxtfiles)} test files.")
if len(tstxtfiles) != len(gttxtfiles):
LOGGER.warning(f"The number of gt files and tracking results files differ.")
LOGGER.warning(f"Proceeding with the calculation of partial results")
LOGGER.info(f"Available LAP solvers {str(mm.lap.available_solvers)}")
LOGGER.info(f"Default LAP solver \'{mm.lap.default_solver}\'")
LOGGER.info(f'Loading files.')
|
|
Yup. I your classes were 1 index- based and not zero. I.e. your first class has index 1 and not 0. You can delete the rest of the txt files generated by yolo so that you avoid all those warnings |
Great! I will check it. |
1 task
|
Hi @mikel-brostrom (@Aandre99 - Tagging you in case you found a solution for the following question too), I've followed all the steps mentioned in the Wiki and the above thread, but I'm still encountering the issue of TrackEval giving the repetitive 'equal' HOTA, MOTA, etc. results for all classes. 
As you can see, the GT counts (and the other metrics) for both the classes are EXACTLY the same. I verified that this GT-Count is actually just the number of GT of 'pedestrian' in the video. I have two questions -
How to run it on single classes other than 'pedestrian'? Would it be possible for you to specify the steps for the same? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Search before asking
Question
Hi! I am running evaluation on a multiclass MOT dataset using ByteTrack and a trained multiclass object detection model. The model works fine and generated tracking results as images. But it is not calculating MOT metrics and giving error "Invalid class label 0" .
I have changed --classes argument in track.py as '--classes 0 1 2 3 4' but still error presists. Model works fine on single class MOT dataset.
Moreover I would like to know what should be the format of gt.txt for Multiclass and Singleclass MOT dataset.
The text was updated successfully, but these errors were encountered: