Reproducing PSGTR and PSGFormer

[Jingkang50]

If we want to reproduce the results of the paper, we need to change

OpenPSG/openpsg/models/frameworks/psgtr.py

Line 16 in a0e5c8f

def triplet2Result(triplets, use_mask, eval_pan_rels=True):

to eval_pan_rels=False.
The difference between the evaluation of paper and the competition is that:
in the paper, the mask of each triplet can be generated independently, but the competition needs to submit a panseg png file, so that the mask of the triplet comes from a unified panseg, so we set eval_pan_rels=True.
We have provided the log file in this issue #33 for your reference.
We are using the release psgtr code to run the results of the psgtr in the paper. We have compared the logs that other students have not reproduced before, probably because the segmentation loss of psgtr has not converged, it may be recommended to add the focal loss of seg, although we did not have it to achieve the results of the paper.
The r50 and 100 of psgtr are much higher than the corresponding values ​​of R@20.
We think that in the case of eval_pan_rels=False, psgtr has a higher degree of freedom and is not limited by the effect of the panoptic segmentation.
For example, the advantage of psgtr is that a mask can be tagged by many categories in different triplets, which makes it easier to guess correctly. But this advantage is weakened in the competition based on a unified panseg map after fusion.

[ShunchiZhang]

I'm still confused about it...

It seems that both

1. changing the following line to eval_pan_rels=False

OpenPSG/openpsg/models/frameworks/psgtr.py

Line 16 in a0e5c8f

def triplet2Result(triplets, use_mask, eval_pan_rels=True):

2. deleting detection_method='pan_seg' in the configuration file.

only affects the evaluation process rather than the training process.

I cannot see how these explain the difficulty of reproducing the reported results.

[ShunchiZhang]

Also, the reported results are based on the IoU of panoptic masks instead of bounding boxes, which is in accord with the competition. The only reason I can guess is that the training dataset for the competition is smaller.

[Jingkang50]

Yes, eval_pan_rels and detection_method only affect the evaluation process, but different evaluation protocol will bring different scores.
To be more specific, eval_pan_rels=False means every triplet can independently generate subject and object masks, while eval_pan_rels=True requires every triplet to select masks from the fused, unified panoptic segmentation mask. It will downgrade the performance of one-stage models PSGTR and PSGFormer compared to the paper results.
For the sake of the submission file size, the competition requires a unified panseg for each image, rather than two masks per triplet, so the current default eval_pan_rels is set True, which is different from the paper's setting.
Also, the test set is not fully released, which will also affect the results. The full test set will be released in the mid Oct.

All the training data is provided.

[ShunchiZhang]

Thanks for this really detailed explanation! I will change the setting and evaluate again.

[ShunchiZhang]

Hi @Jingkang50, after changing eval_pan_rels into False, I successfully reproduce the results on 8*V100! Thank you guys so much for your excellent work and patient and detailed explanations!

[Jingkang50]

Glad to hear that! Thank you so much for your great questions, which will for sure help many other followers!

[ShunchiZhang]

Another question: the segmentation loss described in the paper is DICE/F-1 Loss, while the implementation seems to show that the segmentation loss is the combination of L1Loss, GIoULoss, and psgtrDiceLoss. Will this discrepancy make any difference?

OpenPSG/configs/_base_/models/psgtr_r50.py

Lines 52 to 67 in a0e5c8f

	sub_loss_cls=dict(type='CrossEntropyLoss',
	use_sigmoid=False,
	loss_weight=1.0,
	class_weight=1.0),
	sub_loss_bbox=dict(type='L1Loss', loss_weight=5.0),
	sub_loss_iou=dict(type='GIoULoss', loss_weight=2.0),
	sub_focal_loss=dict(type='BCEFocalLoss', loss_weight=1.0),
	sub_dice_loss=dict(type='psgtrDiceLoss', loss_weight=1.0),
	obj_loss_cls=dict(type='CrossEntropyLoss',
	use_sigmoid=False,
	loss_weight=1.0,
	class_weight=1.0),
	obj_loss_bbox=dict(type='L1Loss', loss_weight=5.0),
	obj_loss_iou=dict(type='GIoULoss', loss_weight=2.0),
	obj_focal_loss=dict(type='BCEFocalLoss', loss_weight=1.0),
	obj_dice_loss=dict(type='psgtrDiceLoss', loss_weight=1.0),

OpenPSG/openpsg/models/relation_heads/psgtr_head.py

Lines 410 to 440 in a0e5c8f

	loss_dict = dict()
	# loss from the last decoder layer
	loss_dict['s_loss_cls'] = s_losses_cls[-1]
	loss_dict['o_loss_cls'] = o_losses_cls[-1]
	loss_dict['r_loss_cls'] = r_losses_cls[-1]
	loss_dict['s_loss_bbox'] = s_losses_bbox[-1]
	loss_dict['o_loss_bbox'] = o_losses_bbox[-1]
	loss_dict['s_loss_iou'] = s_losses_iou[-1]
	loss_dict['o_loss_iou'] = o_losses_iou[-1]
	if self.use_mask:
	# loss_dict['s_focal_losses'] = s_focal_losses[-1]
	# loss_dict['o_focal_losses'] = o_focal_losses[-1]
	loss_dict['s_dice_losses'] = s_dice_losses[-1]
	loss_dict['o_dice_losses'] = o_dice_losses[-1]
	# loss from other decoder layers
	num_dec_layer = 0
	for s_loss_cls_i, o_loss_cls_i, r_loss_cls_i, \
	s_loss_bbox_i, o_loss_bbox_i, \
	s_loss_iou_i, o_loss_iou_i in zip(s_losses_cls[:-1], o_losses_cls[:-1], r_losses_cls[:-1],
	s_losses_bbox[:-1], o_losses_bbox[:-1],
	s_losses_iou[:-1], o_losses_iou[:-1]):
	loss_dict[f'd{num_dec_layer}.s_loss_cls'] = s_loss_cls_i
	loss_dict[f'd{num_dec_layer}.o_loss_cls'] = o_loss_cls_i
	loss_dict[f'd{num_dec_layer}.r_loss_cls'] = r_loss_cls_i
	loss_dict[f'd{num_dec_layer}.s_loss_bbox'] = s_loss_bbox_i
	loss_dict[f'd{num_dec_layer}.o_loss_bbox'] = o_loss_bbox_i
	loss_dict[f'd{num_dec_layer}.s_loss_iou'] = s_loss_iou_i
	loss_dict[f'd{num_dec_layer}.o_loss_iou'] = o_loss_iou_i
	num_dec_layer += 1
	return loss_dict

[GSeanCDAT]

Hi Shunchi,
Thanks for your continuous interest in our work!

1. In our implementation of PSGTR, we predict bounding boxes as well as segmentation masks, and the L1Loss and GIoULoss are losses for bounding boxes. Those losses can help optimize our output triplet queries and thus help the segmentation prediction.
2. Besides helping segmentation indirectly, those bounding boxes prediction are used in our Hungarian Triplet Matcher. Compared with using segmentation masks in the matching operation which are not computably affordable, those bounding-box triplets can literally reduce computations.
3. As for psgtrDiceLoss, this is just our implementation of DICE Loss.
   Our model is trained with those three losses and we get the results reported in our paper from the model. Thus there are neither discrepancies nor differences in our practice.

[ShunchiZhang]

Thanks for the comprehensive reply! I am much clearer now.

[sirosann]

I tried to turn the eval_pan_rels to Flase,But I found this to be slow to val after training epoch, and the process would automatically be killed after a few epochs(often in 1th or 3th epoch).
The following is the log after being stopped from training.By the way I trained with 8*NVIDIA RTX A6000.