Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incoherent logged val/iou value despite coherent confusion matrix and per-class IoUs when training model in DDP #122

Open
CharlesGaydon opened this issue Apr 23, 2024 · 1 comment
Labels
invalid This doesn't seem right

Comments

@CharlesGaydon
Copy link
Collaborator

CharlesGaydon commented Apr 23, 2024

Observed in DDP mode with 2x3 GPUs (not tested on single node training).

This is surprising, because the per-class IoUs are computed with a bespoke method from the confusion matrix of the MultiClassJaccardIndex object that computes mean IoU

self.log("val/iou", iou_epoch, on_step=False, on_epoch=True, prog_bar=True)

Mean IoU:
image

Per-class:
image

Might be linked to #108

@CharlesGaydon CharlesGaydon added the invalid This doesn't seem right label Apr 23, 2024
@CharlesGaydon CharlesGaydon changed the title Incoherent val/iou value despie coherent per-class IoUs when training model in DDP Incoherent logged val/iou value despite coherent confusion matrix and per-class IoUs when training model in DDP Apr 23, 2024
@CharlesGaydon
Copy link
Collaborator Author

Note: maybe #124 has corrected this since the computation of the IoU is now performed using a different function. Feel free to close if you do not observe this behavior again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
invalid This doesn't seem right
Projects
None yet
Development

No branches or pull requests

1 participant