Skip to content

Latest commit

 

History

History

AFAT

Folders and files

NameName
Last commit message
Last commit date

parent directory

..

README.md

README.md

 
 

k_pred_net.py

k_pred_net.py

 
 

sinkhorn_topk.py

sinkhorn_topk.py

 
 

README.md

AFAT

Our implementation of the following paper:

  • Runzhong Wang, Ziao Guo, Shaofei Jiang, Xiaokang Yang, Junchi Yan. "Deep Learning of Partial Graph Matching via Differentiable Top-K." CVPR 2023. [paper]

AFAT is a partial matching handling approach, used to resolve partial graph matching problem, i.e., graph matching with the existence of outliers. It includes a topk-GM algorithm to suppress matchings with low confidence, and two AFA modules to predict the number of inliers k. The implementation of topk-GM algorithm and AFA modules is shown in models.AFAT.sinkhorn_topk.py and models.AFAT.k_pred_net.py respectively.

AFAT has the following components:

  • An Optimal Transport (OT) layer utilizing Sinkhorn algorithm with marginal distributions to resolve topk selection problem
  • AFA-U (unified bipartite graph modeling) module to predict the number of inliers k
  • AFA-I (individual graph modeling) module to predict the number of inliers k
  • Greedy-topk algorithm to select matches with topk confidences

We also show that such a general scheme can be readily plugged into SOTA deep graph matching pipelines. We adopt quadratic matching network NGMv2 as the backend graph matching network, and the file models.NGM.model_v2_topk.py can be referred to for details. We also choose linear matching network GCAN as the backend graph matching network, and the file models.GCAN.GCAN_model_topk.py can be referred to for details.

Benchmark Results

PascalVOC - 2GM (unfiltered)

  • NGMv2-AFAT-U

    experiment config: experiments/ngmv2-afat/vgg16_ngmv2-afat-u_voc_stage1.yaml, experiments/ngmv2-afat/vgg16_ngmv2-afat-u_voc_stage2.yaml, experiments/ngmv2-afat/vgg16_ngmv2-afat-u_voc_stage3.yaml

    pretrained model: google drive

  • NGMv2-AFAT-I

    experiment config: experiments/ngmv2-afat/vgg16_ngmv2-afat-i_voc_stage1.yaml, experiments/ngmv2-afat/vgg16_ngmv2-afat-i_voc_stage2.yaml, experiments/ngmv2-afat/vgg16_ngmv2-afat-i_voc_stage3.yaml

    pretrained model: google drive

  • GCAN-AFAT-U

    experiment config: experiments/gcan-afat/vgg16_gcan-afat-u_voc_stage1.yaml, experiments/gcan-afat/vgg16_gcan-afat-u_voc_stage2.yaml, experiments/gcan-afat/vgg16_gcan-afat-u_voc_stage3.yaml

    pretrained model: google drive

  • GCAN-AFAT-I

    experiment config: experiments/gcan-afat/vgg16_gcan-afat-i_voc_stage1.yaml, experiments/gcan-afat/vgg16_gcan-afat-i_voc_stage2.yaml, experiments/gcan-afat/vgg16_gcan-afat-i_voc_stage3.yaml

    pretrained model: google drive

model year aero bike bird boat bottle bus car cat chair cow table dog horse mbkie person plant sheep sofa train tv mean
NGMv2-AFAT-U 2023 0.457 0.677 0.573 0.449 0.901 0.655 0.499 0.593 0.440 0.620 0.549 0.584 0.586 0.638 0.459 0.948 0.509 0.373 0.742 0.828 0.602
NGMv2-AFAT-I 2023 0.450 0.673 0.559 0.456 0.903 0.646 0.487 0.580 0.447 0.602 0.548 0.572 0.575 0.634 0.452 0.953 0.493 0.416 0.736 0.824 0.599
GCAN-AFAT-U 2023 0.471 0.708 0.581 0.458 0.908 0.665 0.496 0.588 0.506 0.646 0.472 0.605 0.623 0.657 0.463 0.954 0.527 0.474 0.742 0.838 0.620
GCAN-AFAT-I 2023 0.461 0.699 0.561 0.466 0.907 0.661 0.481 0.579 0.499 0.639 0.504 0.590 0.616 0.650 0.447 0.955 0.509 0.492 0.740 0.838 0.616

Willow Object Class - 2GM & MGM

  • NGMv2-AFAT-U

    experiment config: experiments/ngmv2-afat/vgg16_ngmv2-afat-u_willow_stage1.yaml, experiments/ngmv2-afat/vgg16_ngmv2-afat-u_willow_stage2.yaml, experiments/ngmv2-afat/vgg16_ngmv2-afat-u_willow_stage3.yaml

    pretrained model: google drive

  • NGMv2-AFAT-I

    experiment config: experiments/ngmv2-afat/vgg16_ngmv2-afat-i_willow_stage1.yaml, experiments/ngmv2-afat/vgg16_ngmv2-afat-i_willow_stage2.yaml, experiments/ngmv2-afat/vgg16_ngmv2-afat-i_willow_stage3.yaml

    pretrained model: google drive

  • GCAN-AFAT-U

    experiment config: experiments/gcan-afat/vgg16_gcan-afat-u_willow_stage1.yaml, experiments/gcan-afat/vgg16_gcan-afat-u_willow_stage2.yaml, experiments/gcan-afat/vgg16_gcan-afat-u_willow_stage3.yaml

    pretrained model: google drive

  • GCAN-AFAT-I

    experiment config: experiments/gcan-afat/vgg16_gcan-afat-i_willow_stage1.yaml, experiments/gcan-afat/vgg16_gcan-afat-i_willow_stage2.yaml, experiments/gcan-afat/vgg16_gcan-afat-i_willow_stage3.yaml

    pretrained model: google drive

model year Car Duck Face Motorbike Winebottle mean
NGMv2-AFAT-U 2023 0.826 0.745 0.906 0.739 0.870 0.817
NGMv2-AFAT-I 2023 0.846 0.757 0.920 0.745 0.886 0.831
GCAN-AFAT-U 2023 0.801 0.780 0.906 0.760 0.870 0.823
GCAN-AFAT-I 2023 0.822 0.777 0.927 0.772 0.886 0.837

SPair-71k - 2GM (unfiltered)

  • NGMv2-AFAT-U

    experiment config: experiments/ngmv2-afat/vgg16_ngmv2-afat-u_spair71k_stage1.yaml, experiments/ngmv2-afat/vgg16_ngmv2-afat-u_spair71k_stage2.yaml, experiments/ngmv2-afat/vgg16_ngmv2-afat-u_spair71k_stage3.yaml

    pretrained model: google drive

  • NGMv2-AFAT-I

    experiment config: experiments/ngmv2-afat/vgg16_ngmv2-afat-i_spair71k_stage1.yaml, experiments/ngmv2-afat/vgg16_ngmv2-afat-i_spair71k_stage2.yaml, experiments/ngmv2-afat/vgg16_ngmv2-afat-i_spair71k_stage3.yaml

    pretrained model: google drive

  • GCAN-AFAT-U

    experiment config: experiments/gcan-afat/vgg16_gcan-afat-u_spair71k_stage1.yaml, experiments/gcan-afat/vgg16_gcan-afat-u_spair71k_stage2.yaml, experiments/gcan-afat/vgg16_gcan-afat-u_spair71k_stage3.yaml

    pretrained model: google drive

  • GCAN-AFAT-I

    experiment config: experiments/gcan-afat/vgg16_gcan-afat-i_spair71k_stage1.yaml, experiments/gcan-afat/vgg16_gcan-afat-i_spair71k_stage2.yaml, experiments/gcan-afat/vgg16_gcan-afat-i_spair71k_stage3.yaml

    pretrained model: google drive

model year aero bike bird boat bottle bus car cat chair cow dog horse mtbike person plant sheep train tv mean
NGMv2-AFAT-U 2023 0.503 0.435 0.638 0.324 0.590 0.601 0.397 0.686 0.361 0.636 0.565 0.463 0.514 0.433 0.770 0.512 0.811 0.894 0.563
NGMv2-AFAT-I 2023 0.504 0.436 0.639 0.321 0.612 0.585 0.380 0.684 0.357 0.627 0.564 0.477 0.519 0.443 0.785 0.507 0.792 0.912 0.564
GCAN-AFAT-U 2023 0.467 0.433 0.658 0.333 0.615 0.549 0.352 0.684 0.377 0.599 0.560 0.476 0.472 0.435 0.803 0.477 0.838 0.890 0.557
GCAN-AFAT-I 2023 0.468 0.443 0.659 0.324 0.615 0.538 0.337 0.684 0.381 0.601 0.563 0.479 0.483 0.438 0.812 0.484 0.829 0.880 0.557

IMC-PT-SparseGM-50 - 2GM (unfiltered)

  • NGMv2-AFAT-U

    experiment config: experiments/ngmv2-afat/vgg16_ngmv2-afat-u_imcpt_50_stage1.yaml, experiments/ngmv2-afat/vgg16_ngmv2-afat-u_imcpt_50_stage2.yaml, experiments/ngmv2-afat/vgg16_ngmv2-afat-u_imcpt_50_stage3.yaml

    pretrained model: google drive

  • NGMv2-AFAT-I

    experiment config: experiments/ngmv2-afat/vgg16_ngmv2-afat-i_imcpt_50_stage1.yaml, experiments/ngmv2-afat/vgg16_ngmv2-afat-i_imcpt_50_stage2.yaml, experiments/ngmv2-afat/vgg16_ngmv2-afat-i_imcpt_50_stage3.yaml

    pretrained model: google drive

  • GCAN-AFAT-U

    experiment config: experiments/gcan-afat/vgg16_gcan-afat-u_imcpt_50_stage1.yaml, experiments/gcan-afat/vgg16_gcan-afat-u_imcpt_50_stage2.yaml, experiments/gcan-afat/vgg16_gcan-afat-u_imcpt_50_stage3.yaml

    pretrained model: google drive

  • GCAN-AFAT-I

    experiment config: experiments/gcan-afat/vgg16_gcan-afat-i_imcpt_50_stage1.yaml, experiments/gcan-afat/vgg16_gcan-afat-i_imcpt_50_stage2.yaml, experiments/gcan-afat/vgg16_gcan-afat-i_imcpt_50_stage3.yaml

    pretrained model: google drive

model year reichstag sacre coeur st peters square mean
NGMv2-AFAT-U 2023 0.905 0.587 0.669 0.720
NGMv2-AFAT-I 2023 0.923 0.587 0.667 0.728
GCAN-AFAT-U 2023 0.869 0.594 0.671 0.711
GCAN-AFAT-I 2023 0.910 0.603 0.673 0.729

IMC-PT-SparseGM-100 - 2GM (unfiltered)

  • NGMv2-AFAT-U

    experiment config: experiments/ngmv2-afat/vgg16_ngmv2-afat-u_imcpt_100_stage1.yaml, experiments/ngmv2-afat/vgg16_ngmv2-afat-u_imcpt_100_stage2.yaml, experiments/ngmv2-afat/vgg16_ngmv2-afat-u_imcpt_100_stage3.yaml

    pretrained model: google drive

  • NGMv2-AFAT-I

    experiment config: experiments/ngmv2-afat/vgg16_ngmv2-afat-i_imcpt_100_stage1.yaml, experiments/ngmv2-afat/vgg16_ngmv2-afat-i_imcpt_100_stage2.yaml, experiments/ngmv2-afat/vgg16_ngmv2-afat-i_imcpt_100_stage3.yaml

    pretrained model: google drive

  • GCAN-AFAT-U

    experiment config: experiments/gcan-afat/vgg16_gcan-afat-u_imcpt_100_stage1.yaml, experiments/gcan-afat/vgg16_gcan-afat-u_imcpt_100_stage2.yaml, experiments/gcan-afat/vgg16_gcan-afat-u_imcpt_100_stage3.yaml

    pretrained model: google drive

  • GCAN-AFAT-I

    experiment config: experiments/gcan-afat/vgg16_gcan-afat-i_imcpt_100_stage1.yaml, experiments/gcan-afat/vgg16_gcan-afat-i_imcpt_100_stage2.yaml, experiments/gcan-afat/vgg16_gcan-afat-i_imcpt_100_stage3.yaml

    pretrained model: google drive

model year reichstag sacre coeur st peters square mean
NGMv2-AFAT-U 2023 0.817 0.570 0.722 0.703
NGMv2-AFAT-I 2023 0.820 0.570 0.714 0.701
GCAN-AFAT-U 2023 0.826 0.582 0.738 0.715
GCAN-AFAT-I 2023 0.827 0.578 0.724 0.709

File Organization

├── k_pred_net.py
|   the implementation of AFA-I and AFA-U modules 
└── sinkhorn_topk.py
    the implementation of topk-GM algorithm and greedy-topk algorithm

Credits and Citation

Please cite the following paper if you use this model in your research:

@InProceedings{WangCVPR23,
    author={Wang, Runzhong and Guo, Ziao and Jiang, Shaofei and Yang, Xiaokang and Yan, Junchi},
    title={Deep Learning of Partial Graph Matching via Differentiable Top-K},
    booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    year={2023},
}