CombOOD results for OpenOOD Leadboard

[rmagesh148]

Hey Hi!

We would like to submit our CombOOD results to the OpenOOD Leaderboard.

Here's the link to my paper (https://epubs.siam.org/doi/abs/10.1137/1.9781611978032.74),

and the GitHub repository (https://github.com/rmagesh148/combood/tree/main).

Please let me know if you want me to open a PR for the same. Thank you!

[Jingkang50]

@zjysteven Hi Jingyang, Please help check this request. If you need some csv files (with easy-paste values) from the authors, we can leave the comments here.

[zjysteven]

Hi @rmagesh148 thank you for reaching out.

For the leaderboard, would you mind sharing us direct csv files containing your results for easy integration? You can find examples here #232, #207 (comment), #193 (comment).

Also feel free to open a PR if you want to integrate your method into OpenOOD.

[rmagesh148]

Hi @zjysteven : Please check this link for the results. https://drive.google.com/drive/folders/1Ec0Iz7-i87RS36uHar6j7otDpKBr113I?usp=drive_link , let me know if you have any questions. Thanks!

[zjysteven]

Hi @rmagesh148 it seems that the results are obtained on v1.0 benchmark while the leaderboard assumes that all methods are compared using v1.5 updated benchmark. For example for ImageNet we have refrained from using MNIST as OOD dataset (see more changes at https://github.com/Jingkang50/OpenOOD/wiki/OpenOOD-v1.5-change-log).

Is it possible to rerun your methods with v1.5 benchmark? I imagine this wouldn't require much extra efforts (since CombOOD is post-hoc, right?), and you can find example of easily doing this in this colab notebook.

[rmagesh148]

@zjysteven : okay, sure! I will rerun for Imagenet and will let you know the results quickly but can you please update the results for other datasets ?

[zjysteven]

There are also some slight changes to the CIFAR benchmarks too in v1.5, so it would be best to rerun on CIFAR10/100 :)

[rmagesh148]

okay, will be back with the results soon! Thank you for the prompt response.

[MdSaifulIslamSajol]

Hello I am Saiful, co -author of the CombOOD paper.

1. I am trying to download the Imagenet200 dataset using the script shown in the colab (https://colab.research.google.com/drive/1tvTpCM1_ju82Yygu40fy7Lc0L1YrlkQF?usp=sharing). However, I didn't find the key to download Imagenet200 from this file (https://github.com/Jingkang50/OpenOOD/blob/main/scripts/download/download.py). Can you please help me download the dataset ?

2. Also, we are not sure which torchvision.transforms.Normalize() values are used to the OpenOOD_v1.5 for each dataset.
   Can you please provide us with the values used during training ?

[zjysteven]

Hi Saiful @MdSaifulIslamSajol,

1. ImageNet200 will be subsampled from ImageNet1K (just 200 subclasses). So actually there is not a "ImageNet200" separately to be downloaded. You only need to have ImageNet1K ready.

2. The preprocessing is encoded here https://github.com/Jingkang50/OpenOOD/blob/main/openood/preprocessors/test_preprocessor.py, and the normalization statistics are here https://github.com/Jingkang50/OpenOOD/blob/main/openood/preprocessors/transform.py. For ImageNet-1K specifically you can see from our example eval script that the associated preprocessing from torchvision will be used

   OpenOOD/scripts/eval_ood_imagenet.py

   Lines 76 to 99 in 18c6f51

   	if args.tvs_pretrained:
   	if args.arch == 'resnet50':
   	net = ResNet50()
   	weights = eval(f'ResNet50_Weights.IMAGENET1K_V{args.tvs_version}')
   	net.load_state_dict(load_state_dict_from_url(weights.url))
   	preprocessor = weights.transforms()
   	elif args.arch == 'swin-t':
   	net = Swin_T()
   	weights = eval(f'Swin_T_Weights.IMAGENET1K_V{args.tvs_version}')
   	net.load_state_dict(load_state_dict_from_url(weights.url))
   	preprocessor = weights.transforms()
   	elif args.arch == 'vit-b-16':
   	net = ViT_B_16()
   	weights = eval(f'ViT_B_16_Weights.IMAGENET1K_V{args.tvs_version}')
   	net.load_state_dict(load_state_dict_from_url(weights.url))
   	preprocessor = weights.transforms()
   	elif args.arch == 'regnet':
   	net = RegNet_Y_16GF()
   	weights = eval(
   	f'RegNet_Y_16GF_Weights.IMAGENET1K_SWAG_E2E_V{args.tvs_version}')
   	net.load_state_dict(load_state_dict_from_url(weights.url))
   	preprocessor = weights.transforms()
   	else:
   	raise NotImplementedError