Test result does not match the paper

[dusigh]

I checked out your code and ran test on the StanfordOnlineProduct dataset. I used your pre-trained model: StanfordProducts_FastPair_cascade_v1_hard_ratio_iter_60000_50_25_10_672.caffemodel.
The test result is as follows:

stanford online products mean recall@ 1 : 0.450450
stanford online products mean recall@ 10 : 0.685800
stanford online products mean recall@ 100 : 0.785500
stanford online products mean recall@ 1000 : 0.860150

This doesn't seem to match the result in the paper:

HDC + Contrastive 384 69.5 (@1) 84.4(@10) 92.8(@100) 97.7(@1000)

I do find out that the labels in your HDC_train.txt file differ from those in Ebay_train.txt by 1. For example: mug_final/151771651316_1.JPG has label 6687 in Evay_train.txt but has label 6686 in HDC_train.txt. Anyway I don't think that matters because as long as the labels are consistent within the same file it should be fine, not to mention that I am using the pre-trained model so those files were not used at all.

Could you give me some hint on what might have gone wrong so that the test results do not match the paper? Thank you!

[PkuRainBow]

hi, I think there is something wrong with your feature test code. I have tested this model and the result can achieve 69.5 Recall@1

[PkuRainBow]

@dusigh Please refer the test code here and check the possible errors . https://github.com/PkuRainBow/Hard-Aware-Deeply-Cascaded-Embedding_release/blob/master/src_code/test_stanford_products.py

[dusigh]

Could you update the data/stanford_products/test_label.dat file in the repository for the stanford_product dataset so that I can make our experiments as comparable as possible? The current one in this repository obviously has some problems: The labels are simply the sequential index: "0, 1, 2, 3, 4 ... 60498".

[PkuRainBow]

@dusigh Hope this file could help you
stanford_products_test_label.zip

[dusigh]

Thank you. Yes, this version works.