Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Question: performance of QuatE vs RotatE #210

Open
dany-nonstop opened this issue Apr 23, 2021 · 0 comments
Open

Question: performance of QuatE vs RotatE #210

dany-nonstop opened this issue Apr 23, 2021 · 0 comments

Comments

@dany-nonstop
Copy link

Issue Template

Performance of QuatE is not comparable to that of RotateE

Describe the bug
I tried to reproduce performance of both QuatE and RotateE

To Reproduce
Steps to reproduce the behavior:

  1. Run these two commands
pykg2vec-train -exp True -mn rotate -ds freebase15k -device cuda
pykg2vec-train -exp True -mn quate-ds freebase15k -device cuda
  1. Performance reported
    for RotatE
------Test Results for freebase15k: Epoch: 320 --- time: 1372.47------------
--# of entities, # of relations: 14951, 1345
--mr,  filtered mr             : 223.5828, 62.3827
--mrr, filtered mrr            : 0.2553, 0.7473
--hits1                        : 0.1424 
--filtered hits1               : 0.6784 
--hits3                        : 0.2839 
--filtered hits3               : 0.7953 
--hits5                        : 0.3800 
--filtered hits5               : 0.8253 
--hits10                        : 0.5106 
--filtered hits10               : 0.8598 

for QuatE

------Test Results for freebase15k: Epoch: 410 --- time: 511.22------------
--# of entities, # of relations: 14951, 1345
--mr,  filtered mr             : 243.7488, 84.9533
--mrr, filtered mrr            : 0.2333, 0.6890
--hits1                        : 0.1218 
--filtered hits1               : 0.6097 
--hits3                        : 0.2576 
--filtered hits3               : 0.7409 
--hits5                        : 0.3542 
--filtered hits5               : 0.7797 
--hits10                        : 0.4899 
--filtered hits10               : 0.8239 

Expected behavior
My impression is that QuatE as an extension to RotatE, should perform better than the latter. In my experiments, RotatE's performance is pretty close to reported by its authors and QuatE authors, whereas QuatE itself is a far cry from its authors' claim.

Desktop (please complete the following information):

  • OS: Ubuntu 20.04
  • pykg2vec version:
Name: pykg2vec
Version: 0.0.52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant