Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reproducible of retrieval_predictions.ipynb #149

Open
apurvak opened this issue Dec 23, 2021 · 0 comments
Open

reproducible of retrieval_predictions.ipynb #149

apurvak opened this issue Dec 23, 2021 · 0 comments

Comments

@apurvak
Copy link

apurvak commented Dec 23, 2021

Hi,
I am trying to reproduce the results directly listed in the paper: https://arxiv.org/pdf/2103.12011.pdf
image

Specifically, I am trying to reproduce the answer to the question "which element is named for the Greek word for green?"

I used all the provided pre-trained models and all provided pre-trained tables just so I can get rid of any variables. I am using the provided collab notebook as it is:
https://colab.research.google.com/github/google-research/tapas/blob/master/notebooks/retrieval_predictions.ipynb#scrollTo=3jiJ5VvrJkih

However, the answer to that question is incorrect from provided model - it is some random values. Generally, I found that quality of results is not good even for basic questions that prior model worked well.

Is there any piece missing here?
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant