New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

the douban testset #13

Open

heyuanw opened this issue Dec 17, 2018 · 1 comment

heyuanw commented Dec 17, 2018

hi,
I have questions about douban test data.

the pos/neg label is not 1:9 ?
I find that some context has all ten neg responses but no pos response, for such case, the denominator will be 0, thus how to calculate p1/10 result by using douban_evaluation.py?
I find that the testset len in data.pkl in douban is 6670, not equal to 10000 in test.txt, how to deal with this? is there any special point of these 6670 context?

thanks

Collaborator

xyzhou-puck commented Dec 18, 2018

Hi,

q1. the pos/neg label is not 1: 9?
No, it is not 1:9, because the label in the douban test-set is manually annotated.

q2 & q3
Those contexts in test-set that only has positive or negative candidate responses are discarded during evaluation, same to what SMN did.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment