Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

QG for domain adaptation of QA (Question Value Estimation) #305

Closed
bhavani105 opened this issue Aug 29, 2022 · 1 comment · May be fixed by #454
Closed

QG for domain adaptation of QA (Question Value Estimation) #305

bhavani105 opened this issue Aug 29, 2022 · 1 comment · May be fixed by #454
Assignees
@xiangyue9607 @avisil @bhavani105
Labels
enhancement New feature or request

Comments

@bhavani105
Copy link
Collaborator

QG for domain adaptation of QA (Question Value Estimation)
Paper: https://arxiv.org/pdf/2203.08926.pdf
Github: https://github.com/xiangyue9607/QVE

Datasets we evaluated the model on in this paper: SQuAD1.1 as the source domain; NewsQA, NaturalQuestions, HotpotQA TriviaQA as the target domain.
@bhavani105 bhavani105 added the enhancement New feature or request label Aug 29, 2022
@bhavani105 bhavani105 mentioned this issue Oct 26, 2022
Closed
@bhavani105
Copy link
Collaborator Author

bhavani105 commented Jan 24, 2023 •
edited

Experiment Results:

Branch: https://github.com/primeqa/primeqa/tree/qve_testing

Dataset: MRQA
Source: SQUAD
Targets: NewsQA, NaturalQuestionsShort, TriviaQA-web, HotpotQA

QA: bert-base-uncased. Source: SQUAD (F1: 88.1 EM:80.3
QG: facebook/bart-base

  HotpotQA HotpotQA NQ NQ NewsQA NewsQA TriviaQA TriviaQA
Model Training F1 EM F1 EM F1 EM F1 EM
Source Only (SQuAD) 57.22 41.40 55.98 41.70 54.53 38.13 58.33 49.27
Source -> Target 1K 65.21 48.40 69.22 55.68 60.10 44.04 63.44 56.62
Source -> QG no filtering 61.15 44.96 69.59 57.18 58.43 43.19 63.68 58.72
Source -> QG no filtering -> Target 1K 66.21 48.69 72.86 60.91 62.37 47.01 65.61 59.40
Source -> QG filtered 63.39 48.67 68.91 56.59 57.43 41.48 60.79 54.00
Source -> QG filtered -> Target 1K 66.56 50.45 71.52 59.26 62.25 46.56 64.62 57.84
Target ALL 72.57 54.48 78.58 66.66 67.57 53.11 69.17 63.73

Train size

Dataset train train-filtered
HotpotQA 71928 40575
NQ 103071 61842
NewsQA 73160 43887
TriviaQA 60688 8037

Issue: Other than for HotpotQA, QVE based filtering of generated questions loses 1-2 points @xiangyue9607

@xiangyue9607 xiangyue9607 linked a pull request Feb 20, 2023 that will close this issue
QVE #454
Draft
15 tasks
@avisil avisil closed this as completed Aug 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@xiangyue9607 xiangyue9607

@avisil avisil

@bhavani105 bhavani105

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

QVE primeqa/primeqa
3 participants
@xiangyue9607 @avisil @bhavani105