-
Notifications
You must be signed in to change notification settings - Fork 32
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Issues with Evaluation Scripts #7
Comments
My predicted string in the BrainTranslator-all_decoding_result.txt file are all the same, strangely. Ex:
Perhaps my terminal message before presenting the BLEU scores may be relevant: [INFO]subjects: ALL [INFO]loading 3 task datasets [INFO]test_set size: 1407 |
I'm facing the same problem as all generated sentences are the same sentence. It seems that the pre-trained encoder makes all the features similar. |
@underkongkong Have you tried playing around with the config file parameters yet? I wasn't sure if that would make a big difference. |
In my case the predicted string was like something: Not sure how to fix this... |
same here |
Anyone solved this problem? |
Thanks for everyone's effort in the discussion; I haven't got time to test out the issue but will work on it later; |
I found how to fix this problem. In eval_decoding.py file,
predicted_string is list, so we put the only string. This code fix the below problem: {'rouge-1': {'r': 0.0960104371521744, 'p': 0.13671808632706614, 'f': 0.10633835733307583}, 'rouge-2': {'r': 0.011719396402741052, 'p': 0.013988694184239035, 'f': 0.01133032845861094}, 'rouge-l': {'r': 0.09090843088332022, 'p': 0.12862700453138184, 'f': 0.10046980133298505}} to (in my case, ) corpus BLEU-1 score: 0.11137150833175373 But in my case, BrainTranslator model generate the only one sentence.. I am doing research with the author's code. We will update in the future if there are any corrections. Thanks to the @MikeWangWZHL . |
I can't find this code in this project: predictions=tokenizer.encode(predicted_string) |
This issue is related to commit fbaf82d
After running the script:
bash ./scripts/eval_decoding.sh
, the results came out to be:corpus BLEU-1 score: 0
corpus BLEU-2 score: 0
corpus BLEU-3 score: 0
corpus BLEU-4 score: 0
{'rouge-1': {'r': 0.0960104371521744, 'p': 0.13671808632706614, 'f': 0.10633835733307583}, 'rouge-2': {'r': 0.011719396402741052, 'p': 0.013988694184239035, 'f': 0.01133032845861094}, 'rouge-l': {'r': 0.09090843088332022, 'p': 0.12862700453138184, 'f': 0.10046980133298505}}
Removing the .squeeze and .tolist may have some affect on the results...
I'll be working on this as well @MikeWangWZHL , thanks for acting fast!
The text was updated successfully, but these errors were encountered: