Hi! thanks for sharing your code!

[vlomshakov]

Hi, Ahmed,
thanks for sharing your code and results!
It seems to me that you've truncated references unintentionally in CoNaLa test. I saw file and noticed that few code snippets with line feed was truncated (for example, for fn in os.listdir('.'):). This means that BLEU score was computed a bit incorrectly, but good news is that number of truncated references is not so big. Please correct me if I'm wrong

[vlomshakov]

and unfortunately I can't reproduce the result, because authors of CoNaLa have used another tokenizer and it affects BLEU score dramatically: 27 vs 35

Update: Actually it was my fault, I've computed BLEU like sentence metric, not corpus.