[FEATURE] Implementation of Language model estimator #1155

liuzh47 · 2020-02-13T12:07:44Z

Description

Implementation of word language model estimator and large rnn language model estimator

Checklist

Essentials

PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage
Code is well-documented

Changes

Feature1, tests, (and when applicable, API doc)
Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

cc @dmlc/gluon-nlp-team

codecov · 2020-02-13T12:07:47Z

Codecov Report

Merging #1155 into master will increase coverage by 7.99%.
The diff coverage is 25.64%.

@@            Coverage Diff             @@
##           master    #1155      +/-   ##
==========================================
+ Coverage   70.58%   78.57%   +7.99%     
==========================================
  Files          72       77       +5     
  Lines        6970     7278     +308     
==========================================
+ Hits         4920     5719     +799     
+ Misses       2050     1559     -491

Impacted Files	Coverage Δ
src/gluonnlp/estimator/__init__.py	`100% <100%> (ø)`
...uonnlp/estimator/language_model_batch_processor.py	`17.89% <17.89%> (ø)`
...gluonnlp/estimator/language_model_event_handler.py	`24.09% <24.09%> (ø)`
src/gluonnlp/loss/joint_loss.py	`34.78% <34.78%> (ø)`
src/gluonnlp/estimator/language_model_estimator.py	`41.17% <41.17%> (ø)`
src/gluonnlp/model/train/cache.py	`25.58% <0%> (-72.1%)`	⬇️
src/gluonnlp/data/batchify/language_model.py	`43.92% <0%> (-52.34%)`	⬇️
src/gluonnlp/model/translation.py	`20.31% <0%> (-51.57%)`	⬇️
src/gluonnlp/embedding/evaluation.py	`40.33% <0%> (-51.27%)`	⬇️
src/gluonnlp/model/language_model.py	`48.87% <0%> (-49.63%)`	⬇️
... and 48 more

mli · 2020-02-13T12:41:51Z

Job PR-1155/1 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1155/1/index.html

mli · 2020-02-13T14:56:42Z

Job PR-1155/2 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1155/2/index.html

mli · 2020-02-14T14:24:57Z

Job PR-1155/4 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1155/4/index.html

mli · 2020-02-14T16:07:56Z

Job PR-1155/5 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1155/5/index.html

mli · 2020-02-14T17:40:20Z

Job PR-1155/6 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1155/6/index.html

mli · 2020-02-14T17:49:15Z

Job PR-1155/7 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1155/7/index.html

eric-haibin-lin · 2020-02-17T01:03:21Z

scripts/language_model/index.rst

@@ -47,35 +47,35 @@ The dataset used for training the models is wikitext-2.

 For all the above model settings, we set Tied = True and NTASGD = True .

-[1] awd_lstm_lm_1150_wikitext-2 (Val PPL 68.71 Test PPL 65.62 )
+[1] awd_lstm_lm_1150_wikitext-2 (Val PPL 68.52 Test PPL 65.68 )


While you're at it - would you mind removing the rows related to hyper-parameters in the table? i.e. Mode, Num_layers, Embed size, Hidden size, Dropout, Dropout_h, Dropout_i, Dropout_e, Weight_drop. After the removal the table will be simpler. Also could you move the command to https://github.com/dmlc/web-data/tree/master/gluonnlp/logs/language_model and reference the links in the table? Currently, the commands take a lot of space. We shall simply the table for cache_lm and large word lm, too.

Doc is updated. I have submitted a new PR dmlc/web-data#232 to move the commands.

eric-haibin-lin · 2020-02-17T01:05:04Z

scripts/language_model/index.rst

-   $ python large_word_language_model.py --gpus 0,1,2,3 --clip=10
-   $ python large_word_language_model.py --gpus 4 --eval-only --batch-size=1
+   $ python large_word_language_model_estimator.py --gpus 0,1,2,3 --clip=10
+   $ python large_word_language_model_estimator.py --gpus 4 --eval-only --batch-size=1


No PPL change for large_word_language_model?

I am still training on large_word_language_model. It takes approximately 6~7 days to finish the whole training. Currently I get a test ppl of 43.98 with the latest model checkpoint which is comparable to the baseline model. I will update it after the training is completed.

mli · 2020-02-17T04:25:16Z

Job PR-1155/8 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1155/8/index.html

mli · 2020-02-17T06:55:26Z

Job PR-1155/9 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1155/9/index.html

mli · 2020-02-17T06:55:28Z

Job PR-1155/10 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1155/10/index.html

liuzh47 and others added 22 commits February 13, 2020 10:17

add language model estimator

89deadc

modify init file

90c5144

update language model estimator metrics computation

f7c730f

fix and update language model estimator

c90509a

remove unnecessary argument from the language model estimator

8540f4b

Add checkpoint handler for word language model

d030199

Add large language model estimator

9aa824d

fix name errors

06295ef

add word language model evaluation code

17ef38c

update parallel language model

87651c5

update large language model estimator

e565723

fix typos

cfc2f6d

fix large language model estimator bugs

3bf7679

some bug fixes on language model estimator

50b3a95

update large language model estimator

275098f

add script files

8780711

remove files

757354c

modify loading the checkpoint

3f08627

Add todo lists for event handlers

48dc1e4

update index.rst

7ac114a

remove temp files

0916452

relocate joint loss file

13891e7

liuzh47 requested a review from a team as a code owner February 13, 2020 12:07

remove temporary fix

7eddd52

Ubuntu added 2 commits February 14, 2020 13:45

fix pylint errors and add docstrings

d5d8148

fix errors due to the pylint fix

3d72a32

fix docstring pylint errors

ca9c9a0

Ubuntu added 2 commits February 14, 2020 17:00

fix script pylint errors

7735fa6

fix pylint errrors

e7f80cb

eric-haibin-lin reviewed Feb 17, 2020

View reviewed changes

remove hyperparameters from the table

a0bc616

Ubuntu added 2 commits February 17, 2020 05:29

update language model commands

934cba6

minor modification

450dee0

update bigrnn final result

159553f

szha changed the base branch from master to v0.x August 13, 2020 02:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEATURE] Implementation of Language model estimator #1155

[FEATURE] Implementation of Language model estimator #1155

liuzh47 commented Feb 13, 2020 •

edited

codecov bot commented Feb 13, 2020 •

edited

mli commented Feb 13, 2020

mli commented Feb 13, 2020

mli commented Feb 14, 2020

mli commented Feb 14, 2020

mli commented Feb 14, 2020

mli commented Feb 14, 2020

eric-haibin-lin Feb 17, 2020

liuzh47 Feb 17, 2020

eric-haibin-lin Feb 17, 2020

liuzh47 Feb 17, 2020

mli commented Feb 17, 2020

mli commented Feb 17, 2020

mli commented Feb 17, 2020

[FEATURE] Implementation of Language model estimator #1155

Are you sure you want to change the base?

[FEATURE] Implementation of Language model estimator #1155

Conversation

liuzh47 commented Feb 13, 2020 • edited

Description

Checklist

Essentials

Changes

Comments

codecov bot commented Feb 13, 2020 • edited

Codecov Report

mli commented Feb 13, 2020

mli commented Feb 13, 2020

mli commented Feb 14, 2020

mli commented Feb 14, 2020

mli commented Feb 14, 2020

mli commented Feb 14, 2020

eric-haibin-lin Feb 17, 2020

Choose a reason for hiding this comment

liuzh47 Feb 17, 2020

Choose a reason for hiding this comment

eric-haibin-lin Feb 17, 2020

Choose a reason for hiding this comment

liuzh47 Feb 17, 2020

Choose a reason for hiding this comment

mli commented Feb 17, 2020

mli commented Feb 17, 2020

mli commented Feb 17, 2020

liuzh47 commented Feb 13, 2020 •

edited

codecov bot commented Feb 13, 2020 •

edited