Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merging various fixes for Colab, Cloud TPU, TPU Pod, ... #247

Open
wants to merge 11 commits into
base: master
Choose a base branch
from
Open

Merging various fixes for Colab, Cloud TPU, TPU Pod, ... #247

wants to merge 11 commits into from
+28 −13

Conversation

vochicong
Copy link

@vochicong vochicong commented Nov 20, 2019 •
edited

Including

bzantium and others added 11 commits July 4, 2019 17:11
@bzantium
Fix prefix in get_input_fn
9965647
@bzantium
fix errors with get_input_fn in train_gpu
1b61bf2
@bzantium
fix minor issue
4c574d6
@bzantium
fix assertion for even number of batch size
7ef2f2d
@bzantium
change default value of uncased to False
1ea884d
@bzantium
Fix curr_loss calculation
0391d1e 
When restart training, since prev_step is -1, curr_loss for the first print would be wrongly calculated.
@vochicong
Config for TPU pod
c4850b9 
I ran `train.py` on a TPU pod v3-256 and got the following error:

    ValueError: TPUConfig.num_shards is not set correctly ....

Found in https://cloud.google.com/tpu/docs/training-on-tpu-pods#providing_the_tpu_name_and_region_to_tpuclusterresolver that
> For single device training, you can specify either the TPU name or an IP address, for example: `grpc://1.2.3.4:8470`.
> For TPU Pods you must use the TPU name so that TensorFlow can discover the IP addresses of all the hosts available for training distribution.

So, in the case of a TPU pod, setting `master` doesn't work. I just tried setting `cluster` and it worked, all 32 hosts in the TPU pod were detected and used correctly.
@vochicong
Merge from bzantium/master
1f8f347
@vochicong
Merge pull request #2 from vochicong/patch-1
5a109ec 
Config for TPU pod
@vochicong
Add pass_id and task in flags to train.py
0265911
@vochicong
Add flag n_token to data_utils.py
c4966b6
@vochicong vochicong changed the title Combine #121 and #239 Merging various fixes #121, #239, ... Nov 21, 2019
@vochicong vochicong changed the title Merging various fixes #121, #239, ... Merging various fixes for Colab, Cloud TPU, TPU Pod, ... Nov 21, 2019
@vochicong vochicong mentioned this pull request Nov 21, 2019
Closed
@guochao006
Copy link

aaa

@LifeIsStrange
Copy link

You should contribute to the XLnet implementation of the transformers library
https://huggingface.co/transformers/model_doc/xlnet.html
It is the defacto standard

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@vochicong @guochao006 @LifeIsStrange @graykode @bzantium