Fix prefix in get_input_fn / Fix assertion for even number of batch size

[bzantium]

This may be the solution for Issues #120, #111, #85

[bzantium]

add pass_id and task in flags to manipulate file path in train_gpu.py

[3NFBAGDU]

This may be the solution for Issues #120, #111, #85

I have changed code , but i have the same error. TypeError: filenamesmust be atf.data.Datasetoftf.string elements

[3NFBAGDU]

i have changed eeverything correct, but i have error :
TypeError: filenamesmust be atf.data.Datasetoftf.string elements.

[bzantium]

i have changed eeverything correct, but i have error :
TypeError: filenamesmust be atf.data.Datasetoftf.string elements.

Can you show me your command line for running data_utils.py and train_gpu.py ?
and the file names in the "tfrecords" folder like:
record_info-train-0-0.bsz-32.seqlen-512.reuse-256.bi.alpha-6.beta-1.fnp-85.json
train-0-0.bsz-32.seqlen-512.reuse-256.bi.alpha-6.beta-1.fnp-85.tfrecords

[3NFBAGDU]

i have changed eeverything correct, but i have error :
TypeError: filenamesmust be atf.data.Datasetoftf.string elements.

Can you show me your command line for running data_utils.py and train_gpu.py ?

step 1 - python data_utils.py --bsz_per_host=32 --num_core_per_host=16 --seq_len=512 --reuse_len=256 --input_glob=books-sentences.txt --save_dir=fix --num_passes=20 --bi_data=True --sp_path=/home/ubuntu/giorgi/sp10m.cased.v3.model --mask_alpha=6 --mask_beta=1 --num_predict=85.
/home/ubuntu/xlnet/fix/tfrecords this is directory where tfrecords is stored.

step 2 - sudo python3 train_gpu.py --record_info_dir=/home/ubuntu/xlnet/fix/tfrecords --train_batch_size=32 --seq_len=512 --reuse_len=256 --mem_len=384 --perm_size=256 --n_layer=24 --d_model=1024 --d_embed=1024 --n_head=16 --d_head=64 --d_inner=4096 --untie_r=True --model_dir='my_model'

[bzantium]

i have changed eeverything correct, but i have error :
TypeError: filenamesmust be atf.data.Datasetoftf.string elements.

Can you show me your command line for running data_utils.py and train_gpu.py ?

step 1 - python data_utils.py --bsz_per_host=32 --num_core_per_host=16 --seq_len=512 --reuse_len=256 --input_glob=books-sentences.txt --save_dir=training --num_passes=20 --bi_data=True --sp_path=/home/ubuntu/giorgi/sp10m.cased.v3.model --mask_alpha=6 --mask_beta=1 --num_predict=85.
/home/ubuntu/xlnet/training/tfrecords this is directory where tfrecords is stored.

step 2 - sudo python3 train_gpu.py --record_info_dir=/home/ubuntu/xlnet/training/tfrecords --train_batch_size=32--seq_len=512 --reuse_len=256 --mem_len=384 --perm_size=256 --n_layer=24 --d_model=1024 --d_embed=1024 --n_head=16 --d_head=64 --d_inner=4096 --untie_r=True --model_dir='my_model'

Can you add space between --train_batch_size=32 and --seq_len=512 for step 2?

[3NFBAGDU]

i have changed eeverything correct, but i have error :
TypeError: filenamesmust be atf.data.Datasetoftf.string elements.

Can you show me your command line for running data_utils.py and train_gpu.py ?

step 1 - python data_utils.py --bsz_per_host=32 --num_core_per_host=16 --seq_len=512 --reuse_len=256 --input_glob=books-sentences.txt --save_dir=training --num_passes=20 --bi_data=True --sp_path=/home/ubuntu/giorgi/sp10m.cased.v3.model --mask_alpha=6 --mask_beta=1 --num_predict=85.
/home/ubuntu/xlnet/training/tfrecords this is directory where tfrecords is stored.
step 2 - sudo python3 train_gpu.py --record_info_dir=/home/ubuntu/xlnet/training/tfrecords --train_batch_size=32--seq_len=512 --reuse_len=256 --mem_len=384 --perm_size=256 --n_layer=24 --d_model=1024 --d_embed=1024 --n_head=16 --d_head=64 --d_inner=4096 --untie_r=True --model_dir='my_model'

Can you add space between --train_batch_size=32 and --seq_len=512 for step 2?

I made a mistake in copy.

step 2 - sudo python3 train_gpu.py --record_info_dir=/home/ubuntu/xlnet/fix/tfrecords --train_batch_size=32 --seq_len=512 --reuse_len=256 --mem_len=384 --perm_size=256 --n_layer=24 --d_model=1024 --d_embed=1024 --n_head=16 --d_head=64 --d_inner=4096 --untie_r=True --model_dir=my_model .

[3NFBAGDU]

This is the tfrecords, which was generated by running data_utils.py.

[bzantium]

This is the tfrecords, which was generated by running data_utils.py.

I think you should change flags("uncased") to False in data_utils.py. You can simply change file names
from: train-0-0.bsz-32.seqlen-512.reuse-256.uncased.bi.alpha-6.beta-1.fnp-85.tfrecords
record_info-train-0-0.bsz-32.seqlen-512.reuse-256.uncased.bi.alpha-6.beta-1.fnp-85.json
to: train-0-0.bsz-32.seqlen-512.reuse-256.bi.alpha-6.beta-1.fnp-85.tfrecords
record_info-train-0-0.bsz-32.seqlen-512.reuse-256.bi.alpha-6.beta-1.fnp-85.json
if you don't want to run data_utils.py again.

[3NFBAGDU]

This is the tfrecords, which was generated by running data_utils.py.

I think you should change flags.("uncased") to False in data_utils.py. You can simply change file name
from: train-0-0.bsz-32.seqlen-512.reuse-256.uncased.bi.alpha-6.beta-1.fnp-85.tfrecords
record_info-train-0-0.bsz-32.seqlen-512.reuse-256.uncased.bi.alpha-6.beta-1.fnp-85.json
to: train-0-0.bsz-32.seqlen-512.reuse-256.uncased.bi.alpha-6.beta-1.fnp-85.tfrecords
record_info-train-0-0.bsz-32.seqlen-512.reuse-256.bi.alpha-6.beta-1.fnp-85.json

I have changed flags.DEFINE_bool("uncased", False, help="Use uncased inputs or not."), but i have same error: TypeError: filenamesmust be atf.data.Datasetoftf.stringelements.

When i train in gpu, in data_utils.py use_tpu FLAG should be False, i think.

[3NFBAGDU]

This is the tfrecords, which was generated by running data_utils.py.

I think you should change flags("uncased") to False in data_utils.py. You can simply change file names
from: train-0-0.bsz-32.seqlen-512.reuse-256.uncased.bi.alpha-6.beta-1.fnp-85.tfrecords
record_info-train-0-0.bsz-32.seqlen-512.reuse-256.uncased.bi.alpha-6.beta-1.fnp-85.json
to: train-0-0.bsz-32.seqlen-512.reuse-256.bi.alpha-6.beta-1.fnp-85.tfrecords
record_info-train-0-0.bsz-32.seqlen-512.reuse-256.bi.alpha-6.beta-1.fnp-85.json
if you don't want to run data_utils.py again.

Hi, Is here any news about my case?

[bzantium]

Hi, Is here any news about my case?

I think everything looks ok... but can you try relative path? like:
sudo python3 train_gpu.py --record_info_dir=fix2/tfrecords --train_batch_size=32 --seq_len=512 --reuse_len=256 --mem_len=384 --perm_size=256 --n_layer=24 --d_model=1024 --d_embed=1024 --n_head=16 --d_head=64 --d_inner=4096 --untie_r=True --model_dir='my_model'

[Aaradhyaiitr]

Inorder to do pre-training:

Apart from the above mentioned changes, notice and change the following things as well:

1. Batchsize should be same in both train and data_utils.
2. In Line 776 data_utils.py change uncased to None (uncased=None).
3. In Line 236 modelling.py change assert bsz%2 == 0 to tf.debugging.assert_equal(bsz%2,0).
4. add --uncased=True as an argument in train_gpu.

[bzantium]

Inorder to do pre-training:

Apart from the above mentioned changes, notice and change the following things as well:

1. Batchsize should be same in both train and data_utils.
2. In Line 776 data_utils.py change uncased to None (uncased=None).
3. In Line 236 modelling.py change assert bsz%2 == 0 to tf.debugging.assert_equal(bsz%2,0).
4. add --uncased=True as an argument in train_gpu.

Thank you for the commits summarization!