Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pre-Training TAPAS on new data #143

Open
naserahmadi opened this issue Oct 26, 2021 · 1 comment
Open

Pre-Training TAPAS on new data #143

naserahmadi opened this issue Oct 26, 2021 · 1 comment

Comments

@naserahmadi
Copy link

Hello,
I am trying to pre-train a tapas model on my data (a set of queries and a table). My first question is that is there any function that can help me to convert my data into a format readable for create_pretrain_examples_main? or I should convert it myself to a format like interactions_sample.txtpb?

Also when I want to run tapas_pretraining_experiment on generated tfrecords, it returns the following error:

WARNING:tensorflow:From /home/pignal/tapas/tapas/utils/experiment_utils.py:149: The name tf.estimator.tpu.InputPipelineConfig is deprecated. Please use tf.compat.v1.estimator.tpu.InputPipelineConfig instead.

W1026 10:43:37.885343 140090917570368 module_wrapper.py:138] From /home/pignal/tapas/tapas/utils/experiment_utils.py:149: The name tf.estimator.tpu.InputPipelineConfig is deprecated. Please use tf.compat.v1.estimator.tpu.InputPipelineConfig instead.

WARNING:tensorflow:From /home/pignal/tapas/tapas/utils/experiment_utils.py:150: The name tf.estimator.tpu.RunConfig is deprecated. Please use tf.compat.v1.estimator.tpu.RunConfig instead.

W1026 10:43:37.885531 140090917570368 module_wrapper.py:138] From /home/pignal/tapas/tapas/utils/experiment_utils.py:150: The name tf.estimator.tpu.RunConfig is deprecated. Please use tf.compat.v1.estimator.tpu.RunConfig instead.

WARNING:tensorflow:From /home/pignal/tapas/tapas/utils/experiment_utils.py:158: The name tf.estimator.tpu.TPUConfig is deprecated. Please use tf.compat.v1.estimator.tpu.TPUConfig instead.

W1026 10:43:37.885647 140090917570368 module_wrapper.py:138] From /home/pignal/tapas/tapas/utils/experiment_utils.py:158: The name tf.estimator.tpu.TPUConfig is deprecated. Please use tf.compat.v1.estimator.tpu.TPUConfig instead.

WARNING:tensorflow:From /home/pignal/tapas/tapas/utils/experiment_utils.py:165: The name tf.estimator.tpu.TPUEstimator is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimator instead.

W1026 10:43:37.885888 140090917570368 module_wrapper.py:138] From /home/pignal/tapas/tapas/utils/experiment_utils.py:165: The name tf.estimator.tpu.TPUEstimator is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimator instead.

INFO:tensorflow:Using config: {'_model_dir': '...', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 5000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true
graph_options {
 rewrite_options {
   meta_optimizer_iterations: ONE
 }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 4.0, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=5000, num_shards=None, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1), '_cluster': None}
I1026 10:43:37.886492 140090917570368 estimator.py:191] Using config: {'_model_dir': '...', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 5000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true
graph_options {
 rewrite_options {
   meta_optimizer_iterations: ONE
 }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 4.0, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=5000, num_shards=None, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1), '_cluster': None}
INFO:tensorflow:_TPUContext: eval_on_tpu True
I1026 10:43:37.887225 140090917570368 tpu_context.py:216] _TPUContext: eval_on_tpu True
WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.
W1026 10:43:37.887610 140090917570368 tpu_context.py:218] eval_on_tpu ignored because use_tpu is False.
WARNING:tensorflow:From /home/pignal/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
W1026 10:43:37.893979 140090917570368 deprecation.py:506] From /home/pignal/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
WARNING:tensorflow:From /home/pignal/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
W1026 10:43:37.894331 140090917570368 deprecation.py:323] From /home/pignal/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
2021-10-26 10:43:37.900831: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2021-10-26 10:43:37.900858: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2021-10-26 10:43:37.900884: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (pignal): /proc/driver/nvidia/version does not exist
WARNING:tensorflow:From /home/pignal/tapas/tapas/datasets/dataset.py:57: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_deterministic`.
W1026 10:43:37.925668 140090917570368 deprecation.py:323] From /home/pignal/tapas/tapas/datasets/dataset.py:57: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_deterministic`.
WARNING:tensorflow:From /home/pignal/tapas/tapas/datasets/dataset.py:73: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map(map_func, num_parallel_calls)` followed by `tf.data.Dataset.batch(batch_size, drop_remainder)`. Static tf.data optimizations will take care of using the fused implementation.
W1026 10:43:38.022828 140090917570368 deprecation.py:323] From /home/pignal/tapas/tapas/datasets/dataset.py:73: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map(map_func, num_parallel_calls)` followed by `tf.data.Dataset.batch(batch_size, drop_remainder)`. Static tf.data optimizations will take care of using the fused implementation.
INFO:tensorflow:Calling model_fn.
I1026 10:43:38.306126 140090917570368 estimator.py:1169] Calling model_fn.
INFO:tensorflow:Running train on CPU/GPU
I1026 10:43:38.306341 140090917570368 tpu_estimator.py:3171] Running train on CPU/GPU
INFO:tensorflow:*** Features ***
I1026 10:43:38.306876 140090917570368 tapas_pretraining_model.py:137] *** Features ***
INFO:tensorflow:  name = column_ids, shape = (512, 128)
I1026 10:43:38.306996 140090917570368 tapas_pretraining_model.py:139]   name = column_ids, shape = (512, 128)
INFO:tensorflow:  name = column_ranks, shape = (512, 128)
I1026 10:43:38.307089 140090917570368 tapas_pretraining_model.py:139]   name = column_ranks, shape = (512, 128)
INFO:tensorflow:  name = input_ids, shape = (512, 128)
I1026 10:43:38.307202 140090917570368 tapas_pretraining_model.py:139]   name = input_ids, shape = (512, 128)
INFO:tensorflow:  name = input_mask, shape = (512, 128)
I1026 10:43:38.307282 140090917570368 tapas_pretraining_model.py:139]   name = input_mask, shape = (512, 128)
INFO:tensorflow:  name = inv_column_ranks, shape = (512, 128)
I1026 10:43:38.307380 140090917570368 tapas_pretraining_model.py:139]   name = inv_column_ranks, shape = (512, 128)
INFO:tensorflow:  name = masked_lm_ids, shape = (512, 20)
I1026 10:43:38.307460 140090917570368 tapas_pretraining_model.py:139]   name = masked_lm_ids, shape = (512, 20)
INFO:tensorflow:  name = masked_lm_positions, shape = (512, 20)
I1026 10:43:38.307562 140090917570368 tapas_pretraining_model.py:139]   name = masked_lm_positions, shape = (512, 20)
INFO:tensorflow:  name = masked_lm_weights, shape = (512, 20)
I1026 10:43:38.307645 140090917570368 tapas_pretraining_model.py:139]   name = masked_lm_weights, shape = (512, 20)
INFO:tensorflow:  name = next_sentence_labels, shape = (512, 1)
I1026 10:43:38.307728 140090917570368 tapas_pretraining_model.py:139]   name = next_sentence_labels, shape = (512, 1)
INFO:tensorflow:  name = numeric_relations, shape = (512, 128)
I1026 10:43:38.307821 140090917570368 tapas_pretraining_model.py:139]   name = numeric_relations, shape = (512, 128)
INFO:tensorflow:  name = prev_label_ids, shape = (512, 128)
I1026 10:43:38.307901 140090917570368 tapas_pretraining_model.py:139]   name = prev_label_ids, shape = (512, 128)
INFO:tensorflow:  name = row_ids, shape = (512, 128)
I1026 10:43:38.307995 140090917570368 tapas_pretraining_model.py:139]   name = row_ids, shape = (512, 128)
INFO:tensorflow:  name = segment_ids, shape = (512, 128)
I1026 10:43:38.308069 140090917570368 tapas_pretraining_model.py:139]   name = segment_ids, shape = (512, 128)
INFO:tensorflow:training_loop marked as finished
I1026 10:43:38.317348 140090917570368 error_handling.py:115] training_loop marked as finished
WARNING:tensorflow:Reraising captured error
W1026 10:43:38.317499 140090917570368 error_handling.py:149] Reraising captured error
Traceback (most recent call last):
 File "/home/pignal/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/nest.py", line 378, in assert_same_structure
   expand_composites)
ValueError: The two structures don't have the same nested structure.

First structure: type=list str=[<tf.Tensor 'IteratorGetNext:12' shape=(512, 128) dtype=int32>, <tf.Tensor 'IteratorGetNext:0' shape=(512, 128) dtype=int32>, <tf.Tensor 'IteratorGetNext:11' shape=(512, 128) dtype=int32>, <tf.Tensor 'IteratorGetNext:10' shape=(512, 128) dtype=int32>, <tf.Tensor 'IteratorGetNext:1' shape=(512, 128) dtype=int32>, <tf.Tensor 'IteratorGetNext:4' shape=(512, 128) dtype=int32>, <tf.Tensor 'IteratorGetNext:9' shape=(512, 128) dtype=int32>]

Second structure: type=int str=2

More specifically: Substructure "type=list str=[<tf.Tensor 'IteratorGetNext:12' shape=(512, 128) dtype=int32>, <tf.Tensor 'IteratorGetNext:0' shape=(512, 128) dtype=int32>, <tf.Tensor 'IteratorGetNext:11' shape=(512, 128) dtype=int32>, <tf.Tensor 'IteratorGetNext:10' shape=(512, 128) dtype=int32>, <tf.Tensor 'IteratorGetNext:1' shape=(512, 128) dtype=int32>, <tf.Tensor 'IteratorGetNext:4' shape=(512, 128) dtype=int32>, <tf.Tensor 'IteratorGetNext:9' shape=(512, 128) dtype=int32>]" is a sequence, while substructure "type=int str=2" is not

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "tapas/experiments/tapas_pretraining_experiment.py", line 157, in <module>
   app.run(main)
 File "/home/pignal/anaconda3/lib/python3.6/site-packages/absl/app.py", line 303, in run
   _run_main(main, args)
 File "/home/pignal/anaconda3/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
   sys.exit(main(argv))
 File "tapas/experiments/tapas_pretraining_experiment.py", line 117, in main
   input_fn=train_input_fn, max_steps=experiment_utils.num_train_steps())
 File "/home/pignal/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3083, in train
   rendezvous.raise_errors()
 File "/home/pignal/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/tpu/error_handling.py", line 150, in raise_errors
   six.reraise(typ, value, traceback)
 File "/home/pignal/.local/lib/python3.6/site-packages/six.py", line 703, in reraise
   raise value
 File "/home/pignal/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3078, in train
   saving_listeners=saving_listeners)
 File "/home/pignal/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 349, in train
   loss = self._train_model(input_fn, hooks, saving_listeners)
 File "/home/pignal/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1182, in _train_model
   return self._train_model_default(input_fn, hooks, saving_listeners)
 File "/home/pignal/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1211, in _train_model_default
   self.config)
 File "/home/pignal/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 2915, in _call_model_fn
   config)
 File "/home/pignal/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1170, in _call_model_fn
   model_fn_results = self._model_fn(features=features, **kwargs)
 File "/home/pignal/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3173, in _model_fn
   features, labels, is_export_mode=is_export_mode)
 File "/home/pignal/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 1700, in call_without_tpu
   return self._call_model_fn(features, labels, is_export_mode=is_export_mode)
 File "/home/pignal/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 2031, in _call_model_fn
   estimator_spec = self._model_fn(features=features, **kwargs)
 File "/home/pignal/tapas/tapas/models/tapas_pretraining_model.py", line 157, in model_fn
   proj_value_length=proj_value_length,
 File "/home/pignal/tapas/tapas/models/bert/table_bert.py", line 94, in create_model
   proj_value_length=proj_value_length,
 File "/home/pignal/tapas/tapas/models/bert/modeling.py", line 228, in __init__
   dropout_prob=config.hidden_dropout_prob)
 File "/home/pignal/tapas/tapas/models/bert/modeling.py", line 569, in embedding_postprocessor
   tf.nest.assert_same_structure(token_type_ids, token_type_vocab_size)
 File "/home/pignal/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/nest.py", line 385, in assert_same_structure
   % (str(e), str1, str2))
ValueError: The two structures don't have the same nested structure.

First structure: type=list str=[<tf.Tensor 'IteratorGetNext:12' shape=(512, 128) dtype=int32>, <tf.Tensor 'IteratorGetNext:0' shape=(512, 128) dtype=int32>, <tf.Tensor 'IteratorGetNext:11' shape=(512, 128) dtype=int32>, <tf.Tensor 'IteratorGetNext:10' shape=(512, 128) dtype=int32>, <tf.Tensor 'IteratorGetNext:1' shape=(512, 128) dtype=int32>, <tf.Tensor 'IteratorGetNext:4' shape=(512, 128) dtype=int32>, <tf.Tensor 'IteratorGetNext:9' shape=(512, 128) dtype=int32>]

Second structure: type=int str=2

More specifically: Substructure "type=list str=[<tf.Tensor 'IteratorGetNext:12' shape=(512, 128) dtype=int32>, <tf.Tensor 'IteratorGetNext:0' shape=(512, 128) dtype=int32>, <tf.Tensor 'IteratorGetNext:11' shape=(512, 128) dtype=int32>, <tf.Tensor 'IteratorGetNext:10' shape=(512, 128) dtype=int32>, <tf.Tensor 'IteratorGetNext:1' shape=(512, 128) dtype=int32>, <tf.Tensor 'IteratorGetNext:4' shape=(512, 128) dtype=int32>, <tf.Tensor 'IteratorGetNext:9' shape=(512, 128) dtype=int32>]" is a sequence, while substructure "type=int str=2" is not
Entire first structure:
[., ., ., ., ., ., .]
Entire second structure:
.

Can you help me with these questions?
thanks
@SyrineKrichene
Copy link
Collaborator

SyrineKrichene commented Nov 23, 2021 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@naserahmadi @SyrineKrichene