Unexpected behavior in tf.data.Dataset.from_generator when there is a naming collision

[kralka]

Issue type

Documentation Bug

Have you reproduced the bug with TensorFlow Nightly?

Yes

Source

source

TensorFlow version

2.15.0

Custom code

Yes

OS platform and distribution

colab

Mobile device

No response

Python version

3.10.12

Bazel version

No response

GCC/compiler version

11.4.0

CUDA/cuDNN version

No response

GPU model and memory

No response

Current behavior?

When creating a tf.data.Dataset object using from_generator a naming collision with an iterable used in the generator parameter causes a crash.

On Ubuntu I got the following error (not in colab):
date: F external/local_tsl/tsl/platform/default/env.cc:74] Check failed: ret == 0 (11 vs. 0)Thread tf_data_private_threadpool creation via pthread_create() failed.

Standalone code to reproduce the issue

import sys
print(f"Python version: {sys.version}")

import tensorflow as tf
print(f"{tf.__version__ = }")

# This runs as expected:

ds_no_name_collision = list(range(100))

def gen():
  print("gen called")
  for x in ds_no_name_collision:
    print(f"{x = }")
    yield x

# Notice there is no name collision:
ds_without_name_collision = tf.data.Dataset.from_generator(gen, output_signature=tf.TensorSpec(shape=(), dtype=tf.int32))

print("All good so far")

for x in ds_without_name_collision.take(10):
  print(x)



# This fails:

ds_name_collision = list(range(100))

def gen():
  print("gen called")
  for x in ds_name_collision:
    print(f"{x = }")
    yield x

# Notice the name collision:
ds_name_collision = tf.data.Dataset.from_generator(gen, output_signature=tf.TensorSpec(shape=(), dtype=tf.int32))

print("All good so far")

for x in ds_name_collision.take(10):
  print(x)

Relevant log output

Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
tf.__version__ = '2.15.0'
All good so far
gen called
x = 0
x = 1
tf.Tensor(0, shape=(), dtype=int32)
tf.Tensor(1, shape=(), dtype=int32)
x = 2
x = 3
x = 4
tf.Tensor(2, shape=(), dtype=int32)
x = 5
tf.Tensor(3, shape=(), dtype=int32)
tf.Tensor(4, shape=(), dtype=int32)
tf.Tensor(5, shape=(), dtype=int32)
x = 6
x = 7
x = 8
x = 9
tf.Tensor(6, shape=(), dtype=int32)
tf.Tensor(7, shape=(), dtype=int32)
tf.Tensor(8, shape=(), dtype=int32)
tf.Tensor(9, shape=(), dtype=int32)
All good so far
gen called
gen called
gen called
gen called
gen called
gen called
gen called
[log truncated]

[SuryanarayanaY]

Hi @kralka ,

I have replicated the issue with TF2.15v and tf-nightly as well. On Colab the code executes indefinitely due to naming collision of generator and finally runtime will crash without any logs. Attaching gist for reference.

[kralka]

@SuryanarayanaY thank you for your response. One can get a crash (as opposed to indefinite execution without errors) in an official docker (docker pull tensorflow/tensorflow:latest-gpu). I will provide concrete steps if needed.

[kralka]

Hi @SuryanarayanaY is there some plan to resolve this issue? Do you know what causes it? In case resolving would require too large changes do you think it is worth putting a note in the documentation since it is rather hard to figure out what causes this behavior?

Personally I would be interested in the cause of this since some magic needs to be happening here :-)