Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ABORTED: Shutdown in tf.raw_ops.PriorityQueueV2 #67642

Closed
x0w3n opened this issue May 15, 2024 · 6 comments
Closed

ABORTED: Shutdown in tf.raw_ops.PriorityQueueV2 #67642

x0w3n opened this issue May 15, 2024 · 6 comments
Assignees
Labels
comp:ops OPs related issues stale This label marks the issue/pr stale - to be closed automatically if no activity stat:awaiting response Status - Awaiting response from author TF 2.16 type:bug Bug

Comments

@x0w3n
Copy link

x0w3n commented May 15, 2024

Issue type

Bug

Have you reproduced the bug with TensorFlow Nightly?

Yes

Source

source

TensorFlow version

tf 2.16

Custom code

Yes

OS platform and distribution

Linux Ubuntu 22.04.3 LTS (x86_64)

Mobile device

No response

Python version

3.11

Bazel version

No response

GCC/compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current behavior?

When the component_types parameter of tf.raw_ops.PriorityQueueV2 contains some special characters, such as "-", "*", "!" etc., it will trigger "Local rendezvous is aborting with status: ABORTED: Shutdown" error.

Standalone code to reproduce the issue

import tensorflow as tf
shapes = [1, 1]
component_types = [tf.float32, tf.int32]
capacity = 10
container = "test_container" 
shared_name = "testsharedname"
queue_handle = tf.raw_ops.PriorityQueueV2(
    shapes=shapes,
    component_types=component_types,
    capacity=capacity,
    container=container,
    shared_name=shared_name
)

Relevant log output

2024-05-15 14:25:39.269296: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at resource_op_kernel.h:76 : INVALID_ARGUMENT: container contains invalid characters: test!container
2024-05-15 14:25:39.269340: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: container contains invalid characters: test!container
Traceback (most recent call last):
  File "/home/tf/test.py", line 10, in <module>
    queue_handle = tf.raw_ops.PriorityQueueV2(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/tf2.16/lib/python3.11/site-packages/tensorflow/python/util/tf_export.py", line 377, in wrapper
    return f(**kwargs)
           ^^^^^^^^^^^
  File "/root/anaconda3/envs/tf2.16/lib/python3.11/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 3240, in priority_queue_v2
    return priority_queue_v2_eager_fallback(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/tf2.16/lib/python3.11/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 3312, in priority_queue_v2_eager_fallback
    _result = _execute.execute(b"PriorityQueueV2", 1, inputs=_inputs_flat,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/tf2.16/lib/python3.11/site-packages/tensorflow/python/eager/execute.py", line 53, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__PriorityQueueV2_device_/job:localhost/replica:0/task:0/device:CPU:0}} container contains invalid characters: test!container [Op:PriorityQueueV2]
2024-05-15 14:25:39.422495: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: ABORTED: Shutdown
@sushreebarsa
Copy link
Contributor

@x0w3n tf.raw_ops.PriorityQueueV2 function creates a priority queue data structure. The container name "test_container" contains characters that are not allowed. Could we try to rename the container to use only alphanumeric characters (a-z, A-Z, 0-9) and underscores (_). Avoid special characters like "-", "*", "!". I was able to replicate the issue reported here. Thank you!

@sushreebarsa sushreebarsa added the stat:awaiting response Status - Awaiting response from author label May 16, 2024
@x0w3n
Copy link
Author

x0w3n commented May 16, 2024

@x0w3n tf.raw_ops.PriorityQueueV2 function creates a priority queue data structure. The container name "test_container" contains characters that are not allowed. Could we try to rename the container to use only alphanumeric characters (a-z, A-Z, 0-9) and underscores (_). Avoid special characters like "-", "*", "!". I was able to replicate the issue reported here. Thank you!

Noted with thanks.

@google-ml-butler google-ml-butler bot removed the stat:awaiting response Status - Awaiting response from author label May 16, 2024
@sushreebarsa
Copy link
Contributor

@x0w3n Could you please let us know if the issue got resolved ?

@sushreebarsa sushreebarsa added the stat:awaiting response Status - Awaiting response from author label May 16, 2024
Copy link

This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you.

@github-actions github-actions bot added the stale This label marks the issue/pr stale - to be closed automatically if no activity label May 24, 2024
@x0w3n
Copy link
Author

x0w3n commented May 24, 2024

@x0w3n Could you please let us know if the issue got resolved ?

I have got it. Thank you.

@x0w3n x0w3n closed this as completed May 24, 2024
Copy link

Are you satisfied with the resolution of your issue?
Yes
No

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp:ops OPs related issues stale This label marks the issue/pr stale - to be closed automatically if no activity stat:awaiting response Status - Awaiting response from author TF 2.16 type:bug Bug
Projects
None yet
Development

No branches or pull requests

2 participants