How to use PublisherServiceAsyncClient

[skhaz]

I'm currently working on an asynchronous project and seeking guidance on implementing PublisherServiceAsyncClient for message publishing in PubSub. Given that we're in the year 2023 and asyncio is widely used, the traditional client doesn't seem feasible for our needs. Could you please provide an example of how PublisherServiceAsyncClient can be used in an asyncio context?

Thank you in advance.

[adriangb]

This would be great.

[pradn]

There's an auto-generated async client here. A sample for using it is here.

However, I don't recommend using it. The auto-generated layers do not have the custom code required to support complex features like publishing with ordering keys (see sample here). There are other smaller differences as well. The Publish call here returns a future, so you could possible massage that into the async API you prefer.

[adriangb]

More documentation and some higher level wrappers are needed. I’m an advanced python user and kept running into issues with rate limiting, etc. I probably needed to implement exponential backoff and such but gave up and run the sync client in threads.

In general the documentation for this library is quite poor. It’s mostly API docs and a handful of examples with little explanation.

[skhaz]

I agree. I just gave up on using these APIs.

[pradn]

@adriangb :

kept running into issues with rate limiting, etc.

Can you please give me some more detail about this?

I probably needed to implement exponential backoff

The lower-level APIs will automatically do retries for you, with the parameters/settings specified here.

In general the documentation for this library is quite poor. It’s mostly API docs and a handful of examples with little explanation.

Sorry about this - we're especially weak in async API docs. I'll bring this up with leads and see what we can do. We haven't gotten that much demand for async APIs generally over the past few years, but this will likely continue to increase as it becomes more idiomatic.

[pradn]

@skhaz

I just gave up on using these APIs.

Can you please give us some more detail about the specific issues you saw? We, of course, want to make the libraries easy to use.

[adriangb]

Can you please give me some more detail about this?

Well I didn't spend that much time troubleshooting but I can say I was getting a lot of 503 and other errors and moving the sync APIs fixed it. Was it because I didn't put sleeps between yielding pull requests? Not sure, there's not docs on it...

[pradn]

503 Unavailable errors should be temporary backend errors. They'll be retried under the hood, but at some point the error will bubble up to you. You should generally use exponential backoff, but the retry schedule is up to you. If you have sample code, I can take a look. :)

[steve-marmalade]

Linking my comment here in case it helps anyone else who wants to use GCP Pub/Sub in an async environment (such as a FastAPI web server): #389 (comment)

[skhaz]

This is not async.

[steve-marmalade]

This is not async.

@skhaz , is this in response to my comment? If so, can you expand a little bit on what you mean?

Here's what I'm suggesting:

from google.cloud import pubsub_v1

publisher_client = pubsub_v1.PublisherClient()
topic_path = "..."

async def send_message():
  future = publisher_client.publish(topic_path, b'Hello, World')
  message_id = await asyncio.wrap_future(future)
  return message_id