feat(pubsub): publish-side changes for ordering keys

[pradn]

• Create a new concept: sequencers. A sequencer orders messages published by the publishing client. Messages with ordering keys use the OrderedSequencer. Those without ordering keys use the UnorderedSequencer, which behaves just like before.
• The OrderedSequencer is paused when it encounters a permanent error. It must be unpaused manually by the user calling client.resumePublish() to continue sending messages for that ordering key. This is done to prevent out-of-order sending.
• Move the commit thread to the publisher client. This one thread commits batches in all sequencers. Before, each batch had its own commit thread.
• Clean up ordering-key enabled sequencers after all messages are sent. (If paused, the OrderedSequencer is not cleaned-up.)
• Added tests for all the changes.

Implements half of #9928

[plamut]

In general the design looks good and is understandable IMO. I did not try using the new feature, focused only on the code for now.

There are quite a few suggestions/comments, but the PR itself is also quite sizable. Some of them are minor/non-essential, but it's my last review cycle of the year, thus a bit biased towards verbosity. :)

On top of everything, this is a non-trivial feature that would highly benefit from at least a basic system test or two (if the ordering keys feature is not yet enabled on the test project, we must enable them first).

[pradn]

Thanks for the review. A system test is forthcoming, after the subscriber side changes.

[kamalaboulhosn]

Generally looks good!

[plamut]

The updates look good, thanks!

There are still a few comments from the original review that have not been addressed yet (most of those that were hidden by default). Please just check them out, or mark as resolved if you think they aren't relevant (enough).

[plamut]

Apart for a minor docstring thingy, it now looks good IMO.

Disclaimer - did not run the code, we need the subscriber side for ordering keys as well (the same reason why there are no system tests yet).

[plamut]

What's the status of this? I see that error handling was changed in one particular case, and that a deadlock was fixed.

@pradn Would you mind elaborating a bit on the latter? Looking at the state machine and the commit message, it appears to be something with publishing in a STOPPED state, and a new state FINISHED had to be introduced? Or why the deadlock could occur? Thanks!

[plamut]

As per offline discussion, we want to wait for the subscriber side of this, as well as for a fix for a performance regression that was discovered.

[plamut]

Superseded by googleapis/python-pubsub#26