Request & Response sync.Pool usage

[jzelinskie]

Is your feature request related to a problem? Please describe.

High performance gRPC/Connect services need to avoid allocations for request and response messages.
In the gRPC ecosystem, it's possible to pool request messages with plugins like vtprotobuf, but impossible to pool response messages, as you do not control the lifetime of the response object.

Describe the solution you'd like

Ideally pooling is just built into connect-go so that no one has to think about it.
We live in a world of generics now, so this might be possible.

Describe alternatives you've considered

An alternative would be to provide callback/interface that executes when the library needs a request and is finished with a response.

[jzelinskie]

Related planetscale/vtprotobuf#59

[akshayjshah]

Pooling is certainly a nice optimization, but it's error-prone in this context: no matter how we structure connect-go, we'd need to be completely sure that no interceptor has retained a reference to the message or any of its sub-structures. Because pooling errors are so irritating to debug, I've always avoided exposing pooled objects across team boundaries.

I'm hopeful that arenas will be a safer route to improved overall performance.

...but impossible to pool response messages, as you do not control the lifetime of the response object.

Why not use runtime.SetFinalizer? Finalizers are somewhat distasteful and have some sharp edges, but they're a good fit for this use case. You'd still need to ensure the interceptors you use don't retain references to any sub-structures, byte slices, etc., but you wouldn't need any cooperation from the connect-go runtime.

[jzelinskie]

Finalizers have their own overhead cost to performance and are not always viable.

Even if errors are difficult to debug, there are going to be services where coordinating with the GC is the bottleneck in performance. Aren't arenas going to have the same issue of retaining references? Is the plan to simply bulk allocate with arenas to avoid overhead?

[akshayjshah]

Finalizers do have a cost, and they've got plenty of sharp edges. And yes, arenas will have the same issues as pools with retaining references - but the errors are loud, explicit, and much easier to address than silent data corruption. Arenas were introduced specifically for request-scoped protobuf messages within google3, so I'm hopeful that they'll work well with protobuf in OSS too.

A big part of my reluctance to take on response pooling in connect-go is that it's error-prone and not the typical bottleneck. We'll have exposed an API that's unsafe and difficult to debug when it goes sideways, while still eating tons of small allocations in net/http and being relatively slow compared to higher-performance protocols that don't use HTTP (e.g., drpc).

I'd much rather plow our effort into one of the many places we can safely get significant performance improvements. There's plenty of headroom! As an example, the MarshalAppend APIs introduced in the latest release of the protobuf runtime were contributed by a teammate. We'll start using them in the next connect-go release.