Q: preferred way to implement GPU communication

[angainor]

I am implementing a communication pattern where GPUs exchange parts of their local data vector. The exchanged vector entries are 'unstructured' (arbitrary indices) with block size of ~8KB: for each communication index the sender GPU sends ~1k contiguous doubles.

I implemented this pattern with MPI_Type_indexed + MPI_Isend and it works. My question is, which of the following implementations is expected to be most efficient:

• use MPI_Isend directly, without packing, with the newly defined indexed type (I guess OpenMPI allocates some internal buffer?)
• use MPI_Pack with the newly defined type, copying data to a GPU buffer, then use MPI_Isend on the packed buffer
• use MPI_Pack with the newly defined type, copying data to a CPU buffer, then use MPI_Isend on the packed buffer

Or is there any other, better way to implement such scenario?

Thank you!