about patchset of "enable p2p for 4090"

[yongmei04]

I read the patchset. it' brlliant. congrads:)
I wanna confirm whether my understanding is right.

First of all, the basic idea to enable p2p is to use bus addr (mmio addr for another gpu) to simulate pcie dma transaction between gpu and cpu.
so, it changes the peer address in gmmu pte entry to bus address with aperture sys non-coherent. it's similar as static bar1 mapping.
but it touches the gmmu part of codes.

I did similar things in the past months. but I failed to setup mailbox p2p. No code shows how to setup peer id mappings.
I found the pattern to allocate vidmemory is as belows:
first, allocate 2M based vidmem via IOCTL (NV_ESC_RM_VID_HEAP_CONTROL)
then, allocate range in UVM via UVM_CREATE_EXTERNAL_RANGE
and then, map vidmem to gmmu via UVM_MAP_EXTERNAL_ALLOCATION: this one in fact setups gmmu pte.

if p2p is support and enabled, peer replation should be established in map_rm_pt_range: buld pte in memory and the copy to the gmmu.
copy_pte uses gpu channel to DMA copy the pte content to the physical addr.

There're lots of hardware thing behind. It's very hard to understand the whole process.

So, from my debugging experiences, it's the uvm code that does the actally dma mapping instead of issuing NV_ESC_RM_MAP_MEMORY_DMA.

Maybe there's complicated logic in cuda umd driver to handle allocation and mapping.

So far so good. But, I found you comment out the static bar1 related code in _kbusMapAperture_GM107 and _kbusUnmapAperture_GM107.

Does that mean if the cuda uvm driver doesn't handle dma mapping. simpleP2P test would fail. because you don't pass bus addr in kbus_map/unmap code. Is my understanding is correct?

Looking forward to your reply:)