prov/verbs;ofi_rxm: FI_HMEM fails to be detected when out of cache

[thomasgillis]

Hi all,

I am reaching out with a question that might also be an issue in verbs;ofi_rxm:
when exceeding the cache size (default is 1024), the address is interpreted as a system address by default in fi_writemsg, instead of a device one. Increasing the size of the cache with FI_MR_CACHE_MAX_COUNT solves the problem.

Is my issue a direct consequence of FI_MR_HMEM and the limit on the cache size? Or is it a missing detection of the pointer type in the provider?

Thanks for your time and your help :-)