Unclear how libmoon is supposed to interface with Mellanox dev=mlx5_core

[michael-kaplan]

I successfully built libmoon (19.05 branch) with dpdk (19.05 libmoon branch) on Ubuntu 20.04 with Mellanox ConnectX-5 (MLNX_OFED_LINUX-5.3-1.0.0.1-ubuntu20.04-x86_64) firmware 16.28.4512. Note I additionally installed mlnx-ofed-kernel-only deb package (did not install by default) and the libmlx5 libraries seem to be bundled with ibverbs (the libs are present but I do not see any libmlx5 packages).

That said, when I run my libmoon based application, it does not find any usable devices. I am unclear on how this should work given the cards are using dev=mlx5_core which is not custom built by libmoon (unlike igb_uio.ko and rte_kni.ko which are present in the x86_64-native-linux-gcc/kmod directory). Since the cards are not using a newly compiled libmoon dpdk dev driver, I'm not surprised my libmoon application doesn't find the devices. Am I misunderstanding how this is supposed to work?

Here is my output from using MoonCookie (libmoon-based application at https://github.com/syn-proxy/MoonCookie)

[INFO]  Initializing DPDK. This will take a few seconds...
EAL: Detected 64 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /run/user/10002/dpdk/rte/mp_socket
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: Cannot obtain physical addresses: Success. Only vfio will function.
EAL: Couldn't get fd on hugepage file
EAL: Couldn't get fd on hugepage file
error allocating rte services array
EAL: FATAL: rte_service_init() failed
EAL: rte_service_init() failed
[INFO]  Found 0 usable devices:
[FATAL] Lua error in task master
../MoonCookie/libmoon/build/../lua/device.lua:100: there are only 0 ports, tried to configure port id 0
Stack Traceback
===============
(2) Lua method 'fatal' at file '/MoonCookie/libmoon/build/../lua/log.lua:129'
	Local variables:
	 self = table: 0x411091f8  {DEBUG:0, fatal:function: 0x41109580, writeToLog:function: 0x41109560, INFO:1 (more...)}
	 str = string: "there are only 0 ports, tried to configure port id 0"
(3) Lua field 'config' at file 'MoonCookie/libmoon/build/../lua/device.lua:100'
	Local variables:
	 args = table: 0x41b40aa0  {rssQueues:1, rxQueues:1, port:0, txQueues:1}
(4) Lua function 'master' at file 'mooncookie.lua:48' (best guess)
	Local variables:
	 args = table: 0x407b6ca8  {devR:1, batch:63, strategy:2, threads:1, devL:0}
(5) global C function 'xpcall'
(6) Lua upvalue 'master' at file '/MoonCookie/libmoon/build/../lua/main.lua:96'
	Local variables:
	 _ = string: "./libmoon/build/libmoon"
	 file = string: "mooncookie.lua"
	 args = table: 0x403b47c0  {1:-s, 2:auth_invalid, 3:0, 4:1}
	 cfgFile = nil
	 ok = boolean: true
	 parsedArgs = table: 0x407b52f0  {1:table: 0x407b6ca8}
(7) Lua function 'main' at file '../MoonCookie/libmoon/build/../lua/main.lua:146' (best guess)
	Local variables:
	 task = string: "master"
	...MoonCookie/libmoon/build/../lua/device.lua:100: there are only 0 ports, tried to configure port id 0

[emmericp]

mellanox drivers are not compiled in by default (because of their dependencies), you need to pass --mlx5 to the build script, see https://github.com/libmoon/libmoon/blob/master/install-mlx.md

[michael-kaplan]

Thank you for the response. I typically pass "-m" which seems to be the equivalent of --mlx5 to build.sh. I confirm that CONFIG_RTE_LIBRTE_MLX_PMD is changed to y

# Compile burst-oriented Mellanox ConnectX-4, ConnectX-5,
# ConnectX-6 & Bluefield (MLX5) PMD
CONFIG_RTE_LIBRTE_MLX5_PMD=y
CONFIG_RTE_LIBRTE_MLX5_DEBUG=n
# Linking method for mlx4/5 dependency on ibverbs and related libraries
# Default linking is dynamic by linker.
# Other options are: dynamic by dlopen at run-time, or statically embedded.
CONFIG_RTE_IBVERBS_LINK_DLOPEN=n
CONFIG_RTE_IBVERBS_LINK_STATIC=n

Any other suggestions?

[amosawi9999]

@michael-kaplan do have any progress on this? I've faced similar problem with ConnectX-6, OFED 5.4
I can manage compile the libmoon with dpdk 21, the dpdk 21 comes with working testpmd but libmoon could not find any mellanox cards.