ncm-network: nmstate ipoib support

[kwaegema]

current ncm-network using nmstate doesn't seem to support ipoib.
@aka7 Is there a way to do this on EL9 using ncm-network?

I've seen two issues:

• interface got renamed

mlx5_core 0000:a1:00.0 ibp161s0: renamed from ib0

So ib0 in profile is not recognized as ib device, and it got treated as a bond interface :

[ERROR] Error '/usr/bin/nmstatectl apply /etc/nmstate/ib0.yml' output: [2024-02-28T16:29:13Z ERROR nmstate::ifaces::bond] InvalidArgument: Bond mode is mandatory for new bond interface: ib0
NmstateError: InvalidArgument: Bond mode is mandatory for new bond interface: ib0

When I manually rename the interface

ip link set dev ibp161s0 name ib0

this changes to

2024/03/01-16:53:11 [VERB] Getting output of command: /usr/bin/nmstatectl apply /etc/nmstate/ib0.yml
2024/03/01-16:53:11 [ERROR] Error '/usr/bin/nmstatectl apply /etc/nmstate/ib0.yml' output: NmstateError: InvalidArgument: Invalid propriety: unknown field `infiniband

• The generated yaml is not correct for IB devices

I can manually start the ib interface using nmstatectl apply when i change the yaml

[root@node4230 ~]# diff -bu ib0.yml ib0.yml_ok
--- ib0.yml	2024-03-01 17:11:50.585099054 +0100
+++ ib0.yml_ok	2024-03-01 17:16:50.498239387 +0100
@@ -7,12 +7,13 @@
       prefix-length: 19
     dhcp: false
     enabled: true
-  link-aggregation:
-    port: []
   name: ib0
   profile-name: ib0
   state: up
-  type: bond
+  type: infiniband
+  infiniband:
+    mode: datagram
+    pkey: '0xffff'
 routes:
   config:
   - next-hop-interface: ib0

[aka7]

You are right in the fact that nmstste module doesn't support this type. In fact not sure if ncm-network supported it? Maybe maintainers can confirm?

There has been few fixes went into master that would have stopped it from being configured as type bond though.

Regarding renaming of interface, nmstate module in ncm-network doesn't rename interfaces, so that's probably happening outside of ncm-network.

[aka7]

@kwaegema sorry, to answer your question, I think we can look to add the support in nmstate, if you provide full details of config needed for a infiniband type configuration,
and if we can separate the schema for nmstate only options, which @jrha is going to see if it's possible.

[stdweird]

@aka7 we will investigate, but this was all working in old ncm-network. but some of the details are handled by OFED config files and services, maybe something is also going wrong there as well (eg we are missing some rpms that handle this). there is interface renaming code in the old ncm-network though; not sure how this works anymore

[kwaegema]

Hi @aka7
thanks! i don't find a lot of docs on ib config with nmstate, but this is a basic example what is needed:

https://github.com/nmstate/nmstate/blob/base/examples/infiniband_pkey_ipoib_create.yml

interfaces:
  - name: mlx5_ib0.80ff
    type: infiniband
    state: up
    infiniband:
      base-iface: mlx5_ib0
      mode: datagram
      pkey: '0x80ff'

right now I use this to get it running when i manually renamed the interface (that part is to be investigated)

interfaces:
- ipv4:
    address:
    - ip: 10.143.7.160
      prefix-length: 19
    dhcp: false
    enabled: true
  name: ib0
  profile-name: ib0
  state: up
  type: infiniband
  infiniband:
    mode: datagram
    pkey: '0xffff'

[stdweird]

@kwaegema some of this data comes from ncm-ofed, like the datagram mode. not sure how we can mix this

[kwaegema]

@kwaegema some of this data comes from ncm-ofed, like the datagram mode. not sure how we can mix this

we already set some ncm-network params based on this, like:
SELF[bonddev]["mtu"] = if (IPOIB_CM) { ...
so we'll set this in the profile i guess

[kwaegema]

Hi @aka7 Would you have time to look into this #1665 (comment)
No problem if not, but then we know we have to look into it ourselves :)
Thanks!

[aka7]

@kwaegema I thought there were questions round how this can be mix with ncm-network and ncm-fed, question from @stdweird ?

I think its probably worth you looking into this setup since I don't really have any test host to do these type of test. other then looking at generating the expected output.
But there are few things that needs to be discussed though, such as how we separate the core-schama and nmstate only schema. something which was raised before, anything we do to support this issue will be nmstate specific only.

[stdweird]

@aka7 wrt infiniband, i just checked: for initscript network config, there is no explicit infiniband support; and all is left to (ncm-)ofed.
i can add infiniband data to the core network schema, and change ncm-ofed to read some stuff from the network schema instead of it's own schema; or some clever mix to make it compatible.
i'll try to work this out next week