Follow-ups on dual-node deployment

[kevinxyliu]

Hi,

I started a discussion regarding how to configure the CDF file for dual node deployment in the previous weeks (based on the guide from cortx-hare), but I have not heard back from that thread, and I'm following up with some additional details about my current setup. It would be greatly appreciated if anyone can provide some insights!

I have two nodes with different hostnames and IP addresses that can ssh into each other. In particular, suppose that the hostname of the first node is xinyu-cortx-1.novalocal, the IP address of the first node is 10.52.3.24, the hostname of the second node is xinyu-cortx-2.novalocal, and the IP address of the second node is 10.52.0.80. The following two operations are successful without requiring passwords:
ssh cc@10.52.0.80 # from first node to second node
ssh cc@10.52.3.24 # from second node to first node

When configuring the CDF file for dual-node setup, I'm mostly confused with the following two issues:

1. Do I need to have cortx-hare installed on the second node?
2. I'm currently configuring the CDF in the first node as follows:

nodes:
  - hostname: localhost        # [user@]hostname
    data_iface: eth0     # name of data network interface
    transport_type: libfab
    m0_servers:
      - runs_confd: true
        io_disks:
          data: []
      - io_disks:
          data:
            - path: /dev/sdd
            - path: /dev/sde
            - path: /dev/sdf
            - path: /dev/sdg
    m0_clients:
      - name: m0_client_other  # name of the motr client
        instances: 2   # Number of instances, this host will run
  - hostname: xinyu-cortx-2.novalocal
    data_iface: eth1
    data_iface_ip_addr: 10.52.0.80
    transport_type: libfab
    m0_servers:
      - runs_confd: true
        io_disks:
          data: []
      - io_disks:
          data:
            - path: /dev/sdh
            - path: /dev/sdi
            - path: /dev/sdj
            - path: /dev/sdk
    m0_clients:
      - name: m0_client_other  # name of the motr client
        instances: 2   # Number of instances, this host will run

My question is, the data interface was originally set to be eth0 or eth1, but how should I put in the IP address information here? Should it belong to data_iface_ip_addr, or data_iface? Are there other fields that I need to change in order to make this work?

Thanks a lot! I have been stuck on this for more than one week, so any help would be greatly appreciated!

[huanghua78]

Please take a look at the example in cortx-hare:

cfgen/examples/ci-boot3.yaml

nodes:
  - hostname: localhost        # [user@]hostname
    data_iface: eth0     # name of data network interface
    transport_type: libfab

As I said in your previous post, the 'hostname' is the hostname of your node, here it should be xinyu-cortx-1.
data_iface, as comment says, this is the network interface you are using.
Please find out which interface you are using and its IP address it 10.52.3.24
Modify this according to your local config. Please refer to the output of ifconfig command.

Don't introduce some new key/val. data_iface_ip_addr: 10.52.0.80 will not be accepted.
Remove this.
No need to add IP address in the CDF file.

[kevinxyliu]

Thanks so much for the response! The example at cfgen/examples/ci-boot3.yaml is very helpful in explaining how the CDF file should look like. Indeed, I'm still confused how the two nodes would communicate with each other without myself plugging in the IP address of the second node. Currently my CDF file is as follows:

nodes:
  - hostname: xinyu-cortx.novalocal
    data_iface: eth3
    transport_type: libfab
    m0_servers:
      - runs_confd: true
        io_disks:
          data: []
      - io_disks:
          data:
            - path: /dev/nvme0n1
            - path: /dev/nvme1n1
    m0_clients:
      - name: m0_client_other  # name of the motr client
        instances: 2   # Number of instances, this host will run
  - hostname: xinyu-cortx-2.novalocal
    data_iface: eth3
    transport_type: libfab
    m0_servers:
      - runs_confd: true
        io_disks:
          data: []
      - io_disks:
          data:
            - path: /dev/nvme2n1
    m0_clients:
      - name: m0_client_other  # name of the motr client
        instances: 2   # Number of instances, this host will run
pools:
  - name: the pool
    disk_refs:
      - { path: /dev/nvme0n1, node: xinyu-cortx.novalocal }
      - { path: /dev/nvme1n1, node: xinyu-cortx.novalocal }
      - { path: /dev/nvme2n1, node: xinyu-cortx-2.novalocal }
    #type: sns  # optional; supported values: "sns" (default), "dix", "md"
    data_units: 1
    parity_units: 0
    #allowed_failures: { site: 0, rack: 0, encl: 0, ctrl: 0, disk: 0 }

Here, eth3 is the network interface associated with the IP address of the first node (the output of ifconfig), and "xinyu-cortx.novalocal" and "xinyu-cortx-2.novalocal" are the hostnames that I got by executing "hostname" on node 1 and node 2. I think this should look the same as the example, but the hostname for node 2 is not recognized. I tried deleting the second node and the one-node setup is working appropriately. Just to clarify, am I supposed to put in the IP address of the second node somewhere else? What other things might be going wrong here? Thanks a lot!

[hessio]

No HARE does not need to be installed on the second node. You need to change the localhost name for node 1 to the output of hostname command. You also will need to remove this line data_iface_ip_addr: 10.52.0.80

[hessio]

Hi @kevinxyliu feel free to reach out again if you are still having issues

[kevinxyliu]

Thanks so much for the response! The example at cfgen/examples/ci-boot3.yaml is very helpful in explaining how the CDF file should look like. Indeed, I'm still confused how the two nodes would communicate with each other without myself plugging in the IP address of the second node. Currently my CDF file is as follows:

nodes:
  - hostname: xinyu-cortx.novalocal
    data_iface: eth3
    transport_type: libfab
    m0_servers:
      - runs_confd: true
        io_disks:
          data: []
      - io_disks:
          data:
            - path: /dev/nvme0n1
            - path: /dev/nvme1n1
    m0_clients:
      - name: m0_client_other  # name of the motr client
        instances: 2   # Number of instances, this host will run
  - hostname: xinyu-cortx-2.novalocal
    data_iface: eth3
    transport_type: libfab
    m0_servers:
      - runs_confd: true
        io_disks:
          data: []
      - io_disks:
          data:
            - path: /dev/nvme2n1
    m0_clients:
      - name: m0_client_other  # name of the motr client
        instances: 2   # Number of instances, this host will run
pools:
  - name: the pool
    disk_refs:
      - { path: /dev/nvme0n1, node: xinyu-cortx.novalocal }
      - { path: /dev/nvme1n1, node: xinyu-cortx.novalocal }
      - { path: /dev/nvme2n1, node: xinyu-cortx-2.novalocal }
    #type: sns  # optional; supported values: "sns" (default), "dix", "md"
    data_units: 1
    parity_units: 0
    #allowed_failures: { site: 0, rack: 0, encl: 0, ctrl: 0, disk: 0 }

Here, eth3 is the network interface associated with the IP address of the first node (the output of ifconfig), and "xinyu-cortx.novalocal" and "xinyu-cortx-2.novalocal" are the hostnames that I got by executing "hostname" on node 1 and node 2. I think this should look the same as the example, but the hostname for node 2 is not recognized. I tried deleting the second node and the one-node setup is working appropriately. Just to clarify, am I supposed to put in the IP address of the second node somewhere else? What other things might be going wrong here? Thanks a lot!

[huanghua78]

but the hostname for node 2 is not recognized

You should config the hostname as accessible from each other.

Add entries in /etc/hosts to solve this.

[kevinxyliu]

Thanks a lot! It seems that configuring the hostname in /etc/hosts solves my previous issue. (I am following the linux manual for the hosts call to add the entry, and I modified both /etc/hosts files for the two nodes with an additional entry "the other node's IP address + the other node's hostname). However, when running hctl bootstrap, although both hostnames are recognized, I got the following error messages, which is very confusing to me:
[ERROR] Exiting with FAILURE status. Reason: invalid literal for int() with base 10: 'Please'

This looks like an error message from python when someone calls int('Please'). However, I looked through the CDF file and did not find any single 'please' word in it. I have also never modified other files where this world might appear. Had anyone encountered similar issues before? Any insights on where the issue comes up would be helpful! And thanks again for the previous instructions on how to make hostnames accessible to each other.

[hessio]

could you paste the entire error output message so we can look?