Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

failed to create s390x libvirt cluster using kcli #1835

Closed
lysliu opened this issue May 7, 2024 · 10 comments
Closed

failed to create s390x libvirt cluster using kcli #1835

lysliu opened this issue May 7, 2024 · 10 comments

Comments

@lysliu
Copy link
Contributor

lysliu commented May 7, 2024

When I follow the guide https://github.com/confidential-containers/cloud-api-adaptor/tree/main/src/cloud-api-adaptor/libvirt#create-the-kubernetes-cluster
try to create libvirt cluster
I got error as

~/src/confidential-containers/cloud-api-adaptor/src/cloud-api-adaptor# ./libvirt/kcli_cluster.sh create
Download ubuntu2204 s390x image
Using pool default
Grabbing image ubuntu2204 from url https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-s390x.img
Image ubuntu2204 already there.Leaving...
Using 192.168.122.253 as api_ip
Using keepalived virtual_router_id 15
Using version v1.26.7
Deploying Vms...
Traceback (most recent call last):
  File "/usr/local/bin/kcli", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/usr/local/lib/python3.12/site-packages/kvirt/cli.py", line 5512, in cli
    args.func(args)
  File "/usr/local/lib/python3.12/site-packages/kvirt/cli.py", line 1887, in create_generic_kube
    create_kube(args)
  File "/usr/local/lib/python3.12/site-packages/kvirt/cli.py", line 1866, in create_kube
    result = config.create_kube(cluster, kubetype, overrides=overrides)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kvirt/config.py", line 2688, in create_kube
    result = self.create_kube_generic(cluster, overrides=overrides)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kvirt/config.py", line 2713, in create_kube_generic
    return kubeadm.create(self, plandir, cluster, overrides)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kvirt/cluster/kubeadm/__init__.py", line 219, in create
    result = config.plan(plan, inputfile=f'{plandir}/bootstrap.yml', overrides=data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kvirt/config.py", line 2157, in plan
    result = self.create_vm(name, profilename, overrides=currentoverrides, customprofile=profile, k=z,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kvirt/config.py", line 478, in create_vm
    full_volumes = self.k.volumes()
                   ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kvirt/providers/kvm/__init__.py", line 1980, in volumes
    if self.get_capabilities()['arch'] == 'aarch64':
       ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kvirt/providers/kvm/__init__.py", line 188, in get_capabilities
    cpuxml = self.conn.baselineCPU([ET.tostring(cpu, encoding='unicode')], 1)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/libvirt.py", line 4288, in baselineCPU
    raise libvirtError('virConnectBaselineCPU() failed')
libvirt.libvirtError: XML error: Missing CPU model name
@lysliu
Copy link
Contributor Author

lysliu commented May 7, 2024

I also tried to set cpumodel follow https://kcli.readthedocs.io/en/latest/#available-parameters-for-client-profile-plan-files
still no lucky.

@stevenhorsman
Copy link
Member

@karmab - Hey Karim, I know we got this working on s390x a few months ago. Any idea if something has gone in that might have led to the issues that @lysliu is seeing?

@lysliu lysliu changed the title failed to create s390x libvirt cluster using failed to create s390x libvirt cluster using kcli May 9, 2024
@karmab
Copy link

karmab commented May 9, 2024

I think it's a consequence of the change done in here to properly detect nested feature
I can capture the exception but can you share the output of the following commands?

virsh capabilities
virsh capabilities | virsh cpu-baseline --features /dev/stdin

@stevenhorsman
Copy link
Member

I think it's a consequence of the change done in here to properly detect nested feature I can capture the exception but can you share the output of the following commands?

virsh capabilities
virsh capabilities | virsh cpu-baseline --features /dev/stdin

This is the output:

# virsh capabilities
<capabilities>

  <host>
    <uuid>1b6fbde4-0fdb-45ee-a5aa-31e6ca618e50</uuid>
    <cpu>
      <arch>s390x</arch>
      <topology sockets='16' dies='1' cores='1' threads='1'/>
      <pages unit='KiB' size='4'/>
      <pages unit='KiB' size='1024'/>
    </cpu>
    <power_management/>
    <iommu support='no'/>
    <migration_features>
      <live/>
      <uri_transports>
        <uri_transport>tcp</uri_transport>
        <uri_transport>rdma</uri_transport>
      </uri_transports>
    </migration_features>
    <topology>
      <cells num='1'>
        <cell id='0'>
          <memory unit='KiB'>32954940</memory>
          <pages unit='KiB' size='4'>8238735</pages>
          <pages unit='KiB' size='1024'>0</pages>
          <distances>
            <sibling id='0' value='10'/>
          </distances>
          <cpus num='16'>
            <cpu id='0' socket_id='0' die_id='0' core_id='0' siblings='0'/>
            <cpu id='1' socket_id='1' die_id='0' core_id='1' siblings='1'/>
            <cpu id='2' socket_id='2' die_id='0' core_id='2' siblings='2'/>
            <cpu id='3' socket_id='3' die_id='0' core_id='3' siblings='3'/>
            <cpu id='4' socket_id='4' die_id='0' core_id='4' siblings='4'/>
            <cpu id='5' socket_id='5' die_id='0' core_id='5' siblings='5'/>
            <cpu id='6' socket_id='6' die_id='0' core_id='6' siblings='6'/>
            <cpu id='7' socket_id='7' die_id='0' core_id='7' siblings='7'/>
            <cpu id='8' socket_id='8' die_id='0' core_id='8' siblings='8'/>
            <cpu id='9' socket_id='9' die_id='0' core_id='9' siblings='9'/>
            <cpu id='10' socket_id='10' die_id='0' core_id='10' siblings='10'/>
            <cpu id='11' socket_id='11' die_id='0' core_id='11' siblings='11'/>
            <cpu id='12' socket_id='12' die_id='0' core_id='12' siblings='12'/>
            <cpu id='13' socket_id='13' die_id='0' core_id='13' siblings='13'/>
            <cpu id='14' socket_id='14' die_id='0' core_id='14' siblings='14'/>
            <cpu id='15' socket_id='15' die_id='0' core_id='15' siblings='15'/>
          </cpus>
        </cell>
      </cells>
    </topology>
    <secmodel>
      <model>apparmor</model>
      <doi>0</doi>
    </secmodel>
    <secmodel>
      <model>dac</model>
      <doi>0</doi>
      <baselabel type='kvm'>+64055:+108</baselabel>
      <baselabel type='qemu'>+64055:+108</baselabel>
    </secmodel>
  </host>

  <guest>
    <os_type>hvm</os_type>
    <arch name='s390x'>
      <wordsize>64</wordsize>
      <emulator>/usr/bin/qemu-system-s390x</emulator>
      <machine maxCpus='248'>s390-ccw-virtio-jammy</machine>
      <machine canonical='s390-ccw-virtio-jammy' maxCpus='248'>s390-ccw-virtio</machine>
      <machine maxCpus='248'>s390-ccw-virtio-4.0</machine>
      <machine maxCpus='248'>s390-ccw-virtio-5.2</machine>
      <machine maxCpus='248'>s390-ccw-virtio-artful</machine>
      <machine maxCpus='248'>s390-ccw-virtio-3.1</machine>
      <machine maxCpus='248'>s390-ccw-virtio-groovy</machine>
      <machine maxCpus='248'>s390-ccw-virtio-hirsute</machine>
      <machine maxCpus='248'>s390-ccw-virtio-2.6</machine>
      <machine maxCpus='248'>s390-ccw-virtio-disco</machine>
      <machine maxCpus='248'>s390-ccw-virtio-2.12</machine>
      <machine maxCpus='248'>s390-ccw-virtio-yakkety</machine>
      <machine maxCpus='248'>s390-ccw-virtio-2.9</machine>
      <machine maxCpus='248'>s390-ccw-virtio-eoan</machine>
      <machine maxCpus='248'>s390-ccw-virtio-6.0</machine>
      <machine maxCpus='248'>s390-ccw-virtio-5.1</machine>
      <machine maxCpus='248'>s390-ccw-virtio-3.0</machine>
      <machine maxCpus='248'>s390-ccw-virtio-4.2</machine>
      <machine maxCpus='248'>s390-ccw-virtio-2.5</machine>
      <machine maxCpus='248'>s390-ccw-virtio-2.11</machine>
      <machine maxCpus='248'>s390-ccw-virtio-xenial</machine>
      <machine maxCpus='248'>s390-ccw-virtio-impish</machine>
      <machine maxCpus='248'>s390-ccw-virtio-focal</machine>
      <machine maxCpus='248'>s390-ccw-virtio-2.8</machine>
      <machine maxCpus='248'>s390-ccw-virtio-bionic</machine>
      <machine maxCpus='248'>s390-ccw-virtio-5.0</machine>
      <machine maxCpus='248'>s390-ccw-virtio-6.2</machine>
      <machine maxCpus='248'>s390-ccw-virtio-zesty</machine>
      <machine maxCpus='248'>s390-ccw-virtio-4.1</machine>
      <machine maxCpus='248'>s390-ccw-virtio-cosmic</machine>
      <machine maxCpus='248'>s390-ccw-virtio-2.4</machine>
      <machine maxCpus='248'>s390-ccw-virtio-2.10</machine>
      <machine maxCpus='248'>s390-ccw-virtio-2.7</machine>
      <machine maxCpus='248'>s390-ccw-virtio-6.1</machine>
      <domain type='qemu'/>
      <domain type='kvm'/>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <disksnapshot default='on' toggle='no'/>
    </features>
  </guest>

</capabilities>


root@sh-libvirt-s390x:~# virsh capabilities | virsh cpu-baseline --features /dev/stdin
error: XML error: Missing CPU model name

@karmab
Copy link

karmab commented May 9, 2024

ok, karmab/kcli@9b41c02 should cover it then

@stevenhorsman
Copy link
Member

ok, karmab/kcli@9b41c02 should cover it then

Awesome. I'll give it a try once it's through the pipeline!

@stevenhorsman
Copy link
Member

@karmab - I'm getting a different, but similar failure now with the latest code:

# kcli version
version: 99.0 commit: 9b41c02 2024/05/09 Available Updates: False
root@sh-libvirt-s390x:~/go/src/github.com/confidential-containers/cloud-api-adaptor/src/cloud-api-adaptor# kcli create kube generic -P image=ubuntu2204 test2
Using 192.168.122.253 as api_ip
Using keepalived virtual_router_id 113
Using version v1.30.0
Deploying Vms...
Hypervisor not compatible with nesting. Skipping
Traceback (most recent call last):
  File "/usr/local/bin/kcli", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.10/dist-packages/kvirt/cli.py", line 5415, in cli
    args.func(args)
  File "/usr/local/lib/python3.10/dist-packages/kvirt/cli.py", line 1887, in create_generic_kube
    create_kube(args)
  File "/usr/local/lib/python3.10/dist-packages/kvirt/cli.py", line 1866, in create_kube
    result = config.create_kube(cluster, kubetype, overrides=overrides)
  File "/usr/local/lib/python3.10/dist-packages/kvirt/config.py", line 2688, in create_kube
    result = self.create_kube_generic(cluster, overrides=overrides)
  File "/usr/local/lib/python3.10/dist-packages/kvirt/config.py", line 2713, in create_kube_generic
    return kubeadm.create(self, plandir, cluster, overrides)
  File "/usr/local/lib/python3.10/dist-packages/kvirt/cluster/kubeadm/__init__.py", line 219, in create
    result = config.plan(plan, inputfile=f'{plandir}/bootstrap.yml', overrides=data)
  File "/usr/local/lib/python3.10/dist-packages/kvirt/config.py", line 2157, in plan
    result = self.create_vm(name, profilename, overrides=currentoverrides, customprofile=profile, k=z,
  File "/usr/local/lib/python3.10/dist-packages/kvirt/config.py", line 957, in create_vm
    result = k.create(name=name, virttype=virttype, plan=plan, profile=profilename, flavor=flavor,
  File "/usr/local/lib/python3.10/dist-packages/kvirt/providers/kvm/__init__.py", line 1345, in create
    conn.defineXML(vmxml)
  File "/usr/local/lib/python3.10/dist-packages/libvirt.py", line 4441, in defineXML
    raise libvirtError('virDomainDefineXML() failed')
libvirt.libvirtError: XML error: No PCI buses available

@karmab
Copy link

karmab commented May 9, 2024

hum, that one looks like unrelated to the fix, I would say you need to force the machine type
can you share the xml of a running vm?

@stevenhorsman
Copy link
Member

The latest version with karmab/kcli@f4d5c01 is working for me locally now. Thanks as always Karim. @lysliu - you should be unblocked now if you upgrade with pip :)

@lysliu
Copy link
Contributor Author

lysliu commented May 21, 2024

I confirmed, the latest pip fixed the issue now.

@lysliu lysliu closed this as completed May 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@karmab @lysliu @stevenhorsman