lvmd does not recognize volume group #815

ryanfaircloth · 2024-01-14T13:13:20Z

ryanfaircloth
Jan 14, 2024

Output from pvdisplay

debug-pv-pod:/# pvdisplay 
  --- Physical volume ---
  PV Name               /dev/nvme1n1
  VG Name               instancestore
  PV Size               <872.65 GiB / not usable 4.83 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              223397
  Free PE               223397
  Allocated PE          0
  PV UUID               M6L8Ul-AYKH-5BV1-vGzH-fKgz-DtoV-Ip0Aph

output from lvmd

{"level":"info","ts":"2024-01-14T12:52:45Z","msg":"configuration file loaded","device_classes":[{"name":"default","volume-group":"instancestore","default":true,"spare-gb":10,"stripe":null,"stripe-size":"","lvcreate-options":null,"type":"","thin-pool":null}],"socket_name":"/run/topolvm/lvmd.sock","file_name":"/etc/topolvm/lvmd.yaml"}
{"level":"info","ts":"2024-01-14T12:52:45Z","msg":"invoking LVM command","args":["fullreport","--reportformat","json","--units","b","--nosuffix","--configreport","vg","-o","vg_name,vg_uuid,vg_size,vg_free","--configreport","lv","-o","lv_uuid,lv_name,lv_full_name,lv_path,lv_size,lv_kernel_major,lv_kernel_minor,origin,origin_size,pool_lv,lv_tags,lv_attr,vg_name,data_percent,metadata_percent,pool_lv","--configreport","pv","-o,","--configreport","pvseg","-o,","--configreport","seg","-o,"]}
not found
{"level":"error","ts":"2024-01-14T12:52:45Z","msg":"volume group not found","volume_group":"instancestore","error":"not found","stacktrace":"github.com/topolvm/topolvm/pkg/lvmd/cmd.subMain\n\t/workdir/pkg/lvmd/cmd/root.go:70\ngithub.com/topolvm/topolvm/pkg/lvmd/cmd.glob..func2\n\t/workdir/pkg/lvmd/cmd/root.go:45\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992\ngithub.com/topolvm/topolvm/pkg/lvmd/cmd.Execute\n\t/workdir/pkg/lvmd/cmd/root.go:130\nmain.main\n\t/workdir/pkg/hypertopolvm/main.go:38\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
Error: not found

values from helm chart

    cert-manager:
      enabled: false
    scheduler:
      enabled: false
      type: Deployment
      service:
        type: ClusterIP
      updateStrategy:
        rollingUpdate:
          maxUnavailable: 1
        type: RollingUpdate
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app.kubernetes.io/name
                    operator: In
                    values:
                      - topolvm
                  - key: app.kubernetes.io/component
                    operator: In
                    values:
                      - scheduler
              topologyKey: kubernetes.io/hostname
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: kubernetes.io/os
                    operator: In
                    values:
                      - linux
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              preference:
                matchExpressions:
                  - key: role
                    operator: NotIn
                    values:
                      - system
            - weight: 100
              preference:
                matchExpressions:
                  - key: kubernetes.io/arch
                    operator: In
                    values:
                      - arm64

      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: topolvm
              app.kubernetes.io/component: scheduler
          matchLabelKeys:
            - pod-template-hash
      priorityClassName: "system-cluster-critical"
    lvmd:
     
      deviceClasses:
        - name: default
          volume-group: instancestore
          default: true
          spare-gb: 10
      priorityClassName: system-node-critical
      tolerations:
        - key: ebs.csi.aws.com/agent-not-ready
          operator: Exists
        - key: ebs.csi.aws.com/agent-not-ready
          operator: Exists
        - key: topolvm.github.io/agent-not-ready
          operator: Exists
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: kubernetes.io/os
                    operator: In
                    values:
                      - linux
                  - key: karpenter.k8s.aws/instance-local-nvme
                    operator: Exists
    node:
      priorityClassName: system-node-critical
      tolerations:
        - key: ebs.csi.aws.com/agent-not-ready
          operator: Exists
        - key: ebs.csi.aws.com/agent-not-ready
          operator: Exists
        - key: topolvm.github.io/agent-not-ready
          operator: Exists
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: kubernetes.io/os
                    operator: In
                    values:
                      - linux
                  - key: karpenter.k8s.aws/instance-local-nvme
                    operator: Exists
    controller:
      updateStrategy:
        rollingUpdate:
          maxUnavailable: 1
        type: RollingUpdate
      affinity: |
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app.kubernetes.io/name
                    operator: In
                    values:
                      - topolvm
                  - key: app.kubernetes.io/component
                    operator: In
                    values:
                      - controller
              topologyKey: kubernetes.io/hostname
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: kubernetes.io/os
                    operator: In
                    values:
                      - linux
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              preference:
                matchExpressions:
                  - key: role
                    operator: NotIn
                    values:
                      - system
            - weight: 100
              preference:
                matchExpressions:
                  - key: kubernetes.io/arch
                    operator: In
                    values:
                      - arm64

      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: topolvm
              app.kubernetes.io/component: controller
          matchLabelKeys:
            - pod-template-hash
      priorityClassName: "system-cluster-critical"
    storageClasses:
      - name: nvme-xfs # Defines name of storage class.
        storageClass:
          # Supported filesystems are: ext4, xfs, and btrfs.
          fsType: xfs
          # reclaimPolicy
          reclaimPolicy: Delete
          # Additional annotations
          annotations: {}
          # Default storage class for dynamic volume provisioning
          # ref: https://kubernetes.io/docs/concepts/storage/dynamic-provisioning
          isDefaultClass: false
          # volumeBindingMode can be either WaitForFirstConsumer or Immediate. WaitForFirstConsumer is recommended because TopoLVM cannot schedule pods wisely if volumeBindingMode is Immediate.
          volumeBindingMode: WaitForFirstConsumer
          # enables CSI drivers to expand volumes. This feature is available for Kubernetes 1.16 and later releases.
          allowVolumeExpansion: true
          additionalParameters:
            "topolvm.io/device-class": "default"
          # mount options
          mountOptions: []
          allowedTopologies:
            - matchLabelExpressions:
                - key: karpenter.k8s.aws/instance-local-nvme
                  operator: Exists

Answered by pluser

Jan 17, 2024

So can we then close this discussion by assuming that TopoLVM was irrelevant?

View full answer

ryanfaircloth · 2024-01-14T14:35:37Z

ryanfaircloth
Jan 14, 2024
Author

ran the lvm report command used by lvd

lvm fullreport --reportformat json --units b --nosuffix --configreport vg -o vg_name,vg_uuid,vg_size,vg_free --configreport lv -o lv_uuid,lv_name,lv_full_name,lv_path,lv_size,lv_kernel_major,lv_kernel_minor,origin,origin_size,pool_lv,lv_tags,lv_attr,vg_name,data_percent,metadata_percent,pool_lv --configreport pv -o, --configreport pvseg -o, --configreport seg -o,

output

{
      "report": [
          {
              "vg": [
                  {"vg_name":"instancestore", "vg_uuid":"LQgWX0-SuDR-QZx2-4wmL-PpBU-4tll-81fvKP", "vg_size":"936994930688", "vg_free":"936994930688"}
              ]
              ,
              "pv": [
                  {}
              ]
              ,
              "lv": [
              ]
              ,
              "pvseg": [
                  {}
              ]
              ,
              "seg": [
              ]
          }
      ]
  }

0 replies

ryanfaircloth · 2024-01-14T23:05:26Z

ryanfaircloth
Jan 14, 2024
Author

Additional context I am currently using AWS EKS 1.28 with AL2 host instances I did some additional testing by running my own instance of the lvmd pod with a bash entry point. I found when I ran the container lvmd correctly identified my vgs in comparing my basic debug pod and the lvmd pod from the ds I noticed hostpid: true. I patched the manifest and redeployed with hostpid false and it works. Not sure I understand why true would be a problem. I have another customer with the 13.0.0 chart on AKS v1.27 without an issue

3 replies

pluser Jan 15, 2024
Maintainer

Thank you for your report.
TopoLVM does not officially support Kubernetes 1.28 yet. We are in the process of testing the product, so it may not behave as expected.

ryanfaircloth Jan 15, 2024
Author

Understood, and that was missed but clearly documented in the root README. Thank you in the mean time can I help/participate in 1.28 qualification as this is important to cloud IAAS K8s deployments that will see end of support for older releases soonish?

pluser Jan 16, 2024
Maintainer

Support for kubernetes 1.28 is about to be worked on. We can't promise anything about a timing, but you can track the status of the work on this issue.

ryanfaircloth · 2024-01-15T13:17:19Z

ryanfaircloth
Jan 15, 2024
Author

Has there been any consideration to mapping /dev into the pod and including lvm2 in the pod so commands don't require nsenter?

3 replies

toshipp Jan 16, 2024
Maintainer

LVM command uses posix semaphores, it should run in the host namespaces.
https://access.redhat.com/solutions/6073491

ryanfaircloth Jan 16, 2024
Author

I can't access this page I do have a developer network subscription

toshipp Jan 17, 2024
Maintainer

The page says what I said, lvm should run in the host.

ryanfaircloth · 2024-01-16T19:48:17Z

ryanfaircloth
Jan 16, 2024
Author

Well It seems I was mistaken I did not have AKS and GKE deployments the EKS deployment was not using topolvm but OpenEBS running my builds seems to work back to the drawing board for what is wrong

1 reply

pluser Jan 17, 2024
Maintainer

So can we then close this discussion by assuming that TopoLVM was irrelevant?

Answer selected by ryanfaircloth

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lvmd does not recognize volume group #815

{{title}}

Replies: 4 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

lvmd does not recognize volume group #815

ryanfaircloth Jan 14, 2024

Replies: 4 comments · 7 replies

ryanfaircloth Jan 14, 2024 Author

ryanfaircloth Jan 14, 2024 Author

pluser Jan 15, 2024 Maintainer

ryanfaircloth Jan 15, 2024 Author

pluser Jan 16, 2024 Maintainer

ryanfaircloth Jan 15, 2024 Author

toshipp Jan 16, 2024 Maintainer

ryanfaircloth Jan 16, 2024 Author

toshipp Jan 17, 2024 Maintainer

ryanfaircloth Jan 16, 2024 Author

pluser Jan 17, 2024 Maintainer

ryanfaircloth
Jan 14, 2024

Replies: 4 comments 7 replies

ryanfaircloth
Jan 14, 2024
Author

ryanfaircloth
Jan 14, 2024
Author

pluser Jan 15, 2024
Maintainer

ryanfaircloth Jan 15, 2024
Author

pluser Jan 16, 2024
Maintainer

ryanfaircloth
Jan 15, 2024
Author

toshipp Jan 16, 2024
Maintainer

ryanfaircloth Jan 16, 2024
Author

toshipp Jan 17, 2024
Maintainer

ryanfaircloth
Jan 16, 2024
Author

pluser Jan 17, 2024
Maintainer