kDbench: Kubernetes Stroage Performance Test

dbench source image from leeliu/dbench

kDbench DEMO

---

Requirement

k8s's API >= 1.21.x or feature gate ttlSecondsAfterFinished is enable

[root@m-k8s ~]# kubectl get nodes
NAME     STATUS   ROLES                  AGE   VERSION
m-k8s    Ready    control-plane,master   54d   v1.23.1
w1-k8s   Ready    <none>                 54d   v1.23.1
w2-k8s   Ready    <none>                 54d   v1.23.1
w3-k8s   Ready    <none>                 54d   v1.23.1

Less than kubernetes version in 1.21.x, it won't be work properly due to the ttlSecondsAfterFinished feature cannot enable as a default on Managed k8s(eks,aks,gke)
Thus in case of self-hostsed or others should turn on ttlSecondsAfterFinished

Purpose

Simple and automatically benchmark for your k8s storageclasses.
(i.e. for lazy engineer who like me)

---

Usage

Highly recommanedation to install fzf before you run.
fzf could support interactively select a storageclass.

  kdbench                           : Select storageclass with fzf
  kdbench <STORAGECLASS NAME>       : Select storageclass manually (w/o fzf)
  kdbench -q, --quick               : Quick run mode        (w/ fzf)
  kdbench -d, --direct              : Direct fio write mode (w/ fzf)
  kdbench -a  <rwo,rox,rwx,rwop>    : Set access mode       (w/ fzf)
  kdbench -s  <storage size>        : Set stroage size      (w/ fzf)
  kdbench -f  <YAML FILE>           : Run as a file         (w/ fzf)
  kdbench reset                     : Reset all of resources
  kdbench -h, --help                : Show this message

Interactive mode

$ kdbench 
# select one of them 
efs-sc 
gp2

# run automatically 
persistentvolumeclaim/kdbench-pv-claim created
job.batch/kdbench created
Waiting for kdbench's load....Working dir: /ON-efs-sc

Testing Read IOPS...
read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.29-148-ge3de2
Starting 1 process
read_iops: Laying out IO file (1 file / 2048MiB)
<snipped>
All tests complete for [efs-sc]

=============================
= Kubernetes Dbench Summary =
=============================
Random Read/Write IOPS............................ [24.4k / 7330]
Bandwidth Read/Write.............................. [300MiB/s / 100MiB/s]
Average Latency Read(usec)/Write(usec)............ [2542.07 / 7907.11]
Sequential Read/Write............................. [301MiB/s / 102MiB/s]
Mixed Random Read/Write IOPS...................... [7121 / 2391]
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Waiting for job finished by ttlSecondsAfterFinished...

# delete automatically
kdbench finsihed and pvc will delete
persistentvolumeclaim "kdbench-pv-claim" deleted

Manual mode

$ kdbench <storageclass name>
<snipped>
All tests complete for [standard-rwx]

=============================
= Kubernetes Dbench Summary =
=============================
Random Read/Write IOPS............................ [53.3k / 1572]
Bandwidth Read/Write.............................. [942MiB/s / 126MiB/s]
Average Latency Read(usec)/Write(usec)............ [264.60 / 2812.10]
Sequential Read/Write............................. [1228MiB/s / 128MiB/s]
Mixed Random Read/Write IOPS...................... [4674 / 1575]
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Waiting for job finished by ttlSecondsAfterFinished...

# delete automatically 
kdbench finsihed and pvc will delete
persistentvolumeclaim "kdbench-pv-claim" deleted

Other options

Quick mode

When you choose quick-mode, Only IOPS and Bandwidth will be tested

$ kdbench -q 

OR

$ kdbench --quick 

Direct fio write

If value is true, use non-buffered I/O. This is usually O_DIRECT. Note that OpenBSD and ZFS on Solaris don’t support direct I/O. On Windows the synchronous ioengines don’t support direct I/O. Default: false. https://fio.readthedocs.io/en/latest/fio_doc.html

$ kdbench -d 

OR

$ kdbench --direct 

Set access-mode

The access modes is changable.

- RWO  | rwo   (ReadWriteOnce) <<<<< **default** 
- ROX  | rox   (ReadOnlyMany)              
- RWX  | rwx   (ReadWriteMany)             
- RWOP | rwop  (ReadWriteOncePod)          

Set storage size

Size is matter somtimes. For instance GKE's filestore

$ kdbench -s <storage size> 

Run as a file

If you want to run various options. you could change whatever you want. Here is kdbench-sample.yaml in working directory. Free to change but you should input storageclass manually.

$ kdbench -f <YAML FILE>

kdbench-sample.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: kdbench-pv-claim
  labels:
    fio: kdbench
spec:
  storageClassName: << INPUT STORAGECLASS >> # Must change in here 
  # storageClassName: managed-nfs-storage 
  # storageClassName: gp2 
  # storageClassName: default 
  # storageClassName: standard 
  # storageClassName: nks-block-storage
  accessModes:
    - ReadWriteOnce   # (optional) RWO - ReadWriteOnce, ROX - ReadOnlyMany, RWX - ReadWriteMany, RWOP - ReadWriteOncePod
  resources:
    requests:
      storage: 100Gi  # (optional) filestore standard - 1000Gi, filestore premium - 2560Gi 
---
<snipped>

From fio file

[TBD] by ConfigMap

Reset

When abnormal behavior has been occurred, please run reset and then rerun kdbench command.

$ kdbench reset

---

Install kDbench

Using git

git clone --depth 1 https://github.com/sysnet4admin/kdbench.git  ~/.kdbench
~/.kdbench/install

---

Uninstall kDbench

~/.kdbench/uninstall