kcg on k8s

[networkop]

Hey @lspgn 👋
I've done a POC recently of one of your example pipelines (Flows + Kafka + Clicklhouse + Grafana) deployed on top of k8s.
Would you be interested in that? I still have all of the commands and files available.

[lspgn]

Hi @networkop, apologies for the delay. This is great!
Kafka and Clickhouse are big deployments and require operators: it might be interesting to have a Deployment/Helm for GoFlow.
Do you have a repository with the files?

[networkop]

so far I have a bunch of interactive steps. with a bit of effort they can be turned into some flux manifests to deploy all this stuff in one go. Here's how it looks like

Install kafka

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install sflow bitnami/kafka \
--set persistence.enabled=false \
--set zookeeper.persistence.enabled=false

# The resulting kafka API endpoint will be sflow-kafka.sflow.svc.cluster.local:9092

Install clickhouse

curl -LO https://raw.githubusercontent.com/netsampler/goflow2/main/compose/kcg/clickhouse/create.sh
curl -LO  https://raw.githubusercontent.com/netsampler/goflow2/main/compose/kcg/clickhouse/protocols.csv
curl -LO https://raw.githubusercontent.com/netsampler/goflow2/main/pb/flow.proto

sed -i 's/kafka:9092/sflow-kafka.sflow.svc.cluster.local:9092/' create.sh

# this is needed because configmaps are mounted as read-only and clickhouse expects 
# to chown files in these directories and will crash if it can't do it
sed -i '1 a cp /tmp/flow.proto /var/lib/clickhouse/format_schemas/flow.proto' create.sh
sed -i '1 a cp /tmp/protocols.csv /var/lib/clickhouse/user_files/protocols.csv' create.sh

kubectl -n sflow create cm initdb --from-file=create.sh
kubectl -n sflow create cm proto --from-file=flow.proto
kubectl -n sflow create cm csv --from-file=protocols.csv

helm repo add slamdev https://slamdev.github.io/helm-charts/
helm install sflow-db slamdev/altinity-clickhouse-operator --version 0.0.5 
cat << EOF > db.yml
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "sflow-db"
spec:
  configuration:
    users:
      grafana/password: grafana
      grafana/profile: default
      grafana/quota: default
      grafana/networks/ip: "::/0"
    clusters:
      - name: "clickhouse"
        templates:
          podTemplate: db-with-goflow
          serviceTemplates: chi-service-template
        layout:
          shardsCount: 1
          replicasCount: 1
  templates:
    serviceTemplates:
    - name: chi-service-template
      spec:
        type: ClusterIP
    podTemplates:
    - name: db-with-goflow
      spec:
          containers:
            - name: clickhouse
              image: yandex/clickhouse-server:21.5.6.6
              ports:
                - name: http
                  containerPort: 8123
              volumeMounts:
                - name: volume-initdb
                  mountPath: /docker-entrypoint-initdb.d/
                - name: volume-proto
                  mountPath: /tmp/flow.proto
                  subPath: flow.proto
                  readOnly: true
                - name: volume-csv
                  mountPath: /tmp/protocols.csv
                  subPath: protocols.csv
                  readOnly: false
          volumes:
                - name: volume-initdb
                  configMap:
                    name: initdb
                - name: volume-proto
                  configMap:
                    name: proto
                - name: volume-csv
                  configMap:
                    name: csv
EOF

Install goflow2

cat << EOF > deploy.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: goflow
  name: goflow
  namespace: sflow
spec:
  selector:
    matchLabels:
      app: goflow
  template:
    metadata:
      labels:
        app: goflow
    spec:
      containers:
      - image: netsampler/goflow2:latest
        name: goflow
        args:
        - -transport.kafka.brokers=sflow-kafka.sflow.svc.cluster.local:9092
        - -transport=kafka
        - -transport.kafka.topic=flows
        - -format=pb
        - -format.protobuf.fixedlen=true
EOF

k expose -n sflow deploy goflow --type NodePort --port 6343 --protocol UDP
k -n sflow get svc goflow
NAME     TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
goflow   NodePort   10.107.183.222   <none>        6343:30323/UDP   30s

Install Grafana

curl -LO https://raw.githubusercontent.com/netsampler/goflow2/main/compose/kcg/grafana/dashboards/viz-ch.json
curl -LO https://raw.githubusercontent.com/netsampler/goflow2/main/compose/kcg/grafana/datasources-ch.yml
curl -LO https://raw.githubusercontent.com/netsampler/goflow2/main/compose/kcg/grafana/dashboards.yml

sed -i 's/db:8123/clickhouse-sflow-db:8123/' datasources-ch.yml
sed -i "s/user: ''/user: 'grafana'/" datasources-ch.yml
sed -i "s/password: ''/password: 'grafana'/" datasources-ch.yml

kubectl -n sflow create cm grafana-ch --from-file=viz-ch.json
kubectl -n sflow create cm grafana-source --from-file=datasources-ch.yml
kubectl -n sflow create cm grafana-dashboards --from-file=dashboards.yml

kubectl label cm grafana-source grafana_datasource=include
kubectl label cm grafana-ch  grafana_dashboard=include

helm repo add grafana https://grafana.github.io/helm-charts
helm install grafana grafana/grafana \
--set sidecar.datasources.enabled=true \
--set sidecar.datasources.labelValue=include \
--set sidecar.dashboards.enabled=true \
--set sidecar.dashboards.labelValue=include \
--set service.type=NodePort \
--set adminPassword=grafana \
--set plugins=vertamedia-clickhouse-datasource \
--set env.GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=vertamedia-clickhouse-datasource

k get svc grafana
NAME      TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
grafana   NodePort   10.102.189.108   <none>        80:31174/TCP   37s