Setup Pixie locally with Colima fails to initialise ElasticSearch

[dcfranca]

Hello team,
First, I'd like to say that I'm very excited to try out Pixie

I was trying to install locally on a local Kubernetes cluster on top of Colima, is it supported? I was not sure but I tried anyway as this is how the local development happens at my company

I'm running on a MacOS M3 Max with Sonoma 14.4

Colima version

colima version 0.6.8
git commit: 9b0809d0ed9ad3ff1e57c405f27324e6298ca04f

runtime: docker
arch: aarch64
client: v25.0.0
server: v24.0.7

Then I have the first issue on the Install Pixie Cloud section

A few pods in CrashLoopBackoff

postgres-56f89f7d58-vlmns                  1/1     Running            1 (49m ago)      75m
artifact-tracker-server-76ddc4d6bc-5d4dp   1/1     Running            1 (49m ago)      75m
config-manager-server-6d4767f49b-w8ddd     1/1     Running            1 (49m ago)      75m
hydra-7c757587bf-pxrh8                     2/2     Running            2 (49m ago)      75m
profile-server-7bdbb6b487-dmf96            1/1     Running            2 (49m ago)      75m
cron-script-server-6545944d64-tmgjp        1/1     Running            2 (49m ago)      75m
plugin-server-6fc5474d7b-kr2rw             1/1     Running            2 (49m ago)      75m
auth-server-54cd5cd974-rptf4               1/1     Running            2 (49m ago)      75m
kratos-5fd6cd6c7d-ltbmr                    2/2     Running            2 (49m ago)      75m
metrics-server-6dc964b448-prsfm            1/1     Running            2 (49m ago)      75m
project-manager-server-5d798ff9f6-sm2m5    1/1     Running            2 (49m ago)      75m
pl-nats-2                                  1/1     Running            1 (49m ago)      75m
pl-nats-1                                  1/1     Running            1 (49m ago)      75m
pl-nats-0                                  1/1     Running            1 (49m ago)      75m
vzmgr-server-6db898b648-dj8zl              1/1     Running            3 (48m ago)      75m
vzconn-server-6499968767-4zln8             1/1     Running            3 (48m ago)      75m
scriptmgr-server-5b468f58ff-xlrfk          1/1     Running            3 (48m ago)      75m
pl-elastic-es-data-0                       0/1     CrashLoopBackOff   23 (2m35s ago)   75m
pl-elastic-es-master-0                     0/1     CrashLoopBackOff   11 (2m9s ago)    33m
indexer-server-58f7846bb7-5hzj9            0/1     CrashLoopBackOff   25 (97s ago)     75m
api-server-5ccbc5b66c-gs78n                0/1     CrashLoopBackOff   26 (50s ago)     75m
cloud-proxy-7897b497cb-d6mdk               1/2     CrashLoopBackOff   32 (33s ago)     68m
pl-elastic-es-master-1                     0/1     Running            12 (5m6s ago)    36m
plugin-db-updater-job-bd6mf                0/1     PodInitializing    0                2s

It seems to be a cascade issue due to the elastic search pods crashing
Logs from the elastic search:

sysctl vm.max_map_count = 262144
Stream closed EOF for plc/pl-elastic-es-master-0 (sysctl)
elasticsearch Exception in thread "main" java.io.IOException: Cannot run program "/usr/share/elasticsearch/jdk/bin/java": error=0, Failed to exec spawn helper.
elastic-internal-init-filesystem Starting init script
elastic-internal-init-filesystem Linking /mnt/elastic-internal/xpack-file-realm/users to /usr/share/elasticsearch/config/users
elastic-internal-init-filesystem Linking /mnt/elastic-internal/xpack-file-realm/roles.yml to /usr/share/elasticsearch/config/roles.yml
elastic-internal-init-filesystem Linking /mnt/elastic-internal/xpack-file-realm/users_roles to /usr/share/elasticsearch/config/users_roles
elastic-internal-init-filesystem Linking /mnt/elastic-internal/elasticsearch-config/elasticsearch.yml to /usr/share/elasticsearch/config/elasticsearch.yml
elasticsearch     at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
elasticsearch     at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
elasticsearch     at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:111)
elasticsearch     at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:88)
elasticsearch     at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:59)
elasticsearch     at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:95)
elasticsearch Caused by: java.io.IOException: error=0, Failed to exec spawn helper.
elasticsearch     at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
elasticsearch     at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:319)
elasticsearch     at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:250)
elasticsearch     at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
elasticsearch     ... 5 more
elastic-internal-init-filesystem Linking /mnt/elastic-internal/unicast-hosts/unicast_hosts.txt to /usr/share/elasticsearch/config/unicast_hosts.txt
elastic-internal-init-filesystem File linking duration: 1 sec.
elastic-internal-init-filesystem Copying /usr/share/elasticsearch/config/* to /mnt/elastic-internal/elasticsearch-config-local/
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/elasticsearch.yml' -> '/mnt/elastic-internal/elasticsearch-config-local/elasticsearch.yml'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs/ca.crt' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/ca.crt'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs/tls.key' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/tls.key'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs/tls.crt' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/tls.crt'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs/..data' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/..data'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs/..2024_03_15_18_24_41.3325295662' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/..2024_03_15_18_24_41.3325295662'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs/..2024_03_15_18_24_41.3325295662/ca.crt' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/..2024_03_15_18_24_41.3325295662/ca.crt'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs/..2024_03_15_18_24_41.3325295662/tls.key' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/..2024_03_15_18_24_41.3325295662/tls.key'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs/..2024_03_15_18_24_41.3325295662/tls.crt' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/..2024_03_15_18_24_41.3325295662/tls.crt'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/jvm.options' -> '/mnt/elastic-internal/elasticsearch-config-local/jvm.options'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/log4j2.properties' -> '/mnt/elastic-internal/elasticsearch-config-local/log4j2.properties'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/role_mapping.yml' -> '/mnt/elastic-internal/elasticsearch-config-local/role_mapping.yml'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/roles.yml' -> '/mnt/elastic-internal/elasticsearch-config-local/roles.yml'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/transport-remote-certs' -> '/mnt/elastic-internal/elasticsearch-config-local/transport-remote-certs'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/transport-remote-certs/ca.crt' -> '/mnt/elastic-internal/elasticsearch-config-local/transport-remote-certs/ca.crt'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/transport-remote-certs/..data' -> '/mnt/elastic-internal/elasticsearch-config-local/transport-remote-certs/..data'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/transport-remote-certs/..2024_03_15_18_24_41.3095262462' -> '/mnt/elastic-internal/elasticsearch-config-local/transport-remote-certs/..2024_03_15_18_24_41.3095262462'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/transport-remote-certs/..2024_03_15_18_24_41.3095262462/ca.crt' -> '/mnt/elastic-internal/elasticsearch-config-local/transport-remote-certs/..2024_03_15_18_24_41.3095262462/ca.crt'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/unicast_hosts.txt' -> '/mnt/elastic-internal/elasticsearch-config-local/unicast_hosts.txt'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/users' -> '/mnt/elastic-internal/elasticsearch-config-local/users'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/users_roles' -> '/mnt/elastic-internal/elasticsearch-config-local/users_roles'
elastic-internal-init-filesystem Empty dir /usr/share/elasticsearch/plugins
elastic-internal-init-filesystem Copying /usr/share/elasticsearch/bin/* to /mnt/elastic-internal/elasticsearch-bin-local/
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-certgen' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-certgen'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-certutil' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-certutil'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-cli' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-cli'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-croneval' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-croneval'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-env' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-env'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-env-from-file' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-env-from-file'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-keystore' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-keystore'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-migrate' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-migrate'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-node' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-node'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-plugin' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-plugin'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-saml-metadata' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-saml-metadata'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-setup-passwords' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-setup-passwords'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-shard' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-shard'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-sql-cli' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-sql-cli'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-sql-cli-7.6.0.jar' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-sql-cli-7.6.0.jar'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-syskeygen' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-syskeygen'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-users' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-users'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/x-pack-env' -> '/mnt/elastic-internal/elasticsearch-bin-local/x-pack-env'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/x-pack-security-env' -> '/mnt/elastic-internal/elasticsearch-bin-local/x-pack-security-env'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/x-pack-watcher-env' -> '/mnt/elastic-internal/elasticsearch-bin-local/x-pack-watcher-env'
elastic-internal-init-filesystem Files copy duration: 1 sec.
elastic-internal-init-filesystem chowning /usr/share/elasticsearch/data to elasticsearch:elasticsearch
elastic-internal-init-filesystem ownership of '/usr/share/elasticsearch/data' retained as elasticsearch:elasticsearch
elastic-internal-init-filesystem chowning /usr/share/elasticsearch/logs to elasticsearch:elasticsearch
elastic-internal-init-filesystem changed ownership of '/usr/share/elasticsearch/logs' from root:root to elasticsearch:elasticsearch
elastic-internal-init-filesystem chown duration: 1 sec.
elastic-internal-init-filesystem waiting for the transport certificates (/mnt/elastic-internal/transport-certificates/pl-elastic-es-master-0.tls.key)
elastic-internal-init-filesystem wait duration: 0 sec.
elastic-internal-init-filesystem Linking /usr/share/elasticsearch/config/transport-certs/pl-elastic-es-master-0.tls.crt to /mnt/elastic-internal/elasticsearch-config-local/node-transport-cert/transport.tls.crt
elastic-internal-init-filesystem Linking /usr/share/elasticsearch/config/transport-certs/pl-elastic-es-master-0.tls.crt to /mnt/elastic-internal/elasticsearch-config-local/node-transport-cert/transport.tls.crt
elastic-internal-init-filesystem Certs linking duration: 1 sec.
elastic-internal-init-filesystem Init script successful
elastic-internal-init-filesystem Script duration: 4 sec.
Stream closed EOF for plc/pl-elastic-es-master-0 (elastic-internal-init-filesystem)
Stream closed EOF for plc/pl-elastic-es-master-0 (elasticsearch)

I tried to continue the setup, but then I get other errors, I believe this one should be fixed first
For example, running dev_dns_updater gets stuck after typing the password

Any idea what could be causing ElasticSearch failure? What I found googling was related to not working on Apple Silicion, but it seems that since then it was fixed: https://stackoverflow.com/questions/65962810/m1-mac-issue-bringing-up-elasticsearch-cannot-run-jdk-bin-java

[dcfranca]

Any idea?

[dcfranca]

Anyone?

[JamesMBartlett]

The version of Elasticsearch in the stackoverflow you linked is 7.10.2. Pixie currently uses 7.6. It's possible this is an issue with elasticsearch that got fixed between 7.6 and 7.10.2.

You could try changing this line

pixie/k8s/cloud_deps/base/elastic/cluster/elastic_cluster.yaml

Line 8 in cf88e33

image: gcr.io/pixie-oss/pixie-dev-public/elasticsearch:7.6.0-patched1@sha256:f734909115be9dba66736c4b7356fd52da58b1ffdb895ba74cb5c2fca2b133dd

to use the image suggested in the stackoverflow post.

[dcfranca]

@JamesMBartlett If I change the image to the one from the StackOverflow answer I get

chroot: failed to set supplemental groups: Operation not permitted

Not sure if it might be because the one from the SO answer is a public image, while the Pixie one is a patched image

[dcfranca]

Ok, I managed to make it work removing the securityContext capabilities restrictions, which is not ideal, but it should work for local tests, if you know a Pixie patched image that I can use instead it would help

             allowPrivilegeEscalation: false                                                                                                                │   • (×) Install SWE app
-            capabilities:                                                                                                                                  │   • (×) Check Active OrderUp
-              add:                                                                                                                                         │   • (×) List deployments in execution (spin)
-              - SYS_CHROOT                                                                                                                                 │   • (×) Grab latest build in Teran
-              - SETUID                                                                                                                                     │
-              drop:                                                                                                                                        │   • (×) OrderUp: Python Client
-              - ALL