Failed to deploy stackStorm HA through MicroK8S #281
Comments
|
First, ensure the backends like Redis, RabbitMQ, MongoDB are working. Are those backends up and running? Check the logs. |
|
I think RabbitMQ is working. And I looked back at the error message and it showed a line about rabbitMQ : amqp.exceptions.NotFound: Queue.declare: (404) NOT_FOUND - home node 'rabbit@stackstorm-rabbitmq-0.stackstorm-rabbitmq-headless.default.svc.cluster.local' of durable queue 'st2.preinit' in vhost '/' is down or inaccessible Is it caused by this problem? How to fix it |
|
@armab 2022-02-06 08:35:45.319 [info] <0.2547.0> accepting AMQP connection <0.2547.0> (10.1.16.138:44746 -> 10.1.16.181:5672) I tried to solve the problem by restarting the server. It seems to me that the time out was due to a service requiring RabbitMQ that had not yet been started. I do not know whether my analysis is correct, please help to answer it. In addition, if a server suddenly goes down in the reproduction environment, it is unacceptable if the problem I described occurs after the restart. Can I adjust a parameter value to avoid this problem? I am a beginner stackStorm please help me thanks. |
|
StackStorm services are failing and restarting by K8s until they reach a working connection from the backends. So as long as you fix RabbitMQ, StackStorm cluster auto heals and recovers. |
|
@armab I fixed the problem this time by restarting it again. However, if this problem is encountered in the production environment, I think it cannot be solved by restarting the service again. Is there another way to do it? |
|
In your case Redis backend was failing too: As said before, first you need to ensure all the backends (Redis, MongoDB, RabbitMQ) are operable and available. Can you check what was the problem with Redis from the |
@armab I used this commend |
|
I think the port is working |
|
And she kept displaying the following warning.
|
|
I believe You showed the logs for the |
|
[root@centos-master ~]# kubectl logs stackstorm-redis-node-0 -c sentinel |
|
I'm experiencing the exact same issue with Microk8s and Stackstorm HA. All the errors ShimingLee is displaying is exactly what I see. Only difference is, when I type "kubectl logs stackstorm-redis-node-0 -c sentinel", I don't see anything. Doesn't say it can't connect to Redis or anything. Now, I've gotten MongoDB, RabbitMQ and Redis working. Here is what they look like: MongoDB: RabbitMQ: Redis: Yet the waiting-for-queue container looks like this: The container runs this weird command to determine if RabbitMQ is active: I tried changing the stackstorm-rabbitmq to stackstorm-rabbitmq-headless and it didn't work. My guess is some sort of DNS issue, because sometimes the wait-for-db container also gets stuck on waiting for MongoDB connection. |
|
OK. I got some progress. Come to find out, the stackstorm-ha helm chart is just poorly written (no surprise). So, for those of you who have Microk8s, this information may be helpful.
|
|
@QuintonMcLeod - Are you suggesting the way helm deploys StackStorm-HA causes the issue with |
I'm suggesting both.
|
|
Thanks @QuintonMcLeod. Definitely an interesting find. The reason for my ask was because several of us have not faced this and was wondering if there is something different in the way this works in Btw, if the script is POC, it's fine. Otherwise, you have hardwired the service name variables with |
Changing the name wasn't anything I did "directly", as the name is totally based on the name you provide during the helm chart install. As for your experience using other Kubernetes deployments: Microk8s always seems to be the underdog. No one sees issues with it because they don't use it. Kubernetes deployments differ, and unique problems occur. |
|
Anyway, I just ended up using the docker-compose version on a spare server and abandoned the Stackstorm-HA helm chart. It just plain doesn't work on Microk8s. |
|
We've seen in the past issues with The e2e tests in CircleCI are running with So yeah, indeed the probability to hit the edge case increases with the different K8s variants. The chart is supported by the community who's using it and contributing back fixes and improvements. |
|
Hi, can u help me with st2 on microk8s? I have problem |
I ended up abandoning StackStorm on MicroK8s, because it's not worth the headache. Of course, this was 2 years ago, so maybe they fixed those issues? No idea. I went after the docker-compose version because it was better maintained, plus it was contained. I could easily delete the folder if I didn't like it anymore. |
Thanks for reply. But now I don't sure that stackstorm it's helpfully tool. I think more easy solve is python script. Stackstorm it's just python blocks of if then |
This is a strange way of looking at things. StackStorm allows you to set up users, set up roles for those users, it allows you to set up sensors, rules, schedules, configuration in the datastore for different processes and other things. If you just need to execute a python script, then for sure ST2 is a bit overkill for the task. However, if you need to have a leveraged environment where different users can do various things in a relatively user friendly way, just raw python doesn't do it. |
U are right. In future the tool is scalable for the project. I want to use that, but I have only problems with install. Maybe u can help me. StackStorm/st2#6198 |
|
If anyone wants to work on fixing support for micro-k8s, please start with submitting a PR that adds CI testing with micro-k8s. Then you can work on identifying which part(s) of the helm chart are not compatible with micro-k8s to see if fixing that is feasible. I don't use micro-k8s, and I don't have the bandwidth to investigate anything with it. Volunteers welcome! |
Maybe u can help me to get messages from queue rabbitmq with st2? Because I have st2 on docker compose and I have rabbitmq on my desktop. And I want to get new messages from queue. I install plugin rabbitmq, I can publish message in the queue. I use rule if rabbitmq.new_message. St2 connected to with rabbitmq but in 1 minute, connection is disappeared. And st2 doesn't see new messages. What I do wrong? |
Hi. Thanks for your interesting my question. Maybe u can help me? I write problem with st2-rabbitmq in previous message. |
If you want to use your locally installed Rabbit-MQ install, you'll need to allow StackStorm (or whatever pods you're running) see outside of its cluster. There's several different ways to do this. The way I do it is by letting the deployment use my host's networking. If you're using Docker-Compose, I don't recommend you alter it to use your Rabbit-MQ, because the Docker-Compose instance is designed in such a way to be used and then thrown away like used toilet paper. |
How about NOT using Bitnami as your template? Bitnami always has permission issues in their releases (especially within Microk8s), because their releases always want full unmolested access to the storage. Microk8s HATES this - as it should, because from a security standpoint, a deployment has no business having the type of permission Bitnami always demands for. When I worked at University on our Kubernetes cluster, we ALWAYS avoided Bitnami releases for this very reason! |
Several pods failed to start when the server was restarted.
view pod the error log :
How can I solve this problem?????
The text was updated successfully, but these errors were encountered: