glusterfs pods will not deploy #341
Comments
|
Looks like you've run this more than once? What happened the first time you ran it? Otherwise you somehow manually added a BTW this error should be able to be overcome if you have a recent enough version of this repo that has #339 merged. |
|
@jarrpa i was getting the same the first time i ran it.. do you mind elaborating the last part about overcoming it? |
|
Given that your log ended with If you're raising the issue about the |
|
@jarrpa i tried it and was gettng this... |
|
...yes, that's normal output. Did the deployment fail somehow? |
|
it failed :( |
|
...okay, reset your environment:
Then run |
|
hi @jarrpa it failed |
|
...Hm. So two of your GlusterFS pods are failing, and the third one is missing entirely. Is there anything useful if you run "kubectl describe" on the daemonset or pods? |
@jarrpa heres the output |
|
Does the OS you are using support systemd? |
|
@Justluckyg can you do a describe on the pod, we are looking for an event that references 'dbus' |
|
@erinboyd I don't see how the OS supporting systemd matters, systemd is in the container. And he just gave us the describe of the pod. @Justluckyg sorry for the delay, can you also do a describe of the glusterfs daemonset when it reaches such a state? |
|
@jarrpa we ran into a similar issue with the service broker integration on a non-RHEL OS. The error was bubbling up via the container... |
@erinboyd @jarrpa pls see describe pod of glusterfs daemonset, im using centos 7 core. @erinboyd also to answer your question about systemd, yes. [[ |
|
@Justluckyg That's not the daemonset, that's the pod. I'd like the output of |
|
@jarrpa ohh i didnt know that command, here you go: |
|
@Justluckyg The gluster containers write their logs to the |
|
hi @jarrpa theres nothing in the /var/log/glusterfs, here what i saw in the /var/log/containers |
|
What's your version of Kube? |
|
You might be running into this: #298 |
|
@jarrpa Docker version 1.13.1, build 092cba3 I did try downgrading to 1.12 but that version conflicts with the kubespray ansible i ran to deploy the cluster https://github.com/kubernetes-incubator/kubespray |
|
Yes, but you don't want to do that. :) The systemd folks have said that they do not support running under Your other option is to wait for another release of Kube v1.8, as this PR should also solve the problem: kubernetes/kubernetes#51634 Finally, if you want to get something working now, you can try out these experimental images I've put together more-or-less just for fun: #298 (comment) |
|
@jarrpa appreciate your response.. when will 1.8 be released? |
|
Looks like no date has been set yet. And yeah, replace the glusterfs-daemonset.yaml file. |
|
@jarrpa hi again.. i think im making some progress after cloning your yaml file. I was getting below, how do i change the request of resources to something lower. i testing this in a desktop server only before testing in our production env. |
|
@jarrpa but then the 2 pods that has the sufficient resources are still crashing, logs here below |
|
@jarrpa does having a working glusterfs cluster a pre req before i could even run gk-deploy? and is the instructions below the same ? one more rather "dumb" question, accdg to the requirements, it should have a completely empty block device storage. im running this on a test desktop server with a single disk and using vsphere esxi to create vm's and create a cluster for these vm's. i only added a virtual disk to this nodes and declared those in the topology.json. am i required to use a totally different physical block storage device for this to work? |
nope.. gk-deploy helps in setting up glusterfs cluster setup. Refer this link for pre-requisites:
not required. You can mention virtual disk in topology.json |
|
@Justluckyg As I suspected. Yes, follow the guidance we just gave: run the abort, the check to see if there are any heketi svc or ep resources left. Report if there are any. Run the |
|
@jarrpa i just did the above, i was the getting the same. i manually deleted the heketi service that didnt get deleted by the gk-deploy command. but the heketi-storage is still failing and this is the glusterd logs from that node: >/lib64/libglusterfs.so.0(runner_log+0x105) [0x7f63d0781ee5] ) 0-management: Failed to execute script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=heketidbstorage --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd I did the mount -t glusterfs from node2, didnt make any difference... |
|
@Justluckyg Okay, but you were able to mount, yes? And what was the exact name of the service you had to delete? |
|
@jarrpa with the mount -t glusterfs? i only did that after trying to run the gk-deploy -gvy script while the heketi storage copy job was containercreating. the heketi-service was the one i deleted. |
|
@Justluckyg Right, but it worked? And what was the EXACT name of the service, was it |
|
@jarrpa when i entered the command |
|
@Justluckyg Idea: Try mounting the GlusterFS volume again, on a node not running the copy job pod, and see if you can do an |
|
@jarrpa should i just do |
|
@Justluckyg Yes: Create a new directory somewhere, then run |
|
@jarrpa its unable to mount.. |
|
@Justluckyg Ah-ha! try running the |
|
@jarrpa it didnt take it |
|
@Justluckyg Irritating... see if there are any logs in |
|
@Justluckyg Good start. Look for errors towards the end of |
|
@Justluckyg Also |
|
@jarrpa no error there. all Informational line. But in glusterfs/glusterd.log, im getting this: |
|
@Justluckyg You can ignore those errors, they're "normal". :) The root-test log would only appear on nodes where you've mounted a gluster volume to Do me a favor and check the version of Gluster running in the pods ( |
|
@jarrpa same on all 3 |
|
@Justluckyg Hmm... that might be too old. Do you have any way of updating that? |
|
@jarrpa i noticed upgrading it is not as easy as just doing a yum upgrade as it throws a lot of dependency errors. |
|
@Justluckyg Darn. Well, that's my current thinking, unfortunately. Can you try to see if you can resolve the dependency issues? |
|
@jarrpa sure ill try that, do you have any preferred stable version? |
|
@Justluckyg Preferably at or newer than the version in the GlusterFS pods, which I think is 3.10.5. |
|
whew... after a month of testing and your unwavering support @jarrpa @SaravanaStorageNetwork... I finally successfully completed the deployment and deployed my first storage pvc using your sample nginx hello world application! I cant thank you enough for helping me! Now I am off to testing this further, with that I am glad to close this issue now :) |
|
YEEEESSS!! HAH! :D Happy to hear that! Always feel free to come back any time you need additional help. Or if you just want to gice us praise, we won't turn that down either. ;) |
|
@jarrpa I have the right version of mount.glusterfs, and glusterFS is running on 3 nodes. However, I still see the error: "Waiting for GlusterFS pods to start ... pods not found." |
|
@verizonold If you are still having trouble please open a new issue and provide any information about your environment, what you've done, as well as the output of |
Im trying to deploy this glusterfs in a kubernetes cluster that was deployed via kubespray. I have 3 vm's (on baremetal running centos 7. I believe I followed all the prerequisites but im getting this once i run the ./gk-deploy-g
The text was updated successfully, but these errors were encountered: