Helm CDS problems

[sean-abbott]

Hello! I'm completely new to CDS and mostly new to kubernetes, so I apologize if this is the wrong place to ask.

I'm trying to use your helm chart to install this into a private only EKS cluster. So far so good, everything looks happy excpet the UI, which gets a response of
Error creating load balancer (will retry): failed to ensure load balancer for service default/thing: could not find any suitable subnets for creating the ELB

I've tried dropping the eks private load balancer annotation in in a couple of spots (service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0), specifically just in the ui manifest, but it has not worked.

If anyone has any suggestions I would very much appreciate them.

[bnjjj]

Hello @sean-abbott no problem for your issue. It seems that it's more an AWS issue. I make some research and found this issue kubernetes/kubernetes#29298 . If I have a good understanding it seems that AWS doesn't support loadbalancer with a subnet. Maybe you can to use our helm with an ingress configuration as explained here https://github.com/ovh/cds/blob/master/contrib/helm/cds/README.md#configuration

Let me know if it helps you.

[sean-abbott]

Hmmm...I actually wonder if I needed to make sure the subnets were tagged with kubernetes.io/role/internal-elb or something similar...

[sean-abbott]

Ok, so, i switched to kops which I've had working before, and I'm getting

$ kubectl get pods
NAME                                           READY     STATUS             RESTARTS   AGE
my-cds-cds-api-ff69c8786-smrvm                 0/1       CrashLoopBackOff   6          7m
my-cds-cds-elasticsearch-57b78bc666-j4bwp      0/1       CrashLoopBackOff   5          7m
my-cds-cds-hatchery-k8s-64699d59bb-zfsxv       0/1       CrashLoopBackOff   6          7m
my-cds-cds-hooks-57fbcc66bc-6cwdd              0/1       CrashLoopBackOff   5          7m
my-cds-cds-repositories-f59db9ddd-r2f7j        0/1       CrashLoopBackOff   5          7m
my-cds-cds-ui-64c7995b59-jmwgc                 1/1       Running            0          7m
my-cds-cds-vcs-674b647b7f-29z8s                0/1       CrashLoopBackOff   5          7m
my-cds-elasticsearch-client-577cdf9dc5-tkshc   1/1       Running            0          7m
my-cds-elasticsearch-data-0                    1/1       Running            0          7m
my-cds-elasticsearch-master-0                  1/1       Running            0          7m
my-cds-postgresql-86bfc9749c-d2hgh             1/1       Running            0          7m
my-cds-redis-master-0                          1/1       Running            0          7m
my-cds-redis-slave-589bfc9778-dr5tw            1/1       Running            0          7m

with errors either like

$ kubectl logs my-cds-cds-hatchery-k8s-64699d59bb-zfsxv
Starting service hatchery:kubernetes
Unable to init service hatchery:kubernetes: ApplyConfiguration: internal server error (caused by: Cannot create client with newForConfig: tls: failed to find any PEM data in certificate input)

Or

$ kubectl logs my-cds-cds-vcs-674b647b7f-29z8s
Reading configuration file config.toml
Starting service vcs
2019-04-05 15:54:04 [ERROR] vcs> Unable to register: Register: internal server error (caused by: Post http://my-cds-cds-api/services/register: dial tcp 100.64.42.180:80: i/o timeout)
Service has been stopped: vcs Register: internal server error (caused by: Post http://my-cds-cds-api/services/register: dial tcp 100.64.42.180:80: i/o timeout)

But trcking back to everything trying to register to the api, and the api complaining about

base64: invalid option -- 'o'
Try 'base64 --help' for more information.

Grabbing the container, it doesn't look like the version of base64 in that container has that option:

nobody@935a74383f33:/app$ base64 --help
Usage: base64 [OPTION]... [FILE]
Base64 encode or decode FILE, or standard input, to standard output.

Mandatory arguments to long options are mandatory for short options too.
  -d, --decode          decode data
  -i, --ignore-garbage  when decoding, ignore non-alphabet characters
  -w, --wrap=COLS       wrap encoded lines after COLS character (default 76).
                          Use 0 to disable line wrapping

      --help     display this help and exit
      --version  output version information and exit

With no FILE, or when FILE is -, read standard input.

After I changed -o to > (like the other examples), I still get
Unable to init service api: your CDS configuration seems to be empty. Please use environment variables, file or Consul to set your configuration

However, I can get the contents of the my-cds-cds secret and there is a yaml file stored in cds_config_file, so I'm not sure why it's not finding it.

[sean-abbott]

Ok, I got to a point where I can cerify that I am creating config.toml and it's got the contents from the config.toml here in the helm chart.

But cds still thinks it's empty.

args: ["-c", "echo \"$CDS_CONFIG_FILE\" > config.toml && cat config.toml && /app/cds-engine-linux-amd64 start api --config config.toml || sleep 600"]

yields

 kubectl logs my-cds-api-6b9674d4b4-ld8dq
######################
# CDS Hooks Settings
#######################
[hooks]
  URL = "http://localhost:8083"
...
    #       # showDetail = false

Reading configuration file config.toml
Starting service api
Unable to init service api: your CDS configuration seems to be empty. Please use environment variables, file or Consul to set your configuration

[yesnault]

@sean-abbott Thank you for debug about base64. It's fixed with this PR #4164

About that: "Unable to init service api: your CDS configuration seems to be empty" -> seems that variable api in section api.url is empty OR environment variable CDS_API_URL_API is empty on your pod. I'll check that now, as a workaround, you can delete the variable CDS_CONFIG_FILE in the file api-deployment.yaml

[yesnault]

@sean-abbott so, this PR #4164 contains fixes about your issue

FYI, the master branch contains latest revisions of CDS. For a more stable helm version, you can checkout the version 0.38.1 on https://github.com/ovh/cds/tree/0.38.1

[bnjjj]

PR merged @sean-abbott let us know if it's better for you ! Sorry for this regression

[sean-abbott]

Thank you very much for your help! I'm hoping to test this again soon.

[yesnault]

feel free to reopen an issue if needed