Unable to create loadbalancer in kubernetes :: Tags kubernetes.io/cluster/<id> missing on public subnet

[timharsch]

I had an issue trying to create a Load Balancer in a cluster created with the config file below. I was using a VPC create with the Cloud Formation template for VPC in widdix (https://github.com/widdix/aws-cf-templates/blob/v11.4.0/vpc/vpc-2azs.yaml). It creates two public and two private subnets. I then use the two public subnets to create a config file below (obscured). I was unable to get a working load balancer (however using the same approach with an EC2 node group seems to work fine).

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: patient-portal-fargate
  region: us-east-1
vpc:
  id: "vpc-MYVPC"
  cidr: "10.191.0.0/16"
  subnets:
    public:
      us-east-1a:
          id: "subnet-pub1"
          cidr: "10.191.16.0/20"
      us-east-1b:
          id: "subnet-pub2"
          cidr: "10.191.48.0/20"
fargateProfiles:
  - name: fargate-default
    selectors:
      - namespace: default
      - namespace: kube-system

And this was how my service was being created in kubernetes (which is for a DJANGO rest api deployment)

apiVersion: v1
kind: Service
metadata:
  name: my-api-service
  labels:
    io.kompose.service: api-loadbalancer
spec:
  type: LoadBalancer
  selector:
    io.kompose.service: api
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8000

The first thing I discovered is that the load balancer could not create (sorry message lost now). I solved that with the help of this comment.
kubernetes/kubernetes#29298 (comment) by tagging both my public subnet groups with kubernetes.io/cluster/ = owned. That allowed the load balancer to create. I also went to the Private subnets and saw that eksctl had added a kubernetes.io/cluster/ = shared on the private subnets. (perhaps from a previous attempt to build my cluster in the private subnets). So, somehow it seems as if eksctl has forgotten to apply the tag to the public subnets identified in my config yaml file.

Then my error became if I attempt to use the my-api-service I got a

curl: (52) Empty reply from server

Anything else we need to know?
What OS are you using, are you using a downloaded binary or did you compile eksctl, what type of AWS credentials are you using (i.e. default/named profile, MFA) - please don't include actual credentials though!

Versions
Please paste in the output of these commands:

$ eksctl version
$ kubectl version

$ uname -a
Linux ip-10-0-13-180.ec2.internal 4.14.173-137.228.amzn2.x86_64 #1 SMP Thu Mar 19 16:50:21 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ eksctl version
0.15.0
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.7-eks-1861c5", GitCommit:"1861c597586f84f1498a9f2151c78d8a6bf47814", GitTreeState:"clean", BuildDate:"2019-09-24T22:12:08Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.9-eks-502bfb", GitCommit:"502bfb383169b124d87848f89e17a04b9fc1f6f0", GitTreeState:"clean", BuildDate:"2020-02-07T01:31:02Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

Logs
Include the output of the command line when running eksctl. If possible, eksctl should be run with debug logs. For example:
eksctl get clusters -v 4
Make sure you redact any sensitive information before posting.
If the output is long, please consider a Gist.

[MartinEmrich]

Same (or related) issue here: the tag kubernetes.io/role/internal-elb is not set.

% eksctl version
0.15.0

[cPu1]

It appears that the LoadBalancer creation failed because you did not tag your subnets. These tags are used by the cloud-controller-manager to determine the subnets to use for creating load balancers. Since you have specified a custom VPC (as opposed to letting eksctl create a dedicated VPC), you'll have to ensure the VPC and subnets meet the requirements documented here: https://eksctl.io/usage/vpc-networking/#use-existing-vpc-any-custom-configuration

This is the intended behaviour and not a bug.

I also went to the Private subnets and saw that eksctl had added a kubernetes.io/cluster/ = shared on the private subnets.

If a custom VPC is specified, eksctl does not modify any of its properties, including the tags on the subnets, so that tag might have been added externally.

[cPu1]

Same (or related) issue here: the tag kubernetes.io/role/internal-elb is not set.

Did you supply a custom VPC? If you did, you'll have to add that tag to private subnets as eksctl does not tag/modify subnets if a custom VPC is specified. The full list of requirements for a custom VPC is documented here: https://eksctl.io/usage/vpc-networking/#use-existing-vpc-any-custom-configuration

[MartinEmrich]

@cPu1 Indeed I use a custom VPC with both a set of private and public subnets specified.

Apparently I remebered it wrong, I must have added the tag on older clusters.
kubernetes.io/cluster/ is correctly set to "shared" for me, so it looks like everything is in order for me.
Sorry to bother you.

[timharsch]

@cPu1 Thanks for the suggestions. As stated I was successful when I switched to EC2 node groups, which will have to be my method going forward until our next deployment effort in 2 weeks or so. When that happens I will read the guide and try to switch back to fargate and update this issue with my findings.

I'm thinking that perhaps my unwanted tags on the private subnets were causing issues.

[timharsch]

Postmortem note for future travelers:
I did not get a chance to retest this particular issue using fargate as promised. But, I did eventually get fargate with a custom VPC working (tested via deploying some busybox pods and exec shell on them). See my answer in issue #2746
You'll note after reading that issue that I would not have gotten much farther using public subnets as I was attempting to do in this issue. #2746 has a working eksctl config file for fargate using custom VPC.