Cannot create load balancer in AWS

[texasmichelle]

I'm seeing the following error when installing Kubeflow on a cluster generated with Kubicorn:

  Normal   EnsuringLoadBalancer        2m (x19 over 1h)   service-controller  Ensuring load balancer
  Warning  CreatingLoadBalancerFailed  2m (x10 over 47m)  service-controller  (combined from similar events): Error creating load balancer (will retry): failed to ensure load balancer for service kubeflow-chasm/tf-hub-lb: AccessDenied: User: arn:aws:sts::015345740490:assumed-role/kubeflow-demo-KubicornMasterRole/i-0c953d8dc191ce89e is not authorized to perform: iam:CreateServiceLinkedRole on resource: arn:aws:iam::015345740490:role/aws-service-role/elasticloadbalancing.amazonaws.com/AWSServiceRoleForElasticLoadBalancing
           status code: 403, request id: b6003393-8fb1-11e8-aebb-c1258eb69b8d

The user I created to run kubicorn has IAMFullAccess, as well as AutoScalingFullAccess, AmazonVPCFullAccess, & AmazonEC2FullAccess.

These are the steps I take to generate the cluster:

kubicorn create kubeflow-demo --profile aws
kubicorn apply kubeflow-demo

These are the commands for installing kubeflow with ksonnet:

ks init kubeflow
cd kubeflow
VERSION=v0.1.3
ks registry add kubeflow github.com/kubeflow/kubeflow/tree/${VERSION}/kubeflow
ks pkg install kubeflow/core@${VERSION}
ks pkg install kubeflow/tf-serving@${VERSION}
ks pkg install kubeflow/tf-job@${VERSION}
ks generate core kubeflow-core --name=kubeflow-core
ks env add aws
ks param set --env aws kubeflow-core cloud "aws"
ks param set --env aws kubeflow-core jupyterHubServiceType "LoadBalancer"
ks apply aws -c kubeflow-core

When the pods come up, I see the above error in kubectl describe svc tf-hub-lb.
The external IP in the service remains in pending state indefinitely.

Before installing kubeflow on a GKE cluster, I also run this command to assign cluster-wide RBAC privs:

kubectl create clusterrolebinding cluster-admin-binding-kubeflow-admin \
  --clusterrole cluster-admin \
  --user $(gcloud config get-value account)

It's possible that something similar is required for AWS.

[stevenoctopus]

I'm not using kubicorn but I just ran into this today walking through the Getting Started with Amazon EKS guide. I followed the same exact steps I did last week, where I did not receive this error.

I was able to get around it by attaching a new policy to the role I assigned to my EKS cluster but I wonder what changed?

[texasmichelle]

@stevenoctopus Which policy fixed the issue?

[stevenoctopus]

I created a new policy, which allowed the service role I created for eks to perform iam:CreateServiceLinkedRole on the AWSServiceRoleForElasticLoadBalancing role in my account, and then attached that to the eks service role.

However, I don't think that is correct to do. I found out AWS automatically created the AWSServiceRoleForElasticLoadBalancing role if you created a load balancer before January of this year. (I'm on my phone or I would link the docs where I found this)

Nothing in my account was using the AWSServiceRoleForElasticLoadBalancing role so I wanted to try deleting it and starting the process over. I will try on Monday and report back my results.

[stevenoctopus]

No luck deleting the role and trying again. I get the same error even when the AWSServiceRoleForElasticLoadBalancing role doesn't exist in my account. Very strange.

There is a post about this in the AWS forums that I am also watching.

[stevenoctopus]

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "iam:CreateServiceLinkedRole",
            "Resource": "arn:aws:iam::015345740490:role/aws-service-role/elasticloadbalancing.amazonaws.com/AWSServiceRoleForElasticLoadBalancing"
        }
    ]
}

Here is a policy you can attach to the EKS service role as a workaround for the time being. I subbed in the correct account ID for you already.

[texasmichelle]

Thanks for posting the additional context. Is iam:CreateServiceLinkedRole not covered by IAMFullAccess?

[stevenoctopus]

It is, but it would be unwise to give EKS IAMFullAccess as it could possibly delete/modify existing IAM services.

…

On Mon, Jul 30, 2018, 6:09 PM Michelle Casbon ***@***.***> wrote: Thanks for posting the additional context. Is iam:CreateServiceLinkedRole not covered by IAMFullAccess? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#656 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD_VWrErdpnbYq3YdiSFYlnJ3rvms9RPks5uL4QggaJpZM4VfVLn> .

[texasmichelle]

Makes sense - this particular example is for a demo, so I'm less concerned for now & can reduce the scope once it's all working. The service account I created for kubicorn already has IAMFullAccess, so I'm not sure I understand how to apply this workaround.