Insufficient compute.networks.updatePeering rights for creating GKE Autopilot using FAST TF module running on Cloudbuild with FAST created SA

[fbadso]

Hey all,

I want to achieve private GKE Autopilot clusters per projects with masters available via VPN:

In 1-resman I created a team folder(tf.tvfars) with corresponding Cloudbuild Trigger and Source Repo:

team_folders = {

  ejp = {
    descriptive_name = "EJP"
    group_iam = {
      "ejp-developers@somedomain.com" = [
        "roles/viewer"
      ]
    }
    impersonation_groups = ["gcp-organization-admins@somedomain.com"]

    cicd = {
      branch            = "master"
      identity_provider = null
      type              = "sourcerepo"
      name              = "ejp-infra"
    }
  }

Using 2-networking I created a subnet

somename.yaml

region: europe-west3
description: Subnet for project somesuffix-dev-ejpcc-0
ip_cidr_range: 10.120.3.0/24
secondary_ip_ranges:
   pods: 100.70.0.0/16
   services: 100.71.3.0/24

Using 3-project-factory I created a project for that team:

somename.yaml:

labels:
 team: ejp
parent: folders/somefolder
services:
- compute.googleapis.com
- container.googleapis.com
- secretmanager.googleapis.com
- sqladmin.googleapis.com

shared_vpc_service_config:
  host_project: someprefix-dev-net-spoke-0
  service_identity_iam:
    "roles/compute.networkUser":
      - cloudservices
      - container-engine
    "roles/container.hostServiceAgentUser":
      - container-engine

So far, so good - I have a project, shared VPC with a subnet ready for GKE and a CI/CD pipeline created via teamfolder and an SA that can create clusters within that project.

Within the Cloudbuild / Cloud Source CI/CD pipeline, I create a GKE Autopilot cluster:

module "gke-autopilot" {
  source              = "git::https://source.developers.google.com/p/someprefix-prod-iac-core-0/r/fast-modules//modules//gke-cluster-autopilot?ref=v1.0"
  name                = "gke-ejpcc-dev"
  project_id          = var.project_id
  location            = var.default_region
  deletion_protection = false
  labels              = { environment = "dev" }

  vpc_config = {
    host_project_id = var.host_project_ids.dev-spoke-0
    network         = var.vpc_self_links.dev-spoke-0
    subnetwork      = var.subnet_self_links.dev-spoke-0["europe-west3/dev-ejp-ew3"]

    master_authorized_ranges = {
      someprefix_onprem = var.someprefix_onprem
    }

    master_ipv4_cidr_block = "10.120.4.0/28"

    secondary_range_names = {
      pods     = var.secondary_range_names.pods
      services = var.secondary_range_names.services
    }
  }
  private_cluster_config = {
    enable_private_endpoint = true

    master_global_access = false
    peering_config = {
      export_routes = true
      import_routes = true
      project_id    = var.host_project_ids.dev-spoke-0
    }

  }
}

I get the following error during creation:

{
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "status": {
      "code": 7,
      "message": "Required 'compute.networks.updatePeering' permission for 'projects/someprefix-dev-net-spoke-0/global/networks/dev-spoke-0'"
    },
    "authenticationInfo": {
      "principalEmail": "wlc-prod-teams-ejp-0@someprefix-prod-iac-core-0.iam.gserviceaccount.com",
      "serviceAccountDelegationInfo": [
        {
          "firstPartyPrincipal": {
            "principalEmail": "someprefix-prod-teams-ejp-1@wlc-prod-iac-core-0.iam.gserviceaccount.com"
          }
        }
      ],
      "principalSubject": "serviceAccount:someprefix-prod-teams-ejp-0@wlc-prod-iac-core-0.iam.gserviceaccount.com"
    },
    "requestMetadata": {
      "callerIp": "34.82.26.198",
      "callerSuppliedUserAgent": "Terraform/1.7.2 (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 terraform-provider-google/5.17.0,gzip(gfe)",
      "requestAttributes": {
        "time": "2024-02-27T13:03:24.960551Z",
        "auth": {}
      },
      "destinationAttributes": {}
    },
    "serviceName": "compute.googleapis.com",
    "methodName": "v1.compute.networks.updatePeering",
    "authorizationInfo": [
      {
        "resource": "projects/someprefix-dev-net-spoke-0/global/networks/dev-spoke-0",
        "permission": "compute.networks.updatePeering",
        "resourceAttributes": {
          "service": "compute",
          "name": "projects/someprefix-dev-net-spoke-0/global/networks/dev-spoke-0",
          "type": "compute.networks"
        }
      }
    ],
    "resourceName": "projects/someprefix-dev-net-spoke-0/global/networks/dev-spoke-0",
    "request": {
      "@type": "type.googleapis.com/compute.networks.updatePeering",
      "networkPeering": {
        "importCustomRoutes": true,
        "name": "gke-n229c879f402c4a7831d-a51e-d9f7-peer",
        "exportCustomRoutes": true
      }
    },
    "response": {
      "@type": "type.googleapis.com/error",
      "error": {
        "message": "Required 'compute.networks.updatePeering' permission for 'projects/someprefix-dev-net-spoke-0/global/networks/dev-spoke-0'",
        "code": 403,
        "errors": [
          {
            "reason": "forbidden",
            "domain": "global",
            "message": "Required 'compute.networks.updatePeering' permission for 'projects/someprefix-dev-net-spoke-0/global/networks/dev-spoke-0'"
          }
        ]
      }
    },
    "resourceLocation": {
      "currentLocations": [
        "global"
      ]
    }
  },
  "insertId": "-4f7kxhd3sp8",
  "resource": {
    "type": "gce_network",
    "labels": {
      "network_id": "3122607721743879350",
      "project_id": "prefix-dev-net-spoke-0"
    }
  },
  "timestamp": "2024-02-27T13:03:24.744539Z",
  "severity": "ERROR",
  "labels": {
    "compute.googleapis.com/root_trigger_id": "5bf15a3d-2f63-4226-b1eb-a66a1926be5b"
  },
  "logName": "projects/someprefix-dev-net-spoke-0/logs/cloudaudit.googleapis.com%2Factivity",
  "receiveTimestamp": "2024-02-27T13:03:25.500466053Z"
}

Obviously, the service account which runs the CloudBuild trigger, doesn't have the permission compute.networks.updatePeering for the VPC in the host project.

As a workaround, I assigned the FAST fabric custom role "serviceProjectNetworkAdmin" (which has the compute.networks.updatePeering permission) manually to someprefix-prod-teams-ejp-0@wlc-prod-iac-core-0.iam.gserviceaccount.com (ServiceAccount, which is used in the CloudBuild trigger) and everything is being created fine and I can access the cluster from the OnPrem location via VPN.

Without the peering configuration to import/export routes everything works fine, but enabling

    peering_config = {
      export_routes = true
      import_routes = true
      project_id    = var.host_project_ids.dev-spoke-0

throws the error.

Is this a bug / missing feature?
What's the best approach to grant the Cloudbuild SA the compute.networks.updatePeering permissions?

Best,
Frederik

[juliocc]

Is this a bug / missing feature?

This is working as intended. The project factory, gke and data platform service accounts are all assigned serviceProjectNetworkAdmin exactly for this reason.

What's the best approach to grant the Cloudbuild SA the compute.networks.updatePeering permissions?

What you're doing makes sense (i.e. grant serviceProjectNetworkAdmin to the team's SA). I'd do that directly in resman when the team SA is created. How are you doing it?

[fbadso]

Hmmm - first of all thanks for your quick reply. But your answer confuses me a little bit: Don't we have a chicken / egg problem when integrate the access right for the 1-resman created SA on a project / VPC resource that is managed further in stage 2-networking?
To continue working I had granted the right temporary manually but I'll check the right place to put this into TF code..
Isn't 2-networking the right stage to apply the iam binding on the network project to the team CI/CD SA which is being created in 1-resman?

I granted the right as follows:

❯ gcloud projects get-iam-policy someprefix-dev-net-spoke-0
bindings:

• members:
  • serviceAccount:someprefix-prod-teams-ejp-0@someprefix-prod-iac-core-0.iam.gserviceaccount.com
    role: organizations/someorgnumber/roles/serviceProjectNetworkAdmin

[juliocc]

Don't we have a chicken / egg problem when integrate the access right for the 1-resman created SA on a project / VPC resource that is managed further in stage 2-networking?

Yes and no. As you mention, resman doesn't create any projects/vpcs but you can still grant permissions on the folders. The easiest approach is to grant the teams service accounts serviceProjectNetworkAdmin on one of those folders (net/dev, net/prod, or the level one containing the other two)

[fbadso]

Thanks Julio, that absolutely makes sense. :-)