kubernetes_cluster_node_pool: Fix race condition with virtual network status when creating node pool

[c4milo]

Related to #13105. We are able to consistently trigger it in our testing.

❯  make acctests SERVICE='containers' TESTARGS='-run=TestAccKubernetesClusterNodePool_virtualNetworkOwnershipRaceCondition' TESTTIMEOUT='60m'
==> Checking that code complies with gofmt requirements...
==> Checking that Custom Timeouts are used...
==> Checking that acceptance test packages are used...
TF_ACC=1 go test -v ./internal/services/containers -run=TestAccKubernetesClusterNodePool_virtualNetworkOwnershipRaceCondition -timeout 60m -ldflags="-X=github.com/hashicorp/terraform-provider-azurerm/version.ProviderVersion=acc"
=== RUN   TestAccKubernetesClusterNodePool_virtualNetworkOwnershipRaceCondition
=== PAUSE TestAccKubernetesClusterNodePool_virtualNetworkOwnershipRaceCondition
=== CONT  TestAccKubernetesClusterNodePool_virtualNetworkOwnershipRaceCondition
--- PASS: TestAccKubernetesClusterNodePool_virtualNetworkOwnershipRaceCondition (1395.32s)
PASS
ok  	github.com/hashicorp/terraform-provider-azurerm/internal/services/containers	1398.094s

Error

    Error: creating Agent Pool (Subscription: "312d56f6-697b-4377-9a9a-83257ce33066"
        Resource Group Name: "acctestRG-aks-240506121623880995"
        Managed Cluster Name: "acctestaks240506121623880995"
        Agent Pool Name: "internal3"): polling after CreateOrUpdate: polling failed: the Azure API returned the following error:

        Status: "SetVNetOwnershipFailed"
        Code: ""
        Message: "Set virtual network ownership failed. Subscription: 312d56f6-697b-4377-9a9a-83257ce33066; resource group: acctestRG-aks-240506121623880995; virtual network name: acctestvirtnet240506121623880995. autorest/azure: Service returned an error. Status=400 Code=\"VirtualNetworkNotInSucceededState\" Message=\"Virtual network /subscriptions/312d56f6-697b-4377-9a9a-83257ce33066/resourceGroups/acctestRG-aks-240506121623880995/providers/Microsoft.Network/virtualNetworks/acctestvirtnet240506121623880995 is in Updating state. It needs to be in Succeeded state in order to set resource ownership.\" Details=[]\nVirtual network /subscriptions/312d56f6-697b-4377-9a9a-83257ce33066/resourceGroups/acctestRG-aks-240506121623880995/providers/Microsoft.Network/virtualNetworks/acctestvirtnet240506121623880995 is in Updating state. It needs to be in Succeeded state in order to set resource ownership."
        Activity Id: ""

        ---

        API Response:

        ----[start]----
        {
          "name": "da6f9f40-6f18-4380-b252-d516cd3659f0",
          "status": "Failed",
          "startTime": "2024-05-06T16:21:01.6132767Z",
          "endTime": "2024-05-06T16:21:09.0898922Z",
          "error": {
           "code": "SetVNetOwnershipFailed",
           "message": "Set virtual network ownership failed. Subscription: 312d56f6-697b-4377-9a9a-83257ce33066; resource group: acctestRG-aks-240506121623880995; virtual network name: acctestvirtnet240506121623880995. autorest/azure: Service returned an error. Status=400 Code=\"VirtualNetworkNotInSucceededState\" Message=\"Virtual network /subscriptions/312d56f6-697b-4377-9a9a-83257ce33066/resourceGroups/acctestRG-aks-240506121623880995/providers/Microsoft.Network/virtualNetworks/acctestvirtnet240506121623880995 is in Updating state. It needs to be in Succeeded state in order to set resource ownership.\" Details=[]",
           "details": [
            {
             "code": "",
             "message": "Virtual network /subscriptions/312d56f6-697b-4377-9a9a-83257ce33066/resourceGroups/acctestRG-aks-240506121623880995/providers/Microsoft.Network/virtualNetworks/acctestvirtnet240506121623880995 is in Updating state. It needs to be in Succeeded state in order to set resource ownership."
            }
           ]
          }
         }
        -----[end]-----


          with azurerm_kubernetes_cluster_node_pool.test3,
          on terraform_plugin_test.tf line 97, in resource "azurerm_kubernetes_cluster_node_pool" "test3":
          97: resource "azurerm_kubernetes_cluster_node_pool" "test3" {

Community Note

• Please vote on this PR by adding a 👍 reaction to the original PR to help the community and maintainers prioritize for review
• Please do not leave "+1" or "me too" comments, they generate extra noise for PR followers and do not help prioritize for review

Description

PR Checklist

• I have followed the guidelines in our Contributing Documentation.
• I have checked to ensure there aren't other open Pull Requests for the same update/change.
• I have checked if my changes close any open issues. If so please include appropriate closing keywords below.
• I have used a meaningful PR title to help maintainers and other users understand this change and help prevent duplicate work.
  For example: “resource_name_here - description of change e.g. adding property new_property_name_here”

Changes to existing Resource / Data Source

• I have added an explanation of what my changes do and why I'd like you to include them (This may be covered by linking to an issue above, but may benefit from additional explanation).
• I have written new tests for my resource or datasource changes & updated any relevent documentation.
• I have successfully run tests with my changes locally. If not, please provide details on testing challenges that prevented you running the tests.

Testing

• My submission includes Test coverage as described in the Contribution Guide and the tests pass. (if this is not possible for any reason, please include details of why you did or could not add test coverage)

Change Log

Below please provide what should go into the changelog (if anything) conforming to the Changelog Format documented here.

• azurerm_kubernetes_cluster_node_pool - Fix race condition between virtual network status and node pool creation. The virtual network must be in Succeeded state for an AKS node pool to be correctly created.

This is a (please select all that apply):

• Bug Fix

Related Issue(s)

#13105

Note

If this PR changes meaningfully during the course of review please update the title and description as required.

[stephybun]

Thanks for this PR @c4milo.

I left some review comments and suggestions in-line. Could you take a look and fix those up? Once that's done we can run the rests and give this another review.

[c4milo]

@stephybun, I've made the changes you kindly requested. Thanks for reviewing!

[tombuildsstuff]

hey @c4milo

Thanks for pushing those changes, I've taken a look through and left a few comments inline, but this otherwise LGTM 👍

Thanks!

[c4milo]

Thanks for taking the time to review @tombuildsstuff. I've addressed the comments from the last review. Please let me know if there is anything else you all will want me to correct.

[stephybun]

We have a test failure:

------- Stdout: -------
=== RUN   TestAccKubernetesClusterNodePool_virtualNetworkOwnershipRaceCondition
=== PAUSE TestAccKubernetesClusterNodePool_virtualNetworkOwnershipRaceCondition
=== CONT  TestAccKubernetesClusterNodePool_virtualNetworkOwnershipRaceCondition
    testcase.go:113: Step 1/1 error: Error running pre-apply refresh: exit status 1
        Error: Invalid index
          on terraform_plugin_test.tf line 67, in resource "azurerm_kubernetes_cluster" "test":
          67:     vnet_subnet_id = azurerm_subnet.test["19"].id
            ├────────────────
            │ azurerm_subnet.test is tuple with 8 elements
        The given key does not identify an element in this collection value.
        Error: Invalid index
          on terraform_plugin_test.tf line 68, in resource "azurerm_kubernetes_cluster" "test":
          68:     pod_subnet_id  = azurerm_subnet.test["18"].id
            ├────────────────
            │ azurerm_subnet.test is tuple with 8 elements
        The given key does not identify an element in this collection value.
--- FAIL: TestAccKubernetesClusterNodePool_virtualNetworkOwnershipRaceCondition (12.73s)
FAIL

[c4milo]

@stephybun, thank you, I think it is due to the PR review request of lowering the number of subnets. I will fix the tests.

[c4milo]

@stephybun, I fixed the tests. Thank you!

❯  make acctests SERVICE='containers' TESTARGS='-run=TestAccKubernetesClusterNodePool_virtualNetworkOwnershipRaceCondition' TESTTIMEOUT='60m'
==> Checking that code complies with gofmt requirements...
==> Checking that Custom Timeouts are used...
==> Checking that acceptance test packages are used...
TF_ACC=1 go test -v ./internal/services/containers -run=TestAccKubernetesClusterNodePool_virtualNetworkOwnershipRaceCondition -timeout 60m -ldflags="-X=github.com/hashicorp/terraform-provider-azurerm/version.ProviderVersion=acc"
=== RUN   TestAccKubernetesClusterNodePool_virtualNetworkOwnershipRaceCondition
=== PAUSE TestAccKubernetesClusterNodePool_virtualNetworkOwnershipRaceCondition
=== CONT  TestAccKubernetesClusterNodePool_virtualNetworkOwnershipRaceCondition
--- PASS: TestAccKubernetesClusterNodePool_virtualNetworkOwnershipRaceCondition (1163.83s)
PASS
ok  	github.com/hashicorp/terraform-provider-azurerm/internal/services/containers	1166.633s

[stephybun]

Thanks @c4milo LGTM 🍍