VPCPeeringConnection not created with Observed a panic

[JaniszM]

What happened?

I'm trying to create just a VPCpeering, both VPCs in the same region and account.

VPCpeering is partially created, and waiting for acceptance.

My setup is:

1. VPCPeeringConnection CRD is deployed on the so-called control plane EKS cluster cp (AWS account A).
2. From there cp cluster deploys CR on another EKS cluster using providerConfigRef (AWS account B).

How can we reproduce it?

Sample object:

apiVersion: ec2.aws.crossplane.io/v1alpha1
kind: VPCPeeringConnection
metadata:
  name: janisz-test-vpc-peering
spec:
  deletionPolicy: Delete
  forProvider:
    region: eu-central-1
    acceptRequest: true
    vpcIDRef:
      name: cp-dev-vpc
    peerVPCIDRef:
      name: vpn-dev-vpc
  providerConfigRef:
    name: aws-provider-dev

Executed on cp cluster.

Produces the following error on the cp cluster level:

2024-04-12T14:13:52.242Z	DEBUG	provider-aws	Reconciling	{"controller": "managed/vpcpeeringconnection.ec2.aws.crossplane.io", "request": {"name":"janisz-test-vpc-peering"}}
E0412 14:13:52.402745       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 1576 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x6047040?, 0xab0fd00})
	k8s.io/apimachinery@v0.28.3/pkg/util/runtime/runtime.go:75 +0x85
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:108 +0xb2
panic({0x6047040?, 0xab0fd00?})
	runtime/panic.go:914 +0x21f
github.com/crossplane-contrib/provider-aws/pkg/controller/ec2/vpcpeeringconnection.(*custom).postObserve(0xc008e8ff00, {0x786d3d8?, 0xc00a5e5ab0?}, 0xc0024c8d80, 0xc007e83d60, {0x1, 0x1, 0x0, 0x0, {0x0, ...}}, ...)
	github.com/crossplane-contrib/provider-aws/pkg/controller/ec2/vpcpeeringconnection/setup.go:99 +0x71
github.com/crossplane-contrib/provider-aws/pkg/controller/ec2/vpcpeeringconnection.(*external).Observe(0xc0003f3180, {0x786d3d8, 0xc00a5e5ab0}, {0x78ae6c0?, 0xc0024c8d80})
	github.com/crossplane-contrib/provider-aws/pkg/controller/ec2/vpcpeeringconnection/zz_controller.go:104 +0x469
github.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile(0xc000837b30, {0x786d330, 0xc006c5d740}, {{{0x0, 0x0}, {0xc00b0b8960, 0x17}}})
	github.com/crossplane/crossplane-runtime@v1.14.1/pkg/reconciler/managed/reconciler.go:903 +0x2b08
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x7873c38?, {0x786d330?, 0xc006c5d740?}, {{{0x0?, 0xb?}, {0xc00b0b8960?, 0x0?}}})
	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00087a000, {0x786d368, 0xc000021ae0}, {0x64d0240?, 0xc004e5eb40?})
	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316 +0x3c5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00087a000, {0x786d368, 0xc000021ae0})
	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266 +0x1c9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 481
	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:223 +0x565
2024-04-12T14:13:52.403Z	ERROR	Reconciler error	{"controller": "managed/vpcpeeringconnection.ec2.aws.crossplane.io", "controllerGroup": "ec2.aws.crossplane.io", "controllerKind": "VPCPeeringConnection", "VPCPeeringConnection": {"name":"janisz-test-vpc-peering"}, "namespace": "", "name": "janisz-test-vpc-peering", "reconcileID": "613066c0-bb2d-4985-b4fd-e3e2e2feae4c", "error": "panic: runtime error: invalid memory address or nil pointer dereference [recovered]"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227

There are no logs from provider-aws working on the target cluster (AWS Account B).

This is weird because I see the resource is partially created on that account. So were are logs?

What environment did it happen in?

Crossplane version: https://charts.crossplane.io/stable 1.13.2
xpkg.upbound.io/crossplane-contrib/provider-aws:v0.47.1
kubectl version:
Client Version: v1.29.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.6-eks-508b6b3