Crossplane pods are failed with TLS handshake error

[balajikandan]

Crossplane pods are running and throws below error. The issue does not happen if I point and pull image from docker hub

Failed to pull image "xpkg.upbound.io/crossplane/crossplane:v1.14.0": rpc error: code = Unknown desc = failed to pull and unpack image "xpkg.upbound.io/crossplane/crossplane:v1.14.0": failed to resolve reference "xpkg.upbound.io/crossplane/crossplane:v1.14.0": failed to do request: Head "https://xpkg.upbound.io/v2/crossplane/crossplane/manifests/v1.14.0": net/http: TLS handshake timeout

[jbw976]

Is this consistently happening for you @balajikandan, or did you notice it to be temporary/transient?

[balajikandan]

@jastang That is correct, for crossplane we have opened 443 for upbound.io domains

[thephred]

The domains we can see being used for a pull are xpkg.upbound.io and d3qrbvrml4iuq4.cloudfront.net. Could you verify that with those allowed it works?

[balajikandan]

Currently xpkg.upbound.io is open, still seeing the issue mentioned. Do I need to exclude this domain too d3qrbvrml4iuq4.cloudfront.net ?

[thephred]

Yes, that is where the actual oci layers are pulled from.

[balajikandan]

We have whitelisted both the domains. Still we are facing the same issue. TLS handshake is failing with no CA or client certs available.

From EKS cluster, we ran this command to check HTTP connectivity. It looks like connectivity exists but the SSL certs are not available.

~ openssl s_client -connect xpkg.upbound.io:443 -tls1_2

CONNECTED(00000003)
write:errno=0

no peer certificate available

No client certificate CA names sent

SSL handshake has read 0 bytes and written 218 bytes
Verification: OK

New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1706046183
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no

Am I missing something here ?

[balajikandan]

It is a Operating system issue, Required CA certs were missing. We are able to download the images from the xpkg.upbound.io

[jbw976]

Thank you very much for following up on this @balajikandan to help explain the root cause. If there's any other description you can provide about how you installed the required CA certs, feel free to share them here as they may prove useful to others in the future 🙏