Bigtable client connection issue from GKE

[Naidile-P-N]

What version of gRPC-Java are you using?
1.55.1

What is your environment?
OS -On GKE - Linux[gke-os-distribution - cos]
JDK 15 with maven

What did you expect to see?
Connect to Bigtable instance in the same GCP project from the application.
I'm able to connect to Bigtable using my account running in local environment.

What did you see instead?
Connection to bigtable fails with exception at this line

val settings: BigtableDataSettings =
                BigtableDataSettings.newBuilder().setProjectId(projectId).setInstanceId(instanceId).build()
val client = BigtableDataClient.create(settings)

Dependency used

<dependency> <groupId>com.google.cloud</groupId> <artifactId>google-cloud-bigtable</artifactId> <version>2.24.1</version> </dependency>

Steps to reproduce the bug

1. Deploy the service in GKE
2. Attempt to connect to Bigtable

Caused by: java.lang.IllegalStateException: Default config is invalid: Status{code=UNKNOWN, description=None of [grpclb] specified by Service Config are available., cause=null} at com.google.common.base.Preconditions.checkState(Preconditions.java:590) at io.grpc.internal.ManagedChannelImpl.<init>(ManagedChannelImpl.java:642) at io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:631) at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:297) at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:391) at com.google.api.gax.grpc.ChannelPool.<init>(ChannelPool.java:107) at com.google.api.gax.grpc.ChannelPool.create(ChannelPool.java:85) at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:237) at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:231) at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:201) at com.google.cloud.bigtable.data.v2.stub.EnhancedBigtableStub.create(EnhancedBigtableStub.java:175) at com.google.cloud.bigtable.data.v2.BigtableDataClient.create(BigtableDataClient.java:165)

Note : Error still persists after modifications to service account and workload identity setup

[Naidile-P-N]

grpc/grpc-java#10377

[Naidile-P-N]

• tried adding the below service configurations and the issue still persists

META-INF/services/
META-INF/services/io.grpc.LoadBalancerProvider
META-INF/services/io.grpc.NameResolverProvider

[Naidile-P-N]

I finally resolved the error by modifying the files as below and enabling attemptDirectPathXds during Bigtable client creation.

Is this approach right to use in production ? Or will there be any fixes or updates made in the google-cloud-bigtable package ?

---

io.grpc.LoadBalancerProvider

io.grpc.internal.PickFirstLoadBalancerProvider
io.grpc.util.SecretRoundRobinLoadBalancerProvider$Provider
io.grpc.util.OutlierDetectionLoadBalancerProvider
io.grpc.grpclb.GrpclbLoadBalancerProvider
io.grpc.xds.ClusterManagerLoadBalancerProvider
io.grpc.xds.ClusterResolverLoadBalancerProvider
io.grpc.xds.CdsLoadBalancerProvider
io.grpc.xds.PriorityLoadBalancerProvider
io.grpc.xds.ClusterImplLoadBalancerProvider
io.grpc.xds.WrrLocalityLoadBalancerProvider
io.grpc.xds.WeightedTargetLoadBalancerProvider
io.grpc.xds.WeightedRoundRobinLoadBalancerProvider
io.grpc.xds.RingHashLoadBalancerProvider
io.grpc.xds.LeastRequestLoadBalancerProvider
io.grpc.rls.RlsLoadBalancerProvider
io.grpc.protobuf.services.internal.HealthCheckingRoundRobinLoadBalancerProvider

io.grpc.NameResolverProvider

io.grpc.googleapis.GoogleCloudToProdExperimentalNameResolverProvider
io.grpc.googleapis.GoogleCloudToProdNameResolverProvider
io.grpc.xds.XdsNameResolverProvider
io.grpc.internal.DnsNameResolverProvider
io.grpc.grpclb.SecretGrpclbNameResolverProvider$Provider
io.grpc.netty.shaded.io.grpc.netty.UdsNameResolverProvider