spanner.it.ITDatabaseAdminTest: listPagination failed #655
Assignees
Labels
api: spanner
Issues related to the googleapis/java-spanner API.
priority: p1
Important issue which blocks shipping the next release. Will be fixed prior to next release.
🚨
This issue needs some love.
type: bug
Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Comments
flaky-bot
bot
added
buildcop: issue
priority: p1
product-auto-label
bot
added
the
api: spanner
|
commit: b791c86 Test outputcom.google.cloud.spanner.SpannerException: RESOURCE_EXHAUSTED: Failed to list the databases of projects/gcloud-devel/instances/spanner-testing with pageToken AN4G3x_Tf-HkS3TJRB5EUShnqGjjIcnvd2tCeQ0akyd4fV9ohoAWBWI-jI3gXC0o: RESOURCE_EXHAUSTED: com.google.api.gax.rpc.ResourceExhaustedException: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Quota exceeded for quota metric 'Administrative requests' and limit 'Administrative requests per minute' of service 'spanner.googleapis.com' for consumer 'project_number:1016721519174'. at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:236) at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:56) at com.google.cloud.spanner.DatabaseAdminClientImpl$12.getNextPage(DatabaseAdminClientImpl.java:358) at com.google.cloud.spanner.SpannerImpl$PageFetcher.getNextPage(SpannerImpl.java:292) at com.google.cloud.PageImpl.getNextPage(PageImpl.java:116) at com.google.cloud.spanner.it.ITDatabaseAdminTest.listPagination(ITDatabaseAdminTest.java:207) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.ParentRunner.run(ParentRunner.java:413) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.ParentRunner.run(ParentRunner.java:413) at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeLazy(JUnitCoreWrapper.java:119) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:87) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75) at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138) at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451) Caused by: com.google.cloud.spanner.SpannerException: RESOURCE_EXHAUSTED: com.google.api.gax.rpc.ResourceExhaustedException: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Quota exceeded for quota metric 'Administrative requests' and limit 'Administrative requests per minute' of service 'spanner.googleapis.com' for consumer 'project_number:1016721519174'. at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:236) at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:56) at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:176) at com.google.cloud.spanner.spi.v1.GapicSpannerRpc.get(GapicSpannerRpc.java:1407) at com.google.cloud.spanner.spi.v1.GapicSpannerRpc.listDatabases(GapicSpannerRpc.java:829) at com.google.cloud.spanner.DatabaseAdminClientImpl$12.getNextPage(DatabaseAdminClientImpl.java:356) ... 46 more Caused by: java.util.concurrent.ExecutionException: com.google.api.gax.rpc.ResourceExhaustedException: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Quota exceeded for quota metric 'Administrative requests' and limit 'Administrative requests per minute' of service 'spanner.googleapis.com' for consumer 'project_number:1016721519174'. at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at com.google.cloud.spanner.spi.v1.GapicSpannerRpc.get(GapicSpannerRpc.java:1401) ... 48 more Caused by: com.google.api.gax.rpc.ResourceExhaustedException: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Quota exceeded for quota metric 'Administrative requests' and limit 'Administrative requests per minute' of service 'spanner.googleapis.com' for consumer 'project_number:1016721519174'. at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:57) at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72) at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60) at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97) at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68) at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1041) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1215) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:563) at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:533) at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor$1$1.onClose(SpannerErrorInterceptor.java:100) at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:617) at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:803) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:782) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Quota exceeded for quota metric 'Administrative requests' and limit 'Administrative requests per minute' of service 'spanner.googleapis.com' for consumer 'project_number:1016721519174'. at io.grpc.Status.asRuntimeException(Status.java:533) ... 18 more |
|
Looks like this issue is flaky. 😟 I'm going to leave this open and stop commenting. A human should fix and close this. When run at the same commit (b791c86), this test passed in one build (Build Status, Sponge) and failed in another build (Build Status, Sponge). |
olavloite
added a commit
that referenced
this issue
Nov 24, 2020
Automatically retry requests that fail because the admin requests per seconds limit has been exceeded using an exponential backoff. Fixes #655 and others
thiagotnunes
pushed a commit
that referenced
this issue
Nov 30, 2020
* feat: retry admin request limit exceeded error Automatically retry requests that fail because the admin requests per seconds limit has been exceeded using an exponential backoff. Fixes #655 and others * fix: remove unused variable * fix: extract strings to constants
thiagotnunes
pushed a commit
that referenced
this issue
May 6, 2021
* feat: retry admin request limit exceeded error Automatically retry requests that fail because the admin requests per seconds limit has been exceeded using an exponential backoff. Fixes #655 and others * fix: remove unused variable * fix: extract strings to constants
ansh0l
pushed a commit
to ansh0l/java-spanner
that referenced
this issue
Nov 10, 2022
This is an auto-generated regeneration of the .pb.go files by cloud.google.com/go/internal/gapicgen. Once this PR is submitted, genbot will update the corresponding PR to depend on the newer version of go-genproto, and assign reviewers. Whilst this or any regen PR is open in go-genproto, genbot will not create any more regeneration PRs. If all regen PRs are closed, gapicgen will create a new set of regeneration PRs once per night. If you have been assigned to review this PR, please: - Ensure that CI is passing. If it's failing, it requires your manual attention. - Approve and submit this PR if you believe it's ready to ship. That will prompt genbot to assign reviewers to the google-cloud-go PR. Corresponding google-cloud-go PR: googleapis/google-cloud-go#4615 Changes: feat:Publish Workflows call logging proto for documentation PiperOrigin-RevId: 390391198 Source-Link: googleapis/googleapis@c47ff19 fix: Update DocumentUnderstandingService default_host in v1beta2 from a regional endpoint, to non-regional Committer: @nicain PiperOrigin-RevId: 390219495 Source-Link: googleapis/googleapis@09a6183 chore: regenerate API index Source-Link: googleapis/googleapis@08ae683 feat(googleads): protos for v8.1 of Google Ads API Committer: @aohren PiperOrigin-RevId: 390218328 Source-Link: googleapis/googleapis@6ef6c29 chore: regenerate API index Source-Link: googleapis/googleapis@eb7c9d1 chore(secretmanager): no-op proto generator run to pickup Locations service metadata PiperOrigin-RevId: 390124312 Source-Link: googleapis/googleapis@f5e39a2
rajatbhatta
pushed a commit
to rajatbhatta/java-spanner
that referenced
this issue
Nov 17, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Note: #570 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.
commit: 0f9b7bd
buildURL: Build Status, Sponge
status: failed
Test output
The text was updated successfully, but these errors were encountered: