Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Errors in storage acceptance test #11224

Closed
dazuma opened this issue Apr 26, 2021 · 25 comments · Fixed by #11243
Closed

Errors in storage acceptance test #11224

dazuma opened this issue Apr 26, 2021 · 25 comments · Fixed by #11243
Assignees
Labels
api: storage Issues related to the Cloud Storage API. priority: p2 Moderately-important priority. Fix may not be included in next release. 🚨 This issue needs some love. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@dazuma
Copy link
Member

dazuma commented Apr 26, 2021

On Ruby 2.5, we get this. This is from the nightly on 2021-04-26.

  1) Error:
Google::Cloud::Storage::Bucket::generate_signed_post_policy_v4::storage#test_0002_generates a signed post object v4 using signBlob API:
ArgumentError: unknown keyword: payload
    /h/.gem/gems/google-apis-iamcredentials_v1-0.3.0/lib/google/apis/iamcredentials_v1/service.rb:149:in `sign_service_account_blob'
    /workspace/google-cloud-storage/acceptance/storage/bucket_generate_signed_post_policy_v4_test.rb:79:in `block (3 levels) in <top (required)>'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v4.rb:355:in `generate_signature'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v4.rb:69:in `post_object'
    /workspace/google-cloud-storage/lib/google/cloud/storage/bucket.rb:1971:in `generate_signed_post_policy_v4'
    /workspace/google-cloud-storage/acceptance/storage/bucket_generate_signed_post_policy_v4_test.rb:83:in `block (2 levels) in <top (required)>'

  2) Error:
Google::Cloud::Storage::Bucket::post_object::v2::storage#test_0002_generates a signed post object using signBlob API:
ArgumentError: unknown keyword: payload
    /h/.gem/gems/google-apis-iamcredentials_v1-0.3.0/lib/google/apis/iamcredentials_v1/service.rb:149:in `sign_service_account_blob'
    /workspace/google-cloud-storage/acceptance/storage/bucket_post_object_v2_test.rb:79:in `block (3 levels) in <top (required)>'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v2.rb:135:in `generate_signature'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v2.rb:113:in `post_object'
    /workspace/google-cloud-storage/lib/google/cloud/storage/bucket.rb:1811:in `post_object'
    /workspace/google-cloud-storage/acceptance/storage/bucket_post_object_v2_test.rb:83:in `block (2 levels) in <top (required)>'

  3) Error:
Google::Cloud::Storage::signed_url::v2::storage::Google::Cloud::Storage::Bucket::signed_url#test_0002_should create a signed read url using IAM signBlob API:
ArgumentError: unknown keyword: payload
    /h/.gem/gems/google-apis-iamcredentials_v1-0.3.0/lib/google/apis/iamcredentials_v1/service.rb:149:in `sign_service_account_blob'
    /workspace/google-cloud-storage/acceptance/storage/signed_url_v2_test.rb:127:in `block (4 levels) in <top (required)>'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v2.rb:135:in `generate_signature'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v2.rb:128:in `signed_url'
    /workspace/google-cloud-storage/lib/google/cloud/storage/bucket.rb:1619:in `signed_url'
    /workspace/google-cloud-storage/acceptance/storage/signed_url_v2_test.rb:132:in `block (3 levels) in <top (required)>'

  4) Error:
Google::Cloud::Storage::signed_url::v4::storage::Google::Cloud::Storage::Bucket::signed_url#test_0002_should create a signed read url v4 using IAM signBlob API:
ArgumentError: unknown keyword: payload
    /h/.gem/gems/google-apis-iamcredentials_v1-0.3.0/lib/google/apis/iamcredentials_v1/service.rb:149:in `sign_service_account_blob'
    /workspace/google-cloud-storage/acceptance/storage/signed_url_v4_test.rb:127:in `block (4 levels) in <top (required)>'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v4.rb:214:in `block in service_account_signer'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v4.rb:124:in `signed_url'
    /workspace/google-cloud-storage/lib/google/cloud/storage/bucket.rb:1632:in `signed_url'
    /workspace/google-cloud-storage/acceptance/storage/signed_url_v4_test.rb:132:in `block (3 levels) in <top (required)>'

On Ruby 3.0, we get this. From the same nightly.

  1) Error:
Google::Cloud::Storage::signed_url::v2::storage::Google::Cloud::Storage::Bucket::signed_url#test_0002_should create a signed read url using IAM signBlob API:
Google::Apis::ClientError: Invalid request
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:228:in `check_status'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/api_command.rb:134:in `check_status'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:194:in `process_response'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:310:in `execute_once'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:113:in `block (2 levels) in execute'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:110:in `block in execute'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:102:in `execute'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/base_service.rb:377:in `execute_or_queue_command'
    /h/.gem/gems/google-apis-iamcredentials_v1-0.3.0/lib/google/apis/iamcredentials_v1/service.rb:158:in `sign_service_account_blob'
    /workspace/google-cloud-storage/acceptance/storage/signed_url_v2_test.rb:127:in `block (4 levels) in <top (required)>'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v2.rb:135:in `generate_signature'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v2.rb:128:in `signed_url'
    /workspace/google-cloud-storage/lib/google/cloud/storage/bucket.rb:1619:in `signed_url'
    /workspace/google-cloud-storage/acceptance/storage/signed_url_v2_test.rb:132:in `block (3 levels) in <top (required)>'

  2) Error:
Google::Cloud::Storage::Bucket::post_object::v2::storage#test_0002_generates a signed post object using signBlob API:
Google::Apis::ClientError: Invalid request
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:228:in `check_status'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/api_command.rb:134:in `check_status'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:194:in `process_response'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:310:in `execute_once'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:113:in `block (2 levels) in execute'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:110:in `block in execute'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:102:in `execute'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/base_service.rb:377:in `execute_or_queue_command'
    /h/.gem/gems/google-apis-iamcredentials_v1-0.3.0/lib/google/apis/iamcredentials_v1/service.rb:158:in `sign_service_account_blob'
    /workspace/google-cloud-storage/acceptance/storage/bucket_post_object_v2_test.rb:79:in `block (3 levels) in <top (required)>'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v2.rb:135:in `generate_signature'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v2.rb:113:in `post_object'
    /workspace/google-cloud-storage/lib/google/cloud/storage/bucket.rb:1811:in `post_object'
    /workspace/google-cloud-storage/acceptance/storage/bucket_post_object_v2_test.rb:83:in `block (2 levels) in <top (required)>'

  3) Error:
Google::Cloud::Storage::signed_url::v4::storage::Google::Cloud::Storage::Bucket::signed_url#test_0002_should create a signed read url v4 using IAM signBlob API:
Google::Apis::ClientError: Invalid request
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:228:in `check_status'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/api_command.rb:134:in `check_status'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:194:in `process_response'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:310:in `execute_once'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:113:in `block (2 levels) in execute'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:110:in `block in execute'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:102:in `execute'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/base_service.rb:377:in `execute_or_queue_command'
    /h/.gem/gems/google-apis-iamcredentials_v1-0.3.0/lib/google/apis/iamcredentials_v1/service.rb:158:in `sign_service_account_blob'
    /workspace/google-cloud-storage/acceptance/storage/signed_url_v4_test.rb:127:in `block (4 levels) in <top (required)>'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v4.rb:214:in `block in service_account_signer'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v4.rb:124:in `signed_url'
    /workspace/google-cloud-storage/lib/google/cloud/storage/bucket.rb:1632:in `signed_url'
    /workspace/google-cloud-storage/acceptance/storage/signed_url_v4_test.rb:132:in `block (3 levels) in <top (required)>'

  4) Error:
Google::Cloud::Storage::Bucket::generate_signed_post_policy_v4::storage#test_0002_generates a signed post object v4 using signBlob API:
Google::Apis::ClientError: Invalid request
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:228:in `check_status'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/api_command.rb:134:in `check_status'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:194:in `process_response'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:310:in `execute_once'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:113:in `block (2 levels) in execute'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:110:in `block in execute'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:102:in `execute'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/base_service.rb:377:in `execute_or_queue_command'
    /h/.gem/gems/google-apis-iamcredentials_v1-0.3.0/lib/google/apis/iamcredentials_v1/service.rb:158:in `sign_service_account_blob'
    /workspace/google-cloud-storage/acceptance/storage/bucket_generate_signed_post_policy_v4_test.rb:79:in `block (3 levels) in <top (required)>'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v4.rb:355:in `generate_signature'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v4.rb:69:in `post_object'
    /workspace/google-cloud-storage/lib/google/cloud/storage/bucket.rb:1971:in `generate_signed_post_policy_v4'
    /workspace/google-cloud-storage/acceptance/storage/bucket_generate_signed_post_policy_v4_test.rb:83:in `block (2 levels) in <top (required)>'
@dazuma dazuma added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. api: storage Issues related to the Cloud Storage API. priority: p2 Moderately-important priority. Fix may not be included in next release. labels Apr 26, 2021
@quartzmo quartzmo self-assigned this Apr 27, 2021
@quartzmo
Copy link
Member

I could not reproduce the Google::Apis::ClientError: Invalid request locally with Ruby 3.0.1.

I did reproduce the ArgumentError: unknown keyword: payload locally with Ruby 2.6.6 and 2.7.3.

The errors come from this usage pattern:

    signer = lambda do |string_to_sign|
      request = {
        "payload": string_to_sign,
      }
      resource = "projects/-/serviceAccounts/#{issuer}"
      response = iam_client.sign_service_account_blob resource, request
      response.signed_blob
    end

@dazuma Do you know of any reason why the hash form of the request would no longer be accepted? The payload attribute appears unchanged in SignBlobRequest. The signature of sign_service_account_blob also appears unchanged since googleapis/google-api-ruby-client#2139.

@quartzmo
Copy link
Member

The only change to google-apis-iamcredentials_v1 was the trivial doc update in googleapis/google-api-ruby-client#3603 that was released in v0.3.0.

@quartzmo
Copy link
Member

Before the v0.3.0 release, I thoroughly checked for keyword argument errors in acceptance tests when I did #10575.

quartzmo added a commit to quartzmo/google-cloud-ruby that referenced this issue Apr 27, 2021
@quartzmo
Copy link
Member

The unknown keyword: payload error can be avoided by using a string key in the request hash:

request = {
  "payload" => string_to_sign,
}

However, once this problem is solved on the Ruby 2.x runs, the Google::Apis::ClientError: Invalid request is then seen on those Ruby versions as well.

I am now able to reproduce the error on Ruby 3.0.1. The tests fail for me as shown above with Google::Apis::ClientError: Invalid request.

@quartzmo
Copy link
Member

@frankyn Any idea why we might be seeing errors in calls to IAMCredentialsService#sign_service_account_blob?

@frankyn
Copy link
Member

frankyn commented Apr 28, 2021

Hi @quartzmo,

This has not come up (yet) in other languages. Could you log request JSON bodies for payload : and payload =>? I'm wondering if there's a gotcha / issue with the json body between the two.

@frankyn frankyn self-assigned this Apr 28, 2021
@quartzmo
Copy link
Member

Thanks @frankyn, will do! I'll also log the unencoded string_to_sign.

@quartzmo
Copy link
Member

quartzmo commented Apr 29, 2021

@frankyn, here are the logged requests for the three different styles of request object, using Ruby 3.0.1 to avoid ArgumentError: unknown keyword: payload in Ruby 2.x.

Style 1

request = { "payload": string_to_sign }

request object in Google::Apis::Core::ApiCommand:

{:payload=>"GOOG4-RSA-SHA256\n20210429T153232Z\n20210429/auto/storage/goog4_request\n1e2a0ed22c06dac793d44cc09b10935d002b14116e606ea8c40f20129c343b22"}

HTTP request body (Update, from a later test run):

{"payload":"R09PRzQtUlNBLVNIQTI1NgoyMDIxMDQyOVQyMTU4MjlaCjIwMjEwNDI5L2F1dG8vc3RvcmFnZS9nb29nNF9yZXF1ZXN0CmVkMzA1ODJjZTU3Njg4NzUwMjdlNzE3NTg1NWJhNzNkMGU2ZmFjZDI3ZmY1OWI2ZWIzZjhjYWJmNGEyOTdjNjk="}

Result: Works in Ruby 3.0. Causes ArgumentError: unknown keyword: payload in Ruby 2.x.

#<Google::Apis::Core::ApiCommand:0x00007fbd7b2966f8 @options=#<struct Google::Apis::RequestOptions authorization=#<Signet::OAuth2::Client:0x00007fbd7c17e698 @authorization_uri=nil, @token_credential_uri=#<Addressable::URI:0xb7c URI:https://oauth2.googleapis.com/token>, @client_id=nil, @client_secret=nil, @code=nil, @expires_at=2021-04-29 10:32:29.584605 -0600, @issued_at=2021-04-29 09:32:30.584605 -0600, @issuer="542339357638-cr0dserr2evg7sv1meghqeu703274f3h@developer.gserviceaccount.com", @password=nil, @principal=nil, @redirect_uri=nil, @scope=["https://www.googleapis.com/auth/devstorage.full_control", "https://www.googleapis.com/auth/iam"], @target_audience=nil, @state=nil, @username=nil, @access_type=:offline, @expiry=60, @audience="https://oauth2.googleapis.com/token", @signing_key=#<OpenSSL::PKey::RSA:0x00007fbd7c17e788 oid=rsaEncryption>, @extension_parameters={}, @additional_parameters={}, @connection_info=nil, @grant_type=nil, @refresh_token=nil, @access_token="ya29.c.KtUB-wf94v8nv0vQnQ5RQDZYw76KqNMWU48u6U4uDlYo_zqwuQdoLKeRYeD1jYimRZBW_s-r_efcaB3LuccRu62b6InEZCbfMoGExT5sHZPMEDp8ib7og3CGyPGCKTpwRmPDJ055FDqAZKnfsKJHisqdppzsVM1EgxRhfAeR2BVX1grrAlBTw0ZL78vTgoWmIGF26bVPQ2gRwnndFKuKL8tynUeL8WpBQyIf3OxU8sVcMhCTqOoq4espDyFgkvekThRY3mlGT2mocHG-0Q-YDpQpUOWszlEx">, retries=0, header=nil, normalize_unicode=false, skip_serialization=false, skip_deserialization=false, api_format_version=nil, use_opencensus=true, quota_project=nil, query=nil>, @url=#<Addressable::Template:0xb90 PATTERN:https://iamcredentials.googleapis.com/v1/{+name}:signBlob>, @method=:post, @header={}, @body=nil, @query={}, @params={}, @opencensus_span=nil, @client_version="0.3.0", @request_representation=Google::Apis::IamcredentialsV1::SignBlobRequest::Representation, @request_object={:payload=>"GOOG4-RSA-SHA256\n20210429T153232Z\n20210429/auto/storage/goog4_request\n1e2a0ed22c06dac793d44cc09b10935d002b14116e606ea8c40f20129c343b22"}>
D, [2021-04-29T08:51:46.295629 #89655] DEBUG -- : Sending HTTP post https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/542339357638-cr0dserr2evg7sv1meghqeu703274f3h@developer.gserviceaccount.com:signBlob?
D, [2021-04-29T08:51:46.579239 #89655] DEBUG -- : 200
D, [2021-04-29T08:51:46.579488 #89655] DEBUG -- : #<HTTP::Message:0x00007fa92021bc60 @http_header=#<HTTP::Message::Headers:0x00007fa92021bb20 @http_version="1.1", @body_size=0, @chunked=false, @request_method="POST", @request_uri=#<Addressable::URI:0xb90 URI:https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/542339357638-cr0dserr2evg7sv1meghqeu703274f3h@developer.gserviceaccount.com:signBlob?>, @request_query=nil, @request_absolute_uri=nil, @status_code=200, @reason_phrase="OK", @body_type=nil, @body_charset=nil, @body_date=nil, @body_encoding=#<Encoding:UTF-8>, @is_request=false, @header_item=[["Content-Type", "application/json; charset=UTF-8"], ["Vary", "Origin"], ["Vary", "X-Origin"], ["Vary", "Referer"], ["Content-Encoding", "gzip"], ["Date", "Thu, 29 Apr 2021 14:51:46 GMT"], ["Server", "scaffolding on HTTPServer2"], ["Cache-Control", "private"], ["X-XSS-Protection", "0"], ["X-Frame-Options", "SAMEORIGIN"], ["X-Content-Type-Options", "nosniff"], ["Alt-Svc", "h3-29=\":443\"; ma=2592000,h3-T051=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\""], ["Transfer-Encoding", "chunked"]], @dumped=false>, @peer_cert=#<OpenSSL::X509::Certificate: subject=#<OpenSSL::X509::Name CN=upload.video.google.com,O=Google LLC,L=Mountain View,ST=California,C=US>, issuer=#<OpenSSL::X509::Name CN=GTS CA 1O1,O=Google Trust Services,C=US>, serial=#<OpenSSL::BN:0x00007fa924260410>, not_before=2021-04-13 10:16:07 UTC, not_after=2021-07-06 10:16:06 UTC>, @http_body=#<HTTP::Message::Body:0x00007fa92021b8f0 @body="{\n  \"keyId\": \"153f4805d285f5df11d46b88c6c8d53006e7a6c3\",\n  \"signedBlob\": \"ljejz/RkCSuwZboDqokVdl/hu88RYPoz9FNRTI+k6QAWvFqVN66NZQGncQx5m8qnRrQ12LlbDuK8lIazF7XtZq8HC9m3yTvNkzHOwylWXSARGVSFGosRwCbj9hh+531aLRiE/jC2kTnOQV1qACiRWSHIZ6VkR0QccmelH89j5eZPVioafAGQgU0fpnJoDY6EbIIxZsyR3mLEH0KpSudH0iUYcyO+SptWpdKRQ8pgSL0sLVuA4Ai5uAiN5Im3f0OFSda3Z2tnR6iITKcucw4dmoFqJm8RkNeuVekZPP3QjX9GZ+1ipY3IgAI5fUpErV4kmx2Eo75zHFUX//1ncRu84Q==\"\n}\n", @size=0, @positions=nil, @chunk_size=nil>, @previous=nil>
D, [2021-04-29T08:51:46.579884 #89655] DEBUG -- : Success - #<Google::Apis::IamcredentialsV1::SignBlobResponse:0x00007fa9231fddc0
 @key_id="153f4805d285f5df11d46b88c6c8d53006e7a6c3",
 @signed_blob=
  "\x967\xA3\xCF\xF4d\t+\xB0e\xBA\x03\xAA\x89\x15v_\xE1\xBB\xCF\x11`\xFA3\xF4SQL\x8F\xA4\xE9\x00\x16\xBCZ\x957\xAE\x8De\x01\xA7q\fy\x9B\xCA\xA7F\xB45\xD8\xB9[\x0E\xE2\xBC\x94\x86\xB3\x17\xB5\xEDf\xAF\a\v\xD9\xB7\xC9;\xCD\x931\xCE\xC3)V] \x11\x19T\x85\x1A\x8B\x11\xC0&\xE3\xF6\x18~\xE7}Z-\x18\x84\xFE0\xB6\x919\xCEA]j\x00(\x91Y!\xC8g\xA5dGD\x1Crg\xA5\x1F\xCFc\xE5\xE6OV*\x1A|\x01\x90\x81M\x1F\xA6rh\r\x8E\x84l\x821f\xCC\x91\xDEb\xC4\x1FB\xA9J\xE7G\xD2%\x18s#\xBEJ\x9BV\xA5\xD2\x91C\xCA`H\xBD,-[\x80\xE0\b\xB9\xB8\b\x8D\xE4\x89\xB7\x7FC\x85I\xD6\xB7gkgG\xA8\x88L\xA7.s\x0E\x1D\x9A\x81j&o\x11\x90\xD7\xAEU\xE9\x19<\xFD\xD0\x8D\x7FFg\xEDb\xA5\x8D\xC8\x80\x029}JD\xAD^$\x9B\x1D\x84\xA3\xBEs\x1CU\x17\xFF\xFDgq\e\xBC\xE1">

Style 2

request = { "payload" => string_to_sign }

request object in Google::Apis::Core::ApiCommand:

{"payload"=>"GOOG4-RSA-SHA256\n20210429T153000Z\n20210429/auto/storage/goog4_request\n0cb37904dfe5281443f3be44e31ba639c037164be0bc88850f48f9e6c600efc0"}

HTTP request body (Update, from a later test run):

{}

Result: Causes Google::Apis::ClientError: Invalid request in all Ruby versions.

 #<Google::Apis::Core::ApiCommand:0x00007f83073cf460 @options=#<struct Google::Apis::RequestOptions authorization=#<Signet::OAuth2::Client:0x00007f8306242918 @authorization_uri=nil, @token_credential_uri=#<Addressable::URI:0xb7c URI:https://oauth2.googleapis.com/token>, @client_id=nil, @client_secret=nil, @code=nil, @expires_at=2021-04-29 10:29:58.01145 -0600, @issued_at=2021-04-29 09:29:59.01145 -0600, @issuer="542339357638-cr0dserr2evg7sv1meghqeu703274f3h@developer.gserviceaccount.com", @password=nil, @principal=nil, @redirect_uri=nil, @scope=["https://www.googleapis.com/auth/devstorage.full_control", "https://www.googleapis.com/auth/iam"], @target_audience=nil, @state=nil, @username=nil, @access_type=:offline, @expiry=60, @audience="https://oauth2.googleapis.com/token", @signing_key=#<OpenSSL::PKey::RSA:0x00007f8306242a08 oid=rsaEncryption>, @extension_parameters={}, @additional_parameters={}, @connection_info=nil, @grant_type=nil, @refresh_token=nil, @access_token="ya29.c.KtUB-wdfdrg6zZuRmfEfAUo_jGphTLmXGGab94fv4mAHL-kUSJ22eMfCdaYrK_26kPRw-tW8WUJ6vmAyJ0xMO8Hw5na32C1lyn9Zcu_ZU5-8ijFhM8LFExi1df8u3zdOMSKMC5SkX35dcJuPuK07EQrDY2iWOjPouaLje2P5WD08oxOMaJGBxpF_I2hE6EkyKIkjIx3vBvsQKHsbQL0-d8MHoyf6LTdw3Q9VTwiE5X7uALY2R6U5jrq3hvLIavaaNywh80wVWoKmuIOrTcMlCuOndbYrdcAF">, retries=0, header=nil, normalize_unicode=false, skip_serialization=false, skip_deserialization=false, api_format_version=nil, use_opencensus=true, quota_project=nil, query=nil>, @url=#<Addressable::Template:0xb90 PATTERN:https://iamcredentials.googleapis.com/v1/{+name}:signBlob>, @method=:post, @header={}, @body=nil, @query={}, @params={}, @opencensus_span=nil, @client_version="0.3.0", @request_representation=Google::Apis::IamcredentialsV1::SignBlobRequest::Representation, @request_object={"payload"=>"GOOG4-RSA-SHA256\n20210429T153000Z\n20210429/auto/storage/goog4_request\n0cb37904dfe5281443f3be44e31ba639c037164be0bc88850f48f9e6c600efc0"}>
D, [2021-04-29T08:53:11.085236 #90013] DEBUG -- : Sending HTTP post https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/542339357638-cr0dserr2evg7sv1meghqeu703274f3h@developer.gserviceaccount.com:signBlob?
D, [2021-04-29T08:53:11.345776 #90013] DEBUG -- : 400
D, [2021-04-29T08:53:11.346028 #90013] DEBUG -- : #<HTTP::Message:0x00007fe06f166140 @http_header=#<HTTP::Message::Headers:0x00007fe06f165ce0 @http_version="1.1", @body_size=0, @chunked=false, @request_method="POST", @request_uri=#<Addressable::URI:0xb90 URI:https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/542339357638-cr0dserr2evg7sv1meghqeu703274f3h@developer.gserviceaccount.com:signBlob?>, @request_query=nil, @request_absolute_uri=nil, @status_code=400, @reason_phrase="Bad Request", @body_type=nil, @body_charset=nil, @body_date=nil, @body_encoding=#<Encoding:UTF-8>, @is_request=false, @header_item=[["Vary", "Origin"], ["Vary", "X-Origin"], ["Vary", "Referer"], ["Content-Type", "application/json; charset=UTF-8"], ["Content-Encoding", "gzip"], ["Date", "Thu, 29 Apr 2021 14:53:11 GMT"], ["Server", "scaffolding on HTTPServer2"], ["Cache-Control", "private"], ["X-XSS-Protection", "0"], ["X-Frame-Options", "SAMEORIGIN"], ["X-Content-Type-Options", "nosniff"], ["Alt-Svc", "h3-29=\":443\"; ma=2592000,h3-T051=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\""], ["Transfer-Encoding", "chunked"]], @dumped=false>, @peer_cert=#<OpenSSL::X509::Certificate: subject=#<OpenSSL::X509::Name CN=upload.video.google.com,O=Google LLC,L=Mountain View,ST=California,C=US>, issuer=#<OpenSSL::X509::Name CN=GTS CA 1O1,O=Google Trust Services,C=US>, serial=#<OpenSSL::BN:0x00007fe06fa5b688>, not_before=2021-04-13 10:16:07 UTC, not_after=2021-07-06 10:16:06 UTC>, @http_body=#<HTTP::Message::Body:0x00007fe06f164a48 @body="{\n  \"error\": {\n    \"code\": 400,\n    \"message\": \"Request contains an invalid argument.\",\n    \"status\": \"INVALID_ARGUMENT\"\n  }\n}\n", @size=0, @positions=nil, @chunk_size=nil>, @previous=nil>

Style 3

request = Google::Apis::IamcredentialsV1::SignBlobRequest.new(payload: string_to_sign)

request object in Google::Apis::Core::ApiCommand:

#<Google::Apis::IamcredentialsV1::SignBlobRequest:0x00007ff8fda7bce8 @payload="GOOG4-RSA-SHA256\n20210429T153408Z\n20210429/auto/storage/goog4_request\n502e22058701e6935ea79fb232270d7e37e1e40a1ab1bb001c384c739856471c">

HTTP request body (Update, from a later test run):

{"payload":"R09PRzQtUlNBLVNIQTI1NgoyMDIxMDQyOVQyMjA0MTdaCjIwMjEwNDI5L2F1dG8vc3RvcmFnZS9nb29nNF9yZXF1ZXN0CjdmOWUwYTIyM2RiMGRjMmIwMGU3ZDgwNWQzMGM1YTEzODI4ODJlODNiZjc2MzUxOWZlMmUxNTI4NWRiNjk2MTQ="}

Result: Works in all Ruby versions.

 #<Google::Apis::Core::ApiCommand:0x00007ff8fda7ad20 @options=#<struct Google::Apis::RequestOptions authorization=#<Signet::OAuth2::Client:0x00007ff8fd9b9ff8 @authorization_uri=nil, @token_credential_uri=#<Addressable::URI:0xb7c URI:https://oauth2.googleapis.com/token>, @client_id=nil, @client_secret=nil, @code=nil, @expires_at=2021-04-29 10:34:05.785206 -0600, @issued_at=2021-04-29 09:34:06.785206 -0600, @issuer="542339357638-cr0dserr2evg7sv1meghqeu703274f3h@developer.gserviceaccount.com", @password=nil, @principal=nil, @redirect_uri=nil, @scope=["https://www.googleapis.com/auth/devstorage.full_control", "https://www.googleapis.com/auth/iam"], @target_audience=nil, @state=nil, @username=nil, @access_type=:offline, @expiry=60, @audience="https://oauth2.googleapis.com/token", @signing_key=#<OpenSSL::PKey::RSA:0x00007ff8fd9ba0c0 oid=rsaEncryption>, @extension_parameters={}, @additional_parameters={}, @connection_info=nil, @grant_type=nil, @refresh_token=nil, @access_token="ya29.c.KtUB-wee4j4_EyLtaW5pxwbfZ4_Kf5jHJ-E8p-DXGq4-V_lqKnyRDc0mi51_KxfLnk_uE5c7A8K3EzomYjIeCHfnavMX807F8GtkKhObo9dQ5GIEyU-pZ8tNCU4j6NAAGsqnOE3AEvy9uihyk1fdOBQM8bAGbWMMw43p300ueFUvBwHNw94qXc_QMgWXU0bASqFgJugesePrHo5EXaJmuuYLg1_v3ndQsbGIoWBGJnG4dCOxVC_AtlSsKIdMiojrLU7e8tpL2Ozz5Vtbfu_Vl7e9shHuNDXo">, retries=0, header=nil, normalize_unicode=false, skip_serialization=false, skip_deserialization=false, api_format_version=nil, use_opencensus=true, quota_project=nil, query=nil>, @url=#<Addressable::Template:0xb90 PATTERN:https://iamcredentials.googleapis.com/v1/{+name}:signBlob>, @method=:post, @header={}, @body=nil, @query={}, @params={}, @opencensus_span=nil, @client_version="0.3.0", @request_representation=Google::Apis::IamcredentialsV1::SignBlobRequest::Representation, @request_object=#<Google::Apis::IamcredentialsV1::SignBlobRequest:0x00007ff8fda7bce8 @payload="GOOG4-RSA-SHA256\n20210429T153408Z\n20210429/auto/storage/goog4_request\n502e22058701e6935ea79fb232270d7e37e1e40a1ab1bb001c384c739856471c">>
D, [2021-04-29T08:48:20.236387 #89235] DEBUG -- : Sending HTTP post https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/542339357638-cr0dserr2evg7sv1meghqeu703274f3h@developer.gserviceaccount.com:signBlob?
D, [2021-04-29T08:48:20.508003 #89235] DEBUG -- : 200
D, [2021-04-29T08:48:20.508295 #89235] DEBUG -- : #<HTTP::Message:0x00007fdbbc168c68 @http_header=#<HTTP::Message::Headers:0x00007fdbbc168b78 @http_version="1.1", @body_size=0, @chunked=false, @request_method="POST", @request_uri=#<Addressable::URI:0xb90 URI:https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/542339357638-cr0dserr2evg7sv1meghqeu703274f3h@developer.gserviceaccount.com:signBlob?>, @request_query=nil, @request_absolute_uri=nil, @status_code=200, @reason_phrase="OK", @body_type=nil, @body_charset=nil, @body_date=nil, @body_encoding=#<Encoding:UTF-8>, @is_request=false, @header_item=[["Content-Type", "application/json; charset=UTF-8"], ["Vary", "Origin"], ["Vary", "X-Origin"], ["Vary", "Referer"], ["Content-Encoding", "gzip"], ["Date", "Thu, 29 Apr 2021 14:48:20 GMT"], ["Server", "scaffolding on HTTPServer2"], ["Cache-Control", "private"], ["X-XSS-Protection", "0"], ["X-Frame-Options", "SAMEORIGIN"], ["X-Content-Type-Options", "nosniff"], ["Alt-Svc", "h3-29=\":443\"; ma=2592000,h3-T051=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\""], ["Transfer-Encoding", "chunked"]], @dumped=false>, @peer_cert=#<OpenSSL::X509::Certificate: subject=#<OpenSSL::X509::Name CN=upload.video.google.com,O=Google LLC,L=Mountain View,ST=California,C=US>, issuer=#<OpenSSL::X509::Name CN=GTS CA 1O1,O=Google Trust Services,C=US>, serial=#<OpenSSL::BN:0x00007fdbbd29f518>, not_before=2021-04-13 10:16:07 UTC, not_after=2021-07-06 10:16:06 UTC>, @http_body=#<HTTP::Message::Body:0x00007fdbbc168628 @body="{\n  \"keyId\": \"153f4805d285f5df11d46b88c6c8d53006e7a6c3\",\n  \"signedBlob\": \"HTG1qu8fq0WZUL8/JT8uBJRP0LYh8TDEkoEW9+5T7qaq7hMEsgbB5dnrDMW8Dz+LlcjCeY/JuSlCLWdyW1X+AbKDeKwS1MQ9xw3X3oAXtNU+P/eQaESYRZGy8YhT8iSWaR7oIMc/095fMaAsQuu+7tk2tBIKaydjpacjriCFUQl/C7RKCAdXqvcuy9XInmAGARol+RdgnvRqiOvS9yhGZyM58cuu1G2L552DeldCMbx+B3mZ2M1zGCzXyAe1XOkqPKVB18Ah+QYqKrPVFORcAWbK8X4Ia7zSbtT6hcxmRO2ntXEt2bLvTag+t38SJUiBxLGWhtl9/jcO4O9MjRDmxg==\"\n}\n", @size=0, @positions=nil, @chunk_size=nil>, @previous=nil>

@frankyn
Copy link
Member

frankyn commented Apr 29, 2021

Oh interesting, TIL that the Ruby gem google-api-client was split into multiple gems and mono deprecated!

The PRs are very lengthy, but didn't pick up on anything that might be relevant. I referring to the following PRs:

  1. chore: Split google-apis-core into a separate gem google-api-ruby-client#2134
  2. chore: Extract generator into google-apis-generator gem google-api-ruby-client#2143

@dazuma i'm assuming no major logic changes were made, but wondering if you may have more context.

@quartzmo
Copy link
Member

@frankyn I've been trying to find a change that caused the ArgumentError: unknown keyword: payload errors, but so far I haven't.

@quartzmo
Copy link
Member

I also got the ArgumentError: unknown keyword: payload error on Ruby 2.7.2.

@frankyn
Copy link
Member

frankyn commented Apr 29, 2021

@quartzmo for the logs, I see that we only have response JSON body, is there an equivalent request JSON structure?

BTW: Thanks for getting the additional logging!!

@quartzmo
Copy link
Member

@frankyn I wasn't able to figure out how to log the request. I can try again this evening.

@quartzmo
Copy link
Member

@frankyn I updated the logs comment above with the request bodies.

It turns out that in Style 2, with the { "payload" => string_to_sign } request object argument, the body is {}.

@frankyn
Copy link
Member

frankyn commented Apr 29, 2021

Thanks for the additional context @quartzmo!

IIUC the tests and rubydoc samples should use the class data model representation instead of a Ruby Hash to construct the request payload.

The request bodies are equivalent, so wondering if there's something going on the URL because it contains special characters in the URI with the Google Service account email.

BTW: I'm okay with shelving this and moving forward with the PR you have open at this point to address the broken tests.

@quartzmo
Copy link
Member

I reproduced what's happening inside ApiCommand:

irb(main):005:0> request_object = {"payload" => "123"}
=> {"payload"=>"123"}
irb(main):006:0> request_representation = Google::Apis::IamcredentialsV1::SignBlobRequest::Representation
=> Google::Apis::IamcredentialsV1::SignBlobRequest::Representation
irb(main):007:0> body = request_representation.new(request_object).to_json(user_options: { skip_undefined: true })
=> "{}"

@quartzmo
Copy link
Member

@frankyn Keep in mind that we didn't use Style 2 originally, we used Style 1. Which still works if you can get past the ArgumentError: unknown keyword: payload errors:

irb(main):021:0> body = request_representation.new({"payload": "123"}).to_json(user_options: { skip_undefined: true })
=> "{\"payload\":\"MTIz\"}"

@quartzmo
Copy link
Member

Thank you @frankyn for finding the cause of the ArgumentError: unknown keyword: payload errors in the Ruby 2.x versions: In #10575, I removed the trailing empty hash ({}) in test calls to sign_service_account_blob, in order to fix the Ruby 3.0 error ArgumentError: wrong number of arguments (given 3, expected 1..2). I must have forgotten to regression-test this change against the Ruby 2.x versions, since it introduced the unknown keyword error in those versions.

@quartzmo
Copy link
Member

quartzmo commented Jun 29, 2021

These failures still occur in the Kokoro CI environment.

 1) Error:
Google::Cloud::Storage::Bucket::post_object::v2::storage#test_0002_generates a signed post object using signBlob API:
Google::Apis::ClientError: Invalid request
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:228:in `check_status'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/api_command.rb:134:in `check_status'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:194:in `process_response'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:310:in `execute_once'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:113:in `block (2 levels) in execute'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:110:in `block in execute'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:102:in `execute'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/base_service.rb:377:in `execute_or_queue_command'
    /h/.gem/gems/google-apis-iamcredentials_v1-0.4.0/lib/google/apis/iamcredentials_v1/service.rb:158:in `sign_service_account_blob'
    /workspace/google-cloud-storage/acceptance/storage/bucket_post_object_v2_test.rb:79:in `block (3 levels) in <top (required)>'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v2.rb:135:in `generate_signature'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v2.rb:113:in `post_object'
    /workspace/google-cloud-storage/lib/google/cloud/storage/bucket.rb:1913:in `post_object'
    /workspace/google-cloud-storage/acceptance/storage/bucket_post_object_v2_test.rb:83:in `block (2 levels) in <top (required)>'

  2) Error:
Google::Cloud::Storage::Bucket::generate_signed_post_policy_v4::storage#test_0002_generates a signed post object v4 using signBlob API:
Google::Apis::ClientError: Invalid request
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:228:in `check_status'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/api_command.rb:134:in `check_status'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:194:in `process_response'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:310:in `execute_once'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:113:in `block (2 levels) in execute'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:110:in `block in execute'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:102:in `execute'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/base_service.rb:377:in `execute_or_queue_command'
    /h/.gem/gems/google-apis-iamcredentials_v1-0.4.0/lib/google/apis/iamcredentials_v1/service.rb:158:in `sign_service_account_blob'
    /workspace/google-cloud-storage/acceptance/storage/bucket_generate_signed_post_policy_v4_test.rb:79:in `block (3 levels) in <top (required)>'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v4.rb:357:in `generate_signature'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v4.rb:69:in `post_object'
    /workspace/google-cloud-storage/lib/google/cloud/storage/bucket.rb:2073:in `generate_signed_post_policy_v4'
    /workspace/google-cloud-storage/acceptance/storage/bucket_generate_signed_post_policy_v4_test.rb:83:in `block (2 levels) in <top (required)>'

  3) Error:
Google::Cloud::Storage::signed_url::v4::storage::Google::Cloud::Storage::Bucket::signed_url#test_0002_should create a signed read url v4 using IAM signBlob API:
Google::Apis::ClientError: Invalid request
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:228:in `check_status'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/api_command.rb:134:in `check_status'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:194:in `process_response'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:310:in `execute_once'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:113:in `block (2 levels) in execute'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:110:in `block in execute'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:102:in `execute'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/base_service.rb:377:in `execute_or_queue_command'
    /h/.gem/gems/google-apis-iamcredentials_v1-0.4.0/lib/google/apis/iamcredentials_v1/service.rb:158:in `sign_service_account_blob'
    /workspace/google-cloud-storage/acceptance/storage/signed_url_v4_test.rb:127:in `block (4 levels) in <top (required)>'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v4.rb:216:in `block in service_account_signer'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v4.rb:124:in `signed_url'
    /workspace/google-cloud-storage/lib/google/cloud/storage/bucket.rb:1734:in `signed_url'
    /workspace/google-cloud-storage/acceptance/storage/signed_url_v4_test.rb:132:in `block (3 levels) in <top (required)>'

  4) Error:
Google::Cloud::Storage::signed_url::v2::storage::Google::Cloud::Storage::Bucket::signed_url#test_0002_should create a signed read url using IAM signBlob API:
Google::Apis::ClientError: Invalid request
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:228:in `check_status'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/api_command.rb:134:in `check_status'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:194:in `process_response'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:310:in `execute_once'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:113:in `block (2 levels) in execute'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:110:in `block in execute'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
    /h/.gem/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/http_command.rb:102:in `execute'
    /h/.gem/gems/google-apis-core-0.3.0/lib/google/apis/core/base_service.rb:377:in `execute_or_queue_command'
    /h/.gem/gems/google-apis-iamcredentials_v1-0.4.0/lib/google/apis/iamcredentials_v1/service.rb:158:in `sign_service_account_blob'
    /workspace/google-cloud-storage/acceptance/storage/signed_url_v2_test.rb:127:in `block (4 levels) in <top (required)>'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v2.rb:135:in `generate_signature'
    /workspace/google-cloud-storage/lib/google/cloud/storage/file/signer_v2.rb:128:in `signed_url'
    /workspace/google-cloud-storage/lib/google/cloud/storage/bucket.rb:1721:in `signed_url'
    /workspace/google-cloud-storage/acceptance/storage/signed_url_v2_test.rb:132:in `block (3 levels) in <top (required)>'

128 runs, 1119 assertions, 0 failures, 4 errors, 2 skips

@tritone
Copy link

tritone commented Nov 1, 2021

Hi, I see this is now OOSLO. @bajajneha27 do you need any support on next steps here? I'm on duty this week for the storage DPE team so happy to assist.

@bajajneha27
Copy link
Contributor

Hi,

I'm sorry, I haven't been able to look at it.

I tried debugging this and I'm running into the issue:
Google::Apis::ClientError (forbidden: The caller does not have permission)
when I try running this code manually in irb. I'm not sure if this is the actual issue or something I'm missing.

@bajajneha27
Copy link
Contributor

Hi,
I've recently worked on couple of storage PRs : 14407 , 14176 and 14141 . I did not see the CI failing due to these reasons.
@quartzmo Could you please point me to the link where these tests are failing?

Also, these tests may fail in your local as they were failing for me due to permission issue Google::Apis::ClientError (forbidden: The caller does not have permission). In order to fix the error, I had to add Service Account Token Creator role to the service account as suggested by @frankyn.

@yoshi-automation yoshi-automation added the 🚨 This issue needs some love. label Nov 3, 2021
@quartzmo
Copy link
Member

quartzmo commented Nov 3, 2021

@bajajneha27 I also see the successful CI acceptance test run now for your open PR #14407, thank you for providing the links. I would recommend closing this issue now.

@tritone
Copy link

tritone commented Nov 4, 2021

Great, thanks for following up! I will close this.

@tritone tritone closed this as completed Nov 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the Cloud Storage API. priority: p2 Moderately-important priority. Fix may not be included in next release. 🚨 This issue needs some love. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
6 participants