storage: Loading creds during NewClient breaks application without default creds

[Xuanwo]

Client

storage

Environment

Self-hosted CI (not on GCE)

Go Environment

$ go version
go 1.16

Code

package gcs

import (
	"context"
	"net/http"
	"testing"

	gs "cloud.google.com/go/storage"
	"golang.org/x/oauth2/google"
	"google.golang.org/api/option"
	htransport "google.golang.org/api/transport/http"
)

func TestClient(t *testing.T) {
	ctx := context.Background()

	var err error

	// We will have a costumed http client for timeout control.
	hc := http.DefaultClient
	hc.Transport, err = htransport.NewTransport(ctx, hc.Transport,
		// The cred could be loaded from file or JSON that user input.
		option.WithCredentials(&google.Credentials{}))
	if err != nil {
		t.Errorf("new transport error")
	}

	_, err = gs.NewClient(ctx, option.WithHTTPClient(hc))
	if err != nil {
		t.Errorf("new client failed: %v", err)
	}
}

func TestClientWithDuplicatedCredentials(t *testing.T) {
	ctx := context.Background()

	var err error

	// We will have a costumed http client for timeout control.
	hc := http.DefaultClient
	hc.Transport, err = htransport.NewTransport(ctx, hc.Transport,
		// The cred could be loaded from file or JSON that user input.
		option.WithCredentials(&google.Credentials{}))
	if err != nil {
		t.Errorf("new transport error")
	}

	// The trick way
	_, err = gs.NewClient(ctx, option.WithHTTPClient(hc), option.WithCredentials(&google.Credentials{}))
	if err != nil {
		t.Logf("init via set duplicated credentials could succeed")
	}
}

Expected behavior

Init with success.

Actual behavior

Output error:

    utils_test.go:30: new client failed: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

Additional context

We met this error during our depandabot upgrade from 1.17 to 1.18: beyondstorage/go-service-gcs#80

Our failed build log: https://teamcity.beyondstorage.io/buildConfiguration/Services_Gcs_IntegrationTests/934?showLog=934_67_67

---

This problem is introduced by #4604:

google-cloud-go/storage/storage.go

Lines 134 to 140 in 0f7457c

	c, err := transport.Creds(ctx, opts...)
	if err != nil {
	return nil, err
	}
	creds = c
	opts = append(opts, internaloption.WithCredentials(creds))

Do we have a better solution to handle the situation when user doesn't pass credential during storage.NewClient?

[tritone]

Thanks for reporting. @BrennaEpp is investigating this; @codyoss FYI as well.

[Xuanwo]

Bravo!

[tritone]

This is released in storage/v1.18.1. Thanks again for your report, we definitely missed this case.