cmd/go: git remote (Gerrit) did not send all necessary objects

[ukai]

What version of Go are you using (go version)?

$ go version
go version go1.12 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/usr/local/google/home/ukai/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/usr/local/google/home/ukai/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/lib/google-golang"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/google-golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build741696436=/tmp/go-build -gno-record-gcc-switches"

What did you do?

$ ls /tmp/go
ls: cannot access '/tmp/go': No such file or directory
$ mkdir /tmp/go
$ GOPATH=/tmp/go GO111MODULE=on go install go.chromium.org/goma/server/cmd/remoteexec_proxy

What did you expect to see?

go modules downloaded and binary is built.

What did you see instead?

...
go: finding grpc.go4.org v0.0.0-20170609214715-11d0a25b4919
go: google.golang.org/api@v0.0.0-20181030000543-1d582fd0359e: git -c protocol.version=0 fetch --unshallow -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /tmp/go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0: exit status 128:
        fatal: bad object c7964aab01c00a07d0cc6216a78a5570459cebfd
        error: remote did not send all necessary objects
..
go: error loading module requirements

I could fix it by

$ cd /tmp/go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0; git fetch --unshallow

then go install works as expected.

https://bugs.chromium.org/p/chromium/issues/detail?id=951546

[agnivade]

@bcmills @jayconrod

[KillerX]

I can confirm that this is happening to me as well. It is not 100% constant. My docker builds experience it about 98% of the time.

go: google.golang.org/api@v0.0.0-20180910000450-7ca32eb868bf: git -c protocol.version=0 fetch --unshallow -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0: exit status 1:
	fatal: bad object c7964aab01c00a07d0cc6216a78a5570459cebfd
	error: https://code.googlesource.com/google-api-go-client did not send all necessary objects

A git fetch seems to fix it.

Same issues with go 1.11.5, 1.11.8, 1.12.3 in Docker (golang:1.12.3), and installed locally on Ubuntu.

Local git:

➜   git --version 
git version 2.19.1

I also less frequently get this issue:

go: cloud.google.com/go@v0.0.0-20180131234750-2de512d2700d: git -c protocol.version=0 fetch --unshallow -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/mod/cache/vcs/b0e27935eb83c1d7843713bafab507e95768b550f0552cb42d9f41e5fd9c8375: exit status 1:
        fatal: bad object 3fdddcac5af93820a98679aa0008db7413a13265
        error: https://code.googlesource.com/gocloud did not send all necessary objects

EDIT:

It appears that this patch resolves the issue for my builds:

--- a/packages/go/packagename/go.mod
+++ b/packages/go/packagename/go.mod
@@ -7,7 +7,7 @@ require (
        redacted/logging v0.0.0
        golang.org/x/net v0.0.0-20190320064053-1272bf9dcd53 // indirect
        golang.org/x/oauth2 v0.0.0-20190319182350-c85d3e98c914 // indirect
-       google.golang.org/api v0.3.1
+       google.golang.org/api  v0.0.0-20180603000442-8e296ef26005
        google.golang.org/genproto v0.0.0-20190404172233-64821d5d2107
        google.golang.org/grpc v1.19.1
 )

[jayconrod]

Reproduced locally with the same error.

Also seen with the same instructions as above:

go: golang.org/x/exp@v0.0.0-20190121172915-509febef88a4: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /var/folders/rq/x0692kqj6ml8cvrhcqh5bswc008xj1/T/tmp.yf9bsDvK/pkg/mod/cache/vcs/abc0aaaae9fbb1a906994bfb84d444789fc395ac48e2b740ecbbfeebd3047a32: exit status 128:
	fatal: remote error: authentication error

Perhaps Gerrit is acting up again?

cc @heschik @andybons @hakim @dmitshur

[jayconrod]

I've traced this to a sequence of Git commands that reproduces this issue outside of Go. The Gerrit team has been notified and is working to resolve the issue.

[mr-tron]

Is it some workaround for the problem?

[heschi]

The latest we have from the Gerrit team is that this might be a bug specific to the content of the google.golang.org/api repository. As suggested above, a git fetch should work around it. I think fetching a different version might help too.

[mr-tron]

Yes. Second running also helps, but it very ugly hack. I added vendor dir to my projects. It looks like better workaround.

[draftcode]

Hello, this is the team that manages googlesource.com. *.googlesource.com is hosted by using JGit, and we believe that there's a bug in upstream JGit that cannot handle unshallow git-fetch. We recognize that this is reproducible easily, but we cannot spot the buggy code in the JGit. If there's an option to do a full clone always instead of partial clone, that can be a workaround.