ggcr: Image write concurrency errors

[phillebaba]

Describe the bug

Currently it is not possible to write images with overlapping layers concurrently. The reason it is not possible is because two layers could write to the same file at the same time. This would result in data being overwritten and the incorrect written byte count returned.

go-containerregistry/pkg/v1/layout/write.go

Lines 235 to 239 in ff385a9

	if n, err := io.Copy(w, rc); err != nil || renamer == nil {
	return err
	} else if size != -1 && n != size {
	return fmt.Errorf("expected blob size %d, but only wrote %d", size, n)
	}

To Reproduce

The simplest way to reproduce this issue is to attempt to write the same image twice concurrently.

package main

import (
	"context"
	"fmt"
	"os"
	"path/filepath"

	"github.com/google/go-containerregistry/pkg/crane"
	"github.com/google/go-containerregistry/pkg/v1/cache"
	"github.com/google/go-containerregistry/pkg/v1/empty"
	"github.com/google/go-containerregistry/pkg/v1/layout"
	"golang.org/x/sync/errgroup"
)

func main() {
	fmt.Println("running test")
	err := run()
	if err != nil {
		fmt.Println(err)
		os.Exit(1)
		return
	}
	fmt.Println("completed test")
}

func run() error {
	fmt.Println("pulling image")
	img, err := crane.Pull("ghcr.io/spegel-org/benchmark:v1-10MB-4")
	if err != nil {
		return err
	}
	layers, err := img.Layers()
	if err != nil {
		return err
	}
	for _, l := range layers {
		dgst, err := l.Digest()
		if err != nil {
			return err
		}
		fmt.Println(dgst.String())
	}
	fmt.Println("pulled image")
	tmpDir, err := os.MkdirTemp("", "")
	if err != nil {
		return err
	}
	fmt.Println("temp dir", tmpDir)
	layoutPath, err := layout.Write(filepath.Join(tmpDir, "store"), empty.Index)
	if err != nil {
		return err
	}
	g, _ := errgroup.WithContext(context.TODO())
	for i := range 10 {
		i := i
		g.Go(func() error {
			img = cache.Image(img, cache.NewFilesystemCache(filepath.Join(tmpDir, fmt.Sprintf("cache-%d", i))))
			fmt.Printf("writing iteration %d\n", i)
			err = layoutPath.AppendImage(img, layout.WithAnnotations(map[string]string{"foo": "bar"}))
			if err != nil {
				fmt.Println("write error", err)
				return err
			}
			fmt.Printf("completed iteration %d\n", i)
			return nil
		})
	}
	err = g.Wait()
	if err != nil {
		return err
	}
	err = os.RemoveAll(tmpDir)
	if err != nil {
		return err
	}
	return nil
}

Running this code results in the following errors.

write error error writing layer: expected blob size 2622396, but only wrote 23
write error error writing layer: expected blob size 307023, but only wrote 23
write error error writing layer: expected blob size 307023, but only wrote 23
write error error writing layer: expected blob size 307023, but only wrote 23
write error error writing layer: expected blob size 2622396, but only wrote 45
write error error writing layer: expected blob size 2622395, but only wrote 23
write error error writing layer: expected blob size 307023, but only wrote 23
write error error writing layer: expected blob size 2622399, but only wrote 23
write error error writing layer: expected blob size 2622396, but only wrote 23

Expected behavior

Images with overlapping layers should successfully write when done concurrently.

Additional context

My suggestion to solve this issue is to introduce file locks to the write blob logic to make sure that a single file could only be written by one thread. This would solve the issue for concurrently writing multiple images with overlapping files and also images with duplicate layers. This could in theory solve the issues described in #1920.