Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tarexport: Plumb ctx, add OTEL spans, handle cancellation #47629

Merged
merged 1 commit into from
May 14, 2024

Conversation

vvoland
Copy link
Contributor

@vvoland vvoland commented Mar 25, 2024

Pass context.Context through tarexport.Load and tarexport.Save. Create OTEL spans for the most time consuming operations.

Also, handle context cancellations to actually end saving/loading when the operation is cancelled - before this PR the daemon would still be performing the operation even though the user already cancelled it.

- How to verify it

- Description for the changelog

Fix `docker save` and `docker load` not ending on the daemon side when the operation was cancelled (eg. Ctrl+C)

- A picture of a cute animal (not mandatory but encouraged)

@vvoland vvoland added status/2-code-review kind/enhancement Enhancements are not bugs or new features but can improve usability or performance. impact/changelog area/images labels Mar 25, 2024
@vvoland vvoland added this to the 27.0.0 milestone Mar 25, 2024
@vvoland vvoland self-assigned this Mar 25, 2024
@vvoland vvoland modified the milestones: 26.1.0, 27.0.0 Apr 10, 2024
@@ -393,6 +423,11 @@ func (s *saveSession) saveImage(id image.ID) (map[layer.DiffID]distribution.Desc
var layers []layer.DiffID
var foreignSrcs map[layer.DiffID]distribution.Descriptor
for i, diffID := range img.RootFS.DiffIDs {
select {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we check this down at L581 as well? (sorry GH won't let me add a comment there)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure - it's calling system.Chtimes on two paths. Doesn't look like it has a potential to take a noticeable amount of time.

r io.Reader
}

func NewCtxReader(ctx context.Context, r io.Reader) io.Reader {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need a doc string here.

Also I think something like an io.PipeReader would be a better for the use case which you can call CloseWithError(ctx.Err()) on and handle cancellation a little more completely (rather than be potentially blocked on reading the underlying reader).

e.g.

pr, pw := io.Pipe()

go func() {
  <-ctx.Done()
  pr.CloseWithError(ctx.Err())
}()

// ...

Or we can forgo the pipe and just close the reader normally when the context is cancelled and check after the io.Copy if it was cancelled.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if the pipe actually solves anything though.

If the underlying reader Read method blocks indefinitely, then there's actually no way of stopping it, other than calling Close on the reader if it's actually also an io.Closer and hoping that ends all pending Reads.

I would prefer to avoid doing any Close inside the wrapper, because of the defer closer.Close() being actually the most common way of handling calling close. Doing that could potentially result in Close being called twice (first in the wrapper Read call, and then in the defer), which is implementation specific and can lead to subtle bugs like #46418.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a CopyCtx as a replacement for the io.Copy as a best-effort recovery in a situation where Read blocks completely - in this case the calling goroutine won't be blocked, only the child goroutine that does the io.Copy.

@vvoland vvoland force-pushed the tarexport-tracing-ctx-cancel branch 2 times, most recently from 3b2bc73 to 2ca48aa Compare May 10, 2024 08:07
@vvoland vvoland requested a review from cpuguy83 May 13, 2024 09:52
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the only part I have some questions about; not necessarily a blocker, but just cautious where we extend anything in pkg/;

  • Do we expect these functions to be used externally (i.e., will we need these in, e.g. buildkit or elsewhere)?
  • ☝️ if not (or at least not currently) perhaps we should consider moving this to an internal/ package
  • ☝️ if we DO need those externally; is pkg/ioutils the correct package for this, or should it be a separate package? (e.g. contextreader / contextreader.New() (I'd like to be sure we don't end up making ioutils too much of a "utils" package, and it to become a "grab bag" or only "somewhat" related functionality).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm right, I don't expect it to be used outside of moby, so I'm definitely +1 on making it internal.

We can always make it public if we need to, but going the reverse is always harder.

Let me move it to internal.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Yes, trying to be more conservative a bit on expanding things in pkg/ ❤️

Comment on lines 37 to 41
// NewCtxReader wraps the given reader with a reader that doesn't proceed with
// reading if the context is done.
//
// Note: Read will still block if the underlying reader blocks.
func NewCtxReader(ctx context.Context, r io.Reader) io.Reader {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was curious as well if we checked if the Go team is considering something along these lines (and/or if there's proposals), and if we in that case could align to whatever design may be worked on.

I did find this proposal, but haven't read in-depth if there's any progress / good proposal on design there;

Copy link
Member

@laurazard laurazard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Pass `context.Context` through `tarexport.Load` and `tarexport.Save`.
Create OTEL spans for the most time consuming operations.

Also, handle context cancellations to actually end saving/loading when
the operation is cancelled - before this PR the daemon would still be
performing the operation even though the user already cancelled it.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Copy link
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@thaJeztah thaJeztah merged commit ae976b9 into moby:master May 14, 2024
132 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/images area/metrics/otel area/metrics impact/changelog kind/enhancement Enhancements are not bugs or new features but can improve usability or performance. status/2-code-review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants