Upload size estimate scans original directory even when folder override or filesystem snapshotting is used

[Hakkin]

While investigating an issue with Volume Shadow Copy, I noticed that Kopia was still scanning/reading from the original directory even when I specified to use a filesystem snapshot (VSS). I thought this was odd so I looked into it, it seems the culprit is here:

kopia/snapshot/snapshotfs/upload.go

Lines 1289 to 1301 in 2b92388

	go func() {
	defer scanWG.Done()
	wrapped := u.wrapIgnorefs(estimateLog(ctx), entry, policyTree, false /* reportIgnoreStats */)
	ds, _ := u.scanDirectory(scanctx, wrapped, policyTree)
	u.Progress.EstimatedDataSize(ds.numFiles, ds.totalFileSize)
	}()
	wrapped := u.wrapIgnorefs(uploadLog(ctx), entry, policyTree, true /* reportIgnoreStats */)
	s.RootEntry, err = u.uploadDirWithCheckpointing(ctx, wrapped, policyTree, previousDirs, sourceInfo)

A go routine for scanDirectory is started before uploadDirWithCheckpointing, and the processing of filesystem snapshotting and directory overrides is only done in uploadDirWithCheckpointing.

kopia/snapshot/snapshotfs/upload.go

Lines 594 to 597 in 2b92388

	overrideDir, err := u.executeBeforeFolderAction(ctx, "before-snapshot-root", policyTree.EffectivePolicy().Actions.BeforeSnapshotRoot, localDirPathOrEmpty, &hc)
	if err != nil {
	return nil, dirReadError{errors.Wrap(err, "error executing before-snapshot-root action")}
	}

kopia/snapshot/snapshotfs/upload.go

Line 610 in 2b92388

switch osSnapshotDir, cleanup, err := createOSSnapshot(ctx, rootDir, p); {

This means the upload estimates when using folder overrides (including filesystem snapshots) will be incorrect.

I'm not very familiar with the codebase, so it may not be this simple, but it seems like this might be able to be fixed by simply moving the folder override and filesystem snapshot handling that is in uploadDirWithCheckpointing to before the filesize estimate go routine, and then pass the resulting fs.Directory to both the EstimatedDataSize and uploadDirWithCheckpointing functions.