Detect file moves in local source tracking #574

iowillhoit · 2024-05-03T16:07:42Z

What does this PR do?

when a file is moved from one location to another within a project (ex: from force-app/main/default/classes to other-dir/classes) but has the same filename and content, it is NOT considered an add and a delete. Source tracking updates its local filesystem tracking to "clear" those changes.

This does not apply if you move files between different parents (ex: moving a field from Foo__c to Bar__c) or moving a file from one LWC to another--that still counts as an add a delete.

What issues does this PR fix or reference?

forcedotcom/cli#2682
@W-15354803@

…g into ew/file-moves

src/shared/localShadowRepo.ts

mshanemc · 2024-05-08T13:49:13Z

src/shared/localShadowRepo.ts

+      const message = 'Files were found that have the same basename and hash. Skipping the commit of these files';
+      this.logger.warn(message);
+      await Lifecycle.getInstance().emitWarning(message);
+      await Lifecycle.getInstance().emitTelemetry({ eventName: 'moveFileHashBasenameCollisionsDetected' });


if we're capturing telemetry anyway, let's capture the filenames

if we're capturing telemetry anyway, let's capture the filenames.

and always Promise.all unrelated consecutive awaits

The "ignore" Maps will not contain all of the filenames since we are "setting" over the top of the new values as we go. We could add some more logic to calculate them, but i don't know if it is worth the extra complexity. Telemetry will let us know how often this happens

src/shared/localShadowRepo.ts

refactor: more functions!

mshanemc · 2024-05-08T18:29:59Z

QA notes:
using deploy-retrieve 3.6.12-dev.2 (dev) w/ dreamhouse. Unless otherwise stated, using deploy preview to not bother the org about this stuff.

preview looks good before a deploy.
✅ running both this change and regular, the result is consistently 91 items to deploy.

retrieve preview
✅ shows empty, as does current

actual deploy
✅ correct, matches current sf
preview after deploy
✅ correct, matches current sf

create new packageDir and move a file to the same dir over there (other-dir/classes/FileUtilities.cls and its cls.-meta.xml)
✅ shows no changes.
📓 we should probably tell people in the instructions to keep stuff like that together

move GeoCodingService files the same way, with SF_LOG_LEVEL=trace DEBUG=* sf project deploy preview to view logs

[16:30:13.767] TRACE (sf:ShadowRepo): start: getStatus (noCache = false)
[16:30:13.800] TRACE (sf:ShadowRepo): start: getStatus (noCache = false)
[16:30:13.800] TRACE (sf:ShadowRepo): done: getStatus (noCache = false)
[16:30:13.848] DEBUG (sf:ShadowRepo): Files have moved. Committing moved files:
[16:30:13.848] DEBUG (sf:ShadowRepo): File 'force-app/main/default/classes/GeocodingServiceTest.cls' was moved to 'other-dir/classes/GeocodingServiceTest.cls'
File 'force-app/main/default/classes/GeocodingServiceTest.cls-meta.xml' was moved to 'other-dir/classes/GeocodingServiceTest.cls-meta.xml'
[16:30:13.848] DEBUG (sf:ShadowRepo): adding 2 files of 2 deployedFiles to git
[16:30:13.864] TRACE (sf:ShadowRepo): start: commitChanges git.commit
[16:30:13.885] TRACE (sf:ShadowRepo): start: getStatus (noCache = true)
[16:30:13.911] TRACE (sf:ShadowRepo): start: getStatus (noCache = false)
[16:30:13.911] TRACE (sf:ShadowRepo): done: getStatus (noCache = false)
[16:30:13.911] TRACE (sf:ShadowRepo): done: getStatus (noCache = true)
[16:30:13.911] TRACE (sf:ShadowRepo): done: commitChanges git.commit
[16:30:13.911] TRACE (sf:ShadowRepo): done: getStatus (noCache = false)

📓 the seems like a lot of getStatus being called. At least they're fast.

renaming an existing file [should still show up as a add + delete]
✅ yep

✅ put the name back and it disappears

moving a field from 1 custom object to another (test case from @VivekMChawla)
❌ removes from tracking. It probably shouldn't

File 'force-app/main/default/objects/Broker__c/fields/Title__c.field-meta.xml' was moved to 'force-app/main/default/objects/Property__c/fields/Title__c.field-meta.xml'

move an unresolvable file (README.md moved from force-app/main/default/objects/ to other-dir/classes/
✅ doesn't show up when created in force-app
✅ doesn't show up when moved

move an entire bundle

make lwc folder in other-dir (✅ preview works fine with empty folder)
move PropertySummary to new lwc folder
original lists the original LWC (📓 but twice, probably the result of various files resolving to the meta.xml. We should probably deduplicate those when type+path+fullname match)

Will Deploy [2] files.
 Type                     Fullname        Path                                                      
 ──────────────────────── ─────────────── ───────────────────────────────────────────────────────── 
 LightningComponentBundle propertySummary other-dir/lwc/propertySummary/propertySummary.js-meta.xml 
 LightningComponentBundle propertySummary other-dir/lwc/propertySummary/propertySummary.js-meta.xml

✅ moving a whole LWC gives you the expected results and logs

File 'force-app/main/default/lwc/propertyLocation/propertyLocation.html' was moved to 'other-dir/lwc/propertyLocation/propertyLocation.html'
File 'force-app/main/default/lwc/propertyLocation/propertyLocation.js' was moved to 'other-dir/lwc/propertyLocation/propertyLocation.js'
File 'force-app/main/default/lwc/propertyLocation/propertyLocation.js-meta.xml' was moved to 'other-dir/lwc/propertyLocation/propertyLocation.js-meta.xml'

move customLabels file
✅ expected output, expected logs

deploy preview where we move a file and it's changed in the org [expect a conflict in the new location]

move customlabels AND edit it in the org
✅ shows as a retrieve from org (no conflict!)

refactor: registry from STL

mshanemc · 2024-05-14T21:55:28Z

QA notes:

✅ copy paste is treated as an add
✅ move classes to a different dir
✅ move a field from one obj in force-app to another
✅ move the entire object into other-dir
✅ move one field back to an empty object in force-app

mshanemc · 2024-05-15T20:40:00Z

test/nuts/local/localTrackingFileMovesImage.nut.ts

@@ -75,5 +79,7 @@ describe('it detects image file moves ', () => {
    expect(await repo.getChangedFilenames())
      .to.be.an('array')
      .with.lengthOf(0);
+
+    delete process.env.SF_BETA_TRACK_FILE_MOVES;


nit: I have a slight preference for doing these in an before|afterEach so that they next test added doesn't have to remember to unset it.

mshanemc · 2024-05-15T21:16:15Z

QA:
✅ tested with env true, false and not set

iowillhoit and others added 22 commits May 3, 2024 11:07

chore: clean up

392d14a

fix: ignore type linting

4cdd5b8

Merge branch 'main' into ew/file-moves

bb190c7

chore: merge main, fix types

ff3bdd3

chore: bump deps for xnuts

976dc9b

chore(release): 6.0.5-dev.0 [skip ci]

49cb802

chore: bump deps

cb5fb6f

fix: add performance markers

8f78094

chore: bump deps

fdf85b1

chore(release): 6.0.5-dev.1 [skip ci]

7b78bc2

chore: pjson cleanup

dda9d0b

Merge branch 'ew/file-moves' of github.com:forcedotcom/source-trackin…

8755abe

…g into ew/file-moves

chore: bump deps for xnuts

1999927

chore: early exits

78ad8b1

chore: check includes first

34820cc

chore: promise all getInfo

fe52fe5

feat: add env var to skip moved file detection

73a70fd

chore: use sets for quick lookup

49e4899

chore: slight perf improvment using set in getInfo - 5 percent

7cfe207

chore: emit telem if file move detection is skipped

9f82c49

refactor: rework how we compare matches

2d00028

chore: const name

9e842e9

mshanemc reviewed May 8, 2024

View reviewed changes

mshanemc and others added 3 commits May 8, 2024 10:45

refactor: more functions!

fbe3be3

Merge pull request #578 from forcedotcom/ew/file-moves-refactor

dd5bde9

refactor: more functions!

chore(release): 6.0.5-dev.2 [skip ci]

7a3bced

iowillhoit mentioned this pull request May 8, 2024

Detect file moves salesforcecli/plugin-deploy-retrieve#1000

Merged

iowillhoit changed the title ~~WIP: File moves~~ Detect file moves in local source tracking May 8, 2024

chore: tslint

82df743

iowillhoit requested a review from a team as a code owner May 10, 2024 21:06

iowillhoit and others added 19 commits May 13, 2024 10:59

chore: clean up

7b8d5f3

test: temp logs

2b31f22

test: temp logs

361940f

test: more temp logs

f385cbb

test: temp logs

d7e69c3

refactor: registry from STL

967ad14

chore: debugging windows

c312403

chore: remove logs

87fcfdb

Merge branch 'main' into ew/file-moves

3813a9a

fix: merge conflict

7e78049

Merge pull request #582 from forcedotcom/sm/file-move-composability

f986a1b

refactor: registry from STL

fix: fix file move debug

e7d0591

chore: testing windows filepaths

4945b95

refactor: win paths

127e640

chore: return order consistency

6bb7217

chore: bump deps for xNuts

c5cb867

chore(release): 6.0.5-dev.3 [skip ci]

1e16f7b

chore: bump deps for xNuts

f58b518

chore(release): 6.0.5-dev.4 [skip ci]

cf30b8b

mshanemc approved these changes May 14, 2024

View reviewed changes

chore: opt-in with beta env var

1c987a8

mshanemc reviewed May 15, 2024

View reviewed changes

mshanemc approved these changes May 15, 2024

View reviewed changes

mshanemc merged commit 06fdb1b into main May 15, 2024
32 checks passed

mshanemc deleted the ew/file-moves branch May 15, 2024 21:16

iowillhoit mentioned this pull request May 15, 2024

Bump STL salesforcecli/plugin-source#1183

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Detect file moves in local source tracking #574

Detect file moves in local source tracking #574

iowillhoit commented May 3, 2024 •

edited by mshanemc

mshanemc May 8, 2024

mshanemc May 8, 2024

iowillhoit May 8, 2024

mshanemc commented May 8, 2024 •

edited

mshanemc commented May 14, 2024

mshanemc May 15, 2024 •

edited

mshanemc commented May 15, 2024

Detect file moves in local source tracking #574

Detect file moves in local source tracking #574

Conversation

iowillhoit commented May 3, 2024 • edited by mshanemc

What does this PR do?

What issues does this PR fix or reference?

mshanemc May 8, 2024

Choose a reason for hiding this comment

mshanemc May 8, 2024

Choose a reason for hiding this comment

iowillhoit May 8, 2024

Choose a reason for hiding this comment

mshanemc commented May 8, 2024 • edited

mshanemc commented May 14, 2024

mshanemc May 15, 2024 • edited

Choose a reason for hiding this comment

mshanemc commented May 15, 2024

iowillhoit commented May 3, 2024 •

edited by mshanemc

mshanemc commented May 8, 2024 •

edited

mshanemc May 15, 2024 •

edited