Dependencies.swift resolution improvements

[danieleformichelli]

The current implementation of tuist dependencies fetch command does:

1. create a temporary directory
2. copy the Tuist/Dependencies/SwiftPackageManager folder in the .build subfodler of the temporary directory
3. copy the Tuist/Dependencies/Lockfiles/Package.resolved file in the temporary directory
4. create the Package.swift file in the temporary directory
5. Run swift package resolve in the temporary folder
6. copy the .build subfodler of the temporary directory in the Tuist/Dependencies/SwiftPackageManager folder
7. copy the Package.resolved from the temporary directory to the Tuist/Dependencies/Lockfiles folder
8. map the SPM to a tuist graph

Ignoring step 7, which is not affected by this proposal, after the first tuist dependencies fetch, the subsequent ones take more than 1 minute in our project, and most of the time is spent moving around the SPM resolve output (points 2 and 6), which in our case is about 1.5GB, but I guess can be also bigger in other cases.

I have done a quick test refactoring the logic as:

1. copy the Tuist/Dependencies/Lockfiles/Package.resolved file in the Tuist/Dependencies/SwiftPackageManager directory
2. Run swift package resolve in the Tuist/Dependencies/SwiftPackageManager directory
3. copy the Package.resolved from theTuist/Dependencies/SwiftPackageManager directory to the Tuist/Dependencies/Lockfiles folder

This way, the command is much faster, and takes about 1 second.

Pros:

• commands is much faster
  Cons:
• if tuist dependencies fetch it fails halfway, the contents of dependencies directory will be in a weird non-functioning state -> if you are running a tuist dependencies fetch it probably means that what you have in the directory is no longer what you want. In any case, such a performance improvement would probably be worth anyway

Open points:

1. Do we keep Package.swift and Package.resolved in the Tuist/Dependencies/SwiftPackageManager folder? They don't harm and might be useful for debugging purposes
2. Do we offer a command to clean the Tuist/Dependencies folder? Options would be :

• support a new category tuist clean dependencies, which would clean the local dependencies folder -> it will be slightly different from all other tuist clean categories as it would be the only one acting on the local folder
• add a tuist dependencies clean subcommand, it would be probably easier to discover for people playing with tuist dependencies for the first time

[danieleformichelli]

PR implementing the change, using tuist clean dependencies for the clean part: #3418

[laxmorek]

Hey @danyf90 it is a very very good idea!

My thoughts:

1. I think we should mimic this improvement for Carthage as well.
2. What do you think about making something like tuist dependencies clean instead of tuist clean dependencies ?

if tuist dependencies fetch it fails halfway, the contents of dependencies directory will be in a weird non-functioning state

I think we could print to console suggestion that user should call tuist dependencies clean (or tuist clean dependencies) in order to fix the Dependencies/ directory. (I know it may be a bad UX, we can think about it more and try to find something better).

[danieleformichelli]

I agree we should do the same for Carthage, and that tuist dependencies clean instead of tuist clean dependencies might be a better fit 🙌

[pepicrft]

I'd suggest doing tuist clean dependencies. The main reason is that tuist clean is a namespace already known by the users. Being consistent with that will make it easier for users to find where the cleaning subcommand is, removing thus any cognitive load of having to learn that there's a new clean command nested elsewhere.

[danieleformichelli]

Makes sense, let's stick to tuist clean dependencies then!

[pepicrft]

I like the idea. Tuist has to be fast, and therefore any proposal to improve the performance is well received. My only concern regarding the approach is that if the process receives a kill signal, the dependencies directory might end up with intermediate artifacts. I think we should detect that scenario somehow and make sure we clean that, maybe the next time we run the command? What about adding a lockfile in that directory that tells Tuist: hey! I was doing something here, but the operation was terminated and therefore I did not delete the lockfile. Therefore tell the user they need to run the command again.

[danieleformichelli]

I agree and I like the idea, not sure if it is in the scope of this RFC/PR, and I would rather tackle it separately, as the same might make sense for other tuist commands, or for tuist in general

[danieleformichelli]

As discussed, I'll open a separate RFC for this 🙌

[pepicrft]

Do we keep Package.swift and Package.resolved in the Tuist/Dependencies/SwiftPackageManager folder? They don't harm and might be useful for debugging purposes

I can't think of any side effects in doing so. What about adding a note to the files to make it explicit that we don't expect users to modify those files? Just in case developers see those files there and assume they can make changes there directly?

[danieleformichelli]

mmm I see your point and I agree it might be confusing, maybe I would just remove the files then

[laxmorek]

same for Cartfile

[luispadron]

Thanks for the RFC! Agree that improving performance on fetch is crucial, since SPM resolution is already pretty slow.

In terms of where to copy files Id say sticking to whatever each package manager does by default is probably best. One odd thing i've noticed is the .package.resolved file tuist creates for the non dependencies version of the package support. Id like to see us keep these file names standard so they can work as easily with the 3rd party tools.

So in this case i'd say we should probably move away from the temp directory and let SPM resolve to the standard .build directory of the current project folder.

I don't have too much context on how we're doing dependencies now as the project has evolved a lot, so i trust you all to come up with a great solution

[danieleformichelli]

Yep, that's exactly what this proposal an PR does:
BEFORE:

• resolution in temporary folder
• copy tmp/.build in Tuist/Dependencies

AFTER:

• resolution in Tuist/Dependencies
• resolved files will end up in Tuist/Dependencies/.build

[luispadron]

@danyf90 Awesome! One question i have remaining is: Why store the .build in Tuist/Dependencies vs. letting SPM decide at the root of the project? Is there a reason we need to do it? Just wondering because some users might rely on the .build being in the standard location

[danieleformichelli]

Not sure I got your question, the .build folder is generated by SPM in the folder you run the swift package resolve command.
In this case, we generate the Package.swift in Tuist/Dependencies/SwiftPackageManager and then we run swift package resolve in that folder, which ends up having .build in Tuist/Dependencies/SwiftPackageManager/.build