Carthage doesn't cache build across projects (or CI builds)

[kenji21]

• carthage install method: [ ] .pkg, [X] homebrew, [ ] source
• which carthage: /usr/local/bin/carthage
• carthage version: 0.29.0
• xcodebuild -version: Xcode 9.3 Build version 9E145
• Are you using --no-build? no
• Are you using --no-use-binaries? yes
• Are you using --use-submodules? no
• Are you using --cache-builds? yes
• Are you using --new-resolver? no

Cartfile

github "SVProgressHUD/SVProgressHUD" ~> 2.0

Carthage Output

10:35:30 kenji@macboo:~/Desktop/CarthageCacheTest
cat Cartfile
github "SVProgressHUD/SVProgressHUD" ~> 2.0

10:35:33 kenji@macboo:~/Desktop/CarthageCacheTest
cat Cartfile.resolved
github "SVProgressHUD/SVProgressHUD" "2.2.4"

10:35:34 kenji@macboo:~/Desktop/CarthageCacheTest
rm -Rf Carthage ; time carthage bootstrap --no-use-binaries --cache-builds --platform iOS
*** Checking out SVProgressHUD at "2.2.4"
*** No cache found for SVProgressHUD, building with all downstream dependencies
*** xcodebuild output can be found in /var/folders/7v/dfg0p2vj1z58n65d99l2n1lh0000gp/T/carthage-xcodebuild.5CyxXz.log
*** Building scheme "SVProgressHUD-Framework" in SVProgressHUD.xcodeproj

real	0m25.314s
user	0m24.218s
sys	0m7.705s

10:36:21 kenji@macboo:~/Desktop/CarthageCacheTest
ls -l /Users/kenji/Library/Caches/org.carthage.CarthageKit/DerivedData/9.3_9E145/SVProgressHUD/
total 0
drwxrwxr-x@ 6 kenji  staff  192 Apr  1 10:36 2.2.4
drwxrwxr-x@ 6 kenji  staff  192 Apr  1 10:19 2.2.5

10:36:28 kenji@macboo:~/Desktop/CarthageCacheTest
rm -Rf Carthage ; time carthage bootstrap --no-use-binaries --cache-builds --platform iOS
*** Checking out SVProgressHUD at "2.2.4"
*** No cache found for SVProgressHUD, building with all downstream dependencies
*** xcodebuild output can be found in /var/folders/7v/dfg0p2vj1z58n65d99l2n1lh0000gp/T/carthage-xcodebuild.fpieKP.log
*** Building scheme "SVProgressHUD-Framework" in SVProgressHUD.xcodeproj

real	0m25.840s
user	0m25.477s
sys	0m6.808s

Actual outcome
Carthage did not use cached build if I rm the Carthage directory of my working copy (what our CI does before each build)

Expected outcome
Carthage should use the previously compiled framework stored in ~/Library/Caches/org.carthage.CarthageKit/DerivedData when running the second bootstrap command

I used SVProgressHUD here to have a "quickly recompiled" framework, but it is also the other dependencies (and our CI jobs are taking ages to recompile each time all frameworks)

[tmspzz]

Thanks for reporting, I'm not sure this behavior is expected. Further investigation is needed.

@kenji21 for time being you this might help you.

[mdiep]

This is the expected behavior. --cache-builds caches build products, but does not alter the behavior of using a clean vs. a dirty build.

[kenji21]

@blender, thanks for pointing me to Rome, this is an interesting tool to share builds via the CI server (I'll need to think a bit more about what to do with this tool)

Usage of submodules have the same behavior :

$ carthage update --no-use-binaries --cache-builds --platform iOS --use-submodules
$ rm -Rf Carthage/Build/ ; time carthage bootstrap --no-use-binaries --cache-builds --platform iOS
*** Checking out SVProgressHUD at "2.2.5"
*** No cache found for SVProgressHUD, building with all downstream dependencies
*** xcodebuild output can be found in /var/folders/7v/dfg0p2vj1z58n65d99l2n1lh0000gp/T/carthage-xcodebuild.HYI0Hr.log
*** Building scheme "SVProgressHUD-Framework" in SVProgressHUD.xcodeproj

it is the Build directory which is "key" to cache (deleting this folder or the framework subfolder lead to a "Invalid cache found for SVProgressHUD, rebuilding with all downstream dependencies")

[olejnjak]

@kenji21 on our Jenkins I do something similar like Rome, but less sophisticated, when Jenkins workspace is wiped, then I look into predefined cache directory (e.g. ~/Library/Caches/ci/<project_name>) and then sync it to the workspace. After dependency artifacts are built, I sync them back to the cache directory.

[kenji21]

I like the way Carthage caches (e.g. using Xcode/swift version and dependency version in the path), does a PR adding an option like --use-cached-builds (only on bootstrap command, and why not on update, let's say when 2 projects have similar dependencies and are updated the same day) has a chance to be merged @mdiep ? I think it can because it explicitly shows that it will use cached build (if available, reusing the ProjectEvent.buildingUncached if not already compiled)

[mdiep]

I'm not sure what you're asking.

[olejnjak]

What I see in this is that @kenji21 would like to cache build artifacts in ~/Library/Caches/<carthagekit>. And the question would be if it makes any sense to you 🤷‍♂️

[kenji21]

With the new Xcode / swift version (4.1), we have to "bootstrap or update" our dependencies on different projects having common dependencies (Alamofire, RxSwift...), same could occurs if Alamofire post a security update

I want to share carthage builds across projects (or when runing CI builds with Carthage/Build not checked in, which have a simpler way to reproduce the cache usage / non-usage)

[kenji21]

One more use case: if a compilation fails during update (let's say the 7th out of 10), relaunching carthage update will recompile all dependencies, with --use-cached-builds it won't be the case and fixing the failing depency will be easier (real use case as it happened to me with a swiftlint script phase making a xcodebuild command fail)

[ikesyo]

We does not specify clean action explicitly but Xcode may silently do clean build for archive action ( carthage uses archive action for device builds).

https://stackoverflow.com/questions/46211606/xcodebuild-archive-always-performs-a-clean-build

So this will not be an our fault I think.

[kenji21]

I think I understand what carthage means by "Valid cache":

carthage bootstrap --no-use-binaries --cache-builds --platform iOS
*** Checking out SVProgressHUD at "2.2.5"
*** xcodebuild output can be found in /var/folders/7v/dfg0p2vj1z58n65d99l2n1lh0000gp/T/carthage-xcodebuild.MYO3mI.log
*** Valid cache found for SVProgressHUD, skipping build

It is the $PWD/Carthage/Build/iOS/SVProgressHUD.framework*** files that is called "cache" by carthage, I was thinking of cache to be located at ~/Library/Caches/org.carthage.CarthageKit/DerivedData/9.3_9E145/SVProgressHUD/2.2.5/

[ikesyo]

Yes, since you are deleting Carthage directory at all, Carthage can't find any cached builds. So this is the expected behavior.

[kenji21]

So the aim of the PR adding option --use-cached-builds will be to copy ~/Library/Caches/org.carthage.CarthageKit/DerivedData/9.3_9E145/SVProgressHUD/2.2.5/....framework + dSYM to $PWD/Carthage/Build/iOS/ instead of running xcodebuild archive

As I assume that --cache-builds do the opposite (copy the builded framework + dSYM to ~/Library/Caches/Carthage...)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[kenji21]

Am I right when I say:

I assume that --cache-builds do the opposite (copy the builded framework + dSYM to ~/Library/Caches/Carthage...)

I still like to have the --use-cached-builds (e.g. make a PR) in comment #2400 (comment) but I don't want to spend time on such a "non-mergeable request" PR

[olejnjak]

Well, I recently switched to Rome and now I think that such PR would be redundant as it is much more powerful than behavior described above 🤷‍♂️

[kenji21]

To sum up:

--cache-build doesn't use files from ~/Library/Caches/org.carthage.CarthageKit/DerivedData/, but only compares .version files in $PWD/Carthage/Builds (

Carthage/Source/CarthageKit/Project.swift

Lines 1016 to 1026 in 0ec71e1

	guard options.cacheBuilds && projects.isDisjoint(with: projectsToBeBuilt) else {
	return dependenciesIncludingNext
	}
	guard let versionFileMatches = matches else {
	self._projectEventsObserver.send(value: .buildingUncached(nextDependency.0))
	return dependenciesIncludingNext
	}
	if versionFileMatches {
	self._projectEventsObserver.send(value: .skippedBuildingCached(nextDependency.0))

)

When building, the DerivedData path used is generated:

Carthage/Source/CarthageKit/Project.swift

Lines 1067 to 1071 in 0ec71e1

	let baseURL = options.derivedDataPath.flatMap(URL.init(string:)) ?? Constants.Dependency.derivedDataURL
	let derivedDataPerXcode = baseURL.appendingPathComponent(self.xcodeVersionDirectory, isDirectory: true)
	let derivedDataPerDependency = derivedDataPerXcode.appendingPathComponent(dependency.name, isDirectory: true)
	let derivedDataVersioned = derivedDataPerDependency.appendingPathComponent(version.commitish, isDirectory: true)
	options.derivedDataPath = derivedDataVersioned.resolvingSymlinksInPath().path

As xcodebuild archive makes a clean build, even specifying same DerivedData path make all projects being rebuild from scratch...

So, you're right @olejnjak, my needs are what Rome provides, but as I only want to keep it "local" (no need for S3, minio or ceph), I was thinking by error that Carthage with --cache-builds provides this feature, as carthage help bootstrap (with latest 0.30.1) states:

[--cache-builds]
	use cached builds when possible

My bad here, because I haven't RTFM enough the README.md#caching-builds (maybe a link to this can be added to the help?)

And Rome doesn't know which Xcode (so, swift version) build your framework, so we should always use [--cache-prefix](https://github.com/blender/Rome#cache-prefix) $(xcodebuild -version | xargs | sed -E 's# #_#g') to prevent, (Carthage know this). And Rome doesn't fix the issue of #issuecomment-378474277 with the way it tell us to update (using xargs to tell carthage to rebuild all missing one in one step, it should be done in a shell while loop)

I still think that a local cache can easily be included within Carthage, maybe copying built frameworks/dSYMs in a directory, with same path generation as DerivedData, but prefixed by SharedCache instead of DerivedData, plus platform-commitish of .version, with a new option like --share-builds which doesn't interfere with dev workflow, and makes frameworks not being recompiled for a single platform if already in the "SharedCache"

Exemple:

$ carthage bootstrap --no-use-binaries --cache-builds --platform iOS
*** No Cartfile.resolved found, updating dependencies
*** Fetching SVProgressHUD
*** Checking out SVProgressHUD at "2.2.5"
*** No cache found for SVProgressHUD, building with all downstream dependencies
*** xcodebuild output can be found in /var/folders/7v/dfg0p2vj1z58n65d99l2n1lh0000gp/T/carthage-xcodebuild.JzLTs2.log
*** Building scheme "SVProgressHUD-Framework" in SVProgressHUD.xcodeproj
$ ls Carthage/Build/iOS/
BBE05204-6206-30DE-B075-9009DFB73858.bcsymbolmap  FF12DFCB-55B3-38C3-BDBB-F1CF41A02B7E.bcsymbolmap  SVProgressHUD.framework/                          SVProgressHUD.framework.dSYM/
$ ls ~/Library/Caches/org.carthage.CarthageKit/SharedCache/9.3_9E145/SVProgressHUD/2.2.5/iOS/
BBE05204-6206-30DE-B075-9009DFB73858.bcsymbolmap FF12DFCB-55B3-38C3-BDBB-F1CF41A02B7E.bcsymbolmap SVProgressHUD.framework                          SVProgressHUD.framework.dSYM

Important command here is : ls ~/Library/Caches/org.carthage.CarthageKit/SharedCache/9.3_9E145/SVProgressHUD/2.2.5/iOS/

Second "clean" bootstrap (as jenkins would do):

$ rm -Rf Carthage
$ carthage bootstrap --no-use-binaries --cache-builds --platform iOS
*** Checking out SVProgressHUD at "2.2.5"
*** xcodebuild output can be found in /var/folders/7v/dfg0p2vj1z58n65d99l2n1lh0000gp/T/carthage-xcodebuild.HYzyWv.log
*** Valid shared cache found for SVProgressHUD, skipping build

And all files from ~/Library/Caches/org.carthage.CarthageKit/SharedCache/9.3_9E145/SVProgressHUD/2.2.5/ have been copied to $PWD/Carthage/Build/ (depends on --platform filter)

[tmspzz]

--cache-build doesn't use files from ~/Library/Caches/org.carthage.CarthageKit/DerivedData/, but only compares .version files in $PWD/Carthage/Builds

Correct

my needs are what Rome provides, but as I only want to keep it "local"

You can only specify a local cache to keep it local. You don't need to provide S3 or other caches.

#issuecomment-378474277

Can you fix this link please so that I can read the comment?

[kenji21]

Fixed, I was thinking anchor link on "local page" was working

[tmspzz]

From #2400 (comment)

carthage update will recompile all dependencies

It won't if you use --cache-builds. That's the point of it.

This case is already covered. Your problem is a cold bootstrap.

From #2400 (comment)

And Rome doesn't know which Xcode (so, swift version) build your framework, so we should always use --cache-prefix $(xcodebuild -version | xargs | sed -E 's# #_#g') to prevent, (Carthage know this)

Carthage doesn't know either, it uses the current xcode-selected version.

I'll follow up with another comment formalizing your problem.

[tmspzz]

So, to formalize a little for @mdiep

Problem

• Project A uses carthage
• Project B uses carthage
• Project A and B don't depend on each other, they are separate, for example two distinct apps
• Project A and B happen to have a common dependency, Framework C (same version in both projects)

When running carthage boostrap on Project A, Framework C is built.
When running carthage boostrap on Project B, Framework C is built.

Current Status

• Carthage does not share Framework C across boostraps in different projects
• Extra tools are needed to share frameworks across boostraps like Rome
• Carthage only provides a cache for rebuilds in the same project if bootstrap or update are called with. --cache-builds

Improvement Suggestion

Add a feature to share artefacts across boostraps (and updates)

--cache-builds is still relevant and needed for binary artifacts.

Implementation

• Introduce a new flag --share-builds to active the feature
• When the flag is active, check a known location on disk (cache) for previous artifacts
• Copy artifacts from the known location

About the bootstrap cache

• Location: ~/Library/Caches/org.carthage.CarthageKit/SharedCache
• Structure: <xcode-version>/<repository-name>/<comittish-or-tag>/<platform>/

[tmspzz]

My personal comment:

• What you need is already covered by the extra tool (the cache structure is almost the same), however I can see how this can be a nice addition
• I'm not a big fan of adding yet another flag to the command line. It's becoming a jungle imho.
• I'm worried about the confusion between this and --cache-builds, so I would rather improve that instead to achieve what is proposed here

[ikesyo]

This would be a duplicate of #1588.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[kenji21]

Updated the PR to current master : #2608

[kenji21]

Updated the PR again as Xcode 10.1 is out (and our CI was failing building dependencies with the simulator not found issue, so I have run it with PR #2631):

$ ls -1 ~/Library/Caches/org.carthage.CarthageKit/SharedCache/
10.0_10A255
10.1_10B61
9.4.1_9F2000
$ ls -1 ~/Library/Caches/org.carthage.CarthageKit/SharedCache/10.1_10B61/Alamofire
4.7.3

[kenji21]

Here is an example of CI clean build times (on an iMac Pro) with a project having 17 "first level" dependencies (RxSwift, AlamofireObjectMapper...):

carthage 0.31.2 with the "SharedCache" :
ios-app - #646 Success after 3 min 11 sec (Open)
ios-app - #647 Success after 3 min 15 sec (Open)

carthage 0.32:
ios-app - #648 Success after 15 min (Open)

[kenji21]

Created a new PR (#2716) since I moved my commit to a branch (in order to send another PR fixing issue #2677)

[Jeehut]

Anybody coming across this issue nowadays, please checkout my comment here.

[kenji21]

As 0.35 is out, updated the PR #2716 for this issue:

rm -Rf Carthage/Build ; time carthage bootstrap --no-use-binaries --cache-builds --platform iOS
*** Checking out SVProgressHUD at "2.2.4"
*** No cache found for SVProgressHUD, building with all downstream dependencies
*** xcodebuild output can be found in /var/folders/7v/dfg0p2vj1z58n65d99l2n1lh0000gp/T/carthage-xcodebuild.OMZOeA.log
*** Building scheme "SVProgressHUD-Framework" in SVProgressHUD.xcodeproj

real	0m20.356s
user	0m12.260s
sys	0m4.874s

Second run:

rm -Rf Carthage/Build ; time carthage bootstrap --no-use-binaries --cache-builds --platform iOS
*** Checking out SVProgressHUD at "2.2.4"
*** No cache found for SVProgressHUD, building with all downstream dependencies
*** xcodebuild output can be found in /var/folders/7v/dfg0p2vj1z58n65d99l2n1lh0000gp/T/carthage-xcodebuild.BPoHAd.log
*** Copying from shared cache for SVProgressHUD, skipping build

real	0m0.392s
user	0m0.230s
sys	0m0.172s

Executed 265 tests, with 0 failures (0 unexpected) in 677.467 (677.479) seconds
``