Issue #2400 Shared cache for built dependencies #2716

kenji21 · 2019-03-01T14:58:57Z

see discussion in issue #2400

nikolaeu · 2019-03-03T10:58:46Z

Nice one @kenji21! How about some details in readme.md?

tmspzz · 2019-03-04T15:18:57Z

LGTM.

My only concern is that this would be enabled by default.

How does @Carthage/carthage feel about it?

killobatt · 2019-03-05T15:30:54Z

Hello. Merging of this PR can reduce our clean/build time from 22minutes to 5-6 minutes, our team really waits for it.
It there something we can do to help it be merged faster?

joshuapoq · 2019-03-05T18:40:06Z

For clarity, (as I understand it) this allows Carthage to share already built dependencies across projects.

We have a workspace that has Y many targets across many projects that all depend downwards on X number of common lower level targets. Am I correct in thinking this will now only build all X of these common frameworks once instead of X * Y times?

kenji21 · 2019-03-06T07:56:07Z

For clarity, (as I understand it) this allows Carthage to share already built dependencies across projects.

We have a workspace that has Y many targets across many projects that all depend downwards on X number of common lower level targets. Am I correct in thinking this will now only build all X of these common frameworks once instead of X * Y times?

That's the point, here is how @blender nicely recap it : #2400 (comment)

Jeehut · 2019-04-03T11:10:30Z

Anybody coming across this issue nowadays, please checkout my comment here.

tmspzz · 2019-04-05T09:23:10Z

@mdiep ping for review :)

nikolaeu · 2019-04-05T10:10:54Z

@mdiep double ping ;)

Source/CarthageKit/Project.swift

Source/CarthageKit/Xcode.swift

Source/CarthageKit/Project.swift

mdiep · 2019-04-08T12:54:50Z

Looks pretty good overall, but I think we need a couple more items.

kenji21 · 2019-04-11T09:56:44Z

Hi, I updated code responding to reviews, and rebased my feature branch on top of current master, after a make install on my machine:

3m1.871s for initial carthage bootstrap, 0m15.297s for the second one

Full log:

time carthage bootstrap --no-use-binaries --cache-builds --platform iOS
*** Checking out Archimedes at "1.1.5"
*** Checking out AlamofireObjectMapper at "5.2.0"
*** Checking out ObjectMapper at "3.4.2"
*** Checking out GCDWebServer at "3.5.2"
*** Checking out SwiftiumKit at "v1.4.0"
*** Checking out Alamofire at "4.8.1"
*** No cache found for Alamofire, building with all downstream dependencies
*** No cache found for Archimedes, building with all downstream dependencies
*** No cache found for GCDWebServer, building with all downstream dependencies
*** No cache found for ObjectMapper, building with all downstream dependencies
*** No cache found for SwiftiumKit, building with all downstream dependencies
*** xcodebuild output can be found in /var/folders/7v/dfg0p2vj1z58n65d99l2n1lh0000gp/T/carthage-xcodebuild.9Ea0OC.log
*** Building scheme "Alamofire iOS" in Alamofire.xcworkspace
*** Building scheme "Archimedes iOS" in Archimedes.xcworkspace
*** Building scheme "GCDWebServers (iOS)" in GCDWebServer.xcodeproj
*** Building scheme "ObjectMapper-iOS" in ObjectMapper.xcworkspace
*** Building scheme "AlamofireObjectMapper iOS" in AlamofireObjectMapper.xcworkspace
*** Building scheme "SwiftiumKit" in SwiftiumKit.xcodeproj

real	3m1.871s
user	1m48.996s
sys	0m32.073s
kenji@macboo:~/Desktop/cart-issue-2400 [master] $
rm -Rf Carthage/Build/
kenji@macboo:~/Desktop/cart-issue-2400 [master] $
time carthage bootstrap --no-use-binaries --cache-builds --platform iOS
*** Checking out Alamofire at "4.8.1"
*** Checking out GCDWebServer at "3.5.2"
*** Checking out Archimedes at "1.1.5"
*** Checking out SwiftiumKit at "v1.4.0"
*** Checking out AlamofireObjectMapper at "5.2.0"
*** Checking out ObjectMapper at "3.4.2"
*** No cache found for Alamofire, building with all downstream dependencies
*** No cache found for Archimedes, building with all downstream dependencies
*** No cache found for GCDWebServer, building with all downstream dependencies
*** No cache found for ObjectMapper, building with all downstream dependencies
*** No cache found for SwiftiumKit, building with all downstream dependencies
*** Copying from shared cache for Alamofire, skipping build
*** Copying from shared cache for Archimedes, skipping build
*** Copying from shared cache for GCDWebServer, skipping build
*** xcodebuild output can be found in /var/folders/7v/dfg0p2vj1z58n65d99l2n1lh0000gp/T/carthage-xcodebuild.dMgJ11.log
*** Copying from shared cache for ObjectMapper, skipping build
*** Copying from shared cache for AlamofireObjectMapper, skipping build
*** Copying from shared cache for SwiftiumKit, skipping build

real	0m15.297s
user	0m4.562s
sys	0m3.595s

Source/CarthageKit/Xcode.swift

stale · 2019-05-12T15:22:27Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

kenji21 · 2019-05-13T06:30:34Z

Oups, didn't saw that CI was failing, I will update my commit again

werner77 · 2019-05-13T12:41:46Z

Source/CarthageKit/Xcode.swift

+/// ~/Library/Caches/org.carthage.CarthageKit/SharedCache/9.4.1_9F2000/SwiftyUserDefaults/3.0.1
+internal func sharedCacheURLByXcodeVersionFor(dependency: Dependency, pinnedVersion: PinnedVersion) -> URL {
+	let sharedCacheURL = Constants.userCachesURL.appendingPathComponent("SharedCache")
+	let sharedCachePerXcode = sharedCacheURL.appendingPathComponent(XcodeVersion.makeString(), isDirectory: true)


I think you'll want to use the swift toolchain version, not the xcode version. You could switch toolchains within xcode leading to incompatible builds.

you're right, i've replaced Xcode version with the toolchain one if it is set (kept XcodeVersion if no toolchain set in BuildOptions)

new version visible here : 23d433f#diff-f66644e8cfbcad3c3826395a690ddfbcR12

Why not just use the xcrun swift --version like other places in the code? Even if not set explicitly (default) it will yield the correct swift toolchain version used.

kenji21 · 2019-05-23T15:25:36Z

So two commits behind master, current tests are fine
Executed 236 tests, with 0 failures (0 unexpected) in 672.691 (672.723) seconds

kenji21 · 2019-05-23T15:43:10Z

rebased to current master

kenji21 · 2019-05-24T06:48:39Z

The reason I don't want to zip / unzip is that APFS makes copy almost instantly, so when a dependency at is compiled at version X.Y with swift version Y.Z for one project, it is available for another project using the same dependency (let's say Alamofire, which can be in many project for example)

The other aim of the "shared cache" is to speed up Continuous Integration "clean" builds, by not having to recompile again and again dependencies that are not changing (see results in comment : #2400 (comment))

we discussed this in issue #2400 specifically well defined in this comment: #2400 (comment)

tmspzz · 2019-05-29T10:42:53Z

@jdhealy do you want to give an opinion? I'll be reviewing again ASAP.

DerFlash · 2019-07-04T17:43:09Z

This PR fails to build against official Carthage master because it's been updated for Swift 5.1 already.

.../Source/CarthageKit/Xcode.swift:972:12: error: ambiguous use of 'then'
                                return copyToSharedCache.then(
                                       ^
ReactiveSwift.SignalProducer<τ_0_0, τ_0_1>:5:17: note: found this candidate
    public func then<U>(_ replacement: ReactiveSwift.SignalProducer<U, Result.NoError>) -> ReactiveSwift.SignalProducer<U, Error>
                ^
ReactiveSwift.SignalProducer<τ_0_0, τ_0_1>:6:17: note: found this candidate
    public func then<Replacement>(_ replacement: Replacement) -> ReactiveSwift.SignalProducer<Replacement.Value, Error> where Replacement : SignalProducerConvertible, Replacement.Error == Result.NoError
                ^
ReactiveSwift.SignalProducer<τ_0_0, τ_0_1>:7:17: note: found this candidate
    public func then<U>(_ replacement: ReactiveSwift.SignalProducer<U, Error>) -> ReactiveSwift.SignalProducer<U, Error>
                ^
ReactiveSwift.SignalProducer<τ_0_0, τ_0_1>:8:17: note: found this candidate
    public func then<Replacement>(_ replacement: Replacement) -> ReactiveSwift.SignalProducer<Replacement.Value, Error> where Error == Replacement.Error, Replacement : SignalProducerConvertible
                ^

make: *** [installables] Error 1

Anyway, thanks for the good work. Build against Xcode 10.1 straight from the PR branch. Works like a charm!

Regards
Björn

kenji21 · 2019-07-11T11:28:35Z

it seems that using Xcode 10.2.1 wasn't a good idea... CI is down...

kenji21 · 2019-07-30T20:43:37Z

updated to current master, running SWIFTPM_BOOTSTRAP=true make test on my macbook :

	 Executed 236 tests, with 0 failures (0 unexpected) in 896.945 (896.963) seconds

DerFlash · 2019-12-12T14:00:52Z

PR seems to be removed from the roadmap again. Any reason why?

joshuapoq · 2020-01-10T09:37:15Z

Any chance of a rereview @mdiep @ikesyo? I’ve been watching this PR for years now as it will save weeks if not months of CI and developer time for our SDK.

kenji21 mentioned this pull request Mar 1, 2019

Carthage doesn't cache build across projects (or CI builds) #2400

Open

tmspzz self-requested a review March 4, 2019 15:19

tmspzz approved these changes Mar 4, 2019

View reviewed changes

jdhealy self-requested a review March 4, 2019 15:21

mdiep self-requested a review March 7, 2019 13:28

Jeehut mentioned this pull request Apr 3, 2019

Exclude specific schemes via Cartfile.ignore #1990

Closed

2 tasks

mdiep requested changes Apr 8, 2019

View reviewed changes

kenji21 force-pushed the feature-2400-shared-cache branch 3 times, most recently from 02cdd14 to a14f503 Compare April 11, 2019 09:26

ikesyo requested changes Apr 11, 2019

View reviewed changes

Source/CarthageKit/Xcode.swift Outdated Show resolved Hide resolved

kenji21 force-pushed the feature-2400-shared-cache branch from a14f503 to a6eb330 Compare April 12, 2019 15:11

stale bot added the stale label May 12, 2019

ikesyo added enhancement and removed stale labels May 12, 2019

werner77 reviewed May 13, 2019

View reviewed changes

kenji21 force-pushed the feature-2400-shared-cache branch 2 times, most recently from 9c21079 to 23d433f Compare May 23, 2019 15:07

kenji21 force-pushed the feature-2400-shared-cache branch 2 times, most recently from 5c60486 to 4f2f56b Compare May 23, 2019 15:42

kenji21 force-pushed the feature-2400-shared-cache branch from 4f2f56b to e9a33a6 Compare July 11, 2019 07:56

kenji21 force-pushed the feature-2400-shared-cache branch from e9a33a6 to 8e548ca Compare July 30, 2019 20:42

kenji21 force-pushed the feature-2400-shared-cache branch from 8e548ca to 0570cb8 Compare September 30, 2019 06:57

kenji21 requested review from ikesyo and mdiep September 30, 2019 07:49

tmspzz mentioned this pull request Oct 10, 2019

Roadmap Q4 2019 - Q1 2020 #2890

Closed

kenji21 force-pushed the feature-2400-shared-cache branch 2 times, most recently from 774b756 to fe1e041 Compare June 19, 2020 14:19

Issue Carthage#2400 Shared cache for built dependencies

f61f66e

kenji21 force-pushed the feature-2400-shared-cache branch from fe1e041 to f61f66e Compare June 19, 2020 14:55

kenji21 mentioned this pull request Aug 19, 2020

Issue #2400 Shared cache for built dependencies #2608

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue #2400 Shared cache for built dependencies #2716

Issue #2400 Shared cache for built dependencies #2716

kenji21 commented Mar 1, 2019

nikolaeu commented Mar 3, 2019

tmspzz commented Mar 4, 2019

killobatt commented Mar 5, 2019

joshuapoq commented Mar 5, 2019 •

edited

kenji21 commented Mar 6, 2019

Jeehut commented Apr 3, 2019

tmspzz commented Apr 5, 2019

nikolaeu commented Apr 5, 2019

mdiep commented Apr 8, 2019

kenji21 commented Apr 11, 2019

stale bot commented May 12, 2019

kenji21 commented May 13, 2019

werner77 May 13, 2019

kenji21 May 23, 2019

kenji21 May 23, 2019

werner77 May 23, 2019

kenji21 commented May 23, 2019

kenji21 commented May 23, 2019

kenji21 commented May 24, 2019

tmspzz commented May 29, 2019

DerFlash commented Jul 4, 2019

kenji21 commented Jul 11, 2019

kenji21 commented Jul 30, 2019

DerFlash commented Dec 12, 2019

joshuapoq commented Jan 10, 2020

Issue #2400 Shared cache for built dependencies #2716

Are you sure you want to change the base?

Issue #2400 Shared cache for built dependencies #2716

Conversation

kenji21 commented Mar 1, 2019

nikolaeu commented Mar 3, 2019

tmspzz commented Mar 4, 2019

killobatt commented Mar 5, 2019

joshuapoq commented Mar 5, 2019 • edited

kenji21 commented Mar 6, 2019

Jeehut commented Apr 3, 2019

tmspzz commented Apr 5, 2019

nikolaeu commented Apr 5, 2019

mdiep commented Apr 8, 2019

kenji21 commented Apr 11, 2019

stale bot commented May 12, 2019

kenji21 commented May 13, 2019

werner77 May 13, 2019

Choose a reason for hiding this comment

kenji21 May 23, 2019

Choose a reason for hiding this comment

kenji21 May 23, 2019

Choose a reason for hiding this comment

werner77 May 23, 2019

Choose a reason for hiding this comment

kenji21 commented May 23, 2019

kenji21 commented May 23, 2019

kenji21 commented May 24, 2019

tmspzz commented May 29, 2019

DerFlash commented Jul 4, 2019

kenji21 commented Jul 11, 2019

kenji21 commented Jul 30, 2019

DerFlash commented Dec 12, 2019

joshuapoq commented Jan 10, 2020

joshuapoq commented Mar 5, 2019 •

edited