Build caching

[rstacruz]

Is it possible to make subsequent builds faster?

I have a project that takes 3 minutes to compile (mostly sass -> autoprefixer -> postcss -> css). Every time I deploy, that's 3 minutes just for brunch compilation! I'd like to be able to cache builds of sass files that haven't changed.

Normally, watcher makes instantaneous rebuilds, but brunch build --production will rebuild everything from scratch.

[paulmillr]

I think an optional build cache would be something good to have. Why limit ourselves to non-production mode though? If the build process is smart enough, one cache always means one thing.

[rstacruz]

That would be fantastic.

[rstacruz]

How can I help make this happen?

[paulmillr]

@rstacruz let's discuss the possible architecture here then. What's on your mind?

So, suppose we would have some kind of temporary dir for files.

1. Where would the dir be stored? Project dir or OS tmp dir?
2. Which steps of build process would require its own file? For example if we have compile -> lint -> optimize; would it mean we would serialize outputs for both compile and optimize steps?
3. Any other thoughts on the cache?

Once we answer those questions thoroughly, I think we can go ahead and implement it.

[rstacruz]

I haven't dug too far into brunch internals yet, sorry. But here's what I
know others do:

• Rails stores theirs in tmp/assets/
• Ruby Sass into .sass-cache
• browserify-incremental to ./browserify_cache.json (User configurable but
  they recommend something like that)

On Sat, Jun 4, 2016, 9:41 AM Paul Miller notifications@github.com wrote:

@rstacruz https://github.com/rstacruz let's discuss the possible
architecture here then. What's on your mind?

So, suppose we would have some kind of temporary dir for files.

1. Where would the dir be stored? Project dir or OS tmp dir?
2. Which steps of build process would require its own file? For
   example if we have compile -> lint -> optimize; would it mean we would
   serialize outputs for both compile and optimize steps?
3. Any other thoughts on the cache?

Once we answer those questions thoroughly, I think we can go ahead and
implement it.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
#1379 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAEikTdT2kOefkuqzR-D8A8S_YwkDHgTks5qINfLgaJpZM4InvpL
.

[paulmillr]

thoughts @goshakkk?

[goshacmd]

A few random thoughts: plugins can depend on environment, etc; also configs between dev and prod can be different, which means the caches won't necessarily be universal (take this — process.env.NODE_ENV can be different things depending on env) — which ultimately will mean dev caches won't be of much use for production builds.

(Another concern is not everything might be serializable?)

Also the versions of each plugin will have to be taken into account, in addition to exact env/config.

So we'll have to somehow account for this, and also have a way to prune caches no longer in use as to not clutter the fs.

As for which steps, I think it will also make sense to do that for file's dependencies resolved through compiler, as well as for every compiler/linter maybe?

So that what we cache is: [env, source] -> output for each file and compiler — so if we had a project wilt compilers A and B, linter C, and a file z.js, the following will be cached:

• [env, z,js] -> lint result
• [env, z.js] -> A result
• [env, A result] -> B result

If implemented like this, we could theoretically be able to change all calls to compiler methods to check the cache first, and if there is anything of interest — to load it straight away without going into the compiler — which should probably localize the scope of change to just plugins.js

Although it should be noted that not all compilers are pure, and some rely on plugin API methods to perform side effects (e.g. in compile).

Yet another thing to consider is how we're going to make that work for npm integration — could be the case that even with caching enabled, we'd still have to resolve the deps each and every time (because the locations of the deps could have changed, etc).

All in all, this is probably a complex feature to implement, that can potentially slow us down and make us lag behind if we want to add some fundamentally new features later on, so we should carefully evaluate our options and the implications.

[es128]

Prior discussion from very early days of my involvement in Brunch at #651. The conversation kind of devolved, but maybe there are still some worthwhile ideas that can be lifted from there if this will be pursued.