Build optimization

[jimafisk]

The biggest bottleneck currently is compiling components in V8. We're just using one giant VM for this, which makes builds very slow for larger sites and seems to have a memory leak that can timeout once you have a couple of thousand pages. It would be nice to make this process more consistent in general.

The initial issue on concurrent build steps has some interesting suggestions: #26

There are other smaller things that we could do to be more efficient as well. Originally posted in #191:

Currently we're performing EjectCopy separately from the Client build process where we prep ejected svelte files that need to get compiled. Seems redundant to walk these files twice, eject_copy should probably just be a func call during the Client build when non .svelte files are found. This would speed up the build process.

I placed the new --minify processing inside of Gopack for now. I didn't want to insert it before Gopack ran because it could screw up the regexp needed to update paths for ESM. Made me think the way we're running Gopack in general is redundant: we're re-reading files already written to the FS during the Client build, updating the content and re-writing them back to the filesystem. We could probably just do this when the files are still []byte after being compiled initially, before they are written to the filesystem. We could do the minification there as well. This would speed up the build process.

[jimafisk]

Discussion about leveraging Bud's Svelte compiler API: livebud/bud#353

[jimafisk]

the way we're running Gopack in general is redundant: we're re-reading files already written to the FS during the Client build, updating the content and re-writing them back to the filesystem

I'm working on inserting Gopack directly into the Svelte compilation step so it can update the import paths to be ESM ready before they are ever written to the filesystem. This will make builds faster, the reduction of read/writes alone should cut ~200ms from a basic site.

An advantage of doing this is we no longer would need to follow imports recursively and resolved them because every single Svelte file (even core svelte files) goes through our compiler and would get resolved at this step.

The challenge I'm running into is that when running Gopack on the built "public/spa" output, we're actually going the extra step and verifying the that import paths actually are resolvable on the FS, or else we throw an error. Running Gopack directly on the layouts source files causes this check to thrown tons of errors like:

Could not compile 'core/cms/file_upload.svelte' Svelte component: Could not add ESM support via Gopack: 
Import path './media_filters.js' not resolvable from file 'public/spa/core/cms/file_upload.js'

We could just remove this check and folks would hit the issue in the browser when it can't import, but it does seem less intuitive that way (you'd have to realize hydration failed, then chase down console errors).

Another consideration I was removing the alreadyConverted check which prevented trying to re-convert files in the case of circular dependencies, but we still need to be carefully not to continually re-copy node_modules from their source.

Build times with Gopack read/write

✗ redesign (master) plenti build -b
Setting up core NPM packages took 6.575µs
Copying ejectable core files for build took 1.112961ms
Copying static files into build dir took 558.675µs
Copying media files into build dir took 28.51263ms
Compiling client SPA with Svelte took 7.873396605s
Creating data_source took 252.66484ms
Running Gopack took 195.751049ms
Running Minification took 287.048154ms
Total build took 8.709126753s

✔︎ redesign (master) plenti build -b
Setting up core NPM packages took 8.995µs
Copying ejectable core files for build took 855.654µs
Copying static files into build dir took 652.603µs
Copying media files into build dir took 26.046429ms
Compiling client SPA with Svelte took 7.644657324s
Creating data_source took 254.050294ms
Running Gopack took 209.537345ms
Running Minification took 250.86314ms
Total build took 8.450171841s

✔︎ redesign (master) plenti build -b
Setting up core NPM packages took 397.155µs
Copying ejectable core files for build took 1.929068ms
Copying static files into build dir took 2.535535ms
Copying media files into build dir took 62.592133ms
Compiling client SPA with Svelte took 8.27419362s
Creating data_source took 273.012041ms
Running Gopack took 207.906777ms
Running GopackDynamic took 24.708µs
Running Minification took 301.125134ms
Total build took 9.472522308s

✔︎ redesign (master) plenti build -b
Setting up core NPM packages took 8.532µs
Copying ejectable core files for build took 1.124508ms
Copying static files into build dir took 648.823µs
Copying media files into build dir took 29.479363ms
Compiling client SPA with Svelte took 8.035595389s
Creating data_source took 248.817516ms
Running Gopack took 196.228081ms
Running GopackDynamic took 38.705µs
Running Minification took 12.711357ms
Total build took 8.592700582s

✔︎ redesign (master) plenti build -b
Setting up core NPM packages took 17.126µs
Copying ejectable core files for build took 887.507µs
Copying static files into build dir took 440.375µs
Copying media files into build dir took 38.038872ms
Compiling client SPA with Svelte took 8.173244671s
Creating data_source took 266.956432ms
Running Gopack took 276.94514ms
Running GopackDynamic took 35.021µs
Running Minification took 204.146286ms
Total build took 9.018460135s

✔︎ redesign (master) plenti build -b
Setting up core NPM packages took 7.54µs
Copying ejectable core files for build took 997.751µs
Copying static files into build dir took 917.684µs
Copying media files into build dir took 31.004583ms
Compiling client SPA with Svelte took 9.836673025s
Creating data_source took 378.221473ms
Running Gopack took 369.148032ms
Running GopackDynamic took 50.932µs
Running Minification took 184.328047ms
Total build took 10.853923843s

✔︎ redesign (master) plenti build -b
Setting up core NPM packages took 8.802µs
Copying ejectable core files for build took 1.328467ms
Copying static files into build dir took 954.474µs
Copying media files into build dir took 32.341559ms
Compiling client SPA with Svelte took 7.967233175s
Creating data_source took 255.949105ms
Running Gopack took 192.941359ms
Running GopackDynamic took 27.733µs
Running Minification took 279.658121ms
Total build took 8.82839593s

✔︎ redesign (master) plenti build -b
Setting up core NPM packages took 26.62µs
Copying ejectable core files for build took 1.069938ms
Copying static files into build dir took 529.613µs
Copying media files into build dir took 27.148774ms
Compiling client SPA with Svelte took 7.682901592s
Creating data_source took 255.460832ms
Running Gopack took 193.457158ms
Running GopackDynamic took 25.813µs
Running Minification took 289.478755ms
Total build took 8.50300487s

Build times inserting Gopack into compileSvelte

✗ redesign (master) plenti build -b
Setting up core NPM packages took 51.363µs
Copying ejectable core files for build took 1.109412ms
Copying static files into build dir took 1.699563ms
Copying media files into build dir took 62.233031ms
Compiling client SPA with Svelte took 7.671154945s
Creating data_source took 277.735738ms
Running Minification took 263.55875ms
Total build took 8.34493142s

✔︎ redesign (master) plenti build -b
Setting up core NPM packages took 5.921µs
Copying ejectable core files for build took 891.178µs
Copying static files into build dir took 634.44µs
Copying media files into build dir took 33.320886ms
Compiling client SPA with Svelte took 7.814004783s
Creating data_source took 256.741523ms
Running Minification took 297.361407ms
Total build took 8.452695027s

✔︎ redesign (master) plenti build -b
Setting up core NPM packages took 12.711µs
Copying ejectable core files for build took 914.703µs
Copying static files into build dir took 569.266µs
Copying media files into build dir took 29.191774ms
Compiling client SPA with Svelte took 8.682224217s
Creating data_source took 276.377309ms
Running Minification took 295.692885ms
Total build took 9.357606016s

✔︎ redesign (master) plenti build -b
Setting up core NPM packages took 7.064µs
Copying ejectable core files for build took 1.681486ms
Copying static files into build dir took 666.181µs
Copying media files into build dir took 34.773694ms
Compiling client SPA with Svelte took 7.70330875s
Creating data_source took 260.727824ms
Running Minification took 258.19201ms
Total build took 8.312428906s

✔︎ redesign (master) plenti build -b
Setting up core NPM packages took 9.293µs
Copying ejectable core files for build took 965.471µs
Copying static files into build dir took 550.287µs
Copying media files into build dir took 33.899756ms
Compiling client SPA with Svelte took 7.70834438s
Creating data_source took 257.22186ms
Running Minification took 258.596448ms
Total build took 8.304791122s

✔︎ redesign (master) plenti build -b
Setting up core NPM packages took 10.902µs
Copying ejectable core files for build took 2.166673ms
Copying static files into build dir took 738.223µs
Copying media files into build dir took 32.011525ms
Compiling client SPA with Svelte took 7.98073117s
Creating data_source took 252.857005ms
Running Minification took 280.231275ms
Total build took 8.594021007s

✔︎ redesign (master) plenti build -b
Setting up core NPM packages took 11.495µs
Copying ejectable core files for build took 1.117306ms
Copying static files into build dir took 621.368µs
Copying media files into build dir took 32.423262ms
Compiling client SPA with Svelte took 7.907739249s
Creating data_source took 258.847729ms
Running Minification took 264.172016ms
Total build took 8.551841419s

✔︎ redesign (master) plenti build -b
Setting up core NPM packages took 6.39µs
Copying ejectable core files for build took 1.204373ms
Copying static files into build dir took 599.809µs
Copying media files into build dir took 30.984003ms
Compiling client SPA with Svelte took 7.734580337s
Creating data_source took 257.540978ms
Running Minification took 290.458521ms
Total build took 8.364721055s

[jimafisk]

WIP Bud Compiler: abfbe95