Migrate custom scripts and styles to compiled static files

[kawapure]

Background

Rehike currently uses Twig templates and generates inline script and style content for all custom purposes (Rebug, /rehike/ pages, and the classic player). This is a misuse of the templating engine and results in regenerating and sending a lot of the same content over and over again. There have been plans over time to migrate some content to a proper build pipeline. For example, template/hitchhiker/rehike/debugger/css/main.css.twig explicitly states that it should be ported to SCSS.

Proposal

Begin work on moving this content to a proper build pipeline.

We already have a folder for storing static content in the main repo, static/. This is where we can store compiled binaries of things as we need them.

I propose that we create a common folder in the main repo with a common name, something like src/, for storing the source code of these. This will be the new home of non-PHP source code in Rehike.

Gulp is something I like to use for build pipelines. Yeah, forgive me, it is a Node.js thing, but it's pretty easy to work with and surprisingly flexible. I use it in my personal projects and so do some other people on the Rehike team, so I think it would be an appropriate tool to use for the job. Node modules and whatnot should be added to .gitignore though.

Additionally, I think that we have room to migrate Rehike/protos, should we make this change.

Given this setup, I propose that the root folder src/ contains the setup for the build environment (package.json, gulpfile.js, build_tools/ subdirectory, etc.). It will additionally contain subdirectories for each language, let's say:

• src/js - JavaScript/TypeScript source code
• src/css - SCSS source code
• src/protos - new home of Rehike/protos

[YukisCoffee]

Yes. I've been meaning to get around to this for a while (if you couldn't tell lol), but if you want to begin the work on this, I'd appreciate it. Otherwise, I'll do it when I get a chance.

[kawapure]

Okay, I'll start now. Don't you privately have some branding resources and whatnot? Do you mind checking them in when you get a chance?

[YukisCoffee]

Yeah, I'll upload them when I get home later today. src/img is good?

[YukisCoffee]

Actually, I'll just commit them rn, I'll create that folder and you can change it if you want

[YukisCoffee]

Okay done f367e89

[kawapure]

I forgot to comment, but I continued setting up this folder in 252dd87.

Any suggestions on what JS compiler we should? You've used Google Closure Compiler in the past, but I'm curious if you think that's a good idea for Rehike now. Ideally, we want something that will work on old browsers, since that's a surprisingly common reason why people use Rehike.

[YukisCoffee]

Unless there are any complications, use Closure Compiler. It's what I designed a lot of the Rebug JS (which then became Rehike JS) with, so it should be easiest to port that there.

Btw, for frontend reporting, we should add a VFL-like system. It wouldn't be a difficult system at all. We would just need to get the MD5 sum of a file at runtime and then have the static router handler -vflXXXXXX file name suffix, entirely virtually. Maybe a cache list that gets updated with builds for efficiency?

[kawapure]

Alright, I'm working on the general build system now. May be a few days until it's ready, and it might diverge a bit from the original plans. I'm considering moving away from Gulp because I don't feel it will be flexible enough for our future needs.

[kawapure]

Build system RehikeBuild introduced in ed9c7b6.

I didn't move away from Gulp, but instead wrote some wrappers for it where we have subprojects with definition files. This is kinda like Firefox's MozillaBuild, but the definition files are instead included via a global glob match for .rhbuild files. I guess one issue with this approach is that we can't control the order, but hopefully we never need to.

See the commit message for more information.

[YukisCoffee]

Great! I tried my hand at porting something rq here: de174c6

Rebug CSS is now SCSS (as was planned) and is compiled to a CSS file. I had to modify the static router a little to handle this, but it isn't versioned at the moment.

[kawapure]

Looking into it, Closure Compiler is really complicated. I think we should just settle for a simpler bundling/minification tool.

[YukisCoffee]

I suppose there could be complications. I know Closure Compiler works best with the Closure Library, and we don't use that for Rehike JS. Realistically, any build environment will work, but most bundlers in my experience are shit that use a weird module pattern.

I just like Closure Compiler because it includes things more like other languages do, and it also has a module include format that doesn't require relative paths by default (god why is that a problem with modern JS build systems? Do they seriously want you to use node_modules for your own source code too?). I wrote a script a long time ago to work around this, but we'd probably be best off restarting that from scratch.

Here's my old "relativepather" script I made for that purpose: https://gist.github.com/YukisCoffee/47311168fc383715108aa2500f0357fb

By the way, in the project for which I wrote this script (which I can't publish for privacy reasons, but I might redact the names and publish it anyways), I used Browserify + Babel for compiling JS. I hated this combination and it still didn't result in code that worked on Internet Explorer. That's one of the reasons I ended up abandoning it.

[YukisCoffee]

Okay, here is the publication of the ybTube project from which the code originates. I only redacted some words from source code comments which contained private names, and a private IP address leak in the source code.

https://github.com/YukisCoffee/ybTube

[kawapure]

god why is that a problem with modern JS build systems? Do they seriously want you to use node_modules for your own source code too?

Yes lol

I agree that JS bundlers tend to be trash. For the record, I don't think that Closure Compiler is that bad, but we'll definitely need to figure out a way to work with it. Its design pretty much requires that you write code to a certain specification, and that either means we add our own code generation feature on top or spend a lot of time rewriting the JS.

Alternatively, I guess we could write our own JS compiler...

[kawapure]

I am really suffering trying to implement Closure Compiler support: 27132b0

[kawapure]

Update:

I think I'm going to rewrite what's done so far. I think it would be a better idea to work with Gulp streams more directly and write a wrapper class for them rather than the current idea of identifying files with handles. This would certainly simplify the issue of identification.

For the record, Gulp itself is actually very simple. The Gulp API itself is mostly just a renamed wrapper for vinyl-fs abstractions, which don't seem too complicated to reimplement either. Perhaps it would also be a good idea to reimplement these APIs (gulp.src and gulp.dest) to work better with direct file paths.

There is an interesting problem I'm faced with, though: what if we need to support multiple output files? It's actually a pretty complicated matter, because the Vinyl transform stream concept only really works for performing transforms on a bunch of input files to make a bunch of output files (with a few nuances seemingly). Perhaps the best idea would be to simply split these off into separate build tasks, so it's still a 1:1 correspondence.

[kawapure]

Further restructuring in a9ba6de. Read the commit message for more information.

[kawapure]

Okay, I settled on a basic approach to dynamically add tasks during compilation in 35cf995.

Building still doesn't work yet, but I don't see why we can't do a rough port of other parts of the codebase in the meantime. It'd probably be best to stick to CSS for now, but we have some other parts (especially version) that can be ported over.

[YukisCoffee]

Alright. I'll go ahead and port version CSS later.

[kawapure]

More work done in a4a9618. We're getting closer to working JS compilation, but there are seemingly some complications with how the externs files are handled.

The Closure Compiler documentation says to use @externs for this purpose, but it doesn't seem to work. I wonder if Gulp is fucking this up somehow.

[YukisCoffee]

I went ahead and "ported" some other CSS to SCSS. More of moved it, to be fair. Since the code can't be compiled, I didn't bother with restructuring anything just yet.

By the way, if you feel it'd be too difficult to use Closure Compiler, we could totally move to a different compiler. I tried to design our existing code so that a transition to Closure Compiler would be easy, but maybe something else would be better. It's up to you, really.

[kawapure]

Working JS compilation and file output in 36ee816. See the commit message for more information.

Additionally, the rehike-core JS package was moved to RHBuild in 90e1db7 and ae1780b.

[kawapure]

Gulp CLI abandoned in 4e30421.

[kawapure]

I think the bulk of the work is finished. We still need to add support for protobuf compilation, but the core of RehikeBuild is decently finished now. At least to the extent that you can build all migrated packages with it.

As such, I will consider closing this issue.