RFC: gatsby-config/node typescript support

[Js-Brecht]

Summary

It would be advantageous for Gatsby to provide native Typescript (or simply ES6 support, even) for gatsby-config and gatsby-node through the core. Because of the Gatsby's design, its pluggable nature and the GraphQL source layer, using Typescript in gatsby-config and gatsby-node provides a number of advantages, to include:

• Strong types and build-time validation for the configuration of plugins & other gatsby-config settings
  • The same applies to the consumption of Gatsby APIs via gatsby-node.
• Improved developer experience all around, especially for gatsby-node, which can get quite extensive.

The way I see it, this could be done a couple different ways, with increasing levels of difficulty. I would be able to handle the development of this feature, but before I begin, I would like to get some feedback on the following implementation options.

Simple - JIT

The simplest method for implementing Typescript support would be to use an always-on JIT transpiler: babel-register. This would guarantee ES6/Typescript support for any files run from node during the build process.

This would simply involve starting babel-register prior to collecting plugins/themes. babel-preset-gatsby-package could be used for the majority of cases (with some minor updates). To reduce the impact on performance, babel-register can be configured to ignore files outside of the directories of the default site and each of its plugins.

Caveats

• Because there is additional (synchronous) transpiling happening during the sourcing of gatsby-config and gatsby-node files, there will be a (sometimes) noticeable hang time during startup.
• This doesn't provide much flexibility to the consumer. They are locked into using the presets/plugins defined in the core, unless they use their own .babelrc... which comes with all of its own issues, of course.
• Due to the nature of babel-register, this entails utilizing Node's require.extensions. require.extensions is deprecated and has been for some time, but I don't think there's any real plans to get rid of it completely... yet. I also haven't seen any plans for a replacement. This means that JIT transpiling may not be completely future-proof.

Complex - AOT

Ahead-of-time transpiling would be a more "future-proof" method, but would increase the complexity of the implementation considerably. To remain compatible with all package managers, the transpiled source modules would need to be dumped into the default site cache. This creates a lot of problems with module scope:

• All imports would need to be converted to fully-qualified fs paths to remain compatible with pnpm/yarn2
• All imported .ts files would also need to be transpiled and dumped next to the entry file, either in the same folder structure or using some kind of uid with the import changed to match

These problems will either result in a great deal of limitations, or a prohibitive level of complexity (IMO). Would not recommend this approach.

---

I would definitely recommend using the JIT method. I have been using it in gatsby-plugin-ts-config for some time now, with excellent results. My first approach was to use AOT, but it started to turn into a "reinventing the wheel" thing so I abandoned it.

While the simple/always-on method will be functional, and I'd be happy to implement it, I feel like there should also be some extensibility/configuration options available to the consumer. I'd like to get some feedback on these, and then see if there is a good way to include them. The only one that I'd like to be decided on before the initial implementation would be the opt-in/out option. Everything else could be added in another iteration.

• Make the default transpiler opt-in (or out).
  • This way people can use their own transpilers if they wish, without concern for them conflicting.
  • This also means that Themes/Plugins could leave their gatsby-* files in Typescript/ES6 and signal to the core that they should be transpiled. If they don't, then the transpiler will ignore them.
• Allow using ts-node instead of locking it in to only babel-register
  • The Typescript compiler allows more functionality than babel does, especially transformers for working with types.
  • ts-node is not as performant as babel-register, but I feel strongly that it should be up to the consumer whether they want that performance hit in exchange for the added functionality.
  • Neither ts-node nor Typescript itself need to be shipped with Gatsby. Can simply look for them in the consumer's dependencies to see if the desired run type is supported.
• Allow extending the default transpiler
  • For babel-register, this would mean allowing the consumer to add their own presets / plugins / other options to the babel process
  • Of course for ts-node, this means allowing them to some additional options for the ts-node process. This like the compiler or the transformers mentioned earlier.

This raises a new question: how to include this configuration. I have considered a few methods in order of (IMO) their efficacy:

.gatsbyrc

Because gatsby-config should also be .ts compatible, it doesn't make a lot of sense to configure the bootstrap from it. Instead, perhaps Gatsby's own .rc file would make sense.

Like most other .rc files, this would define configurations for the running Gatsby process. In this case, it defines options that are used for configuring the bootstrap process itself, prior to consuming gatsby-config and gatsby-node.

// .gatsbyrc

/** Only used to expose type definitions for the .rc options */
const { createRcOptions } = require("gatsby");

// Could also do a JSDoc for the type definitions
/** @type {import("gatsby").GatsbyRC} */
module.exports = createRcOptions({
  /** opt-out of the core transpiler */
  // transpiler: false

  /** opt-in to the core transpiler */
  // transpiler: true

  /** define transpiler options (opts-in automatically) */
  transpiler: {
    type: "babel" // or "ts-node"
    options: {
      // Transpiler options
    }
  }
});

An alternative to having a createRcOptions type function would be to document the usage with a JSDoc. Their sole purpose is to provide types for the configuration, simply to make it easier to set up.

gatsby-config extends

An alternative configuration method may be to allow gatsby-config to "extend" another config file. This way, the initial gatsby-config could be written in .js, and "extend" the .ts one.

// gatsby-config.js

module.exports = {
  extends: "./some-other-config.ts",
  /** opt-out of the core transpiler */
  // transpiler: false

  /** opt-in to the core transpiler */
  // transpiler: true

  /** define transpiler options (opts-in automatically) */
  transpiler: {
    type: "babel" // or "ts-node"
    options: {
      // Transpiler options
    }
  }
}

package.json configuration

This would involve adding a key to package.json for defining initial configurations for Gatsby. I don't recommend this method: since it is a simple .json file, there will be a lot of things you can't do.

opt-out of the transpiler

{
  "gatsby": {
    "transpiler": false
  }
}

Include transpiler options
This is where it gets trickier. A lot of configuration options for babel-register and ts-node need functions. Can't define functions in .json.

{
  "gatsby": {
    // Define transpiler options & opt-in
    "transpiler": {
      "type": "babel",
      "options": { ... }
    }
  }
}

[KyleAMathews]

Thanks for this writeup!

👍 to simple so @babel/register.

How about we do this in two stages? We first ship a zero-config option and get it out there and then follow up with a way for people to configure Typescript compilation? That way we prioritize fixing ASAP the most basic problem here of being able to use Typescript for gatsby-* files(which is all most people would want) and then we'll get a lot more community input about ts-node and ways to configure things.

I'm not sure I love the idea of compiling Typescript files in plugins. There is a perf hit on startup for each Typescript file so it'd be best to keep this to a minimum. SWC might change that though https://github.com/TypeStrong/ts-node#bundled-swc-integration — benchmarking the cost would be helpful too — but again in the interest of keeping things simple initially, having plugins continue to need a build step seems fine. Almost every plugin in the monorepo here has a build step.

[Js-Brecht]

I'm good with dropping support for 3rd party plugins; it makes sense to me for them to have their own build process. Recommend keeping support for local plugins, however.

Regarding the opt-in/out? Since it will be zero-config to start, do you see an issue with using an env var to turn it on or off? Also, would it be opt-in, or opt-out?

[KyleAMathews]

👍 to supporting local plugins.

We'd ship it behind a flag to start with so it'd be opt-in initially. Once it's been well-tested in the wild and released, I'd assume it'd just be on for everyone (well on as long as they create a typescript file).

[Js-Brecht]

on as long as they create a typescript file

Is that to mean that the transpiler should only be triggered if there is a .ts file? Personally, I would just turn on @babel/register whether there's a .ts file or not (assuming they've opted in, of course). Its performance impact is negligible... barely noticeable in most cases. It has the added benefit of allowing consumers to use ES6+ if they didn't want to use Typescript. It would also handle edge cases where the consumer has a gatsby-*.js, but those import a Typescript file somewhere along the line.

[KyleAMathews]

Ah ok. Makes sense 👍

[wardpeet]

Thanks for the write-up! What I didn't 100% got was, do you want syntax support or also linting support? The former makes a lot of sense. Gatsbyrc goes a bit to far, I don't believe you'll need to configure custom plugins, ...

I'm not a big fan of JIT, it's easy to implement but you don't know what it transpiles to. Every update of the library can give you different results and it's hard to figure out why things went wrong. AOT make sense and isn't that expensive if you look at rollup and it would help us improve our caching layer too, if onCreateWebpackConfig changed, we don't need to throw away the node store, ...

[Js-Brecht]

What I didn't 100% got was, do you want syntax support or also linting support?

Typescript syntax support. Type-checking is a plus as well, but probably to start that can be done separately (tsc --noEmit, essentially).

I don't believe you'll need to configure custom plugins

I'm already an example of somebody who's going to want custom plugins. Not only do I like to use my own set of plugins (which probably wouldn't be included in the core) when I'm using babel, but I also prefer using ts-node.

While babel is faster, a real Typescript compiler has type reflection/resolution capability during the build process. This allows you to transform them in a way so they can be used during runtime, which means much less code duplication & maintenance. To name a few:

• I can generate a schema to check object shapes based on type structures,
• I can get a string array of keys of a type/interface (and use that to process a dynamic object),
• I can resolve import paths to match tsconfig aliases

I also prefer built-in typechecking.

Not only this, but when you use an actual Typescript compiler for the site bundling, you can share configuration between the bootstrap & the build... which means I can consume build files during bootstrap (since they are getting processed the same as during the build), and use them to apply constraints or build data structures, etc... all without needing to duplicate any code, or maintain that duplicated code. There's many benefits.

I will gladly take the 3 second hit from using ts-node during the bootstrap startup.

If I were not able to customize the transpiler, the first thing I would do is find a way to disable the core transpiler and use my own.

However, I do understand that it means added complexity, so it makes sense to have a general consensus from the team & community that it's what they want. I only ask that the ability to turn off the default transpiler stays. It will be rather hard for me to work on something when I know it could potentially cripple a lot of my other work.

I'm not a big fan of JIT, it's easy to implement but you don't know what it transpiles to.

This is true, you don't know immediately what the transpiled code looks like. However, you can take your @babel/register configuration, plug it into @babel/core the same way, and get the same output; same goes for ts-node.. Under the hood, @babel/register just uses @babel/core, and ts-node just uses typescript (or a typescript like compiler, if you've decided to use one). I have done this a few times for troubleshooting, but it's been rare that I needed to.

AOT make sense and isn't that expensive if you look at rollup

What makes AOT more complex is the fact that you're picking those files up from where they are, and dumping them somewhere else. Now you need to point everything that needs those files to the new location, and you need to make sure that everything those transplanted files use is available. There's also other things that depend on fs context (like __dirname).

If you mean that gatsby-* files could instead be bundled by rollup... that is a possibility, and one that I tried out at one point. It does add more build time than JIT, and it also has its own problems with things like node globals that depend on fs context. Some plugins are supposed to help with that, but IME they don't or are broken. It may require designing custom plugins to resolve the issues, which adds its own complexity.

it would help us improve our caching layer too, if onCreateWebpackConfig changed, we don't need to throw away the node store

I'm not sure I fully understand. Do you mean that you would do analysis of the AST during the transpiling process (or bundling, if doing it with rollup) to see what's changed? That would be pretty neat... tracing the symbols to look for changes would be... interesting 😆

[wardpeet]

I'm already an example of somebody who's going to want custom plugins. Not only do I like to use my own set of plugins (which probably wouldn't be included in the core) when I'm using babel, but I also prefer using ts-node.

The way you're explaining seems more like plugins able to get more build information and generating extra type information to consume during development/production. Not so much on enabling typescript syntax/type checking in gatsby-node, gatsby-ssr, gatsby-browser & gatsby-config. Did I read that right?

This is true, you don't know immediately what the transpiled code looks like. However, you can take your @babel/register configuration, plug it into @babel/core the same way, and get the same output; same goes for ts-node.. Under the hood, @babel/register just uses @babel/core, and ts-node just uses typescript (or a typescript like compiler, if you've decided to use one). I have done this a few times for troubleshooting, but it's been rare that I needed to.

I'm well aware :) what scares me a bit is the node_modules multiple versions aspect of NPM/yarn where debugging and issue triaging becomes difficult. I do think JIT is probably the first POC to go forward with

I'm not sure I fully understand. Do you mean that you would do analysis of the AST during the transpiling process (or bundling, if doing it with rollup) to see what's changed? That would be pretty neat... tracing the symbols to look for changes would be... interesting 😆

I agree but I do think it's worthwhile in the future.

I'm not sure I fully understand. Do you mean that you would do analysis of the AST during the transpiling process (or bundling, if doing it with rollup) to see what's changed? That would be pretty neat... tracing the symbols to look for changes would be... interesting 😆

exactly, when gatsby-nod gets changed, we have to clear the full cache as we do not know which lifecycles got updated. When I change onCreateWebpackConfig, there is no need to clear the node store as nothing relevant changed. Bundling these lifecycles separately would give us an easy way to get a dependency tree.

[Js-Brecht]

The way you're explaining seems more like plugins able to get more build information and generating extra type information to consume during development/production. Not so much on enabling typescript syntax/type checking in gatsby-node, gatsby-ssr, gatsby-browser & gatsby-config. Did I read that right?

Yes, in essence that's what it is. The two, however, are inextricably linked, though the former is optional. You cannot have the former unless you plug it into the transpiler that is providing the typescript support. It is possible to layer the transpiler processes, but then it's doing a lot of extra work that doesn't need to be done, and results in more potential points of failure.

[wardpeet]

I think we can start drafting something around the JIT/plugin approach to enable typescript support in gatsby-node, etc...

[alvis]

I'd say, having a choice on the compiler and its options is quite essential. There are many different desires and use cases out there and allowing the user to configure what they need is the only solution to avoid troubles in implementation.

From a user point of view, any solution that results in a minimal change is always the best. @Js-Brecht may not recommend, but I actually support the notion of a .gatsbyrc file or gatsby field in package.json.

However, they should be strictly static and serve the purpose of configuring the transpiler only. The best and cleanest implementation should enable user just simply rename gatsby-config.js to gatsby-config.ts without any further configuration than just setting "transpiler": true in the config file.

Today, I have a tacky way to have all gatsby-* in .ts extension without a single .js file to build a gatsby site.

node_modules/.bin/babel-node --presets @babel/preset-env --presets @babel/preset-typescript --extensions '.ts,.tsx' node_modules/.bin/gatsby build

It doesn't work on windows, but it does the job for cloud deployment and more importantly on my local machine.
The only problem, however, is that I'm struggling to find a way to keep my gatsby-*.ts working on Gatsby cloud as it doesn't allow me to configure a custom build script...

[Js-Brecht]

@Js-Brecht may not recommend, but I actually support the notion of a .gatsbyrc file or gatsby field in package.json.

I meant that I don't recommend using package.json; I would prefer .gatsbyrc. Transpilers frequently have configurations parameters that want functions, and you can't define functions in package.json.

That said, @KyleAMathews mentioned in #31587 (reply in thread) that it will be behind a flag to start, so it can be enabled by an environment variable.

Once it's out of the experimental stage, it will be on all the time (perhaps with an option to opt-out); this means that users can write their gatsby-node and gatsby-config in Typescript, or even ES6 if they wish.

[phil-lgr]

I don't know why I didn't try this sooner, but this works really well in my monorepo:

TS_NODE_PROJECT=../../tsconfig.node-commonjs.json node -r ts-node/register ./node_modules/.bin/gatsby build

[pragmaticpat]

@Js-Brecht and others in this discussion - Typescript support is something we've prioritized for 2022.Q1, and we're looking for folks that would like to participate. We will have a Gatsby core maintainer focused on this feature, and it'd be great to team with our community members to bring this all to life.

@Js-Brecht - are you still up for contributing in this area?

[MarkBennett]

Thanks for that awesome reply @Js-Brecht!

I'll check out your branch and see what we can contribute.

[pragmaticpat]

@MarkBennett - if you have some folks available to work on this, we'll be starting on the project at Gatsby in the next couple of weeks, and would love to team up so you have maintainer support as well! Maybe we can work together to bring this one to reality 😀

[pragmaticpat]

@phil-lgr - we're going to begin working on this RFC soon ( in some form ). You mentioned you were very interested in helping on this feature. Are you still interested in working with us on this? More to come early next week!

[MarkBennett]

Hey @pragmaticpat. I'll keep an eye on this dicussion. Let me know if there's an associated PR and what I can do to help, then I'll see if I can contribute some dev resources. Would like to support Gatsby for sure!

[njbmartin]

Keen on seeing this implemented, so happy to test it out.

[LekoArts]

Hello all 👋

Thanks @Js-Brecht again for your WIP branch. I took the changes and put it into its own branch on our repo: https://github.com/gatsbyjs/gatsby/compare/poc/ts-jit

I've published the contents of it under the alpha tag alpha-ts-jit, so you can do a npm install gatsby@alpha-ts-jit and try out using a gatsby-config.ts and gatsby-node.ts. I've tested it on a rather small project that used some plugins, but I didn't do much with the gatsby-node.ts file. So this is still all in all not production ready! But please try it out!

With the help of this RFC I've written an internal tech spec for this project and I plan on writing a new RFC that contains contents from that and how we envision things going forward. We'll want to do AOT compilation for various reasons so I didn't feel like editing this RFC here is appropriate since it suggests JIT.

So my plan of action is:

• Write new RFC so that y'all know what we want to do for initial release
• You can weigh in on things there
• Write a Proof of Concept for the AOT implementation and let people test it out
• Figure out remaining things to do and if y'all are interested you can work on PRs helping with it (But testing it out is also super appreciated!)
• Get to an actual GA release

[LekoArts]

The new RFC including a new canary for AOT is out here: #34613