outputReferences & iterative lifecycle restructure

[jorenbroekema]

This issue is a proposal for changing some of Style-Dictionary's core architecture with regards to resolving references and transforms.

It goes over two topics, but they are quite interconnected hence why I include both in this issue.

Lifecycle

As I understood, the lifecycle of style-dictionary is roughly as follows:

1. Combine/parse JSONs from include/source, run any custom parsers
2. When building a platform, run all transforms on the dictionary, deferring tokens with values that contain references
3. Then, resolve all references on the dictionary
4. Then, run transitive transforms (after references are resolved) on the deferred tokens that we skipped in the previous cycle of transforms. For chained references, this cycle continues until all transitive references have gone through.
5. Then, go through "files" and run the formats for the transformed and resolved dictionary

However, a crucial thing to understand is that step 2 till 4 runs iteratively for each token, in sequence, rather than phased for the whole dictionary, and the way that tokens with references are deferred continuously for transitive transforms to deal with, makes this quite complicated.

It can easily lead to transforms misbehaving and stacking up on one another, depending on the order of tokens that use references. This is explained in the docs, but I would like to challenge this architecture.

Problematic example configurator link

{
  "foo": {
    "value": "2"
  },
  "bar": {
    "value": "{foo}"
  },
  "qux": {
    "value": "{bar}"
  }
}

Now imagine we have a transitive transform that puts !default after each value. Due to the references and the fact that transforms run iteratively per token, the !default suffix is stacking up once for every reference layer, resulting in pretty awkward output:

$sd-foo: 2 !default;
$sd-bar: 2 !default !default;
$sd-qux: 2 !default !default !default;

There are also use cases where this stacking of transforms is intended:

{
  "color": {
    "red": { "value": "#f00" },
    "danger": { "value": "{color.red}", "darken": 0.75 },
    "error": { "value": "{color.danger}", "darken": 0.5 }
  }
}

error.value should be darken(darken('#f00', 0.75), 0.5), so stacked.

One solution, I think, would be to run every step as a phase over the whole dictionary. This simplifies the lifecycle conceptually as well and ensures that the order of tokens doesn't change the output:

1. Combine/parse JSONs from include/source, run any custom parsers
2. Resolve all references on the dictionary (or not)
3. Then, run transitive transforms (after references are resolved) on the deferred tokens that we skipped in the previous cycle of transforms. For chained references, this cycle continues until all transitive references have gone through.
4. Then, go through "files" and run the formats for the transformed and resolved dictionary

But this would break use cases where the stacking of transitive transforms is intended, so that's important to note.

Output References

Right now, outputReferences is an option on format-level, only enabled for CSS-like languages: css, sass (aka scss), stylus and less. Output references has had its challenges when combining with value transforms (transitive ones specifically), some of which have been fixed by me in version 3.9.0 and will be forward-ported to 4.0.0-prerelease.1, but I think outputReferences struggles more fundamentally, because it only applies at the format lifecycle, which is actually a bit too late to work reliably.

I would like to propose that outputting references should be split into separate parts.

API proposal:

{
  "source": ["tokens.json"],
  "references": {
    "resolve": false, // true by default, can be (token) => boolean function to be selective
    "warnOnly": true // false by default, will usually always check references and throw if a broken one is found, but can be (selectively?) disabled
  }
}

"references" prop is valid on global dictionary config level, as well as on platform specific level.

Putting resolve on false will mean that any values that contain references, will just keep those references as is, rather than resolving them.

The other part of this proposal is that in the future all transforms will happen after reference resolution. If reference resolution is turned off, transforms will not apply to tokens with references.

{
  "math": {
    "value": "{foo} * {bar}"
  }
}

The math expression transform cannot do much with this value pre-reference-resolution. It needs to happen afterwards, where both foo and bar are resolved to numbers.

Finally, when references are resolved (or not!), transforms are ran (or not, if value is/contains a reference), we get to the format stage, where any format can ensure that it understands how to deal with references, e.g. for ES6 variables:

StyleDictionary.registerFormat({
  name: `es6WithReferences`,
  formatter: function({dictionary}) {
    return dictionary.allTokens.map(token => {
      // supports object type token values
      let value = JSON.stringify(token.value);
      // format-agnostic function which checks if there are references, if so,
      // runs callback for each reference to replace the ref string with something format-specific
      // e.g. for CSS it would be (ref) => `var(--${ref.name})` or (ref) => `var(--${ref.name}, ${ref.value})`
      value = dictionary.processReferences(token.value, (ref) => `${ref.name}`);
      return `export const ${token.name} = ${value};`
    }).join(`\n`)
  }
});

This simplifies the format logic a bit, and makes the reference replacement logic more agnostic to what the format is, because all it does is identify references ('{...}') and run a callback for each reference to replace it with something format-specific.

Note: transitive transforms and outputReferences don't go hand in hand, as outputReferences would undo transitive transforms work, in this case it makes more sense to know beforehand if you're going to output references, and skip value transforms altogether for values containing references.

In summary I think this restructure of transforms and reference resolution could simplify the developer experience and reduce the amount of bugs/issues we have related to the current structure, but we have to find a way to still allow stacked transitive transforms to work.

Configuring reference resolution on a global and platform level makes sense I think, opening up the outputting of references in the final format to more platforms than just CSS.

Disclaimer:
I don't have full context on the history of the token transform/resolve lifecycle or the addition of transitive transforms, so please do let me know if there's something I'm missing here.

[jorenbroekema]

tokens-studio/sd-transforms#211 another related issue, which essentially boils down to that:

{
  "color": {
    "red": { "value": "#f00" },
    "danger": { "value": "{color.red}", "darken": 0.75 },
    "error": { "value": "{color.danger}", "darken": 0.5 }
  }
}

stops working when the "darken" value also is or contains a reference 😅 and it's really quite tricky to fix that as far as I can see

[jorenbroekema]

tokens-studio/sd-transforms#198 related topic for outputReferences, which begs the question:

If we allow not resolving references to begin with, will we allow transforms that apply on values with references, useful for token values:

• "{dimension.lg} * {dimension.scale}" -> calc(var(--dimension-lg) * var(--dimension-scale))
• { value: "{colors.red}", "lighten": "0.5" } -> color-mix(in srgb, var(--colors.red), #fff 50%)

For the second case, you could imagine that there is a reference in the lighten prop:

• { value: "{colors.red}", "lighten": "{lighten.half}" } -> color-mix(in srgb, var(--colors.red), #fff 50%)

Which could then be tackled by allowing resolve to be a callback function resolve: ({ propName }) => propName !== 'value' which excludes references inside value props to be resolved, but resolves it in other props.