Make front matter easier to understand, extensible, and programmatically editable

[michaelherold]

In the interest of furthering the Content Authoring Track goal of "[a]dding advanced interactions with file-based content structures and Git state," this proposal outlines a plan to make it possible to programmatically work with front matter.

Right now, the management of Bridgetown front matter has two key areas for improvement:

1. The code scatters itself across four areas of bridgetown-core [1, 2, 3, 4] and not designed for being open to extension. This prevents adding new flavors of front matter (e.g. TOML).
2. It reduces down to a hash, which makes the data accessible but loses the context required to programmatically update and write data back to the originating resource.

I will address these two pieces separately as they represent two separate, but related, flows of work.

Open to extension

The current state of front matter management was organically grown from its Jekyll lineage. It is something that has not had much attention so far other than adding Ruby front matter. This has led to patterns like the following snippet for checking whether a file is a resource or otherwise has front matter:

if Utils.has_yaml_header?(file) || Utils.has_rbfm_header?(file)

In order to add a new type of front matter to Bridgetown, we would have to go through and find all of the pertinent areas of bridgetown-core, update them individually, and ensure we keep them all in sync. In particular with the Brigetown::FrontMatterImporter, this will lead to an increase in complexity for code that is already complex.

To make front matter handling open to extension, we must:

1. Refactor all front matter-related code into a Bridgetown::FrontMatter module (or similar) to make the behavior discoverable
2. Create an abstraction to describe the handling of a particular type of front matter (e.g. YAML or Ruby right now)
3. Document the means by which you can add a new handler

Reorganize the existing code

One approach we could take to this work that aligns with the core principle of "[g]row[ing] the Ruby ecosystem" would be to extract all front matter handling into a gem that is usable outside of Bridgetown. I think this is a laudable long-term goal, but one that we should wait we've proved the approach.

Another approach would be to extract front matter handling into a bridgetown-frontmatter gem. I think the value-add of this approach is small because front matter is core to Bridgetown and its resources. Since it is core behavior, I believe it belongs in bridgetown-core as it does currently.

The final approach, and the one that I recommend, is to use a simple Bridgetown::FrontMatter module to contain all front matter-related code. Specifically, I would map the following:

• Bridgetown::FrontmatterDefaults → Bridgetown::FrontMatter::Defaults
• Bridgetown::FrontMatter::Importer → Bridgetown::FrontMatter::Importer
• Bridgetown::Utils::RubyFrontMatterDSL → Bridgetown::FrontMatter::RubyDSL
• Bridgetown::Utils::RubyFrontMatter → Bridgetown::FrontMatter::RubyFrontMatter

Introduce front matter loaders

Currently, the responsibility for loading front matter chiefly falls to the Bridgetown::FrontMatterImporter#read_front_matter method. This method extracts either YAML or Ruby front matter. By introducing a new concept, "front matter loaders," into the core domain, we can achieve the following:

• Create an abstraction for other gems to add front matter types to Bridgetown through Bridgetown::FrontMatter::Loaders::Base.
• Simplify the extraction of front matter from content by using a loader registry to iterate through in priority order instead of having a single method handle all of it.
• Deprecate Bridgetown::Utils.has_rbfm_headers? and Bridgetown::Utils.has_yaml_headers? in favor of Bridgetown::FrontMatter::Loaders.front_matter? which is closer to the usage of the headers checks.

Why should we do this?

By making front matter open to extension, we can:

1. Make adoption easier for people moving from Hugo since we could have a TOML-based front matter loader to read Hugo front matter.
2. Improve the maintainability of front matter code by making it easier to isolate bugs.
3. Introduce a singular place to make changes to front matter to open up opportunities like the following.

Programmatic updates

So far, this proposal only talks about the work that has to happen to enable the ability to programmatically update front matter. What does it mean to do that?

Currently, Bridgetown processes a resource by:

1. Detecting and extracting front matter
2. Converting that front matter into a HashWithDotAccess::Hash and setting it as the #data attribute of the resource
3. Taking the rest of the content of the file and setting it as the #content attribute of the resource

Step (2) is the source of the friction in programmatically updating the front matter of a resource.

Take, for example, the following three examples of resources:

Example 1: Traditional YAML front matter

---
date: 2023-07-03T12:00:00-05:00
tags:
  - bridgetown
  - ruby
---

This is an article

Example 2: Bridgetown's front matter DSL

~~~ruby
front_matter do
  date "2023-07-03T12:00:00-05:00"
  tags ["bridgetown", "ruby"]
end
~~~

This is an article

Example 3: A straight Ruby hash of front matter

~~~{%
{
  date: Time.new(2023, 7, 3, 12, 0, 0, in: "-05:00"),
  tags: ["bridgetown", "ruby"]
}
%}~~~

This is an article

Each of these resources reduces to the same HashWithDotAccess::Hash once processed by the front matter system.

When working with a HashWithDotAccess::Hash, you can easily tell what key(s) a programmatic update would add to the front matter. However, if you were to then go and attempt to serialize that HashWithDotAccess::Hash back into the file, there's no straightforward way to do so since the formats are so different.

Why should we do this?

To "[add] advanced interactions with file-based content structures and Git state," of course! But what does that mean?

Consider two use cases:

1. Upon publishing, you want to send Webmentions to each link that you have on the page. To save processing time, however, you don't want to process any links that you've previously sent (e.g. upon last publish a page was down). After processing, you want to update the front matter of your resource to include a property that notes which links you've processed, then commit that to the repository.
2. You want to create a Micropub endpoint that allows you to update your website outside of an editor. As part of this endpoint, you want to be able to publish draft posts, which requires updating the front matter of said post.

Each of these use cases requires a programmatic update of front matter for a resource.

First steps

This section is more informal as I've only started thinking about this recently.

In order to be able to update front matter, we must have an abstraction exposed on the resource. The abstraction needs to track which type of front matter it is (e.g. YAML, Ruby, etc.). It also needs to be able to accept updates and appropriately render the update into the backing file.

Bridgetown::Resource#front_matter could return a front matter object. It needs to be aware of the defaults from the collection and/or site. You should either be able to update the object and track whether it is dirty or it should track the updates and later commit them to the file.

[brandondrew]

This seems to be extremely well thought-out, so I hope my reply is not half-baked, but it seems to me that if we are going to programmatically write front matter, we'll want to know

• which type of front matter was originally used so that we can write it in the same format,
• whether any given values (or keys, for that matter) were computed, (and if so possibly their AST, or perhaps literal code, or both)

(My apologies if you made either of those points and I overlooked them.)

I am 100% persuaded that having the ability to write the front matter is a good thing, although the use case that I find compelling is slightly different from the ones you mention. Writing posts/articles/whatever in markdown files seems optimized—as far as I can see—only for single-person workflows. If you ever want multi-person workflows, where you add a proof-reader, an editor, an approver, a co-author, a different contributor, etc., that could be quite tricky without the ability to programmatically write front matter. It would be nice to be able to have this sort of workflow without throwing out the standard approach of markdown files with front matter. (I assume using a headless CMS allows you to have multi-person workflows only if any content handled by multiple people lives 100% in the CMS and not in your (e.g.) Obsidian vault. Someone please correct me if I'm mistaken in this.)

[michaelherold]

I have been digging into how we could make front matter programmatically update front matter, in particular, YAML front matter so far because that's what I currently use on my site.

The news is ... not good.

Psych, Ruby's standard YAML parser and the one that Bridgetown uses, is built upon libyaml. libyaml appears to have been unmaintained since mid-2020. That's not too much of a problem as long as it does what it needs to do.

However, take this simple example:

You have a post with the following front matter:

---
tags:
  - ruby

You want to programmatically add a bridgetown tag to the post, yielding:

---
tags:
  - ruby
  - bridgetown

This isn't possible with libyaml right now (and thus is not possible with Psych using only the AST). You can only output:

---
tags:
- ruby
- bridgetown

because libyaml does not allow controlling the indentation of sequences.

There is an an open issue about this problem and a pull request that may resolve the issue, but those have been open since late 2021 and early 2022, respectively.

That means that for YAML, at least, we'd have to shave many yaks to accomplish the goal. What comes to mind to me is:

1. Implement a YAML parser, possibly based on libfyaml; big, scary, and a very hairy yak
2. Attempt to work around the problem by handling indentation in a custom Psych::Visitors::Emitter, but I'm unsure how this will go because we'd be interweaving writing into a buffer from C and Ruby
3. Combine the output of a Psych dump with something like yamllint to do the formatting (yamllint is GPL and written in Python so that would require a whole farm's worth of bailing wire)

I'm not throwing in the towel yet, but I wanted to share my findings in case anyone else has thoughts.

[brandondrew]

If I understand the problem correctly, the choice of whether to add spaces before the hyphen is a personal aesthetic choice that doesn't change how the YAML is interpreted.

There are other examples that might be more important to preserve, such as whether to put a collection inline instead of across multiple lines.

I think the problem is not so much finding a YAML library that supports the extra indentation. It's a matter of finding a YAML library that supports targeted updating of the source YAML, which means it must preserve the exact source YAML and not merely parse it and then forget the source. For most applications the only thing that matters is the interpretation of the YAML, and the individual aesthetic choices that were made can be safely forgotten. In this case we want to preserve everything about the original YAML except the bit that we update.

My guess is that this will require a new parser that preserves the literal form as well as the interpreted form of everything.

[brandondrew]

To put it in slightly different words:

• YAML is a data structure, so we can think of it like a tiny database.
• We want to be able to update just one field in our tiny YAML database without altering anything else, and without any "schema changes" (changes to aesthetic choices made in creation of the YAML).
• Normally (AFAIK) YAML libraries are mainly concerned with serializing or deserializing the entire YAML "database" and never with surgically changing only one field.
• Therefore it is likely that we would need to fork an existing YAML library to accomplish the goals described here.

If anyone wants to shave that yak, the Rust YAML library supports YAML 1.2, so it might be a good starting point for a Ruby gem that

• is super fast & safe
• supports the latest specification
• supports surgical updates

[michaelherold]

I looked at the yaml-rust library and it coincidentally has the opposite problem: it forces indents to 2 spaces regardless of the spacing as the tree was parsed. This is a feature as it's codified in the tests.

The tree you get from Psych has the spacing information in it. I think the fastest way forward is likely going to be hand-rolling an emitter that does not use libyaml's emitter.

Since the nodes have the spacing information on them, tracking the rolling indent from the parent should be possible.

As to your summary, I agree with it. The key thing here is that no line of the front matter should change that does not have to. I like the database analogy; you wouldn't want your SQL database to taint unrelated tuples every time you edit a row!