The Gatsby monorepo: up- and downsides, DX, and reenabling "spaces" for every plugin

[Hahlh]

Hey dear Gatsby people,

I would like to make the subject of this discussion the up- and downsides of the monorepo and how to compensate for a few of its downsides.

Let's outline the context along the lines of one potential solution for compensating for the downsides of the monorepo for Gatsby plugins:

Keeping the plugins in the mono repo for one unified place for development, but mirror them into separate repos for people to contribute, discuss and collect issues.

That would maintain the benefits of the mono repo, while enabling much cleaner community interaction again (and likely much more, because probably lots of people currently don't use the issue tracker as it is impossible to find things and get an overview over items specific to a plugin).

Specifically, the issue tracking as a resource for debugging and discussion.

Even with the topic tags, it is very hard to find issues related to a specific plugin. A search for a specific behavior won't yield any useful results for the most part and I would argue that most people don't use the issue tracker as a 'space' for specific plugins because of the mono repo.

It would be interesting to compare some key engagement data like the number of issues created, comments created by community members, characters written for a specific plugin (using the topic tag), and amount of individuals contributing and taking part in threads and compare it with other repos that see similar npm download numbers.

One example would be gatsby-source-shopify and the related topic:
https://github.com/gatsbyjs/gatsby/issues?q=is%3Aopen+is%3Aissue+label%3A%22topic%3A+source-shopify%22

There is one open issue (created June 14th 22) and 36 closed. All closed issues are from 2021 or earlier.
The Github discussion section shows 6 threads for the topic gatsby-source-shopify, all of them from 2021 (gatsby-source-shopify was moved into the mono repo in December 2018).

I would assume that this means that there aren't many from the community using the repo in the "normal" way you would use the issue tracker of external code you use.
When I run into issues with some code, one of the first things to do would be to check whether the issue is tracked in the repo already and to get a feeling for what issues are currently open and what is being worked on (f.e. did the maintainers already react or did other people using that code already describe workarounds). This is not possible with the current setup.
At the moment it feels like the issue tracking for Gatsby plugins is more of a “tell the Gatsby team about an issue as they develop in public” than a “here is a representation of the current state of the code in terms of issues where you can find workarounds, find workaround together with other people using the code, and help others with their issues”. There are a lot of DX issues created by not having the code in its own “encapsulated” space. Some of the normal functions of a code-specific repo are drowned in all the noise.

And this is for a plugin that has a topic. For other plugins like gatsby-remark-prismjs the search results look like this:

gatsby-issue-tracking-2023-01-02_10.32.13.1.mp4

There are possibly not many issues for that specific plugin, but it’s hard to know for sure whether that is really the case.
For one of the most popular plugins, gatsby-source-contentful, the results look like this:

gatsby-issue-tracking-contentful-2023-01-02_10.37.40.1.mp4

With a search for gatsby-source-contentful seemingly only a handful of related issues show up in a mountain of unrelated issues. Even with the topic tag, it is still difficult to handle as all the development-related issues are mixed in (and I assume that most are not aware of the topic tags).

NPM weekly downloads:
33.110
Issues created in 2022:
39
Issues created by community members in 2022:
13

Compare that with a repo that is only dedicated to its specific code:

https://github.com/prismicio/prismic-gatsby/issues

NPM weekly downloads:
6985
Issues in 2022:
23
Issues created by community members in 2022:
22

Another interesting approach would be to simply ask developers using Gatsby how they interact with the issue tracker and what problems they have with it (please add your voices below if you have an opinion or experiences to share).

If it can be confirmed that the mono repo setup stifles community interaction and DX, then there currently isn't a public space for the plugins.
Discord is great, but it is not public in the sense that it is searchable from the web and it is also not structured in a way that allows to use it easily as a source for debugging (although threads already improved that).

---

I assume the big questions here are:

1. In terms of community interactions: what is the GitHub issue tracker supposed to achieve for Gatsby plugins?
2. How can that be measured & quantified?
3. Is the current mono repo setup hurting those metrics and how can that be confirmed with data and tests?
4. If the current mono repo setup is hurting those metrics, how does that influence DX and the usage and growth of Gatsby?
5. Is the size of the impact it has on Gatsby making it worth to do a pilot test project or consider adapting the approach overall?

---

This is my thought process and meant to be a starting point for a discussion about this, not a proposal of a specific solution.

I would love to hear your thoughts:

• Do you see similar issues in comparison with how you interact with other repos / smaller monorepos?
• How do you currently interact with the Gatsby monorepo? How does it influence your DX?
• What changes would you wish for?
• If you agree that there is an issue, what would be potential solutions?

[LekoArts]

Hi!

First of all, thanks for the write up and caring about this. I want to give some insights on why we continue to use a monorepo and why it works for us.

While gatsby is the main package that receives most issues and bug fixes, another core part of Gatsby are plugins, more specifically source plugins for important CMSs. Those plugins have their own topic label, as they also often receive new features. A lot of other plugins rarely receive any updates, as they are "finished". There's no need to change things and thus you also can't introduce bugs. So unless something upstream changed, some of the plugins were written 3-4 years ago and just continue to work. So I think a metric about how many issues are opened and correlating that to activity/interest etc. is incorrect as most plugins just work and there's no need to discuss anything. (With the release of GitHub discussion we moved all questions/discussion to there anyway, so the issue tracker is only bugs.)

Having separate repos for each package creates a massive organizational overhead for us and since we're the people who maintain it and need to be happy with the setup we're the priority. If you'd have a separate repo for a package, you'd need to:

• Check the issue and PR tracker for each repo (so checking 100 repos each day)
• Setup CI for each repo
• Have the overhead of issues/PRs created in the wrong place and moving them around
• Non-Working package management as our packages rely on being in a monorepo. Local development will be a huge change for us

Even if the repos are mirrors of the source code in the monorepo, the overhead of then checking each repo is still there. We have had separate repos in the past for some things and it just didn't work out.

With the monorepo we have everything in one place, can fix things across packages in one swoop. So from a development standpoint, using this monorepo is not going to change.

Some of the problems are just current GitHub limitations. While we use issue forms, we e.g. can't populate/enforce a prefix for the title. In theory there could be a dropdown in the issue form where you have to specify which plugin and then you'd always have [plugin-name] in front. But as long as you can't enforce this, people won't use it always.
Adding labels for every package is also not an option since we still have to use that list daily. We did a cleanup at some point and that improved usability.

We've added unique error IDs to gatsby errors so that when you google an error, you find more similar stuff more easily. Not all plugins that throw their own errors have those custom error maps yet, that could help with discoverability via Google. From my experience most people run into the same, similar problems and via Google you then often find stuff. Not saying that first issues, niche bugs don't exist. I've been in the same boat searching an issue tracker to see if someone else had this bug before. But then when creating those issues, my only worry was if the person would see it.