Introduce a full stack web framework

[thisislawatts]

The Problem

The current Platform software is built off of the Create React App (CRA) v4 boilerplate. CRA provides a useful starting point for prototypes and early stage products, the CRA tooling has been instrumental in allowing us to demonstrate the value of the Platform and build out features for our members.

However we are now facing a number of challenges introduced through our current use of CRA, and I believe we're in the perfect place to make a change which supports the next iteration of this project.

• SEO – the pages are rendered on the client side, which makes them harder to parse for search engines. We have a few outstanding issues to improve this behaviour. SEO is a first class use case for all of the full stack frameworks included in this proposal.
• Unstructured application– the Platform maintainer team is comparatively small, so the best use of our time is delivering value to users on the Platform not making architectural decisions or guiding external contributors. Aside from enforcing a Single Page architecture, there is little guidance for contributors on how best to introduce new functionality. Adopting an existing web framework will allow us benefit from wider community learnings and approaches for similar projects.
• Platform complexity – The Platform team have adopted yarn, Typescript and a mono-repo code structure. These decisions have all taken us further away from the ideal starting point for a CRA application.
• Slow release cycles – The CRA project saw a 12 month delay between v4 and v5, so far exploratory work by both @chrismclarke and @thisislawatts shows that this migration will be non-trivial. This means the Platform technical core falls further behind the latest version.

Proposal

Adopt one of the emergent full stack open-source frameworks to provide a foundation for the next iteration of Platform. There are plenty to choose from:

• https://astro.build
• https://blitzjs.com
• https://nextjs.org
• https://redwoodjs.com
• https://remix.run

We would initially evaluate these solutions by building out a deployable instance which allows us to render the logged out view for our How to and Research pages.

A decision of the viability of this approach should be made by the end of May/June 2022

Solution

Once we've reached a decision on which of the above stacks to proceed with we would want to move the existing functionality. This could then be deployed in production alongside our existing site using a URL like: https://community.preciousplastic.com/next/how-tos/abc

Out of Scope

During this exercise we will not be consider changes to the folllowing:

• Data layer: Firebase datastore will still be the primary source of data
• UI rendering: React is our UI framework of choice
• CSS-in-JS: emotion continues to be our preferred tool for styling components.
• Deployment: We will continue to target Google Firebase Hosting and the serverless Cloud Functions platform
• Backend language: Node.js and Typescript

During our assessment of tools going forward we should evaluate solutions on the flexibiity they offer in changing any of the above at a later date.

Risks

• Increased hosting costs, as we currently leverage static hosting to minimise our costs introducing a server component will result in higher server costs.
• Backing the wrong framework, given the range of options currently available it stands to reason that one or more of these will fall by the way side over the next few years.
• Partial migration – we are unable to see the migration through to completion in a timely fashion due to limited resources. This leaves us in a partially migrated state unable to make the jump to the next version.
• Continued contributions – each change merged to the existing Platform application also needs to be migrated to the new build.

[davehakkens]

thanks for brining it up.
Makes sense on many points. But hard for me to give technical input. Would be curious what @chrismclarke thinks.
I'd be mainly worried about the size of the operation, small team. And meanwhile we also have the regular development.
I could take the risk away of increased server costs. Pretty sure we would be able to handle this.

[franknoirot]

Thanks for this thorough write-up @thisislawatts, I agree with your argument and support this effort. A few questions:

1. will the evaluation tests be handled in other issues or within this one?
2. is there any support work that can be done to help determine level of effort for adoption? Anything like mapping out components, views, or doing write-ups on the frameworks?
3. was there discussion about how consensus on the framework will be chosen at the end of the month?

[thisislawatts]

3. I have no strong view atm on how we should evaluate the various frameworks available, I think we need general consensus that this is a project we're willing to take on then we can commit more time/energy to next steps.

[chrismclarke]

Thanks for raising the discussion, there's a huge amount to unpack here but what it really feels like is the question of server-side rendering vs single-page apps (as that is what most of the tools mentioned are geared towards). I would say all the problems identified can be solved reasonably well within our current systems, just that there may be added benefits from introducing something new. More specifically:

• SEO is indeed harder with single-page apps, but by no means necessitates large architectural changes to fix (it is more a by-product that typically comes out-of-the-box with SSR). Right now we could still generate sitemaps and individualised seo pages, however that would require investing more in our own tooling (or something like react-helmet) as opposed to leveraging tooling built by others so there is a tradeoff

• Definitely agreed adding more structure to the repo would be great, although I'd be wary of relying on wider community learnings as that is where most of our mixed structures comes from. There will always be lots of potential approaches and people making cases for which is best - we've often tried to follow such guidance in the past however that's ever-changing (e.g. starting with class-based components when they were all the rage, then moving towards functional components when released, following the best practice guidance for mobx datastores when introduced for v2 and then trying to refactor for v4/v5 etc.). But really what I think your point gets at is a preference for tooling that is opinionated (which I'd be strongly in favour of), so that we don't have to make so many decisions and there are clear, preferred ways of contributing.

• The issue of complexity I'd say is much less from the side of yarn and typescript (both of which react has strong support for and actively maintains templates supporting), more the monorepo structures. Particularly the fact that we have separate workspaces for core components compared that need to be compiled and included in the runtime application, and additional shared definitions and methods used across both frontend and backend. I do not think any of the solutions mentioned will actually go all that far to removing much of this complexity, instead I think the discussion here really needs to be on better monorepo tools (e.g. lerna, turborepo, nx). I'm not sure if any of the tools suggested use these out of the box (guessing next has strong turoborepo support given they are both produced by vercel) ,

• As for slow release cycles, this I think is somewhat misleading as our releases are independent to the underpinning version of react scripts. No matter what system we use, we will always be at the mercy of the bundling tools (webpack, parcel, rollup, vite, esbuild etc.), and it just so happens that the major update from webpack v4 to v5 came with a whole host of breaking changes. If you look through the github of nextjs they had a very long cycle in migrating their support from webpack 4 to 5, in the same way that react script did. I would say the discussion to be had here is if we want to (a) rely on webpack and (b) rely on the version of webpack that is shipped with CRA. If the answer to either of these questions is no then we should be looking to eject and migrate, which as identified is non-trivial but pretty much all the issues we would have in doing so would also be experienced migrating to any other system that has underlying build tools (e.g. nextjs using webpack 5, remix using rollup, astro with vite etc.).

I also think on top of the points raised there are a number of additional potential benefits to be had, particularly around speed and content optimisations, and providing a more secure pattern for server-client data access and modification. Additionally I think there would be some additional risks to consider such as whether the tools we use are likely to be supported in the future or not, and what our options are if we were to determined the tooling we want to use won't support something we need (e.g. how much control/customisation do we have over underlying structures) - given that some of the tools mentioned have up to 1k open github issues (next) I'm sure we won't be alone with our concerns.

If I were to cast a vote in the issue overall I'd probably say that nextJS is the only of the options listed that I feel stable enough to commit to longer term (with the added bonus of strong support for turborepo which I would hope could simply our monorepo setup considerably). Although I'm aware @thisislawatts has already spent a considerable amount of time trying to test a migration and not sure where they landed in terms of feasibility. In either case I think I'd still be tempted to implement smaller solutions within our existing ecosystem first as many would facilitate future migration anyways (particularly ejecting CRA and upgrading webpack 4->5)

[thisislawatts]

Thanks for the detailed feedback @chrismclarke, much appreciated! You certainly helped flag some details that I had originally missed and highlighted the pitch is too broad.

So I think that there is a two stage approach required here:

1. Refine requirements and evaluate available options
2. Pick a solution to migrate to and build a more sophisticated MVP that aims to replicate one of our existing pages eg. https://community.preciousplastic.com/how-to/set-up-a-collection-point

Do those two steps represent project you'd be willing to back as useful investment for the future of the platform?

[chrismclarke]

I'd probably add another point somewhere on collecting some useful reference links on understanding pros/cons of the different approaches to help build out a common understanding for decision-making, but apart from that I think the approach sounds good to me.

I think SSR definitely provides an interesting and exciting pathway for delivering platform content, I just don't have enough experience with any of the platforms to really have a strong grasp on the amount of work it would take and how much we will be able to work within the structures created by these platforms vs creating our own methods/workarounds/hacks to get things working the way we want to. So I'd still likely want to have a plan to address many of the issues identified independent of overall architecture (e.g. continuing the work started to separate out core components.

As for useful investment, I'm pretty much always in favour of anything that other people feel passionately about bringing to the project - I've always seen the platform as a place to come outside my regular day job to explore new and interesting ideas and seek ways to make my contributions, so really happy if others are able to do the same too. I see no real downside to at least attempting these things, best case we get a really smooth-running and performant platform upgrade, worst case we still end up with a lot of core improvements, code tidying and some new targets for the future.

[kfern]

Hi!

I use a different stack on the backend with very good results:

• A linux vps hosting with docker-compose or kubernetes (Ubuntu microk8s)
• Feathersjs
• MongoDB.

Feathersjs is "A framework for real-time applications and REST APIs". https://dove.docs.feathersjs.com/
Important: This is the documentation for the upcoming Feathers v5 (Dove). See docs.feathersjs.com for the current version.

On the frontend, React and Redux.

[adrianduke]

We used https://prerender.io/ for https://gethigher.io (which is all Firebase + Functions). May be worth taking a look at, it was really painless to integrate.

[davehakkens]

@adrianduke we are now actually looking into implementing prerender.io as a temp fix until we have our SEO fully intergrated.
thanks for the tip. Would you have some suggestions/tricks on how to intergrate with firebase?

@chrismclarke you can (hopefully) get more info here :)

[adrianduke]

This is how we have it setup, @yourheropaul anymore you'd add?

package.json

"prerender-node": "^3.2.5",

firebase.json

  "hosting": {
    "public": "build",
    "ignore": ["firebase.json", "**/.*", "**/node_modules/**"],
    "rewrites": [
      {
        "source": "/endorse/**",
        "function": "preRender"
      },

functions.ts

const renderer = express();
const prerenderToken = functions.config().prerender.token;
renderer.use(require("prerender-node").set("prerenderToken", prerenderToken));

/*eslint-disable*/
renderer.get("*", (req: any, res: any) => {
  res.status(200).send(fs.readFileSync("./web/index.html").toString());
});

export const preRender = functions.https.onRequest(renderer);

[chrismclarke]

Thanks for sharing @adrianduke! Yeah I definitely like the look of pre-render as a quick-fix for serving some better-cached SEO content

I think that makes sense to me, am I right in thinking you only using pre-render for some specific paths (/endorse/**) or is your entire firebase app hosted on the path and you are just redirecting all content through the pre-render proxy?

For the platform in the short term I'd probably suggest we use a similar function at a testing endpoint (e.g. /seo/howto/some-howto) and see if we can get prerender to return parsed content from the /howto/some-howto URL), and maybe also add a filter function to only use prerender if a bot user agent.

[chrismclarke]

Just saw this summary pop-up for redwoodJS
https://www.youtube.com/watch?v=o5Mwa_TJ3HM&ab_channel=Fireship

I'd still be a bit hesitant given that v1.0.0 only released back in April, but still an interesting-looking project

Here's the link for the overview the same channel gives for remix. They mention a key use-case they see being for sites that have a lot of statically generated pages, so maybe that does even have the edge of nextJS (?)
https://www.youtube.com/watch?v=r4B69HAOXnA&ab_channel=Fireship

[thisislawatts]

Thanks for sharing those links. A bit of experimentation yesterday has unearthed that Remix bundles React v18 which may complicate any gradual migration as we will have two versions of React in use.

[mariojsnunes]

I have experience with Remix.
I could do a PoC to test:

• Firebase Authentication
• Firestore requests
• Conditional rendering if logged in
• Deployment to firebase

Can you think of anything else?

[thisislawatts]

@mariojsnunes in the interest of keeping the PoC scoped to the smallest possible set of changes. Consider that any migration would need to be done gradually, and with firebase we can offload one route at a time 🤞🏻

So some additional framing for the PoC might be focusing on how much effort to recreate a profile page, an example: /u/metamorf-precious-plastic-touraine.

Open questions:

• How much (if any) of the existing component library can be reused?
• Can we extract our app shell (header/footer) to make it work in CRA and Remix context during migration process?

[mariojsnunes]

The Remix team is working on a PR that will enable exactly what we want... a gradual migration.
We could use Remix in SPA mode until everything is ready for SSR.
remix-run/remix#8338

But it's still a Draft PR, no idea when it will be merged.

For now, there is another option: ClientOnly components.
https://remix.run/docs/en/main/guides/migrating-react-router-app#client-only-components
But all routes would still need SSR. Not sure how much work this requires.

Meanwhile we could focus on improving the codebase and see how that PR develops.

[thisislawatts]

Looks link like the PR has merged ✨

[mariojsnunes]

will be easier to try after the vite migration, is there anything I can do to help with that?

[thisislawatts]

@mariojsnunes the branch has gone stale so it will require some rework. I have some time opening up for this from next Wednesday. But if you've got any time before then please feel free to pick it up.