This document describes the high-level architecture of this MozFest plaza deployment. If you want to familiarize yourself with the code or learn how everything works, you're in the right place!
There are several components that the system is broken into:
- client — Vue.js Single Page App (SPA)
- server — Node.js server
- sockets — Socket.io server
- jobs — Node.js scripts to be run periodically
- content — Localised copy and configuration
- config — Kubernetes manifests to deploy instances of the Plaza
- deconf-ui — the OSS project the UI uses to display schedules, sessions and etc
- deconf-api — the OSS project the API uses to process data and serve authenticated information to clients
- deconf-shared — an OSS project between the ui & api varients to share code and types
- cache — Redis server for key-value paris and message brokering
- database — Postgres database to store relational information
Here's a rough structure of the deployed components. This architecture is designed to have as much content in the cache to be served really fast. Then each of the public facing components (server, client & sockets) scale horizontally to meet demand.
flowchart TD
W[WWW]
C[Client]
S[Server]
IO[Sockets]
R[Cache]
P[Database]
FS[Fetch Schedule]
FU[Fetch users]
FC[Fetch content]
G[GitHub repo]
W --- C
W --- S
W --- IO
S --- R
S --- P
IO --- R
IO --- P
FS --- R
FU --- R
FC --- R
FC --- G
copy and past into mermaid.live if you can't see a preview
The client, ./client, is a Vue.js 2 web app that uses deconf-ui and adds custom logic. A lot of the code is importing Deconf components, configuring them for the Plaza and serving them under vue-router routes.
Cusom code for each iteration of the plaza has existed in this repo and has been contributed back to Deconf and reimplemented using the OSS version.
The server, ./server, is a Node.js backend that uses deconf-api and adds custom logic. Lots of the endpoints serve things from the cache to be as fast as possible, other endpoints may touch the database to modify state or provide something custom.
Cusom code for each iteration of the plaza has existed in this repo and has been contributed back to Deconf and reimplemented using the OSS version.
There are also some webhooks to better integration with external services:
- tito — when a user registers, this records the registration locally and sends them a magic link to login straight away.
For deployment, the server is split into this server and sockets to allow each to scale seperately. This is done through reverse-proxying to multiple deployments of the same container with different configurations.
Technically part of the server, but split out here because they are deployed seperately. The sockets provide live interactions on the plaza using socket.io that is replicated using the cache.
The main example of the live interactions is the "live visitors" widget on the Plaza homepage, sockets has been used for other liveness including live interpretation of events.
The sockets are also used for in-Plaza metrics to collect information on how the Plaza is used.
Jobs are scripts within the server to perform specific jobs on a regular basis. They are run via a CLI that is designed to be deployed as Kubernetes CronJob resources. This measn the same server image can be used to run any of the jobs.
They may use a cache-based semaphore to prevent concurrency.
The fetch-schedule job queries the Pretalx API for the latest information, formats it ready to be served by the server and puts it into the cache for quick access. This job runs all-in-one so all information is pulled down that is needed, it is parsed and formatted into deconf structures and it completely replaces whatever was in cache.
The fetch-users job queries the TiTo API for registrations since the last scrape. If there wasn't a "last scrape" it queries for all registrtaions. If new registrations are found, it merges them into the existing copy in the cache. Once the registration is in the cache, the user can log in to the Plaza.
The fetch-content job clones the content repository (originally it was a different git repository) parses out the configuration and copy and puts them into the cache.
The ./content contains configuration for the plaza and localised copy for the client. The fetch-content job gets these static files into the cache to be served by the server and rendered by the client.
The use of this structure is to allow flexibility in the content and copy on the site. This means that a new version of the app doesn't have to be released to change and update the copy or configuration of the site and the copy and configuration can easily be changed through the GitHub web interface.
The settings.json config is used throughout the client:
- to determine which pages are turned on or visible
- to configure the widgets on the atrium
- to toggle navigation features in the main navigation bar
- to set the main video on the Plaza homepage
Copy is localised markdown files in folders. For example atrium/en.md and atrium/es.md are the atrium copy with two localisations, English and Spanish.
There is then an endpoint in the server to request the copy for atrium.
The mozfest-config defines how to deploy entire instances of the plaza using an infrastructure-as-code approach. More info in Deployment.
Deconf is a collection of libraries for running virtual conferences, broken up into the domain they run in (i.e. frontend vs backend).
The frontend library of deconf provides vue.js components and logic to do most of what the Plaza does in-browser. It's designed to be customisable but it could be better.
The library provides the UI components and base styles for most of the Plaza, exposed in a way that allows them to be customized by library consumers.
The backend library of deconf provides logic to process information and interact with the database or cache. It's also the logic for most jobs.
This bit of deconf is for sharing code and TypeScript types between the front- and backend libraries
The cache is a Redis database to store information for quick retrieval by the server. It is designed to work as much as possible without it but things will be broken if certain keys haven't been set yet.
The cache is also used for semaphores in some occasions to prevent concurrency when it is needed. e.g. when database migrations are happening or for certain jobs.
The cache is also used with by the sockets to allow it to scale horizontally. Messages are broadcasted across the server and key-values are stored to allow this.
The database is a Postgres database to store things that persist, like users or which sessions they've added to their schedule. The database uses migrations to setup the schema which are run automatically when a container starts up. You can find the base migrations here.
The idea is that features for deconf are developed in the deployments that use the libraries then they are contributed back to the OSS project in the most configurable way possible. The OSS project should be for any conference and not include deployment (MozFest) specific things.
The client and server share a version number and their containers are built at the same time.
The repository as a whole uses the top-level package.json as the version. When a new version is pushed to GitHub, it automatically builds the containers and pushes them to the container registry, ghcr.io.
The next version is automatically determined by the commits that have happened since the last version. To do this, commits follow conventional commits. This is set up with standard-version and commitlint to ensure the commit messages are correct and to run the release.
The deployment of different instances of the plaza are defined as Kubernetes manifests using Kustomize. All of the configuration is in the mozfest-config repository.
Once a new version is released, it can be deployed to an instance by modifying it's images section in the kustomization.yml then applying the configuration to Kubernetes.
There has been automatic deployments of these manifests to the cluster but it ended up slowing rapid changes down. For now, these manifests should be the source of truth to deploy from.
In the config repository, there is the base folder which is what each instance starts with.
It contains a blank-ish set of Kubernetes manifests to deploy the instance:
- A Deployment+Service resource for the client
- A Deployment+Service resource for the server
- A Deployment+Service resource for the cache (sometimes a managed redis server has been used)
- A Deployment+Service resource for "sockets", a server container but only exposing the the socket.io stuff
- CronJob resources for the jobs
- A ConfigMap with the DigitalOcean root certificate authority needed to connect to a DO database
Overlays are used to create instances from the "base" configuration. They customise the resources above and add new resources too. This means there can be a production, development or staging instance within the same Kubernetes cluster.
production
- Specific ConfigMaps are created to configure the containers to be production
- ExternalSecrets are added to mount in secrets from an external secret store, more info in dependencies.
- HorizontalPodAutoscalers are used to automatically scale the number of pods serving the client, server or sockets.
- There are some http redirects served by a plain nginx container
- The schedule of the jobs may be configured to change the frequency around key events or times.
- Ingress resources are added to expose the containers to the world wide web.
- The version of container images to use is set in the
kustomization.yml
other instances
There have been other ephemeral instances, e.g. to test features, which use the same approach as production with different images and probably without the HPA resources.
The resources above depend of a few things in the cluster.
For Ingress resources, there needs to be an IngressController in the cluster, ingress-nginx has been used with success so far.
For secret management, ExternalSecrets needs to be installed into the cluster and configured to talk to the secrets provider being referenced. This means all configuration for instances can be defined in the.
Cert Manager is expected to be installed and if so it will automatically create Let's Encrypt certificates to satisfy the Ingress resources.
HorizontalPodAutoscalers only work if there is something to provide metrics to the Kubernetes scheduler, we use metrics-server to provide them.
The plaza pulls information from two external sources: Pretalx and Tito.
The plaza pulls the schedule from pretalx, effectively using it as the content CMS.
Deconf provides PretalxService which is used to query the Pretalx API with a configured PRETALX_API_TOKEN environment variable.
The plaza currently uses these endpoints:
This whole process is run through the CLI's fetch-schedule command which itself is ran as a Kubernetes CronJob. The frequency of the cronjob is often changed to be more-frequent during a live event and less frequent when not.
There is also an administration tool to force the schedule to update that can be ran by allow-listed people on the Plaza.
This list of people is based on a list of email addresses defined in the JSON configuration that gives them an admin role when they sign in.
As an admin, you can visit the plaza with ?dev on the end of the URL and there are special controls including a button to run the fetch-schedule command.
The plaza pulls the people that are registered for the live event from Tito using it's API. There is also a webhook that is configured through Tito to notify the plaza of new registrations.
API
Through the CLI, like with Pretalx registrations are fetched with fetch-users which is ran as a Kubernetes CronJob.
The fetch is differential so only tickets are fetched since the last fetch occured. This is to speed everything up as this job is designed to run very frequently, e.g. up to every 2 minutes.
These endpoints are used:
Webhook
The webhook is a newer integration which was added for MozFest 2023. It runs alongside the regular scrape to speed up the registration-to-sign-in flow. When the webhook is triggered it automatically sends the participant the magic link email to get them straight into the Plaza.
Webhooks:
https://schedule.mozillafestival.org/api/tito/webhook/forticket.completed