About Uesio

Uesio is a low-code application development platform.

Code style

As much as possible, our code style and format is codified with eslint and Prettier. We cherry-picked some rules from the Airbnb JavaScriopt Style Guide, Airbnb React/JSX Style Guide and the React+TypeScript Cheatsheets.

For Redux, we follow the Redux Style Guide with some exceptions. More details on that here.

For Go package naming, we follow this guideline.

Tech Stack

Backend

• Cobra. CLI for Go application.
• gorilla/mux. Web framework in Go.
• goja. JavaScript engine implemented in Go.

Frontend

• Node.js. For package management, building process, and development.
• TypeScript. For strong typing of JavaScript code.
• React. Framework for UI components.
• Redux. State store for the application's frontend data.
• Redux-toolkit. Bootstrap for Redux.

Redux architecture

See the Uesio Specific Redux Docs.

Monorepo structure

The present monorepo hosts several standalone applications, such as the cli.

Standalone libraries are located in the libs folder. These libs are components of the applications or container for sharing code between applications and libs.

The monorepo is managed by a tool called nx. With nx, there is a single package.json for the whole monorepo.

Set up dev environment

Required

• Install homebrew (for macOS user)
• Install git
• Install GitHub Desktop GitHub Desktop
• Install nvm (for ensuring that your version of Node.js matches the version used in the repo): nvm install
• Install Go
• Install the following brew packages:
  • hurl (for integration tests): brew install hurl
  • jq (for JSON manipulation in Shell): brew install jq
  • wget (for fetching URLs): brew install wget
• Start dependencies here.
• Create a symlink for the Uesio CLI into your bin (NOT an alias, which won't work with nx):
  • Mac OS: sudo ln -s ~/git/uesio/dist/cli/uesio /usr/local/bin
  • Windows: mklink C:\bin\uesio C:\Users\<USERNAME>\git\uesio\dist\cli\uesio, and ensure bin is on path: setx PATH "C:\bin;%PATH%
• Build and run here.

Optional

• Set up SSL here. If you don't set up SSL locally and you still want to run multiple sites locally in addition to the ues.io studio, you will need to set the UESIO_ALLOW_INSECURE_COOKIES environment variable to true
• Set up local DNS here This is also necessary if you want to run multiple sites locally in addition to the ues.io studio. By default, you can access the studio at http://localhost:3000
• Install VS Code and plugins (ESLint, Prettier, Go, GitLens). Do enable format on save in conjunction with the Prettier. Set up the code environment variable.
• Install the following Google Chrome plugins : React Developers Tools, Redux DevTools.
• Install Oh My Zsh
• Add a SSH key to your github account
• Install the nx cli globally: npm i -g nx
• An alternative to installing nx globally is to set an alias in your ~/.zshrc file or equivalent: alias nx="npx nx". This way your global nx version will always be the correct version.

    npm run dev

• Optional. Create a file called launch.json located in apps/.vscode for the uesio server debugger in Go and paste the following :

{
  // Use IntelliSense to learn about possible attributes.
  // Hover to view descriptions of existing attributes.
  // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
  "version": "0.2.0",
  "configurations": [
    {
    "name": "Launch",
    "type": "go",
    "request": "launch",
    "mode": "debug",
    "program": "${workspaceRoot}",
    "env": {},
    "args": ["serve"]
    }
  ]
}

Build

• Download and install the npm module dependencies

npm install

Build all applications and libs

npm run build-all

Build a dedicated app (no watcher and no source map)

cd ./libs/apps/uesio/studio && uesio pack

// or
npm run nx -- build apps-uesio-studio

// or, if you have nx installed globally (or aliased)
nx build apps-uesio-studio

Watch mode (for development)

While developing you may want the entire monorepo to rebuild changed files. You can do this with:

npm run watch-all

Start dependencies

1. Launch all local dependencies (e.g. Postgres) with Docker Compose:

docker compose up -d

2. Seed your local Postgres database with everything Uesio needs for local development

npm run migrations
npm run seeds

Run the web application locally

npm start
open http://localhost:3000

If you have SSL and local DNS configured, you can access the Studio via the following "local" DNS:

open https://studio.uesio-dev.com:3000

To run the app in Docker locally:

npm run in-docker
open https://studio.uesio-dev.com:3000

NOTE: Docker Compose aggressively caches, so to force the app to rebuild the image (e.g. to rebuild JS / Go source), use this instead:

npm run in-docker-force-build

Set up SSL

SSL is optional for local development. It is enabled using by setting the environment variable UESIO_USE_HTTPS=true

npm run setup-ssl

This script should create the certificate.crt and private.key files in the apps/platform/ssl directory.

On Windows/Linux, you will need to manually trust this self-signed certificate. On Mac OS, this is done automatically.

• Windows: double-click certificate.crt in the File Explorer. Click "Install Certificate..." Then place the certificate in the "Trusted Root Certification Authorities".

Set up your local DNS

If you just want to work in the Uesio Studio site, local DNS setup is not necessary, you can just access "http://localhost:3000".

However, to simmulate how Uesio routes DNS domains and subdomains to Uesio sites, you need to configure your OS to properly route "local" DNS domains (e.g. "uesio-dev.com") to the Uesio app server running on localhost.

There are two ways to do this, you'll need to pick one:

1. Modify /etc/hosts directly

   On Mac/Linux, modify the /etc/hosts file to resolve local subdomains to 127.0.0.1

   bash ./scripts/seed-etc-hosts.sh

2. Use DNSMasq

   brew install dnsmasq
   

   The installation process will output several commands that you can use to start Dnsmasq automatically with a default configuration. I used the following commands but you should use whichever commands brew tells you to:

   sudo brew services start dnsmasq
   

Worker jobs

There are a number of worker jobs which Uesio runs in production, to handle things such as usage event aggregation, daily invoice generation, and potentially other future use cases (such as scheduled bots).

To run the worker process, use npm run nx -- worker platform (Or nx worker platform if you have nx installed globally):

> nx run platform:worker

{"message":"Running Uesio worker process","severity":"INFO"}
{"message":"Scheduling job Invoices with schedule: @daily","severity":"INFO"}
{"message":"Scheduling job Usage with schedule: * * * * *","severity":"INFO"}
{"message":"Finished loading all jobs, starting scheduler now...","severity":"INFO"}
{"message":"Cron job Invoices (1) next run will be at: Mar 22 00:00:00","severity":"INFO"}
{"message":"Cron job Usage (2) next run will be at: Mar 21 20:09:00","severity":"INFO"}

(Optional) Environment Variables

The following environment variables can optionally be configured in your Shell (e.g. in ~/.zshenv if you are using Zsh)

Environment Variable	Description	Default	Examples, Values, and Help
UESIO_USE_HTTPS	Whether or not to serve with TLS	false	true / false
HOST	Host to use for HTTP server	""	Set to "localhost" for local development
UESIO_PRIMARY_DOMAIN	The primary domain to use for site identification purposes (e.g. for ues.io cloud, this is "ues.io")	localhost (or "uesio-dev.com" if `UESIO_DEV=true`)	If running ues.io on your own infrastructure, set to a 2-part TLD that you own.
UESIO_SESSION_STORE	Allows you to specify the storage location for user sessions	redis	redis, filesystem, ""
UESIO_USERFILES_BUCKET_NAME	The Bucket in AWS / local folder where user-uploaded files will be stored.	""
UESIO_BUNDLES_BUCKET_NAME	The Bucket in AWS / local folder where bundles will be stored.	""
UESIO_STATIC_ASSETS_HOST	Host from which to serve static files, including vendored JS (React) and Uesio assets	""	By default, assets are served from the local filesystem / Docker container. Alternately, set this to a valid URL where Uesio static assets live, e.g. "https://www.ues.io"
UESIO_ALLOW_INSECURE_COOKIES	Allows cookies without the secure flag	true	Useful in local docker development
UESIO_PLATFORM_FILESOURCE_TYPE	Controls where user-uploaded files are stored	uesio.local	Either "uesio.local" (filesystem) or "uesio.s3" (store in AWS S3)
UESIO_PLATFORM_FILESOURCE_CREDENTIALS	The name of the Uesio credential to use for saving user-uploaded files	uesio/core.aws	Must be a fully-qualified Uesio credential name
UESIO_PLATFORM_BUNDLESTORE_TYPE	Controls where Uesio bundles are stored	uesio.local	Either "uesio.local" (filesystem) or "uesio.s3" (store in AWS S3)
UESIO_PLATFORM_BUNDLESTORE_CREDENTIALS	The name of the Uesio credential to use for saving bundlestore files	uesio/core.aws	Must be a fully-qualified Uesio credential name
UESIO_DEV	Enable various features for use in local development of Uesio	false	Set to "localhost" for local development
UESIO_DEBUG_SQL	Enable detailed SQL query debugging	false	If enabled, all Wire loads will return a `debugQueryString` property containing the SQL queries made
UESIO_MOCK_AUTH	Enables you to login with mock user accounts (which can be specified with `UESIO_MOCK_AUTH_USERNAMES`)	false	Only for local dev / unit tests
UESIO_MOCK_AUTH_USERNAMES	A comma-separated list of usernames to use for mock authentication (requires `UESIO_MOCK_AUTH=true`)	ben,abel,wessel,baxter,zach,uesio	Only for local dev / unit tests
UESIO_GRACEFUL_SHUTDOWN_SECONDS	The number of seconds to wait before terminating the Uesio app / worker process	5	Should be less than whatever the ECS / Kubernetes / etc shutdown window is (usually 30)
UESIO_USAGE_JOB_RECURRENCE_MINUTES	The number of minutes to wait between runs of the Usage worker job	10	Usage data (stored in Redis) will only be aggregated and committed to Postgres as often as this job is run by the worker process. Set to a lower window for more frequent checks.

In addition, all Uesio Secrets can have their default value set by setting a corresponding UESIO_SECRET_<namespace>_<name> environment variable. Any value set for these secrets in a Site/Workspace will override the environment variable default, but it can often be useful, especially for local development, to configure a default value, so that you don't have to populate these secrets in every site. (Note: there is no corresponding feature for Config Values, because you can define a Config Value's default directly in the metadata definition).

For example, the uesio/core.sendgridkey secret's default value can be configured with export UESIO_SECRET_UESIO_CORE_SENDGRIDKEY=your-send-grid-key

npm dependencies

As mentioned in the monorepo section, a single package.json file describes the npm dependencies for the whole monorepo.

All npm modules we used are installed as development dependency since uesio is not intended to be released as standalone npm module.

Migrations

We use golang-migrate package for running SQL migrations. This package maintains the current state of migration runs via a schema_migrations table.

Migrations can be run against your local using npm run migrations, or you can use ./uesio migrate [up|down] [NUMBER] to manually run a specific number of migrations, e.g. to undo the most recent migration, you can run ./uesio migrate down 1.

To run migrations in ECS, create a new one-off Task, using the latest Task Definition for "uesio", and modify the command to be ./uesio,migrate (or ./uesio,migrate,down,1) to undo one migration)

adding migrations

New migrations can be created using npm run migrate:create -- <SOME_NAME>

manually setting the migration "pointer"

To forcibly set the migration version to latest (currently 4), you can either use pgcli or some other DB tool to manually run the command update schema_migrations set version = 4, dirty = false against your database, or use this (assuming you install golang-migrate with brew):

brew install golang-migrate
export CONN_STR="postgres://postgres:mysecretpassword@localhost:5432/postgresio?sslmode=disable"
migrate -path apps/platform/migrations -database "$CONN_STR" force 4

This will skip running any migrations but update schema_migrations table to think you've run them all up through 4

testing migrations

To test running migrations (against a separate pgtest database alongside your main postgresio database for dev), do the following (run from THIS top-level directory!):

docker compose up -d
bash apps/platform/migrations_test/test_migrations.sh

End-to-end Testing and Integration testing

To run E2E and Integration tests locally, there are a number of commands available:

1. npm run tests-all
   • Runs all Integration and E2E tests against your local Uesio app.
   • Use this when writing and debugging tests locally
2. npm run tests-ci
   • This is what we run in Github Actions on master branch builds. It spins up all dependencies, and a Dockerized version of the Uesio app, runs integration and E2E tests against the app, and then spins down all Docker containers.
3. npm run tests-integration
   • Runs just the Integration Tests (against your local app).
4. npm run tests-e2e
   • Runs just the E2E Tests (against your local app).

TO run just an individual E2E or Integration test, see the sections below.

E2E testing with Cypress

We use Cypress for writing end-to-end tests of the Uesio app. All E2E tests are defined in cypress/e2e directory.

E2E tests are the most expensive and most brittle, and as such should be used sparingly.

If you're running Uesio locally, you can use npx cypress open to launch Cypress' visual UI for running tests, or npm run tests-e2e to just run the tests in a headless runner.

Running a single E2E spec

If you want to visually run a single spec, use the Cypress visual UI and then select the individual spec.

Or, use npx cypress run --spec <path to spec> to run a specific file in a headless Electron instance, e.g.

npx cypress run --spec cypress/e2e/builder.cy.ts

Integration / API testing with Hurl

We use Hurl for running integration tests against Uesio APIs, and for performing load testing against APIs. Hurl provides a powerful text-based abstraction over curl suitable for defining suites of HTTP requests and assertions to make upon the responses.

To run API integration tests locally against your running Uesio container, use npm run tests-integration

Running a single Integration Test

The easiest way to run a single Integration Test is to go into the run-integration-tests.sh file and comment out the lines where we run all tests, and uncomment the lines here:

hurl --very-verbose -k --variable host=studio.uesio-dev.com --variable domain=uesio-dev.com --variable port=3000 hurl_specs/wire_collection_dependencies.hurl

You could run this from the CLI, but you would have to make sure that you are (a) in the right directory (b) have the right environment variables set up. (See the top of this run-integration-tests.sh for a better understanding).

Continous integration (CI)

We use GitHub Actions for automated builds. All of our workflows live in ./github/workflows.

Creating a new release

We use Github Releases to manage releases of:

(a) the Uesio web/worker app - as a Docker image (b) the Uesio CLI - as a platform-specific binary

Here are the steps to create a new release:

1. In Github, go to Draft a new release
2. Enter a new tag name, using Semver names (e.g. v0.5.0)
3. Click Generate release notes
4. Click Publish release

That's it! This will kick off the "Release" Github Action, which will download the corresponding Docker image from AWS ECR and re-publish it to Github Container Registry with the corresponding version tag, as well as the latest tag. It will also generate CLI binaries for Linux, Windows, and Mac OS.