Data Science Portal

This repository provides a reference implementation for a self-service portal to deploy and manage GCP projects and infrastructure.

Table of Contents

• Developer Guide
  • Setup
    • Bootstrap the Cluster
  • ...
• Technical Design
  • Authentication in Backstage
  • Integrations
    • GitHub
  • ...

Developer Guide

Setup

Bootstrap the Cluster

Before we can use Backstage or the infrastructure templates we need a cluster deployed with Config Sync for GitOps, Crossplane for the control plane, and additional infrastructure to build and run Backstage. This is only required the first time a cluster starts up.

The process to bootstrap the cluster is documented in bootstrap/README.md.

Technical Design

Authentication in Backstage

• Using the Google Authentication Provider which uses OAuth Credentials.
• The Sign-in Identities and Resolvers docs state that we must explicitly configure the provider to map external identities to user identities in Backstage. This is not provided by the Google Authentication provider.
• Options:
  • A custom sign-in resolver with domain validation that populates User entities in the Catalog
  • Use the Google Secure LDAP service to get LDAP Organizational Data

Integrations

GitHub

We use a GitHub integration to:

• Create PRs from a template

Backstage can be configured to use GitHub Apps for backend authentication.

Installation

This project is a monorepo comprising multiple modules, each with its own set of prerequisites for installation. Please refer to the specific installation instructions for each modules below.

Prerequisites

Before proceeding with installation, ensure you have the following tools installed:

• Taskfile: A task runner for your project.
• Node.js v20: JavaScript runtime for executing applications.
• Yarn: Package manager for Node.js projects.

Taskfile

Install Taskfile globally using Yarn:

yarn global add @go-task/cli

For alternative installation methods, please refer to the Taskfile documentation.

To list available tasks, run:

task --list

Node.js

Install Node.js using nvm (Node Version Manager):

nvm use

Verify the installation by checking the Node.js version:

node -v

Yarn v1 (Classic)

Backstage uses Yarn v1. It can be installed globally using corepack:

corepack enable
corepack prepare yarn@1.22.19 --activate

or installed globally:

npm install --global yarn@1.22.19

Verify the installation by checking the Yarn version:

yarn -v

With these prerequisites installed, you're ready to proceed with the installation of individual services within the monorepo.

Modules

• backstage: Module for managing infrastructure tooling, services, and documentation
• bootstrap: Module for bootstrapping Google Kubernetes Engine (GKE) cluster
• root-sync: Module for Config Sync to manage root-level resources in Google Kubernetes Engine (GKE) cluster
• budget-alerts: Module for managing Google Cloud Budget alerts