This part of mds4all is archived for now as it is built on Heroku who cancelled their free tier in 2022.

---

mds4all

Minimalistic and free modern data stack, hence for all.

• one-data-cookie/mds4all-elt:

  • Warehousing: Google BigQuery
  • Orchestrating: GitHub Actions
  • Ingesting: Meltano
  • Transforming: dbt
  • Cataloging: dbt docs on GitHub Pages

• one-data-cookie/mds4all-superset:

  • Visualising: Superset on Heroku

• other:

  • Code-editing: VS Code with:
    • BigQuery Runner
    • dbt Power User
    • dbt Language Server

mds4all-superset

Visualisation layer of mds4all.

Setup

# Set up basics
$ pip install apache-superset
$ pip install psycopg2
$ pip install pybigquery # for BigQuery connection
$ pip freeze > requirements.txt # or just limit to installed above
$ echo "web: superset db upgrade" > Procfile
$ echo "python-3.8.2" > runtime.txt
# add configs to superset_config.py

# Generate secret key
$ python
>>> from cryptography import fernet
>>> fernet.Fernet.generate_key()

# Set up Heroku app
$ heroku login
$ heroku create mds4all-superset
$ heroku config:set SECRET_KEY=<SECRET_KEY>
$ heroku config:set PORT=<PORT>
$ heroku addons:create heroku-postgresql:hobby-dev
$ git push heroku main
$ heroku logs –-tail # check for errors

# Initialise Superset
$ heroku run bash
$ export FLASK_APP=superset
$ superset fab create-admin
$ superset init
$ exit

# Finalise app
$ echo 'web: gunicorn "superset.app:create_app()"' > Procfile
$ git push heroku main
$ heroku logs --tail # check for errors

# Open app
$ heroku open

Limitations

• Sleep after 30 mins of inactivity
• 512 MB RAM
• 4 workers
• Gunicorn server
• No cache (like Redis)
• Database storage with 10K rows and 20 MB

Inspiration

• Chizzy-codes/NBA_Dashboard
• dugjason/superset-on-heroku
• zi-nt/superset-on-heroku