feat: Data Connectors (SQL)

[davidkircos]

Goals
Add a SQL connection
Support Postgres, MySQL
Connection Code Editor
Run code to get data in sheet
Basic schema viewer

• Update branch to main @jimniels
• Merge Connector Service branch @ddimaria

Client

• List connections @jimniels
• Create a connection @jimniels
• Test a connection @jimniels @ddimaria
• Edit a connection @jimniels
  • Done but has security considerations, like which fields are editable based on being sensitive vs. not
• Delete a connection
  • We'll have to handle the lifecycle of a connection, e.g. somebody adds a connection to a file, runs a query, then later deletes that connection so when you run the query again, what should the user see?
• Select a connection in cell type menu @jimniels
• Code editor connection
• Pull data
• Display schema
• Fix all TODO: (connections) in the PR
• Add mix panel events where it makes sense for connection-related interactions

Core

• Add Connector cell type @ddimaria @davidfig

API

• Endpoints for creating, retrieving, and editing connections @jimniels
  • POST /connections for { type: 'postgres' } or { type: 'mysql' }
  • GET /connections
  • GET /connections/:uuid (TODO: should not return sensitive info)
  • PUT /connections/:uuid
  • GET /internal/user/:auth0Id/connections/:uuid (returns sensitive info)
• Permissions for connections
• Tests for above endpoints
• [blocked] Securely store credentials @davidkircos
• Security considerations for sending sensitive info vs. non-sensitive info

Connections Service

• Create connections service (@ddimaria) Axium, Docker, Etc [ similar to Files and Multiplayer ]
• Endpoint for testing connection - Postgres
• Endpoint for retrieving data - fetch credentials from API - Postgres
• Postgres support for schema retrieval
• Mysql support for testing, querying and schema retrieval
• Add supported databases to docker and include in tests

Deployment

• Static IP for server so people can whitelist
• Create/publish docs for connections

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
quadratic	❌ Failed (Inspect)		May 23, 2024 11:05pm

[jimniels]

@ddimaria minor, but can we rename id to uuid? It is a UUID string and that's how it's represented in the API, so it makes the client code a bit more uniform when we're working with this, e.g.

connectionsFromApi.find(connection => connection.uuid === editorInteractionState.mode?.Connection?.uuid)

[jimniels]

Actually, we refer to the kind as type in the API as well, might be worth aligning on this language as wel...

[ddimaria]

Rustland used kind instead of type b/c type is a reserved word. I can look into rewriting during serialization.

I have a small opinion about not changing id to uuid (seems like a database antipattern to have two unique identifiers...a UUID type in Postgres is idiomatic for the PK, I think we use the id as an index or ordering mechanism?). Also, calling it a uuid feels a bit like Hungarian notation.