Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(data-warehouse): data warehouse logic/data refactor #22254

Merged
merged 11 commits into from May 22, 2024

Conversation

Gilbert09
Copy link
Contributor

@Gilbert09 Gilbert09 commented May 10, 2024

Problem

  • The frontend logics for the data warehouse product were getting very messy, lots of intertwined and repeated "logic" going on everywhere
  • We were also calling 3 different endpoints to get the correct data to render the warehouse scene (all returned with different formats that we mangled together)

Changes

  • Builds onto the DatabaseSchema HogQL query and extends it to support all table types along with a new response structure to fulfill all the frontend needs
  • Refactored the frontend logics to use the new endpoint
    • Moves all data querying into a single logic
    • Removes a bunch of redundant code
    • Removes a series of similar typings and uses a single type for all components
    • Updated the product analytics integration to use the new data structure
    • Renamed the SavedQueries logic

TODO

  • Add some backend tests for the DatabaseSchema query endpoint
  • More UI testing

Does this work well for both Cloud and self-hosted?

Yes

How did you test this code?

  • Still testing...

@Gilbert09 Gilbert09 removed the stale label May 21, 2024
@PostHog PostHog deleted a comment from posthog-bot May 21, 2024
@Gilbert09 Gilbert09 changed the title WIP: data warehouse logic/data refactor chore(data-warehouse): WIP: data warehouse logic/data refactor May 21, 2024
Copy link
Contributor

github-actions bot commented May 21, 2024

Size Change: 0 B

Total Size: 1.05 MB

ℹ️ View Unchanged
Filename Size
frontend/dist/toolbar.js 1.05 MB

compressed-size-action

@Gilbert09 Gilbert09 changed the title chore(data-warehouse): WIP: data warehouse logic/data refactor chore(data-warehouse): data warehouse logic/data refactor May 21, 2024
@Gilbert09 Gilbert09 marked this pull request as ready for review May 21, 2024 19:07
@Gilbert09 Gilbert09 requested a review from EDsCODE May 21, 2024 19:07
Copy link
Member

@EDsCODE EDsCODE left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did a bit of QA and looks good! Glad you're getting this refactor in before beta release. Left a comment about tests on one function


def serialize_database(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we test this function at all? Seems like a good bit of logic we want to keep set

@@ -1,99 +0,0 @@
import { afterMount, connect, kea, listeners, path, selectors } from 'kea'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hah finally this horrible name is gone

@Gilbert09 Gilbert09 merged commit 388b6a2 into master May 22, 2024
83 checks passed
@Gilbert09 Gilbert09 deleted the tom/warehouse-ui-refactor branch May 22, 2024 14:36
Copy link

sentry-io bot commented May 22, 2024

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

  • ‼️ ValidationError: 1 validation error for DatabaseSchemaSchema /api/projects/{parent_lookup_team_id}/query/ View Issue
  • ‼️ ValidationError: 1 validation error for S3Table posthog.tasks.tasks.process_query_task View Issue

Did you find this useful? React with a 👍 or 👎

timgl pushed a commit that referenced this pull request May 24, 2024
* WIP

* Removed unused type

* Added view support

* Get join info working better for external tables

* Updated mypy

* Added tests

* Updated mypy

* Fixed tests

* Fixes for visual tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants