Replies: 4 comments 4 replies
-
|
@kislerdm thank-you for this detailed vision for taking SuperDuperDB to the data-mesh. There seem to be some clear takeaways here:
Some comments on this:
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks for your comments Duncan. I feel that you extracted the key takeaways I wanted to convey with the document 🙂 There are still many questions to be answered as pretty much whole RFC is based on assumptions.
It's not a problem, any tech commitment is not beneficial at this stage - we carry on with Rest. We can come back to reassessment if its versioning maintenance costs becomes a bottleneck at some point.
Unfortunately, elegant is not necessarily secure, so we shall rather commit to separating input and output. It will help us to avoid any security, or data integrity issues caused by the source data mutation. For example, the output data can still be stored as a mongo document, but to a different collection to segregate read and write circuits following CQRS. Moreover, we will likely need to collect data from multiple collections/tables to run predictions.
It's a very reasonable approach. However, unfortunately dask is not there yet when it comes to production readiness.
One of the points pro Ray is that it can integrate with Dask giving us known interface for the time being. Later on, we will migrate from Dask completely to simply the system.
Definitely. Ray provides interfaces to integrate with many MLOps tools and some observability tools like grafana. We can start there and extend it as plugins to SDDB. |
Beta Was this translation helpful? Give feedback.
-
BLUFThe key takeaways:
|
Beta Was this translation helpful? Give feedback.
-
|
I love the idea of introducing RFCs and doing it in a public way like this, fantastic work @kislerdm. A lot of what you wrote really got me thinking and out of my comfort zone at times, I really appreciate this. Thanks for taking the time to write this! 🙏 I don't seem to be able to add comments to the various sections (the only drawback that I can see with using GitHub discussions) so I've added a bunch of comments ordered by section below. Most of the comments are mainly meant as food for thought - ways that perhaps the RFC could be refined, but hopefully with enough wiggle room that there are multiple ways to address them. Context
Proposed Solutions
Data Processing Ways
SDDB cluster orchestration
Technical decisions
General |
Beta Was this translation helpful? Give feedback.
-
|
There is something called dask on ray, we can look it as well! |
Beta Was this translation helpful? Give feedback.
-
|
@nenb Hey Nick! Thanks a lot for your feedback! For context, the RFC was aiming to convey a version of the mid-/long-term vision of SDDB platform development in order for us to keep thinking in that direction, and to use this document as a data point. BTW I invite everyone to contribute to it so we converge eventually ;)
The RFC outlines a way for SDDB to become production ready according to the mentioned assumptions. I'll revisit it to add the status quo - thanks for proposing.
Thanks. Point taken. I'd be happy to share my knowledge about diagrams ;)
It will proxy database request to SDDB. The purpose: to enable users to use vector search by querying data from the database. SDK has a similar purpose, but to enable applications to schedule jobs by communicating to SDDB cluster over network. The rationale: to enable teams using the languages other than python.
Dask is a handy instrument for those using PyData stack indeed, but unfortunately it has a huge flaw: lack of mechanism for the scheduler to recover in case of failure (please find details here).
The problem I proposed to solve by employing ray is scalability. We can solve scalability in many ways. I invite you to join me to explore our opportunities. WDYT: shall we commit to Ray, could it facilitate SDDB adoption by the orgs using Databricks?
It was beyond my assessment. The problem I was exploring was scalability. Could you please share you opinion: does Dask fit our AI needs better than Ray?
It's unfortunately the only pragmatic way to avoid avoidable responsibility we were to incur shall we move customers data outside of their perimeter. Anything else requires complex network configuration which is still more risky than deployment to the customer's env. What would you propose?
Indeed :) Although, to reiterate, I see the following arguments pro gRPC:
What do you see as the main blocker for us to commit to using gRPC?
Unfortunately, it's a "chicken-egg" problem: we cannot deliver until there is a customer's input, customer cannot give input until we deliver :) I feel that we shall definitely act proactive, increment, but not over-commit. For example, an alternative to this RFC's proposal would be deploy SDDB using "serverless" to target SME. WDYT? |
Beta Was this translation helpful? Give feedback.
-
|
I think the
Obviously 1. is the crucial argument here, but I think it's probably the case that we have more internal experience with Beyond the short-term (which I know this document is targetting) I'm not sure what is the best option to go for. The only strategy I really have is that we try to list the arguments 'for' and 'against' each pacakge, and make a decision based on that. That is why I think it's so important that the rows in your original table reflect the correct information (see my previous comment for some rows I do not believe to be correct).
I agree that this is a design choice of My personal approach for now is to stick with
Sorry, I just realised that the link I posted earlier to the relevant comment on this is broken. :ups Here is the correct link. Consulting, Funded Development and Support (to name a few) are all other alternatives IMO.
I admit I don't have the experience to confidently say we should go with one or the other. My current reasons in favour of REST are:
Perhaps - I would need more details! But I like the fact that we are considering for SME. For me, our core users are single devs and small organisations, these are the people I want to support and enable over the coming months. |
Beta Was this translation helpful? Give feedback.
-
|
@nenb thank you for your feedback Nick! Based on the argument, I feel in favor of the decisions you shared:
The key: to be "lean and clean", and be ready to execute effectively without over-commitment to a particular decision. |
Beta Was this translation helpful? Give feedback.
-
Table of contents
Context
Following Data Mesh approach, stream-aligned teams will provide domain data as a product. Such transition poses a
bottleneck because the teams, who own the domain logic, may lack extensive expertise in analytics and machine learning.
To address this challenge, SuperDuperDB offers the system to enable stream-aligned teams to delivering data as a product
by applying advanced analytics and machine learning to the business domain's data directly without dependency on cold
data storage a/k/a data lake.
SuperDuperDB solves the three classes of problems:
Problem Statement
The document is intended to address SuperDuperDB product readiness for successful integration into a running system.
Production readiness requirements
The SuperDuperDB product readiness strategies are outlined to satisfy the following requirements.
Goals
The documents addresses design of the SuperDuperDB system.
Non-Goals
The document does not cover implementation details, i.e. priorities and timelines.
Proposed solutions
Assumptions
The following assumptions are made to satisfy the readiness requirements:
coupling to SDDB, nor does it dependent on SDDB.
as k8s custom resource.
modification, and prevent PII leak.
Topology
SuperDuperDB is the stateful application with three inbound dependencies and one outbound dependency.
Inbound dependencies:
Outbound dependency:
Data processing ways
SuperDuperDB operates in two ways for data processing, batch and streaming:
Batch processing is executed asynchronously, or synchronously upon submission of a job to process a batch of data.
The job is submitted follow to the request-response protocol, with the request carrying job configurations, and
the response carrying job's metadata, or its results.
Streaming is executed asynchronously after the SDDB system consumed a pubsub event with the job submission details.
The job result is published back to a dedicated topic/stream.
Batch processing
The mode is used for model training and batch prediction.
There are two scenarios to trigger a job's execution:
The diagrams below illustrate the execution flow.
In-database call
sequenceDiagram title Prediction job in database, sync actor User User->>Database: Executes command Database-->>UDF: Proxies UDF->>+SDDB: Requests SDDB-->>Database: Verifies schema and permissions alt happy path Database-->>SDDB: ok SDDB->>Database: Reads data SDDB-->SDDB: Run model/analytics task else unhappy path Database-->>SDDB: not ok end SDDB->>-UDF: Responses UDF-->>Database: Proxies Database->>User: Returns resultssequenceDiagram title Train job submission in database, async actor User User->>Database: Executes command Database-->>UDF: Proxies UDF->>SDDB: Requests SDDB-->>Database: Verifies schema and permissions alt happy path Database-->>SDDB: ok par SDDB->>Database: Reads data SDDB-->SDDB: Run model/analytics task and SDDB->>UDF: Responses end else unhappy path Database-->>SDDB: not ok SDDB->>UDF: Responses end UDF-->>Database: Proxies Database->>User: Returns jobID and statussequenceDiagram title Train job's status check in database, sync actor User User->>Database: Queries job status Database-->>UDF: Proxies UDF->>SDDB: Requests alt job found SDDB->>UDF: Responses job metadata else job not found SDDB->>UDF: Responses error end UDF-->>Database: Proxies Database->>User: Returns resultsProgrammatic call
sequenceDiagram title Prediction and train jobs submission in app, async App->>SDDB: Execute command SDDB-->>Database: Verifies schema and permissions alt happy path Database-->>SDDB: ok par SDDB->>Database: Reads data SDDB-->SDDB: Run model/analytics task and SDDB->>App: Returns jobID and status end else unhappy path Database-->>SDDB: not ok SDDB->>App: Returns jobID and status endsequenceDiagram title Job's status check in app, sync App->>SDDB: Requests job status alt job found SDDB->>App: Responses job metadata else job not found SDDB->>App: Responses error endStreaming
The mode is used for real-time predictions.
sequenceDiagram title Prediction job - streaming, async AppSource->>PubsubSource: Publishes the job definition PubsubSource->>+SDDB: Consumes the job definition SDDB-->>Database: Verifies schema and permissions alt happy path Database-->>SDDB: ok SDDB->>Database: Reads data SDDB-->SDDB: Run model/analytics task else unhappy path Database-->>SDDB: not ok end SDDB-->>-PubsubSource: Ack SDDB->>PubsubDestination: Publishes result PubsubDestination->>AppDestination: Consumes the job resultsArchitecture design
The diagram's code: here.The diagram highlights three types of teams which are expected to interact with SDDB solution. It illustrates
dependencies among SuperDuperDB the running systems owned by the stream-aligned team.
Container diagram
The diagram's code: here.The container diagram illustrates the following critical points:
and models configurations.
SDDB cluster orchestration
Flyte's focus is on orchestration
Flyte: union.ai
Flyte is focused more on facilitating infrastructure topics, while Dask is focused more on facilitating analytics tasks
definition topics. Ray is somewhere in-between. Moreover, it supports interfaces to integrate with both, Dask and Flyte.
SDDB design
Components
Architecturally, the SDDB runner includes five main components:
IAM: the authN/Z component to ensure zero-trust policy;Compute: to define and configure the model logic;Storage: the interface to communicate to the data source;SerDe: data format definition;Observability: observability and model performance metrics.Technical decisions
Communication protocol for communication between the client and SDDB
- Strong contract guarantees
- Contract maintenance
Questions
Integration checklist
One time:
Repetitive:
Implementation
TBD
Open questions
Glossary
References
Beta Was this translation helpful? Give feedback.
All reactions