Is this project suitable for use with EdgeDB?

[jweckman]

Having tried out EdgeDB and their (IMO potentially revolutionary) EdgeQL, it would seem like it could work very well with this framework after implementing a simple EdgeQL ODM (maybe?). Do you think you could read this first:

https://jcristharif.com/msgspec/examples/edgedb.html

To sum it up, using msgspec we can already map EdgeDB objects to msgspec Structs as follows:

EdgeDB Schema:

module default {

type Person {
required name: str;
}

type Movie {
required title: str;
multi actors: Person;
}
};

msgspec.Struct types mirroring Schema above:

class Person(msgspec.Struct):
    name: str

class Movie(msgspec.Struct):
    title: str
    actors: list[Person]

After this manual mapping we can already get to a point where we can serialize msgspec structs:

msgspec.convert(edgedb.Object('Dune'), Movie, from_attributes=True)

Having come this far are we done? To me it seems like we would want to auto-generate the msgspec.Struct classes from this already auto-generated dataclass by EdgeDB (Person as example since simpler):

@dataclasses.dataclass
class GetPersonResult(NoPydanticValidation):
    id: uuid.UUID
    name: str | None

Edgedb has a separate repo edgedb-py that auto-generates python code from the EdgeDB queries. To me it seems that it should be trivial to extend the EdgeDB generation code to also create msgspec.Structs.

Given your expertise and your seemingly modular approach and focus on simplicity, could you advise me on whether this makes any sense or if i should use an even lighter framework than yours? Let's just say i would be extremely willing to make some contributions myself if my suggestions here look like they could work.

I have difficulties to contain my excitement because it seems to me like i have discovered a solo developers most powerful stack for making small business applications:

EdgeDB: Database with schema,data, automated db migrations, and a lot more validation than plain old postgres. You get the performance of postgres while getting rid of BOTH the ORM AND SQL (both of which have painful tradeoffs)

Esmerald + Jinja2: What looks to me like the more modular successor to FastAPI with support for the faster msgspec library that incidentally has very good EdgeDB support

HTMX + Tailwind CSS: Get rid of javascript and have almost all of your logic in python with a little bit of light client side js scripting without the horrible garbage called node.js and its hundreds of megabytes of dependencies.

Any commentary on feasibility/design is much welcome. If someone can validate my ideas i would be happy to publish a hello-world template and write EdgeDB/Esmerald binding code under this project.

[tarsil]

@jweckman let me give you some good news. We are more than happy to have support for EdgeDB. I've tried myself and I can't deny my excitement as I do also believe the future of DBs as we know it will pass through this revolutionary technology.

We do appreciate you excitement and words and we hope you enjoy Esmerald as this was always the purpose. The modularity and being like that and supporting msgspec helps you a lot 🙂.

In terms of contributions you are more than welcomed to add what you think it would be a amazing feature to Esmerald. In any way you see fit.

You can even create a plugin for it https://esmerald.dev/pluggables

The world is your oyster. I myself have been thinking how we could bring EdgeDB into Esmerald world and give it to the people to actually use both in a clean manner.

Does this answer your question? 🙂

Having a pluggable would be amazing since you decouple the core of Esmerald and you bring EdgeDB into the world of what we want.

https://esmerald.dev/pluggables

See if this make sense to you? Feel free to suggest, play and let's contribute and develop this idea. I do really like it and I'm sure the community will appreciate your contributions and excitement.

P.S.: I think you will like the approach of the pluggable because this will not only allow you to give you the freedom to play around with this amazing functionality but the integration with Esmerald is.... Modular 🙂

See if this and the docs make sense or if you are thinking of something else. I do love your idea a lot.

[tarsil]

@jweckman i also do like your approach of msgspec with EdgeDB. Basically when the msgspec models are generated you can use them directly with Esmerald as payload in the APIs and delegate validations to msgspec itself. Since Esmerald supports msgspec, that is a massive win and since EdgeDB handles with migrations and whatnot, you have the perfect combination there. Fantastic would be, as suggested, having this as pluggable for Esmerald. I don't want to overstep your toes with this and I do believe you could bring invaluable contributions if your were interested in this.

This is amazing, really.

Another way, if it makes it easy, it's to then write an application with Esmerald and EdgeDB, the bellow world and we could perhaps take it from there? Maybe it can be easier for you? Assembling the app should be ok and then we extract what we want and need and add it into Esmerald? Or you can go through the pluggable route where basically you are the boss and owner of this amazing idea and of course we add you into the documentation and references to that same project/pluggable?

It's up to you. Let me know your thoughts on this

[jweckman]

Wonderful!

Without having any experience with Esmerald i first got the single file hello-world example running and extended it to clarify that my ideas actually work. Since i'm more familiar with FastAPI i checked that using the same concepts as in the code below actually works there with msgspec imported, which is already a very good sign. Below is a semi-pseudocode chronological example that explains how data comes in from edgedb and makes its way out of an example endpoint:

import msgspec
import edgedb
from esmerald import Esmerald, Gateway, JSONResponse, Request, get, status

import uuid
from datetime import datetime


# EDGEDB CLIENT SETUP

def get_edgedb_client(request: Request) -> edgedb.AsyncIOClient:
    return request.app.state.edgedb


# EDGEQL TABLE/MODEL EQUIVALENT (NOT PYTHON - THEREFORE COMMENTED OUT)

# type Payment {
#     annotation description := "Payment";
#     required date: datetime {
#       default := datetime_current();
#     };
#     partner_name: str {
#       constraint max_len_value(50);
#       constraint exclusive;
#     };
#     required amount: float64 {
#     };
#     info: str {
#       constraint max_len_value(1000);
#     };
# }


# AUTO-GENERATED EDGEDB PYTHON QUERY MODEL BASED ON PREVIOUS EDGEQL TABLE/MODEL

import dataclasses
@dataclasses.dataclass
class GetPaymentResult():
    id: uuid.UUID
    date: datetime
    partner_name: str
    amount: float
    info: str | None

# AUTO-GENERATED EDGEDB PYTHON METHOD THAT WE USE TO FETCH DATA

async def get_payments(
    executor: edgedb.AsyncIOExecutor,
    # No arguments for simplicity
) -> list[GetPaymentResult]:
    return await executor.query()  # Contains generated EDGEQL query as string in the real world


# PAYMENTSTRUCT BELOW IS WHAT WE WANT TO AUTO-GENERATE, PERHAPS IN DIFFERENT WAYS. LOOKS LIKE THEY SUPPORT METHODS AS WELL

class PaymentStruct(msgspec.Struct):
    id: uuid.UUID
    date: datetime
    partner_name: str
    amount: float
    info: str | None

    def get_repr(self):
        return f"{self.date}: {self.partner_text} | {self.amount}"  # Just something to demo that we can add methods


@get()
async def payments_endpoint(request: Request) -> dict:
    # For really light weight immediate forwarding or pure data-based parsing, we can use
    # the payments as-is or directly serialize them using msgspec
    payments = await get_payments(
        edgedb.AsyncIOExecutor,
    )

    # In case we need logic, we want to be able to convert the edgedb.object records into something
    # that allows methods and serialization. msgspec.struct should be a perfect fit!
    payment_structs = [
        msgspec.convert(
            payment,
            PaymentStruct,
            from_attributes=True
        ) for payment in payments
    ]
    payments_encoded = [msgspec.json.encode(payment_struct) for payment_struct in payment_structs]

    # Return the results in whatever shape Esmerald expect, not part of the problem
    return payments_decoded


@get(status_code=status.HTTP_200_OK)
async def home() -> JSONResponse:
    return JSONResponse({
        "detail": "Hello EdgeWorld"
    })

app = Esmerald(routes=[
    Gateway(handler=home),
    Gateway(handler=payments_endpoint),
])

Before we make any decisions regarding where to implement, i would first like to understand the overall architecture.

I'm fairly sure that it already would be possible to write production apps based on the example above. I'm also sure that i'm probably doing more than needed by hand and that by leveraging what Esmerald has to offer could cut out some lines from the workflow. My philosophy (and practical experience) is that we want two ways to process the data that comes from our DB:

1. Raw recordsets as dictionaries or dataclasses that can be processed as fast as possible
   AND
2. Ability to convert these recordsets into heavy python objects with lots of logic attached when dealing with records that need a lot of complicated behavior attached to them
   OR
3. Some functional style to group functions that operate on certain types of recordsets. I'm a bit tempted to look for Rust style trait implementations in Python but i don't know if this is very wise to go for. Functional programmers would certainly enjoy having a different way than methods to apply logic to pure edgedb.object recordsets that we get for free from EdgeDB.

If i was to solve my own need naively right now with minimal effort, i would probably look into just extending the edgedb-py repo code to also generate the PaymentStruct for us. Given that they are almost identical i don't think this should take more than a couple of hours to figure out.

What i'm most concerned about is whether this mapping makes any sense when we build a complex query that references 5 different objects in the DB and spits out --something-- that i can't even imagine right now. In theory it should always work because whatever it spits out the auto-generated python mapping should be able to map all our classes, dicts, lists or whatever comes out - no matter how nested. It's obvious from my example above that this should work fine when doing SQL style queries on a single "table" but i don't think auto-mapping would make sens if this falls apart as soon as people start "joining" "tables" in edgeql.

It would be very helpful if you could have a look at this:
https://www.edgedb.com/docs/guides/tutorials/rest_apis_with_fastapi#

and then post a modified pseudo-code example (use my stuff above) of how you think this should be done in Esmerald. From what i've seen i agree that a pluggable probably is the correct way to go but for now i just want verification on the parts that should be involved in the flow.

Looks very exciting already :)

[tarsil]

@jweckman thank you for your detailed answer and I will definitely have a look at this as soon as possible as I need to go through some documents but I can see that we can possibly simplify this with Esmerald, of course. Taking advantage of the handlers will help and if msgspec is already compatible with EdgeDB I don't think nesting is a problem whatsoever. What I do believe we can do it's some sorte of a EdgeDAO

https://esmerald.dev/protocols/?h=dao#dao

And we can even take advantage of the Factory

https://esmerald.dev/references/injector/?h=factory

I will have a look at the examples you provided and see what I can do but it looks very promising indeed.

Esmerald is extremely modular and versatile and allows you to build in the way that makes you feel comfortable with but what you need to make it a route is what you used, a handler (get, post, put....) and a Gateway (what connects the handler with the routing system.

Then you have a lot more things you can do with Esmerald like having sub applications using another Esmerald or a ChildEsmerald, context, Include, injectors, permissions and a lot more. The documentation goes into details on how to build simple APIs to design complex, complete fully fledged applications maintaining the clean design and simplicity of the framework 🙂

[tarsil]

@jweckman as far as I could see, we can separate the logic of the payments inside a DAO or in this case an AsyncDAO and what I think you also want (and I agree) is an abstraction to work directly with Esmerald since its 100% compatible with Esmerald in the format of an ODM. Like this one but for EdgeDB.

Then, because the ODM would be msgspec oriented (and not pydantic as the example in the URL), this would integrate seamlessly with Esmerald as model and validator and the queries would run automatically through it. Is this correct?

What I'm failing to see is why not using EdgeDB ORM directly since they provide basically everything you need?

What you did was not too far from what you can do with Esmerald, an example would also be

import dataclasses
import uuid
from datetime import datetime
from typing import List

import edgedb
import msgspec

from esmerald import Esmerald, Gateway, JSONResponse, Request, get

# EDGEDB CLIENT SETUP


def get_edgedb_client(request: Request) -> edgedb.AsyncIOClient:
    return request.app.state.edgedb


# EDGEQL TABLE/MODEL EQUIVALENT (NOT PYTHON - THEREFORE COMMENTED OUT)

# type Payment {
#     annotation description := "Payment";
#     required date: datetime {
#       default := datetime_current();
#     };
#     partner_name: str {
#       constraint max_len_value(50);
#       constraint exclusive;
#     };
#     required amount: float64 {
#     };
#     info: str {
#       constraint max_len_value(1000);
#     };
# }


# AUTO-GENERATED EDGEDB PYTHON QUERY MODEL BASED ON PREVIOUS EDGEQL TABLE/MODEL


@dataclasses.dataclass
class GetPaymentResult:
    id: uuid.UUID
    date: datetime
    partner_name: str
    amount: float
    info: str | None


# AUTO-GENERATED EDGEDB PYTHON METHOD THAT WE USE TO FETCH DATA


async def get_payments(
    executor: edgedb.AsyncIOExecutor,  # No arguments for simplicity
) -> list[GetPaymentResult]:
    return await executor.query()  # Contains generated EDGEQL query as string in the real world


# PAYMENTSTRUCT BELOW IS WHAT WE WANT TO AUTO-GENERATE, PERHAPS IN DIFFERENT WAYS. LOOKS LIKE THEY SUPPORT METHODS AS WELL


class PaymentStruct(msgspec.Struct):
    id: uuid.UUID
    date: datetime
    partner_name: str
    amount: float
    info: str | None

    def get_repr(self):
        return f"{self.date}: {self.partner_text} | {self.amount}"  # Just something to demo that we can add methods


@get()
async def payments_endpoint(request: Request) -> List[PaymentStruct]:
    # For really light weight immediate forwarding or pure data-based parsing, we can use
    # the payments as-is or directly serialize them using msgspec
    payments = await get_payments(
        edgedb.AsyncIOExecutor,
    )

    # In case we need logic, we want to be able to convert the edgedb.object records into something
    # that allows methods and serialization. msgspec.struct should be a perfect fit!
    payment_structs = [
        msgspec.convert(payment, PaymentStruct, from_attributes=True) for payment in payments
    ]
    [msgspec.json.encode(payment_struct) for payment_struct in payment_structs]

    # Return the results in whatever shape Esmerald expect, not part of the problem
    return payments_decoded


@get()  # defaults to 200 (status code)
async def home() -> JSONResponse:
    return JSONResponse({"detail": "Hello EdgeWorld"})


app = Esmerald(
    routes=[
        Gateway(handler=home),
        Gateway(handler=payments_endpoint),
    ]
)

[jweckman]

First thing to make clear is that EdgeDB never needs an ORM and does not work like one. What you get out may look similar but it is not the same.

Another thing i want to point out is that we strive for never having to write a single query on the python side. Querying is exclusive to edgeql and we will have a much more old-school SQL like structure where instead of ORM style "CRUD on the internal logic layer" we can have, say 50 different complex and simple queries that are related to our Payment object on the edgedb side.

The way i see it we mostly want to push all our simple kind of field validation to the DB layer, meaning that we indeed can rely a lot on the edgedb.Object type records that edgedb-py generates for us out of the box. If all you want is records without logic this is indeed enough already.

My vision is to :

a) avoid writing my DB object ("table") fields and validations twice. This seems like the pythonic way and it is effectively what you get with e.g. FastAPI:s sqlmodel. Write the python model once and the ORM does the rest. I want the same here but instead of writing the definitions in python they are only written in the default.esdl file that contains all the "table" definitions such as our "Payment" in the above example.

b ) have the ability to manually/automatically cast the edgedb.Object records into something more powerful that supports methods. The way i see it we cannot really directly add all of our methods to this single object for obvious reasons. No matter what query you make it will always return the same edgedb.Object records with different content (if i interpret things correctly from checking the debugger output).

Point b) is really an attempt to answer/solve the question: What is the most ergonomic way of automatically mapping the edgedb records to logic?

I mentioned Rust traits earlier because it honestly seems like the best fit where we can side-step object instantiation entirely and just group behavior using traits instead. Traits are not parts of python though and i'm not sure about hacking them in. Maybe this really just calls for using functional programming and finding some good way to group the logic. Maybe if we just structure the code such that we create empty python modules that have the same names as our edgedb objects, e.g Payment > payment.py. That way the dev can find which functions are meant for which type of edgeql queries. I suspect i would have to build something with more complex queries first to get a feel for how this could work.

What do you think? Based on this analysis we would not maybe need a pluggable at all but just an article on how to use edgeql with a functional style in python. Not sure if that is satisfactory but it would be very simple to use on small to medium scale. Any ideas are appreciated.

[jweckman]

I played around with this some more and i think i will just have to create a small real app with around 10 tables and make some more complicated queries before i know what i really want :)

What i imagine is that it will probably be best to have two separate approaches. One with an ODM pluggable and one with very little or nothing coupled with a more functional approach. Having programmed using OO in a large code base for around three years now it seems difficult to imagine anything else than OO. I just have a strong intuition that functional way of doing things will make the most sense with the way EdgeQL is set up. If our DB always returns very different recordsets that cannot be mapped to objects, and especially if we have no javascript front-end framework that requires objects, it will be much easier to just take whatever the query returns and feed that directly into our jinja2 template that displays whatever.

I will take some time and experiment with this and come with suggestions after that. Hope i'm not taking up too much of your time with my long and experimental writings :)

[tarsil]

@jweckman not at all, it is indeed exciting and complex and I think Esmerald can perfectly serve for any OO or functional approach to be honest and I will be curious to see how do you do with Esmerald and EdgeDB in your experiment :D.

I hope Esmerald is treating you well too

[jweckman]

I think i figured out how i want to work with this after experimenting with this dummy python script that runs without dependencies:

from statistics import mean

def fake_edgedb_query():
    return [
        {
            'id': 'uuidstring1',
            'num': 1,
            'text': 'hello',
            'related_object_ids': [
                {'id': 'uuidstring3'},
                {'id': 'uuidstring4'}
            ]
        },
        {
            'id': 'uuidstring2',
            'num': 2,
            'text': 'hello2',
            'related_object_ids': [
                {'id': 'uuidstring5'},
                {'id': 'uuidstring6'},
            ]
        }
    ]

def square(num: int | float):
    return num ** 2


d = fake_edgedb_query()

filtered_squares = [square(r['num']) for r in d if r['text'][-1].isdigit()]
filter_related_objects = [r['num'] for r in d if r['related_object_ids']]

print(sum(filtered_squares))
print(mean(filter_related_objects))

To keep it as simple as possible i swapped the EdgeQL objects with a dictionary that has the same structure as actual responses and tried to apply some logic to the results. Let's say that our square() function actually does something a bit more complicated and we don't want/are able to run that logic inside of EdgeDB. Looks to me like we would almost always want to do most of our processing using list comprehensions, where we can very clearly see how we a) apply logic and b) filter the data. Adopting list comprehensions as our primary means of applying logic means that we can do a lot of the simpler kind of filtering on an ad-hoc basis wherever we do it in the code without it being a problem due to nice readability.

Our logic is what i find the most interesting. In a moderately large code base we will probably have a few thousand lines of logic related to our payment records. How should they be structured? Seems like the default EdgeDB FastAPI example already gives most of the clues. Especially if the logic gets very large, we should just put the Payments router in its own folder (namespace) and put all the EdgeDB queries inside a "queries" folder below the logic. This automatically means that our python module imports will tell the story of what the queries are related to. The folder where the Payments router lives will also have plenty of space for all kinds of functions grouped into various different modules depending on demand. This really looks very simple and should in my opinion scale very well, even into huge scale enterprise code bases. It could also work really well to integrating pretty random code that i have lying around that is not web dev related yet.

And that should be it. What got me most excited here was the possibility to ignore "classes and recordsets" thinking completely when creating computational functions on the python side. What i (and probably most web devs at the moment) are used to is that the back-end revolves around recordsets containing classes with difficult to read attribute types and a bunch of methods. In my case this does great harm to type checking and code composability. Not to mention that we have SQL as pure strings as the necessary evil when our ORM gets too slow (which in my case is way too often)

What we can have instead is highly reusable functions that mostly take in primitives. This opens up the door to unprecedented logic sharing compared to OO. Instead of having our custom square() function baked into a bunch of classes all over the code base, we can now easily implement square() once and then apply it on any numerical value in the whole db! Of course square is a bad example because even in an OO code base this would probably be a generic helper function for the whole code base, but if it does something a bit more elaborate we could always generalize it based on which type of generics it takes as arguments.

But enough theory for now. I will report here what problems i encounter in my real-world small problem. Based on that we can see if anything needs to be added to Esmerald. As i've stated earlier, i don't think my stack will benefit from an ODM. I also wonder if even something like React/Vue with back-end and front-end models mapped together will need it, because in that case we have most of our logic in the front-end and with edge-db eating up most of the traditional simple validations etc, python will only be there for routing and serialization purposes, if anything. EdgeDB + Esmerald + Jinja2 + HTMX on the other hand looks like a crazy powerful and simple combination right now :)

[tarsil]

I do love the details you always provide @jweckman , truly and I'm following you. I will also play around but it looks like you are figuring it out what we don't need in order to get what we actually want and this is beautiful. I liked you apprach btw. Truly.

Please continue to keep me up to date, this is very exciting! 🙂.

I love you idea for the tech stack and I agree with it. Simple and powerful in so many levels. I'm glad you like Esmerald for your purposes.