Sub-transactions API #912

yrashk · 2022-12-05T02:30:33Z

Problem: inability to fine-tune transaction boundaries easily

Solution: implement an interface for sub-transactions

It's a fairly simplistic interface at this point, allowing choosing a mode of release on drop (commit by default).

This is an improved version of the original implementation in pgx-contrib-spiext
(https://github.com/supabase/pgx-contrib-spiext/blob/main/src/subtxn.rs)

I welcome feedback as the interface is somewhat opinionated (however, I tried to make it as palatable as possible, and its previous iteration in pgx-contrib-spiext went through a few rounds of improvements to be used in real extensions). I've tried to follow our recent work on SpiClient in spirit.

It'd be great to get additional scrutiny for the safety and soundness of the implementation.

A message to pgx users: my ability to actively contribute to pgx largely depends on your support. Please consider sponsoring my work

Solution: implement an interface for sub-transactions It's a fairly simplistic interface at this point and it allows choosing a mode of release on drop (commit by default). This is an improved version of the original implementation in pgx-contrib-spiext (https://github.com/supabase/pgx-contrib-spiext/blob/main/src/subtxn.rs)

yrashk · 2022-12-11T21:00:12Z

Any feedback I can get on this from the maintainers? Thanks!

pgx/src/subxact.rs

workingjubilee · 2022-12-13T22:14:26Z

pgx/src/subxact.rs

+// This allows SubTransaction to be de-referenced to SpiClient
+impl<'conn, const COMMIT: bool> Deref for SubTransaction<SpiClient<'conn>, COMMIT> {
+    type Target = SpiClient<'conn>;
+
+    fn deref(&self) -> &Self::Target {
+        self.parent.as_ref().unwrap()
+    }
+}
+
+// This allows a SubTransaction of a SubTransaction to be de-referenced to SpiClient
+impl<Parent: SubTransactionExt, const COMMIT: bool> Deref
+    for SubTransaction<SubTransaction<Parent>, COMMIT>
+{
+    type Target = Parent;
+
+    fn deref(&self) -> &Self::Target {
+        self.parent.as_ref().and_then(|p| p.parent.as_ref()).unwrap()
+    }
+}


Wouldn't it be more general for SubTransaction<T, COMMIT> to deref into T? Is there a problem with that approach? Too much recursion?

This is how it was initially https://github.com/supabase/pgx-contrib-spiext/blob/main/src/subtxn.rs#L137-L143

However, this has prevented me from using SubTransaction<T: Deref<SpiClient>> so I had to resort to boxing: https://github.com/supabase/pgx-contrib-spiext/blob/develop/src/subtxn.rs#L169-L178

However, with this special case, I no longer need a Box

pgx/src/subxact.rs

workingjubilee · 2022-12-13T22:32:14Z

pgx/src/subxact.rs

+/// Sub-transaction
+///
+/// Can be created by calling `SpiClient::sub_transaction`, `SubTransaction<Parent>::sub_transaction`
+/// or any other implementation of `SubTransactionExt` and obtaining it as an argument to the provided closure.


My internal grammarian is trying to say the function names should be subtransaction (which is valid in English), whereas the type names should be SubTransaction, but that seems... somehow peculiar, given that normally those would normalize to either sub_transaction and SubTransaction like you have it here or subtransaction and Subtransaction.

I named it sub_transaction for consistency with SubTransaction. But ultimately I don't have a strong opinion about this.

And Postgres calls them "SubTransactions" in their API.

Yeah, I agree that the type/trait names should use SubTransaction. Maybe I will start using sub_transaction in anger and it will bug me enough to want to take the _ out, and I'll just let it be inconsistent-but-natural then. But otherwise it seems fine.

pgx/src/subxact.rs

workingjubilee · 2022-12-13T22:46:58Z

pgx/src/subxact.rs

+    // The reason we are not calling this `released` as we're also using this flag when
+    // we convert between commit_on_drop and rollback_on_drop to ensure it doesn't get released
+    // on the drop of the original value.
+    should_release: bool,


"on the drop of the original value"... hm.

workingjubilee · 2022-12-13T22:49:13Z

pgx/src/subxact.rs

+    fn into(mut self) -> SubTransaction<Parent, false> {
+        let result = SubTransaction {
+            memory_context: self.memory_context,
+            resource_owner: self.resource_owner,
+            should_release: self.should_release,
+            parent: self.parent.take(),
+        };
+        // Make sure original sub-transaction won't commit
+        self.should_release = false;
+        result
+    }


Oh, I see. We copy everything into a new SubTransaction and then let Drop run for the previous one.

Yep. Does this resolve #912 (comment) or do you want to provide additional feedback on this?

It may be unnecessary. Solution: simply derive it

Solution: name it Parent as it is what it is

Also, given `PgMemoryContexts` API is still somewhat rough, it may be not a great idea to expose it. Solution: remove it

…is for Solution: comment it

Solution: provide some reference materials on it

It is only there so that we can move it out of manually copied type that implements Drop Solution: extract release logic into a separate trait

Solution: document them

eeeebbbbrrrr

I think in general this is okay. I've never used any of Postgres subtransaction facilities from C before, and haven't studied their docs, so I'm leaning heavily on the thought that you have.

That said, before we can merge I'd like to see two things...

A lot of additional doc comment documentation in subxact.rs, from the module-level down. I think this needs some exposition around not only what it is, but how it interacts with the database, and how that might impact the user's code and design decisions.
A new example that demonstrates using this, maybe in some kind of fabricated scenario that users could maybe adapt to their own purposes (ie, something a little more polished than, say, our "strings" example). And with that example, a nice README.md that covers topics around proper subxact handling.

This code itself isn't very complex, which is nice. But it's a brand new feature and I'd like it to not only be as immediately useful as possible to users, but also serve as an example of what a new feature PR ought to look like. ;)

Other Thoughts

I also wonder if/how this should interop with pgx' existing subact hooks support (see callbacks.rs/register_subxact_callback() and friends.
What needs to be said in docs (or done in code?) with regards to PgTryBuilder and subtransactions?
Is this also usable as-is by background workers? Is there an opportunity for expanding the existing bgworker example to show off subxact support?
I'm sure I'll come up with other stuff as we get this merged.

eeeebbbbrrrr · 2022-12-29T18:38:30Z

pgx/src/prelude.rs

@@ -47,3 +47,6 @@ pub use crate::pg_sys::{
    check_for_interrupts, debug1, debug2, debug3, debug4, debug5, ereport, error, function_name,
    info, log, notice, warning, FATAL, PANIC,
 };
+
+// Sub-transactions
+pub use crate::subxact::*;


Probably better to limit this to just the module: pub use crate::subxact;.

I think we're trying to be a little better about globbing everything into the current namespace.

I am not sure this is the best. At the very least, I think we should consider pub use crate::subxact::{self, SubTransactionExt} for the sub-transaction functionality to be easily available.

eeeebbbbrrrr · 2022-12-29T18:40:01Z

pgx/src/subxact.rs

+        // Remember the memory context before starting the sub-transaction
+        let ctx = PgMemoryContexts::CurrentMemoryContext.value();
+        // Remember resource owner before starting the sub-transaction
+        let resource_owner = unsafe { pg_sys::CurrentResourceOwner };


I think Jubilee means when might Postgres initially set (or later change) this and how are we certain it even has a valid value at this point in time?

yrashk · 2023-01-02T15:00:55Z

Thank you for your feedback.

I will work on more documentation. I didn't want to invest time into that before we (at least somewhat) agreed on the approach to avoid re-documenting things.
I will write some examples as well. I don't see any reason why this won't be usable in background workers, as long as they are SPI-enabled. I can extend the example there.
PgTryBuilder thing is the most interesting here. I left this out of this PR but there's a case when you want to recover the parent even if your sub-transaction failed. Currently, the parent would be gone, which is not perfect. However, with a trivial addition of another crate-level SubTransactionExt-implementing type (similar to https://github.com/supabase/pgx-contrib-spiext/blob/develop/src/subtxn.rs#L202-L227) one can implement a wrapper around PgTryBuilder that would recover the parent in the event of failure. If you'd like that to be a part of this PR, I will be happy to amend it. Easier to show it in code than explain.
I have not yet considered register_subxact_callback interactions/interop. I'll think about it.

yrashk · 2023-01-14T16:18:33Z

Updated subtransactions to work on recent develop. I will now work on docs/examples and other concerns listed above.

We can do better Solution: export only most common types

eeeebbbbrrrr · 2023-03-06T14:42:02Z

@yrashk are you still planning on taking this over the finish line?

yrashk · 2023-03-06T14:46:08Z

Yes, I've discovered some issues with the API that make its usability reduced (mostly related to mutability) when porting my "contrib" crate to use this core and extending it to essentially allow for error recovery without losing the entire subtransaction stack.

I'm working on rethinking this and attempting to see what I need to change in the core of this PR to make this possible.

eeeebbbbrrrr · 2023-03-06T14:48:08Z

Fair enough. Subxacts are definitely pretty hard to reason about.

yrashk force-pushed the sub-xact branch 2 times, most recently from d2412fe to eba8dc6 Compare December 5, 2022 15:49

yrashk force-pushed the sub-xact branch from eba8dc6 to f7455b1 Compare December 5, 2022 15:51

workingjubilee reviewed Dec 13, 2022

View reviewed changes

yrashk added 6 commits December 13, 2022 15:14

Merge remote-tracking branch 'origin/develop' into sub-xact

a35d24e

Problem: manual implementation of Drop for SubTransaction

aa8ae18

It may be unnecessary. Solution: simply derive it

Problem: SubTransactionExt::T is a non-descriptive name

8998091

Solution: name it Parent as it is what it is

Problem: SubTransaction::memory_context() is not used anywhere

351d157

Also, given `PgMemoryContexts` API is still somewhat rough, it may be not a great idea to expose it. Solution: remove it

Problem: unclear what a null argument to BeginInternalSubTransaction …

b38d9d7

…is for Solution: comment it

Problem: unclear how ResourceOwner should be managed

df654a1

Solution: provide some reference materials on it

yrashk changed the base branch from master to develop December 13, 2022 23:40

Problem: use of Option<Parent> in SubTransaction

d04aea8

It is only there so that we can move it out of manually copied type that implements Drop Solution: extract release logic into a separate trait

yrashk force-pushed the sub-xact branch from 5f102a8 to d04aea8 Compare December 14, 2022 14:32

Problem: meaning of fields in SubTransaction is unclear

1cc57bc

Solution: document them

yrashk mentioned this pull request Dec 15, 2022

Make non-updating queries use readonly unless there was a mutation before #963

Merged

eeeebbbbrrrr requested changes Dec 29, 2022

View reviewed changes

yrashk added 2 commits January 14, 2023 06:13

Merge remote-tracking branch 'origin/develop' into sub-xact

3ed2afe

Update subxact to work on recent develop

94f7cf8

Problem: prelude exports too much from subxact

cb812f4

We can do better Solution: export only most common types

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sub-transactions API #912

Sub-transactions API #912

yrashk commented Dec 5, 2022 •

edited

yrashk commented Dec 11, 2022

workingjubilee Dec 13, 2022

yrashk Dec 13, 2022

workingjubilee Dec 13, 2022

yrashk Dec 13, 2022

yrashk Dec 13, 2022

workingjubilee Dec 14, 2022 •

edited

workingjubilee Dec 13, 2022

workingjubilee Dec 13, 2022

yrashk Dec 13, 2022

eeeebbbbrrrr left a comment

eeeebbbbrrrr Dec 29, 2022

yrashk Jan 2, 2023

eeeebbbbrrrr Dec 29, 2022

yrashk commented Jan 2, 2023 •

edited

yrashk commented Jan 14, 2023

eeeebbbbrrrr commented Mar 6, 2023

yrashk commented Mar 6, 2023

eeeebbbbrrrr commented Mar 6, 2023

Sub-transactions API #912

Are you sure you want to change the base?

Sub-transactions API #912

Conversation

yrashk commented Dec 5, 2022 • edited

yrashk commented Dec 11, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

workingjubilee Dec 14, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eeeebbbbrrrr left a comment

Choose a reason for hiding this comment

Other Thoughts

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yrashk commented Jan 2, 2023 • edited

yrashk commented Jan 14, 2023

eeeebbbbrrrr commented Mar 6, 2023

yrashk commented Mar 6, 2023

eeeebbbbrrrr commented Mar 6, 2023

yrashk commented Dec 5, 2022 •

edited

workingjubilee Dec 14, 2022 •

edited

yrashk commented Jan 2, 2023 •

edited