transactional rollback doesn't seem to be working

[maininformer]

Hello, I hope you are having a great Thursday.

In the old NDB, when there was an exception in a transaction, a rollback was applied, as one does. In this new NDB, even though I see the a call to rollback it does not seem to take effect.

In the following stand alone file, I expect the second transaction to be rolled back, but it does not, after the exception I see the model updated in the datastore.

from google.cloud import ndb

class Library(ndb.Model):
    books = ndb.JsonProperty()


@ndb.transactional()
def add_book(library_id, book_name, pages):
    library = Library.get_by_id(library_id)
    if not library.books:
        library.books = {}
    library.books[book_name] = pages
    library.put()

class LeRandomException(Exception):
    pass

@ndb.transactional()
def add_book_with_exception(library_id, book_name, pages):
    library = Library.get_by_id(library_id)
    library.books[book_name] = pages
    raise LeRandomException


if __name__ == '__main__':
    with ndb.Client().context():
        lib = Library()
        key = lib.put()
        add_book(key.id(), 'behave', 'so many')

        assert Library.get_by_id(key.id()).books['behave'] == 'so many'

        try:
            add_book_with_exception(key.id(), 'hooked', 'less than behave')
        except LeRandomException:
            books = Library.get_by_id(key.id()).books
            assert books.get('hooked', None) is None, books

version is 1.2.0

[danoscarmike]

@andrewsg could you take a look or assign out?

[chrisrossi]

Hi @plumSemPy ,

When I run your test, it passes. IE, the assert on the last line is True. Am I missing something?

[maininformer]

I tried this with master as well, still failing. I will step through and give you the exact failure soon.

[maininformer]

@chrisrossi I stepped through and it seems the rpc call returns successfully, but the error persists. This leads me to believe it could be that I am using the emulator and you are using an actual datastore? Can you please retry with an emulator? Thank you.

[chrisrossi]

It also passes for me with the Datastore Emulator.

[maininformer]

I also tested with a real datastore, and it still fails. I will keep digging.

[maininformer]

@chrisrossi I found the problem. This does seem to be an ndb issue. I broke into my code and queried datastore directly using the datastore client. It was rolledback. This lead me to believe that maybe the in memory cache is not being wiped after a transaction failure. I added a ndb.get_context().clear_cache() right before my query after LeRandomException and sure enough, it passed. Did you maybe not use a cache in your tests?

my passing module now looks like this:

from google.cloud import ndb

class Library(ndb.Model):
    books = ndb.JsonProperty()


@ndb.transactional()
def add_book(library_id, book_name, pages):
    library = Library.get_by_id(library_id)
    if not library.books:
        library.books = {}
    library.books[book_name] = pages
    library.put()

class LeRandomException(Exception):
    pass

@ndb.transactional()
def add_book_with_exception(library_id, book_name, pages):
    library = Library.get_by_id(library_id)
    library.books[book_name] = pages
    raise LeRandomException


if __name__ == '__main__':
    with ndb.Client().context():
        lib = Library()
        key = lib.put()
        add_book(key.id(), 'behave', 'so many')

        assert Library.get_by_id(key.id()).books['behave'] == 'so many'

        try:
            add_book_with_exception(key.id(), 'hooked', 'less than behave')
        except LeRandomException:
            ndb.get_context().clear_cache()  # this is the newly added line
            books = Library.get_by_id(key.id()).books
            assert books.get('hooked', None) is None, books

[chrisrossi]

Yes, you're right. It's something to do with the cache. I can reproduce now! Thank you!