Firestore emulator in deadlock in transaction

[vdvleon]

Environment info

firebase-tools: 8.5.0
Platform: macOS
Node: 10.21.0

Test case

When reading/writing to a document using two simultaneous transactions, the transactions hang (as if in a deadlock).

I added some sample code to illustrate the problem (including some console logs to clarify where the code hangs).

firestore-emulator-bug-showcase.zip

Steps to reproduce

After downloading the sample code run:

npm install
firebase emulators:start --only firestore --debug # Assuming to be at version 8.5.0

While the emulator runs, run the following command in a different terminal:

npm run test

Expected behavior

I would expect all tests to run successfully. This means:

• the 'firestore' test runs successfully.
• in the 'lease single' test true is returned.
• in the 'lease multi' test only one of the transaction returns true while the other returns false.

Actual behavior

But actually the 'lease multi' test hangs and times out. The 'firestore' and 'lease single' tests run as expected.

See the output below:

Output of jest:

> jest

PASS src/firestore.spec.ts
  firestore
    ✓ is able to communicate with emulator (1063 ms)


  console.info
    Before transaction: try to acquire lease in transaction ...

      at acquireLease (src/lease.ts:16:11)

  console.info
    Before transaction: try to acquire lease in transaction ...

      at acquireLease (src/lease.ts:16:11)

  console.info
    Before transaction: try to acquire lease in transaction ...

      at acquireLease (src/lease.ts:16:11)

PASS src/lease.single.spec.ts
  acquireLease single
    ✓ returns true when ran once (1040 ms)

  console.info
    Transaction: acquired a new lease

      at _firestore2.firestore.runTransaction (src/lease.ts:37:15)

  console.info
    Transaction: acquired a new lease

      at _firestore2.firestore.runTransaction (src/lease.ts:37:15)

  console.info
    Transaction: acquired a new lease

      at _firestore2.firestore.runTransaction (src/lease.ts:37:15)

  console.info
    After transaction: will return true

      at acquireLease (src/lease.ts:43:11)

FAIL src/lease.multi.spec.ts
  ● Test suite failed to run

    Timeout - Async callback was not invoked within the 15000 ms timeout specified by jest.setTimeout.Error: Timeout - Async callback was not invoked within the 15000 ms timeout specified by jest.setTimeout.

      at mapper (node_modules/jest-jasmine2/build/queueRunner.js:29:45)

A worker process has failed to exit gracefully and has been force exited. This is likely caused by tests leaking due to improper teardown. Try running with --runInBand --detectOpenHandles to find leaks.
Test Suites: 1 failed, 2 passed, 3 total
Tests:       2 passed, 2 total
Snapshots:   0 total
Time:        32.386 s
Ran all test suites.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! @ test: `jest`
npm ERR! Exit status 1

Output of the emulator:

[2020-07-13T15:28:34.365Z] > command requires scopes: ["email","openid","https://www.googleapis.com/auth/cloudplatformprojects.readonly","https://www.googleapis.com/auth/firebase","https://www.googleapis.com/auth/cloud-platform"]
[2020-07-13T15:28:34.366Z] > authorizing via signed-in user
⚠  Could not find config (firebase.json) so using defaults. 
i  emulators: Starting emulators: firestore {"metadata":{"emulator":{"name":"hub"},"message":"Starting emulators: firestore"}}
⚠  firestore: Did not find a Cloud Firestore rules file specified in a firebase.json config file. {"metadata":{"emulator":{"name":"firestore"},"message":"Did not find a Cloud Firestore rules file specified in a firebase.json config file."}}
⚠  firestore: The emulator will default to allowing all reads and writes. Learn more about this option: https://firebase.google.com/docs/emulator-suite/install_and_configure#security_rules_configuration. {"metadata":{"emulator":{"name":"firestore"},"message":"The emulator will default to allowing all reads and writes. Learn more about this option: https://firebase.google.com/docs/emulator-suite/install_and_configure#security_rules_configuration."}}
[2020-07-13T15:28:34.384Z] Ignoring unsupported arg: projectId {"metadata":{"emulator":{"name":"firestore"},"message":"Ignoring unsupported arg: projectId"}}
[2020-07-13T15:28:34.384Z] Ignoring unsupported arg: auto_download {"metadata":{"emulator":{"name":"firestore"},"message":"Ignoring unsupported arg: auto_download"}}
[2020-07-13T15:28:34.385Z] Starting Firestore Emulator with command {"binary":"java","args":["-Duser.language=en","-jar","/Users/vdvleon/.cache/firebase/emulators/cloud-firestore-emulator-v1.11.4.jar","--host","localhost","--port",8080],"optionalArgs":["port","webchannel_port","host","rules","functions_emulator","seed_from_export"],"joinArgs":false} {"metadata":{"emulator":{"name":"firestore"},"message":"Starting Firestore Emulator with command {\"binary\":\"java\",\"args\":[\"-Duser.language=en\",\"-jar\",\"/Users/vdvleon/.cache/firebase/emulators/cloud-firestore-emulator-v1.11.4.jar\",\"--host\",\"localhost\",\"--port\",8080],\"optionalArgs\":[\"port\",\"webchannel_port\",\"host\",\"rules\",\"functions_emulator\",\"seed_from_export\"],\"joinArgs\":false}"}}
i  firestore: Firestore Emulator logging to firestore-debug.log {"metadata":{"emulator":{"name":"firestore"},"message":"Firestore Emulator logging to \u001b[1mfirestore-debug.log\u001b[22m"}}
[2020-07-13T15:28:35.488Z] API endpoint: http://localhost {"metadata":{"emulator":{"name":"firestore"},"message":"API endpoint: http://localhost"}}
[2020-07-13T15:28:35.488Z] :8080
If you are using a library that supports the FIRESTORE_EMULATOR_HOST environment variable, run:

   export FIRESTORE_EMULATOR_HOST=localhost:8080

Dev App Server is now running.

 {"metadata":{"emulator":{"name":"firestore"},"message":":8080\nIf you are using a library that supports the FIRESTORE_EMULATOR_HOST environment variable, run:\n\n   export FIRESTORE_EMULATOR_HOST=localhost:8080\n\nDev App Server is now running.\n\n"}}

┌──────────────────────────────────────────────────────────────┐
│ ✔  All emulators ready! It is now safe to connect your apps. │
└──────────────────────────────────────────────────────────────┘

┌───────────┬────────────────┐
│ Emulator  │ Host:Port      │
├───────────┼────────────────┤
│ Firestore │ localhost:8080 │
└───────────┴────────────────┘
  Other reserved ports: 

Issues? Report them at https://github.com/firebase/firebase-tools/issues and attach the *-debug.log files.
 
[2020-07-13T15:28:53.466Z] Jul 13, 2020 5:28:53 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
INFO: Detected HTTP/2 connection.
 {"metadata":{"emulator":{"name":"firestore"},"message":"Jul 13, 2020 5:28:53 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead\nINFO: Detected HTTP/2 connection.\n"}}
[2020-07-13T15:28:53.466Z] Jul 13, 2020 5:28:53 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
INFO: Detected HTTP/2 connection.
 {"metadata":{"emulator":{"name":"firestore"},"message":"Jul 13, 2020 5:28:53 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead\nINFO: Detected HTTP/2 connection.\n"}}
[2020-07-13T15:28:53.466Z] Jul 13, 2020 5:28:53 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
INFO: Detected HTTP/2 connection.
 {"metadata":{"emulator":{"name":"firestore"},"message":"Jul 13, 2020 5:28:53 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead\nINFO: Detected HTTP/2 connection.\n"}}
[2020-07-13T15:29:24.261Z] Jul 13, 2020 5:29:24 PM com.google.cloud.datastore.emulator.impl.util.WrappedStreamObserver onError
INFO: operation failed: transaction timeout
 {"metadata":{"emulator":{"name":"firestore"},"message":"Jul 13, 2020 5:29:24 PM com.google.cloud.datastore.emulator.impl.util.WrappedStreamObserver onError\nINFO: operation failed: transaction timeout\n"}}
[2020-07-13T15:29:24.261Z] Jul 13, 2020 5:29:24 PM com.google.cloud.datastore.emulator.impl.util.WrappedStreamObserver onError
INFO: operation failed: transaction timeout
 {"metadata":{"emulator":{"name":"firestore"},"message":"Jul 13, 2020 5:29:24 PM com.google.cloud.datastore.emulator.impl.util.WrappedStreamObserver onError\nINFO: operation failed: transaction timeout\n"}}

[samtstern]

@vdvleon thank you for the reproduction, I was able to see the same result. Assigning to @avolkovi to check it out

[vdvleon]

I wonder if there is progress on this issue? It seems about a year ago the problem occurred as well. Maybe this is related?

[rosalyntan]

Thanks for filing this issue! Filed b/162265825 for internal tracking.

[mikquinlan]

Related?

[vdvleon]

Related?

Could be related yeah. Could this behaviour be changed or is there a way to influence this timeout time? It's not very realistic to let our tests wait for 30 seconds.

[roblaszczak]

Any update on that @samtstern @rosalyntan ? It's a bit critical because it's making Firestore emulator unusable for any integration tests using transactions. Thanks!

[dunyakirkali]

Any progress on this issue?

[lutogin]

Same problem.
Is there a solution? Or a workaround?

[electricmonk]

Same issue here, apparently

[scottfr]

We just ran into this issue too. Looks like we can't use the emulator for logic that has transactions with contention.

[Horsetoast]

Same here. Relevant part of code looks like this:

    await db.runTransaction(async (t) => {
      const gameState = (await t.get(internalRef)).data() as InternalGame
      gameState.players[playerIndex!].ready = true
      gameState.players[playerIndex!].answer = answer

      const progressedGame = progressGame(gameState)
      redactedGame = redactGame(progressedGame)
      
      const p1 = t.update(internalRef, progressedGame);
      const p2 = t.set(redactedRef, redactedGame)
      await Promise.all([p1, p2])
    });

In production, there is no problem. In the emulator, it stalls for maybe 30 seconds, and only then it successfully completes.

[saiichihashimoto]

Is there any way to prioritize this? I can't deploy code that can't be tested.

[saiichihashimoto]

@samtstern @avolkovi @rosalyntan

[Youmoo]

Please, is anyone working on this ?

[saiichihashimoto]

It's been a year since the last member comment, how do we prioritize this? @samtstern @avolkovi @rosalyntan @mbleigh @bkendall @joehan @yuchenshi

[oceanful]

If comments / feedback help with prioritization, I'll add that this is a pretty big thorn in our side.

We have multiple tests that verify that our transactional logic is resilient to concurrent updates. The emulator consistently pauses for 30 seconds (and sometimes 60 or 90 seconds) for each of these tests.

[Poliziano]

My team would also like this issue to be resolved.

[bkendall]

I've spoken with another engineer familiar with the problem. It's not an easy fix to solve this an apparently will take a modest amount of engineering effort to fix completely. The correct team is aware of the bug and has a plan, but prioritization and resourcing hasn't enabled them to address it yet.

I can't comment on any expected timeline, but I'll follow up when there's news. Sorry I don't have a better answer at this point.

[paced]

It is a relief that this issue is known to be only with the Emulator and not in prod. I was going to turn off the test, deploy to prod, and watch the logs for request durations if I didn't find this thread.

Perhaps as a stop-gap, some information can be added to the official documentation? Something like: "Simultaneous requests in a transaction that access the same document may be much slower in the emulator when compared to in production."

[WilliamHarvey97]

I experience this 30 seconds delay using the Firestore emulator even though my two concurrent transactions do not read the same documents. If I execute the same concurrent transactions using a real firestore DB, it works perfectly fine. When using the emulators, the transactions just won't commit and I don't know why...

[dahu33]

Hey Google, how do you expect us to write proper tests without proper support of transactions...? Wait, did I just hear "test in prod!" ? 😱😱😱

[fredzqm]

We have allocated resource to work on it.
If things go well, it should be resolved around April.

[gomezjdaniel]

So I am having the same problem. Hope this gets resolved soon.

[williamjwu]

Hope this issue will resolve soon! 😉