Firebase emulator does not stop when run with Golang exec command

[d3c3balus]

[REQUIRED] Environment info

firebase-tools:
8.12.1

Platform:
Ubuntu

[REQUIRED] Test case

I'm trying to build tests for Firestore using the firebase emulator.
The emulator starts correctly, tests run, but the despite the SIGKILL (and I tried other signals) the emulator is not cleaned up after the tests finish.

[REQUIRED] Steps to reproduce

This is how my main_test.go looks like:

package main

import (
    "fmt"
    "io"
    "log"
    "os"
    "os/exec"
    "strings"
    "syscall"
    "testing"
    "time"
)

func TestMain(m *testing.M) {
    // command to start firestore emulator
    cmd := exec.Command("firebase", "emulators:start")

    // this makes it killable
    cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}

    // we need to capture it's output to know when it's started
    stdout, err := cmd.StdoutPipe()
    if err != nil {
        log.Fatal(err)
    }
    defer stdout.Close()

    if err := cmd.Start(); err != nil {
        log.Fatal(err)
    }

    var result int
    defer func() {
        syscall.Kill(cmd.Process.Pid, syscall.SIGKILL)
        os.Exit(result)
    }()

    started := make(chan bool)
    go func() {
        buf := make([]byte, 512, 512)
        for {
            n, err := stdout.Read(buf[:])
            if err != nil {
                if err == io.EOF {
                    break
                }
                log.Fatalf("reading stdout %v", err)
            }

            if n > 0 {
                d := string(buf[:n])

                // only required if we want to see the emulator output
                fmt.Printf("%s", d)

                // checking for the message that it's started
                if strings.Contains(d, "All emulators ready") {
                    started <- true
                    break
                }
            }
        }
    }()

    done := make(chan error, 1)
    go func() {
        done <- cmd.Wait()
    }()

    select {
    case <-time.After(10 * time.Second):
        log.Fatal("Failed to start the command for 10 seconds")
    case err := <-done:
        log.Fatalf("------\nCommand has finished unexpectedly with error: %v", err)
    case <-started:
        fmt.Println("--------")
        log.Print("Command started successully... running tests")
    }

    log.Print("BEFORE running tests")
    result = m.Run()
    time.Sleep(time.Minute) // to simulate that it take some times to run tests
    log.Print("AFTER running tests")
}

The output of go test . -v is looking ok:

i  emulators: Starting emulators: firestore
i  firestore: Firestore Emulator logging to firestore-debug.log
i  ui: Emulator UI logging to ui-debug.log

┌───────────────────────────────────────────────────────────────────────┐
│ ✔  All emulators ready! View status and logs at http://localhost:4000 │
└────────────────────────────────────────────────────────────--------
2020/10/13 16:20:29 Command started successully... running tests
2020/10/13 16:20:29 BEFORE running tests
testing: warning: no tests to run
PASS
2020/10/13 16:20:29 AFTER running tests
ok      go/src/test (cached) [no tests to run]

But firebase emulator is not properly cleaned up (if I start the same command again, it fails complaining that port 4000 is already in use). The following processes are found lingering:

### snippet from netstat -anp:
tcp        0      0 127.0.0.1:4000          0.0.0.0:*               LISTEN      25187/firebase
tcp6       0      0 127.0.0.1:8080          :::*                    LISTEN      25160/cloud-firesto

### snippet from ps axu:
user 25187 /usr/local/bin/firebase /home/.cache/firebase/emulators/ui-v1.1.1/server.bundle.js
...
user 25160 /home/.cache/firebase/emulators/cloud-firestore-emulator-v1.11.7.jar --launcher_javabase=/usr/local/buildtools/java/jdk11 -Duser.language=en run --host localhost --port 8080 --rules /home/firestore.rules

[REQUIRED] Expected behavior

I expect the SIGINT (or SIGKILL) sent by the program to kill the emulator and not leave any lingering processes.

[REQUIRED] Actual behavior

The following processes are found lingering, after go test finishes (even though SIGKILL was sent):

### snippet from netstat -anp:
tcp        0      0 127.0.0.1:4000          0.0.0.0:*               LISTEN      25187/firebase
tcp6       0      0 127.0.0.1:8080          :::*                    LISTEN      25160/cloud-firesto

### snippet from ps axu:
user 25187 /usr/local/bin/firebase /home/.cache/firebase/emulators/ui-v1.1.1/server.bundle.js
...
user 25160 /home/.cache/firebase/emulators/cloud-firestore-emulator-v1.11.7.jar --launcher_javabase=/usr/local/buildtools/java/jdk11 -Duser.language=en run --host localhost --port 8080 --rules /home/firestore.rules

[google-oss-bot]

This issue does not have all the information required by the template. Looks like you forgot to fill out some sections. Please update the issue with more information.

[samtstern]

@d3c3balus normally we suggest inverting this. Instead of having your Go tests start the emulator (and deal with looking for the log messages, etc) use emulators:exec to have the emulators start your tests:

firebase emulators:exec "go test"

The command above will:

1. Start the emulators
2. Wait for them all to be ready
3. Run your command (in this case, go test)
4. When your command finishes, shut down all the emulators cleanly
5. Return the status code from your command

Does that work for you?

[d3c3balus]

Thanks. I did not know that "the recommended" way to start tests is with emulators:exec

I got inspired from this article where the command to start the emulator is gcloud beta emulators firestore start. With that command, the go test kills the emulator when it exits.

I have the feeling that I'm missing something - can you please help me understand:

1. what is the difference between firebase emulators:start vs. gcloud beta emulators firestore start?
2. is it possible to import data when using gcloud beta emulators firestore start (same way it's done with --import flag when doing firebase emulators:start?

Thanks a lot!

DC

[samtstern]

@d3c3balus great questions, this is definitely a confusing space.

gcloud

gcloud beta emulators firestore start is a very simple command. It basically just downloads and starts up the Firestore emulator jar file. It's the exact same jar that the firebase CLI uses (more on that later) and honestly running that gcloud command could be replaced in most circumstances by running java -jar firestore-emulator.jar directly (if you knew where to get the jar file).

firebase

firebase emulators:start starts what we call the "Emulator Suite" which is a group of emulators and controllers that emulate individual services and their interactions. The individual Firestore emulator is identical, same jar file, but we do a few more things:

• Load security rules with support for hot reload
• Connect any Firestore-triggered Cloud Functions running in the Cloud Functions emulator
• Coordinate import and export
• Allow host and port configuration through the firebase.json file
• Start up a local UI for inspecting and controlling the emulators
• and some more...

So you can see that the firebase version is a much more complex and complete offering on top of the same building blocks. To answer your specific question: can the gcloud version do import and export? I don't think it can, but that's only because the gcloud team hasn't implemented those controls. The underlying jar can do it easily.

[samtstern]

@d3c3balus I hope that answered your question. Unfortunately none of us here can help you with gcloud features or orchestrating emulators through Go ... we're Node.js experts :-)

I hope emulators:exec is useful for you and if you try it and run into any issues please let us know!

[d3c3balus]

Indeed, I tried emulators:exec and it works fine with the following caveats:

#1. I need to put a sleep (~1sec) before starting the testing, in order to allow the emulator to import the mock data. Anyway, this is not a big deal.

#2. from time to time (not always, but often), independent of the tests being run (so for the same suite of tests), the firestore-debug.log shows errors, like the one below (full logfile):

API endpoint: http://localhost:8080
If you are using a library that supports the FIRESTORE_EMULATOR_HOST environment variable, run:

   export FIRESTORE_EMULATOR_HOST=localhost:8080

Dev App Server is now running.

Oct 19, 2020 3:14:01 AM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
INFO: Detected HTTP/2 connection.
Oct 19, 2020 3:14:02 AM io.netty.channel.DefaultChannelPipeline onUnhandledInboundException
WARNING: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.io.IOException: Connection reset by peer
	at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
	at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:233)
	at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:223)
	at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:358)
	at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
	at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)

*** shutting down gRPC server since JVM is shutting down
*** server shut down

Anything about #2?

Thank you!

[samtstern]

@d3c3balus hmmm not quite sure about that second issue .. it could be an issue inside the Firestore emulator or in the Go client library. If you find a way to reproduce it (or a hint of a way) could you let us know in a new issue so we can try to isolate the issue?