FirestoreOptions.setHost causes PERMISSION_DENIED errors from Firestore Emulator

[icoloma]

I'm seeing a PERMISSION DENIED message when trying to clean my test data using collection.listDocuments():

io.grpc.StatusRuntimeException: PERMISSION_DENIED: Metadata operations require admin authentication.
com.google.cloud.firestore.FirestoreException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: Metadata operations require admin authentication.
	at com.google.cloud.firestore.FirestoreException.apiException(FirestoreException.java:94)
	at com.google.cloud.firestore.CollectionReference.listDocuments(CollectionReference.java:141)
	at MyTest.deleteCollections(MyTest.java:21)

This error has been reported before as firebase-tools#1363. The workaround that is mentioned in the referenced issue is not working for me. I have tried three variants of the same thing: setCredentials(), setCredentialsProvider() and setHeaderProvider().

Firestore firestore = FirestoreOptions.getDefaultInstance()
    .toBuilder()
    .setProjectId(Config.PROJECT_ID)
    .setHost("localhost:9000")
    .setChannelProvider(
      InstantiatingGrpcChannelProvider.newBuilder().setEndpoint(Config.FIRESTORE_EMULATOR_URL)
        .setChannelConfigurator(input -> {
          input.usePlaintext();
          return input;
        }).build())
    .setCredentialsProvider(FixedCredentialsProvider.create(new FakeCredentials()))
    .setCredentials(new FakeCredentials())
    .setHeaderProvider(() -> ImmutableMap.of("Authorization", "Bearer owner"))
    .build().getService();

firestore.collection("mycollection").listDocuments();


    public static class FakeCredentials extends Credentials {
        private final Map<String, List<String>> HEADERS =
                ImmutableMap.of("Authorization", Arrays.asList("Bearer owner"));

        @Override
        public String getAuthenticationType() {
            throw new IllegalArgumentException("Not supported");
        }

        @Override
        public Map<String, List<String>> getRequestMetadata(URI uri) {
            return HEADERS;
        }

        @Override
        public boolean hasRequestMetadata() {
            return true;
        }

        @Override
        public boolean hasRequestMetadataOnly() {
            return true;
        }

        @Override
        public void refresh() {
        }
    }

It seems that some of these settings are being ignored by com.google.cloud.firestore.spi.v1.GrpcFirestoreRpc when the URL points to localhost.

The expected behavior would be a documented way to get admin permissions on the emulator, preferably through the Options object. Failing that, this workaround should work, and maybe make public static FirestoreOptions.FakeCredentials instead of just public (as is now), so it can be used by others.

OS: Ubuntu 18.04.4 LTS
Java 11
gcloud version 289.0.0
compile "com.google.firebase:firebase-admin:6.12.2"

[BenWhitehead]

Hi @icoloma,

I tried to reproduce the PERMISSION_DENIED error you ran into but wasn't able to do so. I tried a number of different methods (set, listCollections, listDocuments, get and delete) all were able to complete successfully when using the version of firebase-admin and gcloud you specified.

The code I tested with:

import com.google.cloud.firestore.CollectionReference;
import com.google.cloud.firestore.DocumentReference;
import com.google.cloud.firestore.Firestore;
import com.google.cloud.firestore.FirestoreOptions;
import com.google.common.collect.ImmutableMap;
import java.util.concurrent.ExecutionException;

public final class FirestoreConnect {
  public static void main(String[] args) throws ExecutionException, InterruptedException {
    Firestore fs = FirestoreOptions.getDefaultInstance()
        .toBuilder()
        .setProjectId("something")
        .build()
        .getService();

    var doc1 = fs.collection("coll").document("doc1");
    var set = doc1
        .set(ImmutableMap.of("field", "value"));

    var setResult = set.get();
    System.out.printf("     setResult = %s%n", setResult.getUpdateTime());

    Iterable<CollectionReference> colls = fs.listCollections();
    for (CollectionReference coll : colls) {
      System.out.printf("coll.getPath() = %s%n", coll.getPath());
      Iterable<DocumentReference> docs = coll.listDocuments();
      for (DocumentReference doc : docs) {
        System.out.printf(" doc.getPath() = %s%n", doc.getPath());
      }
    }

    var documentSnapshot = doc1.get().get();
    System.out.println("coll/doc1.data = " + documentSnapshot.getData());

    var deleteResult = doc1.delete().get();
    System.out.println("  deleteResult = " + deleteResult.getUpdateTime());
  }
}

And the output when I ran it:

     setResult = 2020-04-21T21:31:20.560000000Z
coll.getPath() = coll
 doc.getPath() = coll/doc1
coll/doc1.data = {field=value}
  deleteResult = 2020-04-21T21:31:20.666005000Z

When running my program I set the FIRESTORE_EMULATOR_HOST environment variable to the value printed when starting the Firestore emulator via gcloud beta emulators firestore start.

Can you provide any more details about what types of operations you are trying to perform?

[icoloma]

Hi @BenWhitehead, thanks for giving this a look. It seems that the exception is triggered when using options.setHost() instead of using FIRESTORE_EMULATOR_HOST. I created a minimal repro case:

import com.google.cloud.firestore.CollectionReference;
import com.google.cloud.firestore.DocumentReference;
import com.google.cloud.firestore.Firestore;
import com.google.cloud.firestore.FirestoreOptions;
import com.google.common.collect.ImmutableMap;

import java.util.concurrent.ExecutionException;

public class IssueRepro {
    public static void main(String[] args) throws ExecutionException, InterruptedException {
        Firestore fs = FirestoreOptions.getDefaultInstance()
                .toBuilder()
                .setProjectId("foobar")
                // uncomment this line
                //.setHost("localhost:9000")
                .build()
                .getService();
        CollectionReference coll = fs.collection("coll");
        var doc1 = coll.document("doc1");
        var set = doc1.set(ImmutableMap.of("field", "value")).get();
        Iterable<DocumentReference> docs = coll.listDocuments();
        for (DocumentReference doc : docs) {
            System.out.printf(" doc.getPath() = %s%n", doc.getPath());
        }
        doc1.delete();
    }
}

When I uncomment the line I get the PERMISSION_DENIED exception, even if FIRESTORE_EMULATOR_HOST is still set and has the same value.

[BenWhitehead]

Wow that's subtle, thanks for narrowing down the difference.

I'm going to change the name of this issue to reflect it's a config error in the service builder.

[athakor]

@icoloma I guess you need to use local ip instead of localhost and start emulator with --host-port for example gcloud beta emulators firestore start --host-port 127.0.0.1:8000

Sample Code :

import com.google.api.gax.core.FixedCredentialsProvider;
import com.google.api.gax.grpc.InstantiatingGrpcChannelProvider;
import com.google.auth.Credentials;
import com.google.cloud.firestore.CollectionReference;
import com.google.cloud.firestore.DocumentReference;
import com.google.cloud.firestore.Firestore;
import com.google.cloud.firestore.FirestoreOptions;
import com.google.common.collect.ImmutableMap;

import java.io.IOException;
import java.net.URI;
import java.util.Arrays;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ExecutionException;

public class IssueRepro {

    private static final String PROJECT_ID = "project-id";
    private static final String HOST = "127.0.0.1:8000";

    public static void main(String[] args) throws ExecutionException, InterruptedException, IOException, ClassNotFoundException {
        Firestore fs = FirestoreOptions.getDefaultInstance()
                .toBuilder()
                .setProjectId(PROJECT_ID)
                .setHost(HOST)
                .setChannelProvider(
                        InstantiatingGrpcChannelProvider.newBuilder()
                                .setEndpoint(HOST)
                                .setChannelConfigurator(input -> {
                                    input.usePlaintext();
                                    return input;
                                }).build())
                .setCredentialsProvider(FixedCredentialsProvider.create(new FakeCredentials()))
                .setCredentials(new FakeCredentials())
                .setHeaderProvider(() -> ImmutableMap.of("Authorization", "Bearer owner"))
                .build()
                .getService();

        CollectionReference coll = fs.collection("coll");
        var doc1 = coll.document("doc1");
        var set = doc1.set(ImmutableMap.of("field", "value")).get();
        Iterable<DocumentReference> docs = coll.listDocuments();
        for (DocumentReference doc : docs) {
            System.out.printf(" doc.getPath() = %s%n", doc.getPath());
        }
        doc1.delete();
    }


    static class FakeCredentials extends Credentials {
        private final Map<String, List<String>> HEADERS =
                ImmutableMap.of("Authorization", Arrays.asList("Bearer owner"));

        @Override
        public String getAuthenticationType() {
            throw new IllegalArgumentException("Not supported");
        }

        @Override
        public Map<String, List<String>> getRequestMetadata(URI uri) throws IOException {
            return HEADERS;
        }

        @Override
        public boolean hasRequestMetadata() {
            return true;
        }

        @Override
        public boolean hasRequestMetadataOnly() {
            return true;
        }

        @Override
        public void refresh() {
        }
    }
}

Output:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
 doc.getPath() = coll/doc1

Process finished with exit code 0

[icoloma]

This leaves some questions open:

• Why is gax ignoring my settings when I'm using localhost? Leaving options out should be my responsibility, not the library's.
• Using 127.0.0.1 instead of localhost sounds like exploiting a loophole to work around a different bug. Without knowing why these options are ignored for localhost, I don't know if this workaround will break in a future version of the client library.
• com.google.cloud.firestore.FirestoreOptions.FakeCredentials should be public static so that it can be used by others (instead of copied & pasted).
• The error should not be "PERMISSION DENIED" with this config, as the credentials have been provided.

"Admin permissions" sounds like a common case for any tests planning to clean up the database afterwards. Maybe add a specific flag in FirestoreOptions to gain admin permissions against localhost?

Thanks for the workaround, I can use this.

[athakor]

@icoloma Thanks for filling this issue, i think everything is setup in existing firestore client just we need to use FakeCredentials while creating ClientContext so I have already raised PR for that. below is the sample code which tested with localhost by updating current firestore client.

Code :

package com.google.cloud.firestore;

import com.google.cloud.ServiceOptions;
import com.google.common.collect.ImmutableMap;
import java.util.concurrent.ExecutionException;

public class Test {

  private static final String PROJECT_ID = "project-id";
  private static final String HOST = "localhost:8000";

  public static void main(String[] args) throws ExecutionException, InterruptedException {

    Firestore fs =
        FirestoreOptions.getDefaultInstance()
            .toBuilder()
            .setProjectId(PROJECT_ID)
            .setHost(HOST)
            .build()
            .getService();

    CollectionReference coll = fs.collection("coll");
    DocumentReference doc1 = coll.document("doc1");
    WriteResult set = doc1.set(ImmutableMap.of("field", "value")).get();
    Iterable<DocumentReference> docs = coll.listDocuments();
    for (DocumentReference doc : docs) {
      System.out.printf(" doc.getPath() = %s%n", doc.getPath());
    }
    doc1.delete();
  }
}

Output :

 doc.getPath() = coll/doc1

Process finished with exit code 0

[BenWhitehead]

The original code that I assume @icoloma tried by setting the host to localhost:8000 to attempt to use the emulator should not cause the break that it does. There is currently a custom bootstrapping path that specifically replaces credentials with "NoCredentials when the host contains the string localhost This is error prone and what actually needs to change. Additionally, the way emulator connections work into the lifecycle of creating a client should also be cleaned up so that it isn't causing these sorts of errors for users.

The complexity of how the Firestore client is bootstrapped is very high, and is something I am actively looking at seeing if it can be simplified (including other clients as well like Datastore). When using the Firestore emulator, the intention is that the client is functioning as with admin level permissions to allow automatic index creations while developing. Everything @icoloma lists in #190 (comment) is correct, and needs to be accounted for in the fix.

For now, the workaround I recommend is "When using the emulator, only use the environment variable not setHost". I realize it isn't great, but unfortunately it's how the code is written right now.

I hope to have a more straightforward solution in the future, but don't yet have an estimated timeframe.

[crwilcox]

Seems to me #319 will have resolved this.