How to force datastore to use the local development server?

[ericel]

What component of google-cloud-cpp is this feature request for? For
Google cloud datastore(native mode)

Is your feature request related to a problem? Please describe.
I can't seem to be able to get my program to use the local development server.

int main()
{
    // Sets Google Service Account
    setenv("GOOGLE_APPLICATION_CREDENTIALS", "../service-account.json", 1);
    setenv("GOOGLE_CLOUD_PROJECT", "*****my app****", 1);
    // Check if the application is running in a development environment
    const char* environment = getenv("ENVIRONMENT");

    if (environment != nullptr && std::string(environment) == "development") {
        LOG_INFO << "Running *****my app**** APP in development mode";
        // Set the Firestore emulator host (adjust the host and port as necessary127.0.0.1:10901)
        setenv("FIRESTORE_EMULATOR_HOST", "localhost:9019", 1);
        LOG_INFO  << "FIRESTORE_EMULATOR_HOST: " << getenv("FIRESTORE_EMULATOR_HOST");

        // Set Datastore emulator environment variables
        setenv("DATASTORE_EMULATOR_HOST", "localhost:9019", 1);
        setenv("DATASTORE_EMULATOR_HOST_PATH", "localhost:4000/firestore", 1);
        setenv("DATASTORE_HOST", "localhost:9019", 1);
        setenv("DATASTORE_PROJECT_ID", "*****my app****", 1);
        setenv("DATASTORE_USE_PROJECT_ID_AS_APP_ID", "true", 1);
    }
}

Describe the solution you'd like
Okay, I may be out of place here, but I can use the Google-cloud-cpp to make connections to my local cloud storage, which is great.

    // Specify the emulator endpoint. This URL is just an example; adjust the port as needed.
   std::string emulator_endpoint = "http://localhost:9199";
   // Create client options with the emulator endpoint
   auto options = google::cloud::storage::ClientOptions::CreateDefaultClientOptions().value();
   options.set_endpoint(emulator_endpoint);
   // Initialize the Google Cloud Storage client with the configured options
   client_ = google::cloud::storage::Client(options);

I don't seem to find any direct way to do the same through the datastore. I want to use my local datastore or Firebase Firestore emulator as my datastore is in native mode. Is this an overkill, or is there no real need for such usage?

Describe alternatives you've considered
Trying to use just environment variables here, haven't seemed to work for me.

setenv("FIRESTORE_EMULATOR_HOST", "localhost:9019", 1);
     LOG_INFO  << "FIRESTORE_EMULATOR_HOST: " << getenv("FIRESTORE_EMULATOR_HOST");

     // Set Datastore emulator environment variables
     setenv("DATASTORE_EMULATOR_HOST", "localhost:9019", 1);
     setenv("DATASTORE_EMULATOR_HOST_PATH", "localhost:4000/firestore", 1);
     setenv("DATASTORE_HOST", "localhost:9019", 1);
     setenv("DATASTORE_PROJECT_ID", "*****my app****", 1);
     setenv("DATASTORE_USE_PROJECT_ID_AS_APP_ID", "true", 1);

Better
I would like to connect my application to use my Firestore emulator when in development just like I am able to use the Cloud Storage emulator. Any help here would save a life :).

[coryan]

Describe the solution you'd like Okay, I may be out of place here, but I can use the Google-cloud-cpp to make connections to my local cloud storage, which is great.

nit: nowadays I would write:

namespace g = google::cloud;
client_ = g::storage::Client(g::Options{}
    .set<g::storage::RestEndpointOption>("http://localhost:9199")
    .set<g::CredentialsOption>(g::MakeInsecureCredentials()));

I don't seem to find any direct way to do the same through the datastore. I want to use my local datastore or Firebase Firestore emulator as my datastore is in native mode.

This should be possible. The documentation is in:

https://cloud.google.com/cpp/docs/reference/datastore/latest/datastore-override-endpoint

and

https://cloud.google.com/cpp/docs/reference/datastore/latest/datastore-override-authentication

Both use the Admin client as examples, but reference the examples for the DatastoreClient:

https://cloud.google.com/cpp/docs/reference/datastore/latest/datastore_v1_1_1DatastoreClient-service-account-snippet
https://cloud.google.com/cpp/docs/reference/datastore/latest/datastore_v1_1_1DatastoreClient-endpoint-snippet

In summary, I think you want something like this (modulo the endpoint, I am not sure what endpoint is the correct one for DatastoreClient:

  auto client = g::datastore_v1::DatastoreClient(g::datastore_v1::MakeDatastoreConnection(
      g::Options{}
          .set<g::EndpointOption>("localhost:9019")
          .set<g::CredentialsOption>(g::MakeInsecureCredentials())));

You may also find this useful:

https://cloud.google.com/cpp/docs/reference/datastore/latest/datastore-env

[ericel]

@coryan thanks. just a small try i get these errors
version of gc-cpp: google-cloud-cpp_2.23.0_arm64-osx

error: no member named 'EndpointOption' in namespace 'google::cloud'
auto options = google::cloud::Options{}.setgoogle::cloud::EndpointOption(
~~~~~~~~~~~~~~~^
1 error generated.

the same is true with CredentialsOption.
My code:
std::shared_ptr<google::cloud::datastore_v1::DatastoreClient> client_;
and

 client_ = std::make_unique<google::cloud::datastore_v1::DatastoreClient>(
            google::cloud::datastore_v1::MakeDatastoreConnection());
if (debug)
        {
            auto options = google::cloud::Options{}.set<google::cloud::EndpointOption>(
                "localhost:9019");
            client_ = std::make_unique<google::cloud::datastore_v1::DatastoreClient>(google::cloud::datastore_v1::DatastoreClient(google::cloud::datastore_v1::MakeDatastoreConnection(options)));

            //.set<google::cloud::CredentialsOption>(google::cloud::MakeInsecureCredentials())
        }

UPDATE: My bad. After checking the change log, I noticed we are now at version 2.24...

[coryan]

@coryan thanks. just a small try i get these errors version of gc-cpp: google-cloud-cpp_2.23.0_arm64-osx

error: no member named 'EndpointOption' in namespace 'google::cloud'

Sorry about that. That is found in google/cloud/common_options.h:

google-cloud-cpp/google/cloud/common_options.h

Line 41 in bbeffd9

struct EndpointOption {

the same is true with CredentialsOption.

Argh. I should have said UnifiedCredentialsOption:

google-cloud-cpp/google/cloud/credentials.h

Line 73 in bbeffd9

struct UnifiedCredentialsOption {

[ericel]

@coryan Thank you. I managed to get that set up. But it still is not working as smoothly as I would have wanted it to. I thought it would connect seamlessly to the Firebase Firestore emulator at port localhost:9019 as the storage API does with the provided Firebase Storage port 9199.

Instead when I do a simple datastore firestore in native mode database manipulation like updating or querying, I get this error:

google::cloud::Status thrown: INVALID_ARGUMENT: Error in non-idempotent operation: Database mode not set to Datastore Mode. If using the gCloud CLI, please specify the --database-mode flag. error_info={reason=, domain=, metadata={gcloud-cpp.retry.reason=non-idempotent, gcloud-cpp.retry.function=Commit, gcloud-cpp.retry.original-message=Database mode not set to Datastore Mode. If using the gCloud CLI, please specify the --database-mode flag.}}

Just to clear things up, my datastore here is in firestore-native mode. I am using the datastore API because C++ has no Firestore direct app for web.

Is what I am trying to do even possible?
Example code usage:

 if (debug)
        {
            auto options = google::cloud::Options{}.set<google::cloud::EndpointOption>(
                                                       "localhost:9019") // http://[::1]:8035
                               .set<google::cloud::UnifiedCredentialsOption>(google::cloud::MakeInsecureCredentials());

            client_ = std::make_unique<google::cloud::datastore_v1::DatastoreClient>(
                google::cloud::datastore_v1::MakeDatastoreConnection(options));

            try
            {
                google::datastore::v1::Key key;
                key.mutable_partition_id()->set_project_id("ot...");
                auto &path = *key.add_path();
                path.set_kind("Task");
                path.set_name("sampletask1");

                google::datastore::v1::Mutation mutation;
                auto &upsert = *mutation.mutable_upsert();
                *upsert.mutable_key() = key;
                google::datastore::v1::Value value;
                value.set_string_value("Buy milk");
                upsert.mutable_properties()->insert({"description", std::move(value)});

                auto put = client_->Commit(
                    project_id, google::datastore::v1::CommitRequest::NON_TRANSACTIONAL,
                    {mutation});
                if (!put){
                    throw std::move(put).status();
                }

                std::cout << "Saved " << key.DebugString() << " " << put->DebugString()
                          << "\n";

            }
            catch (google::cloud::Status const &status)
            {
                std::cerr << "google::cloud::Status thrown: " << status << "\n";
              
            }
        }

[coryan]

@coryan Thank you. I managed to get that set up. But it still is not working as smoothly as I would have wanted it to. I thought it would connect seamlessly to the Firebase Firestore emulator at port localhost:9019

Apologies, I completely misunderstood what you were trying to do. I may still be wrong about what you are trying to do.

I just sent #14100, with that change I was able to run the quickstart against the emulator embedded with the gcloud CLI.
It occurs to me that this may not be the same emulator you are using, if so, could you share a link to the emulator you do use?

With that said, these two tests worked for me:

# In a separate shell run the datastore emulator.
gcloud beta emulators datastore start --host-port=localhost:9019

DATASTORE_EMULATOR_HOST=localhost:9019 bazel run -- //google/cloud/datastore/quickstart:quickstart foobar

# In a separate shell run the firestore emulator in datastore mode.
gcloud emulators firestore start --database-mode=datastore-mode --host-port=localhost:9019

DATASTORE_EMULATOR_HOST=localhost:9019 bazel run -- //google/cloud/datastore/quickstart:quickstart foobar

as the storage API does with the provided Firebase Storage port 9199.

TIL: firebase has an emulator for Google Cloud Storage. I am glad it works for you, but we don't test against it. If you run into trouble, we (the google-cloud-cpp developers) may not be able to help.

[ericel]

Okay, this is how it goes. I start my Firebase emulators with:
firebase emulators:start --only functions,firestore,storage
My Google Cloud-Cpp storage program is able to connect to and use the Firebase emulator, as I used that as my end point in the C++ storage code config.
Trying to achieve the same results with firestore, which is datastore in firestore-native mode doesn't seem to work, after providing the endpoint from firebase firestore.

└─────────────────────────────────────────────────────────────┘

┌───────────┬────────────────┬─────────────────────────────────┐
│ Emulator │ Host:Port │ View in Emulator UI │
├───────────┼────────────────┼─────────────────────────────────┤
│ Functions │ 127.0.0.1:5001 │ http://127.0.0.1:4000/functions │
├───────────┼────────────────┼─────────────────────────────────┤
│ Firestore │ 127.0.0.1:9019 │ http://127.0.0.1:4000/firestore │
├───────────┼────────────────┼─────────────────────────────────┤
│ Storage │ 127.0.0.1:9199 │ http://127.0.0.1:4000/storage │
└───────────┴────────────────┴─────────────────────────────────┘
Emulator Hub running at 127.0.0.1:4400
Other reserved ports: 4500, 9150

[coryan]

Okay, this is how it goes. I start my Firebase emulators with: firebase emulators:start --only functions,firestore,storage My Google Cloud-Cpp storage program is able to connect to and use the Firebase emulator, as I used that as my end point in the C++ storage code config. Trying to achieve the same results with firestore, which is datastore in firestore-native mode doesn't seem to work, after providing the endpoint from firebase firestore.

I cannot find any documentation that suggests you can run the firebase emulators:start ... and get an emulator in datastore mode. I think you will need to run two commands:

firebase emulators:start --only functions,storage
gcloud emulators firestore start --database-mode=datastore-mode --host-port=localhost:9019