Authenticating using .json file from non-GCE VM not working for me

[kevinlong206]

Greetings,

I am trying to run logstash-input-google_pubsub from a logstash instance that is not hosted in Google Cloud (it is in AWS). The goal is to grab some GKE logs from another environment.

I have tried a few different .json files with various roles for the plugin to authenticate with, and double checked for the path and permissions on these files. I put some extra logger messages in the .rb file, and it seems as though everything is fine with loading my .json file, no errors are thrown during the 'register' hook function.

It appears as though the .json file I specify is simply not being used for the authentication, thus the plugin crashes while executing the 'run' hook:

  `subscriptionAdminClient = SubscriptionAdminClient.create`

https://github.com/logstash-plugins/logstash-input-google_pubsub/blob/master/lib/logstash/inputs/google_pubsub.rb#L241

With the following error:

[2019-11-05T22:25:06,155][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin. Pipeline_id:main Plugin: <LogStash::Inputs::GooglePubSub json_key_file=>"/root/.gce/myproject-qa.json", topic=>"myproject-qa", subscription=>"logstash-sub", id=>"XXXXXXX", project_id=>"myproject-qa", create_subscription=>true, enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_36e3039d-4b33-440e-b0f5-1aff7866d67f", enable_metric=>true, charset=>"UTF-8">, max_messages=>5, include_metadata=>false> Error: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information. Exception: Java::JavaIo::IOException Stack: com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentials(com/google/auth/oauth2/DefaultCredentialsProvider.java:119)

It looks like the ruby code is relying on an google cloud java SDK lib, and that is looking for credentials and not finding them? I assume the environment variable warning is a bit of a red herring? It's not clear to me how the run() function would have access to the .json file which is loaded during register(), but I have never worked with logstash or ruby before.

Thanks for any help!

FYI my config snippet: (.json file in /tmp with 777 permissions purely for troubleshooting purposes, previously was in /root/.gce ).

`[sift-non-prod]root@logstash-bravo-2a-01:/tmp # cat /etc/logstash/conf.d/02-stackdriver.conf
input {
google_pubsub {
# Your GCP project id (name)
project_id => "myproject-qa"

    # The topic name below is currently hard-coded in the plugin. You
    # must first create this topic by hand and ensure you are exporting
    # logging to this pubsub topic.
    topic => "myproject-qa"

    # The subscription name is customizeable. The plugin will attempt to
    # create the subscription (but use the hard-coded topic name above).
    subscription => "logstash-sub"

    # If you are running logstash within GCE, it will use
    # Application Default Credentials and use GCE's metadata
    # service to fetch tokens.  However, if you are running logstash
    # outside of GCE, you will need to specify the service account's
    # JSON key file below.
    json_key_file => "/tmp/myproject-qa.json"

    # Should the plugin attempt to create the subscription on startup?
    # This is not recommended for security reasons but may be useful in
    # some cases.
    create_subscription => true
}

}
[sift-non-prod]root@logstash-bravo-2a-01:/tmp # ls -trlah /tmp/myproject-qa.json
-rwxrwxrwx 1 root root 2.3K Nov 5 22:29 /tmp/myproject-qa.json
[sift-non-prod]root@logstash-bravo-2a-01:/tmp #`

[kevinlong206]

Hmm, I think I found a workaround, setting GOOGLE_APPLICATION_CREDENTIALS=/path/to/myproject.json in the systemd EnvironmentFile for logstash systemd unit. Will post back if it works for sure, but it is getting further now.

[msinghi]

@kevinlong206 I am running into a similar problem. I was able to get around the problem of credentials by setting GOOGLE_APPLICATION_CREDENTIALS in /etc/default/logstash but the plugin still doesn't seem to work. The same config and creds work on my mac but not in AWS. The last message by the pubsub plugin is "[DEBUG] 2020-08-28 00:11:27.994 [[main]<google_pubsub] googlepubsub - Pulling messages from sub " and nothing after that. AWC ec2 instance is running centos 7 Java version 1.8.

On my mac I see this:
[2020-08-27T17:20:48,435][DEBUG][io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler][main] [id: 0x23e0f706, L:/192.168.1.16:54722 - R:pubsub.googleapis.com/172.217.164.106:443] OUTBOUND HEADERS: streamId=3 headers=GrpcHttp2OutboundHeaders[:authority: pubsub.googleapis.com:443, :path: /google.pubsub.v1.Subscriber/StreamingPull, :method: POST, :scheme: https, content-type: application/grpc, te: trailers, user-agent: grpc-java-netty/1.13.1, x-goog-api-client:...

but on ec2, I do not see this message. It seems as if the Nettyclient didn't get created/initialized.