Support for locations in the launch_template method

[laughingman7743]

Environment details

• OS type and version:
• Python version: Python 3.8.11
• pip version: pip 21.0.1
• google-cloud-dataflow-client version: Version: 0.1.2

Steps to reproduce

1. If I specify asia-northeast1 as the location in launch_template, the error occurs.

Code example

dataflow.TemplatesServiceClient().launch_template(
    {
        "project_id": "project"
        "location": "asia-northeast1",
        "gcs_path": "gs://",
        "launch_parameters": LaunchTemplateParameters(
            {
                "job_name": "job_name",
                "parameters": {
                  "foo": "bar"
                },
                "environment": {
                  "service_account_email": "service_account_email",
                },
            }
        ),
    }
)

Stack trace

Task 'dataflow_dag_run': Exception encountered during task execution!
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.FAILED_PRECONDITION
	details = "(1839ef6e6abbd74d): The workflow could not be created, since it was sent to an invalid regional endpoint (asia-northeast1). Please resubmit to a valid Cloud Dataflow regional endpoint. The list of Cloud Dataflow regional endpoints is at https://cloud.google.com/dataflow/docs/concepts/regional-endpoints."
	debug_error_string = "{"created":"@1629880821.759668305","description":"Error received from peer ipv4:172.217.175.42:443","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"(1839ef6e6abbd74d): The workflow could not be created, since it was sent to an invalid regional endpoint (asia-northeast1). Please resubmit to a valid Cloud Dataflow regional endpoint. The list of Cloud Dataflow regional endpoints is at https://cloud.google.com/dataflow/docs/concepts/regional-endpoints.","grpc_status":9}"

If you want to specify Locations, you will likely need to use an API that supports Locations.
https://cloud.google.com/dataflow/docs/reference/rest/#rest-resource:-v1b3.projects.locations.templates

[prgres]

Bump, I’m dealing with the same problem

[meredithslota]

We've identified similar issues in other repos and have escalated this to the API service team since it's not a language-specific library issue. Here are the related issues:

• ListJobsRequest is not using the 'location' filter #31: ListJobsRequest is not using the 'location' filter
• Support for locations in the launch_template method #25: Support for locations in the launch_template method
• The location option of launchFlexTemplate is not working nodejs-dataflow#19: The location option of launchFlexTemplate is not working
• https://github.com/googleapis/java-dataflow/issues/64: Dataflow: The location field of LaunchFlexTemplateRequest building is not working
• ListJobsAsync() returns only us-central1 jobs google-cloud-dotnet#7038: ListJobsAsync() returns only us-central1 jobs (context from issue is that the location parameter isn't working)

Marking this as a P1 bug w/ external label.

[laughingman7743]

It's been abandoned for about two months now, when will it be addressed?

[bbhoss]

I couldn't find an existing issue in the Google Issue Tracker so I created one here. I suggest starring this issue to anyone else that runs across this problem, hopefully it will get some attention from Google soon.

[jcarrizobrass]

Hi guys meanwhile you could use the google-api-python-client.

from googleapiclient.discovery import build
from google.oauth2 import service_account
import os

key_path = os.getenv('GOOGLE_APPLICATION_CREDENTIALS')

credentials = service_account.Credentials.from_service_account_file(
    key_path, scopes=["https://www.googleapis.com/auth/cloud-platform"],
)

service = build('dataflow', 'v1b3',credentials=credentials)

data = {
    "projectId":config["project"],
    "location":"us-east1",
    "body":{
        "environment":{  "tempLocation":"gs://kokoro-rd-2-poc/kokoro/tmp/" },
        "gcsPath":"gs://kokoro-rd-2-poc/kokoro/templates/2bq/template-kokoro-flag",
        "jobName":'python-cloud-function-test-'+str(uuid.uuid4())[:8],
        "parameters":{
            "time": "6 hours",
        }
    }
}

service.projects().locations().templates().create(**data).execute()

service.close()

Just a solution to keep going until someone fix it.

[tgmof]

@jcarrizobrass 's answer is good but I need to work a bit on performances and google-api-python-client is a 50MB library...
@meredithslota The team responsible (https://issuetracker.google.com/issues/208105329) said that they currently have no plan to fix it. The current workaround consist in using another API endpoint which is not exposed by many of the Google client libraries:

v1b3/projects/{projectId}/locations/{location}/templates:launch
https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.templates/launch

additionally to the current:
/v1b3/projects/{projectId}/templates:launch
https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.templates/launch

As a fix, could this endpoint be exposed? Currently my fix is to call v1b3/projects/{projectId}/locations/{location}/templates:launch API directly and it would be great if I could throw away this code to use your pretty-looking and lightweight client.

[tgmof]

For those interested in a quick copy-paste solution, here is a workaround without using the 50MB google-api-python-client.
API doc for parameters: https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.templates/launch

Calling from a Cloud function with the dependency:
google-cloud-dataflow-client==0.3.1

import google
from google.api_core.retry import Retry
from google.cloud import dataflow_v1beta3
from google.auth.transport.requests import AuthorizedSession

dataflow_client = dataflow_v1beta3.TemplatesServiceClient()
credentials, project_id  = google.auth.default()
location = "your-project-location"
http = AuthorizedSession(credentials)
http.mount("https://", HTTPAdapter(max_retries=Retry()))
http.post(
            f"https://{dataflow_client.DEFAULT_ENDPOINT}/v1b3/projects/{project_id}/locations/{location}/templates:launch",
            params={"gcsPath": dataflow_template_path},
            json=body,
        )

[davidcavazos]

The fix has been merged on PR googleapis/gapic-generator-python#1284 and it should be available on release 0.65.2 or later.

I'm closing this, but please re-open if the issue still persists.