Skip to content
This repository has been archived by the owner on Sep 16, 2023. It is now read-only.

Commit

Permalink
feat(v1): add Jobs.SubmitJobAsOperation, SparkR and Presto job types …
Browse files Browse the repository at this point in the history 
…to WorkflowTemplates (#137)

* fix Dataproc: add missing `REQUIRED` annotation.

googleapis/googleapis@7504771
commit 75047719f704d61f405cac6d7439637ab36c1232
Author: Google APIs <noreply@google.com>
Date:   Wed Apr 1 18:40:05 2020 -0700

    fix Dataproc: add missing `REQUIRED` annotation.

    PiperOrigin-RevId: 304305096

* Add v1beta2 SubmitJobAsOperation RPC to Dataproc.

googleapis/googleapis@5378173
commit 5378173a889f9c7d83e36e52d38a6267190de692
Author: Google APIs <noreply@google.com>
Date:   Fri Apr 3 05:15:57 2020 -0700

    Add v1beta2 SubmitJobAsOperation RPC to Dataproc.

    PiperOrigin-RevId: 304594381

* Updates to Dataproc v1beta2 API: - Adds SparkR and Presto job types to WorkflowTemplates - Adds new Optional Components - Clarifies usage of some APIs

googleapis/googleapis@cad0f51
commit cad0f5137a70d0d14a8d9acbfcee98e4cd3e9662
Author: Google APIs <noreply@google.com>
Date:   Mon Apr 6 09:55:41 2020 -0700

    Updates to Dataproc v1beta2 API:
    - Adds SparkR and Presto job types to WorkflowTemplates
    - Adds new Optional Components
    - Clarifies usage of some APIs

    PiperOrigin-RevId: 305053062

* Updates Dataproc v1 API: - Adds Dataproc Jobs.SubmitJobAsOperation RPC - Adds SparkR and Presto job types to WorkflowTemplates - Adds new Optional Components - Clarifies usage of some APIs

googleapis/googleapis@9ff6fd3
commit 9ff6fd3b22f99167827e89aae7778408b5e82425
Author: Google APIs <noreply@google.com>
Date:   Mon Apr 6 09:57:56 2020 -0700

    Updates Dataproc v1 API:
    - Adds Dataproc Jobs.SubmitJobAsOperation RPC
    - Adds SparkR and Presto job types to WorkflowTemplates
    - Adds new Optional Components
    - Clarifies usage of some APIs

    PiperOrigin-RevId: 305053617

* docs: change relative URLs to absolute URLs to fix broken links.

googleapis/googleapis@7445911
commit 744591190e828440f72745aef217f883afd1fd71
Author: Google APIs <noreply@google.com>
Date:   Tue Apr 7 13:06:04 2020 -0700

    docs: change relative URLs to absolute URLs to fix broken links.

    PiperOrigin-RevId: 305323909

* chore: allow proto interface changes

Co-authored-by: Jeff Ching <chingor@google.com>
  • Loading branch information
@yoshi-automation @chingor13
yoshi-automation and chingor13 committed Apr 9, 2020
1 parent 043ab3a commit ac18c87
Show file tree
Hide file tree
Showing 97 changed files with 14,403 additions and 1,913 deletions.
2 changes: 1 addition & 1 deletion google-cloud-dataproc-bom/pom.xml
View file
Expand Up @@ -11,7 +11,7 @@
<version>0.5.0</version>
</parent>

<name>Google Cloud dataproc BOM</name>
<name>Google Cloud Dataproc BOM</name>
<url>https://github.com/googleapis/java-dataproc</url>
<description>
BOM for Google Cloud Dataproc
Expand Down
10 changes: 7 additions & 3 deletions ...d-dataproc/src/main/java/com/google/cloud/dataproc/v1/AutoscalingPolicyServiceClient.java
View file
Expand Up @@ -251,7 +251,7 @@ public final AutoscalingPolicy updateAutoscalingPolicy(UpdateAutoscalingPolicyRe
* has the following format: `projects/{project_id}/regions/{region}`
* <p>&#42; For `projects.locations.autoscalingPolicies.create`, the resource name of the
* location has the following format: `projects/{project_id}/locations/{location}`
* @param policy The autoscaling policy to create.
* @param policy Required. The autoscaling policy to create.
* @throws com.google.api.gax.rpc.ApiException if the remote call fails
*/
public final AutoscalingPolicy createAutoscalingPolicy(
Expand Down Expand Up @@ -284,7 +284,7 @@ public final AutoscalingPolicy createAutoscalingPolicy(
* has the following format: `projects/{project_id}/regions/{region}`
* <p>&#42; For `projects.locations.autoscalingPolicies.create`, the resource name of the
* location has the following format: `projects/{project_id}/locations/{location}`
* @param policy The autoscaling policy to create.
* @param policy Required. The autoscaling policy to create.
* @throws com.google.api.gax.rpc.ApiException if the remote call fails
*/
public final AutoscalingPolicy createAutoscalingPolicy(
Expand Down Expand Up @@ -317,7 +317,7 @@ public final AutoscalingPolicy createAutoscalingPolicy(
* has the following format: `projects/{project_id}/regions/{region}`
* <p>&#42; For `projects.locations.autoscalingPolicies.create`, the resource name of the
* location has the following format: `projects/{project_id}/locations/{location}`
* @param policy The autoscaling policy to create.
* @param policy Required. The autoscaling policy to create.
* @throws com.google.api.gax.rpc.ApiException if the remote call fails
*/
public final AutoscalingPolicy createAutoscalingPolicy(String parent, AutoscalingPolicy policy) {
Expand All @@ -335,8 +335,10 @@ public final AutoscalingPolicy createAutoscalingPolicy(String parent, Autoscalin
* <pre><code>
* try (AutoscalingPolicyServiceClient autoscalingPolicyServiceClient = AutoscalingPolicyServiceClient.create()) {
* RegionName parent = RegionName.of("[PROJECT]", "[REGION]");
* AutoscalingPolicy policy = AutoscalingPolicy.newBuilder().build();
* CreateAutoscalingPolicyRequest request = CreateAutoscalingPolicyRequest.newBuilder()
* .setParent(parent.toString())
* .setPolicy(policy)
* .build();
* AutoscalingPolicy response = autoscalingPolicyServiceClient.createAutoscalingPolicy(request);
* }
Expand All @@ -358,8 +360,10 @@ public final AutoscalingPolicy createAutoscalingPolicy(CreateAutoscalingPolicyRe
* <pre><code>
* try (AutoscalingPolicyServiceClient autoscalingPolicyServiceClient = AutoscalingPolicyServiceClient.create()) {
* RegionName parent = RegionName.of("[PROJECT]", "[REGION]");
* AutoscalingPolicy policy = AutoscalingPolicy.newBuilder().build();
* CreateAutoscalingPolicyRequest request = CreateAutoscalingPolicyRequest.newBuilder()
* .setParent(parent.toString())
* .setPolicy(policy)
* .build();
* ApiFuture&lt;AutoscalingPolicy&gt; future = autoscalingPolicyServiceClient.createAutoscalingPolicyCallable().futureCall(request);
* // Do something
Expand Down
10 changes: 5 additions & 5 deletions ...le-cloud-dataproc/src/main/java/com/google/cloud/dataproc/v1/ClusterControllerClient.java
View file
Expand Up @@ -677,7 +677,7 @@ public final UnaryCallable<GetClusterRequest, Cluster> getClusterCallable() {

// AUTO-GENERATED DOCUMENTATION AND METHOD
/**
* Lists all regions/{region}/clusters in a project.
* Lists all regions/{region}/clusters in a project alphabetically.
*
* <p>Sample code:
*
Expand All @@ -704,7 +704,7 @@ public final ListClustersPagedResponse listClusters(String projectId, String reg

// AUTO-GENERATED DOCUMENTATION AND METHOD
/**
* Lists all regions/{region}/clusters in a project.
* Lists all regions/{region}/clusters in a project alphabetically.
*
* <p>Sample code:
*
Expand Down Expand Up @@ -751,7 +751,7 @@ public final ListClustersPagedResponse listClusters(

// AUTO-GENERATED DOCUMENTATION AND METHOD
/**
* Lists all regions/{region}/clusters in a project.
* Lists all regions/{region}/clusters in a project alphabetically.
*
* <p>Sample code:
*
Expand All @@ -778,7 +778,7 @@ public final ListClustersPagedResponse listClusters(ListClustersRequest request)

// AUTO-GENERATED DOCUMENTATION AND METHOD
/**
* Lists all regions/{region}/clusters in a project.
* Lists all regions/{region}/clusters in a project alphabetically.
*
* <p>Sample code:
*
Expand All @@ -805,7 +805,7 @@ public final ListClustersPagedResponse listClusters(ListClustersRequest request)

// AUTO-GENERATED DOCUMENTATION AND METHOD
/**
* Lists all regions/{region}/clusters in a project.
* Lists all regions/{region}/clusters in a project alphabetically.
*
* <p>Sample code:
*
Expand Down
130 changes: 130 additions & 0 deletions google-cloud-dataproc/src/main/java/com/google/cloud/dataproc/v1/JobControllerClient.java
View file
Expand Up @@ -20,14 +20,18 @@
import com.google.api.core.ApiFutures;
import com.google.api.core.BetaApi;
import com.google.api.gax.core.BackgroundResource;
import com.google.api.gax.longrunning.OperationFuture;
import com.google.api.gax.paging.AbstractFixedSizeCollection;
import com.google.api.gax.paging.AbstractPage;
import com.google.api.gax.paging.AbstractPagedListResponse;
import com.google.api.gax.rpc.OperationCallable;
import com.google.api.gax.rpc.PageContext;
import com.google.api.gax.rpc.UnaryCallable;
import com.google.cloud.dataproc.v1.stub.JobControllerStub;
import com.google.cloud.dataproc.v1.stub.JobControllerStubSettings;
import com.google.common.util.concurrent.MoreExecutors;
import com.google.longrunning.Operation;
import com.google.longrunning.OperationsClient;
import com.google.protobuf.Empty;
import java.io.IOException;
import java.util.List;
Expand Down Expand Up @@ -107,6 +111,7 @@
public class JobControllerClient implements BackgroundResource {
private final JobControllerSettings settings;
private final JobControllerStub stub;
private final OperationsClient operationsClient;

/** Constructs an instance of JobControllerClient with default settings. */
public static final JobControllerClient create() throws IOException {
Expand Down Expand Up @@ -139,12 +144,14 @@ public static final JobControllerClient create(JobControllerStub stub) {
protected JobControllerClient(JobControllerSettings settings) throws IOException {
this.settings = settings;
this.stub = ((JobControllerStubSettings) settings.getStubSettings()).createStub();
this.operationsClient = OperationsClient.create(this.stub.getOperationsStub());
}

@BetaApi("A restructuring of stub classes is planned, so this may break in the future")
protected JobControllerClient(JobControllerStub stub) {
this.settings = null;
this.stub = stub;
this.operationsClient = OperationsClient.create(this.stub.getOperationsStub());
}

public final JobControllerSettings getSettings() {
Expand All @@ -156,6 +163,16 @@ public JobControllerStub getStub() {
return stub;
}

/**
* Returns the OperationsClient that can be used to query the status of a long-running operation
* returned by another API method call.
*/
@BetaApi(
"The surface for long-running operations is not stable yet and may change in the future.")
public final OperationsClient getOperationsClient() {
return operationsClient;
}

// AUTO-GENERATED DOCUMENTATION AND METHOD
/**
* Submits a job to a cluster.
Expand Down Expand Up @@ -709,6 +726,119 @@ public final UnaryCallable<DeleteJobRequest, Empty> deleteJobCallable() {
return stub.deleteJobCallable();
}

// AUTO-GENERATED DOCUMENTATION AND METHOD
/**
* Submits job to a cluster.
*
* <p>Sample code:
*
* <pre><code>
* try (JobControllerClient jobControllerClient = JobControllerClient.create()) {
* String projectId = "";
* String region = "";
* Job job = Job.newBuilder().build();
* Job response = jobControllerClient.submitJobAsOperationAsync(projectId, region, job).get();
* }
* </code></pre>
*
* @param projectId Required. The ID of the Google Cloud Platform project that the job belongs to.
* @param region Required. The Dataproc region in which to handle the request.
* @param job Required. The job resource.
* @throws com.google.api.gax.rpc.ApiException if the remote call fails
*/
@BetaApi(
"The surface for long-running operations is not stable yet and may change in the future.")
public final OperationFuture<Job, JobMetadata> submitJobAsOperationAsync(
String projectId, String region, Job job) {
SubmitJobRequest request =
SubmitJobRequest.newBuilder().setProjectId(projectId).setRegion(region).setJob(job).build();
return submitJobAsOperationAsync(request);
}

// AUTO-GENERATED DOCUMENTATION AND METHOD
/**
* Submits job to a cluster.
*
* <p>Sample code:
*
* <pre><code>
* try (JobControllerClient jobControllerClient = JobControllerClient.create()) {
* String projectId = "";
* String region = "";
* Job job = Job.newBuilder().build();
* SubmitJobRequest request = SubmitJobRequest.newBuilder()
* .setProjectId(projectId)
* .setRegion(region)
* .setJob(job)
* .build();
* Job response = jobControllerClient.submitJobAsOperationAsync(request).get();
* }
* </code></pre>
*
* @param request The request object containing all of the parameters for the API call.
* @throws com.google.api.gax.rpc.ApiException if the remote call fails
*/
@BetaApi(
"The surface for long-running operations is not stable yet and may change in the future.")
public final OperationFuture<Job, JobMetadata> submitJobAsOperationAsync(
SubmitJobRequest request) {
return submitJobAsOperationOperationCallable().futureCall(request);
}

// AUTO-GENERATED DOCUMENTATION AND METHOD
/**
* Submits job to a cluster.
*
* <p>Sample code:
*
* <pre><code>
* try (JobControllerClient jobControllerClient = JobControllerClient.create()) {
* String projectId = "";
* String region = "";
* Job job = Job.newBuilder().build();
* SubmitJobRequest request = SubmitJobRequest.newBuilder()
* .setProjectId(projectId)
* .setRegion(region)
* .setJob(job)
* .build();
* OperationFuture&lt;Job, JobMetadata&gt; future = jobControllerClient.submitJobAsOperationOperationCallable().futureCall(request);
* // Do something
* Job response = future.get();
* }
* </code></pre>
*/
@BetaApi("The surface for use by generated code is not stable yet and may change in the future.")
public final OperationCallable<SubmitJobRequest, Job, JobMetadata>
submitJobAsOperationOperationCallable() {
return stub.submitJobAsOperationOperationCallable();
}

// AUTO-GENERATED DOCUMENTATION AND METHOD
/**
* Submits job to a cluster.
*
* <p>Sample code:
*
* <pre><code>
* try (JobControllerClient jobControllerClient = JobControllerClient.create()) {
* String projectId = "";
* String region = "";
* Job job = Job.newBuilder().build();
* SubmitJobRequest request = SubmitJobRequest.newBuilder()
* .setProjectId(projectId)
* .setRegion(region)
* .setJob(job)
* .build();
* ApiFuture&lt;Operation&gt; future = jobControllerClient.submitJobAsOperationCallable().futureCall(request);
* // Do something
* Operation response = future.get();
* }
* </code></pre>
*/
public final UnaryCallable<SubmitJobRequest, Operation> submitJobAsOperationCallable() {
return stub.submitJobAsOperationCallable();
}

@Override
public final void close() {
stub.close();
Expand Down
28 changes: 28 additions & 0 deletions google-cloud-dataproc/src/main/java/com/google/cloud/dataproc/v1/JobControllerSettings.java
View file
Expand Up @@ -25,10 +25,12 @@
import com.google.api.gax.rpc.ApiClientHeaderProvider;
import com.google.api.gax.rpc.ClientContext;
import com.google.api.gax.rpc.ClientSettings;
import com.google.api.gax.rpc.OperationCallSettings;
import com.google.api.gax.rpc.PagedCallSettings;
import com.google.api.gax.rpc.TransportChannelProvider;
import com.google.api.gax.rpc.UnaryCallSettings;
import com.google.cloud.dataproc.v1.stub.JobControllerStubSettings;
import com.google.longrunning.Operation;
import com.google.protobuf.Empty;
import java.io.IOException;
import java.util.List;
Expand Down Expand Up @@ -99,6 +101,19 @@ public UnaryCallSettings<DeleteJobRequest, Empty> deleteJobSettings() {
return ((JobControllerStubSettings) getStubSettings()).deleteJobSettings();
}

/** Returns the object with the settings used for calls to submitJobAsOperation. */
public UnaryCallSettings<SubmitJobRequest, Operation> submitJobAsOperationSettings() {
return ((JobControllerStubSettings) getStubSettings()).submitJobAsOperationSettings();
}

/** Returns the object with the settings used for calls to submitJobAsOperation. */
@BetaApi(
"The surface for long-running operations is not stable yet and may change in the future.")
public OperationCallSettings<SubmitJobRequest, Job, JobMetadata>
submitJobAsOperationOperationSettings() {
return ((JobControllerStubSettings) getStubSettings()).submitJobAsOperationOperationSettings();
}

public static final JobControllerSettings create(JobControllerStubSettings stub)
throws IOException {
return new JobControllerSettings.Builder(stub.toBuilder()).build();
Expand Down Expand Up @@ -227,6 +242,19 @@ public UnaryCallSettings.Builder<DeleteJobRequest, Empty> deleteJobSettings() {
return getStubSettingsBuilder().deleteJobSettings();
}

/** Returns the builder for the settings used for calls to submitJobAsOperation. */
public UnaryCallSettings.Builder<SubmitJobRequest, Operation> submitJobAsOperationSettings() {
return getStubSettingsBuilder().submitJobAsOperationSettings();
}

/** Returns the builder for the settings used for calls to submitJobAsOperation. */
@BetaApi(
"The surface for long-running operations is not stable yet and may change in the future.")
public OperationCallSettings.Builder<SubmitJobRequest, Job, JobMetadata>
submitJobAsOperationOperationSettings() {
return getStubSettingsBuilder().submitJobAsOperationOperationSettings();
}

@Override
public JobControllerSettings build() throws IOException {
return new JobControllerSettings(this);
Expand Down

0 comments on commit ac18c87

Please sign in to comment.