Skip to content
This repository has been archived by the owner on Nov 29, 2023. It is now read-only.

feat: add SparkR and Presto jobs to WorkflowTemplates; add new optional components; add submit_job_as_operation to v1 (via synth) #21

Merged
merged 3 commits into from May 19, 2020
Merged

feat: add SparkR and Presto jobs to WorkflowTemplates; add new optional components; add submit_job_as_operation to v1 (via synth) #21

merged 3 commits into from May 19, 2020

Conversation

yoshi-automation
Copy link
Contributor

This PR was generated using Autosynth. 🌈

Log from Synthtool 
2020-05-13 05:23:56,490 synthtool [DEBUG] > Executing /home/kbuilder/.cache/synthtool/python-dataproc/synth.py.
On branch autosynth
nothing to commit, working tree clean
2020-05-13 05:23:56,631 synthtool [DEBUG] > Ensuring dependencies.
2020-05-13 05:23:57,261 synthtool [DEBUG] > Cloning googleapis.
2020-05-13 05:23:57,933 synthtool [DEBUG] > Generating code for: //google/cloud/dataproc/v1beta2:dataproc-v1beta2-py.
2020-05-13 05:24:03,770 synthtool [DEBUG] > Copy: google/cloud/dataproc/v1beta2/clusters.proto to /tmpfs/tmp/tmpups1ph99/google/cloud/dataproc_v1beta2/proto/clusters.proto
2020-05-13 05:24:03,770 synthtool [DEBUG] > Copy: google/cloud/dataproc/v1beta2/jobs.proto to /tmpfs/tmp/tmpups1ph99/google/cloud/dataproc_v1beta2/proto/jobs.proto
2020-05-13 05:24:03,770 synthtool [DEBUG] > Copy: google/cloud/dataproc/v1beta2/operations.proto to /tmpfs/tmp/tmpups1ph99/google/cloud/dataproc_v1beta2/proto/operations.proto
2020-05-13 05:24:03,771 synthtool [DEBUG] > Copy: google/cloud/dataproc/v1beta2/autoscaling_policies.proto to /tmpfs/tmp/tmpups1ph99/google/cloud/dataproc_v1beta2/proto/autoscaling_policies.proto
2020-05-13 05:24:03,771 synthtool [DEBUG] > Copy: google/cloud/dataproc/v1beta2/workflow_templates.proto to /tmpfs/tmp/tmpups1ph99/google/cloud/dataproc_v1beta2/proto/workflow_templates.proto
2020-05-13 05:24:03,771 synthtool [DEBUG] > Copy: google/cloud/dataproc/v1beta2/shared.proto to /tmpfs/tmp/tmpups1ph99/google/cloud/dataproc_v1beta2/proto/shared.proto
2020-05-13 05:24:03,771 synthtool [SUCCESS] > Placed proto files into /tmpfs/tmp/tmpups1ph99/google/cloud/dataproc_v1beta2/proto.
2020-05-13 05:24:03,772 synthtool [SUCCESS] > Generated code into /tmpfs/tmp/tmpups1ph99.
2020-05-13 05:24:03,801 synthtool [INFO] > Replaced 'metadata_type=operations_pb2.ClusterOperationMetadata,' in google/cloud/dataproc_v1beta2/gapic/cluster_controller_client.py.
2020-05-13 05:24:03,804 synthtool [WARNING] > No replacements made in google/cloud/dataproc_v1beta2/gapic/cluster_controller_client.py for pattern \s+<strong>Note:</strong>.*
(.*
)+?.*types.FieldMask., maybe replacement is no longer needed?
2020-05-13 05:24:03,805 synthtool [INFO] > Replaced ', and must\n\\s+conform to the following PCRE regular expression:(.*\n)+?.*No more than 32' in google/cloud/dataproc_v1beta2/proto/workflow_templates_pb2.py.
2020-05-13 05:24:03,807 synthtool [INFO] > Replaced ', and must conform to\n\\s+the following regular expression:(.*\n)+?.* No more than' in google/cloud/dataproc_v1beta2/proto/workflow_templates_pb2.py.
2020-05-13 05:24:03,807 synthtool [DEBUG] > Generating code for: //google/cloud/dataproc/v1:dataproc-v1-py.
2020-05-13 05:24:09,752 synthtool [DEBUG] > Copy: google/cloud/dataproc/v1/clusters.proto to /tmpfs/tmp/tmpvaa4eyup/google/cloud/dataproc_v1/proto/clusters.proto
2020-05-13 05:24:09,753 synthtool [DEBUG] > Copy: google/cloud/dataproc/v1/jobs.proto to /tmpfs/tmp/tmpvaa4eyup/google/cloud/dataproc_v1/proto/jobs.proto
2020-05-13 05:24:09,753 synthtool [DEBUG] > Copy: google/cloud/dataproc/v1/operations.proto to /tmpfs/tmp/tmpvaa4eyup/google/cloud/dataproc_v1/proto/operations.proto
2020-05-13 05:24:09,754 synthtool [DEBUG] > Copy: google/cloud/dataproc/v1/autoscaling_policies.proto to /tmpfs/tmp/tmpvaa4eyup/google/cloud/dataproc_v1/proto/autoscaling_policies.proto
2020-05-13 05:24:09,754 synthtool [DEBUG] > Copy: google/cloud/dataproc/v1/workflow_templates.proto to /tmpfs/tmp/tmpvaa4eyup/google/cloud/dataproc_v1/proto/workflow_templates.proto
2020-05-13 05:24:09,754 synthtool [DEBUG] > Copy: google/cloud/dataproc/v1/shared.proto to /tmpfs/tmp/tmpvaa4eyup/google/cloud/dataproc_v1/proto/shared.proto
2020-05-13 05:24:09,754 synthtool [SUCCESS] > Placed proto files into /tmpfs/tmp/tmpvaa4eyup/google/cloud/dataproc_v1/proto.
2020-05-13 05:24:09,754 synthtool [SUCCESS] > Generated code into /tmpfs/tmp/tmpvaa4eyup.
2020-05-13 05:24:09,788 synthtool [INFO] > Replaced 'metadata_type=operations_pb2.ClusterOperationMetadata,' in google/cloud/dataproc_v1/gapic/cluster_controller_client.py.
2020-05-13 05:24:09,789 synthtool [WARNING] > No replacements made in google/cloud/dataproc_v1/gapic/cluster_controller_client.py for pattern \s+<strong>Note:</strong>.*
(.*
)+?.*types.FieldMask., maybe replacement is no longer needed?
2020-05-13 05:24:09,791 synthtool [INFO] > Replaced ', and must\n\\s+conform to the following PCRE regular expression:(.*\n)+?.*No more than 32' in google/cloud/dataproc_v1/proto/workflow_templates_pb2.py.
2020-05-13 05:24:09,792 synthtool [INFO] > Replaced ', and must conform to\n\\s+the following regular expression:(.*\n)+?.* No more than' in google/cloud/dataproc_v1/proto/workflow_templates_pb2.py.
2020-05-13 05:24:09,793 synthtool [INFO] > Replaced '# Generated by the protocol buffer compiler.  DO NOT EDIT!' in google/cloud/dataproc_v1beta2/proto/clusters_pb2.py.
.coveragerc
.flake8
.github/CONTRIBUTING.md
.github/ISSUE_TEMPLATE/bug_report.md
.github/ISSUE_TEMPLATE/feature_request.md
.github/ISSUE_TEMPLATE/support_request.md
.github/PULL_REQUEST_TEMPLATE.md
.github/release-please.yml
.gitignore
.kokoro/build.sh
.kokoro/continuous/common.cfg
.kokoro/continuous/continuous.cfg
.kokoro/docs/common.cfg
.kokoro/docs/docs.cfg
.kokoro/presubmit/common.cfg
.kokoro/presubmit/presubmit.cfg
.kokoro/publish-docs.sh
.kokoro/release.sh
.kokoro/release/common.cfg
.kokoro/release/release.cfg
.kokoro/trampoline.sh
CODE_OF_CONDUCT.md
CONTRIBUTING.rst
LICENSE
MANIFEST.in
docs/_static/custom.css
docs/_templates/layout.html
docs/conf.py.j2
docs/multiprocessing.rst
noxfile.py.j2
renovate.json
setup.cfg
Running session blacken
Creating virtual environment (virtualenv) using python3.6 in .nox/blacken
pip install black==19.3b0
black docs google tests noxfile.py setup.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/__init__.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/__init__.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/__init__.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/gapic/autoscaling_policy_service_client_config.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/docs/conf.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/gapic/cluster_controller_client_config.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/gapic/enums.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/gapic/job_controller_client_config.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/gapic/transports/autoscaling_policy_service_grpc_transport.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/gapic/autoscaling_policy_service_client.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/gapic/cluster_controller_client.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/gapic/transports/cluster_controller_grpc_transport.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/gapic/transports/job_controller_grpc_transport.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/gapic/transports/workflow_template_service_grpc_transport.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/gapic/workflow_template_service_client_config.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/gapic/job_controller_client.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/proto/autoscaling_policies_pb2_grpc.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/proto/clusters_pb2_grpc.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/gapic/workflow_template_service_client.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/proto/jobs_pb2_grpc.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/proto/autoscaling_policies_pb2.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/proto/operations_pb2_grpc.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/proto/operations_pb2.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/proto/shared_pb2_grpc.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/proto/shared_pb2.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/proto/workflow_templates_pb2_grpc.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/types.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/__init__.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/gapic/autoscaling_policy_service_client.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/gapic/autoscaling_policy_service_client_config.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/gapic/cluster_controller_client.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/gapic/cluster_controller_client_config.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/gapic/enums.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/gapic/job_controller_client.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/gapic/job_controller_client_config.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/gapic/transports/autoscaling_policy_service_grpc_transport.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/gapic/transports/cluster_controller_grpc_transport.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/gapic/transports/job_controller_grpc_transport.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/gapic/transports/workflow_template_service_grpc_transport.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/gapic/workflow_template_service_client.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/gapic/workflow_template_service_client_config.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/proto/clusters_pb2.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/proto/autoscaling_policies_pb2_grpc.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/proto/workflow_templates_pb2.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/proto/clusters_pb2_grpc.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/proto/autoscaling_policies_pb2.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/proto/jobs_pb2_grpc.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/proto/operations_pb2.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/proto/operations_pb2_grpc.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/proto/shared_pb2.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/proto/shared_pb2_grpc.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1/proto/jobs_pb2.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/proto/workflow_templates_pb2_grpc.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/types.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/noxfile.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/tests/system/gapic/v1/test_system_cluster_controller_v1.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/tests/system/gapic/v1beta2/test_system_cluster_controller_v1beta2.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/tests/unit/gapic/v1/test_autoscaling_policy_service_client_v1.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/tests/unit/gapic/v1/test_cluster_controller_client_v1.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/tests/unit/gapic/v1/test_job_controller_client_v1.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/tests/unit/gapic/v1/test_workflow_template_service_client_v1.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/tests/unit/gapic/v1beta2/test_autoscaling_policy_service_client_v1beta2.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/tests/unit/gapic/v1beta2/test_cluster_controller_client_v1beta2.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/tests/unit/gapic/v1beta2/test_job_controller_client_v1beta2.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/proto/clusters_pb2.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/proto/workflow_templates_pb2.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/tests/unit/gapic/v1beta2/test_workflow_template_service_client_v1beta2.py
reformatted /home/kbuilder/.cache/synthtool/python-dataproc/google/cloud/dataproc_v1beta2/proto/jobs_pb2.py
All done! ✨ 🍰 ✨
69 files reformatted, 7 files left unchanged.
Session blacken was successful.
2020-05-13 05:24:27,535 synthtool [DEBUG] > Wrote metadata to synth.metadata.

@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label May 13, 2020
@busunkim96 busunkim96 changed the title [CHANGE ME] Re-generated to pick up changes in the API or client library generator. feat: add SparkR and Presto jobs to WorkflowTemplates; add new optional components; add SubmitJobAsOperation to v1 (via synth) May 19, 2020
@busunkim96 busunkim96 changed the title feat: add SparkR and Presto jobs to WorkflowTemplates; add new optional components; add SubmitJobAsOperation to v1 (via synth) feat: add SparkR and Presto jobs to WorkflowTemplates; add new optional components; add submit_job_as_operation to v1 (via synth) May 19, 2020
busunkim96
busunkim96 approved these changes May 19, 2020
View reviewed changes
google/cloud/dataproc_v1/gapic/autoscaling_policy_service_client.py
"""Return a fully-qualified autoscaling_policy string."""
return google.api_core.path_template.expand(
"projects/{project}/regions/{region}/autoscalingPolicies/{autoscaling_policy}",
"projects/{project}/locations/{location}/autoscalingPolicies/{autoscaling_policy}",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check why this resource path helper method is changing

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bradmiro Are region and location interchangeable?

The proto annotation specifies two kinds of patterns.
https://github.com/googleapis/googleapis/blob/7a8875d4bcf51993f6f7aa181066cc5ec95539d5/google/cloud/dataproc/v1/autoscaling_policies.proto#L106-L111

The older config only had region.
https://github.com/googleapis/googleapis/blob/7a8875d4bcf51993f6f7aa181066cc5ec95539d5/google/cloud/dataproc/v1/dataproc_gapic.legacy.yaml#L37-L38

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Luke says that region and zone were used by APIs before this was standardized to location. Given that the proto annotation specifies both paths this change should not impact users.

@busunkim96 busunkim96 requested a review from bradmiro May 19, 2020 22:36
@busunkim96 busunkim96 merged commit 1cf10b6 into master May 19, 2020
gcf-merge-on-green bot pushed a commit that referenced this pull request May 19, 2020
@release-please
chore: release 0.8.0 (#22)
e959874 
🤖 I have created a release \*beep\* \*boop\* 
---
## [0.8.0](https://www.github.com/googleapis/python-dataproc/compare/v0.7.0...v0.8.0) (2020-05-19)


### Features

* add SparkR and Presto jobs to WorkflowTemplates; add new optional components; add submit_job_as_operation to v1 (via synth) ([#21](https://www.github.com/googleapis/python-dataproc/issues/21)) ([1cf10b6](https://www.github.com/googleapis/python-dataproc/commit/1cf10b6b127a63dbeb34958771c2cc8d8cb37099))
---


This PR was generated with [Release Please](https://github.com/googleapis/release-please).
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@busunkim96 busunkim96 busunkim96 approved these changes

@bradmiro bradmiro Awaiting requested review from bradmiro

Assignees
No one assigned
Labels
cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@yoshi-automation @busunkim96 @googlebot