Skip to content

Commit

Permalink
fix: remove local copy of beam I/O now that it is merged to upstream (#…
Browse files Browse the repository at this point in the history
…873)

* fix: remove local copy of beam I/O now that it is merged to upstream

* fix: add README back and update

* chore: cleanup after sync

* chore: cleanup after sync
  • Loading branch information
dpcollins-google committed Nov 22, 2021
1 parent 61d17ef commit 0bd9d52
Show file tree
Hide file tree
Showing 58 changed files with 12 additions and 4,667 deletions.
3 changes: 0 additions & 3 deletions README.md
Expand Up @@ -372,9 +372,6 @@ Samples are in the [`samples/`](https://github.com/googleapis/java-pubsublite/tr
| Subscriber Example | [source code](https://github.com/googleapis/java-pubsublite/blob/master/samples/snippets/src/main/java/pubsublite/SubscriberExample.java) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/java-pubsublite&page=editor&open_in_editor=samples/snippets/src/main/java/pubsublite/SubscriberExample.java) |
| Update Subscription Example | [source code](https://github.com/googleapis/java-pubsublite/blob/master/samples/snippets/src/main/java/pubsublite/UpdateSubscriptionExample.java) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/java-pubsublite&page=editor&open_in_editor=samples/snippets/src/main/java/pubsublite/UpdateSubscriptionExample.java) |
| Update Topic Example | [source code](https://github.com/googleapis/java-pubsublite/blob/master/samples/snippets/src/main/java/pubsublite/UpdateTopicExample.java) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/java-pubsublite&page=editor&open_in_editor=samples/snippets/src/main/java/pubsublite/UpdateTopicExample.java) |
| Read Messages Example | [source code](https://github.com/googleapis/java-pubsublite/blob/master/samples/snippets/src/main/java/pubsublite/beam/ReadMessagesExample.java) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/java-pubsublite&page=editor&open_in_editor=samples/snippets/src/main/java/pubsublite/beam/ReadMessagesExample.java) |
| Write Messages Example | [source code](https://github.com/googleapis/java-pubsublite/blob/master/samples/snippets/src/main/java/pubsublite/beam/WriteMessagesExample.java) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/java-pubsublite&page=editor&open_in_editor=samples/snippets/src/main/java/pubsublite/beam/WriteMessagesExample.java) |



## Troubleshooting
Expand Down
1 change: 0 additions & 1 deletion pom.xml
Expand Up @@ -152,7 +152,6 @@
<module>google-cloud-pubsublite</module>
<module>grpc-google-cloud-pubsublite-v1</module>
<module>proto-google-cloud-pubsublite-v1</module>
<module>pubsublite-beam-io</module>
<module>google-cloud-pubsublite-bom</module>
</modules>

Expand Down
36 changes: 12 additions & 24 deletions pubsublite-beam-io/README.md
@@ -1,28 +1,27 @@
# Instructions for PubsubLiteIO usage.

NOTE: Drain is not currently supported on `PubsubLiteIO`. Once Beam 2.29.0 is released, this will be supported
only when `--experiments=use_runner_v2` is enabled. Please contact dataflow customer support to request adding projects
into the allowlist and launch the pipeline with `--experiments=use_runner_v2`.
The minimum version you should use for Pub/Sub Lite IO with beam is 2.34.0.

1. Add the following to your `pom.xml` file to download the Pub/Sub Lite I/O.
1. Add the following to your `pom.xml` file to download the Pub/Sub Lite I/O
included with Beam.
```xml
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>pubsublite-beam-io</artifactId>
<version>0.15.0</version>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-google-cloud-platform</artifactId>
<version>2.34.0</version>
</dependency>
```
1. Create a topic using [`gcloud pubsub lite-topics create`](https://cloud.google.com/sdk/gcloud/reference/pubsub/lite-topics/create)
1. Write some messages using:

```java
import com.google.cloud.pubsublite.beam.PubsubLiteIO;
import com.google.cloud.pubsublite.beam.PublisherOptions;
import com.google.cloud.pubsublite.proto.PubSubMessage;
import com.google.cloud.pubsublite.CloudZone;
import com.google.cloud.pubsublite.ProjectIdOrNumber;
import com.google.cloud.pubsublite.TopicName;
import com.google.cloud.pubsublite.TopicPath;
import org.apache.beam.sdk.io.gcp.pubsublite.PubsubLiteIO;
import org.apache.beam.sdk.io.gcp.pubsublite.PublisherOptions;

...

Expand All @@ -45,13 +44,13 @@ into the allowlist and launch the pipeline with `--experiments=use_runner_v2`.
1. Read some messages using:

```java
import com.google.cloud.pubsublite.beam.PubsubLiteIO;
import com.google.cloud.pubsublite.beam.SubscriberOptions;
import com.google.cloud.pubsublite.proto.SequencedMessage;
import com.google.cloud.pubsublite.CloudZone;
import com.google.cloud.pubsublite.ProjectIdOrNumber;
import com.google.cloud.pubsublite.SubscriptionName;
import com.google.cloud.pubsublite.SubscriptionPath;
import org.apache.beam.sdk.io.gcp.pubsublite.PubsubLiteIO;
import org.apache.beam.sdk.io.gcp.pubsublite.SubscriberOptions;

...

Expand All @@ -72,16 +71,5 @@ into the allowlist and launch the pipeline with `--experiments=use_runner_v2`.
```

### Known issues and workarounds
1. If you encounter `IllegalAccessError` as shown below, place the Pub/Sub Lite
I/O connector as the first dependency before all other Beam dependencies in
your `pom.xml`.

>`java.lang.IllegalAccessError: failed to access class com.google.cloud.
pubsublite.internal.wire.RoutingMetadata from class com.google.cloud.
pubsublite.beam.SubscriberOptions (com.google.cloud.pubsublite.internal.
wire.RoutingMetadata and com.google.cloud.pubsublite.beam.SubscriberOptions
are in unnamed module of loader java.net.URLClassLoader @ff2266c)`

1. Aggregate transforms after fixed windowing will not emit elements in the
DirectRunner for Pub/Sub Lite I/O unless you explicitly fire after watermark
triggers, or use processing-time triggers instead.
1. Drain does not work on the default dataflow runner. You must set the option
`--experiments=use_runner_v2` for draining to function correctly.
19 changes: 0 additions & 19 deletions pubsublite-beam-io/clirr-ignored-differences.xml

This file was deleted.

174 changes: 0 additions & 174 deletions pubsublite-beam-io/pom.xml

This file was deleted.

This file was deleted.

This file was deleted.

0 comments on commit 0bd9d52

Please sign in to comment.