Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: update README about section #16

Merged
merged 12 commits into from Jan 27, 2021
7 changes: 7 additions & 0 deletions .kokoro/nightly/java7.cfg
@@ -0,0 +1,7 @@
# Format: //devtools/kokoro/config/proto/build.proto

# Configure the docker image for kokoro-trampoline.
env_vars: {
key: "TRAMPOLINE_IMAGE"
value: "gcr.io/cloud-devrel-kokoro-resources/java7"
}
3 changes: 3 additions & 0 deletions .kokoro/release/publish_javadoc.cfg
Expand Up @@ -27,3 +27,6 @@ before_action {
}
}
}

# Downloads docfx doclet resource. This will be in ${KOKORO_GFILE_DIR}/<doclet name>
gfile_resources: "/bigstore/cloud-devrel-kokoro-resources/docfx"
11 changes: 11 additions & 0 deletions .readme-partials.yaml
Expand Up @@ -97,3 +97,14 @@ custom_content: |
```java
spark.readStream.format("pubsublite").option("gcp.credentials.key", "<SERVICE_ACCOUNT_JSON_IN_BASE64>")
```
about: |
[Google Cloud Pub/Sub Lite][product-docs] is a zonal, real-time messaging
service that lets you send and receive messages between independent
applications. You can manually configure the throughput and storage capacity
for Pub/Sub Lite systems.

The Pub/Sub Lite Spark connector supports Pub/Sub Lite as an input source to
Apache Spark Structured Streaming in both the default micro-batch processing
mode and the _experimental_ continous processing mode. The connector works in
all Apache Spark distributions, including [Google Cloud Dataproc](https://cloud.google.com/dataproc/docs/), Databricks,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove Databricks for now. This should only be added after the announcement.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

and manual Spark installations.
4 changes: 2 additions & 2 deletions .repo-metadata.json
Expand Up @@ -2,7 +2,7 @@
"name": "pubsublite-spark",
"name_pretty": "Pub/Sub Lite Spark Connector",
"product_documentation": "https://cloud.google.com/pubsub/lite/docs",
"api_description": "is designed to provide reliable,\nmany-to-many, asynchronous messaging between applications. Publisher\napplications can send messages to a topic and other applications can\nsubscribe to that topic to receive the messages. By decoupling senders and\nreceivers, Google Cloud Pub/Sub allows developers to communicate between\nindependently written applications.\n\nCompared to Google Pub/Sub, Pub/Sub Lite provides partitioned zonal data\nstorage with predefined capacity. Both products present a similar API, but\nPub/Sub Lite has more usage caveats.\n\nSee the [Google Pub/Sub Lite docs](https://cloud.google.com/pubsub/quickstart-console#before-you-begin) for more details on how to activate\nPub/Sub Lite for your project, as well as guidance on how to choose between\nCloud Pub/Sub and Pub/Sub Lite.",
"api_description": "Pub/Sub Lite is a zonal, real-time messaging service that lets you send and receive messages between independent applications. You can manually configure the throughput and storage capacity for Pub/Sub Lite systems.",
"client_documentation": "https://googleapis.dev/java/google-cloud-pubsublite/latest/index.html",
"release_level": "alpha",
"transport": "grpc",
Expand All @@ -11,7 +11,7 @@
"min_java_version": 8,
"repo": "googleapis/java-pubsublite-spark",
"repo_short": "java-pubsublite-spark",
"distribution_name": "com.google.cloud:pubsublite-spark",
"distribution_name": "com.google.cloud:pubsublite-spark-sql-streaming",
"codeowner_team": "@googleapis/api-pubsub",
"api_id": "pubsublite.googleapis.com"
}
39 changes: 16 additions & 23 deletions README.md
Expand Up @@ -19,19 +19,19 @@ If you are using Maven, add this to your pom.xml file:
```xml
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>pubsublite-spark</artifactId>
<artifactId>pubsublite-spark-sql-streaming</artifactId>
<version>0.0.0</version>
</dependency>
```

If you are using Gradle without BOM, add this to your dependencies
```Groovy
compile 'com.google.cloud:pubsublite-spark:0.0.0'
compile 'com.google.cloud:pubsublite-spark-sql-streaming:0.0.0'
```

If you are using SBT, add this to your dependencies
```Scala
libraryDependencies += "com.google.cloud" % "pubsublite-spark" % "0.0.0"
libraryDependencies += "com.google.cloud" % "pubsublite-spark-sql-streaming" % "0.0.0"
```

## Authentication
Expand All @@ -50,29 +50,22 @@ You will need to [enable billing][enable-billing] to use Google Pub/Sub Lite Spa

### Installation and setup

You'll need to obtain the `pubsublite-spark` library. See the [Quickstart](#quickstart) section
to add `pubsublite-spark` as a dependency in your code.
You'll need to obtain the `pubsublite-spark-sql-streaming` library. See the [Quickstart](#quickstart) section
to add `pubsublite-spark-sql-streaming` as a dependency in your code.

## About Pub/Sub Lite Spark Connector

[Google Cloud Pub/Sub Lite][product-docs] is a zonal, real-time messaging
service that lets you send and receive messages between independent
applications. You can manually configure the throughput and storage capacity
for Pub/Sub Lite systems.

[Pub/Sub Lite Spark Connector][product-docs] is designed to provide reliable,
many-to-many, asynchronous messaging between applications. Publisher
applications can send messages to a topic and other applications can
subscribe to that topic to receive the messages. By decoupling senders and
receivers, Google Cloud Pub/Sub allows developers to communicate between
independently written applications.
The Pub/Sub Lite Spark connector supports Pub/Sub Lite as an input source to
Apache Spark Structured Streaming in both the default micro-batch processing
mode and the _experimental_ continous processing mode. The connector works in
all Apache Spark distributions, including [Google Cloud Dataproc](https://cloud.google.com/dataproc/docs/), Databricks,
anguillanneuf marked this conversation as resolved.
Show resolved Hide resolved
or manual Spark installations.

Compared to Google Pub/Sub, Pub/Sub Lite provides partitioned zonal data
storage with predefined capacity. Both products present a similar API, but
Pub/Sub Lite has more usage caveats.

See the [Google Pub/Sub Lite docs](https://cloud.google.com/pubsub/quickstart-console#before-you-begin) for more details on how to activate
Pub/Sub Lite for your project, as well as guidance on how to choose between
Cloud Pub/Sub and Pub/Sub Lite.

See the [Pub/Sub Lite Spark Connector client library docs][javadocs] to learn how to
use this Pub/Sub Lite Spark Connector Client Library.


## Requirements
Expand Down Expand Up @@ -238,8 +231,8 @@ Java is a registered trademark of Oracle and/or its affiliates.
[kokoro-badge-image-5]: http://storage.googleapis.com/cloud-devrel-public/java/badges/java-pubsublite-spark/java11.svg
[kokoro-badge-link-5]: http://storage.googleapis.com/cloud-devrel-public/java/badges/java-pubsublite-spark/java11.html
[stability-image]: https://img.shields.io/badge/stability-alpha-orange
[maven-version-image]: https://img.shields.io/maven-central/v/com.google.cloud/pubsublite-spark.svg
[maven-version-link]: https://search.maven.org/search?q=g:com.google.cloud%20AND%20a:pubsublite-spark&core=gav
[maven-version-image]: https://img.shields.io/maven-central/v/com.google.cloud/pubsublite-spark-sql-streaming.svg
[maven-version-link]: https://search.maven.org/search?q=g:com.google.cloud%20AND%20a:pubsublite-spark-sql-streaming&core=gav
[authentication]: https://github.com/googleapis/google-cloud-java#authentication
[developer-console]: https://console.developers.google.com/
[create-project]: https://cloud.google.com/resource-manager/docs/creating-managing-projects
Expand Down
16 changes: 5 additions & 11 deletions pom.xml
Expand Up @@ -8,7 +8,7 @@
<modelVersion>4.0.0</modelVersion>
<groupId>com.google.cloud</groupId>
<artifactId>pubsublite-spark-sql-streaming</artifactId>
<version>0.1.0-SNAPSHOT</version><!-- {x-version-update:pubsublite-spark-sql-streaming:current} -->
<version>0.0.1-SNAPSHOT</version><!-- {x-version-update:pubsublite-spark-sql-streaming:current} -->
<packaging>jar</packaging>
<name>Pub/Sub Lite Spark SQL Streaming</name>
<url>https://github.com/googleapis/java-pubsublite-spark</url>
Expand Down Expand Up @@ -99,20 +99,14 @@
<artifactId>api-common</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.12.1</version>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
</dependency>
<dependency>
<groupId>com.github.ben-manes.caffeine</groupId>
<artifactId>caffeine</artifactId>
<version>2.8.8</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.12.1</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
Expand All @@ -130,14 +124,14 @@
<dependency>
<groupId>com.google.truth</groupId>
<artifactId>truth</artifactId>
<version>1.1</version>
<version>1.1.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<scope>test</scope>
<version>3.7.0</version>
<version>3.7.7</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
Expand Down
55 changes: 55 additions & 0 deletions samples/pom.xml
@@ -0,0 +1,55 @@
<?xml version='1.0' encoding='UTF-8'?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-pubsublite-spark-samples</artifactId>
<version>0.0.1-SNAPSHOT</version><!-- This artifact should not be released -->
<packaging>pom</packaging>
<name>Google Pub/Sub Lite Spark Connector Samples Parent</name>
<url>https://github.com/googleapis/java-pubsublite-spark</url>
<description>
Java idiomatic client for Google Cloud Platform services.
</description>

<!--
The parent pom defines common style checks and testing strategies for our samples.
Removing or replacing it should not affect the execution of the samples in anyway.
-->
<parent>
<groupId>com.google.cloud.samples</groupId>
<artifactId>shared-configuration</artifactId>
<version>1.0.18</version>
</parent>

<properties>
<maven.compiler.target>1.8</maven.compiler.target>
<maven.compiler.source>1.8</maven.compiler.source>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

<modules>
<module>snapshot</module>
<module>snippets</module>
</modules>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-deploy-plugin</artifactId>
<version>2.8.2</version>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
<plugin>
<groupId>org.sonatype.plugins</groupId>
<artifactId>nexus-staging-maven-plugin</artifactId>
<version>1.6.8</version>
<configuration>
<skipNexusStagingDeployMojo>true</skipNexusStagingDeployMojo>
</configuration>
</plugin>
</plugins>
</build>
</project>
83 changes: 83 additions & 0 deletions samples/snapshot/pom.xml
@@ -0,0 +1,83 @@
<?xml version='1.0' encoding='UTF-8'?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.google.cloud</groupId>
<artifactId>pubsublite-spark-snapshot</artifactId>
<packaging>jar</packaging>
<name>Google Pub/Sub Lite Spark Connector Snapshot Samples</name>
<url>https://github.com/googleapis/java-pubsublite-spark</url>

<!--
The parent pom defines common style checks and testing strategies for our samples.
Removing or replacing it should not affect the execution of the samples in anyway.
-->
<parent>
<groupId>com.google.cloud.samples</groupId>
<artifactId>shared-configuration</artifactId>
<version>1.0.12</version>
</parent>

<properties>
<maven.compiler.target>1.8</maven.compiler.target>
<maven.compiler.source>1.8</maven.compiler.source>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

<!-- {x-version-update-start:pubsublite-spark:current} -->
<dependencies>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>pubsublite-spark-sql-streaming</artifactId>
<version>0.0.1-SNAPSHOT</version>
</dependency>
<!-- {x-version-update-end} -->

<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.google.truth</groupId>
<artifactId>truth</artifactId>
<version>1.0.1</version>
<scope>test</scope>
</dependency>
</dependencies>

<!-- compile and run all snippet tests -->
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>3.1.0</version>
<executions>
<execution>
<id>add-snippets-source</id>
<goals>
<goal>add-source</goal>
</goals>
<configuration>
<sources>
<source>../snippets/src/main/java</source>
</sources>
</configuration>
</execution>
<execution>
<id>add-snippets-tests</id>
<goals>
<goal>add-test-source</goal>
</goals>
<configuration>
<sources>
<source>../snippets/src/test/java</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
51 changes: 51 additions & 0 deletions samples/snippets/pom.xml
@@ -0,0 +1,51 @@
<?xml version='1.0' encoding='UTF-8'?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.google.cloud</groupId>
<artifactId>pubsublite-spark-snippets</artifactId>
<packaging>jar</packaging>
<name>Google Pub/Sub Lite Spark Connector Snippets</name>
<url>https://github.com/googleapis/java-pubsublite-spark</url>

<!--
The parent pom defines common style checks and testing strategies for our samples.
Removing or replacing it should not affect the execution of the samples in anyway.
-->
<parent>
<groupId>com.google.cloud.samples</groupId>
<artifactId>shared-configuration</artifactId>
<version>1.0.12</version>
</parent>

<properties>
<maven.compiler.target>1.8</maven.compiler.target>
<maven.compiler.source>1.8</maven.compiler.source>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

<dependencies>
<!-- TODO: switch to libraries-bom after this artifact is included -->
<!-- [START pubsublite-spark_install_without_bom] -->
<!-- [START pubsublite-spark_java_dependencies] -->
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>pubsublite-spark-sql-streaming</artifactId>
<version>0.0.0</version>
</dependency>
<!-- [END pubsublite-spark_java_dependencies] -->
<!-- [END pubsublite-spark_install_without_bom] -->

<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.google.truth</groupId>
<artifactId>truth</artifactId>
<version>1.0.1</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>