Skip to content
This repository has been archived by the owner on Nov 28, 2023. It is now read-only.

Revive connector with updates in pom.xml and README #8

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Revive connector with updates in pom.xml and README #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
662e1a6
chore(deps): update Spark version+nits
anguillanneuf Oct 20, 2021
0a468b2
docs: set decodeData to true in message receiver
anguillanneuf Oct 20, 2021
398be83
update Spark to 2.4.8
anguillanneuf Oct 21, 2021
ee7284d
chore(deps): update scala to 2.12
anguillanneuf Oct 21, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ import org.apache.spark.rdd._
val SUBSCRIPTION = "<My SUBSCRIPTION>"

var ssc = new StreamingContext(sc,Seconds(5))
var pubsubReceiver = new PubsubReceiver(SUBSCRIPTION, 10)
var pubsubReceiver = new PubsubReceiver(SUBSCRIPTION, 10, true)
val customReceiverStream = ssc.receiverStream(pubsubReceiver)
customReceiverStream.map(x => x).foreachRDD((x: RDD[String]) => println(x.count))
customReceiverStream.map(x => x).foreachRDD((x: RDD[String]) => println(x))
Expand Down Expand Up @@ -85,7 +85,7 @@ from signifai.pubsub import PubsubUtils
SUBSCRIPTION = "<MY SUBSCRIPTION>"

ssc = StreamingContext(sc, 1)
pubsubStream = PubsubUtils.createStream(ssc, SUBSCRIPTION, 5)
pubsubStream = PubsubUtils.createStream(ssc, SUBSCRIPTION, 5, True)
pubsubStream.flatMap(lambda x: x).pprint()
ssc.start()
```
Expand Down
11 changes: 6 additions & 5 deletions java/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,12 +81,13 @@
<maven.shade.plugin.version>3.2.0</maven.shade.plugin.version>
<maven.site.plugin>3.5.1</maven.site.plugin>
<maven.taglist.plugin>2.4</maven.taglist.plugin>
<maven.findbugs.plugin>4.4.2</maven.findbugs.plugin>
<maven.versions.plugin>2.2</maven.versions.plugin>
<apache.maven.rat.version>0.12</apache.maven.rat.version>

<spark.version>2.3.2</spark.version>
<scala.binary.version>2.11</scala.binary.version>
<gcloud.pubsub.version>v1-rev403-1.25.0</gcloud.pubsub.version>
<spark.version>2.4.8</spark.version>
<scala.binary.version>2.12</scala.binary.version>
<gcloud.pubsub.version>v1-rev452-1.25.0</gcloud.pubsub.version>
<jackson.databind.version>2.9.7</jackson.databind.version>
<apache.commons.lang3.version>3.8.1</apache.commons.lang3.version>
<apache.commons.collections4.version>4.2</apache.commons.collections4.version>
Expand Down Expand Up @@ -355,7 +356,7 @@
<plugin>
<groupId>org.apache.rat</groupId>
<artifactId>apache-rat-plugin</artifactId>
<version>${apapache.maven.rat.version}</version>
<version>${apache.maven.rat.version}</version>
</plugin>
</reportPlugins>
<locales>en</locales>
Expand Down Expand Up @@ -453,4 +454,4 @@
<url>s3://maven.signifai.io/release</url>
</repository>
</distributionManagement>
</project>
</project>