A quickstart demo to showcase Hudi functionalities using docker along with support for integration-tests #455

bvaradar · 2018-09-11T13:23:13Z

Comes with foundations for adding docker integration tests. Docker images built with Hadoop 2.8.4 Hive 2.3.3 and Spark 2.3.1

Demo using docker containers with documentation

bvaradar · 2018-09-11T13:24:43Z

@vinothchandar @n3nash : Rebased this branch against master. Tests passes locally. Please use this instead of vinothchandar#4

bvaradar · 2018-09-11T14:11:02Z

@vinothchandar : The build is failing here

[ERROR] Failed to execute goal on project hoodie-hadoop-base-docker: Could not resolve dependencies for project com.uber.hoodie:hoodie-hadoop-base-docker:pom:0.4.4-SNAPSHOT: Could not transfer artifact com.uber.hoodie:hoodie-hadoop-docker:jar:0.4.4-SNAPSHOT from/to Maven repository (https://central.maven.org/maven2/): Host name 'central.maven.org' does not match the certificate subject provided by the peer (CN=repo1.maven.org, O="Sonatype, Inc", L=Fulton, ST=MD, C=US) -> [Help 1]
[ERROR]

This is a brand new module. If I run "mvn clean package", it passes but "mvn clean install", it fails. Have you noticed this before when new modules were added to hudi ?

n3nash · 2018-09-11T20:55:21Z

docs/quickstart.md

+
+Stock Tracker data will be used to showcase both different Hudi Views and the effects of Compaction. 
+
+Take a look at the director `docker/demo/data`. There are 2 batches of stock data - each at 1 minute granularity. 


docs/quickstart.md

sungjuly

This is a huge help to understand the Hoodie, thanks!

docs/quickstart.md

sungjuly · 2018-09-15T04:05:46Z

docs/quickstart.md

+# Schedule a compaction. This will use Spark Launcher to schedule compaction
+hoodie:stock_ticks->compaction schedule
+....
+Compaction instance : 20180910234509


It seems compaction schedule is trying to kick off YARN instead of spark cluster. Here's a log when testing

2018-09-15 03:59:11 INFO TimelineClientImpl:297 - Timeline service address: http://historyserver:8188/ws/v1/timeline/ 2018-09-15 03:59:11 INFO RMProxy:98 - Connecting to ResourceManager at resourcemanager:8032 2018-09-15 03:59:11 INFO AHSProxy:42 - Connecting to Application History server at historyserver/172.22.0.7:10200 2018-09-15 04:14:10 ERROR SparkContext:91 - Error initializing SparkContext. java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: "resourcemanager":8032; java.net.UnknownHostException; For more details see: http://wiki.apache.org/hadoop/UnknownHost at sun.reflect.GeneratedConstructorAccessor20.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

Thanks @sungjuly . I have rebased and this issue should be fixed now.

sungjuly · 2018-09-18T18:43:42Z

docs/quickstart.md

@@ -14,11 +14,11 @@ Check out code and pull it into Intellij as a normal maven project.

 Normally build the maven project, from command line
 ```
-$ mvn clean install -DskipTests
+$ mvn clean install -DskipTests -skipITs


-skipITs should be -DskipITs

bvaradar · 2018-09-21T16:45:34Z

@sungjuly @n3nash : Thanks a lot for providing the feedback. I have revamped this PR and made few improvements. Here is the list:

Reimplement docker files to reduce image sizes. I am seeing 3-4x improvements in docker sizes (from 4 - 5 GB to 0.8-1.5GB).
I have published all the docker images to docker-hub (temporarily using my docker-hub account) so that there is no need to build the docker images as part of running the script.
Made both docker images and compose follow convention so that a single image can satisfy both use-cases
(a) Standalone - Docker images that are published to Docker-hub have inbuilt hudi jars so that they are self-contained.
(b) Mounting Convention - The images allow for overriding those hudi jars with locally built jars allowing for the same setup to be used for running integ-testing against code-changes locally (look at the compose script and hoodie-integ/pom.xml)
Updates quickstart docs to reflect the changes above

I have marked this PR as WIP since I need to showcase incremental view as part of demo.

@n3nash @sungjuly : Can you please follow the quickstart steps and let me know your experience.
Our aim is to have an easy-to-use and hassle-free hudi quickstart. I would appreciate your feedback on that.

sungjuly · 2018-09-21T21:32:08Z

docs/quickstart.md

+
+```
+cd docker
+./setup_demo.sh


I've encountered this. It seems it's related docker version. Here's my env to figure out problems.

docker-engine: 18.06.1-ce (there's no way to downgrade 17.12 since the official site only supports the latest version for Mac)

docker-compose: 1.22.0

➜ docker git:(docker) ✗ ./setup_demo.sh Removing network compose_default Creating network "compose_default" with the default driver Creating kafkabroker ... Creating zookeeper ... error Creating hive-metastore-postgresql ... Creating namenode ... Creating kafkabroker ... error ERROR: for zookeeper Cannot create container for service zookeeper: Conflict. The container name "/zookeeper" is already in use by container "8fb174b9eb8c40bdc20e0ed1b042a793fe49ec3dcea1f353c250b482c7c80019". You have to remove (or rename) that container to be able to reuse that name. ERROR: for kafkabroker Cannot create container for service kafka: Conflict. The container name "/kafkabroker" is alreCreating hive-metastore-postgresql ... error ame) that container to be able to reuse that name.

@sungjuly : ok, can you check "docker container ls" and see if there are stopped/running containers with the same name. If you find them, can you remove the containers using "docker rm".

This is an one-time issue for those who tried this PR before. I have moved the compose script to a different file but the container names are same. Removing the docker containers should hopefully fix the issue.

For a different reason, I reverted my docker (mac) installation to "factory defaults" and that also fixed the problem.

docker env was cleaned up all things before I tested. Here's output for docker container ls

➜ docker git:(docker) ✗ docker container ls CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

It worked after resetting to factory defaults on docker! thank you! @bvaradar

bvaradar · 2018-09-21T21:45:07Z

@sungjuly: I am also online in hoodie slack channel. Ping me there if you need quicker reply.

Thanks,
Balaji.V

sungjuly · 2018-09-21T21:51:33Z

@bvaradar would you please share the link for hoodie slack channel? I don't see any information, thank you!

sungjuly · 2018-09-21T22:00:15Z

nvmd, I found this - #143 (comment)

sungjuly

huge thank you @bvaradar. It's super helpful to understand hoodie more!

sungjuly · 2018-09-22T01:26:18Z

docs/quickstart.md

+
+
+# Execute the compaction
+hoodie:stock_ticks->compaction run --compactionInstant  20180910234509 --parallelism 2 --sparkMemory 1G  --schemaFilePath /var/demo/config/schema.avsc --retry 1  


It would be great if you can add more description compactionInstant value should be updated based on previous results of compactions show all.

sungjuly · 2018-09-22T01:29:58Z

docs/quickstart.md

+
+```
+cd docker
+./setup_demo.sh


It worked after resetting to factory defaults on docker! thank you! @bvaradar

bvaradar

@vinothchandar @n3nash @sungjuly : Updated PR with incremental view demo in quickstart. Ready for review.

bvaradar · 2018-09-24T07:29:25Z

Tests were failing because of log-size limit issue. Fixed as part of PR : #465

n3nash · 2018-09-24T15:33:01Z

docker/compose/docker-compose_hadoop284_hive233_spark231.yml

+services:
+
+  namenode:
+    image: varadarb/hudi-hadoop_2.8.4-namenode:latest


Is this correct ? Should the username be here ? @bvaradar

@n3nash :
I am not aware of any headless docker hub account . I have created a ticket to provide more context and assigned to @vinothchandar

#469

Once the ticket is resolved, we can replace the image locations in Dockerfile, pom.xml and in compose scripts.

n3nash · 2018-09-24T15:33:12Z

docker/compose/docker-compose_hadoop284_hive233_spark231.yml

+      - /tmp/hadoop_data:/hadoop/dfs/data
+
+  historyserver:
+    image: varadarb/hudi-hadoop_2.8.4-history:latest


bvaradar · 2018-09-25T05:41:29Z

docs/quickstart.md

+exit
+```
+
+#### Step 5 (b): Run Spark-SQl Queries


@sungjuly @vinothchandar @n3nash : added spark-sql example here

👍 I've tested with new scenarios. It worked properly! Separately creating MOR/COW tables is a good idea to explain internal works. Thank you for your work as a user!

+1 It is so much better to test stuff now.. :) Thank you for doing this, even as co-creator of the project :) ha ha.

vinothchandar

Works really well. Took me 20 mins to get containers up, from a decent internet connection in India.

vinothchandar · 2018-09-30T04:29:56Z

docs/quickstart.md

+2018-09-24 22:20:00 INFO  OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
+2018-09-24 22:20:00 INFO  SparkContext:54 - Successfully stopped SparkContext
+# Run the following spark-submit command to execute the delta-streamer and ingest to stock_ticks_mor dataset in HDFS
+spark-submit --class com.uber.hoodie.utilities.deltastreamer.HoodieDeltaStreamer /var/hoodie/ws/hoodie-utilities/target/hoodie-utilities-0.4.4-SNAPSHOT.jar --source-class com.uber.hoodie.utilities.sources.JsonKafkaSource --source-ordering-field ts  --target-base-path /user/hive/warehouse/stock_ticks_mor --target-table stock_ticks_mor --props /var/demo/config/kafka-source.properties


this is now hoodie-utilities-0.4.4-SNAPSHOT.jar with the new release. same comment on other 3 occurrances as well.. Can we name the jar differently inside the container without the version? that way it will continue to work? (I understand it takes away knowing what version is being tested.) thoughts?

Also DeltaStreamer now takes a --storage-type COPY_ON_WRITE/MERGE_ON_READ argument which is required.. we can use this and get rid of steps to create the dataset manually via CLI? DeltaStreamer will create the dataset if the basePath does not exist.

@vinothchandar : I have made changes to treat hoodie-utilities jar in the same was as other bundle jars. The utilities jar will now be part of docker image with version removed. It is also available using the alias $HUDI_UTILITIES_BUNDLE. I have updated quickstart and uploaded newer docker images with "latest" tag so you should see the changes when you setup demo again. The 2nd step of explicitly initializing Hudi datasets using Hudi CLI is also removed.

@bvaradar docker/compose#3574 don't think compose up will pull in latest images.. I am going remove all containers and retry.. We probably need to have a better support in the script for optionally also doing a compose pull before?

That worked @bvaradar .. One more issue I found was between recreating containers, the path on host machine /tmp/hadoop* needs to be blown away.. Can we add a rm -rf before the mkdir -p in the setup script.. other than that, verified that building and deltastreamer work..

vinothchandar · 2018-09-30T04:32:06Z

docs/quickstart.md

+Lets run similar queries against M-O-R dataset. Lets look at both 
+ReadOptimized and Realtime views supported by M-O-R dataset
+
+# Run agains ReadOptimized View. Notice that the latest timestamp is 10:29


typo: against

vinothchandar · 2018-09-30T04:32:22Z

docs/quickstart.md

+1 row selected (6.326 seconds)
+
+
+# Run agains Realtime View. Notice that the latest timestamp is again 10:29


typo: against

vinothchandar · 2018-09-30T04:33:18Z

docs/quickstart.md

+exit
+```
+
+#### Step 5 (b): Run Spark-SQl Queries


+1 It is so much better to test stuff now.. :) Thank you for doing this, even as co-creator of the project :) ha ha.

vinothchandar · 2018-09-30T04:33:59Z

docs/quickstart.md

+running in spark-sql
+
+```
+$SPARK_INSTALL/bin/spark-shell --jars $HUDI_SPARK_BUNDLE --master local[2] --driver-class-path $HADOOP_CONF_DIR --conf spark.sql.hive.convertMetastoreParquet=false --deploy-mode client  --driver-memory 1G --executor-memory 3G --num-executors 1  --packages com.databricks:spark-avro_2.11:4.0.0


add docker exec -it adhoc-1 /bin/bash ?

vinothchandar · 2018-09-30T04:34:40Z

docs/quickstart.md

+
+#### Step 7(b): Run Spark SQL Queries
+
+Running the same queries in Spark-SQl:


vinothchandar

Took a pass. other changes seem ok.. Let me know once you have fixed the minor issues I left in comments..

vinothchandar · 2018-09-30T11:38:20Z

docker/demo/config/kafka-source.properties

+hoodie.deltastreamer.schemaprovider.source.schema.file=/var/demo/config/schema.avsc
+hoodie.deltastreamer.schemaprovider.target.schema.file=/var/demo/config/schema.avsc
+# Kafka Source
+#hoodie.deltastreamer.source.kafka.topic=uber_trips


nit: remove this line?

vinothchandar · 2018-09-30T11:39:52Z

docker/hoodie/hadoop/base/docker-maven-plugin-1.1.0.pom

+  <packaging>maven-plugin</packaging>
+  <name>docker-maven-plugin</name>
+  <description>A maven plugin for docker</description>
+  <url>https://github.com/spotify/docker-maven-plugin</url>


is the plugin not published anywhere we can pull down from?

Good catch. This file was accidentally added

bvaradar · 2018-10-01T02:26:26Z

@vinothchandar : Thanks a lot for the review comments. Incorporated them and updated the PR.

vinothchandar · 2018-10-01T13:04:20Z

@bvaradar reported two issues with respect to reiniting containers.. otherwise its good for merging. I will go ahead and do that

vinothchandar · 2018-10-01T13:15:06Z

docker/hoodie/hadoop/hive_base/README.md

@@ -0,0 +1,49 @@
+[![Gitter chat](https://badges.gitter.im/gitterHQ/gitter.png)](https://gitter.im/big-data-europe/Lobby)
+
+# docker-hive


@bvaradar is this file meant to be checked in?

Missed this. I have removed it now.

vinothchandar · 2018-10-01T13:17:27Z

docker/setup_demo.sh

@@ -0,0 +1,15 @@
+# Create host mount directory and copy
+mkdir -p /tmp/hadoop_name


add a protective rm -rf here? otherwise NN will start in safe mode since it does not recognize the data

once the docker containers are stopped, the files are deleted using "rm -rf ". I did not want to do the deletion before docker compose down so that docker containers can be shutdown more cleanly. Will chat f2f to explain more.

vinothchandar · 2018-10-01T13:18:24Z

@bvaradar actually lets do a quick sync before I merge this

bvaradar · 2018-10-01T16:06:48Z

@vinothchandar : Changed the setup_demo script to first do docker-compose pull before docker-compose up in order to pull the latest version of docker images. Addressed other comments.

…er integration tests. Docker images built with Hadoop 2.8.4 Hive 2.3.3 and Spark 2.3.1 and published to docker-hub Look at quickstart document for how to setup docker and run demo

bvaradar requested review from vinothchandar and n3nash September 11, 2018 13:23

bvaradar mentioned this pull request Sep 11, 2018

Docker Container Build and Run setup with foundations for adding integration-tests vinothchandar/hudi#4

Closed

bvaradar force-pushed the docker branch from 3b12f21 to 360099f Compare September 11, 2018 19:46

n3nash reviewed Sep 11, 2018

View reviewed changes

docs/quickstart.md Show resolved Hide resolved

sungjuly reviewed Sep 15, 2018

View reviewed changes

bvaradar force-pushed the docker branch from 360099f to 4f2f76a Compare September 17, 2018 23:56

sungjuly reviewed Sep 18, 2018

View reviewed changes

bvaradar force-pushed the docker branch from 4f2f76a to 312421a Compare September 21, 2018 16:20

bvaradar changed the title ~~Build and deploy Hoodie Docker containers for integration-tests along with demo~~ [WIP] Build and deploy Hoodie Docker containers for integration-tests along with demo Sep 21, 2018

bvaradar force-pushed the docker branch 4 times, most recently from 3369065 to 0654237 Compare September 21, 2018 21:18

sungjuly reviewed Sep 21, 2018

View reviewed changes

bvaradar mentioned this pull request Sep 22, 2018

Travis CI tests needs to be run in quieter mode (WARN log level) to avoid max log-size errors #465

Merged

sungjuly reviewed Sep 22, 2018

View reviewed changes

bvaradar force-pushed the docker branch from 0654237 to e7c7b67 Compare September 24, 2018 07:23

bvaradar changed the title ~~[WIP] Build and deploy Hoodie Docker containers for integration-tests along with demo~~ A quickstart demo to showcase Hudi functionalities using dockers along with support for integration-tests Sep 24, 2018

bvaradar changed the title ~~A quickstart demo to showcase Hudi functionalities using dockers along with support for integration-tests~~ A quickstart demo to showcase Hudi functionalities using docker along with support for integration-tests Sep 24, 2018

bvaradar commented Sep 24, 2018

View reviewed changes

n3nash reviewed Sep 24, 2018

View reviewed changes

bvaradar mentioned this pull request Sep 24, 2018

Create hub.docker.com account for releasing official Hudi docker images #469

Closed

bvaradar force-pushed the docker branch 2 times, most recently from 13ff672 to 6899513 Compare September 25, 2018 05:31

bvaradar commented Sep 25, 2018

View reviewed changes

bvaradar force-pushed the docker branch 3 times, most recently from 0cb91de to a634dc1 Compare September 29, 2018 14:38

vinothchandar reviewed Sep 30, 2018

View reviewed changes

vinothchandar approved these changes Sep 30, 2018

View reviewed changes

bvaradar force-pushed the docker branch from a634dc1 to f3542ac Compare October 1, 2018 02:25

vinothchandar reviewed Oct 1, 2018

View reviewed changes

vinothchandar mentioned this pull request Oct 1, 2018

Create docker image for Hoodie quickstart #422

Closed

bvaradar force-pushed the docker branch from f3542ac to b3780f0 Compare October 1, 2018 16:00

bvaradar force-pushed the docker branch 2 times, most recently from 0aec4bd to 3f3f97b Compare October 1, 2018 20:20

Docker Container Build and Run setup with foundations for adding dock…

26dffce

…er integration tests. Docker images built with Hadoop 2.8.4 Hive 2.3.3 and Spark 2.3.1 and published to docker-hub Look at quickstart document for how to setup docker and run demo

bvaradar force-pushed the docker branch from 3f3f97b to 26dffce Compare October 1, 2018 23:02

vinothchandar merged commit f3418e4 into apache:master Oct 2, 2018


		Stock Tracker data will be used to showcase both different Hudi Views and the effects of Compaction.

		Take a look at the director `docker/demo/data`. There are 2 batches of stock data - each at 1 minute granularity.



		# Execute the compaction
		hoodie:stock_ticks->compaction run --compactionInstant 20180910234509 --parallelism 2 --sparkMemory 1G --schemaFilePath /var/demo/config/schema.avsc --retry 1

		1 row selected (6.326 seconds)


		# Run agains Realtime View. Notice that the latest timestamp is again 10:29


		#### Step 7(b): Run Spark SQL Queries

		Running the same queries in Spark-SQl:

		@@ -0,0 +1,49 @@
		[![Gitter chat](https://badges.gitter.im/gitterHQ/gitter.png)](https://gitter.im/big-data-europe/Lobby)

		# docker-hive

		@@ -0,0 +1,15 @@
		# Create host mount directory and copy
		mkdir -p /tmp/hadoop_name

A quickstart demo to showcase Hudi functionalities using docker along with support for integration-tests #455

A quickstart demo to showcase Hudi functionalities using docker along with support for integration-tests #455

Conversation

bvaradar commented Sep 11, 2018

bvaradar commented Sep 11, 2018

bvaradar commented Sep 11, 2018

Choose a reason for hiding this comment

sungjuly left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bvaradar commented Sep 21, 2018

sungjuly Sep 21, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bvaradar commented Sep 21, 2018

sungjuly commented Sep 21, 2018

sungjuly commented Sep 21, 2018

sungjuly left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bvaradar left a comment

Choose a reason for hiding this comment

bvaradar commented Sep 24, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vinothchandar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vinothchandar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bvaradar commented Oct 1, 2018

vinothchandar commented Oct 1, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vinothchandar commented Oct 1, 2018

bvaradar commented Oct 1, 2018

sungjuly Sep 21, 2018 •

edited