Setting up the SMACK stack and a sample app with DC/OS

Prerequesites

• install Vagrant and Virtualbox
• install Ansible
• get an AWS Account. The AWS resources used in this example exceed the free tier, regular AWS fees apply. You have been warned!
• Create an EC2 key pair called dcos-intro
• A Twitter access key

Environment variables

Export

• DCOS_INTRO_AWS_REGION, e.g. with "eu-central-1"
• DCOS_INTRO_AWS_ACCESS_KEY, your AWS access key
• DCOS_INTRO_AWS_SECRET_KEY, your AWS secret key

Setup SMACK Frameworks

• SSH into Vagrant box (vagrant ssh)
• call /vagrant/provisioning/install-smack-frameworks.sh

Wait

• Wait in Marathon (http://<master_load_balancer>/marathon) until all apps are healthy

Setup Kafka brokers and topics

• call /vagrant/provisioning/setup-kafka-broker-and-topic.sh

Setup Cassandra keyspace and schema

• SSH into the master node on AWS
• Start cqlsh with docker run -ti cassandra:2.2.5 cqlsh node-0.cassandra.mesos
• Execute the two lines from /vagrant/provisioning/cassandra.cql

Start ingestion job

• edit /vagrant/provisioning/ingestion.json to add your twitter credentials
• Call /vagrant/provisioning/deploy-ingestion.sh

Start digestion job

• Call /vagrant/provisioning/deploy-digestion.sh

Build Fat Jar for example

• cd into "example"
• call sbt assembly
• Fat Jar will contain both Spark job and ingestion job. You would not have this normally, this is a simplification for the sake of the example

Create Docker container

• call sbt docker