twitter-streams

Kafka Connect Twitter demo

Table of content

• Appendix: Running demo on AWS
  • Prerequisites

A demo setup demonstrates processing of twitter streams. Can be used in booth / conferences on analyze real-time social media data.

Example 1. Acknowledgements

The project structure inspired Confluent Platform demo ^[1] Actual demo logic is inspired by Robin’s blog ^[2] Playbook is available!

Appendix: Running demo on AWS

Prerequisites

• Generate ssh key for github access

  ssh-keygen -t rsa -b 4096 -C "github@email"
  cat .ssh/id_rsa.pub

optional install zsh and git

sudo yum install git
sudo yum install zsh  # (1)
sudo yum install jq   # (2)

1. optional install zsh

2. tool to work with JSON

   • clone repo

mkdir ~/temp/demo-scene
cd ~/temp/demo-scene
git init .
git remote add origin -f https://github.com/confluentinc/demo-scene/
git config core.sparsecheckout true
echo "twitter-streams/*" >> .git/info/sparse-checkout
git pull --depth=2 origin master
cd twitter-streams
ls -lh

• intall JDK 8

sudo yum install java-1.8.0-openjdk-devel
sudo /usr/sbin/alternatives --config java
sudo /usr/sbin/alternatives --config javac
sudo yum remove java-1.7.0-openjdk # <1> optional, remove java 7

Note	How find java in debian - JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:bin/java::")

Optional, install micro because it’s awesome

cd ~
mkdir -p .local/bin
cd .local/bin
curl https://getmic.ro | bash

Update twitter connector config

cd ~
cd twitter-streams/
micro connect_twitter.json

• install docker and docker-compose

  sudo yum update
  sudo yum install -y docker
  sudo usermod -aG docker ec2-user
  sudo service docker start
  sudo su
  curl -L "https://github.com/docker/compose/releases/download/1.21.2/docker-compose-$(uname -s)-$(uname -m)" > /usr/local/bin/docker-compose
  chmod +x /usr/local/bin/docker-compose
  exit
  docker-compose --version

---

1. https://github.com/confluentinc/cp-demo

2. https://www.confluent.io/blog/using-ksql-to-analyse-query-and-transform-data-in-kafka