Download-tweets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Usando exemplos de Tweets já baixados\n",
    "Esses tweets foram baixados utilizando o Flume. Caso você queria baixar novos tweets, seguir para o tópico 2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Copiando os arquivos locais para o HDFS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!hdfs dfs -copyFromLocal $(pwd)/examples/* /user/lucasbatista/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Trend Topics com MapReduce"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python mrjob-hashtag-rank.py -r hadoop --hadoop-streaming-jar $HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-2.9.2.jar hdfs:///user/lucasbatista/*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Apagando os vértices e arestas do grafo "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from py2neo import Graph\n",
    "graph = Graph(\"bolt://127.0.0.1:7687\")\n",
    "graph.delete_all()\n",
    "\n",
    "print(\"Number of nodes: {} \".format(len(graph.nodes)))\n",
    "print(\"Number of relationships: {} \".format(len(graph.nodes)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Criando o grafo de Hashtags e Tweets usando MRJob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python twitterMRJob.py -r hadoop --hadoop-streaming-jar $HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-2.9.2.jar hdfs:///user/lucasbatista/*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizando o grafo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0, 'resources/python/')\n",
    "\n",
    "from graphdrawer.visgraph import draw\n",
    "\n",
    "options = {\"User\": \"id\", \"Hashtag\": \"hashtag\"}\n",
    "draw(graph, options, physics=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Criando uma App no Twitter\n",
    "\n",
    "## Acessar o endereço abaixo e criar uma App: https://apps.twitter.com/\n",
    "\n",
    "Criar login, senha e logar\n",
    "\n",
    "Criar uma nova App clicando em Create New App\n",
    "\n",
    "Definir os detalhes da aplicação: nome, descrição, website, etc\n",
    "\n",
    "**No menu \"Keys and Tokens\" gerar as chaves da App para usar na configuração do Flume e substituir no arquivo twitterAgent.conf abaixo.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Configurando o agent, source, channel e sink\n",
    "\n",
    "**Agent**: Apenas um agente chamado *TwitterAgent*\n",
    "\n",
    "**Source**: Twitter\n",
    "\n",
    "**Channel**: Memória\n",
    "\n",
    "**Sink**: Registra os dados no HDFS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat twitterAgent.conf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Nota: \n",
    "Usando as configurações descritas no site do Flume - **TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource** - na configuração do source, o arquivo é gerado com caracteres ilegíveis. Assim, iremos utilizar **TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource** e temos que copiar o arquivo flume-sources-1.0-SNAPSHOT.jar para a pasta do Flume para o seu correto funcionamento."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cp resources/flume-sources-1.0-SNAPSHOT.jar ~/resources/local/flume-${FLUME_VERSION}/lib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Apagando os tweets antigos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!hdfs dfs -rm /user/lucasbatista/*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Abrir um terminal e iniciar o FlumeAgent\n",
    "\n",
    "Espere um tempo para o download de alguns tweets\n",
    "``` bash\n",
    "flume-ng agent --conf conf --conf-file twitterAgent.conf --name TwitterAgent -Dflume.logger=INFO,console\n",
    "``` "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TrendTopics com MapReduce\n",
    "### Criando um rank de hashtags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python mrjob-hashtag-rank.py -r hadoop --hadoop-streaming-jar $HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-2.9.2.jar hdfs:///user/lucasbatista/*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Criando grafo de Hashtags e Usuários usando MRJob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from py2neo import Graph\n",
    "graph = Graph(\"bolt://127.0.0.1:7687\")\n",
    "graph.delete_all()\n",
    "\n",
    "print(\"Number of nodes: {} \".format(len(graph.nodes)))\n",
    "print(\"Number of relationships: {} \".format(len(graph.nodes)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python twitterMRJob.py -r hadoop --hadoop-streaming-jar $HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-2.9.2.jar hdfs:///user/lucasbatista/*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualizando o grafo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0, 'resources/python/')\n",
    "\n",
    "from graphdrawer.visgraph import draw\n",
    "\n",
    "options = {\"User\": \"id\", \"Hashtag\": \"hashtag\"}\n",
    "draw(graph, options, physics=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}