Gera_dados.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 5,
  "metadata": {
    "colab": {
      "name": "Gera_dados.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.8"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/zavaleta/Machine-Learning-and-Fake-News/blob/main/Gera_dados.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "opponent-boston"
      },
      "source": [
        "---\n",
        "# Projeto de Pesquisa \n",
        "> ## Métodos de Machine Learning para Determinar Fake News em Saúde nas Mídias Sociais\n",
        ">> ### Autor: Jorge Zavaleta"
      ],
      "id": "opponent-boston"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "painted-delaware"
      },
      "source": [
        "---\n",
        "[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.4697918.svg)](https://doi.org/10.5281/zenodo.4697918)"
      ],
      "id": "painted-delaware"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0ICW6HeMEJ3l"
      },
      "source": [
        "---\n",
        "# Módulo de Geração de Dados"
      ],
      "id": "0ICW6HeMEJ3l"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7cb29252"
      },
      "source": [
        "> ## Importar bibliotercas necessárias \n"
      ],
      "id": "7cb29252"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b6f759d3"
      },
      "source": [
        "# importar bibliotecas\n",
        "import requests                           #  Usar http\n",
        "from bs4 import BeautifulSoup             #  Usar dados de arquivos HTML e XML\n",
        "from datetime import datetime, date       #  Usar para datas e horários\n",
        "import pandas as pd                       #  Usar com Series e DataFrames\n",
        "import re                                 #  Usar com expressões regulares"
      ],
      "id": "b6f759d3",
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bored-heart"
      },
      "source": [
        "---\n",
        "# 1. Métodos"
      ],
      "id": "bored-heart"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LXU8OyGG1PBH"
      },
      "source": [
        ">## 1.1 Termos de uso e robôs"
      ],
      "id": "LXU8OyGG1PBH"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "medical-conclusion"
      },
      "source": [
        "> A raspagem (**scraping**) quase não tem limitações devido à falta de normas ou leis que definam o que pode ser raspado e o que não pode. No entanto, existem diretrizes que definem o que você deve respeitar. \n",
        "> No Brasil existe la Lei Geral de Proteção de Dados Pessoais (LGPD) com vigência a partir de 2020, que criar um cenário de segurança jurídica para promover a proteção dos dados sensíveis sobre crianças e adolescentes tratados por meio físicos e digitais.\n",
        "\n",
        "> Antes de iniciar qualquer tarefa de raspagem, consulte os **Termos & Condições** e a **Política de Privacidade** do site do qual deseja coletar dados. Se não houver limitação de raspagem, é recomendavél sempre verificar o arquivo **robots.txt** fornecido pelo site, se este for fornecido.\n",
        "\n",
        "> Verificar se existe dados restringidos ou sensíveis\n",
        "> - ```usar: robots.txt```"
      ],
      "id": "medical-conclusion"
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kgLgz3Sr7Ajn",
        "outputId": "b3b285e3-b2c0-487f-cf6b-1453945347fa"
      },
      "source": [
        "# verificando a existencia do arquivo robots.txt\n",
        "url_do_site = 'https://www.globo.com/robots.txt'\n",
        "req = requests.get(url_do_site)\n",
        "print(req.text)"
      ],
      "id": "kgLgz3Sr7Ajn",
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "#\n",
            "# robots.txt\n",
            "#\n",
            "\n",
            "User-Agent: *\n",
            "Disallow: /busca/\n",
            "Disallow: /beta/\n",
            "Disallow: *globo-cdn-src/*\n",
            "Sitemap: http://www.globo.com/sitemap-image.xml\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "current-header"
      },
      "source": [
        "## 1.2 WebScrapping"
      ],
      "id": "current-header"
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "conscious-algebra",
        "outputId": "96ebe17e-79d9-4b06-f562-92ea9b37a787"
      },
      "source": [
        "# metodos webscrapping\n",
        "print('Método Webscrapping')"
      ],
      "id": "conscious-algebra",
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Método Webscrapping\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "included-scheduling"
      },
      "source": [
        "## 1.3 WebCrawler"
      ],
      "id": "included-scheduling"
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "auburn-lindsay",
        "outputId": "bd14345b-3cef-40fb-8516-4ba6cca91401"
      },
      "source": [
        "# metodos webscrapping\n",
        "print('Método WebCrawler')"
      ],
      "id": "auburn-lindsay",
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Método WebCrawler\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "affecting-potential"
      },
      "source": [
        "## 1.4 Selenium"
      ],
      "id": "affecting-potential"
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "beginning-notion",
        "outputId": "e3b8aae8-aa6b-44e3-c80d-3efc5425d810"
      },
      "source": [
        "# metodos webscrapping\n",
        "print('Método Selenium')"
      ],
      "id": "beginning-notion",
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Método Selenium\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "303c8245"
      },
      "source": [
        "# 2. Automatização da Geração de Dados"
      ],
      "id": "303c8245"
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fifth-things",
        "outputId": "b9e23158-75f7-4f36-bd1e-eaf3d3fe8932"
      },
      "source": [
        "# automatizacao\n",
        "print('Automatização da geração de dados')"
      ],
      "id": "fifth-things",
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Automatização da geração de dados\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c8c38f46"
      },
      "source": [
        "# 3. Análise e limpeza de dados"
      ],
      "id": "c8c38f46"
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EA64LzZtGvLs",
        "outputId": "945a1624-f36f-4892-c5b5-9df51d613bca"
      },
      "source": [
        "# Usar dataset do Github\n",
        "!git clone https://github.com/zavaleta/Machine-Learning-and-Fake-News\n"
      ],
      "id": "EA64LzZtGvLs",
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'Machine-Learning-and-Fake-News'...\n",
            "remote: Enumerating objects: 251, done.\u001b[K\n",
            "remote: Counting objects: 100% (251/251), done.\u001b[K\n",
            "remote: Compressing objects: 100% (189/189), done.\u001b[K\n",
            "remote: Total 251 (delta 141), reused 149 (delta 60), pack-reused 0\u001b[K\n",
            "Receiving objects: 100% (251/251), 773.39 KiB | 14.32 MiB/s, done.\n",
            "Resolving deltas: 100% (141/141), done.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 683
        },
        "id": "fe4b5b72",
        "outputId": "3491bd92-c3e0-4591-bc8f-15d0fc25f95e"
      },
      "source": [
        "# leitura do dataset -> github\n",
        "fake_df = pd.read_csv('Machine-Learning-and-Fake-News/data/fake-news-g1_06_03_2021.csv', delimiter =',')\n",
        "fake_df.head()"
      ],
      "id": "fe4b5b72",
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>fonte</th>\n",
              "      <th>author</th>\n",
              "      <th>title</th>\n",
              "      <th>intencionalidade</th>\n",
              "      <th>data</th>\n",
              "      <th>fato</th>\n",
              "      <th>conteudo</th>\n",
              "      <th>autenticidade</th>\n",
              "      <th>tipo</th>\n",
              "      <th>categoria</th>\n",
              "      <th>link</th>\n",
              "      <th>image</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>G1</td>\n",
              "      <td>-</td>\n",
              "      <td>Fato ou Fake lança bot exclusivo para checar c...</td>\n",
              "      <td>xv</td>\n",
              "      <td>2000-01-01 00:00:00</td>\n",
              "      <td>False</td>\n",
              "      <td>É possível por meio do número no WhatsApp faze...</td>\n",
              "      <td>False</td>\n",
              "      <td>noticia</td>\n",
              "      <td>boato</td>\n",
              "      <td>https://g1.globo.com/fato-ou-fake/noticia/2020...</td>\n",
              "      <td>-</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>G1</td>\n",
              "      <td>-</td>\n",
              "      <td>O que é #FATO ou #FAKE sobre o coronavírus</td>\n",
              "      <td>xv</td>\n",
              "      <td>00</td>\n",
              "      <td>False</td>\n",
              "      <td>Vídeos, áudios e posts têm viralizado nas rede...</td>\n",
              "      <td>False</td>\n",
              "      <td>[]</td>\n",
              "      <td>boato</td>\n",
              "      <td>https://g1.globo.com/fato-ou-fake/coronavirus/</td>\n",
              "      <td>https://s2.glbimg.com/vUTxF9w-hBRXE_So7-tFd8nl...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>G1</td>\n",
              "      <td>-</td>\n",
              "      <td>É #FAKE vídeo em que mulher diz que mais de 50...</td>\n",
              "      <td>xv</td>\n",
              "      <td>2000-01-01 00:00:00</td>\n",
              "      <td>False</td>\n",
              "      <td>Números citados em português são referentes a ...</td>\n",
              "      <td>False</td>\n",
              "      <td>noticia</td>\n",
              "      <td>boato</td>\n",
              "      <td>https://g1.globo.com/fato-ou-fake/coronavirus/...</td>\n",
              "      <td>https://s2.glbimg.com/DThH2hqRg3DlLe_9Q7WYpi-n...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>G1</td>\n",
              "      <td>-</td>\n",
              "      <td>É #FAKE que imagens provem que Hospital Moinho...</td>\n",
              "      <td>xv</td>\n",
              "      <td>00</td>\n",
              "      <td>False</td>\n",
              "      <td>Hospital diz que 'o vídeo gravado nas dependên...</td>\n",
              "      <td>False</td>\n",
              "      <td>video</td>\n",
              "      <td>boato</td>\n",
              "      <td>https://g1.globo.com/fato-ou-fake/video/e-fake...</td>\n",
              "      <td>https://s2.glbimg.com/9N-SqOkQrRNvOVtr6hs-31R1...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>G1</td>\n",
              "      <td>-</td>\n",
              "      <td>É #FAKE que imagens provem que Hospital Moinho...</td>\n",
              "      <td>xv</td>\n",
              "      <td>2000-01-01 00:00:00</td>\n",
              "      <td>False</td>\n",
              "      <td>Hospital diz que 'o vídeo gravado nas dependên...</td>\n",
              "      <td>False</td>\n",
              "      <td>noticia</td>\n",
              "      <td>boato</td>\n",
              "      <td>https://g1.globo.com/fato-ou-fake/coronavirus/...</td>\n",
              "      <td>https://s2.glbimg.com/9N-SqOkQrRNvOVtr6hs-31R1...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Unnamed: 0  ...                                              image\n",
              "0           0  ...                                                  -\n",
              "1           1  ...  https://s2.glbimg.com/vUTxF9w-hBRXE_So7-tFd8nl...\n",
              "2           2  ...  https://s2.glbimg.com/DThH2hqRg3DlLe_9Q7WYpi-n...\n",
              "3           3  ...  https://s2.glbimg.com/9N-SqOkQrRNvOVtr6hs-31R1...\n",
              "4           4  ...  https://s2.glbimg.com/9N-SqOkQrRNvOVtr6hs-31R1...\n",
              "\n",
              "[5 rows x 13 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Qlp4ScDqHsA0"
      },
      "source": [
        "# leitura do dataset local\n",
        "#fake_df = pd.read_csv('data/fake-news-g1_06_03_2021.csv', delimiter =',')\n",
        "#fake_df.head()"
      ],
      "id": "Qlp4ScDqHsA0",
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ea19f1f1",
        "outputId": "7c238741-27c8-4efc-8e4e-f1bbf3870f4a"
      },
      "source": [
        "# Metodos de limpeza\n",
        "print('Métodos de limpeza de dados')"
      ],
      "id": "ea19f1f1",
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Métodos de limpeza de dados\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2bae2ce5"
      },
      "source": [
        "# continuar"
      ],
      "id": "2bae2ce5",
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tzXkBLK9CVww"
      },
      "source": [
        "> Citar como:\n",
        ">>  Jorge Zavaleta. (2021). zavaleta/Machine-Learning-and-Fake-News: Repository of Machine Learning and Fake News (ml_fn_v2.0). Zenodo. https://doi.org/10.5281/zenodo.4697918"
      ],
      "id": "tzXkBLK9CVww"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "selected-nurse"
      },
      "source": [
        "---\n",
        "> #### Projeto de Pesquisa CAPES-TecnoDigital,  Copyright &copy;  Jorge Zavaleta, 2021"
      ],
      "id": "selected-nurse"
    }
  ]
}