Day008_HW.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 靜態網頁的資料爬蟲策略\n",
    "\n",
    "\n",
    "* 了解靜態網頁的資料爬蟲策略\n",
    "* 認識適用於靜態網頁爬蟲的相關套件工具：Request\n",
    "* 認識適用於靜態網頁爬蟲的相關套件工具：BeatifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 作業目標\n",
    "\n",
    "利用 Request + BeatifulSoup 爬取下列兩個網站內容並解析：\n",
    "\n",
    "1. Dcared 網址： https://www.dcard.tw/f\n",
    "2. 知乎： https://www.zhihu.com/explore\n",
    "\n",
    "並且回答下面問題：\n",
    "\n",
    "1. Request 取回之後該怎麼取出資料，資料型態是什麼？\n",
    "2. 為什麼要使用 BeatifulSoup 處理？處理後的型態是什麼？\n",
    "3. 觀察一下知乎回來的資料好像有點怪怪的，該怎麼解決？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Dcard 網址： https://www.dcard.tw/f"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE html><html lang=\"zh-TW\"><head prefix=\"og: http://ogp.me/ns#\" itemscope=\"\" itemType=\"https://schema.org/WebSite\"><meta charSet=\"utf-8\"/><meta http-equiv=\"X-UA-Compatible\" content=\"IE=edge\"/><meta name=\"apple-mobile-web-app-status-bar-style\" content=\"default\"/><meta name=\"application-name\" content=\"Dcard\"/><meta name=\"apple-itunes-app\" content=\"app-id=951353454\"/><meta name=\"theme-color\" content=\"#006aa6\"/><meta name=\"mobile-web-app-capable\" content=\"yes\"/><meta name=\"apple-mobile-web-app-capable\" content=\"yes\"/><meta name=\"supported-color-schemes\" content=\"light\"/><meta property=\"fb:app_id\" content=\"211628828926493\"/><meta property=\"fb:pages\" content=\"178875832200695,577748865730563,1333515469994506,619122564952487,804004803032067,178024805867764\"/><meta property=\"al:ios:app_store_id\" content=\"951353454\"/><meta property=\"al:ios:app_name\" content=\"Dcard\"/><meta property=\"al:android:package\" content=\"com.sparkslab.dcardreader\"/><meta property=\"al:android:app_name\" content=\"Dcard\"/><link rel=\"manifest\" href=\"/_next/static/manifest.json\"/><link rel=\"preconnect\" href=\"https://www.googletagmanager.com\" crossorigin=\"true\"/><link rel=\"preconnect\" href=\"https://fonts.googleapis.com\" crossorigin=\"true\"/><link rel=\"preconnect\" href=\"https://fonts.gstatic.com\" crossorigin=\"true\"/><link rel=\"dns-prefetch\" href=\"https://www.googletagmanager.com\"/><link rel=\"dns-prefetch\" href=\"https://fonts.googleapis.com\"/><link rel=\"dns-prefetch\" href=\"https://fonts.gstatic.com\"/><link rel=\"dns-prefetch\" href=\"https://www.google-analytics.com\"/><link rel=\"dns-prefetch\" href=\"https://sb.scorecardresearch.com\"/><link rel=\"dns-prefetch\" href=\"https://certify.alexametrics.com\"/><link rel=\"dns-prefetch\" href=\"https://cloudfront-labs.amazonaws.com\"/><link rel=\"dns-prefetch\" href=\"https://bilanx.dcard.tw\"/><link rel=\"dns-prefetch\" href=\"https://imgur.dcard.tw\"/><link rel=\"dns-prefetch\" href=\"https://megapx-assets.dcard.tw\"/><link rel=\"dns-prefetch\" href=\"https://browser.sentry-cdn.com\"/><link rel=\"dns-prefetch\" href=\"https://gcs.dcard.tw\"/><link rel=\"preload\" href=\"https://fonts.googleapis.com/css2?family=Roboto:wght@400;500;700&amp;display=swap\" as=\"style\"/><link rel=\"stylesheet\" href=\"https://fonts.googleapis.com/css2?family=Roboto:wght@400;500;700&amp;display=swap\"/><link rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/_next/static/media/icon-16-8783abf2c964e446614d4d22e23ca2bb.png\"/><link rel=\"icon\" type=\"image/png\" sizes=\"24x24\" href=\"/_next/static/media/icon-24-32dfd860d03fae33428bcfcabbc07df5.png\"/><link rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/_next/static/media/icon-32-7974ce7fd55403bdd4174737b963d694.png\"/><link rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/_next/static/media/icon-192-8ddcd08c7f194cf623c2e405e1689557.png\"/><link rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"57x57\" href=\"/_next/static/media/icon-57-f33c45592fe2c81b9f6fa9f7689e0d17.png\"/><link rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"72x72\" href=\"/_next/static/media/icon-72-4390490d91681428383d075d41977580.png\"/><link rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"76x76\" href=\"/_next/static/media/icon-76-d6563e5ba905e13955938af1697d8015.png\"/><link rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"114x114\" href=\"/_next/static/media/icon-114-b8fad6b9e2bc12d6516b58612539be50.png\"/><link rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"120x120\" href=\"/_next/static/media/icon-120-f7c73abe77f9f8500bb2ab73cce5fa43.png\"/><link rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"144x144\" href=\"/_next/static/media/icon-144-e29b1dfa92ae6b6288184429b165f543.png\"/><link rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"152x152\" href=\"/_next/static/media/icon-152-298677be6ebcf68acf1efd932b138c47.png\"/><link rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"180x180\" href=\"/_next/static/media/icon-180-53b4506f2ba6156616310137fa6f68a4.png\"/><link rel=\"shortcut icon\" type=\"image/png\" href=\"/_next/static/media/icon-32-7974ce7fd55403bdd4174737b963d694.png\"/><meta charSet=\"utf-8\"/><meta name=\"description\" content=\"廣受年輕人喜愛的 Dcard 是台灣最大的匿名交流平台，舉凡時事話題、感情心情、吃喝玩樂、學習工作等，都有卡友陪你聊！\"/><meta property=\"og:description\" content=\"廣受年輕人喜愛的 Dcard 是台灣最大的匿名交流平台，舉凡時事話題、感情心情、吃喝玩樂、學習工作等，都有卡友陪你聊！\"/><meta property=\"og:type\" content=\"website\"/><meta property=\"twitter:description\" content=\"廣受年輕人喜愛的 Dcard 是台灣最大的匿名交流平台，舉凡時事話題、感情心情、吃喝玩樂、學習工作等，都有卡友陪你聊！\"/><meta name=\"image\" itemProp=\"image\" content=\"https://www.dcard.tw/images/landing.png\"/><meta property=\"og:image\" content=\"https://www.dcard.tw/images/landing.png\"/><meta property=\"og:image:secure_url\" content=\"https://www.dcard.tw/images/landing.png\"/><meta property=\"twitter:image\" content=\"https://www.dcard.tw/images/landing.png\"/><link rel=\"preload\" as=\"image\" href=\"/_next/static/media/logo-2ef7b50901acaa284f36a0484329ec05.svg\"/><title>Dcard</title><link rel=\"canonical\" href=\"https://www.dcard.tw/f\"/><meta name=\"url\" itemProp=\"url\" content=\"https://www.dcard.tw/f\"/><meta name=\"viewport\" content=\"width=device-width, initial-scale=1, minimum-scale=1, viewport-fit=cover\"/><meta property=\"og:site_name\" content=\"Dcard\"/><meta property=\"og:title\" content=\"Dcard\"/><meta property=\"og:url\" content=\"https://www.dcard.tw/f\"/><meta property=\"twitter:title\" content=\"Dcard\"/><meta property=\"al:android:url\" content=\"dcard://category/all/%E5%85%A8%E9%83%A8/hot\"/><meta property=\"al:ios:url\" content=\"dcard://category\"/><script type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@type\":\"CollectionPage\",\"name\":\"Dcard\",\"url\":\"https://www.dcard.tw/f\",\"description\":\"廣受年輕人喜愛的 Dcard 是台灣最大的匿名交流平台，舉凡時事話題、感情心情、吃喝玩樂、學習工作等，都有卡友陪你聊！\"}</script><script type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@type\":\"BreadcrumbList\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Dcard\",\"item\":\"https://www.dcard.tw\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"全部\",\"item\":\"https://www.dcard.tw/f\"}]}</script><script type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@type\":\"Organization\",\"name\":\"Dcard\",\"url\":\"https://www.dcard.tw\",\"logo\":{\"@type\":\"ImageObject\",\"url\":\"https://www.dcard.tw/images/favicon_512.png\",\"width\":\"512\",\"height\":\"512\"}}</script><script>(function(d){var a=d.domain.match(/[-\\w]+\\.(dcard\\.(?:cc|hk|io|tw))$/);if(a){d.domain=a[1]}})(document);</script><script>window.dataLayer = window.dataLayer || [];\n",
      "window.dataLayer.push({'gtm.start': new Date().getTime(), event: 'gtm.js'});</script><link rel=\"preload\" as=\"script\" href=\"https://www.googletagmanager.com/gtm.js?id=GTM-TWLTW4M\"/><script defer=\"\" src=\"https://www.googletagmanager.com/gtm.js?id=GTM-TWLTW4M\"></script><meta name=\"next-head-count\" content=\"27\"/><link rel=\"preload\" href=\"/_next/static/css/61458f81c2387d70bfff.css\" as=\"style\"/><link rel=\"stylesheet\" href=\"/_next/static/css/61458f81c2387d70bfff.css\" data-n-g=\"\"/><link rel=\"preload\" href=\"/_next/static/css/7fbedb2bf2bfe8541bc3.css\" as=\"style\"/><link rel=\"stylesheet\" href=\"/_next/static/css/7fbedb2bf2bfe8541bc3.css\" data-n-p=\"\"/><noscript data-n-css=\"\"></noscript><link rel=\"preload\" href=\"/_next/static/chunks/c8f7fe3b0e41be846d5687592cf2018ff6e22687.cba5cb18ac53b79e0e64.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/78d62a35dcc6dedda879910d5b00bd9d449f477e.2b732ea3787b71e58eaa.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/197.5ab87ff47998f4d6f5af.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/163.4a6509c1efeaf7b61f96.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/167.9ca87376eb75ce037e3d.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/3c4859d19f00ad0f5d804de5d236492f40221cb1.61f2f6bc58c2f57b45ec.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/148.7702bc66e6f785824844.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/54.a47ebe9a43911af2884d.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/commons.ca60b6b1ed7d1ab10d11.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/147.1dcd14c069cd038b27e9.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/ac134e40a4ebb2560c9fcdaf12815ad8d162f977.fd24b28472608478a36e.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/ddeb8ccf10a924ccb84e738c520b209d064c92ec.6a5ad665c719475d45d3.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/07ddc8feebba80388bd103699a5e8a2319231b85.6bb329780b731d00a4c4.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/134.372a4d524fefbc3316bb.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/main-2cc8e6610c933369369b.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/webpack-7d103a9d37a758ae2536.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/framework.40eb3c23eef57826c213.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/commons.ca60b6b1ed7d1ab10d11.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/8862a45f59e6b7768410ac3c552121a01e403207.8fa8b25d7a50f967cf07.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/da1128d44cccd5597fb803fb5b683820da185f90.bb0896f524063e854df3.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/f1cc47844cc655ebc1f22fd6781ed86fcd30cfb5.983d75536aa1d6e4c567.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/0927f891e69f6097b2ccfeccb526cee7fa0486df.6c09e50b86893ee4be91.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/75f47be49fc36684a8da0f32687a806aea800236.86a7fad411481b5ce190.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/cc0a0ecf8c272400e541e5e9855fb9efa77e0c73.1c6a2680f1f4bd48497a.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/cab84e0748f9c746e2b1c7ab0e8c907de64d82e0.08638bed3414f540c041.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/5f2b7d946865b7bb2cc171aecd190871cd9195c7.e7cdb752f367b43947a7.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/c99843dc412a5296a43acb9c0ca329ecf0217027.4dfe410ab91165ec1085.js\" as=\"script\"/><link rel=\"preload\" href=\"/_next/static/chunks/ac134e4\n"
     ]
    }
   ],
   "source": [
    "url = 'https://www.dcard.tw/f'\n",
    "r=requests.get(url)\n",
    "\n",
    "r.encoding = 'utf-8'\n",
    "print(r.text[0:10000])\n",
    "html=r.text\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Request 取回之後該怎麼取出資料，資料型態是什麼？ =>\n",
      "出現台灣的次數有11次\n"
     ]
    }
   ],
   "source": [
    "print('Request 取回之後該怎麼取出資料，資料型態是什麼？ =>')\n",
    "\n",
    "type(html)\n",
    "\n",
    "print(\"出現台灣的次數有{}次\".format(html.count(\"台灣\"))) #format的用法會使用{}裝"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "為什麼要使用 BeatifulSoup 處理？處理後的型態是什麼？ => \n",
      "Beautifulsoup處理後的資料型態： <class 'bs4.BeautifulSoup'>\n"
     ]
    }
   ],
   "source": [
    "print('為什麼要使用 BeatifulSoup 處理？處理後的型態是什麼？ => ')\n",
    "\n",
    "html_doc=r.text\n",
    "soup=BeautifulSoup(html_doc,\"html.parser\") #html的字串轉為物件\n",
    "\n",
    "\n",
    "\n",
    "print(\"Beautifulsoup處理後的資料型態：\",type(soup))\n",
    "\n",
    "title=soup.find_all(\"h2\")\n",
    "\n",
    "# print(\"使用BeatifulSoup可以利用html tag去找出所需要的資料\")\n",
    "\n",
    "for i in title:\n",
    "    print(i.span.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. 知乎： https://www.zhihu.com/explore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<html>\r\n",
      "<head><title>400 Bad Request</title></head>\r\n",
      "<body bgcolor=\"white\">\r\n",
      "<center><h1>400 Bad Request</h1></center>\r\n",
      "<hr><center>openresty</center>\r\n",
      "</body>\r\n",
      "</html>\r\n",
      "\n"
     ]
    }
   ],
   "source": [
    "url = 'https://www.zhihu.com/explore'\n",
    "r = requests.get(url)\n",
    "r.encoding = 'utf-8'\n",
    "\n",
    "print(r.text[0:600])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. 觀察一下知乎回來的資料好像有點怪怪的，該怎麼解決？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!doctype html>\n",
      "<html lang=\"zh\" data-hairline=\"true\" data-theme=\"light\"><head><meta charSet=\"utf-8\"/><title data-react-helmet=\"true\">发现 - 知乎</title><meta name=\"viewport\" content=\"width=device-width,initial-scale=1,maximum-scale=1\"/><meta name=\"renderer\" content=\"webkit\"/><meta name=\"force-rendering\" content=\"webkit\"/><meta http-equiv=\"X-UA-Compatible\" content=\"IE=edge,chrome=1\"/><meta name=\"google-site-verification\" content=\"FTeR0c8arOPKh8c5DYh_9uu98_zJbaWw53J-Sch9MTg\"/><meta name=\"description\" property=\"og:description\" content=\"知乎，中文互联网最大的问答社区和创作者聚集的原创内容平台，于 2011 年 1 月正式上线，以「让人们更好地分享知识、经验和见解，\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "url = 'https://www.zhihu.com/explore'\n",
    "my_header={\"user-agent\":\"000\"}\n",
    "\n",
    "r = requests.get(url,headers=my_header) #bad request 需加入header \n",
    "\n",
    "r.encoding = 'utf-8'\n",
    "print(r.text[0:600])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}