17. Parse HTML by Beautiful Soup 4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 爬虫基础: Requests + BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "“爬虫”，是访问互联网网页 ->定位网页元素 -> 爬取网页内容的过程。<br>\n",
    "在实际工作中，访问的工作交给urllib或者requests完成；爬取的工作则交给xpath, BeatifulSoup乃至正则合作完成。<br>\n",
    "想学习爬虫知识，首先需要了解HTTP基础请求方法：get和post"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HTTP方法：GET和POST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GET方法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color='red'>请注意，查询字符串（名称/值对）是在 GET 请求的 URL 中发送的</font><br>\n",
    "如：```https://www.baidu.com/s?ie=utf-8&wd=python```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "有关 GET 请求的其他一些注释：\n",
    "\n",
    "* GET 请求可被缓存\n",
    "* GET 请求保留在浏览器历史记录中\n",
    "* GET 请求可被收藏为书签\n",
    "* GET 请求不应在处理敏感数据时使用\n",
    "* GET 请求有长度限制\n",
    "* GET 请求只应当用于取回数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### POST方法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color='red'>请注意，查询字符串（名称/值对）是在 POST 请求的 HTTP 消息主体中发送的</font><br>\n",
    "```\n",
    "POST http://dcms-ml-dev.dc68032.easn.morningstar.com/automation/api/namesimilar HTTP/1.1\n",
    "Host: dcms-ml-dev.dc68032.easn.morningstar.com\n",
    "Connection: keep-alive\n",
    "Content-Length: 103\n",
    "accept: application/json\n",
    "Origin: http://dcms-ml-dev.dc68032.easn.morningstar.com\n",
    "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36\n",
    "Content-Type: application/json\n",
    "Referer: http://dcms-ml-dev.dc68032.easn.morningstar.com/apidocs/\n",
    "Accept-Encoding: gzip, deflate\n",
    "Accept-Language: en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7\n",
    "{\n",
    "  \"country\": \"\",\n",
    "  \"onlyactive\": \"0\",\n",
    "  \"querytype\": \"share\",\n",
    "  \"text\": \"S&P 500\",\n",
    "  \"universe\": \"\"\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "有关 POST 请求的其他一些注释：\n",
    "\n",
    "* POST 请求不会被缓存\n",
    "* POST 请求不会保留在浏览器历史记录中\n",
    "* POST 不能被收藏为书签\n",
    "* POST 请求对数据长度没有要求"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./image/2018-09-17 12_15_17-HTTP 方法GET对比POST.png\" width=\"100%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "requests是一个很实用的Python HTTP客户端库，编写爬虫和测试服务器响应数据时经常会用到。可以说，Requests 完全满足如今网络的需求。<br>\n",
    "Requests的官方文档 http://docs.python-requests.org/en/master/ <br>\n",
    "安装方式一般采用```pip install requests``` <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GET请求"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Response [200]>\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "r = requests.get('https://www.baidu.com')\n",
    "print(r)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 带参数的Get请求\n",
    "这里需要加入headers，给一个user-agent伪装一下自己"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Response: <Response [200]>, Whole URL: https://www.baidu.com/s?ie=utf-8&wd=%E7%9C%8B%E7%97%85\n"
     ]
    }
   ],
   "source": [
    "parameters = {'ie': 'utf-8', 'wd':'看病'}\n",
    "headers = {\n",
    "\"User-Agent\": \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36\"}\n",
    "requestforbing = requests.get('https://www.baidu.com/s', \n",
    "                              params=parameters, \n",
    "                              headers=headers)\n",
    "# requestforbing = requests.get('https://www.baidu.com/s', params=parameters)\n",
    "# print('Response: {0}, Whole URL: {1}, Text: {2}'.format(requestforbing, requestforbing.url, requestforbing.content))\n",
    "print('Response: {0}, Whole URL: {1}'.format(requestforbing, requestforbing.url))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 将请求的网页内容保存到本地"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Response [200]>\n"
     ]
    }
   ],
   "source": [
    "rstream = requests.get('https://www.baidu.com/s', \n",
    "                       params=parameters, \n",
    "                       headers=headers, \n",
    "                       stream=True)\n",
    "print(rstream)\n",
    "try:\n",
    "    with open('./html/baiduresult.html', 'wb') as fd:\n",
    "        for chunk in rstream.iter_content(1000):\n",
    "            fd.write(chunk)\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[baiduresult.html](./html/baiduresult.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### POST请求"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "大部分的业务应用，其实都是POST请求。<br>\n",
    "现在POST请求的参数一般通过JSON字符串组成，返回的结果往往也是JSON字符串<br>\n",
    "以下是具体的例子。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以Form的形式发送请求"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"args\": {}, \n",
      "  \"data\": \"\", \n",
      "  \"files\": {}, \n",
      "  \"form\": {\n",
      "    \"key1\": \"value1\", \n",
      "    \"key2\": \"value2\"\n",
      "  }, \n",
      "  \"headers\": {\n",
      "    \"Accept\": \"*/*\", \n",
      "    \"Accept-Encoding\": \"gzip, deflate\", \n",
      "    \"Content-Length\": \"23\", \n",
      "    \"Content-Type\": \"application/x-www-form-urlencoded\", \n",
      "    \"Host\": \"httpbin.org\", \n",
      "    \"User-Agent\": \"python-requests/2.20.0\", \n",
      "    \"X-Bluecoat-Via\": \"fc8ec97cbb42049b\"\n",
      "  }, \n",
      "  \"json\": null, \n",
      "  \"origin\": \"183.62.146.29, 183.62.146.29\", \n",
      "  \"url\": \"https://httpbin.org/post\"\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import json\n",
    "url = 'http://httpbin.org/post'\n",
    "d = {'key1': 'value1', 'key2': 'value2'}\n",
    "r = requests.post(url, data=d)\n",
    "print(r.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以Json的形式发送请求"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"args\": {}, \n",
      "  \"data\": \"{\\\"key1\\\": \\\"value1\\\", \\\"key2\\\": \\\"value2\\\"}\", \n",
      "  \"files\": {}, \n",
      "  \"form\": {}, \n",
      "  \"headers\": {\n",
      "    \"Accept\": \"*/*\", \n",
      "    \"Accept-Encoding\": \"gzip, deflate\", \n",
      "    \"Content-Length\": \"36\", \n",
      "    \"Host\": \"httpbin.org\", \n",
      "    \"User-Agent\": \"python-requests/2.20.0\", \n",
      "    \"X-Bluecoat-Via\": \"fc8ec97cbb42049b\"\n",
      "  }, \n",
      "  \"json\": {\n",
      "    \"key1\": \"value1\", \n",
      "    \"key2\": \"value2\"\n",
      "  }, \n",
      "  \"origin\": \"183.62.146.29, 183.62.146.29\", \n",
      "  \"url\": \"https://httpbin.org/post\"\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "url = 'http://httpbin.org/post'\n",
    "s = json.dumps({'key1': 'value1', 'key2': 'value2'})\n",
    "r = requests.post(url, data=s)\n",
    "print(r.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "简单来说，Beautiful Soup是python的一个库，最主要的功能是从网页抓取数据。官方解释如下：<br>\n",
    "```\n",
    "Beautiful Soup提供一些简单的、python式的函数用来处理导航、搜索、修改分析树等功能。它是一个工具箱，通过解析文档为用户提供需要抓取的数据，因为简单，所以不需要多少代码就可以写出一个完整的应用程序。\n",
    "\n",
    "Beautiful Soup自动将输入文档转换为Unicode编码，输出文档转换为utf-8编码。你不需要考虑编码方式，除非文档没有指定一个编码方式，这时，Beautiful Soup就不能自动识别编码方式了。然后，你仅仅需要说明一下原始编码方式就可以了。\n",
    "\n",
    "Beautiful Soup已成为和lxml、html5lib一样出色的python解释器，为用户灵活地提供不同的解析策略或强劲的速度。\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 安装"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "pip install beautifulsoup4\n",
    "pip install lxml\n",
    "pip install html5lib\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful Soup支持Python标准库中的HTML解析器,还支持一些第三方的解析器，如果我们不安装它，则 Python 会使用 Python默认的解析器，lxml 解析器更加强大，速度更快，推荐安装。<br>\n",
    "<font color='red'>lxml处理具有多个pre标签的html的时候，只能解析第一个pre标签，此时建议还是通过BeautifulSoup(markup, \"html.parser\")进行解析网页</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./image/2018-09-17 20_05_34-Python爬虫利器二之Beautiful Soup的用法.png\" width=\"100%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "特别详细的内容，可以参考官方文档，下面就开始正式举例说明了~~~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[官方文档](https://beautifulsoup.readthedocs.io/zh_CN/latest/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "之前应星宇所托，写了个爬取山东地区汽油柴油价格行情的爬虫示例。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/2018-09-17 20_11_54.png' width='100%'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "点击第一个链接，可以进入明细页面:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/2018-09-17 20_14_16.png' width='100%'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如何实现呢？我们一步一步来~~~"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 导入所需要的python包\n",
    "import re\n",
    "import requests\n",
    "from pyquery import PyQuery as pq\n",
    "from bs4 import  BeautifulSoup\n",
    "import pandas as pd\n",
    "import os\n",
    "from IPython.display import display, HTML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "设置url，header，搜索文本变量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"http://www.baidu.com/s?ie=utf-8&wd={0}&tn=monline_4_dg\"\n",
    "headers = {\n",
    "    \"User-Agent\":\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/600.5.17 (KHTML, like Gecko) Version/8.0.5 Safari/600.5.17\"\n",
    "}\n",
    "searchtext = \"2018年9月17日 山东地区汽油柴油价格行情 东方财富\"\n",
    "url = url.format(searchtext)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过requests完成查询，并且通过pyquery获取特定属性下面的链接信息"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('http://www.baidu.com/link?url=zQboJU2YO411sCt6FT1d2MdXNXflmVRJ0u7oSqDAk1zmHZF-KEd06yrXSRPqt6wy37K0BAPCzEtfPskSLGAHFkzXfmEZVWaURtsIfrpPUGO', b'2018\\xe5\\xb9\\xb49\\xe6\\x9c\\x8817\\xe6\\x97\\xa5\\xe5\\xb1\\xb1\\xe4\\xb8\\x9c\\xe5\\x9c\\xb0\\xe5\\x8c\\xba\\xe6\\xb1\\xbd\\xe6\\xb2\\xb9\\xe6\\x9f\\xb4\\xe6\\xb2\\xb9\\xe4\\xbb\\xb7\\xe6\\xa0\\xbc\\xe8\\xa1\\x8c\\xe6\\x83\\x85 _ \\xe4\\xb8\\x9c\\xe6\\x96\\xb9\\xe8\\xb4\\xa2\\xe5\\xaf\\x8c\\xe7\\xbd\\x91'), ('http://www.baidu.com/link?url=0pFXD-T0SMhYT1LG6d2R_pE5mvaJV3q5t_LZrnG0ZUCKVxqptMTCdFhtYnE02JAVdBJyEjFArqM5kHIvzT53XEw_2OyViTuiRrV_Bg2dqta', b'2018\\xe5\\xb9\\xb49\\xe6\\x9c\\x8817\\xe6\\x97\\xa5\\xe5\\x8d\\x8e\\xe4\\xb8\\x9c\\xe5\\x9c\\xb0\\xe5\\x8c\\xba\\xe6\\xb1\\xbd\\xe6\\xb2\\xb9\\xe6\\x9f\\xb4\\xe6\\xb2\\xb9\\xe4\\xbb\\xb7\\xe6\\xa0\\xbc\\xe8\\xa1\\x8c\\xe6\\x83\\x85 _ \\xe4\\xb8\\x9c\\xe6\\x96\\xb9\\xe8\\xb4\\xa2\\xe5\\xaf\\x8c\\xe7\\xbd\\x91'), ('http://www.baidu.com/link?url=brrbXcVc7O-sWg_obWya7dBcRJOkVPj-8LiiRqjiYUvMvimQgC78VDyMrPdqo6yQXl4LmajSuXYxx1vVSFcslY-niAUqzJ7cOHhCkxVqteW', b'2018\\xe5\\xb9\\xb47\\xe6\\x9c\\x8817\\xe6\\x97\\xa5\\xe9\\x99\\x95\\xe8\\xa5\\xbf\\xe5\\x9c\\xb0\\xe5\\x8c\\xba\\xe6\\xb1\\xbd\\xe6\\xb2\\xb9\\xe6\\x9f\\xb4\\xe6\\xb2\\xb9\\xe4\\xbb\\xb7\\xe6\\xa0\\xbc\\xe8\\xa1\\x8c\\xe6\\x83\\x85 _ \\xe4\\xb8\\x9c\\xe6\\x96\\xb9\\xe8\\xb4\\xa2\\xe5\\xaf\\x8c\\xe7\\xbd\\x91'), ('http://www.baidu.com/link?url=rGnY50yZoeuY__6vS86cJc00foGCYSqG6sq7KkNWBij04Qe-UGuTr93x1DZQbeWC1RDnXchGIg2LU_09OwzLPzSFH_3_JMYol317hUX6ChC', b'2018\\xe5\\xb9\\xb49\\xe6\\x9c\\x8812\\xe6\\x97\\xa5\\xe9\\x99\\x95\\xe8\\xa5\\xbf\\xe5\\x9c\\xb0\\xe5\\x8c\\xba\\xe6\\xb1\\xbd\\xe6\\xb2\\xb9\\xe6\\x9f\\xb4\\xe6\\xb2\\xb9\\xe4\\xbb\\xb7\\xe6\\xa0\\xbc\\xe8\\xa1\\x8c\\xe6\\x83\\x85 _ \\xe4\\xb8\\x9c\\xe6\\x96\\xb9\\xe8\\xb4\\xa2\\xe5\\xaf\\x8c\\xe7\\xbd\\x91'), ('http://www.baidu.com/link?url=t-0eA8AGGRWQ_MQpHfqZxGa3iz7v9Sk1v8c6PGtYC4QyNihFaHB82OpL3uli15RS', b'2015\\xe5\\xb9\\xb411\\xe6\\x9c\\x8817\\xe6\\x97\\xa5\\xe5\\xb1\\xb1\\xe4\\xb8\\x9c\\xe5\\x9c\\xb0\\xe5\\x8c\\xba\\xe6\\xb1\\xbd\\xe6\\xb2\\xb9\\xe6\\x9f\\xb4\\xe6\\xb2\\xb9\\xe4\\xbb\\xb7\\xe6\\xa0\\xbc\\xe8\\xa1\\x8c\\xe6\\x83\\x85'), ('http://www.baidu.com/link?url=J6vdBBbI8X0cOJ4Od41wY_de6KIUBovIWZZ_vQgYmkRVRq6UaE16h1qQduhPP4CYoa8Koc76JsUYn7u6YjgdCq', b'\\xe5\\x85\\xa8\\xe5\\x9b\\xbd\\xe6\\xb2\\xb9\\xe4\\xbb\\xb7\\xe6\\x95\\xb0\\xe6\\x8d\\xae\\xe8\\xb5\\xb0\\xe5\\x8a\\xbf\\xe4\\xb8\\x80\\xe8\\xa7\\x88 _ \\xe6\\x95\\xb0\\xe6\\x8d\\xae\\xe4\\xb8\\xad\\xe5\\xbf\\x83 _ \\xe4\\xb8\\x9c\\xe6\\x96\\xb9\\xe8\\xb4\\xa2\\xe5\\xaf\\x8c\\xe7\\xbd\\x91'), ('http://www.baidu.com/link?url=BdOv2wStwbb1Mz_JnkvA03Ugsk2te8Z_IFvx1rm8CAvogtIxw7B8lNzCKjvIC8ClwjsV0XIJI7PVPJXWp19m6q9N2ExHa-dFoWIVY9POHjW', b'3\\xe6\\x9c\\x889\\xe6\\x97\\xa5\\xe5\\x94\\x90\\xe5\\xb1\\xb1\\xe5\\x9c\\xb0\\xe5\\x8c\\xba\\xe6\\xb1\\xbd\\xe6\\xb2\\xb9\\xe6\\x9f\\xb4\\xe6\\xb2\\xb9\\xe4\\xbb\\xb7\\xe6\\xa0\\xbc\\xe6\\x8c\\x81\\xe7\\xa8\\xb3 \\xe4\\xb8\\xad\\xe5\\x9b\\xbd\\xe8\\xb4\\xa2\\xe7\\xbb\\x8f\\xe9\\x97\\xa8\\xe6\\x88\\xb7,\\xe6\\x8f\\x90\\xe4\\xbe\\x9b\\xe4\\xb8\\x93\\xe4\\xb8\\x9a\\xe7\\x9a\\x84\\xe8\\xb4\\xa2\\xe7\\xbb\\x8f\\xe3\\x80\\x81...'), ('http://www.baidu.com/link?url=S4uCf6LwYdLOaKN4PfjUHn1KXIW6Eghxq6hJfW6HUvGmg8OziHGKLZIlu_qR71JHLPZPwW47ff3xTJH8zImBF_', b'3\\xe6\\x9c\\x8813\\xe6\\x97\\xa5\\xe5\\xb1\\xb1\\xe4\\xb8\\x9c\\xe5\\x9c\\xb0\\xe5\\x8c\\xba\\xe6\\xb1\\xbd\\xe6\\xb2\\xb9\\xe6\\x9f\\xb4\\xe6\\xb2\\xb9\\xe4\\xbb\\xb7\\xe6\\xa0\\xbc\\xe8\\xa1\\x8c\\xe6\\x83\\x85 _ \\xe4\\xb8\\x9c\\xe6\\x96\\xb9\\xe8\\xb4\\xa2\\xe5\\xaf\\x8c\\xe7\\xbd\\x91'), ('http://www.baidu.com/link?url=-wX3N0C11ymWsTaelZE42XPA4Ji1teo-jsiK_P1E73E4ht74SGWdaSk4Gaow7QNtgYArI6l5HzSHt2oKylYc-K', b'2018\\xe5\\xb9\\xb42\\xe6\\x9c\\x8818\\xe6\\x97\\xa5\\xe6\\xb2\\xb3\\xe5\\x8c\\x97\\xe5\\x9c\\xb0\\xe5\\x8c\\xba\\xe6\\xb1\\xbd\\xe6\\xb2\\xb9\\xe6\\x9f\\xb4\\xe6\\xb2\\xb9\\xe4\\xbb\\xb7\\xe6\\xa0\\xbc\\xe8\\xa1\\x8c\\xe6\\x83\\x85 _ \\xe4\\xb8\\x9c\\xe6\\x96\\xb9\\xe8\\xb4\\xa2\\xe5\\xaf\\x8c\\xe7\\xbd\\x91'), ('http://www.baidu.com/link?url=t-0eA8AGGRWQ_MQpHfqZxGa3iz7v9Sk1v8c6PGtYC4Rn5L12F2b4_qXzpVaGLNCb', b'2016\\xe5\\xb9\\xb43\\xe6\\x9c\\x883\\xe6\\x97\\xa5\\xe9\\x99\\x95\\xe8\\xa5\\xbf\\xe5\\x9c\\xb0\\xe5\\x8c\\xba\\xe6\\xb1\\xbd\\xe6\\xb2\\xb9\\xe6\\x9f\\xb4\\xe6\\xb2\\xb9\\xe4\\xbb\\xb7\\xe6\\xa0\\xbc\\xe8\\xa1\\x8c\\xe6\\x83\\x85-\\xe6\\x96\\xb0\\xe9\\x97\\xbb\\xe9\\xa2\\x91\\xe9\\x81\\x93-\\xe6\\x89\\x8b\\xe6\\x9c\\xba\\xe6\\x90\\x9c\\xe7\\x8b\\x90')]\n"
     ]
    }
   ],
   "source": [
    "response = requests.get(url, headers=headers)\n",
    "response.encoding = 'utf-8'\n",
    "# print(response.text)\n",
    "page = pq(response.text)\n",
    "baiduurls = [(site.attr('href'), site.text().encode('utf-8')) for site in\n",
    "                page('div.result.c-container  h3.t  a').items()]\n",
    "print(baiduurls)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因为百度做了url转义，所以需要再用requests做一次get操作，从而获取真正需要获取的链接"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('http://finance.eastmoney.com/news/1356,20180917946941610.html', '2018年9月17日山东地区汽油柴油价格行情 _ 东方财富网'), ('http://finance.eastmoney.com/news/1356,20180917946941397.html', '2018年9月17日华东地区汽油柴油价格行情 _ 东方财富网'), ('http://futures.eastmoney.com/news/1514,20180717908141283.html', '2018年7月17日陕西地区汽油柴油价格行情 _ 东方财富网'), ('http://futures.eastmoney.com/news/1514,20180912944475986.html', '2018年9月12日陕西地区汽油柴油价格行情 _ 东方财富网'), ('http://www.sohu.com/a/42268295_119536', '2015年11月17日山东地区汽油柴油价格行情'), ('http://data.eastmoney.com/cjsj/oil_default.html', '全国油价数据走势一览 _ 数据中心 _ 东方财富网'), ('http://appcert.eastmoney.com/info/detail/201903091064569266', '3月9日唐山地区汽油柴油价格持稳 中国财经门户,提供专业的财经、...'), ('http://forex.eastmoney.com/a/201903131068429498.html', '3月13日山东地区汽油柴油价格行情 _ 东方财富网'), ('http://forex.eastmoney.com/a/201902181046357466.html', '2018年2月18日河北地区汽油柴油价格行情 _ 东方财富网'), ('http://www.sohu.com/a/61671920_119536', '2016年3月3日陕西地区汽油柴油价格行情-新闻频道-手机搜狐')]\n"
     ]
    }
   ],
   "source": [
    "originalURLs = []\n",
    "for tmpurl in baiduurls:\n",
    "    tmpPage = requests.get(tmpurl[0], allow_redirects=False)\n",
    "#     print(tmpPage.text, tmpPage.headers)\n",
    "    if tmpPage.status_code == 200:\n",
    "#         print('200')\n",
    "        urlMatch = re.search(r'URL=\\'(.*?)\\'', tmpPage.text.encode('utf-8'), re.S)\n",
    "        originalURLs.append((urlMatch.group(1), tmpurl[1].decode(\"utf-8\")))\n",
    "    elif tmpPage.status_code == 302:\n",
    "#         print('302')\n",
    "        originalURLs.append((tmpPage.headers.get('location'), tmpurl[1].decode(\"utf-8\")))\n",
    "    else:\n",
    "        print('No URL found!!')\n",
    "print(originalURLs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "既然真正的url已经都拿到了，那么我们就可以进入具体的网页一探究竟"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['2018年9月17日', '山东地区汽油柴油价格行情', '东方财富']\n",
      "东方财富网获得信息：2018年9月17日山东地区汽油柴油价格行情 _ 东方财富网, 网址：http://finance.eastmoney.com/news/1356,20180917946941610.html\n"
     ]
    }
   ],
   "source": [
    "searcharray = searchtext.split()\n",
    "searchdate = ''\n",
    "findurl = ''\n",
    "print(searcharray)\n",
    "for url in originalURLs:\n",
    "#     print(url[1], url[0])\n",
    "    if len(searcharray) >=2 \\\n",
    "            and searcharray[0] in url[1] \\\n",
    "            and searcharray[1] in url[1] \\\n",
    "            and ('futures.eastmoney' in url[0] or \n",
    "                 'finance.eastmoney' in url[0]):\n",
    "        print(\"东方财富网获得信息：{0}, 网址：{1}\".format(url[1], url[0]))\n",
    "        searchdate = searchtext.split()[0]\n",
    "        findurl = url[0]\n",
    "        break\n",
    "#             parsegasinfo(searchdate, url[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "既然拿到最终需要去爬取的URL了，那么就需要BeautifulSoup去拿文本了"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "2018年9月17日山东地区汽油柴油价格行情 _ 东方财富网\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\r\n",
      "        function isMobile() {\r\n",
      "            var ua = navigator.userAgent;\r\n",
      "            var res = false;\r\n",
      "            var ipad = ua.match(/(iPad).*OS\\s([\\d_]+)/),\r\n",
      "                isIphone = !ipad && ua.match(/(iPhone\\sOS)\\s([\\d_]+)/),\r\n",
      "                isAndroid = ua.match(/(Android)\\s+([\\d.]+)/),\r\n",
      "                isMobile = isIphone || isAndroid;\r\n",
      "            if (isMobile) {\r\n",
      "                res = true;\r\n",
      "            } else {\r\n",
      "                res = false;\r\n",
      "            }\r\n",
      "            return res;\r\n",
      "        }\r\n",
      "        if (isMobile()) {\r\n",
      "            location.href = \"https://emwap.eastmoney.com/info/detail/20180917946941610\";\r\n",
      "        }\r\n",
      "    \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 2018年9月17日山东地区汽油柴油价格行情 _ 东方财富网\n"
     ]
    }
   ],
   "source": [
    "fullgastextinfo = requests.get(findurl)\n",
    "fullgastextinfo.encoding = 'utf-8'\n",
    "# 这里使用lxml作为BS4的解析引擎\n",
    "soup = BeautifulSoup(fullgastextinfo.text, features='lxml')\n",
    "print(soup.head.text, soup.title.text)\n",
    "sourcetxtfile = './output/sourcetxt/source_{0}.txt'.format(searchdate)\n",
    "# 将爬取的信息去空白行之后，存入本地文件\n",
    "with open(sourcetxtfile, 'w', encoding='utf-8') as f:\n",
    "    for index, line in enumerate(soup.text.split('\\n')):\n",
    "        if line.split():\n",
    "#             print(\"write to file\")\n",
    "#             print(\"count:\", index + 1, \"content:\", line)\n",
    "            f.write(line + '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "根据之前的知识，查看具体油价信息的地址"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/2018-09-17 20_47_51.png' width='100%' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "获得XPATH示例：\n",
    "```\n",
    "//*[@id=\"ContentBody\"]/p[1]\n",
    "//*[@id=\"ContentBody\"]/p[2]\n",
    "...\n",
    "```\n",
    "既然确定文本一定在非常规矩的段落中，那就可以干活了"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "既然知道文本在div中，且id为ContentBody，那么就可以直接定位div部分"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接着，我们再去找段落，用刚刚的结果，再通过find_all('p')就搞定了"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<p>　　山东东营华星石化今日<span id=\"Info.353\"><a class=\"infokey \" href=\"http://data.eastmoney.com/cjsj/yjtz/default.html\" target=\"_blank\">成品油</a></span>报价：国五92#汽油8850元/吨，密度0.72；国五95#汽油无报价，密度0.72；国六92#汽油(高效)8900元/吨，新出；国五0#普柴7800元/吨，密度0.84；国五0#车柴7900元/吨；国六0#车柴7900元/吨；催柴无报价，停报。</p>, <p>　　山东东营正和集团今日成品油报价：国四90#汽油无货，密度0.73；国四93#汽油无报价，密度0.739；国四97#汽油无报价，密度0.748；国六92#汽油8850元/吨，密度0.74；国六95#汽油9000元/吨，密度0.74；国五0#普通柴油7830元/吨。燃料油13#7830元/吨；国五0#车柴7930元/吨；国六0#车柴7930元/吨。</p>, <p>　　山东东营神驰炼厂今日成品油报价：国六89#组分汽油8700元/吨，密度0.728；国六92#汽油8800元/吨，密度0.737；国六95#汽油8900元/吨，密度0.740；普通0#柴油无报价，密度0.823；国六-0#车柴无报价，密度0.813；国六高标92#汽油无报价，密度0.73；国六高标95#汽油无报价，密度0.73；0#普柴无报价，密度0.823；国六0#车柴无报价，密度0.8139；清洁柴油0#7600。</p>, <p class=\"res-edit\">\r\n",
      "                (责任编辑：DF120)\r\n",
      "            </p>]\n"
     ]
    }
   ],
   "source": [
    "# help(soup.find_all)\n",
    "print(soup.find('div', id='ContentBody').find_all('p'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们接着再往下来~~~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意singlegasprovider.text的用法，其能够去除段落中的所有html标签。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "　　山东东营华星石化今日成品油报价：国五92#汽油8850元/吨，密度0.72；国五95#汽油无报价，密度0.72；国六92#汽油(高效)8900元/吨，新出；国五0#普柴7800元/吨，密度0.84；国五0#车柴7900元/吨；国六0#车柴7900元/吨；催柴无报价，停报。\n",
      "　　山东东营正和集团今日成品油报价：国四90#汽油无货，密度0.73；国四93#汽油无报价，密度0.739；国四97#汽油无报价，密度0.748；国六92#汽油8850元/吨，密度0.74；国六95#汽油9000元/吨，密度0.74；国五0#普通柴油7830元/吨。燃料油13#7830元/吨；国五0#车柴7930元/吨；国六0#车柴7930元/吨。\n",
      "　　山东东营神驰炼厂今日成品油报价：国六89#组分汽油8700元/吨，密度0.728；国六92#汽油8800元/吨，密度0.737；国六95#汽油8900元/吨，密度0.740；普通0#柴油无报价，密度0.823；国六-0#车柴无报价，密度0.813；国六高标92#汽油无报价，密度0.73；国六高标95#汽油无报价，密度0.73；0#普柴无报价，密度0.823；国六0#车柴无报价，密度0.8139；清洁柴油0#7600。\n",
      "\r\n",
      "                (责任编辑：DF120)\r\n",
      "            \n"
     ]
    }
   ],
   "source": [
    "for singlegasprovider in soup.find('div', id='ContentBody').find_all('p'):\n",
    "    print(singlegasprovider.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "讲真，我是期望把数据表格化的，目前来看有点乱，所以还需要加入正则方面的内容。<br>\n",
    "然后pandas就出场了，这里有些大材小用，但是用的顺手就选它了~~~"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "wwqq 油无报rwww ioopp ee\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "temp = r'wwqq；油无报rwww。，：ioopp；。。ee'\n",
    "temp = re.sub('(\\W)', ' ', temp)\n",
    "temp = re.sub('( ){2,}', ' ', temp)\n",
    "print(temp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "国四93#汽油无报价 国四93#汽油\n"
     ]
    }
   ],
   "source": [
    "gasinfo = r'国四93#汽油无报价'\n",
    "pattern = re.compile(r'(.*)(无报价|无货)')\n",
    "gastype = pattern.search(gasinfo).group(1)\n",
    "print(pattern.search(gasinfo).group(0), gastype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后用正则硬刚："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>releasedate</th>\n",
       "      <th>provider</th>\n",
       "      <th>producttype</th>\n",
       "      <th>productprice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2018年9月17日</td>\n",
       "      <td>山东东营华星石化</td>\n",
       "      <td>国五92#汽油</td>\n",
       "      <td>8850元/吨</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2018年9月17日</td>\n",
       "      <td>山东东营华星石化</td>\n",
       "      <td>国五95#汽油</td>\n",
       "      <td>无报价</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2018年9月17日</td>\n",
       "      <td>山东东营华星石化</td>\n",
       "      <td>国六92#汽油(高效)</td>\n",
       "      <td>8900元/吨</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2018年9月17日</td>\n",
       "      <td>山东东营华星石化</td>\n",
       "      <td>国五0#普柴</td>\n",
       "      <td>7800元/吨</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2018年9月17日</td>\n",
       "      <td>山东东营华星石化</td>\n",
       "      <td>国五0#车柴</td>\n",
       "      <td>7900元/吨</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2018年9月17日</td>\n",
       "      <td>山东东营华星石化</td>\n",
       "      <td>国六0#车柴</td>\n",
       "      <td>7900元/吨</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2018年9月17日</td>\n",
       "      <td>山东东营华星石化</td>\n",
       "      <td>催柴</td>\n",
       "      <td>无报价</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2018年9月17日</td>\n",
       "      <td>山东东营正和集团</td>\n",
       "      <td>国四90#汽油</td>\n",
       "      <td>无报价</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2018年9月17日</td>\n",
       "      <td>山东东营正和集团</td>\n",
       "      <td>国四93#汽油</td>\n",
       "      <td>无报价</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2018年9月17日</td>\n",
       "      <td>山东东营正和集团</td>\n",
       "      <td>国四97#汽油</td>\n",
       "      <td>无报价</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2018年9月17日</td>\n",
       "      <td>山东东营正和集团</td>\n",
       "      <td>国六92#汽油</td>\n",
       "      <td>8850元/吨</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2018年9月17日</td>\n",
       "      <td>山东东营正和集团</td>\n",
       "      <td>国六95#汽油</td>\n",
       "      <td>9000元/吨</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>2018年9月17日</td>\n",
       "      <td>山东东营正和集团</td>\n",
       "      <td>国五0#普通柴油</td>\n",
       "      <td>7830元/吨</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>2018年9月17日</td>\n",
       "      <td>山东东营正和集团</td>\n",
       "      <td>燃料油13#</td>\n",
       "      <td>7830元/吨</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>2018年9月17日</td>\n",
       "      <td>山东东营正和集团</td>\n",
       "      <td>国五0#车柴</td>\n",
       "      <td>7930元/吨</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>2018年9月17日</td>\n",
       "      <td>山东东营正和集团</td>\n",
       "      <td>国六0#车柴</td>\n",
       "      <td>7930元/吨</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>2018年9月17日</td>\n",
       "      <td>山东东营神驰炼厂</td>\n",
       "      <td>国六89#组分汽油</td>\n",
       "      <td>8700元/吨</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>2018年9月17日</td>\n",
       "      <td>山东东营神驰炼厂</td>\n",
       "      <td>国六92#汽油</td>\n",
       "      <td>8800元/吨</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>2018年9月17日</td>\n",
       "      <td>山东东营神驰炼厂</td>\n",
       "      <td>国六95#汽油</td>\n",
       "      <td>8900元/吨</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>2018年9月17日</td>\n",
       "      <td>山东东营神驰炼厂</td>\n",
       "      <td>普通0#柴油</td>\n",
       "      <td>无报价</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>2018年9月17日</td>\n",
       "      <td>山东东营神驰炼厂</td>\n",
       "      <td>国六-0#车柴</td>\n",
       "      <td>无报价</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>2018年9月17日</td>\n",
       "      <td>山东东营神驰炼厂</td>\n",
       "      <td>国六高标92#汽油</td>\n",
       "      <td>无报价</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>2018年9月17日</td>\n",
       "      <td>山东东营神驰炼厂</td>\n",
       "      <td>国六高标95#汽油</td>\n",
       "      <td>无报价</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>2018年9月17日</td>\n",
       "      <td>山东东营神驰炼厂</td>\n",
       "      <td>0#普柴</td>\n",
       "      <td>无报价</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>2018年9月17日</td>\n",
       "      <td>山东东营神驰炼厂</td>\n",
       "      <td>国六0#车柴</td>\n",
       "      <td>无报价</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   releasedate  provider  producttype productprice\n",
       "0   2018年9月17日  山东东营华星石化      国五92#汽油      8850元/吨\n",
       "1   2018年9月17日  山东东营华星石化      国五95#汽油          无报价\n",
       "2   2018年9月17日  山东东营华星石化  国六92#汽油(高效)      8900元/吨\n",
       "3   2018年9月17日  山东东营华星石化       国五0#普柴      7800元/吨\n",
       "4   2018年9月17日  山东东营华星石化       国五0#车柴      7900元/吨\n",
       "5   2018年9月17日  山东东营华星石化       国六0#车柴      7900元/吨\n",
       "6   2018年9月17日  山东东营华星石化           催柴          无报价\n",
       "7   2018年9月17日  山东东营正和集团      国四90#汽油          无报价\n",
       "8   2018年9月17日  山东东营正和集团      国四93#汽油          无报价\n",
       "9   2018年9月17日  山东东营正和集团      国四97#汽油          无报价\n",
       "10  2018年9月17日  山东东营正和集团      国六92#汽油      8850元/吨\n",
       "11  2018年9月17日  山东东营正和集团      国六95#汽油      9000元/吨\n",
       "12  2018年9月17日  山东东营正和集团     国五0#普通柴油      7830元/吨\n",
       "13  2018年9月17日  山东东营正和集团       燃料油13#      7830元/吨\n",
       "14  2018年9月17日  山东东营正和集团       国五0#车柴      7930元/吨\n",
       "15  2018年9月17日  山东东营正和集团       国六0#车柴      7930元/吨\n",
       "16  2018年9月17日  山东东营神驰炼厂    国六89#组分汽油      8700元/吨\n",
       "17  2018年9月17日  山东东营神驰炼厂      国六92#汽油      8800元/吨\n",
       "18  2018年9月17日  山东东营神驰炼厂      国六95#汽油      8900元/吨\n",
       "19  2018年9月17日  山东东营神驰炼厂       普通0#柴油          无报价\n",
       "20  2018年9月17日  山东东营神驰炼厂      国六-0#车柴          无报价\n",
       "21  2018年9月17日  山东东营神驰炼厂    国六高标92#汽油          无报价\n",
       "22  2018年9月17日  山东东营神驰炼厂    国六高标95#汽油          无报价\n",
       "23  2018年9月17日  山东东营神驰炼厂         0#普柴          无报价\n",
       "24  2018年9月17日  山东东营神驰炼厂       国六0#车柴          无报价"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 210 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "dfresult = pd.DataFrame(columns=('releasedate',\n",
    "                                 'provider',\n",
    "                                 'producttype',\n",
    "                                 'productprice'))\n",
    "dfindex = 0\n",
    "for singlegasprovider in soup.find('div', id='ContentBody').find_all('p'):\n",
    "    content = singlegasprovider.text\n",
    "    if (len(content) > 0 and \n",
    "       re.match(r'(.*)今日', content) is not None):\n",
    "        content = content.replace('。','；')\n",
    "        provider = re.match(r'(.*)今日', content).group(1)\n",
    "        pattern = re.compile(r'报价：(.*)')\n",
    "        gasinfo = pattern.search(content)\n",
    "        if gasinfo is not None:\n",
    "            gaslist = gasinfo.group(1).split(r'；')\n",
    "            for gas in gaslist:\n",
    "                gastypeprice = gas.split(r'，')\n",
    "                if ('无报价' in gastypeprice[0] or\n",
    "                   '无货' in gastypeprice[0]):\n",
    "                    pattern = re.compile(r'(.*)(无报价|无货)')\n",
    "                    gastype = pattern.search(gastypeprice[0]).group(1)\n",
    "                    gasprice = r'无报价'\n",
    "                else:\n",
    "                    pattern = re.compile(r'([1-9]\\d*元/吨)')\n",
    "                    if pattern.search(gastypeprice[0]) is None:\n",
    "                        continue\n",
    "                    gasprice = pattern.search(gastypeprice[0]).group(1)\n",
    "                    gastype = gastypeprice[0].replace(gasprice, '')\n",
    "                dfresult.loc[dfindex] = {'releasedate': searchdate.strip(),\n",
    "                                         'provider': provider.strip(),\n",
    "                                         'producttype': gastype.strip(),\n",
    "                                         'productprice': gasprice}\n",
    "                dfindex += 1\n",
    "display(dfresult)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后将数据存储到csv中："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfresult.to_csv('./output/result_{0}.csv'.format(searchdate), encoding='utf_8_sig')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}