18.1 Web Crawler - Selenium入门及使用.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Selenium 入门及使用"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 什么是Selenium"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "selenium 是一套完整的web应用程序测试系统，包含了测试的录制（selenium IDE）,编写及运行（Selenium Remote Control）和测试的并行处理（Selenium Grid）。Selenium的核心Selenium Core基于JsUnit，完全由JavaScript编写，因此可以用于任何支持JavaScript的浏览器上。\n",
    "\n",
    "selenium可以模拟真实浏览器，自动化测试工具，支持多种浏览器，爬虫中主要用来解决JavaScript渲染问题。<br>\n",
    "作为爬虫的应用：可以模拟用户的浏览器操作，并定位到含有数据的页面，此时将页面的数据爬取下来，或者干脆将网页html爬取下来，再提供xpath或者bs4提取内容。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 支持多种操作系统"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如Windows、Linux、IOS、Android等"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 安装浏览器驱动"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Selenium支持大部分常见的浏览器，但是需要注意的是，需要选择适合的浏览器驱动。否则可能会调用失败。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看Chrome的版本：<br>\n",
    "<img src='./image/2018-09-23 15_27_17-Start.png' /><br>\n",
    "选择合适的Chrome 驱动（只有32位版本）：<br>\n",
    "<img src='./image/2018-09-23 15_27_36-Start.png' /><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下载完毕后，<font color='red'>将可执行文件拷贝到Python的安装根目录即可。</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/2018-09-23 15_23_25-python36.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Firefox驱动](https://github.com/mozilla/geckodriver/releases)<br>\n",
    "[Chrome驱动](https://sites.google.com/a/chromium.org/chromedriver/home)<br>\n",
    "[IE驱动](http://selenium-release.storage.googleapis.com/index.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 官方说明文档\n",
    "[官方说明文档](https://selenium-python-zh.readthedocs.io/en/latest/index.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selenium安装\n",
    "执行```pip install selenium```即可"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 初次尝试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium import webdriver"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "firefoxdriver = webdriver.Firefox()\n",
    "chromedriver = webdriver.Chrome()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Firefox弹出的空白页面：<br>\n",
    "<img src='./image/2018-09-23 15_36_43-Start.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Chrome弹出的空白页面:<br>\n",
    "<img src='./image/2018-09-23 15_37_16-Start.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selenium快速入门"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. 8种定位方式"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 1. id\n",
    "* 2. name\n",
    "* 3. class name\n",
    "* 4. tag name\n",
    "* 5. link text\n",
    "* 6. partial link text\n",
    "* 7. xpath\n",
    "* 8. css selector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. 定位元素的8种方式"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/2018-09-23 15_45_15-Python.png' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd = webdriver.Chrome()\n",
    "cd.get('https://www.baidu.com')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定位百度的搜索文本框"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/2018-09-23 15_48_51-百度一下，你就知道.png' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<selenium.webdriver.remote.webelement.WebElement (session=\"cc65ed7e6d8fec458e72b574f3e3052f\", element=\"0.7041380235683354-1\")>\n"
     ]
    }
   ],
   "source": [
    "# 通过id定位\n",
    "txtbyid = cd.find_element_by_id('kw')\n",
    "print(txtbyid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<selenium.webdriver.remote.webelement.WebElement (session=\"cc65ed7e6d8fec458e72b574f3e3052f\", element=\"0.7041380235683354-1\")>\n"
     ]
    }
   ],
   "source": [
    "# 通过name定位\n",
    "txtbyname = cd.find_element_by_name('wd')\n",
    "print(txtbyname)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<selenium.webdriver.remote.webelement.WebElement (session=\"cc65ed7e6d8fec458e72b574f3e3052f\", element=\"0.7041380235683354-1\")>\n"
     ]
    }
   ],
   "source": [
    "# 通过class name定位\n",
    "txtbyclassname = cd.find_element_by_class_name('s_ipt')\n",
    "print(txtbyclassname)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<selenium.webdriver.remote.webelement.WebElement (session=\"cc65ed7e6d8fec458e72b574f3e3052f\", element=\"0.7041380235683354-2\")>\n"
     ]
    }
   ],
   "source": [
    "# 通过tag name定位\n",
    "txtbytagname = cd.find_element_by_tag_name('input')\n",
    "print(txtbytagname)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过xpath定位，可以有很多种写法，这里列出几个常用写法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<selenium.webdriver.remote.webelement.WebElement (session=\"cc65ed7e6d8fec458e72b574f3e3052f\", element=\"0.7041380235683354-1\")>\n",
      "<selenium.webdriver.remote.webelement.WebElement (session=\"cc65ed7e6d8fec458e72b574f3e3052f\", element=\"0.7041380235683354-1\")>\n",
      "<selenium.webdriver.remote.webelement.WebElement (session=\"cc65ed7e6d8fec458e72b574f3e3052f\", element=\"0.7041380235683354-1\")>\n",
      "<selenium.webdriver.remote.webelement.WebElement (session=\"cc65ed7e6d8fec458e72b574f3e3052f\", element=\"0.7041380235683354-1\")>\n",
      "<selenium.webdriver.remote.webelement.WebElement (session=\"cc65ed7e6d8fec458e72b574f3e3052f\", element=\"0.7041380235683354-1\")>\n"
     ]
    }
   ],
   "source": [
    "print(cd.find_element_by_xpath(\"//*[@id='kw']\"))\n",
    "print(cd.find_element_by_xpath(\"//*[@name='wd']\"))\n",
    "print(cd.find_element_by_xpath(\"//input[@class='s_ipt']\"))\n",
    "# print(cd.find_element_by_xpath(\"/html/body/form/span/input\"))\n",
    "# print(cd.find_element_by_xpath(\"//span[@class='soutu-btn']/input\"))\n",
    "print(cd.find_element_by_xpath(\"//form[@id='form']/span/input\"))\n",
    "print(cd.find_element_by_xpath(\"//input[@id='kw' and @name='wd']\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过css定位，css定位有N种写法，这里列几个常用写法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<selenium.webdriver.remote.webelement.WebElement (session=\"cc65ed7e6d8fec458e72b574f3e3052f\", element=\"0.7041380235683354-1\")>\n",
      "<selenium.webdriver.remote.webelement.WebElement (session=\"cc65ed7e6d8fec458e72b574f3e3052f\", element=\"0.7041380235683354-1\")>\n",
      "<selenium.webdriver.remote.webelement.WebElement (session=\"cc65ed7e6d8fec458e72b574f3e3052f\", element=\"0.7041380235683354-1\")>\n",
      "<selenium.webdriver.remote.webelement.WebElement (session=\"cc65ed7e6d8fec458e72b574f3e3052f\", element=\"0.7041380235683354-1\")>\n"
     ]
    }
   ],
   "source": [
    "print(cd.find_element_by_css_selector(\"#kw\"))\n",
    "print(cd.find_element_by_css_selector(\"[name=wd]\"))\n",
    "print(cd.find_element_by_css_selector(\".s_ipt\"))\n",
    "# print(cd.find_element_by_css_selector(\"html > body > form > span > input\"))\n",
    "# print(cd.find_element_by_css_selector(\"span.soutu-btn> input#kw\"))\n",
    "print(cd.find_element_by_css_selector(\"form#form > span > input\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "根据文本链接定位，如百度右上角有一组快捷方式链接："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/2018-09-23 16_01_34-百度一下，你就知道.png' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<selenium.webdriver.remote.webelement.WebElement (session=\"cc65ed7e6d8fec458e72b574f3e3052f\", element=\"0.7041380235683354-3\")>\n",
      "<selenium.webdriver.remote.webelement.WebElement (session=\"cc65ed7e6d8fec458e72b574f3e3052f\", element=\"0.7041380235683354-4\")>\n"
     ]
    }
   ],
   "source": [
    "print(cd.find_element_by_link_text('新闻'))\n",
    "print(cd.find_element_by_link_text('hao123'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "也可以通过partial link text定位"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<selenium.webdriver.remote.webelement.WebElement (session=\"cc65ed7e6d8fec458e72b574f3e3052f\", element=\"0.7041380235683354-3\")>\n",
      "<selenium.webdriver.remote.webelement.WebElement (session=\"cc65ed7e6d8fec458e72b574f3e3052f\", element=\"0.7041380235683354-4\")>\n",
      "<selenium.webdriver.remote.webelement.WebElement (session=\"cc65ed7e6d8fec458e72b574f3e3052f\", element=\"0.7041380235683354-4\")>\n"
     ]
    }
   ],
   "source": [
    "print(cd.find_element_by_partial_link_text('新'))\n",
    "print(cd.find_element_by_partial_link_text('hao'))\n",
    "print(cd.find_element_by_partial_link_text('123'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Web Driver模块常用方法的使用"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. 控制浏览器操作的一些方法\n",
    "<img src='./image/2018-09-23 16_07_40-Python.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面是演示打开百度，并点击“新闻”链接的例子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from time import sleep\n",
    "\n",
    "browser = webdriver.Chrome()\n",
    "browser.get('https://www.baidu.com')\n",
    "sleep(3)\n",
    "\n",
    "#刷新浏览器\n",
    "browser.refresh()\n",
    "\n",
    "#设置浏览器大小\n",
    "browser.set_window_size(1400, 800)\n",
    "\n",
    "#设置链接内容\n",
    "element = browser.find_element_by_link_text('新闻')\n",
    "element.click()\n",
    "# browser.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面是演示打开百度，并搜索的例子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from time import sleep\n",
    "\n",
    "browser = webdriver.Chrome()\n",
    "browser.get('https://www.baidu.com')\n",
    "sleep(1)\n",
    "\n",
    "browser.find_element_by_id('kw').send_keys('上证指数')\n",
    "browser.find_element_by_id('su').click()\n",
    "# browser.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们也可以按照如下的写法，直接将搜索内容，与回车指令一句话完成。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "上证指数: 2781.14\n",
      "深成指: 一塌糊涂, 创业板:基本没治\n",
      "Wall time: 5.21 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.keys import Keys\n",
    "from time import sleep\n",
    "\n",
    "browser = webdriver.Chrome()\n",
    "browser.get('https://www.baidu.com')\n",
    "browser.find_element_by_id('kw').send_keys('上证指数', Keys.ENTER)\n",
    "sleep(2)\n",
    "print('上证指数: {0}'.format(\n",
    "    browser.find_element_by_class_name('op-stockdynamic-moretab-cur-num').text))\n",
    "# print('深成指: {0}, 创业板:{1}'.format('一塌糊涂', '基本没治'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. 鼠标事件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在WebDriver中，将这些关于鼠标操作的方法封装在ActionChains类提供"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/2018-09-23 16_29_18-Python.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们演示鼠标悬停的功能"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/2018-09-23 16_31_30-百度一下，你就知道.png' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "#1.引入 ActionChains 类\n",
    "from selenium.webdriver.common.action_chains import ActionChains\n",
    "\n",
    "driver= webdriver.Chrome()\n",
    "\n",
    "driver.get(\"https://www.baidu.cn\")\n",
    "\n",
    "#2.定位到要悬停的元素\n",
    "element= driver.find_element_by_link_text(\"设置\")\n",
    "\n",
    "#3.对定位到的元素执行鼠标悬停操作\n",
    "ActionChains(driver).move_to_element(element).perform()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. 键盘事件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Selenium中的Key模块为我们提供了模拟键盘按键的方法，那就是send_keys()方法。它不仅可以模拟键盘输入，也可以模拟键盘的操作。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "常用的键盘操作如下, 刚刚演示的百度搜索功能，就有一个模拟回车的操作："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/2018-09-23 16_37_30-Python.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. 获取断言信息\n",
    "不管是在做功能测试还是自动化测试，最后一步需要拿实际结果与预期进行比较。这个比较的称之为断言。通过我们获取title 、URL和text等信息进行断言。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/2018-09-23 16_39_39-Python.png' /><br>\n",
    "实例演示，以下代码也演示了如何获取页面源代码，以及页面元素纯文本的方法："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before search================\n",
      "百度一下，你就知道\n",
      "https://www.baidu.com/\n",
      "<!DOCTYPE html><!--STATUS OK--><html xmlns=\"http://www.w3.org/1999/xhtml\"><head><script type=\"text/javascript\" charset=\"gb2312\" src=\"//www.baidu.com/cache/aladdin/ui/scrollbarv/scrollbarv.js?v=20170208\" data-for=\"A.ui\"></script><script type=\"text/javascript\" charset=\"gb2312\" src=\"//www.baidu.com/cache/aladdin/ui/lunar7/lunar7.js?v=20170208\" data-for=\"A.ui\"></script><script type=\"text/javascript\" charset=\"gb2312\" src=\"//www.baidu.com/cache/aladdin/ui/dropdown21/dropdown21.js?v=20170208\" data-for=\"A.ui\"></script><script charset=\"utf-8\" async=\"\" src=\"https://ss0.bdstatic.com/-0U0bnSm1A5BphGlnYG/tam-ogel/5d4e9b24-dcc5-483a-b6da-be1e9e621891.js\"></script>\n",
      "    \n",
      "    <meta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\" /><style data-for=\"result\" id=\"css_result\" type=\"text/css\">body{color:#333;background:#fff;padding:6px 0 0;margin:0;position:relative;min-width:900px}body,th,td,.p1,.p2{font-family:arial}p,form,ol,ul,li,dl,dt,dd,h3{margin:0;padding:0;list-style:none}input{padding-to\n",
      "纯文本 by BS4: 国庆节_百度百科\n",
      "After search================\n",
      "国庆节_百度搜索\n",
      "https://www.baidu.com/s?ie=utf-8&f=8&rsv_bp=0&rsv_idx=1&tn=baidu&wd=%E5%9B%BD%E5%BA%86%E8%8A%82&rsv_pq=eddc7ae700033d6f&rsv_t=9b84kTUoS%2By3fep4bwdQCUHSOKSw46h1kARNsNSE3i4jiM41QBYjlTQCkNM&rqlang=cn&rsv_enter=0&rsv_sug3=3&rsv_sug1=1&rsv_sug7=100&inputT=125&rsv_sug4=125\n",
      "搜索工具\n",
      "百度为您找到相关结果约17,200,000个\n"
     ]
    }
   ],
   "source": [
    "from selenium import webdriver\n",
    "from time import sleep\n",
    "from bs4 import  BeautifulSoup\n",
    "\n",
    "driver = webdriver.Chrome()\n",
    "driver.get(\"https://www.baidu.com\")\n",
    "\n",
    "print('Before search================')\n",
    "\n",
    "# 打印当前页面title\n",
    "title = driver.title\n",
    "print(title)\n",
    "\n",
    "# 打印当前页面URL\n",
    "now_url = driver.current_url\n",
    "print(now_url)\n",
    "\n",
    "driver.find_element_by_id(\"kw\").send_keys(\"国庆节\")\n",
    "driver.find_element_by_id(\"su\").click()\n",
    "sleep(1)\n",
    "## 打印源码前1000个字符\n",
    "sourcecode = driver.page_source\n",
    "print(sourcecode[:1000])\n",
    "\n",
    "## 打印源码纯文本\n",
    "soup = BeautifulSoup(sourcecode, features='lxml')\n",
    "print('纯文本 by BS4: {0}'.format(soup.find('div').find('h3').find('a').text))\n",
    "\n",
    "print('After search================')\n",
    "\n",
    "# 再次打印当前页面title\n",
    "title = driver.title\n",
    "print(title)\n",
    "\n",
    "# 打印当前页面URL\n",
    "now_url = driver.current_url\n",
    "print(now_url)\n",
    "\n",
    "# 获取结果数目\n",
    "nums = driver.find_element_by_class_name('nums').text\n",
    "print(nums)\n",
    "\n",
    "sleep(2)\n",
    "#关闭所有窗口\n",
    "driver.quit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. 设置元素等待"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这个大家自己看一下参考资料即可：[等待页面完成](https://selenium-python-zh.readthedocs.io/en/latest/waits.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6. 定位一组元素"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定位一组元素的方法与定位单个元素的方法类似，唯一的区别是在单词element后面多了一个s表示复数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'list'>\n",
      "StubHub丨巴塞罗那门票_海外赛事门票抢购平台\n",
      "真实url: /baidu.php?sc.Ks0000amqZra9B-65SsInqCasrBTryDEhq-srpGFFizU2brFt2BdTeUJPBBpLyJJSIHTFe-_JDy91beV59sRvTpXKHKi1qOJdJJlwH6nqLxca6LzbberCKqCaVgS286xmJvQF6ORYo7o0-ZIJKdwav0lSOcSVtjQQSKxW0vIni974EDpHs.7D_NR2Ar5Od663rj6tClAaSoKxHK9tqvZdtT5ovmx5I9qVXZx43x5ksSEzseOZj4etr1F9qEZ5o91L_1fGC8Bu13x5x9LSrZven5M33xU8sSLuKa1kvIt7jHzk8sHfGmEukmr8a9G4I2UM3PQZ4E1xjdsRP5Qa1Gk_Edwnxg3Xgduh9HwID1dsRP5QfHPGmeXa1FbLe8or8BkgE3Oo_O3_-xSEGRojPak8vI-Xkf.U1Yk0ZDq_nLcStQXknxg3txzVVn0mywkXHvs1TWesVeDkqxWkO2vYsK9uZ7Y5Hc0TA-W5H00Ijvs1TWesVeDkt57koD0pyYqnWcd0ATqmhNsT100Iybqmh7GuZR0TA-b5Hf0mv-b5Hb10AdY5HDsnHIxnH0krNtznjmz0AVG5H00TMfqPjmY0ANGujYkPjndg1cvrH04g1cknH01g1cznHR30AFG5HcsP7tkPHR0UynqP1TYPWmvrHDYg1Dsnj7xnWfdnWm1PWT3n6KkTA-b5H00TyPGujYs0ZFMIA7M5H00mycqn7ts0ANzu1Ys0ZKs5H00UMus5H08nj0snj0snj00Ugws5H00uAwETjYs0ZFJ5H00uANv5gKW0AuY5H00TA6qn0KET1Ys0AFL5HDs0A4Y5H00TLCq0ZwdT1YkrjnzrH61P1bYnjTLPjcLn16dPfKzug7Y5HDdn1T3P1f3nHnLPjb0Tv-b5yNWPHndPymLnj0snWbzPjc0mLPV5HnYwWRYPYf3wbRLfH6vPHT0mynqnfKsUWYs0Z7VIjYs0Z7VT1Ys0ZGY5HD0UyPxuMFEUHYsg1Kxn0Kbmy4dmhNxTAk9Uh-bT1Ysg1Kxn0Ksmgwxuhk9u1Ys0AwWpyfqn6K-IA-b5iYk0A71TAPW5H00IgKGUhPW5H00Tydh5HDv0AuWIgfqn0KhXh6qn0Khmgfqn0KlTAkdT1Ys0A7buhk9u1Yk0Akhm1Ys0APzm1YYnHR4r0&ck=0.0.0.0.0.0.0.0&shh=www.baidu.com&sht=baidu\n",
      "\n",
      "巴塞罗那足球俱乐部_百度百科\n",
      "真实url: http://baike.baidu.com/link?url=efROeXLMtofvdzwxIBu_eH-SQrBAEBeXgZ4dtd5PXozBv-zNfSLw7uLCamtxLvvJ48S88YPVtVCsIbh77NCoxc23A0ol9Kf3XtzyyKV7mzy1mr4snMnh3Juna0zhwysluVMV5H9VV3HuWDqxBfm6WlFsJO6U5Lu6tioAjSaFdg6hhgBpjLTkfpA8zd_jSbxj\n",
      "\n",
      "巴塞罗那中文官网|FC Barcelona Web Oficial\n",
      "真实url: http://fcbarcelona.qq.com/\n",
      "\n",
      "巴塞罗那足球队_百度百科\n",
      "真实url: https://baike.baidu.com/item/%E5%B7%B4%E5%A1%9E%E7%BD%97%E9%82%A3%E8%B6%B3%E7%90%83%E9%98%9F/9994806\n",
      "\n",
      "足球俱乐部:巴塞罗那足球俱乐部,足球俱乐部的荣耀王者\n",
      "真实url: https://baijiahao.baidu.com/s?id=1609775714742126526&wfr=spider&for=pc\n",
      "\n",
      "Official FC Barcelona Web Site - Barça | FCBarcelona.com -...\n",
      "真实url: https://www.fcbarcelona.com/\n",
      "\n",
      "宇宙队(巴塞罗那足球俱乐部别称)_百度百科\n",
      "真实url: https://baike.baidu.com/item/%E5%AE%87%E5%AE%99%E9%98%9F/6237512\n",
      "\n",
      "巴萨中文官网_俱乐部_历史\n",
      "真实url: http://fcbarcelona.qq.com/isocce/barcelonafc/club1899/historys/index.htm\n",
      "\n",
      "巴塞罗那中文官网_一队_大名单\n",
      "真实url: http://fcbarcelona.qq.com/isocce/barcelonafc/football1899/firstteam/index.htm\n",
      "\n",
      "巴塞罗那足球队_百度图片\n",
      "真实url: http://image.baidu.com/search/index?tn=baiduimage&ct=201326592&lm=-1&cl=2&ie=gb18030&word=%B0%CD%C8%FB%C2%DE%C4%C7%D7%E3%C7%F2%B6%D3&fr=ala&ala=1&alatpl=adress&pos=0&hs=2&xthttps=000000\n",
      "\n",
      "买机票上携程网_携程在手_说走就走\n",
      "真实url: /baidu.php?sc.Ks0000amqZra9B-65ONVYe9Ki8Tvq99tGussDlJdplvT1jPAx0eNrlQnggJmx9sryvPCb2MJ-sTbjRj2BLc3djteHtdk9ST5ZyBQLIt042Z4yt5S7zpyeSlesU993JRYebZQBxvACVGByODWHE5pR4sggjRLCuUW6k_vRq3P2qj5YkTcFf.7R_iRFhmXdnL-Guhx7ZGwKPIL9knI5u_eZqotMHuugdYqEKBmoOReIxWSL8IZdks1mh2lePMugu3qrHFdC3v2J7jZ93SLjbS8ZFqII7SJJ9Z4fYFM3YmTMdstxJ1wCEuB9HBstx-vUPOW9zInXQjkSgWELePOW9zIra9G4I2UM3PrZWtovTyjjblT7jHzs_lTUQqRyZWt_DdAeCs4rMKfYt_U_DY2T5Z4g_3_A5ZjdsHRXy61PJHfxhmoRojPak8___v20.U1Yk0ZDq_nLcStQXknxg3txzVVn0mywkXHvs1TWesVeDkqxWkO2vYsK9uZ7Y5HD0TA-W5HD0Ijvs1TWesVeDksKGUHYznWT0u1dBugK1n0KdpHdBmy-bIfKspyfqP0KWpyfqrHn0UgfqnH0kPdtknjc4g1DsnHPxnH0krNt1PW0k0AVG5H00TMfqPjmY0ANGujYkPjndg1cvrH04g1cknH01g1cznHR30AFG5HcsP0KVm1YLrjT3rHTLnjFxP16LrjbLP103g1cznjDkPjDzn-tknj0kg100TgKGujYs0Z7Wpyfqn0KzuLw9u1Ys0A7B5HKxn0K-ThTqn0KsTjYs0A4vTjYsQW0snj0snj0s0AdYTjYs0AwbUL0qn0KzpWYs0Aw-IWdsmsKhIjYs0ZKC5H00ULnqn0KBI1Ykn0K8IjYs0ZPl5fKYIgnqPWDznHDLn163rj03nj64n1bLr0Kzug7Y5HDdn1T3P1f3nHnLPjb0Tv-b5yNWPHndPymLnj0snWbzPjc0mLPV5HnYwWRYPYf3wbRLfH6vPHT0mynqnfKYIgfqnfKsUWYs0Z7VIjYs0Z7VT1Ys0ZGY5H00UyPxuMFEUHYsg1Kxn7tsg100uA78IyF-gLK_my4GuZnqn7tsg1Kxn7ts0ZK9I7qhUA7M5H00uAPGujYs0ANYpyfqQHD0mgPsmvnqn0KdTA-8mvnqn0KkUymqnHm0uhPdIjYs0AulpjYs0Au9IjYs0ZGsUZN15H00mywhUA7M5H60UAuW5H00mLFW5HfzPHnd&ck=0.0.0.0.0.0.0.0&shh=www.baidu.com&sht=baidu\n",
      "\n",
      "\n",
      "真实url: /baidu.php?sc.Ks0000amqZra9B-65ONVYe9Ki8Tvq99tGussDlJdplvT1jPAx0eNrlQnggJmx9sryvPCb2MJ-sTbjRj2BLc3djteHtdk9ST5ZyBQLIt042Z4yt5S7zpyeSlesU993JRYebZQBxvACVGByODWHE5pR4sggjRLCuUW6k_vRq3P2qj5YkTcFf.7R_iRFhmXdnL-Guhx7ZGwKPIL9knI5u_eZqotMHuugdYqEKBmoOReIxWSL8IZdks1mh2lePMugu3qrHFdC3v2J7jZ93SLjbS8ZFqII7SJJ9Z4fYFM3YmTMdstxJ1wCEuB9HBstx-vUPOW9zInXQjkSgWELePOW9zIra9G4I2UM3PrZWtovTyjjblT7jHzs_lTUQqRyZWt_DdAeCs4rMKfYt_U_DY2T5Z4g_3_A5ZjdsHRXy61PJHfxhmoRojPak8___v20.U1Yk0ZDq_nLcStQXknxg3txzVVn0mywkXHvs1TWesVeDkqxWkO2vYsK9uZ7Y5HD0TA-W5HD0Ijvs1TWesVeDksKGUHYznWT0u1dBugK1n0KdpHdBmy-bIfKspyfqP0KWpyfqrHn0UgfqnH0kPdtknjc4g1DsnHPxnH0krNt1PW0k0AVG5H00TMfqPjmY0ANGujYkPjndg1cvrH04g1cknH01g1cznHR30AFG5HcsP0KVm1YLrjT3rHTLnjFxP16LrjbLP103g1cznjDkPjDzn-tknj0kg100TgKGujYs0Z7Wpyfqn0KzuLw9u1Ys0A7B5HKxn0K-ThTqn0KsTjYs0A4vTjYsQW0snj0snj0s0AdYTjYs0AwbUL0qn0KzpWYs0Aw-IWdsmsKhIjYs0ZKC5H00ULnqn0KBI1Ykn0K8IjYs0ZPl5fKYIgnqPWDznHDLn163rj03nj64n1bLr0Kzug7Y5HDdn1T3P1f3nHnLPjb0Tv-b5yNWPHndPymLnj0snWbzPjc0mLPV5HnYwWRYPYf3wbRLfH6vPHT0mynqnfKYIgfqnfKsUWYs0Z7VIjYs0Z7VT1Ys0ZGY5H00UyPxuMFEUHYsg1Kxn7tsg100uA78IyF-gLK_my4GuZnqn7tsg1Kxn7ts0ZK9I7qhUA7M5H00uAPGujYs0ANYpyfqQHD0mgPsmvnqn0KdTA-8mvnqn0KkUymqnHm0uhPdIjYs0AulpjYs0Au9IjYs0ZGsUZN15H00mywhUA7M5H60UAuW5H00mLFW5HfzPHnd&ck=0.0.0.0.0.0.0.0&shh=www.baidu.com&sht=baidu\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from selenium import webdriver\n",
    "from time import sleep\n",
    "import requests\n",
    "\n",
    "driver = webdriver.Chrome()\n",
    "driver.get(\"https://www.baidu.com\")\n",
    "\n",
    "driver.find_element_by_id(\"kw\").send_keys(\"巴塞罗那足球队\")\n",
    "driver.find_element_by_id(\"su\").click()\n",
    "sleep(1)\n",
    "\n",
    "#1.定位一组元素\n",
    "elements = driver.find_elements_by_xpath('//div/h3/a')\n",
    "print(type(elements))\n",
    "\n",
    "#2.循环遍历出每一条搜索结果的标题\n",
    "for t in elements:\n",
    "    print(t.text)\n",
    "    baiduurl = t.get_attribute('href')\n",
    "    realurl = ''\n",
    "    tmpPage = requests.get(baiduurl, allow_redirects=False)\n",
    "    if tmpPage.status_code == 200:\n",
    "        urlMatch = re.search(r'URL=\\'(.*?)\\'', tmpPage.text.encode('utf-8'), re.S)\n",
    "        realurl = urlMatch.group(1)\n",
    "    elif tmpPage.status_code == 302:\n",
    "        realurl = tmpPage.headers.get('location')\n",
    "    else:\n",
    "        print('No URL found!!')\n",
    "    print('真实url: {0}'.format(realurl))\n",
    "    print()\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 7. 多表单切换"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在Web应用中经常会遇到frame/iframe表单嵌套页面的应用，WebDriver只能在一个页面上对元素识别与定位，对于frame/iframe表单内嵌页面上的元素无法直接定位。这时就需要通过switch_to.frame()方法将当前定位的主体切换为frame/iframe表单的内嵌页面中。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 方法：switch_to.frame()，说明：将当期定位的主体切换为frame/ iframe表单的内嵌页面中\n",
    "* 方法：switch_to.default_content()，说明：跳回最外层的页面"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以126邮箱为例，登录框是通过iframe嵌入的，这就需要使用switch_to的功能："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/2018-09-23 17_35_14-126.png' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from time import sleep\n",
    "\n",
    "driver = webdriver.Chrome()\n",
    "driver.get(\"http://www.126.com\")\n",
    "sleep(5)\n",
    "driver.switch_to.frame('x-URS-iframe')\n",
    "driver.find_element_by_name(\"email\").clear()\n",
    "driver.find_element_by_name(\"email\").send_keys(\"username\")\n",
    "driver.find_element_by_name(\"password\").clear()\n",
    "driver.find_element_by_name(\"password\").send_keys(\"password\")\n",
    "driver.find_element_by_id(\"dologin\").click()\n",
    "sleep(5)\n",
    "driver.switch_to.default_content()\n",
    "\n",
    "driver.quit() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果iframe没有可用的id和name属性，则可以通过xpath进行定位："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "#先通过xpth定位到iframe\n",
    "xf = driver.find_element_by_xpath('//*[@id=\"x-URS-iframe\"]')\n",
    "\n",
    "#再将定位对象传给switch_to.frame()方法\n",
    "driver.switch_to.frame(xf)\n",
    "……\n",
    "driver.switch_to.parent_frame()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 8. 多窗口切换\n",
    "在页面操作过程中有时候点击某个链接会弹出新的窗口，这时就需要主机切换到新打开的窗口上进行操作。WebDriver提供了switch_to.window()方法，可以实现在不同的窗口之间切换。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/2018-09-23 17_41_42-Python.png' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "跳转到注册窗口\n"
     ]
    }
   ],
   "source": [
    "from selenium import webdriver\n",
    "import time\n",
    "driver = webdriver.Chrome()\n",
    "driver.implicitly_wait(10)\n",
    "driver.get(\"http://www.baidu.com\")\n",
    "\n",
    "#1.获得百度搜索窗口句柄\n",
    "search_windows = driver.current_window_handle\n",
    "\n",
    "driver.find_element_by_link_text('登录').click()\n",
    "driver.find_element_by_link_text(\"立即注册\").click()\n",
    "\n",
    "#1.获得当前所有打开的窗口的句柄\n",
    "all_handles = driver.window_handles\n",
    "\n",
    "#3.进入注册窗口\n",
    "for handle in all_handles:\n",
    "    if handle != search_windows:\n",
    "        driver.switch_to.window(handle)\n",
    "        print('跳转到注册窗口')\n",
    "        driver.find_element_by_name(\"userName\").send_keys('register1234567')\n",
    "        driver.find_element_by_name(\"phone\").send_keys('18823456789')\n",
    "#         driver.find_element_by_name('password').send_keys('123456789')\n",
    "        time.sleep(5)\n",
    "    \n",
    "driver.quit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面的代码是演示如何进行百度搜索，并逐条打开搜索结果页面，并最终切回搜索页面的示例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'list'>\n",
      "StubHub丨巴塞罗那门票_海外赛事门票抢购平台\n",
      "真实url: https://www.baidu.com/baidu.php?sc.Ks0000amqZra9B-65dCnQBBsqwJNYgcsOG6c9bV_8yxi2hggCzTBV6nKT_6DTtCxPgajqKpxu4-0GFkae3eBvc7pdzWrWFjkaJ-DcV8W9CKt2XZEYFWRNMI9adxVBHqMfk-eXp5dYrV8Yhq6hDKtclm3Foj7yxOIOv0n5sUpdLJfPapUxf.7D_NR2Ar5Od663rj6tClAaSoKxHK9tqvZdtT5ovmx5I9qVXZx43x5ksSEzseOZj4etr1F9qEZ5o91L_1fGC8Bu13x5x9LSrZven5M33xU8sSLuKa1kvIt7jHzk8sHfGmEukmr8a9G4I2UM3PQZ4E1xjdsRP5Qa1Gk_Edwnxg3Xgduh9HwID1dsRP5QfHPGmeXa1FbLe8or8BkgE3Oo_O3_-xSEGRojPak8vI-Xkf.U1Yk0ZDq_nLcStQXknxg3txzVVn0mywkXHvs1TWesVeDkqxWkO2vYsK9uZ7Y5Hc0TA-W5H00Ijvs1TWesVeDkt57koD0pyYqnWcd0ATqTZFB0ZNG5yF9pywd0ZKGujYY0APGujY4nsKVIjYknjDLg1DsnH-xnW0vn-t1PW0k0AVG5H00TMfqnHf0uy-b5HDYn1uxnWm4nj9xnWDknWwxnW04nWb0mhbqnW0Yg1DdPfKVm1YLP1fvPWm4nHwxnH0snNtzPjRzPWnvP16zg100TgKGujYs0Z7Wpyfqn0KzuLw9u1Ys0A7B5HKxn0K-ThTqn0KsTjYs0A4vTjYsQW0snj0snj0s0AdYTjYs0AwbUL0qn0KzpWYs0Aw-IWdsmsKhIjYs0ZKC5H00ULnqn0KBI1Ykn0K8IjYs0ZPl5fKYIgnqnH61nWb3n1T4Pj0LP1fzP1n3PHR0ThNkIjYkPHnLrjTdn1TsPjnL0ZPGujdbmHmdn16YPH0snjndrAc30AP1UHY1P101nYDkwRF7wHTYPHKa0A7W5HD0TA3qn0KkUgfqn0KkUgnqn0KlIjYk0AdWgvuzUvYqn7tsg1Kxn0Kbmy4dmhNxTAk9Uh-bT1Ysg1Kxn7ts0ZK9I7qhUA7M5H00uAPGujYz0ANYpyfqQHD0mgPsmvnqn0KdTA-8mvnqn0KkUymqnHm0uhPdIjYs0AulpjYs0Au9IjYs0ZGsUZN15H00mywhUA7M5HD0UAuW5H00mLFW5HRdnjc&ck=0.0.0.0.0.0.0.0&shh=www.baidu.com&sht=baidu\n",
      "\n",
      "巴塞罗那足球俱乐部_百度百科\n",
      "真实url: http://baike.baidu.com/link?url=hZSADLVFjklzUceG3v61o6ry9br9qAsstT19tN09-Hwj2dgNZumJTkYYChVySDDcDLwtZJpKnEULCNCDBRhAaMJNl_UqKw8rNkuHSJYiNmEDTfUJpYC8mYQuelvRp_Q2a_CAdNKJcBl27Ao7JI5_RYRmH4VN0AHVMejZuOo_IVIWU_fkuHX-5IHiasX1-QDz\n",
      "\n",
      "巴塞罗那中文官网|FC Barcelona Web Oficial\n",
      "真实url: http://fcbarcelona.qq.com/\n",
      "\n",
      "巴塞罗那足球队_百度百科\n",
      "真实url: https://baike.baidu.com/item/%E5%B7%B4%E5%A1%9E%E7%BD%97%E9%82%A3%E8%B6%B3%E7%90%83%E9%98%9F/9994806\n",
      "\n",
      "足球俱乐部:巴塞罗那足球俱乐部,足球俱乐部的荣耀王者\n",
      "真实url: https://baijiahao.baidu.com/s?id=1609775714742126526&wfr=spider&for=pc\n",
      "\n",
      "巴萨中文官网_俱乐部_历史\n",
      "真实url: http://fcbarcelona.qq.com/isocce/barcelonafc/club1899/historys/index.htm\n",
      "\n",
      "Official FC Barcelona Web Site - Barça | FCBarcelona.com -...\n",
      "真实url: https://www.fcbarcelona.com/\n",
      "\n",
      "宇宙队(巴塞罗那足球俱乐部别称)_百度百科\n",
      "真实url: https://baike.baidu.com/item/%E5%AE%87%E5%AE%99%E9%98%9F/6237512\n",
      "\n",
      "巴塞罗那中文官网_一队_大名单\n",
      "真实url: http://fcbarcelona.qq.com/isocce/barcelonafc/football1899/firstteam/index.htm\n",
      "\n",
      "巴塞罗那足球队_百度图片\n",
      "真实url: http://image.baidu.com/search/index?tn=baiduimage&ct=201326592&lm=-1&cl=2&ie=gb18030&word=%B0%CD%C8%FB%C2%DE%C4%C7%D7%E3%C7%F2%B6%D3&fr=ala&ala=1&alatpl=adress&pos=0&hs=2&xthttps=000000\n",
      "\n",
      "买机票上携程网_携程在手_说走就走\n",
      "真实url: https://www.baidu.com/baidu.php?sc.Ks0000amqZra9B-65YB433FcdLDXDkjMXnAmVap2NPJpm3GgGXeMhCSnyI8oAh5WoQKO3dejRh6P9-dEnApDsXbcsLBR1iHuHsR4REfuq_DneICh1GLEZrqFJSO4J0XOjp53_BtTzfeE-aWvnB3w0vCGDIbQhQl6Bpn6Fxidh95aAQjjy6.7R_iRFhmXdnL-Guhx7ZGwKPIL9knI5u_eZqotMHuugdYqEKBmoOReIxWSL8IZdks1mh2lePMugu3qrHFdC3v2J7jZ93SLjbS8ZFqII7SJJ9Z4fYFM3YmTMdstxJ1wCEuB9HBstx-vUPOW9zInXQjkSgWELePOW9zIra9G4I2UM3PrZWtovTyjjblT7jHzs_lTUQqRyZWt_DdAeCs4rMKfYt_U_DY2T5Z4g_3_A5ZjdsHRXy61PJHfxhmoRojPak8___v20.U1Yk0ZDq_nLcStQXknxg3txzVVn0mywkXHvs1TWesVeDkqxWkO2vYsK9uZ7Y5HD0TA-W5HD0Ijvs1TWesVeDksKGUHYznWT0u1dsThc0Iybqmh7GuZR0TA-b5Hf0mv-b5Hb10AdY5HDsnHIxnH0zrNtknjD1g1DsnH-xn1msnfKopHYs0ZFY5HDY0ANGujYkPjnvg1cvrH03g1cknHcYg1csrHc40AFG5HcsP0KVm1YLrjT3rHTLnjFxP16LrjbLP103g1cznjDkPjDzn-tknj0kg100TgKGujYs0Z7Wpyfqn0KzuLw9u1Ys0A7B5HKxn0K-ThTqn0KsTjYs0A4vTjYsQW0snj0snj0s0AdYTjYs0AwbUL0qn0KzpWYs0Aw-IWdsmsKhIjYs0ZKC5H00ULnqn0KBI1Ykn0K8IjYs0ZPl5fKYIgnqPWDznHDLn163rj03nj64n1bLr0Kzug7Y5HDdn1T3P1R1P10Yn1T0Tv-b5yw9PWR1rjfdnj0sn1R3mW60mLPV5HnLnjn1fH77fbN7P1fdnDc0mynqnfKYIgfqnfKsUWYs0Z7VIjYs0Z7VT1Ys0ZGY5H00UyPxuMFEUHYsg1Kxn7tsg100uA78IyF-gLK_my4GuZnqn7tsg1Kxn7ts0ZK9I7qhUA7M5H00uAPGujYs0ANYpyfqQHD0mgPsmvnqn0KdTA-8mvnqn0KkUymqnHm0uhPdIjYs0AulpjYs0Au9IjYs0ZGsUZN15H00mywhUA7M5H60UAuW5H00mLFW5Hc4PWcs&ck=0.0.0.0.0.0.0.0&shh=www.baidu.com&sht=baidu\n",
      "\n",
      "\n",
      "真实url: https://www.baidu.com/baidu.php?sc.Ks0000amqZra9B-65YB433FcdLDXDkjMXnAmVap2NPJpm3GgGXeMhCSnyI8oAh5WoQKO3dejRh6P9-dEnApDsXbcsLBR1iHuHsR4REfuq_DneICh1GLEZrqFJSO4J0XOjp53_BtTzfeE-aWvnB3w0vCGDIbQhQl6Bpn6Fxidh95aAQjjy6.7R_iRFhmXdnL-Guhx7ZGwKPIL9knI5u_eZqotMHuugdYqEKBmoOReIxWSL8IZdks1mh2lePMugu3qrHFdC3v2J7jZ93SLjbS8ZFqII7SJJ9Z4fYFM3YmTMdstxJ1wCEuB9HBstx-vUPOW9zInXQjkSgWELePOW9zIra9G4I2UM3PrZWtovTyjjblT7jHzs_lTUQqRyZWt_DdAeCs4rMKfYt_U_DY2T5Z4g_3_A5ZjdsHRXy61PJHfxhmoRojPak8___v20.U1Yk0ZDq_nLcStQXknxg3txzVVn0mywkXHvs1TWesVeDkqxWkO2vYsK9uZ7Y5HD0TA-W5HD0Ijvs1TWesVeDksKGUHYznWT0u1dsThc0Iybqmh7GuZR0TA-b5Hf0mv-b5Hb10AdY5HDsnHIxnH0zrNtknjD1g1DsnH-xn1msnfKopHYs0ZFY5HDY0ANGujYkPjnvg1cvrH03g1cknHcYg1csrHc40AFG5HcsP0KVm1YLrjT3rHTLnjFxP16LrjbLP103g1cznjDkPjDzn-tknj0kg100TgKGujYs0Z7Wpyfqn0KzuLw9u1Ys0A7B5HKxn0K-ThTqn0KsTjYs0A4vTjYsQW0snj0snj0s0AdYTjYs0AwbUL0qn0KzpWYs0Aw-IWdsmsKhIjYs0ZKC5H00ULnqn0KBI1Ykn0K8IjYs0ZPl5fKYIgnqPWDznHDLn163rj03nj64n1bLr0Kzug7Y5HDdn1T3P1R1P10Yn1T0Tv-b5yw9PWR1rjfdnj0sn1R3mW60mLPV5HnLnjn1fH77fbN7P1fdnDc0mynqnfKYIgfqnfKsUWYs0Z7VIjYs0Z7VT1Ys0ZGY5H00UyPxuMFEUHYsg1Kxn7tsg100uA78IyF-gLK_my4GuZnqn7tsg1Kxn7ts0ZK9I7qhUA7M5H00uAPGujYs0ANYpyfqQHD0mgPsmvnqn0KdTA-8mvnqn0KkUymqnHm0uhPdIjYs0AulpjYs0Au9IjYs0ZGsUZN15H00mywhUA7M5H60UAuW5H00mLFW5Hc4PWcs&ck=0.0.0.0.0.0.0.0&shh=www.baidu.com&sht=baidu\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from selenium import webdriver\n",
    "from time import sleep\n",
    "import requests\n",
    "\n",
    "driver = webdriver.Chrome()\n",
    "driver.get(\"https://www.baidu.com\")\n",
    "\n",
    "driver.find_element_by_id(\"kw\").send_keys(\"巴塞罗那足球队\")\n",
    "driver.find_element_by_id(\"su\").click()\n",
    "sleep(1)\n",
    "\n",
    "#1.定位一组元素\n",
    "elements = driver.find_elements_by_xpath('//div/h3/a')\n",
    "print(type(elements))\n",
    "search_window = driver.current_window_handle\n",
    "\n",
    "#2.循环遍历出每一条搜索结果的标题\n",
    "for t in elements:\n",
    "    print(t.text)\n",
    "    baiduurl = t.get_attribute('href')\n",
    "    realurl = ''\n",
    "    tmpPage = requests.get(baiduurl, allow_redirects=False)\n",
    "    if tmpPage.status_code == 200:\n",
    "        urlMatch = re.search(r'URL=\\'(.*?)\\'', tmpPage.text.encode('utf-8'), re.S)\n",
    "        realurl = urlMatch.group(1)\n",
    "    elif tmpPage.status_code == 302:\n",
    "        realurl = tmpPage.headers.get('location')\n",
    "    else:\n",
    "        print('No URL found!!')\n",
    "    if realurl.startswith('/'):\n",
    "        realurl = '{0}{1}'.format('https://www.baidu.com', realurl)\n",
    "    print('真实url: {0}'.format(realurl))\n",
    "    # 遍历url并在新标签打开的方法\n",
    "    js = \"window.open('{0}')\".format(realurl) \n",
    "    driver.execute_script(js)\n",
    "    sleep(2)\n",
    "    driver.switch_to.window(search_window)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 9. 警告框处理\n",
    "在WebDriver中处理JavaScript所生成的alert、confirm以及prompt十分简单，具体做法是使用 switch_to.alert 方法定位到 alert/confirm/prompt，然后使用text/accept/dismiss/ send_keys等方法进行操作。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/2018-09-23 18_07_34-Python.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "实例演示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.action_chains import ActionChains\n",
    "import time\n",
    "\n",
    "driver = webdriver.Chrome()\n",
    "driver.implicitly_wait(10)\n",
    "driver.get('http://www.baidu.com')\n",
    "\n",
    "# 鼠标悬停至“设置”链接\n",
    "link = driver.find_element_by_link_text('设置')\n",
    "ActionChains(driver).move_to_element(link).perform()\n",
    "\n",
    "# 打开搜索设置\n",
    "driver.find_element_by_link_text(\"搜索设置\").click()\n",
    "\n",
    "#在此处设置等待2s否则可能报错\n",
    "time.sleep(2)\n",
    "# 保存设置\n",
    "driver.find_element_by_class_name(\"prefpanelgo\").click()\n",
    "time.sleep(2)\n",
    "\n",
    "# 接受警告框\n",
    "driver.switch_to.alert.accept()\n",
    "\n",
    "# driver.quit() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 10. 下拉框选择操作\n",
    "导入选择下拉框Select类，使用该类处理下拉框操作。\n",
    "Select类的方法\n",
    "* 方法：select_by_value(“选择值”)，说明：相当于我们使用鼠标选择下拉框的值"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "有时我们会碰到下拉框，WebDriver提供了Select类来处理下拉框。 网上大量的例子，是百度搜索下拉框的选择。这里我们用Quantum Mutual Fund的站点，同时操作三个下拉框选项，并点击Search按钮, 然后下载文件的模拟。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/2018-09-23 19_10_03.png' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium.webdriver.support.select import Select\n",
    "import selenium\n",
    "# help(selenium.webdriver.support.select)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "downloading for 1\n",
      "Search\n",
      "download file: https://www.quantumamc.com/FileCDN/FactSheet/820ce35c-7f18-49a6-bf57-3a56a0caeed6.xlsx\n",
      "download file: https://www.quantumamc.com/FileCDN/FactSheet/28def8b1-16ef-41a9-90e2-d455051953d9.xlsx\n",
      "downloading for 2\n",
      "Search\n",
      "download file: https://www.quantumamc.com/FileCDN/FactSheet/bb90915e-960c-4295-8c67-c953ef6424e2.xlsx\n",
      "download file: https://www.quantumamc.com/FileCDN/FactSheet/986e9816-3a9e-4686-bee2-d7475783156f.xlsx\n",
      "downloading for 3\n",
      "Search\n",
      "download file: https://www.quantumamc.com/FileCDN/FactSheet/30066725-c31f-450b-b61f-ea43b2c2208b.xlsx\n",
      "download file: https://www.quantumamc.com/FileCDN/FactSheet/501c4e5f-8e7a-435c-ba77-965a7e1ec99b.xlsx\n"
     ]
    }
   ],
   "source": [
    "from selenium import webdriver\n",
    "from selenium.webdriver.support.select import Select\n",
    "from time import sleep\n",
    "\n",
    "driver = webdriver.Chrome()\n",
    "driver.implicitly_wait(10)\n",
    "driver.get('https://www.quantumamc.com/schemeportfolio.aspx?SchemeId=0&FactSheetType=2')\n",
    "sleep(3)\n",
    "sel = driver.find_element_by_xpath(\"//select[@id='ddnScheme'][not(@ip-type)]\")\n",
    "Select(sel).select_by_value('2') # Quantum Long Term Equity Value Fund\n",
    "\n",
    "sel = driver.find_element_by_xpath(\"//select[@id='ddnYear']\")\n",
    "Select(sel).select_by_value('2018')\n",
    "\n",
    "months = ['1', '2', '3']\n",
    "\n",
    "for month in months:\n",
    "    sel = driver.find_element_by_xpath(\"//select[@id='ddnMonth']\")\n",
    "    print('downloading for {0}'.format(month))\n",
    "    Select(sel).select_by_value(month) # May\n",
    "\n",
    "    buttonelement = driver.find_element_by_xpath('/html/body/div[1]/div/div[1]/div/div[1]/ul/li[4]/button')\n",
    "    print(buttonelement.text)\n",
    "    buttonelement.click()\n",
    "    sleep(3)\n",
    "\n",
    "    files = driver.find_elements_by_xpath(\"//*[@id=\\\"dvFactSheetList\\\"]/ul/li/h5/a\")\n",
    "    links = []\n",
    "    for file in files:\n",
    "        href = file.get_attribute('href')\n",
    "        if href not in links:\n",
    "            print('download file: {0}'.format(href))\n",
    "            file.click()\n",
    "            links.append(href)\n",
    "            sleep(2)\n",
    "\n",
    "# sleep(3)\n",
    "# driver.quit() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 11. 文件上传\n",
    "对于通过input标签实现的上传功能，可以将其看作是一个输入框，即通过send_keys()指定本地文件路径的方式实现文件上传。\n",
    "\n",
    "通过send_keys()方法来实现文件上传:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "文件上传的实际场景其实不多，这里仅仅列出相关示例代码， 这里file是文件上传按钮的name\n",
    "```\n",
    "driver.find_element_by_name(\"file\").send_keys('D:\\\\upload_file.txt')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 12. cookie操作"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "有时候我们需要验证浏览器中cookie是否正确，因为基于真实cookie的测试是无法通过白盒和集成测试进行的。WebDriver提供了操作Cookie的相关方法，可以读取、添加和删除cookie信息。\n",
    "\n",
    "WebDriver操作cookie的方法:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/2018-09-23 18_47_54.png' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=====================================\n",
      "打印cookie信息为：\n",
      "<bound method WebDriver.get_cookies of <selenium.webdriver.chrome.webdriver.WebDriver (session=\"7d891edd657c38154a282aa940dc9ca8\")>>\n",
      "=====================================\n",
      "添加cookie信息为：\n",
      "OUTFOX_SEARCH_USER_ID----897134088@183.62.146.29\n",
      "\n",
      "DICT_UGC----be3af0da19b5c5e6aa4e17bd8d90b28a|\n",
      "\n",
      "JSESSIONID----abcc01kB8bcZI9G8-Nqyw\n",
      "\n",
      "___rl__test__cookies----1537877213816\n",
      "\n",
      "OUTFOX_SEARCH_USER_ID_NCOO----566235550.1247661\n",
      "\n",
      "name----Blade\n",
      "\n",
      "=====================================\n",
      "删除一个cookie\n",
      "OUTFOX_SEARCH_USER_ID----897134088@183.62.146.29\n",
      "\n",
      "DICT_UGC----be3af0da19b5c5e6aa4e17bd8d90b28a|\n",
      "\n",
      "JSESSIONID----abcc01kB8bcZI9G8-Nqyw\n",
      "\n",
      "___rl__test__cookies----1537877213816\n",
      "\n",
      "OUTFOX_SEARCH_USER_ID_NCOO----566235550.1247661\n",
      "\n",
      "=====================================\n",
      "删除所有cookie后：\n"
     ]
    }
   ],
   "source": [
    "from selenium import webdriver\n",
    "import time\n",
    "browser = webdriver.Chrome()\n",
    "browser.get(\"http://www.youdao.com\")\n",
    "\n",
    "#1.打印cookie信息\n",
    "print('=====================================')\n",
    "print(\"打印cookie信息为：\")\n",
    "print(browser.get_cookies)\n",
    "\n",
    "#2.添加cookie信息\n",
    "dict={'name':\"name\",'value':'Blade'}\n",
    "browser.add_cookie(dict)\n",
    "\n",
    "print('=====================================')\n",
    "print('添加cookie信息为：')\n",
    "#3.遍历打印cookie信息\n",
    "for cookie in browser.get_cookies():\n",
    "    print('%s----%s\\n' %(cookie['name'],cookie['value']))\n",
    "    \n",
    "#4.删除一个cookie\n",
    "browser.delete_cookie('name')\n",
    "print('=====================================')\n",
    "print('删除一个cookie')\n",
    "for cookie in browser.get_cookies():\n",
    "    print('%s----%s\\n' %(cookie['name'],cookie['value']))\n",
    "\n",
    "print('=====================================')\n",
    "print('删除所有cookie后：')\n",
    "#5.删除所有cookie,无需传递参数\n",
    "browser.delete_all_cookies()\n",
    "for cookie in browser.get_cookies():\n",
    "    print('%s----%s\\n' %(cookie['name'],cookie['value']))\n",
    "\n",
    "time.sleep(3)\n",
    "browser.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 13. 调用javascript代码"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在上面的例子中，其实已经有通过调用javascript代码，实现逐个打开百度搜索结果链接的例子。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "虽然WebDriver提供了操作浏览器的前进和后退方法，但对于浏览器滚动条并没有提供相应的操作方法。在这种情况下，就可以借助JavaScript来控制浏览器的滚动条。WebDriver提供了execute_script()方法来执行JavaScript代码。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用于调整浏览器滚动条位置的JavaScript代码如下：\n",
    "```\n",
    "<!-- window.scrollTo(左边距,上边距); -->\n",
    "window.scrollTo(0,450);\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "window.scrollTo()方法用于设置浏览器窗口滚动条的水平和垂直位置。方法的第一个参数表示水平的左间距，第二个参数表示垂直的上边距。其代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from time import sleep\n",
    "\n",
    "#1.访问百度\n",
    "driver=webdriver.Chrome()\n",
    "driver.get(\"http://www.baidu.com\")\n",
    "\n",
    "#2.搜索\n",
    "driver.find_element_by_id(\"kw\").send_keys(\"中美贸易战\")\n",
    "driver.find_element_by_id(\"su\").click()\n",
    "\n",
    "#3.休眠2s目的是获得服务器的响应内容，如果不使用休眠可能报错\n",
    "sleep(2)\n",
    "\n",
    "#4.通过javascript设置浏览器窗口的滚动条位置\n",
    "js=\"window.scrollTo(100,450);\"\n",
    "driver.execute_script(js)\n",
    "sleep(3)\n",
    "\n",
    "# driver.close() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 14. 窗口截图"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "自动化用例是由程序去执行的，因此有时候打印的错误信息并不十分明确。如果在脚本执行出错的时候能对当前窗口截图保存，那么通过图片就可以非常直观地看出出错的原因。WebDriver提供了截图函数get_screenshot_as_file()来截取当前窗口。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 方法：get_screenshot_as_file(self, filename)，说明：用于截取当期窗口，并把图片保存到本地"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from time import sleep\n",
    "\n",
    "driver =webdriver.Firefox()\n",
    "driver.get('http://www.baidu.com')\n",
    "\n",
    "driver.find_element_by_id('kw').send_keys('python selenium')\n",
    "driver.find_element_by_id('su').click()\n",
    "sleep(2)\n",
    "\n",
    "#1.截取当前窗口，并指定截图图片的保存位置\n",
    "driver.get_screenshot_as_file(\"./image/baidu_img.png\")\n",
    "\n",
    "driver.quit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/baidu_img.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 15. 关闭浏览器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在前面的例子中我们一直使用quit()方法，其含义为退出相关的驱动程序和关闭所有窗口。除此之外，WebDriver还提供了close()方法，用来关闭当前窗口。例多窗口的处理，在用例执行的过程中打开了多个窗口，我们想要关闭其中的某个窗口，这时就要用到close()方法进行关闭了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/2018-09-23 19_01_19.png' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}