16. Parse HTML by XPATH.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 通过XPATH解析HTML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "XPath在Python的爬虫学习中，起着举足轻重的地位，对比正则表达式 re两者可以完成同样的工作，实现的功能也差不多，但XPath明显比re具有优势，在网页分析上使re退居二线。<br>\n",
    "XPath全称为XML Path Language，是一种小型的查询语言 <br>\n",
    "1） 可在XML中查找信息 <br>\n",
    "2） 支持HTML的查找 <br>\n",
    "3） 通过元素和属性进行导航 <br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python开发使用XPath的条件\n",
    "由于XPath属于lxml库模块，所以首先要安装库lxml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## XPath的简单调用方法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先加载lxml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from lxml import etree\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义方法，从html文件读取html源码，并通过etree加载。<br>\n",
    "关键部分在于：<br>\n",
    "* 编码部分，最好显式声明为utf-8\n",
    "* etree.HTML(html源码)，这句代码的目的是将源码转化为能被XPath匹配的格式"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gethtmlrootnodeforxpath(file):\n",
    "    with open(file, 'r', encoding='utf-8', errors='ignore') as f:\n",
    "        htmlbytes = bytes(bytearray(f.read(), encoding='utf-8'))\n",
    "        html = etree.HTML(htmlbytes)\n",
    "        return html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "htmlobject = gethtmlrootnodeforxpath('./html/167711672.htm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Element html at 0x1fe04e97688>\n"
     ]
    }
   ],
   "source": [
    "print(htmlobject)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color='red'>如果有将html文件转为纯文本的需求，以下代码，就可以解决这个问题</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "content = htmlobject.xpath('string(.)')\n",
    "contents = content.split('\\n')\n",
    "for text in contents:\n",
    "    if len(text.strip()) > 0:\n",
    "#         print(text.strip())\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## XPath的基本使用方法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先讲一下XPath的基本语法知识:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "四种标签的使用方法:<br>\n",
    "* 1) // 双斜杠 定位根节点，会对全文进行扫描，在文档中选取所有符合条件的内容，以列表的形式返回。 \n",
    "* 2) / 单斜杠 寻找当前标签路径的下一层路径标签或者对当前路标签内容进行操作 \n",
    "* 3) /text() 获取当前路径下的文本内容 \n",
    "* 4) /@xxxx 提取当前路径下标签的属性文本值\n",
    "* 5) | 可选符 使用|可选取若干个路径 如//p | //div 即在当前路径下选取所有符合条件的p标签和div标签。 \n",
    "* 6) . 点 用来选取当前节点 \n",
    "* 7) .. 双点 选取当前节点的父节点 \n",
    "* 另外还有starts-with(@属性名称,属性字符相同部分)，string(.)两种重要的特殊方法后面将重点讲。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 简单的例子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "html = '''\n",
    "<!DOCTYPE html>\n",
    "<html>\n",
    "    <head lang=\"en\">\n",
    "    <title>测试</title>\n",
    "    <meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\" />\n",
    "</head>\n",
    "<body>\n",
    "    <div id=\"content\">\n",
    "        <ul id=\"ul\">\n",
    "            <li>NO.1</li>\n",
    "            <li>NO.2</li>\n",
    "            <li>NO.3</li>\n",
    "        </ul>\n",
    "        <ul id=\"ul2\">\n",
    "            <li>one</li>\n",
    "            <li>two</li>\n",
    "        </ul>\n",
    "    </div>\n",
    "    <div id=\"url\">\n",
    "        <a href=\"http://www.58.com\" title=\"58\">58</a>\n",
    "        <a href=\"http://www.csdn.net\" title=\"CSDN\">CSDN</a>\n",
    "    </div>\n",
    "</body>\n",
    "</html>\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!DOCTYPE html>\n",
    "<html>\n",
    "    <head lang=\"en\">\n",
    "    <title>测试</title>\n",
    "    <meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\" />\n",
    "</head>\n",
    "<body>\n",
    "    <div id=\"content\">\n",
    "        <ul id=\"ul\">\n",
    "            <li>NO.1</li>\n",
    "            <li>NO.2</li>\n",
    "            <li>NO.3</li>\n",
    "        </ul>\n",
    "        <ul id=\"ul2\">\n",
    "            <li>one</li>\n",
    "            <li>two</li>\n",
    "        </ul>\n",
    "    </div>\n",
    "    <div id=\"url\">\n",
    "        <a href=\"http://www.58.com\" title=\"58\">58</a>\n",
    "        <a href=\"http://www.csdn.net\" title=\"CSDN\">CSDN</a>\n",
    "    </div>\n",
    "</body>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用id属性来定位哪个div和ul被匹配，使用text()获取文本内容"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NO.1\n",
      "NO.2\n",
      "NO.3\n"
     ]
    }
   ],
   "source": [
    "selector = etree.HTML(html)\n",
    "content=selector.xpath('//div[@id=\"content\"]/ul[@id=\"ul\"]/li/text()')\n",
    "for i in content:\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里使用//从全文中定位符合条件的a标签，使用“@标签属性”获取a便签的href属性值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "58\n",
      "CSDN\n"
     ]
    }
   ],
   "source": [
    "con = selector.xpath('//a[(@href)]')\n",
    "for each in con:\n",
    "    print(each.xpath('string(.)'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用绝对路径定位"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://www.58.com\n",
      "http://www.csdn.net\n"
     ]
    }
   ],
   "source": [
    "con=selector.xpath('/html/body/div/a/@href') \n",
    "# print(type(con))\n",
    "# print(len(con))\n",
    "for each in con:\n",
    "    print(each)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 进阶用法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* starts-with 解决标签属性值以相同字符串开头的情况"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "aa\n",
      "ab\n"
     ]
    }
   ],
   "source": [
    "html=\"\"\"\n",
    "    <body>\n",
    "        <div id=\"aa\">aa</div>\n",
    "        <div id=\"ab\">ab<p>ss</p></div>\n",
    "        <div id=\"Cc\">ac</div>\n",
    "    </body>\n",
    "    \"\"\"\n",
    "selector=etree.HTML(html)\n",
    "#这里使用starts-with方法提取div的id标签属性值开头为a的div标签\n",
    "content=selector.xpath('//div[starts-with(@id,\"a\")]/text()') \n",
    "for each in content:\n",
    "    print(each)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*  string(.) 标签套标签，具体含义就是采集xpath路径所包含的所有标签内的文本"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "only input pure text\n",
      "\n",
      "    left\n",
      "    \n",
      "\n",
      "    west\n",
      "\n",
      "output all text\n",
      "\n",
      "    left\n",
      "    \n",
      "        right\n",
      "        \n",
      "        up\n",
      "        down\n",
      "        \n",
      "        this is red text\n",
      "        east\n",
      "    \n",
      "    west\n",
      "\n"
     ]
    }
   ],
   "source": [
    "html=\"\"\"\n",
    "<div id=\"a\">\n",
    "    left\n",
    "    <div id=\"b\">\n",
    "        right\n",
    "        <ul>\n",
    "        up\n",
    "        <li>down</li>\n",
    "        </ul>\n",
    "        <font color='red'>this is red text</font>\n",
    "        east\n",
    "    </div>\n",
    "    west\n",
    "</div>\n",
    "<div id=\"c\">\n",
    "    white\n",
    "    <div id=\"d\">\n",
    "        east\n",
    "        <ul>\n",
    "        up\n",
    "        <li>down</li>\n",
    "        </ul>\n",
    "        <font color='red'>this is red text</font>\n",
    "        east\n",
    "    </div>\n",
    "    west\n",
    "</div>\n",
    "\"\"\"\n",
    "#下面是没有用string方法的输出\n",
    "sel=etree.HTML(html)\n",
    "con=sel.xpath('//div[@id=\"a\"]/text()')\n",
    "print('only input pure text')\n",
    "for i in con:\n",
    "    print(i)   #输出内容为left west\n",
    "\n",
    "print('output all text')\n",
    "data=sel.xpath('//div[@id=\"a\"]')[0]\n",
    "info=data.xpath('string(.)')\n",
    "#输出为 全部内容\n",
    "print(info)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id=\"a\">\n",
    "    left\n",
    "    <div id=\"b\">\n",
    "        right\n",
    "        <ul>\n",
    "        up\n",
    "        <li>down</li>\n",
    "        </ul>\n",
    "        <font color='red'>this is red text</font>\n",
    "        east\n",
    "    </div>\n",
    "    west\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 获得XPath的方法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 使用Chrome浏览器\n",
    "* 在具体要获取的元素，鼠标右键单击\n",
    "* 选择Inspect，定位HTML标签"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/2018-09-08 18_52_55-PL SVUL Protector 2018 Combined Document.png'  />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 定位到标签之后，鼠标右键源码，copy -> copy xpath"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/2018-09-08 18_55_35-PL SVUL Protector 2018 Combined Document.png' >"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "于是我们就得到具体的XPath：/html/body/div[127]/div/table/tbody/tr[6]/td[1]/div/font<br>\n",
    "真实情况中，我们不会这样写绝对定位路径，往往会通过//div/div/table的方式，尝试获得Table列表，然后根据Table中的表头信息，去定位我们想要的table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 实战"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "XPath在实际工作中使用的场景:<br>\n",
    "基金公司公布的Share Name往往集中在某些Table，DIV内，使用XPath能够很方便定位"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Document Id: 167711672"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "需求：获取文档中的Share Name。<br>\n",
    "Share Name位于文章的表格中，且为第一列\n",
    "<img src='./image/2018-08-15 18_26_33-PL SVUL Protector 2018 Combined Document.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将html对象加载到lxml中："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "htmlobject = gethtmlrootnodeforxpath('./html/167711672.htm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "经过分析，我们如果需要得到table，可以通过如下路径：/html/body/div/div/table,得到一堆table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "74\n"
     ]
    }
   ],
   "source": [
    "tablelist = htmlobject.xpath('/html/body/div/div/table')\n",
    "print(len(tablelist))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如何找到我想要的table呢？<br>\n",
    "* 定位表头<br>\n",
    "经过分析，该table的第二列的文字是investment objective summary，这其实可以区分其他非share name描述的table，然后我们可以获取任意该列的xpath<br>\n",
    "/html/body/div[127]/div/table/tbody/tr[4]/td[2]/div/font<br>\n",
    "我们忽略tbody之前的内容，可以发现能够通过tbody/tr[4]/td[2]进行定位了。\n",
    "* 定位share name<br>\n",
    "我们定位任意share name: /html/body/div[127]/div/table/tbody/tr[6]/td[1]/div/font<br>\n",
    "因为share name有很多，所以我们通过/tbody/tr/td[1]进行过滤，获得列表即可\n",
    "* 加上限制<br>\n",
    "因为表格中有跨列的内容，如ADVANCED SERIES TRUST，所以还需要加入span相关的排除描述:<br>\n",
    "tbody/tr/td[not(@colspan)][1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getpureelementtext(element):\n",
    "    text = element.xpath('string(.)').strip()\n",
    "    text = text.replace('\\n', ' ')\n",
    "    text = re.sub('( ){2,}', ' ', text)\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "def searchbykeyword(keyword, text):\n",
    "    pattern = re.compile(keyword)\n",
    "    return re.search(pattern, text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "share 1, name is: AST Advanced Strategies Portfolio\n",
      "share 2, name is: AST Balanced Asset Allocation Portfolio\n",
      "share 3, name is: AST BlackRock Global Strategies Portfolio\n",
      "share 4, name is: AST BlackRock Low Duration Bond Portfolio\n",
      "share 5, name is: AST BlackRock/Loomis Sayles Bond Portfolio\n",
      "share 6, name is: AST Fidelity Institutional AMSM Quantitative Portfolio (formerly AST FI Pyramis® Quantitative Portfolio)\n",
      "share 7, name is: AST Goldman Sachs Mid-Cap Growth Portfolio\n",
      "share 8, name is: AST Hotchkis & Wiley Large-Cap Value Portfolio\n",
      "share 9, name is: AST International Value Portfolio\n",
      "share 10, name is: AST J.P. Morgan International Equity Portfolio\n",
      "share 11, name is: AST J.P. Morgan Strategic Opportunities Portfolio\n",
      "share 12, name is: AST Loomis Sayles Large-Cap Growth Portfolio\n",
      "share 13, name is: AST MFS Global Equity Portfolio\n",
      "share 14, name is: AST MFS Growth Portfolio\n",
      "share 15, name is: AST Preservation Asset Allocation Portfolio\n",
      "share 16, name is: AST Prudential Growth Allocation Portfolio\n",
      "share 17, name is: AST RCM World Trends Portfolio\n",
      "share 18, name is: AST T. Rowe Price Asset Allocation Portfolio\n",
      "share 19, name is: AST T. Rowe Price Large-Cap Growth Portfolio\n",
      "share 20, name is: AST T. Rowe Price Large-Cap Value Portfolio\n",
      "share 21, name is: AST Templeton Global Bond Portfolio\n",
      "share 22, name is: AST Wellington Management Hedged Equity Portfolio\n",
      "share 23, name is: PSF Conservative Balanced Portfolio - Class I\n",
      "share 24, name is: PSF Diversified Bond Portfolio - Class I\n",
      "share 25, name is: PSF Equity Portfolio - Class I\n",
      "share 26, name is: PSF Flexible Managed Portfolio - Class I\n",
      "share 27, name is: PSF Global Portfolio - Class I\n",
      "share 28, name is: PSF Government Money Market Portfolio - Class I\n",
      "share 29, name is: PSF High Yield Bond Portfolio - Class I\n",
      "share 30, name is: PSF Jennison Portfolio - Class I\n",
      "share 31, name is: PSF Jennison 20/20 Focus Portfolio - Class I\n",
      "share 32, name is: PSF SP Prudential U.S. Emerging Growth Portfolio - Class I\n",
      "share 33, name is: PSF SP Small-Cap Value Portfolio - Class I\n",
      "share 34, name is: PSF Stock Index Portfolio - Class I\n",
      "share 35, name is: PSF Value Portfolio - Class I\n",
      "share 36, name is: American Funds Insurance Series® Blue Chip Income and Growth Fund - Class 2\n",
      "share 37, name is: American Funds Insurance Series® Growth Fund - Class 2\n",
      "share 38, name is: American Funds Insurance Series® Growth-Income Fund- Class 2\n",
      "share 39, name is: American Funds Insurance Series® International Fund - Class 2\n",
      "share 40, name is: The Dreyfus Sustainable U.S. Equity Portfolio, Inc. - Service Shares\n",
      "share 41, name is: Fidelity® VIP Index 500 Portfolio - Service Class 2\n",
      "share 42, name is: Franklin Mutual Shares VIP Fund - Class 2\n",
      "share 43, name is: MFS® Utilities Series - Initial Class\n",
      "share 44, name is: Neuberger Berman AMT Sustainable Equity Portfolio (formerly Neuberger Berman AMT Socially Responsive Portfolio) - Class S\n",
      "share 45, name is: TOPS® Aggressive Growth ETF Portfolio - Class 2\n",
      "share 46, name is: TOPS® Balanced ETF Portfolio - Class 2\n",
      "share 47, name is: TOPS® Conservative ETF Portfolio - Class 2\n",
      "share 48, name is: TOPS® Growth ETF Portfolio - Class 2\n",
      "share 49, name is: TOPS® Managed Risk Balanced ETF Portfolio - Class 2\n",
      "share 50, name is: TOPS® Managed Risk Growth ETF Portfolio - Class 2\n",
      "share 51, name is: TOPS® Managed Risk Moderate Growth ETF Portfolio - Class 2\n",
      "share 52, name is: TOPS® Moderate Growth ETF Portfolio - Class 2\n"
     ]
    }
   ],
   "source": [
    "keyword = r'(\\b([0-9A-Zi])\\S*( )\\b){2,}'\n",
    "sharecount = 1\n",
    "for table in tablelist:\n",
    "    header = table.xpath('tbody/tr[4]/td[2]')\n",
    "    if len(header) > 0:\n",
    "        temp = getpureelementtext(header[0])\n",
    "#         print(temp)\n",
    "        # 通过header文字，得到正确的table\n",
    "        if temp.lower() == 'investment objective summary':\n",
    "            # 通过share name的xpath，获得相应的element list\n",
    "            sharelist = table.xpath('tbody/tr/td[not(@colspan)][1]')\n",
    "            for index, share in enumerate(sharelist):\n",
    "                sharename = getpureelementtext(share)\n",
    "                # Share Name一般都是大写字母开头的单词组合而成\n",
    "                if searchbykeyword(keyword, sharename) is not None:\n",
    "                    print('share {0}, name is: {1}'.format(sharecount, getpureelementtext(share)))\n",
    "                    sharecount += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Document Id: 169431423"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "需求：获取文档中的Share Name。<br>\n",
    "Share Name位于文章的DIV中<br>\n",
    "<img src='./image/2018-08-23 10_25_51-BOA CVUL Future.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将html对象加载到lxml中："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "htmlfordivobject = gethtmlrootnodeforxpath('./html/169431423.htm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "经过分析，发现share name都是在div段落中，并且有特定的属性和样式"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/2018-09-08 19_43_32-BOA CVUL Future.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 首先定位div段落: /html/body/div[206]/div[1]，结合属性，我们可以得到这样的xpath: /html/body/div/div[@type=\\\"Block\\\"]\n",
    "* 然后定位share name，结合样式，我们可以得到这样的xpath: div[contains(@style,\\\"FONT-SIZE: 9pt;\\\") and contains(@style, \\\"FONT-WEIGHT: bold;\\\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "具体代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "share 1, name is: AllianceBernstein Variable Products Series Fund, Inc. - AB VPS Growth and Income Portfolio: Class A\n",
      "share 2, name is: AllianceBernstein Variable Products Series Fund, Inc. - AB VPS International Value Portfolio: Class A\n",
      "share 3, name is: AllianceBernstein Variable Products Series Fund, Inc. - AB VPS Small/Mid Cap Value Portfolio: Class A\n",
      "share 4, name is: American Century Variable Portfolios II, Inc. - American Century VP Inflation Protection Fund: Class I\n",
      "share 5, name is: American Century Variable Portfolios, Inc. - American Century VP Capital Appreciation Fund: Class I\n",
      "share 6, name is: American Century Variable Portfolios, Inc. - American Century VP Income & Growth Fund: Class I\n",
      "share 7, name is: American Century Variable Portfolios, Inc. - American Century VP International Fund: Class I\n",
      "share 8, name is: American Century Variable Portfolios, Inc. - American Century VP Mid Cap Value Fund: Class I\n",
      "share 9, name is: American Century Variable Portfolios, Inc. - American Century VP Ultra® Fund: Class I\n",
      "share 10, name is: American Century Variable Portfolios, Inc. - American Century VP Value Fund: Class I\n",
      "share 11, name is: American Funds Insurance Series® - Asset Allocation Fund: Class 2\n",
      "share 12, name is: American Funds Insurance Series® - Bond Fund: Class 2\n",
      "share 13, name is: American Funds Insurance Series® - Global Small Capitalization Fund: Class 2\n",
      "share 14, name is: American Funds Insurance Series® - Growth Fund: Class 2\n",
      "share 15, name is: American Funds Insurance Series® - International Fund: Class 2\n",
      "share 16, name is: American Funds Insurance Series® - New World Fund®: Class 2\n",
      "share 17, name is: BlackRock Variable Series Funds, Inc. - BlackRock Advantage Large Cap Core V.I. Fund: Class II\n",
      "share 18, name is: BlackRock Variable Series Funds, Inc. - BlackRock Global Allocation V.I. Fund: Class II\n",
      "share 19, name is: BlackRock Variable Series Funds, Inc. - BlackRock High Yield V.I. Fund: Class I\n",
      "share 20, name is: Calvert Variable Products, Inc. - Calvert VP S&P 500 Index Portfolio\n",
      "share 21, name is: Davis Variable Account Fund, Inc. - Davis Value Portfolio\n",
      "share 22, name is: Delaware VIP Trust - Delaware VIP Emerging Markets Series: Service Class\n",
      "share 23, name is: Delaware VIP Trust - Delaware VIP Small Cap Value Series: Service Class\n",
      "share 24, name is: Deutsche Variable Series II - Deutsche CROCI® U.S. VIP: Class B\n",
      "share 25, name is: Deutsche Variable Series II - Deutsche Small Mid Cap Value VIP: Class B\n",
      "share 26, name is: Dreyfus Investment Portfolios - MidCap Stock Portfolio: Initial Shares\n",
      "share 27, name is: Dreyfus Investment Portfolios - Small Cap Stock Index Portfolio: Service Shares\n",
      "share 28, name is: Dreyfus Stock Index Fund, Inc.: Initial Shares\n",
      "share 29, name is: Dreyfus Sustainable U.S. Equity Portfolio, Inc. (The): Initial Shares\n",
      "share 30, name is: Dreyfus Variable Investment Fund - Appreciation Portfolio: Initial Shares\n",
      "share 31, name is: Dreyfus Variable Investment Fund - International Value Portfolio: Initial Shares\n",
      "share 32, name is: Eaton Vance Variable Trust - Eaton Vance VT Floating-Rate Income Fund: Initial Class\n",
      "share 33, name is: Federated Insurance Series - Federated Fund for U.S. Government Securities II\n",
      "share 34, name is: Federated Insurance Series - Federated Quality Bond Fund II: Primary Shares\n",
      "share 35, name is: Fidelity Variable Insurance Products - Emerging Markets Portfolio: Service Class\n",
      "share 36, name is: Fidelity Variable Insurance Products Fund - Fidelity VIP Freedom Fund 2015 Portfolio: Service Class\n",
      "share 37, name is: Fidelity Variable Insurance Products Fund - Fidelity VIP Freedom Fund 2020 Portfolio: Service Class\n",
      "share 38, name is: Fidelity Variable Insurance Products Fund - Fidelity VIP Freedom Fund 2025 Portfolio: Service Class\n",
      "share 39, name is: Fidelity Variable Insurance Products Fund - Fidelity VIP Freedom Fund 2030 Portfolio: Service Class\n",
      "share 40, name is: Fidelity Variable Insurance Products Fund - Fidelity VIP Freedom Fund 2035 Portfolio: Service Class\n",
      "share 41, name is: Fidelity Variable Insurance Products Fund - Fidelity VIP Freedom Fund 2040 Portfolio: Service Class\n",
      "share 42, name is: Fidelity Variable Insurance Products Fund - Fidelity VIP Freedom Fund 2045 Portfolio: Service Class\n",
      "share 43, name is: Fidelity Variable Insurance Products Fund - Fidelity VIP Freedom Fund 2050 Portfolio: Service Class\n",
      "share 44, name is: Fidelity Variable Insurance Products Fund - VIP Contrafund® Portfolio: Service Class\n",
      "share 45, name is: Fidelity Variable Insurance Products Fund - VIP Equity-Income Portfolio: Service Class\n",
      "share 46, name is: Fidelity Variable Insurance Products Fund - VIP Growth Opportunities Portfolio: Service Class\n",
      "share 47, name is: Fidelity Variable Insurance Products Fund - VIP Growth Portfolio: Service Class\n",
      "share 48, name is: Fidelity Variable Insurance Products Fund - VIP High Income Portfolio: Service Class\n",
      "share 49, name is: Fidelity Variable Insurance Products Fund - VIP Investment Grade Bond Portfolio: Service Class\n",
      "share 50, name is: Fidelity Variable Insurance Products Fund - VIP Mid Cap Portfolio: Service Class\n",
      "share 51, name is: Fidelity Variable Insurance Products Fund - VIP Overseas Portfolio: Service Class\n",
      "share 52, name is: Fidelity Variable Insurance Products Fund - VIP Real Estate Portfolio: Service Class\n",
      "share 53, name is: Fidelity Variable Insurance Products Fund - VIP Value Strategies Portfolio: Service Class\n",
      "share 54, name is: Franklin Templeton Variable Insurance Products Trust - Franklin Mutual Global Discovery VIP Fund: Class 2\n",
      "share 55, name is: Franklin Templeton Variable Insurance Products Trust - Franklin Small Cap Value VIP Fund: Class 2\n",
      "share 56, name is: Franklin Templeton Variable Insurance Products Trust - Templeton Foreign VIP Fund: Class 2\n",
      "share 57, name is: Franklin Templeton Variable Insurance Products Trust - Templeton Global Bond VIP Fund: Class 1\n",
      "share 58, name is: Franklin Templeton Variable Insurance Products Trust - Templeton Global Bond VIP Fund: Class 2\n",
      "share 59, name is: Goldman Sachs Variable Insurance Trust - Goldman Sachs Growth Opportunities Fund: Service Shares\n",
      "share 60, name is: Goldman Sachs Variable Insurance Trust - Goldman Sachs Small Cap Equity Insights Fund: Institutional Shares\n",
      "share 61, name is: Goldman Sachs Variable Insurance Trust - Goldman Sachs VIT Mid Cap Value Fund: Institutional Shares\n",
      "share 62, name is: Invesco - Invesco V.I. Growth and Income Fund: Series I Shares\n",
      "share 63, name is: Invesco - Invesco V.I. High Yield Fund: Series I Shares\n",
      "share 64, name is: Invesco - Invesco V.I. International Growth Fund: Series I Shares\n",
      "share 65, name is: Invesco - Invesco V.I. Mid Cap Core Equity Fund: Series I Shares\n",
      "share 66, name is: Invesco - Invesco V.I. Mid Cap Growth Fund: Series I Shares\n",
      "share 67, name is: Invesco - Invesco V.I. Small Cap Equity Fund: Series I Shares\n",
      "share 68, name is: Invesco - Invesco V.I. Value Opportunities Fund: Series I Shares\n",
      "share 69, name is: Ivy Variable Insurance Portfolios - Advantus Real Estate Securities: Class II (formerly, Ivy Variable Insurance Portfolios - Advantus Real Estate Securities)\n",
      "share 70, name is: Ivy Variable Insurance Portfolios - Asset Strategy: Class II (formerly, Ivy Variable Insurance Portfolios - Asset Strategy)\n",
      "share 71, name is: Ivy Variable Insurance Portfolios - Growth: Class II (formerly, Ivy Variable Insurance Portfolios - Growth)\n",
      "share 72, name is: Ivy Variable Insurance Portfolios - High Income: Class II (formerly, Ivy Variable Insurance Portfolios - High Income)\n",
      "share 73, name is: Ivy Variable Insurance Portfolios - Mid Cap Growth: Class II (formerly, Ivy Variable Insurance Portfolios - Mid Cap Growth)\n",
      "share 74, name is: Ivy Variable Insurance Portfolios - Science and Technology: Class II (formerly, Ivy Variable Insurance Portfolios - Science and Technology)\n",
      "share 75, name is: Janus Henderson VIT Balanced Portfolio: Service Shares\n",
      "share 76, name is: Janus Henderson VIT Enterprise Portfolio: Service Shares\n",
      "share 77, name is: Janus Henderson VIT Flexible Bond Portfolio: Service Shares\n",
      "share 78, name is: Janus Henderson VIT Forty Portfolio: Service Shares\n",
      "share 79, name is: Janus Henderson VIT Global Technology Portfolio: Service Shares\n",
      "share 80, name is: Janus Henderson VIT Mid Cap Value Portfolio: Service Shares\n",
      "share 81, name is: Janus Henderson VIT Overseas Portfolio: Service Shares\n",
      "share 82, name is: JPMorgan Insurance Trust - JPMorgan Insurance Trust Mid Cap Value Portfolio: Class 1\n",
      "share 83, name is: JPMorgan Insurance Trust - JPMorgan Insurance Trust Small Cap Core Portfolio: Class 1\n",
      "share 84, name is: Lazard Retirement Series, Inc. - Lazard Retirement Emerging Markets Equity Portfolio: Service Shares\n",
      "share 85, name is: Legg Mason Partners Variable Equity Trust - ClearBridge Variable Small Cap Growth Portfolio: Class I\n",
      "share 86, name is: Lincoln Variable Insurance Products Trust - LVIP Baron Growth Opportunities Fund: Service Class\n",
      "share 87, name is: Lord Abbett Series Fund, Inc. - Bond Debenture Portfolio: Class VC\n",
      "share 88, name is: Lord Abbett Series Fund, Inc. - Mid-Cap Stock Portfolio: Class VC\n",
      "share 89, name is: Lord Abbett Series Fund, Inc. - Short Duration Income Portfolio: Class VC\n",
      "share 90, name is: Lord Abbett Series Fund, Inc. - Total Return Portfolio: Class VC\n",
      "share 91, name is: MFS® Variable Insurance Trust - MFS Mid Cap Growth Series: Service Class\n",
      "share 92, name is: MFS® Variable Insurance Trust - MFS New Discovery Series: Service Class\n",
      "share 93, name is: MFS® Variable Insurance Trust - MFS Total Return Bond Series: Service Class\n",
      "share 94, name is: MFS® Variable Insurance Trust - MFS Value Series: Service Class\n",
      "share 95, name is: MFS® Variable Insurance Trust II - MFS Blended Research® Core Equity Portfolio: Service Class\n",
      "share 96, name is: MFS® Variable Insurance Trust II - MFS International Value Portfolio: Service Class\n",
      "share 97, name is: MFS® Variable Insurance Trust II - MFS Research International Portfolio: Service Class\n",
      "share 98, name is: Morgan Stanley Variable Insurance Fund, Inc. - Emerging Markets Debt Portfolio: Class I\n",
      "share 99, name is: Morgan Stanley Variable Insurance Fund, Inc. - Global Real Estate Portfolio: Class II\n",
      "share 100, name is: Morgan Stanley Variable Insurance Fund, Inc. - Growth Portfolio: Class I\n",
      "share 101, name is: Morgan Stanley Variable Insurance Fund, Inc. - Mid Cap Growth Portfolio: Class I\n",
      "share 102, name is: Morgan Stanley Variable Insurance Fund, Inc. - U.S. Real Estate Portfolio: Class I\n",
      "share 103, name is: Nationwide Variable Insurance Trust - BlackRock NVIT Equity Dividend Fund: Class I\n",
      "share 104, name is: Nationwide Variable Insurance Trust - Federated NVIT High Income Bond Fund: Class I\n",
      "share 105, name is: Nationwide Variable Insurance Trust - Neuberger Berman NVIT Socially Responsible Fund: Class I\n",
      "share 106, name is: Nationwide Variable Insurance Trust - NVIT Bond Index Fund: Class I\n",
      "share 107, name is: Nationwide Variable Insurance Trust - NVIT Cardinal(SM) Aggressive Fund: Class I\n",
      "share 108, name is: Nationwide Variable Insurance Trust - NVIT Cardinal(SM) Balanced Fund: Class I\n",
      "share 109, name is: Nationwide Variable Insurance Trust - NVIT Cardinal(SM) Capital Appreciation Fund: Class I\n",
      "share 110, name is: Nationwide Variable Insurance Trust - NVIT Cardinal(SM) Conservative Fund: Class I\n",
      "share 111, name is: Nationwide Variable Insurance Trust - NVIT Cardinal(SM) Moderate Fund: Class I\n",
      "share 112, name is: Nationwide Variable Insurance Trust - NVIT Cardinal(SM) Moderately Aggressive Fund: Class I\n",
      "share 113, name is: Nationwide Variable Insurance Trust - NVIT Cardinal(SM) Moderately Conservative Fund: Class I\n",
      "share 114, name is: Nationwide Variable Insurance Trust - NVIT Emerging Markets Fund: Class I\n",
      "share 115, name is: Nationwide Variable Insurance Trust - NVIT Government Bond Fund: Class I\n",
      "share 116, name is: Nationwide Variable Insurance Trust - NVIT Government Money Market Fund: Class I\n",
      "share 117, name is: Nationwide Variable Insurance Trust - NVIT Government Money Market Fund: Class V\n",
      "share 118, name is: Nationwide Variable Insurance Trust - NVIT International Equity Fund: Class I\n",
      "share 119, name is: Nationwide Variable Insurance Trust - NVIT International Index Fund: Class I\n",
      "share 120, name is: Nationwide Variable Insurance Trust - NVIT International Index Fund: Class II\n",
      "share 121, name is: Nationwide Variable Insurance Trust - NVIT Investor Destinations Aggressive Fund: Class II\n",
      "share 122, name is: Nationwide Variable Insurance Trust - NVIT Investor Destinations Conservative Fund: Class II\n",
      "share 123, name is: Nationwide Variable Insurance Trust - NVIT Investor Destinations Moderate Fund: Class II\n",
      "share 124, name is: Nationwide Variable Insurance Trust - NVIT Investor Destinations Moderately Aggressive Fund: Class II\n",
      "share 125, name is: Nationwide Variable Insurance Trust - NVIT Investor Destinations Moderately Conservative Fund: Class II\n",
      "share 126, name is: Nationwide Variable Insurance Trust - NVIT Large Cap Growth Fund: Class I\n",
      "share 127, name is: Nationwide Variable Insurance Trust - NVIT Mid Cap Index Fund: Class I\n",
      "share 128, name is: Nationwide Variable Insurance Trust - NVIT Multi Sector Bond Fund: Class I\n",
      "share 129, name is: Nationwide Variable Insurance Trust - NVIT Multi-Manager Large Cap Growth Fund: Class I\n",
      "share 130, name is: Nationwide Variable Insurance Trust - NVIT Multi-Manager Large Cap Value Fund: Class I\n",
      "share 131, name is: Nationwide Variable Insurance Trust - NVIT Multi-Manager Mid Cap Growth Fund: Class I\n",
      "share 132, name is: Nationwide Variable Insurance Trust - NVIT Multi-Manager Mid Cap Value Fund: Class I\n",
      "share 133, name is: Nationwide Variable Insurance Trust - NVIT Multi-Manager Small Cap Growth Fund: Class I\n",
      "share 134, name is: Nationwide Variable Insurance Trust - NVIT Multi-Manager Small Cap Value Fund: Class I\n",
      "share 135, name is: Nationwide Variable Insurance Trust - NVIT Multi-Manager Small Company Fund: Class I\n",
      "share 136, name is: Nationwide Variable Insurance Trust - NVIT Nationwide Fund: Class I\n",
      "share 137, name is: Nationwide Variable Insurance Trust - NVIT Real Estate Fund: Class I\n",
      "share 138, name is: Nationwide Variable Insurance Trust - NVIT Short Term Bond Fund: Class I\n",
      "share 139, name is: Neuberger Berman Advisers Management Trust - Guardian Portfolio: Class I\n",
      "share 140, name is: Neuberger Berman Advisers Management Trust - Large Cap Value Portfolio: Class I\n",
      "share 141, name is: Neuberger Berman Advisers Management Trust - Mid Cap Intrinsic Value Portfolio: Class I\n",
      "share 142, name is: Neuberger Berman Advisers Management Trust - Mid-Cap Growth Portfolio: Class I\n",
      "share 143, name is: Neuberger Berman Advisers Management Trust - Mid-Cap Growth Portfolio: Class S\n",
      "share 144, name is: Oppenheimer Variable Account Funds - Oppenheimer Capital Appreciation Fund/VA: Non-Service Shares\n",
      "share 145, name is: Oppenheimer Variable Account Funds - Oppenheimer Discovery Mid Cap Growth Fund/VA: Non-Service Shares\n",
      "share 146, name is: Oppenheimer Variable Account Funds - Oppenheimer Global Fund/VA: Non-Service Shares\n",
      "share 147, name is: Oppenheimer Variable Account Funds - Oppenheimer International Growth Fund/VA: Non-Service Shares\n",
      "share 148, name is: Oppenheimer Variable Account Funds - Oppenheimer Main Street Fund®/VA: Non-Service Shares\n",
      "share 149, name is: Oppenheimer Variable Account Funds - Oppenheimer Main Street Small Cap Fund®/VA: Non-Service Shares\n",
      "share 150, name is: PIMCO Variable Insurance Trust - All Asset Portfolio: Administrative Class\n",
      "share 151, name is: PIMCO Variable Insurance Trust - Foreign Bond Portfolio (U.S. Dollar-Hedged): Administrative Class\n",
      "share 152, name is: PIMCO Variable Insurance Trust - Foreign Bond Portfolio (Unhedged): Administrative Class\n",
      "share 153, name is: PIMCO Variable Insurance Trust - Global Bond Portfolio (Unhedged): Administrative Class\n",
      "share 154, name is: PIMCO Variable Insurance Trust - High Yield Portfolio: Administrative Class\n",
      "share 155, name is: PIMCO Variable Insurance Trust - Long-Term U.S. Government Portfolio: Administrative Class\n",
      "share 156, name is: PIMCO Variable Insurance Trust - Low Duration Portfolio: Administrative Class\n",
      "share 157, name is: PIMCO Variable Insurance Trust - Real Return Portfolio: Administrative Class\n",
      "share 158, name is: PIMCO Variable Insurance Trust - Total Return Portfolio: Administrative Class\n",
      "share 159, name is: Pioneer Variable Contracts Trust - Pioneer High Yield VCT Portfolio: Class I\n",
      "share 160, name is: Putnam Variable Trust - Putnam VT Small Cap Value Fund: Class IB\n",
      "share 161, name is: Royce Capital Fund - Royce Micro-Cap Portfolio: Investment Class\n",
      "share 162, name is: T. Rowe Price Equity Series, Inc. - T. Rowe Price Blue Chip Growth Portfolio\n",
      "share 163, name is: T. Rowe Price Equity Series, Inc. - T. Rowe Price Equity Income Portfolio: II\n",
      "share 164, name is: T. Rowe Price Equity Series, Inc. - T. Rowe Price Mid-Cap Growth Portfolio: II\n",
      "share 165, name is: T. Rowe Price Equity Series, Inc. - T. Rowe Price New America Growth Portfolio\n",
      "share 166, name is: T. Rowe Price Equity Series, Inc. - T. Rowe Price Personal Strategy Balanced Portfolio\n",
      "share 167, name is: T. Rowe Price Fixed Income Series, Inc. - T. Rowe Price Limited-Term Bond Portfolio\n",
      "share 168, name is: VanEck VIP Trust - VanEck VIP Emerging Markets Fund: Initial Class\n",
      "share 169, name is: VanEck VIP Trust - VanEck VIP Global Hard Assets Fund: Initial Class\n",
      "share 170, name is: Wells Fargo Variable Trust - VT Discovery Fund: Class 2\n",
      "share 171, name is: Wells Fargo Variable Trust - VT Opportunity Fund: Class 2\n",
      "share 172, name is: Wells Fargo Variable Trust - VT Small Cap Growth Fund: Class 2\n"
     ]
    }
   ],
   "source": [
    "keyword = r'(\\b([0-9A-Zi])\\S*( )\\b){2,}'\n",
    "sharecount = 1\n",
    "for block in htmlfordivobject.xpath('/html/body/div/div[@type=\\\"Block\\\"]'):\n",
    "    sharelist = block.xpath('div[contains(@style,\\\"FONT-SIZE: 9pt;\\\") and contains(@style, \\\"FONT-WEIGHT: bold;\\\")]')\n",
    "    for share in sharelist:\n",
    "        sharename = getpureelementtext(share)\n",
    "        # Share Name一般都是大写字母开头的单词组合而成\n",
    "        if searchbykeyword(keyword, sharename) is not None:\n",
    "            print('share {0}, name is: {1}'.format(sharecount, getpureelementtext(share)))\n",
    "            sharecount += 1\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 不太适合用XPath的场景"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "刚刚的例子是针对从具体的某一段HTML标签拿取相关信息。<br>\n",
    "但是如果需要从文本中，根据某些关键字，如new, close定位相应的段落，即确定action；<br>并且再从上下文获取具体的share name或者pending date<br>\n",
    "根据实际经验，感觉通过转为纯文本，然后正则定位或许更合适。<br>\n",
    "用XPath可能反而会让事情复杂化，而且<b>XPath的有效性，取决于html的样式不能频繁发生变化。</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}