14. regex samples.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 正则表达式样例解析"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 适用正则表达式的场景\n",
    "我们日常工作或生活，都会遇到大量的文本信息，往往需要做文本查找。<br>\n",
    "对于简单的查询，比如查找\"Close\"这个文字在哪，通过ctrl+F即可做到简单查询。<br>\n",
    "但是或许我们有如下的一些需求，比如获取文档中的日期、数字，包含某个或某些关键字的一段话。<br>\n",
    "这种情况用简单的查询，就很难定位到所有相关的文本信息。<br>\n",
    "那么正则表达式，就可以华丽登场了~~~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 查找SEC文档中的日期\n",
    "SEC文章中的格式，往往是这种格式：英文单词月 数字日1~2位, 数字年4位\n",
    "那么正则表达式可以这样写："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "May 1, 2018\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "def getlistiterbyregex(keyword, text):\n",
    "    pattern = re.compile(keyword, re.I)\n",
    "    return re.finditer(pattern, text)\n",
    "\n",
    "text = r'The effectivedate is May 1, 2018, the disclosed date is April 25, 2018. '\n",
    "keyword = r'effectivedate is ((january|february|march|april|may|june|july|august|september|october|november|december)[\\s]*[0-9]{1,2}[\\s]*,[\\s]*[0-9]{4}),'\n",
    "matchlist = getlistiterbyregex(keyword, text)\n",
    "for match in matchlist:\n",
    "    print(match.group(1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 查找文档中的含有百分号的数字"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10%\n",
      "10%\n",
      "3.8%\n"
     ]
    }
   ],
   "source": [
    "text = r'''Additional Taxes Payable On Withdrawals, Surrenders, Or Annuity Payouts\n",
    "The Code may impose a 10% additional tax on any distribution from your contract which you must include in your gross income. The 10% additional tax does not apply if one of several exceptions exists. These exceptions include withdrawals, surrenders, or Annuity Payouts that:\n",
    "you receive on or after you reach 59 , you receive because you became disabled (as defined in the Code), you receive from an immediate annuity, a Beneficiary receives on or after your death, or you receive as a series of substantially equal periodic payments based on your life or life expectancy (non-natural owners holding as agent for an individual do not qualify).\n",
    "Unearned Income Medicare Contribution\n",
    "Congress enacted the “Unearned Income Medicare Contribution” as a part of the Health Care and Education Reconciliation Act of 2010. This tax, which affects individuals whose modified adjusted gross income exceeds certain thresholds, is a 3.8% tax on the lesser of (i) the individual's “unearned income”, or (ii) the dollar amount by which the individual's modified adjusted gross income exceeds the applicable threshold. Unearned income includes the taxable portion of distributions that you take from your annuity contract. If you take a distribution from your contract that may be subject to the tax, we will include a Distribution Code “D” in Box 7 of the Form 1099-R issued to report the distribution. Please consult your tax advisor to determine whether your annuity distributions are subject to this tax.\n",
    "Special Rules If You Own More Than One Annuity Contract'''\n",
    "keyword = r'[\\d]*(\\.[\\d]*)?\\%'\n",
    "matchlist = getlistiterbyregex(keyword, text)\n",
    "for match in matchlist:\n",
    "    print(match.group(0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 根据分组从一段话中拿到特定内容\n",
    "分组，在正则表达式是通过括号:\\(\\)实现的，如果加入```(?<groupname>正则表达式)```，即可实现有名称的分组。<br>\n",
    "但是如果用python，则需要这样写:```(?P<groupname>正则表达式)```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " JPMorgan Insurance Trust Intrepid Mid Cap Portfolio \n"
     ]
    }
   ],
   "source": [
    "text = r'The JPMorgan Insurance Trust Intrepid Mid Cap Portfolio intends to liquidate.'\n",
    "keyword = 'The(?P<sharename>(.*)?)intends to liquidate'\n",
    "matchlist = getlistiterbyregex(keyword, text)\n",
    "groupname = 'sharename'\n",
    "groupnames = []\n",
    "for match in matchlist:\n",
    "    groupnames = match.groupdict().keys() if len(groupnames) == 0 else groupname\n",
    "    sharename = match.group(groupname) if groupname in groupnames else ''\n",
    "    print(sharename)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 通过或的方式达到匹配多种情况\n",
    "有时一段话的关键字，往往有多种时态，比如过去时，现在进行时等，那么只写一种情况，或许无法匹配多种情况。<br>\n",
    "可以通过或的方式解决这种问题。<br>\n",
    "如下面的例子，是为了获取merge相关的previous与after share name。<br>\n",
    "这个例子还可以达到演示通过多个分组，分别获取相应信息的目的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pendingdate is August 24, 2018\n",
      "previous is Class I shares of the Voya Multi-Manager Large Cap Core Portfolio \n",
      "after is Class I shares of the Voya Index Plus LargeCap Portfolio \n"
     ]
    }
   ],
   "source": [
    "text = r'''On March 15, 2018, the Board of Trustees of Voya Investors Trust approved a proposal to reorganize the Voya Multi-Manager Large Cap Core Portfolio (the “Reorganization”). \n",
    "Subject to shareholder approval, effective after the close of business on or about August 24, 2018 (the “Reorganization Date”), \n",
    "Class I shares of the Voya Multi-Manager Large Cap Core Portfolio (the “Merging Fund”) will be merged into \n",
    "Class I shares of the Voya Index Plus LargeCap Portfolio (the “Surviving Fund”).'''\n",
    "keyword = 'on or about[\\s]*?(?P<pendingdate>(january|february|march|april|may|june|july|august|september|october|november|december)[\\s]*[0-9]{1,2}[\\s]*,[\\s]*[0-9]{4})[\\s]*?\\((.*)?\\),[\\s]*?(?P<previous>(.*)?)\\(the(.*)?fund(”)?\\)[\\s]*?(.*)?merg(e|ed|ing) into[\\s]*?(?P<after>(.*)?)\\(the(.*)?fund(”)?\\)\\.'\n",
    "matchlist = getlistiterbyregex(keyword, text)\n",
    "\n",
    "groupnames = ['pendingdate', 'previous', 'after']\n",
    "for match in matchlist:\n",
    "    groupnamelist = match.groupdict().keys() if len(groupnames) == 0 else groupname\n",
    "    for groupname in groupnames:\n",
    "        print('{0} is {1}'.format(groupname, match.group(groupname) if groupname in groupnames else ''))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 先拿段落再拿明细的正则表达式写法\n",
    "有时候，我们会遇到更复杂的文档。<br>\n",
    "比如确定share name的段落，有明确的标识：UNDERLYING FUNDS:，但是share name是一个列表。<br>\n",
    "这个时候，一般的做法，是先将段落，确定下来。<br>\n",
    "然后再从段落里面，拿share name。<br>\n",
    "示例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                                                                            \n",
      " Franklin Founding Funds Allocation VIP Fund         Franklin Templeton Services, LLC             FTVIPT\n",
      " Franklin Income VIP Fund                            Franklin Advisers, Inc.                      FTVIPT\n",
      " SA Allocation Balanced Portfolio                    SunAmerica Asset Management, LLC             SST\n",
      " SA Allocation Growth Portfolio                      SunAmerica Asset Management, LLC             SST\n",
      " SA Allocation Moderate Portfolio                    SunAmerica Asset Management, LLC             SST\n",
      " SA Allocation Moderate Growth Portfolio             SunAmerica Asset Management, LLC             SST\n",
      " SA American Funds(R) Asset Allocation Portfolio     Capital Research and Management Company      SAST\n",
      " SA BlackRock Multi-Asset Income Portfolio           BlackRock Investment Management, LLC         AST\n",
      " SA Edge Asset Allocation Portfolio                  Principal Global Investors, LLC              AST\n",
      " SA JPMorgan Diversified Balanced Portfolio          J.P. Morgan Investment Management Inc.       SAST\n",
      " SA MFS Total Return Portfolio*                      Massachusetts Financial Services Company     SAST\n"
     ]
    }
   ],
   "source": [
    "text = r'''\n",
    "UNDERLYING FUNDS:                                    MANAGED BY:                                  TRUST\n",
    "--------------------------------------------------   ------------------------------------------   -------\n",
    "                                                                                            \n",
    " Franklin Founding Funds Allocation VIP Fund         Franklin Templeton Services, LLC             FTVIPT\n",
    " Franklin Income VIP Fund                            Franklin Advisers, Inc.                      FTVIPT\n",
    " SA Allocation Balanced Portfolio                    SunAmerica Asset Management, LLC             SST\n",
    " SA Allocation Growth Portfolio                      SunAmerica Asset Management, LLC             SST\n",
    " SA Allocation Moderate Portfolio                    SunAmerica Asset Management, LLC             SST\n",
    " SA Allocation Moderate Growth Portfolio             SunAmerica Asset Management, LLC             SST\n",
    " SA American Funds(R) Asset Allocation Portfolio     Capital Research and Management Company      SAST\n",
    " SA BlackRock Multi-Asset Income Portfolio           BlackRock Investment Management, LLC         AST\n",
    " SA Edge Asset Allocation Portfolio                  Principal Global Investors, LLC              AST\n",
    " SA JPMorgan Diversified Balanced Portfolio          J.P. Morgan Investment Management Inc.       SAST\n",
    " SA MFS Total Return Portfolio*                      Massachusetts Financial Services Company     SAST\n",
    "\n",
    "\n",
    "\n",
    "'''\n",
    "keyword = r'[\\n]UNDERLYING FUNDS(.*)[\\n]-{5,}(.*)?[\\n](?P<multiname>[\\s\\S]*?)[\\n][\\n]'\n",
    "matchlist = getlistiterbyregex(keyword, text)\n",
    "segment = ''\n",
    "for match in matchlist:\n",
    "    segment = match.group('multiname')\n",
    "    print(segment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color='red' size='5'><b>下面这个例子是通过贪婪匹配，直接获得share名称的方式，非常有用！</b></font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "share 1, name is:  Franklin Founding Funds Allocation VIP Fund\n",
      "share 2, name is:  Franklin Income VIP Fund\n",
      "share 3, name is:  SA Allocation Balanced Portfolio\n",
      "share 4, name is:  SA Allocation Growth Portfolio\n",
      "share 5, name is:  SA Allocation Moderate Portfolio\n",
      "share 6, name is:  SA Allocation Moderate Growth Portfolio\n",
      "share 7, name is:  SA American Funds(R) Asset Allocation Portfolio\n",
      "share 8, name is:  SA BlackRock Multi-Asset Income Portfolio\n",
      "share 9, name is:  SA Edge Asset Allocation Portfolio\n",
      "share 10, name is:  SA JPMorgan Diversified Balanced Portfolio\n",
      "share 11, name is:  SA MFS Total Return Portfolio*\n"
     ]
    }
   ],
   "source": [
    "if len(segment) > 0:\n",
    "#      keyword = r'[\\n](?P<sharename>(.*)?)[ ]{2,}'\n",
    "#     keyword = r'[\\n](?P<sharename>(.*)?(fund|portfolio)(\\*)?)'\n",
    "    # 如果想使用贪婪匹配，用(.*)?达不到效果，只能通过[\\s\\S]*?这种方式实现贪婪匹配\n",
    "    # 即遇到超过2个空格的地方，就停下来\n",
    "    keyword = r'[\\n](?P<sharename>[\\s\\S]*?)[ ]{2,}' \n",
    "    matchlist = getlistiterbyregex(keyword, segment)\n",
    "    for index, match in enumerate(matchlist):\n",
    "        print('share {0}, name is: {1}'.format(index + 1, match.group('sharename')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "另一个示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = r'''\n",
    "Variable Account Options\n",
    "--------------------------------------------------------------------------------\n",
    "\n",
    "\n",
    "\n",
    "                                     \n",
    "    VALIC Company I Funds               VALIC Company II Funds\n",
    "    Asset Allocation Fund               Aggressive Growth Lifestyle Fund\n",
    "    Blue Chip Growth Fund               Capital Appreciation Fund\n",
    "    Broad Cap Value Income Fund         Core Bond Fund\n",
    "    Capital Conservation Fund           Conservative Growth Lifestyle Fund\n",
    "    Core Equity Fund                    Government Money Market II Fund\n",
    "    Dividend Value Fund                 High Yield Bond Fund\n",
    "    Emerging Economies Fund             International Opportunities Fund\n",
    "    Foreign Value Fund                  Large Cap Value Fund\n",
    "    Global Real Estate Fund             Mid Cap Growth Fund\n",
    "    Global Social Awareness Fund        Mid Cap Value Fund\n",
    "    Global Strategy Fund                Moderate Growth Lifestyle Fund\n",
    "    Government Money Market I Fund      Small Cap Growth Fund\n",
    "    Government Securities Fund          Small Cap Value Fund\n",
    "    Growth Fund                         Socially Responsible Fund\n",
    "    Growth & Income Fund                Strategic Bond Fund\n",
    "    Health Sciences Fund\n",
    "    Inflation Protected Fund\n",
    "    International Equities Index Fund\n",
    "    International Government Bond Fund\n",
    "    International Growth Fund\n",
    "    Large Cap Core Fund\n",
    "    Large Capital Growth Fund\n",
    "    Mid Cap Index Fund\n",
    "    Mid Cap Strategic Growth Fund\n",
    "    Nasdaq-100(R) Index Fund\n",
    "    Science & Technology Fund\n",
    "    Small Cap Aggressive Growth Fund\n",
    "    Small Cap Fund\n",
    "    Small Cap Index Fund\n",
    "    Small Cap Special Values Fund\n",
    "    Small-Mid Growth Fund\n",
    "    Stock Index Fund\n",
    "    Value Fund\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "Table of Contents\n",
    "--------------------------------------------------------------------------------\n",
    "\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "                                     \n",
      "    VALIC Company I Funds               VALIC Company II Funds\n",
      "    Asset Allocation Fund               Aggressive Growth Lifestyle Fund\n",
      "    Blue Chip Growth Fund               Capital Appreciation Fund\n",
      "    Broad Cap Value Income Fund         Core Bond Fund\n",
      "    Capital Conservation Fund           Conservative Growth Lifestyle Fund\n",
      "    Core Equity Fund                    Government Money Market II Fund\n",
      "    Dividend Value Fund                 High Yield Bond Fund\n",
      "    Emerging Economies Fund             International Opportunities Fund\n",
      "    Foreign Value Fund                  Large Cap Value Fund\n",
      "    Global Real Estate Fund             Mid Cap Growth Fund\n",
      "    Global Social Awareness Fund        Mid Cap Value Fund\n",
      "    Global Strategy Fund                Moderate Growth Lifestyle Fund\n",
      "    Government Money Market I Fund      Small Cap Growth Fund\n",
      "    Government Securities Fund          Small Cap Value Fund\n",
      "    Growth Fund                         Socially Responsible Fund\n",
      "    Growth & Income Fund                Strategic Bond Fund\n",
      "    Health Sciences Fund\n",
      "    Inflation Protected Fund\n",
      "    International Equities Index Fund\n",
      "    International Government Bond Fund\n",
      "    International Growth Fund\n",
      "    Large Cap Core Fund\n",
      "    Large Capital Growth Fund\n",
      "    Mid Cap Index Fund\n",
      "    Mid Cap Strategic Growth Fund\n",
      "    Nasdaq-100(R) Index Fund\n",
      "    Science & Technology Fund\n",
      "    Small Cap Aggressive Growth Fund\n",
      "    Small Cap Fund\n",
      "    Small Cap Index Fund\n",
      "    Small Cap Special Values Fund\n",
      "    Small-Mid Growth Fund\n",
      "    Stock Index Fund\n",
      "    Value Fund\n",
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "keyword = r'[\\n]Variable Account Options[\\n]-{5,}(.*)?[\\n](?P<multiname>[\\s\\S]*?)[\\n][\\n]Table of Contents[\\n]'\n",
    "matchlist = getlistiterbyregex(keyword, text)\n",
    "segmentforname = ''\n",
    "for match in matchlist:\n",
    "    segmentforname = match.group('multiname')\n",
    "    print(segmentforname)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这个例子不仅将share name识别出来，而且将第一行作为share header进行拼接"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sharename is VALIC Company I Funds Asset Allocation Fund\n",
      "sharename is  VALIC Company II Funds  Aggressive Growth Lifestyle Fund\n",
      "sharename is VALIC Company I Funds Blue Chip Growth Fund\n",
      "sharename is  VALIC Company II Funds  Capital Appreciation Fund\n",
      "sharename is VALIC Company I Funds Broad Cap Value Income Fund\n",
      "sharename is  VALIC Company II Funds  Core Bond Fund\n",
      "sharename is VALIC Company I Funds Capital Conservation Fund\n",
      "sharename is  VALIC Company II Funds  Conservative Growth Lifestyle Fund\n",
      "sharename is VALIC Company I Funds Core Equity Fund\n",
      "sharename is  VALIC Company II Funds Government Money Market II Fund\n",
      "sharename is VALIC Company I Funds Dividend Value Fund\n",
      "sharename is  VALIC Company II Funds  High Yield Bond Fund\n",
      "sharename is VALIC Company I Funds Emerging Economies Fund\n",
      "sharename is  VALIC Company II Funds  International Opportunities Fund\n",
      "sharename is VALIC Company I Funds Foreign Value Fund\n",
      "sharename is  VALIC Company II Funds Large Cap Value Fund\n",
      "sharename is VALIC Company I Funds Global Real Estate Fund\n",
      "sharename is  VALIC Company II Funds  Mid Cap Growth Fund\n",
      "sharename is VALIC Company I Funds Global Social Awareness Fund\n",
      "sharename is  VALIC Company II Funds Mid Cap Value Fund\n",
      "sharename is VALIC Company I Funds Global Strategy Fund\n",
      "sharename is  VALIC Company II Funds Moderate Growth Lifestyle Fund\n",
      "sharename is VALIC Company I Funds Government Money Market I Fund\n",
      "sharename is  VALIC Company II Funds Small Cap Growth Fund\n",
      "sharename is VALIC Company I Funds Government Securities Fund\n",
      "sharename is  VALIC Company II Funds Small Cap Value Fund\n",
      "sharename is VALIC Company I Funds Growth Fund\n",
      "sharename is  VALIC Company II Funds  Socially Responsible Fund\n",
      "sharename is VALIC Company I Funds Growth & Income Fund\n",
      "sharename is  VALIC Company II Funds Strategic Bond Fund\n",
      "sharename is VALIC Company I Funds Health Sciences Fund\n",
      "sharename is VALIC Company I Funds Inflation Protected Fund\n",
      "sharename is VALIC Company I Funds International Equities Index Fund\n",
      "sharename is VALIC Company I Funds International Government Bond Fund\n",
      "sharename is VALIC Company I Funds International Growth Fund\n",
      "sharename is VALIC Company I Funds Large Cap Core Fund\n",
      "sharename is VALIC Company I Funds Large Capital Growth Fund\n",
      "sharename is VALIC Company I Funds Mid Cap Index Fund\n",
      "sharename is VALIC Company I Funds Mid Cap Strategic Growth Fund\n",
      "sharename is VALIC Company I Funds Nasdaq-100(R) Index Fund\n",
      "sharename is VALIC Company I Funds Science & Technology Fund\n",
      "sharename is VALIC Company I Funds Small Cap Aggressive Growth Fund\n",
      "sharename is VALIC Company I Funds Small Cap Fund\n",
      "sharename is VALIC Company I Funds Small Cap Index Fund\n",
      "sharename is VALIC Company I Funds Small Cap Special Values Fund\n",
      "sharename is VALIC Company I Funds Small-Mid Growth Fund\n",
      "sharename is VALIC Company I Funds Stock Index Fund\n",
      "sharename is VALIC Company I Funds Value Fund\n"
     ]
    }
   ],
   "source": [
    "if len(segmentforname) > 0:\n",
    "    keyword = '[\\n][ ]{2,}(?P<sharename>(.*)?)'\n",
    "    matchlist = getlistiterbyregex(keyword, segmentforname)\n",
    "    shareheaders = []\n",
    "    isfirstline = True\n",
    "    for match in matchlist:\n",
    "        shareinfo = match.group('sharename')\n",
    "        sharelist = [share for share in shareinfo.split('  ') if len(share.strip()) > 0]\n",
    "        for index, share in enumerate(sharelist):\n",
    "            if len(share.strip()) > 0:\n",
    "                if isfirstline:\n",
    "                    shareheaders.append(share)\n",
    "                else:\n",
    "                    sharename = '{0} {1}'.format(shareheaders[index], share)\n",
    "                    print('sharename is {0}'.format(sharename))\n",
    "        if len(shareheaders) > 0:\n",
    "            isfirstline = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 直接从文本文件做正则表达式匹配"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "this fund will be liquidating on or about june 28, 2018.\n",
      "\n",
      "if, pursuant to sec rules, an underlying money market fund suspends payment of redemption proceeds in connection with a liquidation of the fund, we will delay payment of any transfer, partial withdrawal, surrender, loan, or death benefit from the money market sub-account until the fund is liquidated. payment of contract proceeds from the fixed account may be delayed for up to six months.\n",
      "\n",
      "* the numbers of accumulation units less than 1000 were rounded up to one. 1 the jpmorgan insurance trust intrepid mid cap portfolio was liquidated on may 19, 2017. 2 the lvip aqr enhanced global strategies fund was liquidated on january 10, 2017. 3 on december 9, 2016, this subaccount was closed and the values were transferred to the lvip ssga international managed volatility fund subaccount. 4 the transparent value directional allocation vi portfolio was liquidated on june 6, 2016.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "filename = './docs/txt/168104986/1.txt'\n",
    "if os.path.exists(filename):\n",
    "    with open(filename, 'r', encoding='utf-8', errors='ignore') as txt:\n",
    "        text = txt.read().lower().replace('\\x00', '').strip()\n",
    "        keyword = '[\\n](.*)?(liquidat|transfer[\\s]*date)(.*)?'\n",
    "        matchlist = getlistiterbyregex(keyword, text)\n",
    "        for match in matchlist:\n",
    "            print(match.group(0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 通过正则表达式做结果替换\n",
    "有时，爬下来的文本，有很多特殊符号，或者不想要的部分，可以通过正则表达式相关函数进行批量替换"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 将不需要的特殊符号替换为空字符串"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This    is    test     text,    contain   part of   special     Characters (formerly,   its ok).   (the test text is     for    regex   training)\n"
     ]
    }
   ],
   "source": [
    "text = r'This    is    test  @  - text,    contain   part of  · special     Characters (formerly,   it''s ok).   (the test text is     for    regex   training)'\n",
    "keyword = '(@|-|·)'\n",
    "newtext = re.sub(keyword, '', text)\n",
    "print(newtext)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 去除不需要的括号部分\n",
    "<font color='red'><b>注意：这里依然用```[\\s\\S]*?```做贪婪匹配，用```(.*)?```做贪婪匹配会失败！！！</b></font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This    is    test     text,    contain   part of   special     Characters .   (the test text is     for    regex   training)\n"
     ]
    }
   ],
   "source": [
    "keyword = '\\(formerly[\\s\\S]*?\\)'\n",
    "newtext = re.sub(keyword, '', newtext)\n",
    "print(newtext)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 将2个或2个以上空格，替换为一个空格"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is test text, contain part of special Characters . (the test text is for regex training)\n"
     ]
    }
   ],
   "source": [
    "# 将2个或2个以上的空格，统一替换为一个空格\n",
    "keyword = '( ){2,}'\n",
    "newtext = re.sub(keyword, ' ', newtext)\n",
    "print(newtext)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以通过更通用的方式完成这个替换, 需要注意的是将多个空格替换为一个空格的动作，应该最后完成"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is test text, contain part of special Characters . (the test text is for regex training)\n"
     ]
    }
   ],
   "source": [
    "text = r'This    is    test  @  - text,    contain   part of  · special     Characters (formerly,   it''s ok).   (the test text is     for    regex   training)'\n",
    "# 将2个或2个以上的空格，统一替换为一个空格\n",
    "keywords = ['(@|-|·)', '\\(formerly[\\s\\S]*?\\)', '( ){2,}']\n",
    "for keyword in keywords:\n",
    "    text = re.sub(keyword, ' ', text)\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 性能考虑\n",
    "* 匹配一切字符并且贪婪模式，```[\\s\\S]*?```是性能非常好的\n",
    "* 如果确定是在一行内拿文本，并且不需要贪婪模式，```(.*)?```这种写法非常常见\n",
    "* 如果确定某文本关键字，必然是一段话的开头，那么使用这种方式，性能会较优：```[\\n]Effective date of(.*)?```\n",
    "* 如果重复的符号非常多，其实不需要完全把这些符号集合放在表达式里面。比如:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = '''\n",
    "Variable Account Options\n",
    "--------------------------------------------------------------------------------\n",
    "    VALIC Company I Funds               VALIC Company II Funds\n",
    "    Asset Allocation Fund               Aggressive Growth Lifestyle Fund\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果想确定Variable Account Options开头，然后取之后的share name，如下写法会显得简洁很多, 注意<b>-{10,}</b>的写法, 表示符号：-至少重复10次"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    VALIC Company I Funds               VALIC Company II Funds\n",
      "    Asset Allocation Fund               Aggressive Growth Lifestyle Fund\n",
      "\n"
     ]
    }
   ],
   "source": [
    "keyword = '[\\n]Variable Account Options[\\n]-{10,}[\\n](?P<multiname>[\\s\\S]*)'\n",
    "matchlist = getlistiterbyregex(keyword, text)\n",
    "for match in matchlist:\n",
    "        print(match.group('multiname'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}