notebook/DL_nlp_qa/BERT模型搭建中文问答系统.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# BERT 模型\n",
    "\n",
    "BERT（Bidirectional Encoder Representations from Transformers）是在2018年10月发布的论文[《BERT:Pre-training of Deep Bidirectional Transformers for Language Understanding》](https://arxiv.org/pdf/1810.04805.pdf)中提出。刚发布就刷新了自然语言处理任务的11项记录，到现在很多记录仍然是 BERT 的变种，比如行业内公认的机器阅读理解领域的顶级水平测试——[SQuAD（Stanford Question Answering Dataset）](https://rajpurkar.github.io/SQuAD-explorer/)，最高的成绩已经超过了人类的水平。截至2019年5月，SQuAD的榜单如下：\n",
    "\n",
    " ![squad榜单.png](./img/squad榜单.png)\n",
    "\n",
    "BERT 训练数据采用了英文的开源语料 BooksCropus 以及英文维基百科数据，一共有33亿个词，同时BERT模型的标准版本有1亿的参数量，而BERT的large版本有**3亿多参数量**。其团队训练一个预训练模型需要**在64块TPU芯片上训练4天**完成，而一块 TPU 的速度约是目前主流 GPU 的7-8倍。\n",
    "\n",
    "BERT的相关代码在[github](https://github.com/google-research/bert)，主要程序如下：\n",
    " \n",
    "![BERT图.png](./img/BERT图.png)\n",
    "\n",
    "模型开发团队开源了多个预训练模型，以供多种下游任务需求使用。开源的预训练模型如下：\n",
    "\n",
    "- [BERT-Base, Uncased](https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip): 12-layer, 768-hidden, 12-heads, 110M parameters\n",
    "\n",
    "\n",
    "- [BERT-Large, Uncased](https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-24_H-1024_A-16.zip): 24-layer, 1024-hidden, 16-heads, 340M parameters\n",
    "\n",
    "\n",
    "- [BERT-Base, Cased](https://storage.googleapis.com/bert_models/2018_10_18/cased_L-12_H-768_A-12.zip): 12-layer, 768-hidden, 12-heads , 110M parameters\n",
    "\n",
    "\n",
    "- [BERT-Large, Cased](https://storage.googleapis.com/bert_models/2018_10_18/cased_L-24_H-1024_A-16.zip): 24-layer, 1024-hidden, 16-heads, 340M parameters\n",
    "\n",
    "\n",
    "- [BERT-Base, Multilingual Cased (New, recommended)](https://storage.googleapis.com/bert_models/2018_11_23/multi_cased_L-12_H-768_A-12.zip): 104 languages, 12-layer, 768-hidden, 12-heads, 110M parameters\n",
    "\n",
    "\n",
    "- [BERT-Base, Multilingual Uncased (Orig, not recommended)](https://storage.googleapis.com/bert_models/2018_11_03/multilingual_L-12_H-768_A-12.zip)(Not recommended, use Multilingual Cased instead): 102 languages, 12-layer, 768-hidden, 12-heads, 110M parameters\n",
    "\n",
    "\n",
    "- [BERT-Base, Chinese](https://storage.googleapis.com/bert_models/2018_11_03/chinese_L-12_H-768_A-12.zip): Chinese Simplified and Traditional, 12-layer, 768-hidden, 12-heads, 110M parameters\n",
    "\n",
    "前4个是英文模型，Multilingual 是多语言模型，最后一个是中文模型。其中 Uncased 是字母全部转换成小写，而 Cased 是保留了大小写。\n",
    "\n",
    "### 预训练模型\n",
    "\n",
    "在本实践中研究中文自然语言处理，使用的是中文**BERT-Base,Chinese**预训练模型。从链接[BERT-Base, Chinese](https://storage.googleapis.com/bert_models/2018_11_03/chinese_L-12_H-768_A-12.zip)下载解压后得到如下预训练模型文件\n",
    "\n",
    "- bert_config.json\n",
    "- bert_model.ckpt.data-00000-of-00001\n",
    "- bert_model.ckpt.index\n",
    "- bert_model.ckpt.meta\n",
    "- vocab.txt\n",
    "\n",
    "其中\n",
    "\n",
    "- 一个config file(bert_config.json) ：存储预训练模型超参数\n",
    "- 三个tensorflow checkpoint (bert_model.ckpt) ：包含预训练模型checkpoint\n",
    "- 一个vocab文件(vocab.txt)：将WordPiece映射成word id  \n",
    "\n",
    "### 下游任务\n",
    "\n",
    "由于 BERT 强大的普适性，对于不同的下游任务，BERT 都可以通过改造输入输出来处理。\n",
    "\n",
    "![BERT下游.png](./img/BERT下游.png)\n",
    " \n",
    "- (a) 对于句子关系类任务：MultiNLI 文本蕴含识别（M推理出N，蕴含/矛盾/中立），QQP（文本匹配），QNLI（自然语言问题推理），STS-B（语义文本相似度1-5），MRPC（微软研究释义语料库，判断文本对语音信息是否等价）、RTE（同MNLI，小数据），SWAG（113k多项选择问题组成的数据集，涉及丰富的基础情境）。加上一个起始和终结符号，句子之间加个分隔符即可。对于输出来说，把第一个起始符号对应的Transformer最后一层位置上面串接一个softmax分类层即可。\n",
    "\n",
    "\n",
    "- (b) 对于分类问题：SST-2（斯坦福情感分类树），CoLA（语言可接受性预测）。只需要增加起始和终结符号，输出部分和句子关系判断任务类似改造。\n",
    "\n",
    "\n",
    "- (c) 对于阅读理解问题：SQuAD（斯坦福问答数据集）。就只需要加 softmax 输出层。\n",
    "\n",
    "\n",
    "- (d) 对于序列标注问题：CoNLL-2003 NER（命名实体识别数据）。输入部分和单句分类是一样的，只需要输出部分 Transformer 最后一层每个单词对应位置都进行分类即可。\n",
    "\n",
    "\n",
    "从这里可以看出，NLP四大类任务都可以比较方便地改造成 BERT 能够接受的方式，根据任务选择不同的预训练数据初始化 encoder 和 decoder 即可。这也就是 BERT 的最大的优点之一，几乎可以完成任何 NLP 的下游任务。\n",
    "\n",
    "### BERT输入处理\n",
    "\n",
    "BERT的输入部分是个线性序列，句子通过分隔符分割，句前和句后分别增加两个标识符号[CLS]和[SEP]。\n",
    "\n",
    "![BERT输入](./img/bert输入.png)\n",
    "\n",
    "- **token embeddings**：词向量，将各个词转换成固定维度的向量。在 BERT 中，每个词会被转换成768维的向量表示。输入文本在送入token embeddings 层之前要先进行 tokenization 处理。在目前的 BERT 模型中，文章作者还将英文词汇作进一步切割，划分为更细粒度的语义单位（WordPiece），例如：将 playing 分割为 play 和 ##ing。对于中文，目前作者尚未对输入文本进行分词，而是直接将单字作为构成文本的基本单位。\n",
    "\n",
    "\n",
    "- **position embeddings**：位置向量，表示位置信息，由于词顺序是很重要的特征，出现在文本不同位置的字/词所携带的语义信息存在差异（比如：“我爱你”和“你爱我”），因此，BERT模型对不同位置的字/词分别附加一个不同的向量以作区分。\n",
    "\n",
    "\n",
    "- **segment embeddings**：文本向量，用来区别两种句子，只有两种向量表示。前一个向量是把0赋给第一个句子中的各个 token, 后一个向量是把1赋给第二个句子中的各个 token。如果输入仅仅只有一个句子，那么它的 segment embedding 就是全0。\n",
    "\n",
    "最后，BERT 模型将字向量、文本向量和位置向量的加和作为模型输入。\n",
    "\n",
    "BERT 能够处理最长512个 token 的输入序列。论文作者通过让 BERT 在各个位置上学习一个向量表示来讲序列顺序的信息编码进来。这意味着 Position Embeddings layer 实际上就是一个大小为 (512, 768) 的 lookup 表，表的第一行是代表第一个序列的第一个位置，第二行代表序列的第二个位置，以此类推。因此，如果有这样两个句子“Hello world” 和“Hi there”, “Hello” 和“Hi”会由完全相同的 position embeddings，因为他们都是句子的第一个词。同理，“world” 和“there”也会有相同的 position embedding。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  问答系统\n",
    "\n",
    "在NLP（自然语言处理）领域，与我们生活息息相关的就是问答系统（QA），它是机器与人交互最常见的方式。先来目睹一下问答系统的风采吧：\n",
    "\n",
    "#### 大家最熟悉的手机语音助手\n",
    "\n",
    "![siri](./img/siri.png)\n",
    "\n",
    "#### 智能搜索平台\n",
    "\n",
    "![search](./img/search.png)\n",
    "\n",
    "以上这些可以进行开放域智能问答的问答系统或对话机器人，需要相当庞大的训练数据做基础，需要很复杂的语义理解和转化的实现过程。\n",
    "\n",
    "针对实际生活中的各个领域，可以依靠固定领域的数据库来进行特定问题的回答。例如电商智能客服、银行智能回答等，其实质都是问答系统的流程：\n",
    "\n",
    "- 首先对问题内容识别分析\n",
    "\n",
    "- 再基于先验知识，找到合适的回答\n",
    "\n",
    "我们今天就来学习基于知识库的问答系统构建。\n",
    "\n",
    "## 知识库\n",
    "\n",
    "“第74届奥斯卡金像奖的最佳影片是《美丽心灵》”\n",
    "\n",
    "“《鲁迅全集》这本书是1981年出版的”\n",
    "\n",
    "“乔丹出生在美国纽约布鲁克林”。\n",
    "\n",
    "以上就是三条**知识**，把大量的知识汇聚起来就成为了知识库。我们可以在wiki百科，百度百科等查阅到大量的知识。这些百科的知识组建形式是非结构化的自然语言，这样的组织方式很适合人们阅读，但并不适合计算机去处理。为了方便计算机的处理和理解，我们使用更加形式化、简洁化的方式去表示知识——**三元组（triple）**。知识库通常由大量的三元组组成。\n",
    "\n",
    "“第74届奥斯卡金像奖的最佳影片是《美丽心灵》” 可以用三元组表示为（第74届奥斯卡金像奖, 最佳影片, 美丽心灵）。\n",
    "\n",
    "这里通常把三元组理解为（实体 entity，关系属性 attribute，属性值 value），即“第74届奥斯卡金像奖”为实体, “最佳影片”为关系属性, “美丽心灵”为属性值。\n",
    "\n",
    "进一步的，如果我们把实体看作是结点，把实体关系（包括属性，类别等等）看作是一条边，那么包含了大量三元组的知识库就成为了一个庞大的知识图。如下即为一个小型知识图。\n",
    "\n",
    "![knowledge_graph](./img/knowledge_graph.png)\n",
    "\n",
    "知识图谱又是一个很复杂庞大的领域，近年来也越来越受到人们的重视，在此不进行具体的介绍，感兴趣的同学可以自行学习。\n",
    "\n",
    "\n",
    "## 数据集\n",
    "\n",
    "本教程使用的中文数据集来自NLPCC（自然语言处理与中文计算会议，The Conference on Natural Language Processing and Chinese Computing）2016中的[***Task5：Open Domain Question Answering***](http://tcci.ccf.org.cn/conference/2017/taskdata.php)数据集。数据集包含 14,609 个问答对的训练集和包含 9870 个问答对的测试集。并提供一个知识库，包含 6,502,738 个实体、587,875 个属性以及 43,063,796 个三元组。\n",
    "\n",
    "- 知识库：nlpcc-iccpol-2016.kbqa.kb\n",
    "- 训练集：nlpcc-iccpol-2016.kbqa.traing-data\n",
    "- 测试集：nlpcc-iccpol-2016.kbqa.testing-data\n",
    "\n",
    "知识库文件中每行存储一个事实（fact），即三元组 ( 实体、属性、属性值) 。知识库样例如下所示：\n",
    "\n",
    "```\n",
    "\"希望之星\"英语风采大赛|||中文名|||“希望之星”英语风采大赛\n",
    "\"希望之星\"英语风采大赛|||主办方|||中央电视台科教节目中心\n",
    "\"希望之星\"英语风采大赛|||别名|||\"希望之星\"英语风采大赛\n",
    "\"希望之星\"英语风采大赛|||外文名|||Star of Outlook English Talent Competition\n",
    "\"希望之星\"英语风采大赛|||开始时间|||1998\n",
    "\"希望之星\"英语风采大赛|||比赛形式|||全国选拔\n",
    "\"希望之星\"英语风采大赛|||节目类型|||英语比赛\n",
    "计算机应用基础 ||| 别名 ||| 计算机应用基础\n",
    "计算机应用基础 ||| 中文名 ||| 计算机应用基础\n",
    "计算机应用基础 ||| 作者 ||| 刘晓斌、魏智荣、刘庆生\n",
    "计算机应用基础 ||| 类别 ||| 计算机/网络 > 计算机理论\n",
    "计算机应用基础 ||| 字数 ||| 445000\n",
    "计算机应用基础 ||| ISBN ||| 9787122177513\n",
    "计算机应用基础 ||| 出版社 ||| 化学工业出版社\n",
    "计算机应用基础 ||| 页数 ||| 255\n",
    "计算机应用基础 ||| 开本 ||| 16开\n",
    "计算机应用基础 ||| 出版时间 ||| 2013年\n",
    "计算机应用基础 ||| 装帧 ||| 平装\n",
    "计算机应用基础 ||| 原作者 ||| 刘升贵，黄敏，庄强兵\n",
    "计算机应用基础 ||| 定价 ||| 29.00元\n",
    "```\n",
    "\n",
    "由于原数据集中知识库数据量庞大（2.3G），在本实战中提供的知识库文件仅为节选示例。\n",
    "\n",
    "训练集和测试集为经过[预处理](https://github.com/huangxiangzhou/NLPCC2016KBQA)包含三元组的数据集。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 准备源代码和数据\n",
    "\n",
    "准备案例所需的源代码和数据，相关资源已经保存在OBS中，我们通过ModelArts SDK将资源下载到本地。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully download file modelarts-labs/notebook/DL_nlp_qa/qa.tar.gz from OBS to local ./qa.tar.gz\n",
      "total 375464\r\n",
      "drwxrwxrwx  3 ma-user ma-group      4096 Sep 30 17:31 .\r\n",
      "drwsrwsr-x 20 ma-user ma-group      4096 Sep 30 17:17 ..\r\n",
      "drwxr-x---  2 ma-user ma-group      4096 Sep 30 17:17 .ipynb_checkpoints\r\n",
      "-rw-r-----  1 ma-user ma-group     91607 Sep 30 17:30 qa.ipynb\r\n",
      "-rw-r-----  1 ma-user ma-group 384366229 Sep 30 17:31 qa.tar.gz\r\n"
     ]
    }
   ],
   "source": [
    "from modelarts.session import Session\n",
    "session = Session()\n",
    "\n",
    "if session.region_name == 'cn-north-1':\n",
    "    bucket_path = 'modelarts-labs/notebook/DL_nlp_qa/qa.tar.gz'\n",
    "    \n",
    "elif session.region_name == 'cn-north-4':\n",
    "    bucket_path = 'modelarts-labs-bj4/notebook/DL_nlp_qa/qa.tar.gz'\n",
    "else:\n",
    "    print(\"请更换地区到北京一或北京四\")\n",
    "    \n",
    "session.download_data(bucket_path=bucket_path, path='./qa.tar.gz')\n",
    "\n",
    "!ls -la"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "解压从obs下载的压缩包，解压后删除压缩包。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 108\r\n",
      "drwxrwxrwx  4 ma-user ma-group  4096 Sep 30 17:31 .\r\n",
      "drwsrwsr-x 20 ma-user ma-group  4096 Sep 30 17:17 ..\r\n",
      "drwxr-x---  2 ma-user ma-group  4096 Sep 30 17:17 .ipynb_checkpoints\r\n",
      "drwxr-x---  8 ma-user ma-group  4096 Sep 30 17:15 qa\r\n",
      "-rw-r-----  1 ma-user ma-group 91607 Sep 30 17:30 qa.ipynb\r\n"
     ]
    }
   ],
   "source": [
    "!tar xf ./qa.tar.gz\n",
    "\n",
    "!rm ./qa.tar.gz\n",
    "\n",
    "!ls -la   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "基于知识库的问答（KBQA）拆分为两个主要步骤: **命名实体识别**步骤和**属性映射（文本相似度）**步骤。\n",
    "\n",
    "其中，实体识别步骤的目的是找到问句中询问的实体名称，而属性映射步骤的目的在于找到问句中询问的相关属性，即为文本相似度问题。\n",
    "\n",
    "下面分别进行这两个步骤。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 命名实体识别\n",
    "\n",
    "命名实体识别（Named EntitiesRecognition，NER）是NLP里的一项很基础的任务，就是指从文本中识别出命名性指称项，为信息抽取、信息检索、机器翻译、问答系统等任务做铺垫。其目的是识别语料中人名、地名、组织机构名等命名实体。在特定的领域中，会相应地定义领域内的各种实体类型。在本实践中，命名实体识别目的是找到问句中询问的**实体**名称。\n",
    "\n",
    "构造NER的数据集，需要根据三元组-Enitity 反向标注问题，给数据集中的Question 打标签。在这里采用BIO的标注方式，将每个元素标注为“B-X”、“I-X”或者“O”：\n",
    "\n",
    "- “B-X”（Begin）表示此元素所在的片段属于X类型并且此元素在此片段的开头；\n",
    "\n",
    "- “I-X”（Inside）表示此元素所在的片段属于X类型并且此元素在此片段的中间位置；\n",
    "\n",
    "- “O”（Outside）表示不属于任何类型。\n",
    "\n",
    "其中，“X”为“LOC”代表地名，“PER”代表人名，“ORG”代表组织机构名。\n",
    "\n",
    "由于本任务无需区分地点名、人名和组织名，只需要识别出实体，因此仅**使用B-LOC表示实体首字，I-LOC表示实体非首字**。\n",
    "\n",
    "使用本实践的数据集进行标注，标注前示例：\n",
    "\n",
    "```\n",
    "《机械设计基础》这本书的作者是谁？     机械设计基础\n",
    "\n",
    "《高等数学》是哪个出版社出版的？        高等数学\n",
    " ```\n",
    " \n",
    "标注后为：\n",
    "\n",
    "![ner_tagging](./img/ner_tagging.png)\n",
    "\n",
    "本教程已为开发者做好命名实体的数据标注工作，储存在`data/data_ner`文件夹下的`train.txt`中.\n",
    "\n",
    "执行下面程序，查看前50行标注。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "《 O\n",
      "\n",
      "机 B-LOC\n",
      "\n",
      "械 I-LOC\n",
      "\n",
      "设 I-LOC\n",
      "\n",
      "计 I-LOC\n",
      "\n",
      "基 I-LOC\n",
      "\n",
      "础 I-LOC\n",
      "\n",
      "》 O\n",
      "\n",
      "这 O\n",
      "\n",
      "本 O\n",
      "\n",
      "书 O\n",
      "\n",
      "的 O\n",
      "\n",
      "作 O\n",
      "\n",
      "者 O\n",
      "\n",
      "是 O\n",
      "\n",
      "谁 O\n",
      "\n",
      "？ O\n",
      "\n",
      "   \n",
      "\n",
      "《 O\n",
      "\n",
      "高 B-LOC\n",
      "\n",
      "等 I-LOC\n",
      "\n",
      "数 I-LOC\n",
      "\n",
      "学 I-LOC\n",
      "\n",
      "》 O\n",
      "\n",
      "是 O\n",
      "\n",
      "哪 O\n",
      "\n",
      "个 O\n",
      "\n",
      "出 O\n",
      "\n",
      "版 O\n",
      "\n",
      "社 O\n",
      "\n",
      "出 O\n",
      "\n",
      "版 O\n",
      "\n",
      "的 O\n",
      "\n",
      "？ O\n",
      "\n",
      "   \n",
      "\n",
      "《 O\n",
      "\n",
      "线 B-LOC\n",
      "\n",
      "性 I-LOC\n",
      "\n",
      "代 I-LOC\n",
      "\n",
      "数 I-LOC\n",
      "\n",
      "》 O\n",
      "\n",
      "这 O\n",
      "\n",
      "本 O\n",
      "\n",
      "书 O\n",
      "\n",
      "的 O\n",
      "\n",
      "出 O\n",
      "\n",
      "版 O\n",
      "\n",
      "时 O\n",
      "\n",
      "间 O\n",
      "\n",
      "是 O\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open(\"./qa/data/data_ner/train.txt\", \"r\") as f:\n",
    "    d = f.readlines()\n",
    "    for i in range(50):\n",
    "        print(d[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在本教程中使用**BERT**模型，并用 **BiLSTM+CRF** 作为下游模型，BiLSTM（Bi-directional Long Short-Term Memory）来规范处理输入输出，\n",
    "CRF 为最后预测的标签添加一些约束来保证预测的标签合法。完整的程序封装在`bert_ner.py`文件中，在这里不深入代码细节。\n",
    "\n",
    "参数设置如下：\n",
    "\n",
    "```\n",
    "max_seq_length = 128          #序列最大长度\n",
    "train_batch_size = 64         #训练批尺寸\n",
    "predict_batch_size = 64        #测试批尺寸\n",
    "learning_rate = 5e-5          #学习率\n",
    "num_train_epochs = 5.0         #训练轮数\n",
    "dropout_rate = 1             #随机失活率\n",
    "warmup_proportion = 0.1        #预热训练\n",
    "```\n",
    "\n",
    "执行下面程序块运行，完成命名实体识别的训练和测试。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7f8dc18b8598>) includes params argument, but params are not passed to Estimator.\n",
      "INFO:tensorflow:Using config: {'_model_dir': './qa/output_ner', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true\n",
      "graph_options {\n",
      "  rewrite_options {\n",
      "    meta_optimizer_iterations: ONE\n",
      "  }\n",
      "}\n",
      ", '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f8dc15c8940>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None), '_cluster': None}\n",
      "INFO:tensorflow:_TPUContext: eval_on_tpu True\n",
      "WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.\n",
      "INFO:tensorflow:Writing example 0 of 13637\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: train-0\n",
      "INFO:tensorflow:tokens: 《 机 械 设 计 基 础 》 这 本 书 的 作 者 是 谁 ？\n",
      "INFO:tensorflow:input_ids: 101 517 3322 3462 6392 6369 1825 4794 518 6821 3315 741 4638 868 5442 3221 6443 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label_ids: 9 1 6 7 7 7 7 7 1 1 1 1 1 1 1 1 1 1 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: train-1\n",
      "INFO:tensorflow:tokens: 《 高 等 数 学 》 是 哪 个 出 版 社 出 版 的 ？\n",
      "INFO:tensorflow:input_ids: 101 517 7770 5023 3144 2110 518 3221 1525 702 1139 4276 4852 1139 4276 4638 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label_ids: 9 1 6 7 7 7 1 1 1 1 1 1 1 1 1 1 1 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: train-2\n",
      "INFO:tensorflow:tokens: 《 线 性 代 数 》 这 本 书 的 出 版 时 间 是 什 么 ？\n",
      "INFO:tensorflow:input_ids: 101 517 5296 2595 807 3144 518 6821 3315 741 4638 1139 4276 3198 7313 3221 784 720 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label_ids: 9 1 6 7 7 7 1 1 1 1 1 1 1 1 1 1 1 1 1 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: train-3\n",
      "INFO:tensorflow:tokens: 安 德 烈 是 哪 个 国 家 的 人 呢 ？\n",
      "INFO:tensorflow:input_ids: 101 2128 2548 4164 3221 1525 702 1744 2157 4638 782 1450 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label_ids: 9 6 7 7 1 1 1 1 1 1 1 1 1 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: train-4\n",
      "INFO:tensorflow:tokens: 《 线 性 代 数 》 的 i s b n 码 是 什 么 ？\n",
      "INFO:tensorflow:input_ids: 101 517 5296 2595 807 3144 518 4638 151 161 144 156 4772 3221 784 720 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label_ids: 9 1 6 7 7 7 1 1 1 1 1 1 1 1 1 1 1 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "O B-LOC I-LOC I-LOC I-LOC I-LOC I-LOC O O O O O O O O O O\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Writing example 5000 of 13637\n",
      "INFO:tensorflow:Writing example 10000 of 13637\n",
      "INFO:tensorflow:***** Running training *****\n",
      "INFO:tensorflow:  Num examples = 13637\n",
      "INFO:tensorflow:  Batch size = 64\n",
      "INFO:tensorflow:  Num steps = 1065\n",
      "WARNING:tensorflow:From /home/ma-user/anaconda3/envs/TensorFlow-1.13.1/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "WARNING:tensorflow:From /home/ma-user/work/qa/bert_ner.py:355: map_and_batch (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.experimental.map_and_batch(...)`.\n",
      "WARNING:tensorflow:From /home/ma-user/work/qa/bert_ner.py:342: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Running train on CPU\n",
      "WARNING:tensorflow:From /home/ma-user/work/qa/bert/modeling.py:358: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
      "WARNING:tensorflow:From /home/ma-user/work/qa/bert/modeling.py:671: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.dense instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape of input_ids (64, 128)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/ma-user/anaconda3/envs/TensorFlow-1.13.1/lib/python3.6/site-packages/tensorflow/contrib/crf/python/ops/crf.py:213: dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `keras.layers.RNN(cell)`, which is equivalent to this API\n",
      "WARNING:tensorflow:From /home/ma-user/anaconda3/envs/TensorFlow-1.13.1/lib/python3.6/site-packages/tensorflow/python/training/learning_rate_decay_v2.py:321: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Deprecated in favor of operator or tf.math.divide.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 0 into ./qa/output_ner/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 1.0474\n",
      "INFO:tensorflow:examples/sec: 67.0333\n",
      "INFO:tensorflow:global_step/sec: 1.12179\n",
      "INFO:tensorflow:examples/sec: 71.7943\n",
      "INFO:tensorflow:global_step/sec: 1.12324\n",
      "INFO:tensorflow:examples/sec: 71.8874\n",
      "INFO:tensorflow:global_step/sec: 1.12284\n",
      "INFO:tensorflow:examples/sec: 71.8615\n",
      "INFO:tensorflow:global_step/sec: 1.12351\n",
      "INFO:tensorflow:examples/sec: 71.9046\n",
      "INFO:tensorflow:global_step/sec: 1.12301\n",
      "INFO:tensorflow:examples/sec: 71.8723\n",
      "INFO:tensorflow:Saving checkpoints for 657 into ./qa/output_ner/model.ckpt.\n",
      "WARNING:tensorflow:From /home/ma-user/anaconda3/envs/TensorFlow-1.13.1/lib/python3.6/site-packages/tensorflow/python/training/saver.py:966: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file APIs to delete files with this prefix.\n",
      "INFO:tensorflow:global_step/sec: 1.08961\n",
      "INFO:tensorflow:examples/sec: 69.7349\n",
      "INFO:tensorflow:global_step/sec: 1.12222\n",
      "INFO:tensorflow:examples/sec: 71.8219\n",
      "INFO:tensorflow:global_step/sec: 1.1238\n",
      "INFO:tensorflow:examples/sec: 71.9232\n",
      "INFO:tensorflow:global_step/sec: 1.12346\n",
      "INFO:tensorflow:examples/sec: 71.9013\n",
      "INFO:tensorflow:Saving checkpoints for 1065 into ./qa/output_ner/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 13.384634.\n",
      "INFO:tensorflow:training_loop marked as finished\n",
      "INFO:tensorflow:Writing example 0 of 9016\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: 你 知 道 计 算 机 应 用 基 础 这 本 书 的 作 者 是 谁 吗 ？\n",
      "INFO:tensorflow:input_ids: 101 872 4761 6887 6369 5050 3322 2418 4500 1825 4794 6821 3315 741 4638 868 5442 3221 6443 1408 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label_ids: 9 1 1 1 6 7 7 7 7 7 7 1 1 1 1 1 1 1 1 1 1 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-1\n",
      "INFO:tensorflow:tokens: 计 算 机 应 用 基 础 这 本 书 的 出 版 社 是 那 个 ？\n",
      "INFO:tensorflow:input_ids: 101 6369 5050 3322 2418 4500 1825 4794 6821 3315 741 4638 1139 4276 4852 3221 6929 702 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label_ids: 9 6 7 7 7 7 7 7 1 1 1 1 1 1 1 1 1 1 1 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-2\n",
      "INFO:tensorflow:tokens: 告 诉 我 高 等 数 学 的 出 版 时 间 是 什 么 时 候 ？\n",
      "INFO:tensorflow:input_ids: 101 1440 6401 2769 7770 5023 3144 2110 4638 1139 4276 3198 7313 3221 784 720 3198 952 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label_ids: 9 1 1 1 6 7 7 7 1 1 1 1 1 1 1 1 1 1 1 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-3\n",
      "INFO:tensorflow:tokens: 我 想 知 道 戴 维 斯 是 什 么 国 家 的 人 ？\n",
      "INFO:tensorflow:input_ids: 101 2769 2682 4761 6887 2785 5335 3172 3221 784 720 1744 2157 4638 782 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label_ids: 9 1 1 1 1 6 7 7 1 1 1 1 1 1 1 1 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-4\n",
      "INFO:tensorflow:tokens: 你 知 道 高 等 数 学 的 i s b n 吗 ？\n",
      "INFO:tensorflow:input_ids: 101 872 4761 6887 7770 5023 3144 2110 4638 151 161 144 156 1408 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label_ids: 9 1 1 1 6 7 7 7 1 1 1 1 1 1 1 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "O O O B-LOC I-LOC I-LOC I-LOC I-LOC I-LOC I-LOC O O O O O O O O O O\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Writing example 5000 of 9016\n",
      "INFO:tensorflow:***** Running prediction*****\n",
      "INFO:tensorflow:  Num examples = 9016\n",
      "INFO:tensorflow:  Batch size = 64\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Running infer on CPU\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape of input_ids (?, 128)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "WARNING:tensorflow:From /home/ma-user/anaconda3/envs/TensorFlow-1.13.1/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file APIs to check for files with this prefix.\n",
      "INFO:tensorflow:Restoring parameters from ./qa/output_ner/model.ckpt-1065\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:prediction_loop marked as finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "打印测试评价指标\n",
      "processed 145137 tokens with 9016 phrases; found: 9050 phrases; correct: 8639.\n",
      "accuracy:  99.43%; precision:  95.46%; recall:  95.82%; FB1:  95.64\n",
      "              LOC: precision:  95.46%; recall:  95.82%; FB1:  95.64  9050\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#进行命名实体识别的训练、验证和测试\n",
    "from qa import bert_ner\n",
    "\n",
    "bert_ner.main(_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "至此，问答系统构建的前半部分已经完成，接下来进行后半部分的工作——属性映射（文本相似度）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 属性映射（文本相似度）\n",
    "\n",
    "属性映射目的在于找到问句中询问的相关属性，即为文本相似度问题。\n",
    "\n",
    "在自然语言处理（NLP）过程中，经常会涉及到如何度量两个文本之间的相似性。文本相似度（Semantic Text Similarity）计算就是判断两段文本之间是否相似。文本相似度技术可以应用到信息检索、自动问答、机器翻译和自动文摘等NLP任务中。\n",
    "\n",
    "中文相似度按照长度可以有字与字的相似度、单词与单词的相似度、句子与句子的相似度、段落与段落的相似度和文章与文章的相似度。\n",
    "\n",
    "计算文本相似度的算法有以下3类：\n",
    "\n",
    "* 基于具体字符\n",
    "* 基于概率统计\n",
    "* 基于词嵌入\n",
    "\n",
    "基于词嵌入的方法是指将文本中的单词和上下文信息嵌入到向量中，将向量作为神经网络的输入，然后训练出一个文本相似度预测模型。本案例就是一种基于词嵌入的中文短句文本相似度计算方法。\n",
    "\n",
    "相似度有多种粒度的表示方式，可以使用（1,2,3,4,5）中的一个数字表示相似度，值越大表示越相似；也可以使用（0,1）中的一个数字表示相似度，1表示相似，0表示不相似。在本案例中，我们使用（0,1）表示相似度。\n",
    "\n",
    "下面为两个文本相似度的示例：\n",
    "\n",
    "`\n",
    "句子1: 我曾经帮这位教授整理过稿子。\n",
    "`\n",
    "\n",
    "`\n",
    "句子2: 这位教授的稿子我帮着整理过。\n",
    "`\n",
    "\n",
    "`\n",
    "相似度是：0.9986 ——> 即认为相似\n",
    "`\n",
    "\n",
    "`\n",
    "句子1: 我习惯喝咖啡不放糖。\n",
    "`\n",
    "\n",
    "`\n",
    "句子2: 他边打工挣学费边上学。\n",
    "`\n",
    "\n",
    "`\n",
    "相似度是：0.0003 ——> 即认为不相似\n",
    "`\n",
    "\n",
    "\n",
    "构造用于文本相似度分析的训练集和测试集，构造测试集的整体关系集合，通过提取和去重，获得若干关系集合；每个sample由“问题句+关系属性+label”构成，原始数据中的关系属性的label为 1；从关系集合中随机采样五个属性作为 Negative Samples，label为0。\n",
    "\n",
    "本案例已为开发者做好属性映射（文本相似度）的数据处理，储存在`data/data_sim`文件夹下的`train.txt`中。\n",
    "\n",
    "打印训练集前50行的问题、三元组、答案和属性，示例如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12005\t请问有没有其他出版社出版了东京暗鸦？\t信噪比\t0\n",
      "\n",
      "12006\t请问有没有其他出版社出版了东京暗鸦？\t职位\t0\n",
      "\n",
      "12007\t请问有没有其他出版社出版了东京暗鸦？\t生产年份\t0\n",
      "\n",
      "12008\t请问草根金融论坛的联系方式是什么？\t联络方式\t1\n",
      "\n",
      "12009\t请问草根金融论坛的联系方式是什么？\t设计\t0\n",
      "\n",
      "12010\t请问草根金融论坛的联系方式是什么？\t建置时间\t0\n",
      "\n",
      "12011\t请问草根金融论坛的联系方式是什么？\t原料\t0\n",
      "\n",
      "12012\t请问草根金融论坛的联系方式是什么？\t人口总数\t0\n",
      "\n",
      "12013\t请问草根金融论坛的联系方式是什么？\t产生\t0\n",
      "\n",
      "12014\t请问足跟骨刺是否被纳入了医保体系?\t是否进入医保\t1\n",
      "\n",
      "12015\t请问足跟骨刺是否被纳入了医保体系?\t外观设计\t0\n",
      "\n",
      "12016\t请问足跟骨刺是否被纳入了医保体系?\t初次登场\t0\n",
      "\n",
      "12017\t请问足跟骨刺是否被纳入了医保体系?\t定名人\t0\n",
      "\n",
      "12018\t请问足跟骨刺是否被纳入了医保体系?\tt(厚度)\t0\n",
      "\n",
      "12019\t请问足跟骨刺是否被纳入了医保体系?\t友好城市\t0\n",
      "\n",
      "12020\t请问西北大学的研究生有多少人？\t研究生\t1\n",
      "\n",
      "12021\t请问西北大学的研究生有多少人？\tcpu主频\t0\n",
      "\n",
      "12022\t请问西北大学的研究生有多少人？\t最大输出功率\t0\n",
      "\n",
      "12023\t请问西北大学的研究生有多少人？\t定价\t0\n",
      "\n",
      "12024\t请问西北大学的研究生有多少人？\t注释\t0\n",
      "\n",
      "12025\t请问西北大学的研究生有多少人？\t放大倍率\t0\n",
      "\n",
      "12026\t请问生地饴糖鸡属于什么食物？\t所属类型\t1\n",
      "\n",
      "12027\t请问生地饴糖鸡属于什么食物？\t副职\t0\n",
      "\n",
      "12028\t请问生地饴糖鸡属于什么食物？\tcpu主频\t0\n",
      "\n",
      "12029\t请问生地饴糖鸡属于什么食物？\t青年队\t0\n",
      "\n",
      "12030\t请问生地饴糖鸡属于什么食物？\t主板架构\t0\n",
      "\n",
      "12031\t请问生地饴糖鸡属于什么食物？\t配 音\t0\n",
      "\n",
      "12032\t请问哪个公司出版了《忘忧草》啊？\t音乐公司\t1\n",
      "\n",
      "12033\t请问哪个公司出版了《忘忧草》啊？\t转职业年\t0\n",
      "\n",
      "12034\t请问哪个公司出版了《忘忧草》啊？\t网站性质\t0\n",
      "\n",
      "12035\t请问哪个公司出版了《忘忧草》啊？\t专长\t0\n",
      "\n",
      "12036\t请问哪个公司出版了《忘忧草》啊？\t领导者\t0\n",
      "\n",
      "12037\t请问哪个公司出版了《忘忧草》啊？\t使用平台\t0\n",
      "\n",
      "12038\t请问多花秋海棠属于什么亚组？\t亚组\t1\n",
      "\n",
      "12039\t请问多花秋海棠属于什么亚组？\t产品天线\t0\n",
      "\n",
      "12040\t请问多花秋海棠属于什么亚组？\t人口（2005）\t0\n",
      "\n",
      "12041\t请问多花秋海棠属于什么亚组？\t项目占地\t0\n",
      "\n",
      "12042\t请问多花秋海棠属于什么亚组？\t功能主治\t0\n",
      "\n",
      "12043\t请问多花秋海棠属于什么亚组？\t应用范围\t0\n",
      "\n",
      "12044\t请问广州恒大淘宝足球俱乐部的总经理是谁啊？\t总经理\t1\n",
      "\n",
      "12045\t请问广州恒大淘宝足球俱乐部的总经理是谁啊？\t制作人\t0\n",
      "\n",
      "12046\t请问广州恒大淘宝足球俱乐部的总经理是谁啊？\t属\t0\n",
      "\n",
      "12047\t请问广州恒大淘宝足球俱乐部的总经理是谁啊？\t注册时间\t0\n",
      "\n",
      "12048\t请问广州恒大淘宝足球俱乐部的总经理是谁啊？\t性状\t0\n",
      "\n",
      "12049\t请问广州恒大淘宝足球俱乐部的总经理是谁啊？\t瞄准具型式\t0\n",
      "\n",
      "12050\t请问spark乐驰的驱动方式是什么？\t驱动方式\t1\n",
      "\n",
      "12051\t请问spark乐驰的驱动方式是什么？\t科 室\t0\n",
      "\n",
      "12052\t请问spark乐驰的驱动方式是什么？\t剧本\t0\n",
      "\n",
      "12053\t请问spark乐驰的驱动方式是什么？\t技术人员\t0\n",
      "\n",
      "12054\t请问spark乐驰的驱动方式是什么？\t执行标准\t0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open(\"./qa/data/data_sim/train.txt\", \"r\") as f:\n",
    "    d = f.readlines()\n",
    "    for i in range(50):\n",
    "        print(d[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面进行属性映射（文本相似度）的训练和测试，同样使用**BERT**模型完成，封装到`bert_similarity.py`文件中，在这里不深入代码细节。\n",
    "\n",
    "参数设置如下：\n",
    "\n",
    "```\n",
    "num_train_epochs = 5        #训练轮数\n",
    "batch_size = 64           #批尺寸\n",
    "learning_rate = 0.00005      #学习率\n",
    "gpu_memory_fraction = 0.9     #gpu使用率\n",
    "layer_indexes = [-2]        #句向量选取\n",
    "max_seq_len = 64          #序列最大长度\n",
    "```\n",
    "\n",
    "执行下面程序块运行，完成属性映射（文本相似度）的训练和测试。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using config: {'_model_dir': './qa/output_sim/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 0.9\n",
      "  allow_growth: true\n",
      "}\n",
      ", '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f8dc19d62e8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "INFO:tensorflow:Using config: {'_model_dir': './qa/output_sim/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 0.9\n",
      "  allow_growth: true\n",
      "}\n",
      ", '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f8e5f289b70>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "INFO:tensorflow:Writing example 0 of 75621\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: train-0\n",
      "INFO:tensorflow:tokens: [CLS] 请 问 有 没 有 其 他 出 版 社 出 版 了 东 京 暗 鸦 ？ [SEP] 信 噪 比 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6435 7309 3300 3766 3300 1071 800 1139 4276 4852 1139 4276 749 691 776 3266 7887 8043 102 928 1692 3683 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: train-1\n",
      "INFO:tensorflow:tokens: [CLS] 请 问 有 没 有 其 他 出 版 社 出 版 了 东 京 暗 鸦 ？ [SEP] 职 位 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6435 7309 3300 3766 3300 1071 800 1139 4276 4852 1139 4276 749 691 776 3266 7887 8043 102 5466 855 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: train-2\n",
      "INFO:tensorflow:tokens: [CLS] 请 问 有 没 有 其 他 出 版 社 出 版 了 东 京 暗 鸦 ？ [SEP] 生 产 年 份 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6435 7309 3300 3766 3300 1071 800 1139 4276 4852 1139 4276 749 691 776 3266 7887 8043 102 4495 772 2399 819 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: train-3\n",
      "INFO:tensorflow:tokens: [CLS] 请 问 草 根 金 融 论 坛 的 联 系 方 式 是 什 么 ？ [SEP] 联 络 方 式 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6435 7309 5770 3418 7032 6084 6389 1781 4638 5468 5143 3175 2466 3221 784 720 8043 102 5468 5317 3175 2466 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 1 (id = 1)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: train-4\n",
      "INFO:tensorflow:tokens: [CLS] 请 问 草 根 金 融 论 坛 的 联 系 方 式 是 什 么 ？ [SEP] 设 计 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6435 7309 5770 3418 7032 6084 6389 1781 4638 5468 5143 3175 2466 3221 784 720 8043 102 6392 6369 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:Writing example 10000 of 75621\n",
      "INFO:tensorflow:Writing example 20000 of 75621\n",
      "INFO:tensorflow:Writing example 30000 of 75621\n",
      "INFO:tensorflow:Writing example 40000 of 75621\n",
      "INFO:tensorflow:Writing example 50000 of 75621\n",
      "INFO:tensorflow:Writing example 60000 of 75621\n",
      "INFO:tensorflow:Writing example 70000 of 75621\n",
      "INFO:tensorflow:***** Running training *****\n",
      "INFO:tensorflow:  Num examples = 75621\n",
      "INFO:tensorflow:  Batch size = 64\n",
      "INFO:tensorflow:  Num steps = 5907\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 0 into ./qa/output_sim/model.ckpt.\n",
      "INFO:tensorflow:loss = 0.80406123, step = 0\n",
      "INFO:tensorflow:global_step/sec: 1.56336\n",
      "INFO:tensorflow:loss = 0.12109105, step = 100 (63.966 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17991\n",
      "INFO:tensorflow:loss = 0.014379987, step = 200 (45.873 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18049\n",
      "INFO:tensorflow:loss = 0.037250243, step = 300 (45.861 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17971\n",
      "INFO:tensorflow:loss = 0.004695637, step = 400 (45.878 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17942\n",
      "INFO:tensorflow:loss = 0.0020469145, step = 500 (45.884 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17993\n",
      "INFO:tensorflow:loss = 0.0147804385, step = 600 (45.873 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17918\n",
      "INFO:tensorflow:loss = 0.00692276, step = 700 (45.889 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18035\n",
      "INFO:tensorflow:loss = 0.04618354, step = 800 (45.864 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18008\n",
      "INFO:tensorflow:loss = 0.019808413, step = 900 (45.870 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.1802\n",
      "INFO:tensorflow:loss = 0.0062913853, step = 1000 (45.867 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18033\n",
      "INFO:tensorflow:loss = 0.0133275, step = 1100 (45.864 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17894\n",
      "INFO:tensorflow:loss = 0.037112594, step = 1200 (45.894 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1253 into ./qa/output_sim/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 1.72351\n",
      "INFO:tensorflow:loss = 0.12576269, step = 1300 (58.021 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17812\n",
      "INFO:tensorflow:loss = 0.07432252, step = 1400 (45.911 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17993\n",
      "INFO:tensorflow:loss = 0.18995932, step = 1500 (45.873 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17927\n",
      "INFO:tensorflow:loss = 0.0027036057, step = 1600 (45.887 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18044\n",
      "INFO:tensorflow:loss = 0.0038441897, step = 1700 (45.862 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.181\n",
      "INFO:tensorflow:loss = 0.014303254, step = 1800 (45.851 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.1825\n",
      "INFO:tensorflow:loss = 0.00049416925, step = 1900 (45.819 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18204\n",
      "INFO:tensorflow:loss = 0.056986697, step = 2000 (45.829 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.1825\n",
      "INFO:tensorflow:loss = 0.11168129, step = 2100 (45.819 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 2.18275\n",
      "INFO:tensorflow:loss = 0.00057690404, step = 2200 (45.814 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18302\n",
      "INFO:tensorflow:loss = 0.00024280635, step = 2300 (45.808 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18192\n",
      "INFO:tensorflow:loss = 0.0028881684, step = 2400 (45.831 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18054\n",
      "INFO:tensorflow:loss = 0.006113374, step = 2500 (45.860 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 2536 into ./qa/output_sim/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 1.68529\n",
      "INFO:tensorflow:loss = 0.0013344908, step = 2600 (59.337 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18049\n",
      "INFO:tensorflow:loss = 0.0038962504, step = 2700 (45.861 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18183\n",
      "INFO:tensorflow:loss = 0.006715819, step = 2800 (45.834 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17844\n",
      "INFO:tensorflow:loss = 0.055857826, step = 2900 (45.904 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.1794\n",
      "INFO:tensorflow:loss = 0.00040053533, step = 3000 (45.884 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18079\n",
      "INFO:tensorflow:loss = 0.0020485804, step = 3100 (45.855 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18134\n",
      "INFO:tensorflow:loss = 0.00025651336, step = 3200 (45.844 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18023\n",
      "INFO:tensorflow:loss = 0.00030929683, step = 3300 (45.867 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18217\n",
      "INFO:tensorflow:loss = 0.0010573445, step = 3400 (45.827 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.1811\n",
      "INFO:tensorflow:loss = 0.00039029078, step = 3500 (45.848 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.1793\n",
      "INFO:tensorflow:loss = 0.0008383829, step = 3600 (45.886 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17883\n",
      "INFO:tensorflow:loss = 0.007717145, step = 3700 (45.896 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18169\n",
      "INFO:tensorflow:loss = 0.00049783796, step = 3800 (45.836 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 3815 into ./qa/output_sim/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 1.72628\n",
      "INFO:tensorflow:loss = 0.0031695478, step = 3900 (57.928 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18009\n",
      "INFO:tensorflow:loss = 0.00032910623, step = 4000 (45.869 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18109\n",
      "INFO:tensorflow:loss = 0.00077105424, step = 4100 (45.849 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18079\n",
      "INFO:tensorflow:loss = 0.00023945066, step = 4200 (45.855 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18095\n",
      "INFO:tensorflow:loss = 8.450498e-05, step = 4300 (45.852 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17976\n",
      "INFO:tensorflow:loss = 0.0001537561, step = 4400 (45.876 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17887\n",
      "INFO:tensorflow:loss = 0.00019694449, step = 4500 (45.895 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18123\n",
      "INFO:tensorflow:loss = 0.0006497489, step = 4600 (45.846 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18059\n",
      "INFO:tensorflow:loss = 0.00040960687, step = 4700 (45.859 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.17921\n",
      "INFO:tensorflow:loss = 0.00014905719, step = 4800 (45.888 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.1773\n",
      "INFO:tensorflow:loss = 0.00028763563, step = 4900 (45.928 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18012\n",
      "INFO:tensorflow:loss = 0.00081312575, step = 5000 (45.869 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 5097 into ./qa/output_sim/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 1.71021\n",
      "INFO:tensorflow:loss = 0.000552332, step = 5100 (58.472 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18294\n",
      "INFO:tensorflow:loss = 0.00016604707, step = 5200 (45.810 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18256\n",
      "INFO:tensorflow:loss = 0.00019858233, step = 5300 (45.818 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.1825\n",
      "INFO:tensorflow:loss = 7.9740086e-05, step = 5400 (45.819 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18086\n",
      "INFO:tensorflow:loss = 0.0010153695, step = 5500 (45.853 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18148\n",
      "INFO:tensorflow:loss = 0.00014651103, step = 5600 (45.840 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18126\n",
      "INFO:tensorflow:loss = 7.378128e-05, step = 5700 (45.845 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18139\n",
      "INFO:tensorflow:loss = 0.0005076176, step = 5800 (45.842 sec)\n",
      "INFO:tensorflow:global_step/sec: 2.18048\n",
      "INFO:tensorflow:loss = 9.0745336e-05, step = 5900 (45.862 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 5907 into ./qa/output_sim/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 7.0486276e-05.\n",
      "INFO:tensorflow:Using config: {'_model_dir': './qa/output_sim/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 0.9\n",
      "  allow_growth: true\n",
      "}\n",
      ", '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f8dba63ca90>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "INFO:tensorflow:Writing example 0 of 12005\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 《 机 械 设 计 基 础 》 这 本 书 的 作 者 是 谁 ？ [SEP] 作 者 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 517 3322 3462 6392 6369 1825 4794 518 6821 3315 741 4638 868 5442 3221 6443 8043 102 868 5442 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 1 (id = 1)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-1\n",
      "INFO:tensorflow:tokens: [CLS] 《 机 械 设 计 基 础 》 这 本 书 的 作 者 是 谁 ？ [SEP] 各 方 兵 力 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 517 3322 3462 6392 6369 1825 4794 518 6821 3315 741 4638 868 5442 3221 6443 8043 102 1392 3175 1070 1213 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-2\n",
      "INFO:tensorflow:tokens: [CLS] 《 机 械 设 计 基 础 》 这 本 书 的 作 者 是 谁 ？ [SEP] 地 区 * * [SEP]\n",
      "INFO:tensorflow:input_ids: 101 517 3322 3462 6392 6369 1825 4794 518 6821 3315 741 4638 868 5442 3221 6443 8043 102 1765 1277 115 115 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-3\n",
      "INFO:tensorflow:tokens: [CLS] 《 机 械 设 计 基 础 》 这 本 书 的 作 者 是 谁 ？ [SEP] 音 乐 成 就 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 517 3322 3462 6392 6369 1825 4794 518 6821 3315 741 4638 868 5442 3221 6443 8043 102 7509 727 2768 2218 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-4\n",
      "INFO:tensorflow:tokens: [CLS] 《 机 械 设 计 基 础 》 这 本 书 的 作 者 是 谁 ？ [SEP] 所 有 权 者 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 517 3322 3462 6392 6369 1825 4794 518 6821 3315 741 4638 868 5442 3221 6443 8043 102 2792 3300 3326 5442 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:Writing example 10000 of 12005\n",
      "INFO:tensorflow:***** Running evaluation *****\n",
      "INFO:tensorflow:  Num examples = 12005\n",
      "INFO:tensorflow:  Batch size = 64\n",
      "INFO:tensorflow:Using config: {'_model_dir': './qa/output_sim/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 0.9\n",
      "  allow_growth: true\n",
      "}\n",
      ", '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f8d9d56ada0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "WARNING:tensorflow:From /home/ma-user/anaconda3/envs/TensorFlow-1.13.1/lib/python3.6/site-packages/tensorflow/python/ops/metrics_impl.py:455: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-09-30T10:37:04Z\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from ./qa/output_sim/model.ckpt-5907\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-09-30-10:37:34\n",
      "INFO:tensorflow:Saving dict for global step 5907: eval_accuracy = 0.9876718, eval_auc = 0.9706281, eval_loss = 0.0823417, global_step = 5907, loss = 0.08273375\n",
      "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 5907: ./qa/output_sim/model.ckpt-5907\n",
      "INFO:tensorflow:***** Eval results *****\n",
      "INFO:tensorflow:  eval_accuracy = 0.9876718\n",
      "INFO:tensorflow:  eval_auc = 0.9706281\n",
      "INFO:tensorflow:  eval_loss = 0.0823417\n",
      "INFO:tensorflow:  global_step = 5907\n",
      "INFO:tensorflow:  loss = 0.08273375\n"
     ]
    }
   ],
   "source": [
    "from qa import bert_similarity \n",
    "import tensorflow as tf\n",
    "\n",
    "#BertSim为BERT相似度检验\n",
    "sim = bert_similarity.BertSim()\n",
    "\n",
    "#在训练集上进行相似度模型训练\n",
    "sim.set_mode(tf.estimator.ModeKeys.TRAIN)\n",
    "sim.train()\n",
    "\n",
    "#在验证集上进行相似度验证\n",
    "sim.set_mode(tf.estimator.ModeKeys.EVAL)\n",
    "sim.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，整合前面两个步骤，进行完整的基于知识库的问答系统构建。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 问答系统（QA）\n",
    "\n",
    "整合以上两个步骤，就可以完成一个简单的基于知识库的问答系统。下面为具体说明：\n",
    "\n",
    "####  1. 命名实体识别：输入问题，使用BERT模型得到问题中的实体，在知识库中检索出包含该实体的所有知识组合。\n",
    "\n",
    "#### 2. 属性映射（文本相似度）：在包含实体的知识组合中，进行文本相似度分析寻找答案，又可分为非语义匹配和语义匹配。\n",
    "\n",
    "  - **非语义匹配**：如果一个知识三元组的关系属性是输入问题的子集（相当于字符串匹配），则该三元组对应的答案匹配为正确答案。非语义匹配步骤可以大大加速匹配。\n",
    "\n",
    "  - **语义匹配**：即文本相似度问题，利用BERT模型计算输入问题与知识三元组的相似度，将最相近的三元组对应的答案匹配为正确答案。\n",
    "\n",
    "原数据集中知识库数据量庞大（2.3G），共43,063,796个三元组，在本实战中选择使用训练集的14620个问答对对应的三元组生成知识库，用于完成简易问答任务。若需要使用完整知识库，请自行下载[Task 5: Open Domain Question Answering](http://tcci.ccf.org.cn/conference/2017/taskdata.php)使用。（由于原知识库较大，建议使用数据库进行存储、查找等操作。）\n",
    "\n",
    "本实战已为开发者做好三元组知识库的数据处理，储存在`data/data_qa`文件夹下的`triple.txt`中。\n",
    "\n",
    "执行下面程序，读取三元组知识库，储存在`triple_data`中，并打印前50条知识。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          0     1                                   2\n",
      "0    机械设计基础    作者                         杨可桢，程光蕴，李仲生\n",
      "1      高等数学   出版社                             武汉大学出版社\n",
      "2      线性代数  出版时间                          2013/12/30\n",
      "3       安德烈    国籍                                 摩纳哥\n",
      "4      线性代数  isbn                   978-7-111-36843-4\n",
      "5      高等数学    书名                          高等数学一（微积分）\n",
      "6      万达广场   外文名                    amoy wanda plaza\n",
      "7        李明  出生日期                              1963.1\n",
      "8     韩娱守护力  小说进度                                  连载\n",
      "9        夏想  连载网站                                潇湘书院\n",
      "10  大学计算机基础    页数                                 272\n",
      "11      城关镇   中文名                            城关镇[临澧县]\n",
      "12       李明   出生地                                青海湟源\n",
      "13  大学计算机基础    定价                                25 元\n",
      "14       李军    职业                      河南省林业厅党组成员、副厅长\n",
      "15       文庙  中文名称                                  文庙\n",
      "16       杨明    民族                                  苗族\n",
      "17      毛泽东    装帧                              平装(无盘)\n",
      "18      西游记    类型                                  奇幻\n",
      "19       王娟    性别                                   女\n",
      "20  计算机应用基础    开本                                  16\n",
      "21       周迅  公司名称                      安徽龙润农业科技开发有限公司\n",
      "22     博士来拜    类别                               油画，壁画\n",
      "23       李军  毕业院校                           天津卫生职工医学院\n",
      "24       陈平    国籍                                  中国\n",
      "25     万达广场  总部地点  重庆市南岸区江南大道8号（重庆万达）/重庆市万州区北滨路（万州万达）\n",
      "26       张宇    性别                                   女\n",
      "27      城关镇    面积                          134.27平方公里\n",
      "28      城关镇  所属地区                                中国西南\n",
      "29      和平村  地理位置                           山东省微山县南阳镇\n",
      "30     西城社区  成立时间                             2006年6月\n",
      "31       王辉    职业                        广东省广播电影电视局官员\n",
      "32     东创建国  经营范围                      整车销售、售后服务、配件供应\n",
      "33       茴香  中文学名                                  茴香\n",
      "34       王军  出生年月                             1962年7月\n",
      "35      丹顶鹤     界                                 动物界\n",
      "36       王平    别名                         原名王惟允，曾用名王明\n",
      "37       李军  主要成就                   中国美术学会会员 中国陶瓷协会会员\n",
      "38      tcl  公司性质                                  私企\n",
      "39      城关镇    人口                                  5万\n",
      "40    圆锥花序组     科                  石竹科caryophyllaceae\n",
      "41     匆匆那年   英文名                        back in time\n",
      "42     《兄弟》    类别                                 图书 \n",
      "43       刘勇  代表作品                     《中国现代作家的宗教文化情结》\n",
      "44    多花秋海棠     门                  被子植物门 angiospermae\n",
      "45    类胡萝卜素    性质                                  色素\n",
      "46       王军    籍贯                                甘肃高台\n",
      "47    命运石之门  游戏类型                             假想科学adv\n",
      "48    圆锥花序组     目                  中央种子目centrospermae\n",
      "49      凤凰山    地点                              山西省定襄县\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "triple_data = pd.read_csv('./qa/data/data_qa/triple.txt', encoding='utf-8', sep='\\t', header=None)\n",
    "print(triple_data[:50])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "建立好知识库后，即可进行基于知识库的问答系统搭建，问答步骤依次为：\n",
    "\n",
    "1. 请提问：输入与知识库相关问题\n",
    "\n",
    "- 打印问题token，实体标注结果，并输出识别实体\n",
    "\n",
    "- 找到结果的可能来源知识集合并打印\n",
    "\n",
    "- 属性映射：分别进行语义匹配（查找属性是否在问题中）和非语义匹配（相似度匹配）\n",
    "\n",
    "- 相似度最高即为匹配答案，并打印答案来源知识及用时\n",
    "\n",
    "- 可进行多轮问答，输入回车结束在线问答\n",
    "\n",
    "#### 首先导入依赖包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using config: {'_model_dir': './qa/output_sim/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 0.9\n",
      "  allow_growth: true\n",
      "}\n",
      ", '_keep_checkpoint_max': 1, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f8da77a8b38>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "WARNING:tensorflow:From /home/ma-user/anaconda3/envs/TensorFlow-1.13.1/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:429: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "tf.py_func is deprecated in TF V2. Instead, use\n",
      "    tf.py_function, which takes a python function which manipulates tf eager\n",
      "    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to\n",
      "    an ndarray (just call tensor.numpy()) but having access to eager tensors\n",
      "    means `tf.py_function`s can use accelerators such as GPUs as well as\n",
      "    being differentiable using a gradient tape.\n",
      "    \n",
      "INFO:tensorflow:Calling model_fn.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "checkpoint path:./qa/output_ner/checkpoint\n",
      "going to restore checkpoint\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./qa/output_ner/model.ckpt-1065\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from qa.bert import tokenization\n",
    "from qa.bert_similarity import BertSim\n",
    "from qa.bert_qa import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 读取BERT预训练模型中文字典"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Done calling model_fn.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['今', '天', '的', '天', '气', '真', '好', '！']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer = tokenization.FullTokenizer(vocab_file=vocab_file, do_lower_case=True)\n",
    "\n",
    "tokenizer.tokenize(\"今天的天气真好！\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 分别获得每个字的字向量、位置向量、文本向量和标签"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert(line):\n",
    "    feature = convert_single_example(0, line, label_list, max_seq_length, tokenizer, 'p')\n",
    "    input_ids = np.reshape([feature.input_ids],(batch_size, max_seq_length))\n",
    "    input_mask = np.reshape([feature.input_mask],(batch_size, max_seq_length))\n",
    "    segment_ids = np.reshape([feature.segment_ids],(batch_size, max_seq_length))\n",
    "    label_ids =np.reshape([feature.label_ids],(batch_size, max_seq_length))\n",
    "    return input_ids, input_mask, segment_ids, label_ids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 构建问答系统"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def kbqa():\n",
    "    while True:\n",
    "        print('\\n\\033[1;31m请提问:\\033[0m\\n')\n",
    "        sentence = str(input())\n",
    "        if len(sentence) == 0:\n",
    "            print(\"再见啦！\")\n",
    "            return\n",
    "        \n",
    "        sentence_ = tokenizer.tokenize(sentence)\n",
    "        print('\\n你的问题是:{}'.format(sentence_))\n",
    "        input_ids, input_mask, segment_ids, label_ids = convert(sentence_)\n",
    "\n",
    "        feed_dict = {input_ids_p: input_ids,\n",
    "                     input_mask_p: input_mask,\n",
    "                     segment_ids_p:segment_ids,\n",
    "                     label_ids_p:label_ids}\n",
    "\n",
    "        pred_ids_result = sess.run([pred_ids], feed_dict)\n",
    "        pred_label_result = convert_id_to_label(pred_ids_result, id2label)\n",
    "        print(\"\\n实体标注结果为：\",pred_label_result)\n",
    "        result = strage_combined_link_org_loc(sentence_, pred_label_result[0])\n",
    "        print('\\n识别的实体是：{}'.format(''.join(result)),'\\n')\n",
    "\n",
    "        ans_range = []\n",
    "        for j in range(len(triple_data)):\n",
    "            if triple_data[0][j] == result[0]:\n",
    "                triple_range = [triple_data[0][j],triple_data[1][j],triple_data[2][j]]\n",
    "                print(\"结果可能来自：\",triple_range)\n",
    "                ans_range.append(triple_range)\n",
    "                \n",
    "        ans = None\n",
    "        ans_base = None\n",
    "        score = 0\n",
    "\n",
    "        for k in range(len(ans_range)):\n",
    "            print(\"\\n知识三元组%d：\"%(k+1),ans_range[k][0],ans_range[k][1],ans_range[k][2])\n",
    "\n",
    "            #非语义匹配\n",
    "            if ans_range[k][1] in sentence:\n",
    "                print(\"属性“\",ans_range[k][1],\"”在问题中\")\n",
    "                ans_ = 1\n",
    "\n",
    "            #语义匹配\n",
    "            else:                \n",
    "                ans_ = max(bs.predict(ans_range[k][0]+ans_range[k][1]+ans_range[k][2],sentence)[0])\n",
    "            print(\"相似度为：\",ans_)\n",
    "\n",
    "            if score < ans_:\n",
    "                score = ans_\n",
    "                ans = ans_range[k][2]\n",
    "                ans_base = ans_range[k]\n",
    "\n",
    "        if score < 0.8:\n",
    "            print(\"\\n\\033[1;31m答案不确定\\033[0m\")\n",
    "            \n",
    "        else:\n",
    "            print(\"\\n答案来自三元组：\",ans_base[0],ans_base[1],ans_base[2])\n",
    "            print(\"相似度为：\", score)\n",
    "            print(\"\\n\\033[1;31m答案是：{}\\033[0m\".format(ans))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 下面就可以运行`kbqa()`来进行基于知识库的多轮在线问答。\n",
    "\n",
    "### 注意，由于使用的知识库规格限制，请先大致浏览`./qa/data/data_qa/`文件夹下`triple.txt`文件中的三元组知识，以免无关问题造成问答效果不佳。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[1;31m请提问:\u001b[0m\n",
      "\n",
      "西游记每集多长时间？\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 西 游 记 类 型 奇 幻 [SEP] 西 游 记 每 集 多 长 时 间 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6205 3952 6381 5102 1798 1936 2404 102 6205 3952 6381 3680 7415 1914 7270 3198 7313 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 西 游 记 首 播 时 间 2010 年 1 月 3 日 [SEP] 西 游 记 每 集 多 长 时 间 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6205 3952 6381 7674 3064 3198 7313 8166 2399 122 3299 124 3189 102 6205 3952 6381 3680 7415 1914 7270 3198 7313 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "你的问题是:['西', '游', '记', '每', '集', '多', '长', '时', '间', '？']\n",
      "\n",
      "实体标注结果为： [['B-LOC', 'I-LOC', 'I-LOC', 'O', 'O', 'O', 'O', 'O', 'O', 'O']]\n",
      "\n",
      "识别的实体是：西游记 \n",
      "\n",
      "结果可能来自： ['西游记', '类型', '奇幻']\n",
      "结果可能来自： ['西游记', '首播时间', '2010年1月3日']\n",
      "结果可能来自： ['西游记', '国家／地区', '中国大陆']\n",
      "结果可能来自： ['西游记', '每集长度', '54分钟']\n",
      "结果可能来自： ['西游记', '播映', '中央电视台']\n",
      "\n",
      "知识三元组1： 西游记 类型 奇幻\n",
      "相似度为： 0.97977567\n",
      "\n",
      "知识三元组2： 西游记 首播时间 2010年1月3日\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 西 游 记 国 家 ／ 地 区 中 国 大 陆 [SEP] 西 游 记 每 集 多 长 时 间 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6205 3952 6381 1744 2157 8027 1765 1277 704 1744 1920 7355 102 6205 3952 6381 3680 7415 1914 7270 3198 7313 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 西 游 记 每 集 长 度 54 分 钟 [SEP] 西 游 记 每 集 多 长 时 间 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6205 3952 6381 3680 7415 7270 2428 8267 1146 7164 102 6205 3952 6381 3680 7415 1914 7270 3198 7313 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 西 游 记 播 映 中 央 电 视 台 [SEP] 西 游 记 每 集 多 长 时 间 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6205 3952 6381 3064 3216 704 1925 4510 6228 1378 102 6205 3952 6381 3680 7415 1914 7270 3198 7313 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "相似度为： 0.9963476\n",
      "\n",
      "知识三元组3： 西游记 国家／地区 中国大陆\n",
      "相似度为： 0.675162\n",
      "\n",
      "知识三元组4： 西游记 每集长度 54分钟\n",
      "相似度为： 0.9998462\n",
      "\n",
      "知识三元组5： 西游记 播映 中央电视台\n",
      "相似度为： 0.99562705\n",
      "\n",
      "答案来自三元组： 西游记 每集长度 54分钟\n",
      "相似度为： 0.9998462\n",
      "\n",
      "\u001b[1;31m答案是：54分钟\u001b[0m\n",
      "\n",
      "\u001b[1;31m请提问:\u001b[0m\n",
      "\n",
      "线性代数的出版时间是什么时候？\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 线 性 代 数 isbn ##97 ##8 - 7 - 111 - 368 ##43 - 4 [SEP] 线 性 代 数 的 出 版 时 间 是 什 么 时 候 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 5296 2595 807 3144 8446 9410 8156 118 128 118 8932 118 11642 9433 118 125 102 5296 2595 807 3144 4638 1139 4276 3198 7313 3221 784 720 3198 952 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 线 性 代 数 书 号 368 ##43 [SEP] 线 性 代 数 的 出 版 时 间 是 什 么 时 候 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 5296 2595 807 3144 741 1384 11642 9433 102 5296 2595 807 3144 4638 1139 4276 3198 7313 3221 784 720 3198 952 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "你的问题是:['线', '性', '代', '数', '的', '出', '版', '时', '间', '是', '什', '么', '时', '候', '？']\n",
      "\n",
      "实体标注结果为： [['B-LOC', 'I-LOC', 'I-LOC', 'I-LOC', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O']]\n",
      "\n",
      "识别的实体是：线性代数 \n",
      "\n",
      "结果可能来自： ['线性代数', '出版时间', '2013/12/30']\n",
      "结果可能来自： ['线性代数', 'isbn', '978-7-111-36843-4']\n",
      "结果可能来自： ['线性代数', '书号', '36843']\n",
      "结果可能来自： ['线性代数', '作者', '侯亚君']\n",
      "结果可能来自： ['线性代数', '出版社', '清华大学出版社']\n",
      "结果可能来自： ['线性代数', '定价', '26元']\n",
      "\n",
      "知识三元组1： 线性代数 出版时间 2013/12/30\n",
      "属性“ 出版时间 ”在问题中\n",
      "相似度为： 1\n",
      "\n",
      "知识三元组2： 线性代数 isbn 978-7-111-36843-4\n",
      "相似度为： 0.9995728\n",
      "\n",
      "知识三元组3： 线性代数 书号 36843\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 线 性 代 数 作 者 侯 亚 君 [SEP] 线 性 代 数 的 出 版 时 间 是 什 么 时 候 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 5296 2595 807 3144 868 5442 908 762 1409 102 5296 2595 807 3144 4638 1139 4276 3198 7313 3221 784 720 3198 952 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 线 性 代 数 出 版 社 清 华 大 学 出 版 社 [SEP] 线 性 代 数 的 出 版 时 间 是 什 么 时 候 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 5296 2595 807 3144 1139 4276 4852 3926 1290 1920 2110 1139 4276 4852 102 5296 2595 807 3144 4638 1139 4276 3198 7313 3221 784 720 3198 952 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 线 性 代 数 定 价 26 元 [SEP] 线 性 代 数 的 出 版 时 间 是 什 么 时 候 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 5296 2595 807 3144 2137 817 8153 1039 102 5296 2595 807 3144 4638 1139 4276 3198 7313 3221 784 720 3198 952 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "相似度为： 0.9996922\n",
      "\n",
      "知识三元组4： 线性代数 作者 侯亚君\n",
      "相似度为： 0.97082436\n",
      "\n",
      "知识三元组5： 线性代数 出版社 清华大学出版社\n",
      "相似度为： 0.9994628\n",
      "\n",
      "知识三元组6： 线性代数 定价 26元\n",
      "相似度为： 0.98183894\n",
      "\n",
      "答案来自三元组： 线性代数 出版时间 2013/12/30\n",
      "相似度为： 1\n",
      "\n",
      "\u001b[1;31m答案是：2013/12/30\u001b[0m\n",
      "\n",
      "\u001b[1;31m请提问:\u001b[0m\n",
      "\n",
      "迈克尔·杰克逊的资产是多少？\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 迈 克 尔 · 杰 克 逊 逝 世 2009 年 6 月 25 日 （ 50 岁 ） 美 国 加 利 福 尼 亚 州 洛 杉 矶 市 荷 尔 贝 山 [SEP] 迈 克 尔 · 杰 克 逊 的 资 产 是 多 少 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6815 1046 2209 185 3345 1046 6849 6860 686 8170 2399 127 3299 8132 3189 8020 8145 2259 8021 5401 1744 1217 1164 4886 2225 762 2336 3821 3329 4768 2356 5792 2209 6564 2255 102 6815 1046 2209 185 3345 1046 6849 4638 6598 772 3221 1914 2208 8043 102 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "你的问题是:['迈', '克', '尔', '·', '杰', '克', '逊', '的', '资', '产', '是', '多', '少', '？']\n",
      "\n",
      "实体标注结果为： [['B-LOC', 'I-LOC', 'I-LOC', 'I-LOC', 'I-LOC', 'I-LOC', 'I-LOC', 'O', 'O', 'O', 'O', 'O', 'O', 'O']]\n",
      "\n",
      "识别的实体是：迈克尔·杰克逊 \n",
      "\n",
      "结果可能来自： ['迈克尔·杰克逊', '逝世', '2009年6月25日（50岁） 美国加利福尼亚州洛杉矶市荷尔贝山']\n",
      "结果可能来自： ['迈克尔·杰克逊', '厂牌', '钢城唱片 摩城唱片 史诗唱片 遗产唱片 mjj制作']\n",
      "结果可能来自： ['迈克尔·杰克逊', '净资产', '▲ 11.78亿美元 （2009年估算）[1]']\n",
      "\n",
      "知识三元组1： 迈克尔·杰克逊 逝世 2009年6月25日（50岁） 美国加利福尼亚州洛杉矶市荷尔贝山\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 迈 克 尔 · 杰 克 逊 厂 牌 钢 城 唱 片 摩 城 唱 片 史 诗 唱 片 遗 产 唱 片 m ##j ##j 制 作 [SEP] 迈 克 尔 · 杰 克 逊 的 资 产 是 多 少 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6815 1046 2209 185 3345 1046 6849 1322 4277 7167 1814 1548 4275 3040 1814 1548 4275 1380 6408 1548 4275 6890 772 1548 4275 155 8334 8334 1169 868 102 6815 1046 2209 185 3345 1046 6849 4638 6598 772 3221 1914 2208 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 迈 克 尔 · 杰 克 逊 净 资 产 ▲ 11 . 78 亿 美 元 （ 2009 年 估 算 ） [ 1 ] [SEP] 迈 克 尔 · 杰 克 逊 的 资 产 是 多 少 ？ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 6815 1046 2209 185 3345 1046 6849 1112 6598 772 464 8111 119 8409 783 5401 1039 8020 8170 2399 844 5050 8021 138 122 140 102 6815 1046 2209 185 3345 1046 6849 4638 6598 772 3221 1914 2208 8043 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "相似度为： 0.9995204\n",
      "\n",
      "知识三元组2： 迈克尔·杰克逊 厂牌 钢城唱片 摩城唱片 史诗唱片 遗产唱片 mjj制作\n",
      "相似度为： 0.9938148\n",
      "\n",
      "知识三元组3： 迈克尔·杰克逊 净资产 ▲ 11.78亿美元 （2009年估算）[1]\n",
      "相似度为： 0.9997197\n",
      "\n",
      "答案来自三元组： 迈克尔·杰克逊 净资产 ▲ 11.78亿美元 （2009年估算）[1]\n",
      "相似度为： 0.9997197\n",
      "\n",
      "\u001b[1;31m答案是：▲ 11.78亿美元 （2009年估算）[1]\u001b[0m\n",
      "\n",
      "\u001b[1;31m请提问:\u001b[0m\n",
      "\n",
      "\n",
      "再见啦！\n"
     ]
    }
   ],
   "source": [
    "kbqa()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "至此，使用BERT进行基于知识库的问答系统搭建完毕。\n",
    "\n",
    "\n",
    "## 注意，在使用结束后，请关闭并删除开发环境，以免造成不必要的扣费。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}