notebooks/dataprep/01h-CORDLineages.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Downloads Publication Information for PANGO Lineages from the CORD-19 Data Set\n",
    "**[Work in progress]**\n",
    "\n",
    "This notebook text-mines [PANGO lineage](https://cov-lineages.org/) mentions in the titles and abstracts of publications and preprints from the CORD-19 data set. Note, the text-mined results may contain false positive!\n",
    "\n",
    "Data sources: [PANGO Lineage Designations](https://github.com/cov-lineages/pango-designation), \n",
    "[CORD-19](https://allenai.org/data/cord-19)\n",
    "\n",
    "References:\n",
    "\n",
    "Rambaut A, et al., A dynamic nomenclature proposal for SARS-CoV-2 lineages to assist genomic epidemiology(2020) Nature Microbiology [doi:10.1038/s41564-020-0770-5](https://doi.org/10.1038/s41564-020-0770-5).\n",
    "\n",
    "Lucy Lu Wang, et al., CORD-19: The COVID-19 Open Research Dataset (2020) [arXiv:2004.10706v4](https://arxiv.org/abs/2004.10706).\n",
    "\n",
    "Author: Peter Rose (pwrose@ucsd.edu)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import io\n",
    "import dateutil\n",
    "import re\n",
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.options.display.max_rows = None  # display all rows\n",
    "pd.options.display.max_columns = None  # display all columsns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/peter/Library/Application Support/com.Neo4j.Relate/data/dbmss/dbms-8bf637fc-0d20-4d9f-9c6f-f7e72e92a4da/import\n"
     ]
    }
   ],
   "source": [
    "NEO4J_IMPORT = Path(os.getenv('NEO4J_IMPORT'))\n",
    "print(NEO4J_IMPORT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get PANGO lineages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "pango = pd.read_csv(NEO4J_IMPORT / \"00b-PANGOLineage.csv\", dtype=str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lineage</th>\n",
       "      <th>description</th>\n",
       "      <th>alias</th>\n",
       "      <th>predecessor</th>\n",
       "      <th>l0</th>\n",
       "      <th>l1</th>\n",
       "      <th>l2</th>\n",
       "      <th>l3</th>\n",
       "      <th>levels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1208</th>\n",
       "      <td>M.1</td>\n",
       "      <td>Alias of B.1.1.294.1, Israel</td>\n",
       "      <td>B.1.1.294.1</td>\n",
       "      <td>B.1.1.294</td>\n",
       "      <td>M.1</td>\n",
       "      <td>M</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>966</th>\n",
       "      <td>B.1.510</td>\n",
       "      <td>Iceland, Sweden, North America (could be disso...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>B.1.510</td>\n",
       "      <td>B.1</td>\n",
       "      <td>B</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>622</th>\n",
       "      <td>B.1.177.53</td>\n",
       "      <td>UK lineage</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>B.1.177.53</td>\n",
       "      <td>B.1.177</td>\n",
       "      <td>B.1</td>\n",
       "      <td>B</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1158</th>\n",
       "      <td>C.2</td>\n",
       "      <td>Alias of B.1.1.1.2, South Africa and some Euro...</td>\n",
       "      <td>B.1.1.1.2</td>\n",
       "      <td>B.1.1.1</td>\n",
       "      <td>C.2</td>\n",
       "      <td>C</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>529</th>\n",
       "      <td>B.1.153</td>\n",
       "      <td>English lineage with some Australian, New Zeal...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>B.1.153</td>\n",
       "      <td>B.1</td>\n",
       "      <td>B</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         lineage                                        description  \\\n",
       "1208         M.1                       Alias of B.1.1.294.1, Israel   \n",
       "966      B.1.510  Iceland, Sweden, North America (could be disso...   \n",
       "622   B.1.177.53                                         UK lineage   \n",
       "1158         C.2  Alias of B.1.1.1.2, South Africa and some Euro...   \n",
       "529      B.1.153  English lineage with some Australian, New Zeal...   \n",
       "\n",
       "            alias predecessor          l0       l1   l2   l3 levels  \n",
       "1208  B.1.1.294.1   B.1.1.294         M.1        M  NaN  NaN      2  \n",
       "966           NaN         NaN     B.1.510      B.1    B  NaN      3  \n",
       "622           NaN         NaN  B.1.177.53  B.1.177  B.1    B      4  \n",
       "1158    B.1.1.1.2     B.1.1.1         C.2        C  NaN  NaN      2  \n",
       "529           NaN         NaN     B.1.153      B.1    B  NaN      3  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pango.sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "lineages = set(pango['lineage'].unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get CORD-19 Metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "CACHE = Path(NEO4J_IMPORT / 'cache/cord19/metadata.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata = pd.read_csv(CACHE, dtype='str')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata.fillna('', inplace=True)\n",
    "#convert datetime column to just date\n",
    "metadata['year'] = metadata['publish_time'].apply(lambda d: d[:4] if len(d) > 4 else '')\n",
    "metadata['date'] = metadata['publish_time'].apply(lambda d: dateutil.parser.parse(d) if len(d) > 0 else '')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of papers 536817\n"
     ]
    }
   ],
   "source": [
    "print(\"Total number of papers\", metadata.shape[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extract a list of PANGO lineages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove special characters to simply parsing for lineages in parenthesis, comma-separated lists, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata['title'] = metadata['title'].replace('[()/,]', ' ', regex=True)\n",
    "metadata['abstract'] = metadata['abstract'].replace('[()/,]', ' ', regex=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Match PANGO patterns and check agains list of known lineages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "pattern1 = re.compile(' [A-Z]{1,2}[.]\\d+ ')\n",
    "pattern2 = re.compile(' [A-Z]{1,2}[.]\\d+[.]\\d+ ')\n",
    "pattern3 = re.compile(' [A-Z]{1,2}[.]\\d+[.]\\d+[.]+\\d+ ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_lineages(row):\n",
    "    text = ' ' + row.title + ' ' + row.abstract + ' '\n",
    "    lin = pattern1.findall(text) + pattern2.findall(text) + pattern3.findall(text)\n",
    "    u_lin = set()\n",
    "    \n",
    "    for l in lin:\n",
    "        l = l.strip()\n",
    "        # check if lineage is valid (e.g., not a withdrawn lineage or false positive)\n",
    "        if l in lineages:\n",
    "            u_lin.add(l)\n",
    "            \n",
    "    return \";\".join(u_lin)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata['lineages'] = metadata.apply(get_lineages, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keep only papers that map to PANGO lineages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "hits = metadata[metadata['lineages'].str.len() > 0].copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assign CURIEs from [Identifiers.org](https://identifiers.org)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "hits['doi'] = hits['doi'].apply(lambda x: 'doi:' + x if len(x) > 0 else '')\n",
    "hits['pubmed_id'] = hits['pubmed_id'].apply(lambda x: 'pubmed:' + x if len(x) > 0 else '')\n",
    "hits['pmcid'] = hits['pmcid'].apply(lambda x: 'pmc:' + x if len(x) > 0 else '')\n",
    "hits['arxiv_id'] = hits['arxiv_id'].apply(lambda x: 'arxiv:' + x if len(x) > 0 else '')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hits.sort_values(by=['publish_time'], ascending=False, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of matches 592\n"
     ]
    }
   ],
   "source": [
    "print(\"Number of matches\", hits.shape[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_id(row):\n",
    "    \"\"\"Creates a unique id using the most commonly available id in priority order\"\"\"\n",
    "    if row.doi != '':\n",
    "        return row.doi\n",
    "    elif row.pubmed_id != '':\n",
    "        return row.pubmed_id\n",
    "    elif row.pmcid != '':\n",
    "        return row.pmcid\n",
    "    elif row.arxiv_id != '':\n",
    "        return row.arxiv_id\n",
    "    elif row.url != '':\n",
    "        return row.url\n",
    "    else:\n",
    "        # TODO deal with WHO papers here?\n",
    "        return ''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "hits['id'] = hits.apply(create_id, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cord_uid</th>\n",
       "      <th>sha</th>\n",
       "      <th>source_x</th>\n",
       "      <th>title</th>\n",
       "      <th>doi</th>\n",
       "      <th>pmcid</th>\n",
       "      <th>pubmed_id</th>\n",
       "      <th>license</th>\n",
       "      <th>abstract</th>\n",
       "      <th>publish_time</th>\n",
       "      <th>authors</th>\n",
       "      <th>journal</th>\n",
       "      <th>mag_id</th>\n",
       "      <th>who_covidence_id</th>\n",
       "      <th>arxiv_id</th>\n",
       "      <th>pdf_json_files</th>\n",
       "      <th>pmc_json_files</th>\n",
       "      <th>url</th>\n",
       "      <th>s2_id</th>\n",
       "      <th>year</th>\n",
       "      <th>date</th>\n",
       "      <th>lineages</th>\n",
       "      <th>id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>286738</th>\n",
       "      <td>itikugkk</td>\n",
       "      <td></td>\n",
       "      <td>WHO</td>\n",
       "      <td>An Observational Laboratory-Based Assessment o...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>unk</td>\n",
       "      <td>Information on severe acute respiratory syndro...</td>\n",
       "      <td>2021</td>\n",
       "      <td>Sander, Anna-Lena; Yadouleton, Anges; Moreira-...</td>\n",
       "      <td>MSphere</td>\n",
       "      <td></td>\n",
       "      <td>#1029965</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>231605197</td>\n",
       "      <td></td>\n",
       "      <td>2021-04-23</td>\n",
       "      <td>A.4;B.1</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>247931</th>\n",
       "      <td>ftvatk0a</td>\n",
       "      <td></td>\n",
       "      <td>WHO</td>\n",
       "      <td>Immunoinformatic analysis of structural and ep...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>unk</td>\n",
       "      <td>A newly emerged strain of SARS-CoV-2 of B.1.1....</td>\n",
       "      <td>2021</td>\n",
       "      <td>Hussain, Mushtaq; Shabbir, Sanya; Amanullah, A...</td>\n",
       "      <td>J. med. virol</td>\n",
       "      <td></td>\n",
       "      <td>#1126508</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>232196602</td>\n",
       "      <td></td>\n",
       "      <td>2021-04-23</td>\n",
       "      <td>B.1.1.7</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16254</th>\n",
       "      <td>g157h5sn</td>\n",
       "      <td>cc129c4cd6dc4523658db923b90f269e11d9d09a</td>\n",
       "      <td>PMC</td>\n",
       "      <td>Could it be that the B.1.1.7 lineage is more d...</td>\n",
       "      <td>doi:10.1017/ice.2021.59</td>\n",
       "      <td>pmc:PMC7948098</td>\n",
       "      <td>pubmed:33557960</td>\n",
       "      <td>no-cc</td>\n",
       "      <td></td>\n",
       "      <td>2021-02-09</td>\n",
       "      <td>Kow, Chia Siang; Hasan, Syed Shahzad</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>document_parses/pdf_json/cc129c4cd6dc4523658db...</td>\n",
       "      <td>document_parses/pmc_json/PMC7948098.xml.json</td>\n",
       "      <td>https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7...</td>\n",
       "      <td></td>\n",
       "      <td>2021</td>\n",
       "      <td>2021-02-09</td>\n",
       "      <td>B.1.1.7</td>\n",
       "      <td>doi:10.1017/ice.2021.59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>420424</th>\n",
       "      <td>vw3cuiof</td>\n",
       "      <td>69fa13b80bb407e8bca177cfb906bbf604ffcf63; c0ee...</td>\n",
       "      <td>Elsevier; Medline; PMC</td>\n",
       "      <td>Genomic characteristics and clinical effect of...</td>\n",
       "      <td>doi:10.1016/s1473-3099(21)00170-5</td>\n",
       "      <td>pmc:PMC8041359</td>\n",
       "      <td>pubmed:33857406</td>\n",
       "      <td>no-cc</td>\n",
       "      <td>BACKGROUND: Emergence of variants with specifi...</td>\n",
       "      <td>2021-04-12</td>\n",
       "      <td>Frampton, Dan; Rampling, Tommy; Cross, Aidan; ...</td>\n",
       "      <td>Lancet Infect Dis</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>document_parses/pdf_json/69fa13b80bb407e8bca17...</td>\n",
       "      <td>document_parses/pmc_json/PMC8041359.xml.json</td>\n",
       "      <td>https://www.sciencedirect.com/science/article/...</td>\n",
       "      <td>233215323</td>\n",
       "      <td>2021</td>\n",
       "      <td>2021-04-12</td>\n",
       "      <td>B.1.1.7</td>\n",
       "      <td>doi:10.1016/s1473-3099(21)00170-5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>395962</th>\n",
       "      <td>wun5rwsk</td>\n",
       "      <td>3966d03d72440c939928b54f0d024f49181db116</td>\n",
       "      <td>Elsevier; Medline; PMC</td>\n",
       "      <td>Emergence and rapid transmission of SARS-CoV-2...</td>\n",
       "      <td>doi:10.1016/j.cell.2021.03.052</td>\n",
       "      <td>pmc:PMC8009040</td>\n",
       "      <td>pubmed:33861950</td>\n",
       "      <td>els-covid</td>\n",
       "      <td>The highly transmissible B.1.1.7 variant of SA...</td>\n",
       "      <td>2021-03-30</td>\n",
       "      <td>Washington, Nicole L.; Gangavarapu, Karthik; Z...</td>\n",
       "      <td>Cell</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>document_parses/pdf_json/3966d03d72440c939928b...</td>\n",
       "      <td></td>\n",
       "      <td>https://www.ncbi.nlm.nih.gov/pubmed/33861950/;...</td>\n",
       "      <td>232412011</td>\n",
       "      <td>2021</td>\n",
       "      <td>2021-03-30</td>\n",
       "      <td>B.1.1.7</td>\n",
       "      <td>doi:10.1016/j.cell.2021.03.052</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96776</th>\n",
       "      <td>k3qp2lxc</td>\n",
       "      <td></td>\n",
       "      <td>Medline</td>\n",
       "      <td>Detection of B.1.351 SARS-CoV-2 Variant Strain...</td>\n",
       "      <td>doi:10.15585/mmwr.mm7008e2</td>\n",
       "      <td></td>\n",
       "      <td>pubmed:33630820</td>\n",
       "      <td>unk</td>\n",
       "      <td>The first laboratory-confirmed cases of corona...</td>\n",
       "      <td>2021-02-26</td>\n",
       "      <td>Mwenda, Mulenga; Saasa, Ngonda; Sinyange, Nyam...</td>\n",
       "      <td>MMWR. Morbidity and mortality weekly report</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>https://doi.org/10.15585/mmwr.mm7008e2; https:...</td>\n",
       "      <td>232058475</td>\n",
       "      <td>2021</td>\n",
       "      <td>2021-02-26</td>\n",
       "      <td>B.1.351</td>\n",
       "      <td>doi:10.15585/mmwr.mm7008e2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>477204</th>\n",
       "      <td>g9pwxfd2</td>\n",
       "      <td>5adfe79458623b8b71e1c1b3ccf370a81445a654</td>\n",
       "      <td>Elsevier; Medline; PMC</td>\n",
       "      <td>Characterizing SARS-CoV-2 genome diversity cir...</td>\n",
       "      <td>doi:10.1016/j.ijid.2021.02.073</td>\n",
       "      <td>pmc:PMC7895695</td>\n",
       "      <td>pubmed:33618008</td>\n",
       "      <td>els-covid</td>\n",
       "      <td>Objectives To evaluate the genomic diversity a...</td>\n",
       "      <td>2021-02-20</td>\n",
       "      <td>Muñoz, Marina; Patiño, Luz H.; Ballesteros, Na...</td>\n",
       "      <td>Int J Infect Dis</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>document_parses/pdf_json/5adfe79458623b8b71e1c...</td>\n",
       "      <td></td>\n",
       "      <td>https://api.elsevier.com/content/article/pii/S...</td>\n",
       "      <td>231965833</td>\n",
       "      <td>2021</td>\n",
       "      <td>2021-02-20</td>\n",
       "      <td>B.1.351;B.1.1.7;P.1</td>\n",
       "      <td>doi:10.1016/j.ijid.2021.02.073</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>447927</th>\n",
       "      <td>3ztgrafg</td>\n",
       "      <td></td>\n",
       "      <td>MedRxiv; WHO</td>\n",
       "      <td>SARS-CoV-2 B.1.1.7 lineage-related perceptions...</td>\n",
       "      <td>doi:10.1101/2021.01.19.21250111</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>medrxiv</td>\n",
       "      <td>Background: Healthcare workers'  HCWs'  travel...</td>\n",
       "      <td>2021-01-21</td>\n",
       "      <td>Temsah, M.-H.; Barry, M.; Aljamaan, F.; Alhuza...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>http://medrxiv.org/cgi/content/short/2021.01.1...</td>\n",
       "      <td>231653830</td>\n",
       "      <td>2021</td>\n",
       "      <td>2021-01-21</td>\n",
       "      <td>B.1.1.7</td>\n",
       "      <td>doi:10.1101/2021.01.19.21250111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>169766</th>\n",
       "      <td>h5e8bi73</td>\n",
       "      <td></td>\n",
       "      <td>WHO</td>\n",
       "      <td>Identification of B.1.346 Lineage of SARS-CoV-...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>unk</td>\n",
       "      <td>SARS-CoV-2 whole-genome sequencing of samples ...</td>\n",
       "      <td>2021</td>\n",
       "      <td>Abe, Kodai; Shimura, Takako; Takenouchi, Toshi...</td>\n",
       "      <td>Keio j. med</td>\n",
       "      <td></td>\n",
       "      <td>#1183786</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>231729271</td>\n",
       "      <td></td>\n",
       "      <td>2021-04-23</td>\n",
       "      <td>B.1.1.284;B.1.1.214;B.1.346</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>483453</th>\n",
       "      <td>ccqq5734</td>\n",
       "      <td>38be32b65618c66a2e3d41db5be9cda55a01b324</td>\n",
       "      <td>BioRxiv; WHO</td>\n",
       "      <td>A human antibody with blocking activity to RBD...</td>\n",
       "      <td>doi:10.1101/2021.02.07.429299</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>biorxiv</td>\n",
       "      <td>Severe acute respiratory syndrome coronavirus-...</td>\n",
       "      <td>2021-02-08</td>\n",
       "      <td>Gu, Chunyin; Cao, Xiaodan; Wang, Zongda; Hu, X...</td>\n",
       "      <td>bioRxiv</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>document_parses/pdf_json/38be32b65618c66a2e3d4...</td>\n",
       "      <td></td>\n",
       "      <td>https://doi.org/10.1101/2021.02.07.429299</td>\n",
       "      <td>231885348</td>\n",
       "      <td>2021</td>\n",
       "      <td>2021-02-08</td>\n",
       "      <td>B.1.351</td>\n",
       "      <td>doi:10.1101/2021.02.07.429299</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>452322</th>\n",
       "      <td>770f1qn6</td>\n",
       "      <td>e49ae48fbdccbf290c84f9670a9257f2d3e00fa3</td>\n",
       "      <td>Elsevier; Medline; PMC</td>\n",
       "      <td>Effect of the new SARS-CoV-2 variant B.1.1.7 o...</td>\n",
       "      <td>doi:10.1016/s2352-4642(21)00030-4</td>\n",
       "      <td>pmc:PMC7906637</td>\n",
       "      <td>pubmed:33581054</td>\n",
       "      <td>no-cc</td>\n",
       "      <td></td>\n",
       "      <td>2021-02-10</td>\n",
       "      <td>Brookman, Sarah; Cook, James; Zucherman, Mark;...</td>\n",
       "      <td>Lancet Child Adolesc Health</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>document_parses/pdf_json/e49ae48fbdccbf290c84f...</td>\n",
       "      <td>document_parses/pmc_json/PMC7906637.xml.json</td>\n",
       "      <td>https://api.elsevier.com/content/article/pii/S...</td>\n",
       "      <td>231919047</td>\n",
       "      <td>2021</td>\n",
       "      <td>2021-02-10</td>\n",
       "      <td>B.1.1.7</td>\n",
       "      <td>doi:10.1016/s2352-4642(21)00030-4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96842</th>\n",
       "      <td>6bne3x68</td>\n",
       "      <td></td>\n",
       "      <td>Medline</td>\n",
       "      <td>Sensitivity of infectious SARS-CoV-2 B.1.1.7 a...</td>\n",
       "      <td>doi:10.1038/s41591-021-01318-5</td>\n",
       "      <td></td>\n",
       "      <td>pubmed:33772244</td>\n",
       "      <td>unk</td>\n",
       "      <td>Severe acute respiratory syndrome coronavirus ...</td>\n",
       "      <td>2021-03-26</td>\n",
       "      <td>Planas, Delphine; Bruel, Timothée; Grzelak, Lu...</td>\n",
       "      <td>Nature medicine</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>https://doi.org/10.1038/s41591-021-01318-5; ht...</td>\n",
       "      <td>232375717</td>\n",
       "      <td>2021</td>\n",
       "      <td>2021-03-26</td>\n",
       "      <td>B.1.351;B.1.1.7</td>\n",
       "      <td>doi:10.1038/s41591-021-01318-5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>454903</th>\n",
       "      <td>ufybfmp4</td>\n",
       "      <td></td>\n",
       "      <td>MedRxiv; WHO</td>\n",
       "      <td>Epidemiological and Clinical Characteristics  ...</td>\n",
       "      <td>doi:10.1101/2021.01.06.20249091</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>medrxiv</td>\n",
       "      <td>Background: The earliest coronavirus disease-2...</td>\n",
       "      <td>2021-01-08</td>\n",
       "      <td>Yegorov, S.; Goremykina, M.; Ivanova, R.; Good...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>https://doi.org/10.1101/2021.01.06.20249091; h...</td>\n",
       "      <td>231138384</td>\n",
       "      <td>2021</td>\n",
       "      <td>2021-01-08</td>\n",
       "      <td>B.4.1;B.1.3;B.1.1;B.1;A.2</td>\n",
       "      <td>doi:10.1101/2021.01.06.20249091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>462424</th>\n",
       "      <td>0usbo1yu</td>\n",
       "      <td></td>\n",
       "      <td>BioRxiv; WHO</td>\n",
       "      <td>Phylogenetic analyses of SARS-CoV-2 B.1.1.7 li...</td>\n",
       "      <td>doi:10.1101/2021.01.12.426373</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>biorxiv</td>\n",
       "      <td>The authors have withdrawn this manuscript bec...</td>\n",
       "      <td>2021-01-15</td>\n",
       "      <td>Vrancken, B.; Dellicour, S.; Smith, D.M.; Chai...</td>\n",
       "      <td>bioRxiv</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>https://doi.org/10.1101/2021.01.12.426373</td>\n",
       "      <td>231666904</td>\n",
       "      <td>2021</td>\n",
       "      <td>2021-01-15</td>\n",
       "      <td>B.1.1.7</td>\n",
       "      <td>doi:10.1101/2021.01.12.426373</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>279515</th>\n",
       "      <td>f6yc8ho0</td>\n",
       "      <td></td>\n",
       "      <td>WHO</td>\n",
       "      <td>Detection of B.1.351 SARS-CoV-2 Variant Strain...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>unk</td>\n",
       "      <td>The first laboratory-confirmed cases of corona...</td>\n",
       "      <td>2021</td>\n",
       "      <td>Mwenda, Mulenga; Saasa, Ngonda; Sinyange, Nyam...</td>\n",
       "      <td>MMWR Morb Mortal Wkly Rep</td>\n",
       "      <td></td>\n",
       "      <td>#1102701</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>232058475</td>\n",
       "      <td></td>\n",
       "      <td>2021-04-23</td>\n",
       "      <td>B.1.351</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>529853</th>\n",
       "      <td>k0uolif0</td>\n",
       "      <td>c5bf135c660e1a18623d3f3c40c7b89c9630e310</td>\n",
       "      <td>BioRxiv; WHO</td>\n",
       "      <td>Computational assessment of the spike protein ...</td>\n",
       "      <td>doi:10.1101/2021.03.25.437035</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>biorxiv</td>\n",
       "      <td>Since its emergence into the human population ...</td>\n",
       "      <td>2021-03-25</td>\n",
       "      <td>Ge, Anni; Rioux, Melissa; Kelvin, Alyson A.</td>\n",
       "      <td>bioRxiv</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>document_parses/pdf_json/c5bf135c660e1a18623d3...</td>\n",
       "      <td></td>\n",
       "      <td>https://doi.org/10.1101/2021.03.25.437035</td>\n",
       "      <td>232412338</td>\n",
       "      <td>2021</td>\n",
       "      <td>2021-03-25</td>\n",
       "      <td>B.1.351;B.1.1.7;B.1.429;P.1</td>\n",
       "      <td>doi:10.1101/2021.03.25.437035</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>424949</th>\n",
       "      <td>1ms5m97x</td>\n",
       "      <td>fbf9052e9c2076014217462000f8fdc35464e5dc</td>\n",
       "      <td>Medline; PMC</td>\n",
       "      <td>SARS-CoV-2 Variant of Concern 202 012 01  B.1....</td>\n",
       "      <td>doi:10.1093/jtm/taab032</td>\n",
       "      <td>pmc:PMC7989411</td>\n",
       "      <td>pubmed:33693926</td>\n",
       "      <td>no-cc</td>\n",
       "      <td>We report the complete genome of SARS-CoV-2 VO...</td>\n",
       "      <td>2021-03-08</td>\n",
       "      <td>Yao, Mingxiao; Jiang, Fachun; Zhang, Yuwei; Li...</td>\n",
       "      <td>J Travel Med</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>document_parses/pdf_json/fbf9052e9c20760142174...</td>\n",
       "      <td></td>\n",
       "      <td>https://doi.org/10.1093/jtm/taab032; https://w...</td>\n",
       "      <td>232197534</td>\n",
       "      <td>2021</td>\n",
       "      <td>2021-03-08</td>\n",
       "      <td>B.1.1.7</td>\n",
       "      <td>doi:10.1093/jtm/taab032</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>219772</th>\n",
       "      <td>61ck712f</td>\n",
       "      <td></td>\n",
       "      <td>WHO</td>\n",
       "      <td>Sixteen novel lineages of SARS-CoV-2 in South ...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>unk</td>\n",
       "      <td>The first severe acute respiratory syndrome co...</td>\n",
       "      <td>2021</td>\n",
       "      <td>Tegally, Houriiyah; Wilkinson, Eduan; Lessells...</td>\n",
       "      <td>Nat Med</td>\n",
       "      <td></td>\n",
       "      <td>#1061182</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>231788423</td>\n",
       "      <td></td>\n",
       "      <td>2021-04-23</td>\n",
       "      <td>B.1.1.54;B.1.1.56;C.1;B.1.106</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>331889</th>\n",
       "      <td>3qe1tyur</td>\n",
       "      <td></td>\n",
       "      <td>WHO</td>\n",
       "      <td>Detection and Genome Sequencing of SARS-CoV-2 ...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>unk</td>\n",
       "      <td>Since the emergence of coronavirus disease  CO...</td>\n",
       "      <td>2021</td>\n",
       "      <td>Klaus, Julia; Meli, Marina L; Willi, Barbara; ...</td>\n",
       "      <td>Viruses</td>\n",
       "      <td></td>\n",
       "      <td>#1138760</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>232383967</td>\n",
       "      <td></td>\n",
       "      <td>2021-04-23</td>\n",
       "      <td>B.1.1.39</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>485445</th>\n",
       "      <td>9ju00ryh</td>\n",
       "      <td>5fc2ca54b6535130b999af0cd24f41cbc1321c94</td>\n",
       "      <td>MedRxiv; WHO</td>\n",
       "      <td>Immunodominant B cell epitope in SARS-CoV-2 RB...</td>\n",
       "      <td>doi:10.1101/2021.03.11.21253399</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>medrxiv</td>\n",
       "      <td>Recent SARS-CoV-2 variants pose important conc...</td>\n",
       "      <td>2021-03-12</td>\n",
       "      <td>Santos, K. S.; Oliveira, J. R.; Machado, R. R....</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>document_parses/pdf_json/5fc2ca54b6535130b999a...</td>\n",
       "      <td></td>\n",
       "      <td>https://doi.org/10.1101/2021.03.11.21253399; h...</td>\n",
       "      <td>232204178</td>\n",
       "      <td>2021</td>\n",
       "      <td>2021-03-12</td>\n",
       "      <td>B.1.351;P.1</td>\n",
       "      <td>doi:10.1101/2021.03.11.21253399</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        cord_uid                                                sha  \\\n",
       "286738  itikugkk                                                      \n",
       "247931  ftvatk0a                                                      \n",
       "16254   g157h5sn           cc129c4cd6dc4523658db923b90f269e11d9d09a   \n",
       "420424  vw3cuiof  69fa13b80bb407e8bca177cfb906bbf604ffcf63; c0ee...   \n",
       "395962  wun5rwsk           3966d03d72440c939928b54f0d024f49181db116   \n",
       "96776   k3qp2lxc                                                      \n",
       "477204  g9pwxfd2           5adfe79458623b8b71e1c1b3ccf370a81445a654   \n",
       "447927  3ztgrafg                                                      \n",
       "169766  h5e8bi73                                                      \n",
       "483453  ccqq5734           38be32b65618c66a2e3d41db5be9cda55a01b324   \n",
       "452322  770f1qn6           e49ae48fbdccbf290c84f9670a9257f2d3e00fa3   \n",
       "96842   6bne3x68                                                      \n",
       "454903  ufybfmp4                                                      \n",
       "462424  0usbo1yu                                                      \n",
       "279515  f6yc8ho0                                                      \n",
       "529853  k0uolif0           c5bf135c660e1a18623d3f3c40c7b89c9630e310   \n",
       "424949  1ms5m97x           fbf9052e9c2076014217462000f8fdc35464e5dc   \n",
       "219772  61ck712f                                                      \n",
       "331889  3qe1tyur                                                      \n",
       "485445  9ju00ryh           5fc2ca54b6535130b999af0cd24f41cbc1321c94   \n",
       "\n",
       "                      source_x  \\\n",
       "286738                     WHO   \n",
       "247931                     WHO   \n",
       "16254                      PMC   \n",
       "420424  Elsevier; Medline; PMC   \n",
       "395962  Elsevier; Medline; PMC   \n",
       "96776                  Medline   \n",
       "477204  Elsevier; Medline; PMC   \n",
       "447927            MedRxiv; WHO   \n",
       "169766                     WHO   \n",
       "483453            BioRxiv; WHO   \n",
       "452322  Elsevier; Medline; PMC   \n",
       "96842                  Medline   \n",
       "454903            MedRxiv; WHO   \n",
       "462424            BioRxiv; WHO   \n",
       "279515                     WHO   \n",
       "529853            BioRxiv; WHO   \n",
       "424949            Medline; PMC   \n",
       "219772                     WHO   \n",
       "331889                     WHO   \n",
       "485445            MedRxiv; WHO   \n",
       "\n",
       "                                                    title  \\\n",
       "286738  An Observational Laboratory-Based Assessment o...   \n",
       "247931  Immunoinformatic analysis of structural and ep...   \n",
       "16254   Could it be that the B.1.1.7 lineage is more d...   \n",
       "420424  Genomic characteristics and clinical effect of...   \n",
       "395962  Emergence and rapid transmission of SARS-CoV-2...   \n",
       "96776   Detection of B.1.351 SARS-CoV-2 Variant Strain...   \n",
       "477204  Characterizing SARS-CoV-2 genome diversity cir...   \n",
       "447927  SARS-CoV-2 B.1.1.7 lineage-related perceptions...   \n",
       "169766  Identification of B.1.346 Lineage of SARS-CoV-...   \n",
       "483453  A human antibody with blocking activity to RBD...   \n",
       "452322  Effect of the new SARS-CoV-2 variant B.1.1.7 o...   \n",
       "96842   Sensitivity of infectious SARS-CoV-2 B.1.1.7 a...   \n",
       "454903  Epidemiological and Clinical Characteristics  ...   \n",
       "462424  Phylogenetic analyses of SARS-CoV-2 B.1.1.7 li...   \n",
       "279515  Detection of B.1.351 SARS-CoV-2 Variant Strain...   \n",
       "529853  Computational assessment of the spike protein ...   \n",
       "424949  SARS-CoV-2 Variant of Concern 202 012 01  B.1....   \n",
       "219772  Sixteen novel lineages of SARS-CoV-2 in South ...   \n",
       "331889  Detection and Genome Sequencing of SARS-CoV-2 ...   \n",
       "485445  Immunodominant B cell epitope in SARS-CoV-2 RB...   \n",
       "\n",
       "                                      doi           pmcid        pubmed_id  \\\n",
       "286738                                                                       \n",
       "247931                                                                       \n",
       "16254             doi:10.1017/ice.2021.59  pmc:PMC7948098  pubmed:33557960   \n",
       "420424  doi:10.1016/s1473-3099(21)00170-5  pmc:PMC8041359  pubmed:33857406   \n",
       "395962     doi:10.1016/j.cell.2021.03.052  pmc:PMC8009040  pubmed:33861950   \n",
       "96776          doi:10.15585/mmwr.mm7008e2                  pubmed:33630820   \n",
       "477204     doi:10.1016/j.ijid.2021.02.073  pmc:PMC7895695  pubmed:33618008   \n",
       "447927    doi:10.1101/2021.01.19.21250111                                    \n",
       "169766                                                                       \n",
       "483453      doi:10.1101/2021.02.07.429299                                    \n",
       "452322  doi:10.1016/s2352-4642(21)00030-4  pmc:PMC7906637  pubmed:33581054   \n",
       "96842      doi:10.1038/s41591-021-01318-5                  pubmed:33772244   \n",
       "454903    doi:10.1101/2021.01.06.20249091                                    \n",
       "462424      doi:10.1101/2021.01.12.426373                                    \n",
       "279515                                                                       \n",
       "529853      doi:10.1101/2021.03.25.437035                                    \n",
       "424949            doi:10.1093/jtm/taab032  pmc:PMC7989411  pubmed:33693926   \n",
       "219772                                                                       \n",
       "331889                                                                       \n",
       "485445    doi:10.1101/2021.03.11.21253399                                    \n",
       "\n",
       "          license                                           abstract  \\\n",
       "286738        unk  Information on severe acute respiratory syndro...   \n",
       "247931        unk  A newly emerged strain of SARS-CoV-2 of B.1.1....   \n",
       "16254       no-cc                                                      \n",
       "420424      no-cc  BACKGROUND: Emergence of variants with specifi...   \n",
       "395962  els-covid  The highly transmissible B.1.1.7 variant of SA...   \n",
       "96776         unk  The first laboratory-confirmed cases of corona...   \n",
       "477204  els-covid  Objectives To evaluate the genomic diversity a...   \n",
       "447927    medrxiv  Background: Healthcare workers'  HCWs'  travel...   \n",
       "169766        unk  SARS-CoV-2 whole-genome sequencing of samples ...   \n",
       "483453    biorxiv  Severe acute respiratory syndrome coronavirus-...   \n",
       "452322      no-cc                                                      \n",
       "96842         unk  Severe acute respiratory syndrome coronavirus ...   \n",
       "454903    medrxiv  Background: The earliest coronavirus disease-2...   \n",
       "462424    biorxiv  The authors have withdrawn this manuscript bec...   \n",
       "279515        unk  The first laboratory-confirmed cases of corona...   \n",
       "529853    biorxiv  Since its emergence into the human population ...   \n",
       "424949      no-cc  We report the complete genome of SARS-CoV-2 VO...   \n",
       "219772        unk  The first severe acute respiratory syndrome co...   \n",
       "331889        unk  Since the emergence of coronavirus disease  CO...   \n",
       "485445    medrxiv  Recent SARS-CoV-2 variants pose important conc...   \n",
       "\n",
       "       publish_time                                            authors  \\\n",
       "286738         2021  Sander, Anna-Lena; Yadouleton, Anges; Moreira-...   \n",
       "247931         2021  Hussain, Mushtaq; Shabbir, Sanya; Amanullah, A...   \n",
       "16254    2021-02-09               Kow, Chia Siang; Hasan, Syed Shahzad   \n",
       "420424   2021-04-12  Frampton, Dan; Rampling, Tommy; Cross, Aidan; ...   \n",
       "395962   2021-03-30  Washington, Nicole L.; Gangavarapu, Karthik; Z...   \n",
       "96776    2021-02-26  Mwenda, Mulenga; Saasa, Ngonda; Sinyange, Nyam...   \n",
       "477204   2021-02-20  Muñoz, Marina; Patiño, Luz H.; Ballesteros, Na...   \n",
       "447927   2021-01-21  Temsah, M.-H.; Barry, M.; Aljamaan, F.; Alhuza...   \n",
       "169766         2021  Abe, Kodai; Shimura, Takako; Takenouchi, Toshi...   \n",
       "483453   2021-02-08  Gu, Chunyin; Cao, Xiaodan; Wang, Zongda; Hu, X...   \n",
       "452322   2021-02-10  Brookman, Sarah; Cook, James; Zucherman, Mark;...   \n",
       "96842    2021-03-26  Planas, Delphine; Bruel, Timothée; Grzelak, Lu...   \n",
       "454903   2021-01-08  Yegorov, S.; Goremykina, M.; Ivanova, R.; Good...   \n",
       "462424   2021-01-15  Vrancken, B.; Dellicour, S.; Smith, D.M.; Chai...   \n",
       "279515         2021  Mwenda, Mulenga; Saasa, Ngonda; Sinyange, Nyam...   \n",
       "529853   2021-03-25        Ge, Anni; Rioux, Melissa; Kelvin, Alyson A.   \n",
       "424949   2021-03-08  Yao, Mingxiao; Jiang, Fachun; Zhang, Yuwei; Li...   \n",
       "219772         2021  Tegally, Houriiyah; Wilkinson, Eduan; Lessells...   \n",
       "331889         2021  Klaus, Julia; Meli, Marina L; Willi, Barbara; ...   \n",
       "485445   2021-03-12  Santos, K. S.; Oliveira, J. R.; Machado, R. R....   \n",
       "\n",
       "                                            journal mag_id who_covidence_id  \\\n",
       "286738                                      MSphere                #1029965   \n",
       "247931                                J. med. virol                #1126508   \n",
       "16254                                                                         \n",
       "420424                            Lancet Infect Dis                           \n",
       "395962                                         Cell                           \n",
       "96776   MMWR. Morbidity and mortality weekly report                           \n",
       "477204                             Int J Infect Dis                           \n",
       "447927                                                                        \n",
       "169766                                  Keio j. med                #1183786   \n",
       "483453                                      bioRxiv                           \n",
       "452322                  Lancet Child Adolesc Health                           \n",
       "96842                               Nature medicine                           \n",
       "454903                                                                        \n",
       "462424                                      bioRxiv                           \n",
       "279515                    MMWR Morb Mortal Wkly Rep                #1102701   \n",
       "529853                                      bioRxiv                           \n",
       "424949                                 J Travel Med                           \n",
       "219772                                      Nat Med                #1061182   \n",
       "331889                                      Viruses                #1138760   \n",
       "485445                                                                        \n",
       "\n",
       "       arxiv_id                                     pdf_json_files  \\\n",
       "286738                                                               \n",
       "247931                                                               \n",
       "16254            document_parses/pdf_json/cc129c4cd6dc4523658db...   \n",
       "420424           document_parses/pdf_json/69fa13b80bb407e8bca17...   \n",
       "395962           document_parses/pdf_json/3966d03d72440c939928b...   \n",
       "96776                                                                \n",
       "477204           document_parses/pdf_json/5adfe79458623b8b71e1c...   \n",
       "447927                                                               \n",
       "169766                                                               \n",
       "483453           document_parses/pdf_json/38be32b65618c66a2e3d4...   \n",
       "452322           document_parses/pdf_json/e49ae48fbdccbf290c84f...   \n",
       "96842                                                                \n",
       "454903                                                               \n",
       "462424                                                               \n",
       "279515                                                               \n",
       "529853           document_parses/pdf_json/c5bf135c660e1a18623d3...   \n",
       "424949           document_parses/pdf_json/fbf9052e9c20760142174...   \n",
       "219772                                                               \n",
       "331889                                                               \n",
       "485445           document_parses/pdf_json/5fc2ca54b6535130b999a...   \n",
       "\n",
       "                                      pmc_json_files  \\\n",
       "286738                                                 \n",
       "247931                                                 \n",
       "16254   document_parses/pmc_json/PMC7948098.xml.json   \n",
       "420424  document_parses/pmc_json/PMC8041359.xml.json   \n",
       "395962                                                 \n",
       "96776                                                  \n",
       "477204                                                 \n",
       "447927                                                 \n",
       "169766                                                 \n",
       "483453                                                 \n",
       "452322  document_parses/pmc_json/PMC7906637.xml.json   \n",
       "96842                                                  \n",
       "454903                                                 \n",
       "462424                                                 \n",
       "279515                                                 \n",
       "529853                                                 \n",
       "424949                                                 \n",
       "219772                                                 \n",
       "331889                                                 \n",
       "485445                                                 \n",
       "\n",
       "                                                      url      s2_id  year  \\\n",
       "286738                                                     231605197         \n",
       "247931                                                     232196602         \n",
       "16254   https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7...             2021   \n",
       "420424  https://www.sciencedirect.com/science/article/...  233215323  2021   \n",
       "395962  https://www.ncbi.nlm.nih.gov/pubmed/33861950/;...  232412011  2021   \n",
       "96776   https://doi.org/10.15585/mmwr.mm7008e2; https:...  232058475  2021   \n",
       "477204  https://api.elsevier.com/content/article/pii/S...  231965833  2021   \n",
       "447927  http://medrxiv.org/cgi/content/short/2021.01.1...  231653830  2021   \n",
       "169766                                                     231729271         \n",
       "483453          https://doi.org/10.1101/2021.02.07.429299  231885348  2021   \n",
       "452322  https://api.elsevier.com/content/article/pii/S...  231919047  2021   \n",
       "96842   https://doi.org/10.1038/s41591-021-01318-5; ht...  232375717  2021   \n",
       "454903  https://doi.org/10.1101/2021.01.06.20249091; h...  231138384  2021   \n",
       "462424          https://doi.org/10.1101/2021.01.12.426373  231666904  2021   \n",
       "279515                                                     232058475         \n",
       "529853          https://doi.org/10.1101/2021.03.25.437035  232412338  2021   \n",
       "424949  https://doi.org/10.1093/jtm/taab032; https://w...  232197534  2021   \n",
       "219772                                                     231788423         \n",
       "331889                                                     232383967         \n",
       "485445  https://doi.org/10.1101/2021.03.11.21253399; h...  232204178  2021   \n",
       "\n",
       "             date                       lineages  \\\n",
       "286738 2021-04-23                        A.4;B.1   \n",
       "247931 2021-04-23                        B.1.1.7   \n",
       "16254  2021-02-09                        B.1.1.7   \n",
       "420424 2021-04-12                        B.1.1.7   \n",
       "395962 2021-03-30                        B.1.1.7   \n",
       "96776  2021-02-26                        B.1.351   \n",
       "477204 2021-02-20            B.1.351;B.1.1.7;P.1   \n",
       "447927 2021-01-21                        B.1.1.7   \n",
       "169766 2021-04-23    B.1.1.284;B.1.1.214;B.1.346   \n",
       "483453 2021-02-08                        B.1.351   \n",
       "452322 2021-02-10                        B.1.1.7   \n",
       "96842  2021-03-26                B.1.351;B.1.1.7   \n",
       "454903 2021-01-08      B.4.1;B.1.3;B.1.1;B.1;A.2   \n",
       "462424 2021-01-15                        B.1.1.7   \n",
       "279515 2021-04-23                        B.1.351   \n",
       "529853 2021-03-25    B.1.351;B.1.1.7;B.1.429;P.1   \n",
       "424949 2021-03-08                        B.1.1.7   \n",
       "219772 2021-04-23  B.1.1.54;B.1.1.56;C.1;B.1.106   \n",
       "331889 2021-04-23                       B.1.1.39   \n",
       "485445 2021-03-12                    B.1.351;P.1   \n",
       "\n",
       "                                       id  \n",
       "286738                                     \n",
       "247931                                     \n",
       "16254             doi:10.1017/ice.2021.59  \n",
       "420424  doi:10.1016/s1473-3099(21)00170-5  \n",
       "395962     doi:10.1016/j.cell.2021.03.052  \n",
       "96776          doi:10.15585/mmwr.mm7008e2  \n",
       "477204     doi:10.1016/j.ijid.2021.02.073  \n",
       "447927    doi:10.1101/2021.01.19.21250111  \n",
       "169766                                     \n",
       "483453      doi:10.1101/2021.02.07.429299  \n",
       "452322  doi:10.1016/s2352-4642(21)00030-4  \n",
       "96842      doi:10.1038/s41591-021-01318-5  \n",
       "454903    doi:10.1101/2021.01.06.20249091  \n",
       "462424      doi:10.1101/2021.01.12.426373  \n",
       "279515                                     \n",
       "529853      doi:10.1101/2021.03.25.437035  \n",
       "424949            doi:10.1093/jtm/taab032  \n",
       "219772                                     \n",
       "331889                                     \n",
       "485445    doi:10.1101/2021.03.11.21253399  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hits.sample(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "WHO documents seem to be copies of articles that are already present in the dataset and will be ignored for now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cord_uid</th>\n",
       "      <th>sha</th>\n",
       "      <th>source_x</th>\n",
       "      <th>title</th>\n",
       "      <th>doi</th>\n",
       "      <th>pmcid</th>\n",
       "      <th>pubmed_id</th>\n",
       "      <th>license</th>\n",
       "      <th>abstract</th>\n",
       "      <th>publish_time</th>\n",
       "      <th>authors</th>\n",
       "      <th>journal</th>\n",
       "      <th>mag_id</th>\n",
       "      <th>who_covidence_id</th>\n",
       "      <th>arxiv_id</th>\n",
       "      <th>pdf_json_files</th>\n",
       "      <th>pmc_json_files</th>\n",
       "      <th>url</th>\n",
       "      <th>s2_id</th>\n",
       "      <th>year</th>\n",
       "      <th>date</th>\n",
       "      <th>lineages</th>\n",
       "      <th>id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>278446</th>\n",
       "      <td>84bhzaqw</td>\n",
       "      <td></td>\n",
       "      <td>WHO</td>\n",
       "      <td>SARS-CoV-2 spread across the Colombian-Venezue...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>unk</td>\n",
       "      <td>INTRODUCTION: Venezuela and Colombia both adop...</td>\n",
       "      <td>2020</td>\n",
       "      <td>Paniz-Mondolfi, Alberto; Muñoz, Marina; Florez...</td>\n",
       "      <td>Infect Genet Evol</td>\n",
       "      <td></td>\n",
       "      <td>#907154</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>220444484</td>\n",
       "      <td></td>\n",
       "      <td>2020-04-23</td>\n",
       "      <td>B.1.13</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>518529</th>\n",
       "      <td>0vtredu8</td>\n",
       "      <td>d2d3b8b04378bebdde024760e49c297a24dd2af7</td>\n",
       "      <td>Elsevier; Medline; PMC</td>\n",
       "      <td>SARS-CoV-2 spread across the Colombian-Venezue...</td>\n",
       "      <td>doi:10.1016/j.meegid.2020.104616</td>\n",
       "      <td>pmc:PMC7609240</td>\n",
       "      <td>pubmed:33157300</td>\n",
       "      <td>no-cc</td>\n",
       "      <td>INTRODUCTION: Venezuela and Colombia both adop...</td>\n",
       "      <td>2020-11-04</td>\n",
       "      <td>Paniz-Mondolfi, Alberto; Muñoz, Marina; Florez...</td>\n",
       "      <td>Infect Genet Evol</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>document_parses/pdf_json/d2d3b8b04378bebdde024...</td>\n",
       "      <td>document_parses/pmc_json/PMC7609240.xml.json</td>\n",
       "      <td>https://www.ncbi.nlm.nih.gov/pubmed/33157300/;...</td>\n",
       "      <td>226238553</td>\n",
       "      <td>2020</td>\n",
       "      <td>2020-11-04</td>\n",
       "      <td>B.1.13</td>\n",
       "      <td>doi:10.1016/j.meegid.2020.104616</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        cord_uid                                       sha  \\\n",
       "278446  84bhzaqw                                             \n",
       "518529  0vtredu8  d2d3b8b04378bebdde024760e49c297a24dd2af7   \n",
       "\n",
       "                      source_x  \\\n",
       "278446                     WHO   \n",
       "518529  Elsevier; Medline; PMC   \n",
       "\n",
       "                                                    title  \\\n",
       "278446  SARS-CoV-2 spread across the Colombian-Venezue...   \n",
       "518529  SARS-CoV-2 spread across the Colombian-Venezue...   \n",
       "\n",
       "                                     doi           pmcid        pubmed_id  \\\n",
       "278446                                                                      \n",
       "518529  doi:10.1016/j.meegid.2020.104616  pmc:PMC7609240  pubmed:33157300   \n",
       "\n",
       "       license                                           abstract  \\\n",
       "278446     unk  INTRODUCTION: Venezuela and Colombia both adop...   \n",
       "518529   no-cc  INTRODUCTION: Venezuela and Colombia both adop...   \n",
       "\n",
       "       publish_time                                            authors  \\\n",
       "278446         2020  Paniz-Mondolfi, Alberto; Muñoz, Marina; Florez...   \n",
       "518529   2020-11-04  Paniz-Mondolfi, Alberto; Muñoz, Marina; Florez...   \n",
       "\n",
       "                  journal mag_id who_covidence_id arxiv_id  \\\n",
       "278446  Infect Genet Evol                 #907154            \n",
       "518529  Infect Genet Evol                                    \n",
       "\n",
       "                                           pdf_json_files  \\\n",
       "278446                                                      \n",
       "518529  document_parses/pdf_json/d2d3b8b04378bebdde024...   \n",
       "\n",
       "                                      pmc_json_files  \\\n",
       "278446                                                 \n",
       "518529  document_parses/pmc_json/PMC7609240.xml.json   \n",
       "\n",
       "                                                      url      s2_id  year  \\\n",
       "278446                                                     220444484         \n",
       "518529  https://www.ncbi.nlm.nih.gov/pubmed/33157300/;...  226238553  2020   \n",
       "\n",
       "             date lineages                                id  \n",
       "278446 2020-04-23   B.1.13                                    \n",
       "518529 2020-11-04   B.1.13  doi:10.1016/j.meegid.2020.104616  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hits[hits['abstract'].str.contains('INTRODUCTION: Venezuela and')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "hits.query('id != \"\"', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of matches 449\n"
     ]
    }
   ],
   "source": [
    "print(\"Total number of matches\", hits.shape[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "hits.to_csv(NEO4J_IMPORT / \"01h-CORDLineages.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}