DTW_All_boxplots.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This program creates box plots for the output from the TIME web application's Workflow 5b, which caluclates uses their custom Dynamic Time Warping (DTW) algorithm to calculate the pairwise DTW distance between bacterial taxa. Dynamic Time Warping is a measure of similarity in longitudinal data, and the TIME version of the algorithm ranges from 0 to 1. 0 is the most similar, and 1 is the most different. For more on the algorithm, see the relevant research paper [here](https://www.frontiersin.org/articles/10.3389/fmicb.2018.00036/full).\n",
    "<br>\n",
    "\n",
    "For the CF data, I used the following settings: a taxonomic level of 'Genus,' the default constraint of 2 and a filter for rare taxa of 0.1 (because the input data was already filtered). The input file was not rarefied or normalised in the intitial upload. In the programs Create_TDTW_all_filtered and Create_TDTW_all_example I created csv files for Workflow 5b output data. Now it's time to plot them, with a specific focus on 3 OTU's (operational taxonomic units) of particular interest to our CF research. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This program is written for use with the CF data, and uses the output file from Create_TDTW_all_filtered, which is called TDTW_all_filtered.csv. To make it possible for anyone to run it I have included in this repository an altered version of that file called TDTW_all_filtered2.csv, using randomly generated values. If you are interested in seeing how I created TDTW_all_filtered2.csv, please email me at vtalbot@lesley.edu or vrtalbot@yahoo.com and I will be happy to send you the program.  \n",
    "<br>\n",
    "\n",
    "Users also have the option of doing everything themselves in making the box plots, from running the TIME workflow on their example [repeated antibiotic perturbation](https://web.rniapps.net/time/index.php) data to creating the merged file in Create_TDTW_all_example and making the box plots in DTW_All_boxplots_example. To learn more about that data, click [here to view the antibiotic research paper](https://www.pnas.org/content/108/Supplement_1/4554.long). The program Create_TDTW_all_example also produces input for DTW_boxplots_by_status, which does not have a counterpart for the CF data and uses the antibiotic example data as well. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code may easily be adapted to create box plots for other types of data. It plots a selection of columns from the table rather than grouping by a factor variable (See DTW_boxplots_by_status for box plots which *are* grouped by a factor variable). It also extracts names of bacteria from much longer strings. Finally, as mentioned above, see DTW_All_boxplots_example to run a version of this program using the example file generated in Create_TDTW_all_example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#import necessary libraries\n",
    "import pandas as pd\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#make a list of the OTU's which are of special interest to us, which we wish to plot\n",
    "bacteria=['Pseudomonas','Staphylococcus','Streptococcus']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 1: Focus on 3 OTU's\n",
    "The first few cells modify the main data set created in Create_TDTW_all_filtered to focus on only 3 OTU's, creating a separate file for each of the three which shows the distance from the other bacteria to that one bacteria. You can, of course, easily edit this to focus on any number of selected taxa. \n",
    "\n",
    "<br>\n",
    "In Section 2 there are two versions of the loops that plot the data for both styles of plot. The first version both creates and provides the option to save the filtered files which we set up here in Section 1, and the second version reads them in to plot them. This means that you can run Section 1 every time you use this program and use the first version of either style of plot without saving, or you can save the filtered files as directed when running the first version of either style of plot, then skip this section on the next run and go straight to Section 2 to use the second versions of either style."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#read in the file, created in the program Create_TDTW_all_filtered, which contains all the output from Workflow 5b\n",
    "#here I am using a randomly generated version of TDTW_all_filtered.csv called TDTW_all_filtered2.csv instead\n",
    "df = pd.read_csv(\"Data/TDTW_all_filtered2.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#rename the first column, which has no name, to 'Sample' \n",
    "df.rename(columns={'Unnamed: 0': 'Sample'}, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#extract the participant's clinical status from the sample identifier, and change to title case\n",
    "#I used lower case when I created the sample identifiers because they were easier to read that way\n",
    "df['Status']=df['Sample'].str.split('_').str[2].str.title()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#extract the participant's ID number from the identifier\n",
    "df['ID']=df['Sample'].str.split('_').str[1]\n",
    "#review the changes to confirm that they worked\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What the new columns should look like:\n",
    "<img src='https://imgur.com/KU0nqUf.png' style='height:175px'>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#create a separate data frame for just those entries with status 'All'\n",
    "df2=df[df['Status'] == 'All']\n",
    "#view the head of the new table if desired\n",
    "#df2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that in our CF input data for TIME, taxonomic names are preceded by double underscores at all levels.\n",
    "<br>\n",
    "\n",
    "Example: D_0__Bacteria;D_1__Actinobacteria;D_2__Actinobacteria;D_3__Actinomycetales;D_4__Actinomycetaceae;D_5__Actinomyces\n",
    "<br>\n",
    "\n",
    "This generates output for the genus level with Actinomyces preceded by double underscores in every workflow.\n",
    "We left the underscores in as separators when we merged the output files from TIME in Create_TDTW_all_filtered. The example data lacked these underscores, so we chose to add a single underscore as a separator between bacteria names in Create_TDTW_all_example. This means that, if you are editing this program to run it on the other example data or your own data, the code below may need to be modified to reflect any formatting differences. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#make a list of columns (pairs of OTU's) for only the 3 OTU's of interest, excluding the ones that pair them with themselves \n",
    "clist=[]\n",
    "for i in range(len(bacteria)):\n",
    "    #if you are editing this code to use on your own data, here you may need a minor edit to relect formatting differences\n",
    "    #viewing the head of df or df2 above should tell you if and how you need to change the str.format() function\n",
    "    j=[col for col in df.columns if col.startswith('__{}'.format(bacteria[i])) \n",
    "       and col!= '__{}__{}'.format(bacteria[i],bacteria[i])]\n",
    "    clist.append(j)                                              \n",
    "#view the list, if desired\n",
    "#print(clist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 2: Create the Plots"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I included the code in all of my different plotting programs for GP Microbiome output the creation of keys for bacteria names. The keys pair OTU ID numbers with names of bacteria. The file 'OTU_key_named_selection.csv' contains only the bacteria of particular interest to the CF study. 'OTU_key_named_selection.xlsx' is that file, with information added about the NCBI taxonomic ID and how we want the box plot to look, saved as an Excel workbook. \n",
    "<br>\n",
    "\n",
    "All of my plotting functions save the plots to a folder called 'Plots,' which is in this repository as well. Adjust the file path if you want to save them somewhere else, or comment out the line of code which saves them if you prefer not to save."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For quick basic plots without including the NCBI taxonomic ID's or specifying the order, skip to the last two cells. To specify the order but not the ID number, simply comment out the line in the loop which renames the columns and adjust the figsize as directed in the comments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plots with Taxonomic ID's"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#read in the file and take a look at the columns\n",
    "key=pd.read_excel(\"Data/OTU_key_named_selection.xlsx\")\n",
    "key.head()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#create a dictionary for the taxonomic ID's\n",
    "taxid_key=dict(zip(key['Name'], key['NCBI:txid']))\n",
    "#view dictionary, if desired\n",
    "#taxid_key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#sort by plotting order for those OTU's which are included in the plots\n",
    "taxa=key[key['in_boxplot']=='Y'].sort_values('order for plotting')\n",
    "#view the new data frame\n",
    "taxa.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### First Version\n",
    "You can use this version of the plotting loop if you ran Section 1. Before plotting the data, it filters the data frames and provides the opportunity to save them to csv files so you can skip Section 1 in the future. To save the files, just un-comment the line which does so where indicated. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(len(bacteria)):\n",
    "    df3=df2.filter(clist[i], axis=1)\n",
    "    #since each data frame is for the distance from one bacteria, change the column names to be just the second bacteria\n",
    "    #this code may need adjustment depending on how many underscores are in your bacteria names\n",
    "    df3.columns=[w.replace('__{}__'.format(bacteria[i]),'') for w in df3.columns]\n",
    "    #optional: un-comment the next line to save file if you want to skip Section 1 in the future\n",
    "    #df3.to_csv(\"Data/TDTW_filtered_{}.csv\".format(bacteria[i]), index=False)\n",
    "    #to keep those with at least 4 non-NaN's:\n",
    "    #df3.dropna(thresh=4, axis=1, inplace=True)\n",
    "    df3.dropna(axis=1, inplace=True)\n",
    "    order=list(taxa['Name'][taxa['Name']!=bacteria[i]])\n",
    "    df3=df3.reindex(columns=order)\n",
    "    df3.columns=[col+' NCBI:taxid {}'.format(str(taxid_key[col])) for col in df3.columns]\n",
    "    plt.figure(figsize=(12,14))\n",
    "    #to exclude the taxid, comment out the previous 2 lines and adjust the figsize as follows for better dimensions:\n",
    "    #plt.figure(figsize=(12,10))   \n",
    "    ax=sns.boxplot(x=\"variable\", y=\"value\", data=pd.melt(df3))\n",
    "    plt.setp(ax.get_xticklabels(), rotation=90, size=18, style='italic')\n",
    "    plt.setp(ax.get_yticklabels(), size=16)\n",
    "    plt.title('DTW Distance from {}'.format(bacteria[i]), size=24)\n",
    "    plt.xlabel('')\n",
    "    plt.ylabel('Distance',size=20)\n",
    "    plt.tight_layout()\n",
    "    #save plot if desired - change file path if you want to put it in its own folder\n",
    "    plt.savefig(\"Plots/{}_taxid.png\".format(bacteria[i]), format = \"png\")\n",
    "    #if you set the order but excluded the taxid:\n",
    "    #plt.savefig(\"Plots/{}_ordered.png\".format(bacteria[i]), format = \"png\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Second Version\n",
    "This plotting loop reads in csv files with the filtered versions of the data frames if they have already been created and saved by running the first version of the plotting loop for either style of plot with the line that saves them un-commented."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#loop for if the files have already been created\n",
    "for i in range(len(bacteria)):\n",
    "    #read in the file\n",
    "    df = pd.read_csv(\"Data/TDTW_filtered_{}.csv\".format(bacteria[i]))\n",
    "    df.dropna(axis=1, inplace=True)\n",
    "    order=list(taxa['Name'][taxa['Name']!=bacteria[i]])\n",
    "    df=df.reindex(columns=order)\n",
    "    df.columns=[col+' NCBI:taxid {}'.format(str(taxid_key[col])) for col in df.columns]\n",
    "    plt.figure(figsize=(12,14))\n",
    "    #to exclude the taxid, comment out the previous 2 lines and adjust the figsize as follows for better dimensions:\n",
    "    #plt.figure(figsize=(12,10))  \n",
    "    ax=sns.boxplot(x=\"variable\", y=\"value\", data=pd.melt(df))\n",
    "    plt.setp(ax.get_xticklabels(), rotation=90, size=18, style='italic')\n",
    "    plt.setp(ax.get_yticklabels(), size=16)\n",
    "    plt.title('DTW Distance from {}'.format(bacteria[i]), size=24)\n",
    "    plt.xlabel('')\n",
    "    plt.ylabel('Distance',size=20)\n",
    "    plt.tight_layout()\n",
    "    #save plot if desired - change file path if you want to put it in its own folder\n",
    "    plt.savefig(\"Plots/{}_taxid.png\".format(bacteria[i]), format = \"png\")\n",
    "    #if you set the order but excluded the taxid:\n",
    "    #plt.savefig(\"Plots/{}_ordered.png\".format(bacteria[i]), format = \"png\")\n",
    "    plt.show() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a mini preview of what the three image files from the plots with taxonomic ID's will look like.\n",
    "<table><tr>\n",
    "<td> <img src='https://imgur.com/zBe69B6.png' style='height:400px'></td>\n",
    "<td><img src='https://imgur.com/CY5bF8w.png' style='height:400px'></td>\n",
    "<td><img src='https://imgur.com/3U5Iipa.png' style='height:400px'></td>\n",
    "</tr></table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is what they look like in order, but without the taxonomic IDs.\n",
    "<table><tr>\n",
    "<td> <img src='https://imgur.com/l8R5puP.png' style='height:310px'></td>\n",
    "<td><img src='https://imgur.com/9L9KYcJ.png' style='height:310px'></td>\n",
    "<td><img src='https://imgur.com/sXvRvYa.png' style='height:310px'></td>\n",
    "</tr></table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quick basic plots\n",
    "Please note that these plots do not specify the order of the bacteria. To specify the order, go back to the beginning of Section 2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### First Version\n",
    "You can use this version of the plotting loop if you ran Section 1. Before plotting the data, it filters the data frames and provides the opportunity to save them to csv files so you can skip Section 1 in the future. To save the files, just un-comment the line which does so where indicated. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(len(bacteria)):\n",
    "    df3=df2.filter(clist[i], axis=1)\n",
    "    #since each data frame is for the distance from one bacteria, change the column names to be just the second bacteria\n",
    "    #this code may need adjustment depending on how many underscores are in your bacteria names    \n",
    "    df3.columns=[w.replace('__{}__'.format(bacteria[i]),'') for w in df3.columns]\n",
    "    #optional: un-comment the next line to save file if you want to skip Section 1 in the future\n",
    "    #df3.to_csv(\"Data/TDTW_filtered_{}.csv\".format(bacteria[i]), index=False)\n",
    "    #to keep those with at least 4 non-NaN's:\n",
    "    #df3.dropna(thresh=4, axis=1, inplace=True)\n",
    "    df3.dropna(axis=1, inplace=True)\n",
    "    plt.figure(figsize=(12,10))\n",
    "    ax=sns.boxplot(x=\"variable\", y=\"value\", data=pd.melt(df3))\n",
    "    plt.setp(ax.get_xticklabels(), rotation=90, size=18, style='italic')\n",
    "    plt.setp(ax.get_yticklabels(), size=16)\n",
    "    plt.title('DTW Distance from {}'.format(bacteria[i]), size=24)\n",
    "    plt.xlabel('')\n",
    "    plt.ylabel('Distance',size=20)\n",
    "    plt.tight_layout()\n",
    "    #save plot if desired - you might want to put it in its own folder\n",
    "    plt.savefig(\"Plots/{}_basic.png\".format(bacteria[i]), format = \"png\")\n",
    "    plt.show() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Second Version\n",
    "This plotting loop reads in csv files with the filtered versions of the data frames if they have already been created and saved by running the first version of the plotting loop for either style of plot with the line that saves them un-commented."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#loop for if the files have already been created\n",
    "for i in range(len(bacteria)):\n",
    "    #read in the file\n",
    "    df = pd.read_csv(\"Data/TDTW_filtered_{}.csv\".format(bacteria[i]))\n",
    "    df.dropna(axis=1, inplace=True)\n",
    "    plt.figure(figsize=(12,10))\n",
    "    ax=sns.boxplot(x=\"variable\", y=\"value\", data=pd.melt(df))\n",
    "    plt.setp(ax.get_xticklabels(), rotation=90, size=18, style='italic')\n",
    "    plt.setp(ax.get_yticklabels(), size=16)\n",
    "    plt.title('DTW Distance from {}'.format(bacteria[i]), size=24)\n",
    "    plt.xlabel('')\n",
    "    plt.ylabel('Distance',size=20)\n",
    "    plt.tight_layout()\n",
    "    #save plot if desired - you might want to put it in its own folder\n",
    "    plt.savefig(\"Plots/{}_basic.png\".format(bacteria[i]), format = \"png\")\n",
    "    plt.show() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a mini preview of what the three image files from the basic plots will look like.\n",
    "<table><tr>\n",
    "<td> <img src='https://imgur.com/3R6R0ki.png' style='height:310px'></td>\n",
    "<td><img src='https://imgur.com/2Ua3gKR.png' style='height:310px'></td>\n",
    "<td><img src='https://imgur.com/7JA3bQv.png' style='height:310px'></td>\n",
    "</tr></table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}