notebooks/Checking the modularization based on changes (3D Version).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "In my [previous blog post](https://www.feststelltaste.de/checking-the-modularization-of-software-systems-by-analyzing-co-changing-source-code-files/), we looked at the similarity within and across modules by only looking at the change data of each source code file.\n",
    "\n",
    "In this analysis, we use same data analysis approach, but visualize the result in a 3D scatter plot."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Wrangling\n",
    "We just repeat the stuff explained in the mentioned blog post. The only difference is, that we are going from a 2D representation of the distance matrix to a 3D representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>module</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>file</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>backend/src/main/java/at/dropover/comment/boundary/AddCommentRequestModel.java</th>\n",
       "      <td>0.204507</td>\n",
       "      <td>-0.500405</td>\n",
       "      <td>0.328621</td>\n",
       "      <td>comment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>backend/src/main/java/at/dropover/comment/boundary/ChangeCommentRequestModel.java</th>\n",
       "      <td>0.420250</td>\n",
       "      <td>-0.287899</td>\n",
       "      <td>0.239667</td>\n",
       "      <td>comment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>backend/src/main/java/at/dropover/comment/boundary/CommentData.java</th>\n",
       "      <td>0.384984</td>\n",
       "      <td>-0.418287</td>\n",
       "      <td>0.185055</td>\n",
       "      <td>comment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>backend/src/main/java/at/dropover/comment/boundary/GetCommentRequestModel.java</th>\n",
       "      <td>0.305661</td>\n",
       "      <td>-0.373012</td>\n",
       "      <td>0.299084</td>\n",
       "      <td>comment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>backend/src/main/java/at/dropover/comment/boundary/GetCommentResponseModel.java</th>\n",
       "      <td>0.176830</td>\n",
       "      <td>-0.580814</td>\n",
       "      <td>0.123574</td>\n",
       "      <td>comment</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                           x         y  \\\n",
       "file                                                                     \n",
       "backend/src/main/java/at/dropover/comment/bound...  0.204507 -0.500405   \n",
       "backend/src/main/java/at/dropover/comment/bound...  0.420250 -0.287899   \n",
       "backend/src/main/java/at/dropover/comment/bound...  0.384984 -0.418287   \n",
       "backend/src/main/java/at/dropover/comment/bound...  0.305661 -0.373012   \n",
       "backend/src/main/java/at/dropover/comment/bound...  0.176830 -0.580814   \n",
       "\n",
       "                                                           z   module  \n",
       "file                                                                   \n",
       "backend/src/main/java/at/dropover/comment/bound...  0.328621  comment  \n",
       "backend/src/main/java/at/dropover/comment/bound...  0.239667  comment  \n",
       "backend/src/main/java/at/dropover/comment/bound...  0.185055  comment  \n",
       "backend/src/main/java/at/dropover/comment/bound...  0.299084  comment  \n",
       "backend/src/main/java/at/dropover/comment/bound...  0.123574  comment  "
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from sklearn.metrics.pairwise import cosine_distances\n",
    "from sklearn.manifold import MDS\n",
    "import numpy as np\n",
    "from matplotlib import cm\n",
    "from matplotlib.colors import rgb2hex\n",
    "import ipyvolume as ipv\n",
    "\n",
    "# read, filter and prepare data\n",
    "git_log = pd.read_csv(\"https://git.io/Jez2h\")\n",
    "prod_code = git_log.copy()\n",
    "prod_code = prod_code[prod_code.file.str.endswith(\".java\")]\n",
    "prod_code = prod_code[prod_code.file.str.startswith(\"backend/src/main\")]\n",
    "prod_code = prod_code[~prod_code.file.str.endswith(\"package-info.java\")]\n",
    "prod_code['hit'] = 1\n",
    "\n",
    "# pivot table to get a change vector per file\n",
    "commit_matrix = prod_code.reset_index().pivot_table(\n",
    "    index='file',\n",
    "    columns='sha',\n",
    "    values='hit',\n",
    "    fill_value=0)\n",
    "commit_matrix.iloc[0:5,50:55]\n",
    "\n",
    "# calculate distance between files based on changes\n",
    "dissimilarity_matrix = cosine_distances(commit_matrix)\n",
    "\n",
    "# break down matrix to 3D representation\n",
    "model = MDS(dissimilarity='precomputed', random_state=0, n_components=3)\n",
    "dissimilarity_3d = model.fit_transform(dissimilarity_matrix)\n",
    "\n",
    "# extract module names\n",
    "dissimilarity_3d_df = pd.DataFrame(\n",
    "    dissimilarity_3d,\n",
    "    index=commit_matrix.index,\n",
    "    columns=[\"x\", \"y\", \"z\"])\n",
    "dissimilarity_3d_df['module'] = dissimilarity_3d_df.index.str.split(\"/\").str[6]\n",
    "dissimilarity_3d_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualization\n",
    "So this part is new: We brew a color for each module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>module</th>\n",
       "      <th>color</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>file</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>backend/src/main/java/at/dropover/comment/boundary/AddCommentRequestModel.java</th>\n",
       "      <td>0.204507</td>\n",
       "      <td>-0.500405</td>\n",
       "      <td>0.328621</td>\n",
       "      <td>comment</td>\n",
       "      <td>[0.6196078431372549, 0.00392156862745098, 0.25...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>backend/src/main/java/at/dropover/comment/boundary/ChangeCommentRequestModel.java</th>\n",
       "      <td>0.420250</td>\n",
       "      <td>-0.287899</td>\n",
       "      <td>0.239667</td>\n",
       "      <td>comment</td>\n",
       "      <td>[0.6196078431372549, 0.00392156862745098, 0.25...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>backend/src/main/java/at/dropover/comment/boundary/CommentData.java</th>\n",
       "      <td>0.384984</td>\n",
       "      <td>-0.418287</td>\n",
       "      <td>0.185055</td>\n",
       "      <td>comment</td>\n",
       "      <td>[0.6196078431372549, 0.00392156862745098, 0.25...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>backend/src/main/java/at/dropover/comment/boundary/GetCommentRequestModel.java</th>\n",
       "      <td>0.305661</td>\n",
       "      <td>-0.373012</td>\n",
       "      <td>0.299084</td>\n",
       "      <td>comment</td>\n",
       "      <td>[0.6196078431372549, 0.00392156862745098, 0.25...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>backend/src/main/java/at/dropover/comment/boundary/GetCommentResponseModel.java</th>\n",
       "      <td>0.176830</td>\n",
       "      <td>-0.580814</td>\n",
       "      <td>0.123574</td>\n",
       "      <td>comment</td>\n",
       "      <td>[0.6196078431372549, 0.00392156862745098, 0.25...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                           x         y  \\\n",
       "file                                                                     \n",
       "backend/src/main/java/at/dropover/comment/bound...  0.204507 -0.500405   \n",
       "backend/src/main/java/at/dropover/comment/bound...  0.420250 -0.287899   \n",
       "backend/src/main/java/at/dropover/comment/bound...  0.384984 -0.418287   \n",
       "backend/src/main/java/at/dropover/comment/bound...  0.305661 -0.373012   \n",
       "backend/src/main/java/at/dropover/comment/bound...  0.176830 -0.580814   \n",
       "\n",
       "                                                           z   module  \\\n",
       "file                                                                    \n",
       "backend/src/main/java/at/dropover/comment/bound...  0.328621  comment   \n",
       "backend/src/main/java/at/dropover/comment/bound...  0.239667  comment   \n",
       "backend/src/main/java/at/dropover/comment/bound...  0.185055  comment   \n",
       "backend/src/main/java/at/dropover/comment/bound...  0.299084  comment   \n",
       "backend/src/main/java/at/dropover/comment/bound...  0.123574  comment   \n",
       "\n",
       "                                                                                                color  \n",
       "file                                                                                                   \n",
       "backend/src/main/java/at/dropover/comment/bound...  [0.6196078431372549, 0.00392156862745098, 0.25...  \n",
       "backend/src/main/java/at/dropover/comment/bound...  [0.6196078431372549, 0.00392156862745098, 0.25...  \n",
       "backend/src/main/java/at/dropover/comment/bound...  [0.6196078431372549, 0.00392156862745098, 0.25...  \n",
       "backend/src/main/java/at/dropover/comment/bound...  [0.6196078431372549, 0.00392156862745098, 0.25...  \n",
       "backend/src/main/java/at/dropover/comment/bound...  [0.6196078431372549, 0.00392156862745098, 0.25...  "
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "modules = dissimilarity_3d_df[['module']].drop_duplicates()\n",
    "rgb_colors = [x for x in cm.Spectral(np.linspace(0,1,len(modules)))]\n",
    "modules['color'] = rgb_colors\n",
    "modules = modules.set_index(\"module\", drop=True)\n",
    "dissimilarity_3d_df['color'] = dissimilarity_3d_df['module'].map(modules['color'].to_dict())\n",
    "dissimilarity_3d_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then, we visualize this data with [ipyvolume](https://ipyvolume.readthedocs.io/en/latest/index.html#)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1bd327b383cc46a8a12aecb8c6afdd6b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Figure(camera=PerspectiveCamera(fov=46.0, position=(0.0, 0.0, 2.0), quaternion=(0.0, 0.0, 0.0, …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = dissimilarity_3d_df['x']\n",
    "y = dissimilarity_3d_df['y']\n",
    "z = dissimilarity_3d_df['z']\n",
    "color = dissimilarity_3d_df['color'].values.tolist()\n",
    "ipv.quickscatter(x, y, z, color=color, size=7, marker=\"sphere\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary\n",
    "That's it! A nice 3D representation of our little software program.\n",
    "\n",
    "\n",
    "\n",
    "We see some spheres with the same color near to each other. These modules the were change together in the first place. But there are also some mixed up areas. The reasons for this are explained [here](https://www.feststelltaste.de/checking-the-modularization-of-software-systems-by-analyzing-co-changing-source-code-files/#Modules-%22comment%22,-%22framework%22-and-%22site%22)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}