demos/Demo Analysis IntelliJ IDEA (german).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Programmierbeispiel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Fallbeispiel\n",
    "\n",
    "#### IntelliJ IDEA\n",
    "\n",
    "* IDE für Java-Entwickler\n",
    "* Fast komplett in Java geschrieben\n",
    "* Großes und lang aktives Projekt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### I. Fragestellung (1/3)\n",
    "\n",
    "* Schreibe die Frage explizit auf\n",
    "* Erkläre die Anayseidee verständlich\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### I. Fragestellung (2/3)\n",
    "\n",
    "<b>Frage</b>\n",
    "* Welche Quellcodedateien sind besonders komplex und änderten sich in letzter Zeit häufig?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### I. Fragestellung (3/3)\n",
    "#### Umsetzungsideen\n",
    "* Werkzeuge: Jupyter, Python, pandas, matplotlib\n",
    "* Heuristiken:\n",
    " * \"komplex\": viele Quellcodezeilen\n",
    " * \"ändert ... häufig\": hohe Anzahl Commits\n",
    " * \"in letzter Zeit\": letzte 90 Tage\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Meta-Ziel:** Grundmechaniken kennenlernen."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### II. Explorative Datenanalyse\n",
    "* Finde und lade mögliche Softwaredaten\n",
    "* Bereinige und filtere die Rohdaten"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "*Wir laden einen Datenexport aus einem Git-Repository.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": [
     "delete"
    ]
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "log = pd.read_csv(\"dataset/git_log_intellij.csv.gz\")\n",
    "log.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "*Wir sehen uns Basisinfos über den Datensatz an.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    },
    "tags": [
     "delete"
    ]
   },
   "outputs": [],
   "source": [
    "log.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<b>1</b> **DataFrame** (~ programmierbares Excel-Arbeitsblatt), <b>6</b> **Series** (= Spalten), <b>1128819</b> **entries** (= Reihen)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "*Wir wandeln die Zeitstempel von Texte in Objekte um.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    },
    "tags": [
     "delete"
    ]
   },
   "outputs": [],
   "source": [
    "log['timestamp'] = pd.to_datetime(log['timestamp'])\n",
    "log.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "*Wir sehen uns nur die jüngsten Änderungen an.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    },
    "tags": [
     "delete"
    ]
   },
   "outputs": [],
   "source": [
    "# use log['timestamp'].max() instead of pd.Timedelta('today') to avoid outdated data in the future\n",
    "recent = log[log['timestamp'] > log['timestamp'].max() - pd.Timedelta('90 days')]\n",
    "recent.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "*Wir wollen nur Java-Code verwenden.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    },
    "tags": [
     "delete"
    ]
   },
   "outputs": [],
   "source": [
    "java = recent[recent['filename'].str.endswith(\".java\")].copy()\n",
    "java.head() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### III. Formale Modellierung\n",
    "\n",
    "* Schaffe neue Sichten\n",
    "* Verschneide weitere Daten"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "*Wir zählen die Anzahl der Änderungen je Datei.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    },
    "tags": [
     "delete"
    ]
   },
   "outputs": [],
   "source": [
    "changes = java.groupby('filename')[['sha']].count()\n",
    "changes.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "*Wir holen Infos über die Code-Zeilen hinzu...*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    },
    "tags": [
     "delete"
    ]
   },
   "outputs": [],
   "source": [
    "loc = pd.read_csv(\"dataset/cloc_intellij.csv.gz\", index_col=1)\n",
    "loc.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "*...und verschneiden diese mit den vorhandenen Daten.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    },
    "tags": [
     "delete"
    ]
   },
   "outputs": [],
   "source": [
    "hotspots = changes.join(loc[['code']]).dropna(subset=['code'])\n",
    "hotspots.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### VI. Interpretation\n",
    "* Erarbeite das Kernergebnis der Analyse heraus\n",
    "* Mache die zentrale Botschaft / neuen Erkenntnisse deutlich"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "*Wir zeigen nur die TOP 10 Hotspots im Code an.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    },
    "tags": [
     "delete"
    ]
   },
   "outputs": [],
   "source": [
    "top10 = hotspots.sort_values(by=\"sha\", ascending=False).head(10)\n",
    "top10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### V. Kommunikation\n",
    "* Transformiere die Erkenntnisse in eine verständliche Visualisierung\n",
    "* Kommuniziere die nächsten Schritte nach der Analyse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "*Wir erzeugen ein XY-Diagramm aus der TOP 10 Liste.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    },
    "tags": [
     "delete"
    ]
   },
   "outputs": [],
   "source": [
    "ax = top10.plot.scatter('sha', 'code');\n",
    "\n",
    "for k, v in top10.iterrows():\n",
    "    ax.annotate(k.split(\"/\")[-1], v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Bonus: Welche Dateien ändern sich besonders häufig im Allgemeinen?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    },
    "tags": [
     "delete"
    ]
   },
   "outputs": [],
   "source": [
    "most_changes = hotspots['sha'].sort_values(ascending=False)\n",
    "most_changes.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "*Wir visualisieren dies mit einem einfachen Liniendiagramm.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    },
    "tags": [
     "delete"
    ]
   },
   "outputs": [],
   "source": [
    "most_changes.plot(rot=90);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Ende der Demo"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "rise": {
   "controls": false,
   "showSlideNumber": false,
   "slideNumber ": "false",
   "transition": "none"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}