python/lecture1-eventlogs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is part of a course on Applied Process Mining. The collection of notebooks is a *living document* and subject to change. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lecture 1 - 'Event Logs and Process Discovery' (Python / PM4Py)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "<img src=\"https://pm4py.fit.fraunhofer.de/static/assets/images/pm4py-site-logo-padded.png\" alt=\"PM4Py\" style=\"width: 200px;\"/>\n",
    "\n",
    "In this notebook, we are using several libraries:\n",
    "\n",
    "* [PM4Py](https://pm4py.fit.fraunhofer.de/)\n",
    "* [pandas](https://pandas.pydata.org/)\n",
    "* [plotnine](https://plotnine.readthedocs.io/en/stable/)\n",
    "\n",
    "Often used dependencies are imported:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import pm4py\n",
    "import plotnine\n",
    "from plotnine import ggplot, geom_point, aes, theme_bw, coord_flip, scale_y_discrete, theme, element_text, geom_bin2d, ylab, scale_x_datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Event Logs\n",
    "\n",
    "This part introduces event logs and their unique properties that provide the basis for any Process Mining method. We use the same event logs as provided by `bupaR`. However, we need to load them from the CSV files in the `data` directory of this repository. In this lecture we are going to make use of the following datasets:\n",
    "\n",
    "* Patients, a synthetically generated example event log in a hospital setting.\n",
    "* Sepsis, a real-life event log taken from a Dutch hospital. The event log is publicly available here: https://doi.org/10.4121/uuid:915d2bfb-7e84-49ad-a286-dc35f063a460 and has been used in many Process Mining related publications."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import Patients Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients = pd.read_csv(\"../data/patients.csv\", sep=';')\n",
    "patients['time'] = pd.to_datetime(patients['time'])\n",
    "num_rows = len(patients)\n",
    "print(\"Number of rows: {}\".format(num_rows))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import Sepsis Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sepsis = pd.read_csv(\"../data/sepsis.csv\", sep=';')\n",
    "num_rows = len(sepsis)\n",
    "sepsis['timestamp'] = pd.to_datetime(sepsis['timestamp'])\n",
    "print(\"Number of rows: {}\".format(num_rows))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploring Event Data\n",
    "\n",
    "Let us first explore the event data without any prior knowledge about event log structure or properties. We use standard Pandas and plotnine features to do so. Regarding the choice for plotnine, any other plotting library such as Matplotlib could also be used and it is simply used to deviate as little as possible from the exploration performed with ggplot2 in the R version of this lecture. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The most important ingredient of an event log is the timestamps column `time`. This allows us to establish a sequence of events."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients_sample = patients[patients['time'] < '2017-01-31']\n",
    "(ggplot(patients_sample, aes('time', 0))\n",
    " + geom_point() \n",
    " + theme_bw()\n",
    " + ylab(\"Event\")\n",
    " + scale_x_datetime(date_breaks = \"1 days\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need to have information on the kind of actions or `activities` performed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients.drop_duplicates(subset='handling')[[\"handling\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us have a look at what other data is available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients.drop_duplicates(subset='patient')[[\"patient\"]].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Maybe the patient identifier could be a good candidate for defining a process `case` since this is an 'entity' that we would like to follow. When counting the events that occurred per individual patient it seems that there is a similar number of events for each patient, which is generally a good indicator for a process case identifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients.groupby(['patient'])[\"patient\"].agg(['count']).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let use decide that we want to look at the process by following the patient identifier as `case identifier`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients_sample = patients[patients['time'] < '2017-01-31']\n",
    "(ggplot(patients_sample, aes('time', 'patient', color = 'handling'))\n",
    " + geom_point() \n",
    " + theme_bw()\n",
    " + scale_x_datetime(date_breaks = \"7 days\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The scatterplot above is known as `Dotted Chart` in the process mining community and provides an 'at a glance' overview on the events and their temporal relation when grouped by a case. It seems that each of the sequences of events (also known as `traces`) start with the `Registration` event. Let us have a look at the event data sorted by patient identifier and by time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients.sort_values(['patient', 'time']).head(14)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An individual process execution (e.g., for patient 1) consists of several activities that are done in a sequence. However, we have more information available than simply the sequence of events. For each occurrence of an activity we have two events: a `start` event and a `complete` event as captured in the column `registration_type`. These event refer to the lifecycle of an activity and allow us to capture the `duration` of an activity. Much more complex lifecycles of activities are possible, a general model is described here: http://bupar.net/creating_eventlogs.html#Transactional_life_cycle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Further resources\n",
    "\n",
    "* [XES Standard](http://xes-standard.org/)\n",
    "* [Importing CSV event logs](https://pm4py.fit.fraunhofer.de/documentation#item-import-csv)\n",
    "* [Importing XES event logs](https://pm4py.fit.fraunhofer.de/documentation#item-impoort-xes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Reflection Questions\n",
    "\n",
    "* What could be the reason a column `.order` is included in this dataset?\n",
    "* How could the column `employee` be used?\n",
    "* What is the use of the column `handling_id` and in which situation is it required?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Process Visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set of Traces \n",
    "\n",
    "Exploring traces as a set visualization is currently not implemented in PM4Py. \n",
    "**Challenge** implement a visualization similar to that in bupaR with Python and open a pull request. Here are some reference implementations of a 'trace explorer':\n",
    "\n",
    "* http://bupar.net/trace_explorer.html\n",
    "* https://fmannhardt.de/blog/software/prom/explorer (ProM)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# implement a view of the event log as a set of traces "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dotted Chart\n",
    "\n",
    "The `Dotted Chart` adds the timing aspect of the individual traces and visualized all of them at-a-glance. It can be configured in many different ways and provides a good insight into time-related aspects of the process behavior. PM4Py provides a basic  Dotted Chart visualization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients_log = pm4py.format_dataframe(patients, case_id='patient', activity_key='handling', timestamp_key='time')\n",
    "pm4py.view_dotted_chart(pm4py.filter_time_range(patients_log, \"1970-01-01 00:00:00\", \"2017-01-31 00:00:00\", mode='events'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, to allow for more customization of the visuals, the same view can be simply reproduced using plotnine and allows for some more flexibility in choosing the perspectives:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Necessary pre-processing on the data frame\n",
    "patients_sorted = patients.sort_values(['time'])\n",
    "# Creating categories for the case identifier\n",
    "patients_sorted['patient'] = pd.Categorical(patients_sorted['patient'], \n",
    "                                            categories = patients_sorted['patient'].drop_duplicates().tolist()[::-1], ordered= True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Absolute Time Dimension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(patients_sorted[patients_sorted['time'] < '2017-01-31'], \n",
    "        aes('time', 'patient', color = 'handling'))\n",
    " + geom_point()\n",
    " + theme_bw()\n",
    " + scale_y_discrete(labels = \"\")\n",
    " + theme(axis_text_x=element_text(rotation=45, hjust=1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Relative Time Dimension"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We meed to make the time relative and add a new column `time_relative` for that purpose:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients_sorted['time_relative'] = patients_sorted['time'].sub( patients_sorted.groupby('patient')['time'].transform('first'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(patients_sorted, aes('time_relative', 'patient', color = 'handling'))\n",
    " + geom_point()\n",
    " + theme_bw()\n",
    " + scale_y_discrete(labels = \"\")\n",
    " + theme(axis_text_x=element_text(rotation=45, hjust=1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We still need to sort by the overall duration to replicate the `bupaR`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients_sorted['duration'] = patients_sorted.groupby('patient')['time_relative'].transform('max')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients_sorted_duration = patients_sorted.sort_values(['duration'])\n",
    "patients_sorted_duration['patient'] = pd.Categorical(patients_sorted_duration['patient'], categories = patients_sorted_duration['patient'].drop_duplicates().tolist()[::-1], ordered= True)\n",
    "\n",
    "(ggplot(patients_sorted_duration, aes('time_relative', 'patient', color = 'handling'))\n",
    " + geom_point()\n",
    " + theme_bw()\n",
    " + scale_y_discrete(labels = \"\")\n",
    " + theme(axis_text_x=element_text(rotation=45, hjust=1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check out other basic process visualization options using PM4Py:\n",
    "\n",
    "* [Basic Process Statistics](https://pm4py.fit.fraunhofer.de/documentation#statistics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Process Map Visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, there is no built-in precedence matrix visualization in PM4Py, but it can be replicated easily:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients_sorted['antecedent'] = patients_sorted.groupby([\"patient\"])['handling'].shift(1).fillna(\"Start\")\n",
    "patients_sorted['consequent'] = patients_sorted['handling']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(patients_sorted, aes('consequent', 'antecedent', ))\n",
    " + geom_bin2d() \n",
    " + theme_bw()\n",
    " + theme(axis_text_x=element_text(rotation=45, hjust=1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Directly-follows Graph / Process Map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The process map or directly-follows graph visualization in PM4Py cannot deal yet with `activity instances`, so we need to only focus on the `complete` events."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patients_log = patients_log[patients_log['registration_type'] == 'complete']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfg, sa, ea = pm4py.discover_directly_follows_graph(patients_log)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pm4py.view_dfg(dfg, sa, ea)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pm4py.algo.discovery.dfg import algorithm as dfg_discovery\n",
    "from pm4py.algo.discovery.dfg import algorithm as dfg_discovery\n",
    "from pm4py.visualization.dfg import visualizer as dfg_visualization\n",
    "\n",
    "dfg = dfg_discovery.apply(patients_log)\n",
    "\n",
    "dfg = dfg_discovery.apply(patients_log, variant=dfg_discovery.Variants.PERFORMANCE)\n",
    "gviz = dfg_visualization.apply(dfg, log=patients_log, variant=dfg_visualization.Variants.PERFORMANCE)\n",
    "dfg_visualization.view(gviz)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Further Perspectives and Animation\n",
    "\n",
    "No such feature in PM4Py yet."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Real-life Processes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sepsis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sepsis_sorted = sepsis.sort_values(['timestamp'])\n",
    "sepsis_sorted['timestamp'] = pd.to_datetime(sepsis_sorted['timestamp'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sepsis_sorted['antecedent'] = sepsis_sorted.groupby([\"case_id\"])['activity'].shift(1).fillna(\"Start\")\n",
    "sepsis_sorted['consequent'] = sepsis_sorted['activity']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(ggplot(sepsis_sorted, aes('consequent', 'antecedent', ))\n",
    " + geom_bin2d() \n",
    " + theme_bw()\n",
    " + theme(axis_text_x=element_text(rotation=45, hjust=1)))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "697fc57cee903094f2e79d714169e862761fd41795cb1acde2e554f7b56adc7f"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}