Windfarm_PEGN552.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## AGL Wind Farm Power Prediction Notebook  \n",
    "\n",
    "### This version is a modified version of https://github.com/osisoft/sample-adh-data_views_jupyter-python/blob/main/Wind_Turbine_ADH_Data_ADH_Python_Library.ipynb for the [Academic Hub Wind Farms dataset](https://academic.osisoft.com/datasets)\n",
    "\n",
    "\n",
    "**NOTE: the following cell will install required modules when this notebook runs outside Binder**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "if not os.environ.get(\"BINDER_LAUNCH_HOST\"):\n",
    "    !pip install ocs_academic_hub plotly matplotlib sklearn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import required modules and hub_login"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import json\n",
    "import pandas as pd\n",
    "from datetime import date, timedelta\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "from ocs_academic_hub.datahub import hub_login\n",
    "print(\"-- modules now imported --\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Login to Academic Hub by running the next cell\n",
    "\n",
    "Run the cell and follow the steps "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "widget, hub = hub_login()\n",
    "widget"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get latest dataset information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hub.refresh_datasets()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Standard Hub Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "hub.datasets()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make WindFarm dataset the current dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hub.set_dataset(\"MIT\") \n",
    "hub.set_dataset(\"<dataset>\") # insert the correct name for the dataset to be used\n",
    "hub.current_dataset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List the assets in the dataset\n",
    "\n",
    "There are 10 wind turbines times 5 cluster (total of 50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hub.assets()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assets metadata\n",
    "\n",
    "Store data about cluster no.1 into dataframe `df_meta` for map plot in next section"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_metadata = hub.all_assets_metadata()\n",
    "df_meta = df_metadata[df_metadata.Asset_Id.apply(lambda s: s[:8] == \"cluster5\")] # insert correct cluster\n",
    "df_meta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Code for map generation using Plotly\n",
    "\n",
    "With higher zoom to see turbine location and its ID better"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import plotly.express as px\n",
    "\n",
    "fig = px.scatter_mapbox(\n",
    "    df_meta,\n",
    "    lat=\"Latitude\",\n",
    "    lon=\"Longitude\",\n",
    "    text=\"Asset_Id\",\n",
    "    zoom=12.0,\n",
    "    title=\"Locations of Cluster 1 wind turbines (red dots)\",\n",
    ")\n",
    "fig.update_traces(marker=dict(size=12, color=\"green\"))\n",
    "fig.update_layout(mapbox_style=\"open-street-map\")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the list of all single-asset data views"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hub.asset_dataviews()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the list of all multiple-asset data views \n",
    "\n",
    "There is one per cluster "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataview_ids = hub.asset_dataviews(\"\", multiple_asset=True)\n",
    "dataview_id = dataview_ids[0]  # keep the first one for cluster1\n",
    "dataview_ids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Each dataset exists within a namespace\n",
    "\n",
    "The next line finds it and stores it in a variable for the current dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "namespace_id = hub.namespace_of(hub.current_dataset())\n",
    "namespace_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verify the structure of the data view\n",
    "\n",
    "For wind turbine `cluster1.turb1` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hub.dataview_definition(namespace_id, \"<dataview>\") # insert dataview for cluster1.turb1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE:** data view for the whole `cluster1` (`wind.farms_cluster1`) is similar but with added rows for all turbines of cluster no.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Request data view result\n",
    "\n",
    "For 2 months starting on 2019-01-01, interpolated for every hour. Method `dataview_interpolated_pd` takes care of gathering multiple pages of data and returning a single Pandas dataframe.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "dateFrom = \"2019-01-01\" # set start date\n",
    "dateTo = \"2019-03-01\"  # set end date\n",
    "timeinterval = \"xx:xx:xx\"  # format for interval is hh:mm:ss\n",
    "\n",
    "df = hub.dataview_interpolated_pd(\n",
    "    namespace_id, dataview_id, dateFrom, dateTo, timeinterval\n",
    ")\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Structure of dataframe\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Renaming DataFrame column names to abbreviations, in order to display these column names clearly in\n",
    "# a correlation plot\n",
    "\n",
    "df = df.rename(\n",
    "    columns={\n",
    "        \"Rotor Speed\": \"RS\",\n",
    "        \"State\": \"TS\",\n",
    "        \"Power To Grid\": \"AP\",  # Active Power\n",
    "        \"Pitch Angle\": \"NP\",\n",
    "        \"Ambient Temperature\": \"AT\",\n",
    "        \"Wind Speed\": \"WS\",\n",
    "    }\n",
    ")\n",
    "columns_to_drop = [\n",
    "    c\n",
    "    for c in df.columns\n",
    "    if any(i in c for i in [\"Drive\", \"Nacelle\", \"Yaw\", \"Relative\"])\n",
    "]\n",
    "columns_to_drop\n",
    "df.drop(columns_to_drop, axis=1, inplace=True)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Just keep the rows with the turbine in a good state\n",
    "\n",
    "df = df[df[\"TS\"] == \"OK\"]\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the correlation between Active Power and the rest of the variables\n",
    "\n",
    "# retrieve the correlation table\n",
    "df_corr = df.corr()\n",
    "\n",
    "# increase the size of the figure\n",
    "fig = plt.figure(figsize=(50, 10))\n",
    "ax = fig.add_subplot(111)\n",
    "\n",
    "# set the color pallete (Red, yellow, green)\n",
    "cax = ax.matshow(df_corr, cmap=plt.cm.RdYlGn)\n",
    "fig.colorbar(cax)\n",
    "\n",
    "# configure the labels\n",
    "labels = [c for c in df_corr.columns]\n",
    "\n",
    "# make sure to show all the labels\n",
    "ax.set_xticks(np.arange(len(labels)))\n",
    "ax.set_yticks(np.arange(len(labels)))\n",
    "\n",
    "# Setting labels for the x and y axes of the correlation plot\n",
    "ax.set_xticklabels(labels)\n",
    "ax.set_yticklabels(labels)\n",
    "\n",
    "plt.show(block=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Renaming DataFrame column names from abbreviations back to their updated full names\n",
    "\n",
    "df = df.rename(\n",
    "    columns={\n",
    "        \"RS\": \"Rotor Speed\",\n",
    "        \"TS\": \"State\",\n",
    "        \"AP\": \"Active Power\",\n",
    "        \"NP\": \"Nacelle Position\",\n",
    "        \"AT\": \"Air Temperature\",\n",
    "        \"WS\": \"Wind Speed\",\n",
    "    }\n",
    ")\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plotting Best related variables\n",
    "\n",
    "fig = plt.figure(figsize=(10, 10))\n",
    "ax = fig.add_subplot(111)\n",
    "ax.scatter(df[\"<insert variable 1>\"], df[\"<insert variable 2>\"])\n",
    "ax.set_xlabel(\"<variable 1>\")\n",
    "ax.set_ylabel(\"<variable 2>\")\n",
    "ax.set_title(\"<variable 1> vs <variable 2>\")\n",
    "ax.set_xlim([0,20])\n",
    "\n",
    "plt.show(block=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove any rows with missing data (if any)\n",
    "df_Filter = df.dropna() \n",
    "df_Filter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Filter out negative & excessive Active Power Values\n",
    "filterNegativeActivePower = df_Filter[\"Active Power\"] >= 0\n",
    "df_Filter = df_Filter[filterNegativeActivePower]\n",
    "df_Filter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove the rows where we have a high wind speed and low active power in order to keep only the normal operating conditions\n",
    "filterOutLowPowerHighWindSpeedData = ~(\n",
    "    (df_Filter[\"Wind Speed\"] > 10) & (df_Filter[\"Active Power\"] < 600)\n",
    ")\n",
    "df_Filter = df_Filter[filterOutLowPowerHighWindSpeedData]\n",
    "df_Filter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Filter out high Wind Speeds (> 13 m/s) that do not change the Active Power results\n",
    "filterOutHighWind = df_Filter[\"Wind Speed\"] < 13\n",
    "df_Filter = df_Filter[filterOutHighWind]\n",
    "df_Filter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plotting Active Power versus Wind Speed - filtered data frame representing Normal Operating Conditions\n",
    "\n",
    "fig = plt.figure(figsize=(10, 10))\n",
    "ax = fig.add_subplot(111)\n",
    "ax.scatter(df_Filter[\"Wind Speed\"], df_Filter[\"Active Power\"])\n",
    "ax.set_xlabel(\"Wind Speed (m/s)\")\n",
    "ax.set_ylabel(\"Active Power (kW)\")\n",
    "ax.set_title(\"Active Power vs Wind Speed\")\n",
    "\n",
    "plt.show(block=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare the training & testing/scoring data sets, and split them randomly\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# define the target variable to be predicted\n",
    "y = df_Filter[\"Active Power\"].values\n",
    "# split the dataset randomly into test and train sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    df_Filter[[\"Air Temperature\", \"Wind Speed\"]].values,\n",
    "    y,\n",
    "    test_size=0.25,\n",
    "    random_state=42,\n",
    ")\n",
    "print(\"-- training/testing prepared --\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the Decision Tree Regression Machine Learning model from scikit-learn\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "\n",
    "regr_1 = DecisionTreeRegressor(max_depth=2)\n",
    "regr_2 = DecisionTreeRegressor(max_depth=5)\n",
    "regr_1.fit(X_train, y_train)\n",
    "regr_2.fit(X_train, y_train)\n",
    "\n",
    "# Predict\n",
    "y_1 = regr_1.predict(X_test)\n",
    "y_2 = regr_2.predict(X_test)\n",
    "print(\"-- decision tree regression --\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the results\n",
    "plt.figure(figsize=(10, 10))\n",
    "plt.scatter(\n",
    "    X_train[:, 1], y_train, s=20, edgecolor=\"black\", c=\"darkorange\", label=\"data\"\n",
    ")\n",
    "plt.plot(X_test[:, 1], y_1, color=\"cornflowerblue\", label=\"max_depth=2\", linewidth=2)\n",
    "plt.plot(X_test[:, 1], y_2, color=\"yellowgreen\", label=\"max_depth=5\", linewidth=2)\n",
    "plt.xlabel(\"Wind Speed\")\n",
    "plt.ylabel(\"Active Power\")\n",
    "plt.title(\"Decision Tree Regression\")\n",
    "plt.legend()\n",
    "plt.show(block=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save the machine learning model to disk\n",
    "import pickle\n",
    "\n",
    "filename = \"WT_ActivePower_model.sav\"\n",
    "pickle.dump(regr_2, open(filename, \"wb\"))\n",
    "print(\"-- model saved --\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test the model with the scoring/testing data set\n",
    "loaded_model = pickle.load(open(filename, \"rb\"))\n",
    "global result\n",
    "result = loaded_model.score(X_test, y_test)\n",
    "# print the model score\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sample prediction\n",
    "# define input\n",
    "new_input = [[<temp>, <wind speed>]]  # insert correct values for temp (°C) and wind speed (m/s) without <>\n",
    "# get prediction for new input\n",
    "new_output = regr_2.predict(new_input)\n",
    "print(new_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Call the OpenWeather API to retrieve the forecasted air temperature and wind speed\n",
    "# for Jamestown, Australia for the next 5 days\n",
    "# City code information: http://bulk.openweathermap.org/sample/\n",
    "#\n",
    "url = \"https://api.openweathermap.org/data/2.5/forecast?q=Jamestown,AU,2069194&units=metric&APPID=5dac981ce33f41f61d8d1ea06ee89798\"\n",
    "responseWeatherForecast = requests.get(url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display first 3 results \n",
    "responseWeatherForecast.json()[\"list\"][:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Store the forecasted air temperature, wind speed and timestamp from the API json response in a pandas DataFrame\n",
    "\n",
    "import datetime\n",
    "\n",
    "TempArray = []\n",
    "WindSpeedArray = []\n",
    "TimestampArray = []\n",
    "\n",
    "for val in responseWeatherForecast.json()[\"list\"]:\n",
    "    tempC = val[\"main\"][\"temp\"]\n",
    "    windSpeedMeterPerSec = round(val[\"wind\"][\"speed\"], 2)\n",
    "    np.array(TempArray.append(tempC))\n",
    "    np.array(WindSpeedArray.append(windSpeedMeterPerSec))\n",
    "    np.array(\n",
    "        TimestampArray.append(\n",
    "            datetime.datetime.strptime(val[\"dt_txt\"], \"%Y-%m-%d %H:%M:%S\")\n",
    "        )\n",
    "    )\n",
    "\n",
    "dfWeatherForecast = pd.DataFrame(\n",
    "    {\n",
    "        \"Timestamp\": TimestampArray,\n",
    "        \"Temp (C)\": TempArray,\n",
    "        \"Wind Speed (m/s)\": WindSpeedArray,\n",
    "    }\n",
    ")\n",
    "\n",
    "dfWeatherForecast"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the machine learning model developed previously to predict the Active Power\n",
    "# and add the values to the existing Data Frame\n",
    "\n",
    "import pickle\n",
    "\n",
    "filename = \"WT_ActivePower_model.sav\"\n",
    "loaded_model = pickle.load(open(filename, \"rb\"))\n",
    "\n",
    "PredictedPowerArray = []\n",
    "\n",
    "for index, row in dfWeatherForecast.iterrows():\n",
    "    new_input = [[row[\"Temp (C)\"], row[\"Wind Speed (m/s)\"]]]\n",
    "    result = loaded_model.predict(new_input)\n",
    "    np.array(PredictedPowerArray.append(result))\n",
    "\n",
    "dfWeatherForecast[\"Predicted Active Power (kW)\"] = pd.DataFrame(PredictedPowerArray)\n",
    "\n",
    "dfWeatherForecast"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot predictions\n",
    "\n",
    "import plotly.express as px\n",
    "\n",
    "px.scatter_3d(\n",
    "    dfWeatherForecast,\n",
    "    x=\"Temp (C)\",\n",
    "    y=\"Wind Speed (m/s)\",\n",
    "    z=\"Predicted Active Power (kW)\",\n",
    "    size=\"Predicted Active Power (kW)\",\n",
    "    color=\"Predicted Active Power (kW)\",\n",
    "    log_x=False,\n",
    "    size_max=100,\n",
    "    range_x=[0, 90],\n",
    "    range_y=[0, 12],\n",
    "    height=800,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}