t81_558_class_05_2_kfold.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_05_2_kfold.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 5: Regularization and Dropout**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 5 Material\n",
    "\n",
    "* Part 5.1: Part 5.1: Introduction to Regularization: Ridge and Lasso [[Video]](https://www.youtube.com/watch?v=jfgRtCYjoBs&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_05_1_reg_ridge_lasso.ipynb)\n",
    "* **Part 5.2: Using K-Fold Cross Validation with Keras** [[Video]](https://www.youtube.com/watch?v=maiQf8ray_s&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_05_2_kfold.ipynb)\n",
    "* Part 5.3: Using L1 and L2 Regularization with Keras to Decrease Overfitting [[Video]](https://www.youtube.com/watch?v=JEWzWv1fBFQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_05_3_keras_l1_l2.ipynb)\n",
    "* Part 5.4: Drop Out for Keras to Decrease Overfitting [[Video]](https://www.youtube.com/watch?v=bRyOi0L6Rs8&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_05_4_dropout.ipynb)\n",
    "* Part 5.5: Benchmarking Keras Deep Learning Regularization Techniques [[Video]](https://www.youtube.com/watch?v=1NLBwPumUAs&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_05_5_bootstrap.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' \n",
    "\n",
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 5.2: Using K-Fold Cross-validation with Keras\n",
    "\n",
    "You can use cross-validation for a variety of purposes in predictive modeling:\n",
    "\n",
    "* Generating out-of-sample predictions from a neural network\n",
    "* Estimate a good number of epochs to train a neural network for (early stopping)\n",
    "* Evaluate the effectiveness of certain hyperparameters, such as activation functions, neuron counts, and layer counts\n",
    "\n",
    "Cross-validation uses several folds and multiple models to provide each data segment a chance to serve as both the validation and training set. Figure 5.CROSS shows cross-validation.\n",
    "\n",
    "**Figure 5.CROSS: K-Fold Crossvalidation**\n",
    "![K-Fold Crossvalidation](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_1_kfold.png \"K-Fold Crossvalidation\")\n",
    "\n",
    "It is important to note that each fold will have one model (neural network). To generate predictions for new data (not present in the training set), predictions from the fold models can be handled in several ways:\n",
    "\n",
    "* Choose the model with the highest validation score as the final model.\n",
    "* Preset new data to the five models (one for each fold) and average the result (this is an [ensemble](https://en.wikipedia.org/wiki/Ensemble_learning)).\n",
    "* Retrain a new model (using the same settings as the cross-validation) on the entire dataset. Train for as many epochs and with the same hidden layer structure.\n",
    "\n",
    "Generally, I prefer the last approach and will retrain a model on the entire data set once I have selected hyper-parameters. Of course, I will always set aside a final holdout set for model validation that I do not use in any aspect of the training process.\n",
    "\n",
    "## Regression vs Classification K-Fold Cross-Validation\n",
    "\n",
    "Regression and classification are handled somewhat differently concerning cross-validation. Regression is the simpler case where you can break up the data set into K folds with little regard for where each item lands. For regression, the data items should fall into the folds as randomly as possible. It is also important to remember that not every fold will necessarily have the same number of data items. It is not always possible for the data set to be evenly divided into K folds. For regression cross-validation, we will use the Scikit-Learn class **KFold**.\n",
    "\n",
    "Cross-validation for classification could also use the **KFold** object; however, this technique would not ensure that the class balance remains the same in each fold as in the original. The balance of classes that a model was trained on must remain the same (or similar) to the training set. Drift in this distribution is one of the most important things to monitor after a trained model has been placed into actual use. Because of this, we want to make sure that the cross-validation itself does not introduce an unintended shift. This technique is called stratified sampling and is accomplished by using the Scikit-Learn object **StratifiedKFold** in place of **KFold** whenever you use classification. In summary, you should use the following two objects in Scikit-Learn:\n",
    "\n",
    "* **KFold** When dealing with a regression problem.\n",
    "* **StratifiedKFold** When dealing with a classification problem.\n",
    "\n",
    "The following two sections demonstrate cross-validation with classification and regression. \n",
    "\n",
    "## Out-of-Sample Regression Predictions with K-Fold Cross-Validation\n",
    "\n",
    "The following code trains the simple dataset using a 5-fold cross-validation. The expected performance of a neural network of the type trained here would be the score for the generated out-of-sample predictions. We begin by preparing a feature vector using the **jh-simple-dataset** to predict age. This model is set up as a regression problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from scipy.stats import zscore\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Read the data set\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "# Generate dummies for job\n",
    "df = pd.concat([df,pd.get_dummies(df['job'],prefix=\"job\")],axis=1)\n",
    "df.drop('job', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for area\n",
    "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\")],axis=1)\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for product\n",
    "df = pd.concat([df,pd.get_dummies(df['product'],prefix=\"product\")],axis=1)\n",
    "df.drop('product', axis=1, inplace=True)\n",
    "\n",
    "# Missing values for income\n",
    "med = df['income'].median()\n",
    "df['income'] = df['income'].fillna(med)\n",
    "\n",
    "# Standardize ranges\n",
    "df['income'] = zscore(df['income'])\n",
    "df['aspect'] = zscore(df['aspect'])\n",
    "df['save_rate'] = zscore(df['save_rate'])\n",
    "df['subscriptions'] = zscore(df['subscriptions'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x_columns = df.columns.drop('age').drop('id')\n",
    "x = df[x_columns].values\n",
    "y = df['age'].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the feature vector is created a 5-fold cross-validation can be performed to generate out-of-sample predictions.  We will assume 500 epochs and not use early stopping.  Later we will see how we can estimate a more optimal epoch count."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold #1\n",
      "Fold score (RMSE): 0.6814299426511208\n",
      "Fold #2\n",
      "Fold score (RMSE): 0.45486513719487165\n",
      "Fold #3\n",
      "Fold score (RMSE): 0.571615041876392\n",
      "Fold #4\n",
      "Fold score (RMSE): 0.46416356081116916\n",
      "Fold #5\n",
      "Fold score (RMSE): 1.0426518491685475\n",
      "Final, out of sample score (RMSE): 0.678316077597408\n"
     ]
    }
   ],
   "source": [
    "EPOCHS=500\n",
    "\n",
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from scipy.stats import zscore\n",
    "from sklearn.model_selection import KFold\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "\n",
    "# Cross-Validate\n",
    "kf = KFold(5, shuffle=True, random_state=42) # Use for KFold classification\n",
    "oos_y = []\n",
    "oos_pred = []\n",
    "\n",
    "fold = 0\n",
    "for train, test in kf.split(x):\n",
    "    fold+=1\n",
    "    print(f\"Fold #{fold}\")\n",
    "        \n",
    "    x_train = x[train]\n",
    "    y_train = y[train]\n",
    "    x_test = x[test]\n",
    "    y_test = y[test]\n",
    "    \n",
    "    model = Sequential()\n",
    "    model.add(Dense(20, input_dim=x.shape[1], activation='relu'))\n",
    "    model.add(Dense(10, activation='relu'))\n",
    "    model.add(Dense(1))\n",
    "    model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "    \n",
    "    model.fit(x_train,y_train,validation_data=(x_test,y_test),verbose=0,\n",
    "              epochs=EPOCHS)\n",
    "    \n",
    "    pred = model.predict(x_test)\n",
    "    \n",
    "    oos_y.append(y_test)\n",
    "    oos_pred.append(pred)    \n",
    "\n",
    "    # Measure this fold's RMSE\n",
    "    score = np.sqrt(metrics.mean_squared_error(pred,y_test))\n",
    "    print(f\"Fold score (RMSE): {score}\")\n",
    "\n",
    "# Build the oos prediction list and calculate the error.\n",
    "oos_y = np.concatenate(oos_y)\n",
    "oos_pred = np.concatenate(oos_pred)\n",
    "score = np.sqrt(metrics.mean_squared_error(oos_pred,oos_y))\n",
    "print(f\"Final, out of sample score (RMSE): {score}\")    \n",
    "    \n",
    "# Write the cross-validated prediction\n",
    "oos_y = pd.DataFrame(oos_y)\n",
    "oos_pred = pd.DataFrame(oos_pred)\n",
    "oosDF = pd.concat( [df, oos_y, oos_pred],axis=1 )\n",
    "#oosDF.to_csv(filename_write,index=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the above code also reports the average number of epochs needed.  A common technique is to then train on the entire dataset for the average number of epochs required.\n",
    "\n",
    "## Classification with Stratified K-Fold Cross-Validation\n",
    "\n",
    "The following code trains and fits the **jh**-simple-dataset dataset with cross-validation to generate out-of-sample.  It also writes the out-of-sample (predictions on the test set) results.\n",
    "\n",
    "It is good to perform stratified k-fold cross-validation with classification data.  This technique ensures that the percentages of each class remain the same across all folds.  Use the **StratifiedKFold** object instead of the **KFold** object used in the regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from scipy.stats import zscore\n",
    "\n",
    "# Read the data set\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "# Generate dummies for job\n",
    "df = pd.concat([df,pd.get_dummies(df['job'],prefix=\"job\")],axis=1)\n",
    "df.drop('job', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for area\n",
    "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\")],axis=1)\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "\n",
    "# Missing values for income\n",
    "med = df['income'].median()\n",
    "df['income'] = df['income'].fillna(med)\n",
    "\n",
    "# Standardize ranges\n",
    "df['income'] = zscore(df['income'])\n",
    "df['aspect'] = zscore(df['aspect'])\n",
    "df['save_rate'] = zscore(df['save_rate'])\n",
    "df['age'] = zscore(df['age'])\n",
    "df['subscriptions'] = zscore(df['subscriptions'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x_columns = df.columns.drop('product').drop('id')\n",
    "x = df[x_columns].values\n",
    "dummies = pd.get_dummies(df['product']) # Classification\n",
    "products = dummies.columns\n",
    "y = dummies.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will assume 500 epochs and not use early stopping.  Later we will see how we can estimate a more optimal epoch count."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold #1\n",
      "Fold score (accuracy): 0.6325\n",
      "Fold #2\n",
      "Fold score (accuracy): 0.6725\n",
      "Fold #3\n",
      "Fold score (accuracy): 0.6975\n",
      "Fold #4\n",
      "Fold score (accuracy): 0.6575\n",
      "Fold #5\n",
      "Fold score (accuracy): 0.675\n",
      "Final score (accuracy): 0.667\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "\n",
    "# np.argmax(pred,axis=1)\n",
    "# Cross-validate\n",
    "# Use for StratifiedKFold classification\n",
    "kf = StratifiedKFold(5, shuffle=True, random_state=42) \n",
    "    \n",
    "oos_y = []\n",
    "oos_pred = []\n",
    "fold = 0\n",
    "\n",
    "# Must specify y StratifiedKFold for\n",
    "for train, test in kf.split(x,df['product']):  \n",
    "    fold+=1\n",
    "    print(f\"Fold #{fold}\")\n",
    "        \n",
    "    x_train = x[train]\n",
    "    y_train = y[train]\n",
    "    x_test = x[test]\n",
    "    y_test = y[test]\n",
    "    \n",
    "    model = Sequential()\n",
    "    # Hidden 1\n",
    "    model.add(Dense(50, input_dim=x.shape[1], activation='relu')) \n",
    "    model.add(Dense(25, activation='relu')) # Hidden 2\n",
    "    model.add(Dense(y.shape[1],activation='softmax')) # Output\n",
    "    model.compile(loss='categorical_crossentropy', optimizer='adam')\n",
    "\n",
    "    model.fit(x_train,y_train,validation_data=(x_test,y_test),\n",
    "              verbose=0, epochs=EPOCHS)\n",
    "    \n",
    "    pred = model.predict(x_test)\n",
    "    \n",
    "    oos_y.append(y_test)\n",
    "    # raw probabilities to chosen class (highest probability)\n",
    "    pred = np.argmax(pred,axis=1) \n",
    "    oos_pred.append(pred)  \n",
    "\n",
    "    # Measure this fold's accuracy\n",
    "    y_compare = np.argmax(y_test,axis=1) # For accuracy calculation\n",
    "    score = metrics.accuracy_score(y_compare, pred)\n",
    "    print(f\"Fold score (accuracy): {score}\")\n",
    "\n",
    "# Build the oos prediction list and calculate the error.\n",
    "oos_y = np.concatenate(oos_y)\n",
    "oos_pred = np.concatenate(oos_pred)\n",
    "oos_y_compare = np.argmax(oos_y,axis=1) # For accuracy calculation\n",
    "\n",
    "score = metrics.accuracy_score(oos_y_compare, oos_pred)\n",
    "print(f\"Final score (accuracy): {score}\")    \n",
    "    \n",
    "# Write the cross-validated prediction\n",
    "oos_y = pd.DataFrame(oos_y)\n",
    "oos_pred = pd.DataFrame(oos_pred)\n",
    "oosDF = pd.concat( [df, oos_y, oos_pred],axis=1 )\n",
    "#oosDF.to_csv(filename_write,index=False)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training with both a Cross-Validation and a Holdout Set\n",
    "\n",
    "If you have a considerable amount of data, it is always valuable to set aside a holdout set before you cross-validate. This holdout set will be the final evaluation before using your model for its real-world use. Figure 5. HOLDOUT shows this division.\n",
    "\n",
    "**Figure 5. HOLDOUT: Cross-Validation and a Holdout Set**\n",
    "![Cross Validation and a Holdout Set](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_3_hold_train_val.png \"Cross-Validation and a Holdout Set\")\n",
    "\n",
    "The following program uses a holdout set and then still cross-validates.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from scipy.stats import zscore\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Read the data set\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "# Generate dummies for job\n",
    "df = pd.concat([df,pd.get_dummies(df['job'],prefix=\"job\")],axis=1)\n",
    "df.drop('job', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for area\n",
    "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\")],axis=1)\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for product\n",
    "df = pd.concat([df,pd.get_dummies(df['product'],prefix=\"product\")],axis=1)\n",
    "df.drop('product', axis=1, inplace=True)\n",
    "\n",
    "# Missing values for income\n",
    "med = df['income'].median()\n",
    "df['income'] = df['income'].fillna(med)\n",
    "\n",
    "# Standardize ranges\n",
    "df['income'] = zscore(df['income'])\n",
    "df['aspect'] = zscore(df['aspect'])\n",
    "df['save_rate'] = zscore(df['save_rate'])\n",
    "df['subscriptions'] = zscore(df['subscriptions'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x_columns = df.columns.drop('age').drop('id')\n",
    "x = df[x_columns].values\n",
    "y = df['age'].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the data has been preprocessed, we are ready to build the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold #1\n",
      "Fold score (RMSE): 0.544195299216696\n",
      "Fold #2\n",
      "Fold score (RMSE): 0.48070599342910353\n",
      "Fold #3\n",
      "Fold score (RMSE): 0.7034584765928998\n",
      "Fold #4\n",
      "Fold score (RMSE): 0.5397141785190473\n",
      "Fold #5\n",
      "Fold score (RMSE): 24.126205213080077\n",
      "\n",
      "Cross-validated score (RMSE): 10.801732731207947\n",
      "Holdout score (RMSE): 24.097657947297677\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from scipy.stats import zscore\n",
    "from sklearn.model_selection import KFold\n",
    "\n",
    "# Keep a 10% holdout\n",
    "x_main, x_holdout, y_main, y_holdout = train_test_split(    \n",
    "    x, y, test_size=0.10) \n",
    "\n",
    "\n",
    "# Cross-validate\n",
    "kf = KFold(5)\n",
    "    \n",
    "oos_y = []\n",
    "oos_pred = []\n",
    "fold = 0\n",
    "for train, test in kf.split(x_main):        \n",
    "    fold+=1\n",
    "    print(f\"Fold #{fold}\")\n",
    "        \n",
    "    x_train = x_main[train]\n",
    "    y_train = y_main[train]\n",
    "    x_test = x_main[test]\n",
    "    y_test = y_main[test]\n",
    "    \n",
    "    model = Sequential()\n",
    "    model.add(Dense(20, input_dim=x.shape[1], activation='relu'))\n",
    "    model.add(Dense(5, activation='relu'))\n",
    "    model.add(Dense(1))\n",
    "    model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "    \n",
    "    model.fit(x_train,y_train,validation_data=(x_test,y_test),\n",
    "              verbose=0,epochs=EPOCHS)\n",
    "    \n",
    "    pred = model.predict(x_test)\n",
    "    \n",
    "    oos_y.append(y_test)\n",
    "    oos_pred.append(pred) \n",
    "\n",
    "    # Measure accuracy\n",
    "    score = np.sqrt(metrics.mean_squared_error(pred,y_test))\n",
    "    print(f\"Fold score (RMSE): {score}\")\n",
    "\n",
    "\n",
    "# Build the oos prediction list and calculate the error.\n",
    "oos_y = np.concatenate(oos_y)\n",
    "oos_pred = np.concatenate(oos_pred)\n",
    "score = np.sqrt(metrics.mean_squared_error(oos_pred,oos_y))\n",
    "print()\n",
    "print(f\"Cross-validated score (RMSE): {score}\")    \n",
    "    \n",
    "# Write the cross-validated prediction (from the last neural network)\n",
    "holdout_pred = model.predict(x_holdout)\n",
    "\n",
    "score = np.sqrt(metrics.mean_squared_error(holdout_pred,y_holdout))\n",
    "print(f\"Holdout score (RMSE): {score}\")    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.9 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}