notebooks/tutorial_delira.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "# Delira Introduction\n",
    "\n",
    "*Last updated: 09.05.2019*\n",
    "\n",
    "Authors: Justus Schock, Christoph Haarburger"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "## Loading Data\n",
    "\n",
    "To train your network you first need to load your training data (and probably also your validation data). This chapter will therefore deal with `delira`'s capabilities to load your data (and apply some augmentation). \n",
    "\n",
    "### The Dataset\n",
    "There are mainly two ways to load your data: Lazy or non-lazy. Loading in a lazy way means that you load the data just in time and keep the used memory to a bare minimum. This has, however, the disadvantage that your loading function could be a bottleneck since all postponed operations may have to wait until the needed data samples are loaded. In a no-lazy way, one would preload all data to RAM before starting any other operations. This has the advantage that there cannot be a loading bottleneck during latter operations. This advantage comes at cost of a higher memory usage and a (possibly) huge latency at the beginning of each experiment. Both ways to load your data are implemented in `delira` and they are named `BaseLazyDataset`and `BaseCacheDataset`. In the following steps you will only see the `BaseLazyDataset` since exchanging them is trivial. All Datasets (including the ones you might want to create yourself later) must be derived of `delira.data_loading.AbstractDataset` to ensure a minimum common API.\n",
    "\n",
    "The dataset's `__init__` has the following signature:\n",
    "\n",
    "```python\n",
    "def __init__(self, data_path, load_fn, **load_kwargs):\n",
    "```\n",
    "\n",
    "This means, you have to pass the path to the directory containing your data (`data_path`), a function to load a single sample of your data (`load_fn`). To get a single sample of your dataset after creating it, you can index it like this: `dataset[0]`.\n",
    "Additionally you can iterate over your dataset just like over any other `python` iterator via\n",
    "\n",
    "```python\n",
    "for sample in dataset:\n",
    "    # do your stuff here\n",
    "```\n",
    "\n",
    "or enumerate it via\n",
    "\n",
    "```python\n",
    "for idx, sample in enumerate(dataset):\n",
    "    # do your stuff here\n",
    "```\n",
    ".\n",
    "\n",
    "The missing argument `**load_kwargs` accepts an arbitrary amount of additional keyword arguments which are directly passed to your loading function.\n",
    "\n",
    "An example of how loading your data may look like is given below:\n",
    "```python\n",
    "from delira.data_loading import BaseLazyDataset, default_load_fn_2d\n",
    "dataset_train = BaseLazyDataset(\"/images/datasets/external/mnist/train\",\n",
    "                                default_load_fn_2d, img_shape=(224, 224))\n",
    "```\n",
    "\n",
    "In this case all data lying in `/images/datasets/external/mnist/train` is loaded by `default_load_fn_2d`. The files containing the data must be PNG-files, while the groundtruth is defined in TXT-files. The `default_load_fn_2d` needs the additional argument `img_shape` which is passed as keyword argument via `**load_kwargs`.\n",
    "\n",
    "> **Note:** for reproducability we decided to use some wrapped PyTorch datasets for this introduction. \n",
    "\n",
    "Now, let's just initialize our trainset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "from delira.data_loading import TorchvisionClassificationDataset\n",
    "dataset_train = TorchvisionClassificationDataset(\"mnist\", train=True,\n",
    "                                                 img_shape=(224, 224))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "Getting a single sample of your dataset with dataset_train[0] will produce:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "dataset_train[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "which means, that our data is stored in a dictionary containing the keys `data` and `label`, each of them holding the corresponding numpy arrays. The dataloading works on `numpy` purely and is thus backend agnostic. It does not matter in which format or with which library you load/preprocess your data, but at the end it must be converted to numpy arrays\n",
    "For validation purposes another dataset could be created with the test data like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "dataset_val = TorchvisionClassificationDataset(\"mnist\", train=False,\n",
    "                                               img_shape=(224, 224))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "### The Dataloader\n",
    "The Dataloader wraps your dataset to privode the ability to load whole batches with an abstract interface. To create a dataloader, one would have to pass the following arguments to it's `__init__`: the previously created `dataset`.Additionally, it is possible to pass the `batch_size` defining the number of samples per batch, the total number of batches (`num_batches`), which will be the number of samples in your dataset devided by the batchsize per default, a random `seed`for always getting the same behaviour of random number generators and a [`sampler`]() defining your sampling strategy. This would create a dataloader for your `dataset_train`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "from delira.data_loading import DataLoader\n",
    "\n",
    "batch_size = 32\n",
    "\n",
    "loader_train = DataLoader(dataset_train, batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "Since the batch_size has been set to 32, the loader will load 32 samples as one batch.\n",
    "\n",
    "Even though it would be possible to train your network with an instance of `DataLoader`, `malira` also offers a different approach that covers multithreaded data loading and augmentation:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "### The Datamanager\n",
    "\n",
    "The data manager is implemented as `delira.data_loading.DataManager` and wraps a `DataLoader`. It also encapsulates augmentations. Having a view on the `DataManager`'s signature, it becomes obvious that it accepts the same arguments as the [`DataLoader`](#The-Dataloader). You can either pass a `dataset` or a combination of path, dataset class and load function. Additionally, you can pass a custom dataloder class if necessary and a sampler class to choose a sampling algorithm. \n",
    "\n",
    "The parameter `transforms` accepts augmentation transformations as implemented in `batchgenerators`. Augmentation is applied on the fly using `n_process_augmentation` threads.\n",
    "\n",
    "All in all the DataManager is the recommended way to generate batches from your dataset.\n",
    "\n",
    "The following example shows how to create a data manager instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "from delira.data_loading import DataManager\n",
    "from batchgenerators.transforms.abstract_transforms import Compose\n",
    "from batchgenerators.transforms.sample_normalization_transforms import MeanStdNormalizationTransform\n",
    "\n",
    "batchsize = 64\n",
    "transforms = Compose([MeanStdNormalizationTransform(mean=1*[0], std=1*[1])])\n",
    "\n",
    "data_manager_train = DataManager(dataset_train,  # dataset to use\n",
    "                                    batchsize,  # batchsize\n",
    "                                    n_process_augmentation=1,  # number of augmentation processes\n",
    "                                    transforms=transforms)  # augmentation transforms\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "The approach to initialize a DataManager from a datapath takes more arguments since, in opposite to initializaton from dataset, it needs all the arguments which are necessary to internally create a dataset.\n",
    "\n",
    "Since we want to validate our model we have to create a second manager containing our `dataset_val`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "data_manager_val = DataManager(dataset_val, \n",
    "                                    batchsize, \n",
    "                                    n_process_augmentation=1, \n",
    "                                    transforms=transforms)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "\n",
    "That's it - we just finished loading our data!\n",
    "\n",
    "Iterating over a DataManager is possible in simple loops:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "from tqdm.auto import tqdm # utility for progress bars\n",
    "\n",
    "# create actual batch generator from DataManager\n",
    "batchgen = data_manager_val.get_batchgen()\n",
    "\n",
    "for data in tqdm(batchgen):\n",
    "    pass # here you can access the data of the current batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "### Sampler\n",
    "In previous section samplers have been already mentioned but not yet explained. A sampler implements an algorithm how a batch should be assembled from single samples in a dataset. `delira` provides the following sampler classes in it's subpackage `delira.data_loading.sampler`:\n",
    "\n",
    "* `AbstractSampler`\n",
    "* `SequentialSampler`\n",
    "* `PrevalenceSequentialSampler`\n",
    "* `RandomSampler`\n",
    "* `PrevalenceRandomSampler`\n",
    "* `WeightedRandomSampler`\n",
    "* `LambdaSampler`\n",
    "\n",
    "The `AbstractSampler` implements no sampling algorithm but defines a sampling API and thus all custom samplers must inherit from this class. The `Sequential` sampler builds batches by just iterating over the samples' indices in a sequential way. Following this, the `RandomSampler` builds batches by randomly drawing the samples' indices with replacement. \n",
    "If the class each sample belongs to is known for each sample at the beginning, the `PrevalenceSequentialSampler` and the `PrevalenceRandomSampler` perform a per-class sequential or random sampling and building each batch with the exactly same number of samples from each class. \n",
    "The `WeightedRandomSampler`accepts custom weights to give specific samples a higher probability during random sampling than others.\n",
    "\n",
    "The `LambdaSampler` is a wrapper for a custom sampling function, which can be passed to the wrapper during it's initialization, to ensure API conformity.\n",
    "\n",
    "It can be passed to the DataLoader or DataManager as class (argument `sampler_cls`) or as instance (argument `sampler`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "## Models\n",
    "\n",
    "Since the purpose of this framework is to use machine learning algorithms, there has to be a way to define them. Defining models is straight forward. `delira` provides a class `delira.models.AbstractNetwork`. *All models must inherit from this class*.\n",
    "\n",
    "To inherit this class four functions must be implemented in the subclass:\n",
    "\n",
    "* `__init__`\n",
    "* `closure`\n",
    "* `prepare_batch`\n",
    "* `__call__`\n",
    "\n",
    "\n",
    "### `__init__`\n",
    "The `__init__`function is a classes constructor. In our case it builds the entire model (maybe using some helper functions). If writing your own custom model, you have to override this method.\n",
    "\n",
    "> **Note:** If you want the best experience for saving your model and completely recreating it during the loading process you need to take care of a few things:\n",
    "> * if using `torchvision.models` to build your model, always import it with `from torchvision import models as t_models`\n",
    "> * register all arguments in your custom `__init__` in the abstract class. A init_prototype could look like this:\n",
    ">\n",
    "```python\n",
    "def __init__(self, in_channels: int, n_outputs: int, **kwargs):\n",
    "    \"\"\"\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    in_channels: int\n",
    "        number of input_channels\n",
    "    n_outputs: int\n",
    "        number of outputs (usually same as number of classes)\n",
    "    \"\"\"\n",
    "    # register params by passing them as kwargs to parent class __init__\n",
    "    # only params registered like this will be saved!\n",
    "    super().__init__(in_channels=in_channels,\n",
    "                     n_outputs=n_outputs,\n",
    "                     **kwargs)\n",
    "```\n",
    "\n",
    "\n",
    "### `closure`\n",
    "The `closure`function defines one batch iteration to train the network. This function is needed for the framework to provide a generic trainer function which works with all kind of networks and loss functions.\n",
    "\n",
    "The closure function must implement all steps from forwarding, over loss calculation, metric calculation, logging (for which `delira.logging_handlers` provides some extensions for pythons logging module), and the actual backpropagation.\n",
    "\n",
    "It is called with an empty optimizer-dict to evaluate and should thus work with optional optimizers.\n",
    "\n",
    "### `prepare_batch`\n",
    "The `prepare_batch`function defines the transformation from loaded data to match the networks input and output shape and pushes everything to the right device.\n",
    "\n",
    "\n",
    "## Abstract Networks for specific Backends\n",
    "### PyTorch\n",
    "At the time of writing, PyTorch is the only backend which is supported, but other backends are planned.\n",
    "In PyTorch every network should be implemented as a subclass of `torch.nn.Module`, which also provides a `__call__` method.\n",
    "\n",
    "This results in sloghtly different requirements for PyTorch networks: instead of implementing a `__call__` method, we simply call the `torch.nn.Module.__call__` and therefore have to implement the `forward` method, which defines the module's behaviour and is internally called by `torch.nn.Module.__call__` (among other stuff). To give a default behaviour suiting most cases and not have to care about internals, `delira` provides the `AbstractPyTorchNetwork` which is a more specific case of the `AbstractNetwork` for PyTorch modules.\n",
    "\n",
    "#### `forward`\n",
    "The `forward` function defines what has to be done to forward your input through your network and must return a dictionary. Assuming your network has three convolutional layers stored in `self.conv1`, `self.conv2` and `self.conv3` and a ReLU stored in `self.relu`, a simple `forward` function could look like this:\n",
    "\n",
    "```python\n",
    "def forward(self, input_batch: torch.Tensor):\n",
    "    out_1 = self.relu(self.conv1(input_batch))\n",
    "    out_2 = self.relu(self.conv2(out_1))\n",
    "    out_3 = self.conv3(out2)\n",
    "    \n",
    "    return {\"pred\": out_3}\n",
    "```\n",
    "\n",
    "#### `prepare_batch`\n",
    "The default `prepare_batch` function for PyTorch networks looks like this:\n",
    "\n",
    "```python\n",
    "    @staticmethod\n",
    "    def prepare_batch(batch: dict, input_device, output_device):\n",
    "        \"\"\"\n",
    "        Helper Function to prepare Network Inputs and Labels (convert them to\n",
    "        correct type and shape and push them to correct devices)\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        batch : dict\n",
    "            dictionary containing all the data\n",
    "        input_device : torch.device\n",
    "            device for network inputs\n",
    "        output_device : torch.device\n",
    "            device for network outputs\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        dict\n",
    "            dictionary containing data in correct type and shape and on correct\n",
    "            device\n",
    "\n",
    "        \"\"\"\n",
    "        return_dict = {\"data\": torch.from_numpy(batch.pop(\"data\")).to(\n",
    "            input_device)}\n",
    "\n",
    "        for key, vals in batch.items():\n",
    "            return_dict[key] = torch.from_numpy(vals).to(output_device)\n",
    "\n",
    "        return return_dict\n",
    "\n",
    "```\n",
    "and can be customized by subclassing the `AbstractPyTorchNetwork`.\n",
    "\n",
    "#### `closure example`\n",
    "A simple closure function for a PyTorch module could look like this:\n",
    "```python\n",
    "    @staticmethod\n",
    "    def closure(model: AbstractPyTorchNetwork, data_dict: dict,\n",
    "                optimizers: dict, criterions={}, metrics={},\n",
    "                fold=0, **kwargs):\n",
    "        \"\"\"\n",
    "        closure method to do a single backpropagation step\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        model : :class:`ClassificationNetworkBasePyTorch`\n",
    "            trainable model\n",
    "        data_dict : dict\n",
    "            dictionary containing the data\n",
    "        optimizers : dict\n",
    "            dictionary of optimizers to optimize model's parameters\n",
    "        criterions : dict\n",
    "            dict holding the criterions to calculate errors\n",
    "            (gradients from different criterions will be accumulated)\n",
    "        metrics : dict\n",
    "            dict holding the metrics to calculate\n",
    "        fold : int\n",
    "            Current Fold in Crossvalidation (default: 0)\n",
    "        **kwargs:\n",
    "            additional keyword arguments\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        dict\n",
    "            Metric values (with same keys as input dict metrics)\n",
    "        dict\n",
    "            Loss values (with same keys as input dict criterions)\n",
    "        list\n",
    "            Arbitrary number of predictions as torch.Tensor\n",
    "\n",
    "        Raises\n",
    "        ------\n",
    "        AssertionError\n",
    "            if optimizers or criterions are empty or the optimizers are not\n",
    "            specified\n",
    "\n",
    "        \"\"\"\n",
    "\n",
    "        assert (optimizers and criterions) or not optimizers, \\\n",
    "            \"Criterion dict cannot be emtpy, if optimizers are passed\"\n",
    "\n",
    "        loss_vals = {}\n",
    "        metric_vals = {}\n",
    "        total_loss = 0\n",
    "\n",
    "        # choose suitable context manager:\n",
    "        if optimizers:\n",
    "            context_man = torch.enable_grad\n",
    "\n",
    "        else:\n",
    "            context_man = torch.no_grad\n",
    "\n",
    "        with context_man():\n",
    "\n",
    "            inputs = data_dict.pop(\"data\")\n",
    "            # obtain outputs from network\n",
    "            preds = model(inputs)[\"pred\"]\n",
    "\n",
    "            if data_dict:\n",
    "\n",
    "                for key, crit_fn in criterions.items():\n",
    "                    _loss_val = crit_fn(preds, *data_dict.values())\n",
    "                    loss_vals[key] = _loss_val.detach()\n",
    "                    total_loss += _loss_val\n",
    "\n",
    "                with torch.no_grad():\n",
    "                    for key, metric_fn in metrics.items():\n",
    "                        metric_vals[key] = metric_fn(\n",
    "                            preds, *data_dict.values())\n",
    "\n",
    "        if optimizers:\n",
    "            optimizers['default'].zero_grad()\n",
    "            total_loss.backward()\n",
    "            optimizers['default'].step()\n",
    "\n",
    "        else:\n",
    "\n",
    "            # add prefix \"val\" in validation mode\n",
    "            eval_loss_vals, eval_metrics_vals = {}, {}\n",
    "            for key in loss_vals.keys():\n",
    "                eval_loss_vals[\"val_\" + str(key)] = loss_vals[key]\n",
    "\n",
    "            for key in metric_vals:\n",
    "                eval_metrics_vals[\"val_\" + str(key)] = metric_vals[key]\n",
    "\n",
    "            loss_vals = eval_loss_vals\n",
    "            metric_vals = eval_metrics_vals\n",
    "\n",
    "        for key, val in {**metric_vals, **loss_vals}.items():\n",
    "            logging.info({\"value\": {\"value\": val.item(), \"name\": key,\n",
    "                                    \"env_appendix\": \"_%02d\" % fold\n",
    "                                    }})\n",
    "\n",
    "        logging.info({'image_grid': {\"images\": inputs, \"name\": \"input_images\",\n",
    "                                     \"env_appendix\": \"_%02d\" % fold}})\n",
    "\n",
    "        return metric_vals, loss_vals, preds\n",
    "```\n",
    "\n",
    "> **Note:** This closure is taken from the `delira.models.classification.ClassificationNetworkBasePyTorch`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "### Other examples\n",
    "In `delira.models` you can find exemplaric implementations of generative adversarial networks, classification and regression approaches or segmentation networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "## Training\n",
    "\n",
    "### Parameters\n",
    "Training-parameters (often called hyperparameters) can be defined in the `delira.training.Parameters` class. \n",
    "\n",
    "The class accepts the parameters `batch_size` and `num_epochs` to define the batchsize and the number of epochs to train, the parameters `optimizer_cls` and `optimizer_params` to create an optimizer or training, the parameter `criterions` to specify the training criterions (whose gradients will be accumulated by defaut), the parameters `lr_sched_cls` and `lr_sched_params` to define the learning rate scheduling and the parameter `metrics` to specify evaluation metrics.\n",
    "\n",
    "Additionally, it is possible to pass an aritrary number of keyword arguments to the class\n",
    "\n",
    "It is good practice to create a `Parameters` object at the beginning and then use it for creating other objects which are needed for training, since you can use the classes attributes and changes in hyperparameters only have to be done once:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from delira.training import Parameters\n",
    "from delira.data_loading import RandomSampler, SequentialSampler\n",
    "\n",
    "params = Parameters(fixed_params={\n",
    "    \"model\": {},\n",
    "    \"training\": {\n",
    "        \"batch_size\": 64, # batchsize to use\n",
    "        \"num_epochs\": 2, # number of epochs to train\n",
    "        \"optimizer_cls\": torch.optim.Adam, # optimization algorithm to use\n",
    "        \"optimizer_params\": {'lr': 1e-3}, # initialization parameters for this algorithm\n",
    "        \"criterions\": {\"CE\": torch.nn.CrossEntropyLoss()}, # the loss function\n",
    "        \"lr_sched_cls\": None,  # the learning rate scheduling algorithm to use\n",
    "        \"lr_sched_params\": {}, # the corresponding initialization parameters\n",
    "        \"metrics\": {} # and some evaluation metrics\n",
    "    }\n",
    "}) \n",
    "\n",
    "# recreating the data managers with the batchsize of the params object\n",
    "manager_train = DataManager(dataset_train, params.nested_get(\"batch_size\"), 1,\n",
    "                                transforms=None, sampler_cls=RandomSampler,\n",
    "                                n_process_loading=4)\n",
    "manager_val = DataManager(dataset_val, params.nested_get(\"batch_size\"), 3,\n",
    "                              transforms=None, sampler_cls=SequentialSampler,\n",
    "                              n_process_loading=4)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "### Trainer\n",
    "\n",
    "The `delira.training.NetworkTrainer` class provides functions to train a single network by passing attributes from your parameter object, a `save_freq` to specify how often your model should be saved (`save_freq=1` indicates every epoch, `save_freq=2` every second epoch etc.) and `gpu_ids`. If you don't pass any ids at all, your network will be trained on CPU (and probably take a lot of time). If you specify 1 id, the network will be trained on the GPU with the corresponding index and if you pass multiple `gpu_ids` your network will be trained on multiple GPUs in parallel.\n",
    "\n",
    "> **Note:** The GPU indices are refering to the devices listed in `CUDA_VISIBLE_DEVICES`. E.g if `CUDA_VISIBLE_DEVICES` lists GPUs 3, 4, 5 then gpu_id 0 will be the index for GPU 3 etc.\n",
    "\n",
    "> **Note:** training on multiple GPUs is not recommended for easy and small networks, since for these networks the synchronization overhead is far greater than the parallelization benefit.\n",
    "\n",
    "Training your network might look like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "from delira.training import PyTorchNetworkTrainer\n",
    "from delira.models.classification import ClassificationNetworkBasePyTorch\n",
    "\n",
    "# path where checkpoints should be saved\n",
    "save_path = \"./results/checkpoints\"\n",
    "\n",
    "model = ClassificationNetworkBasePyTorch(in_channels=1, n_outputs=10)\n",
    "\n",
    "trainer = PyTorchNetworkTrainer(network=model,\n",
    "                                save_path=save_path,\n",
    "                                criterions=params.nested_get(\"criterions\"),\n",
    "                                optimizer_cls=params.nested_get(\"optimizer_cls\"),\n",
    "                                optimizer_params=params.nested_get(\"optimizer_params\"),\n",
    "                                metrics=params.nested_get(\"metrics\"),\n",
    "                                lr_scheduler_cls=params.nested_get(\"lr_sched_cls\"),\n",
    "                                lr_scheduler_params=params.nested_get(\"lr_sched_params\"),\n",
    "                                gpu_ids=[0]\n",
    "                        )\n",
    "\n",
    "#trainer.train(params.nested_get(\"num_epochs\"), manager_train, manager_val)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "### Experiment\n",
    "The `delira.training.AbstractExperiment` class needs an experiment name, a path to save it's results to, a parameter object, a model class and the keyword arguments to create an instance of this class. It provides methods to perform a single training and also a method for running a kfold-cross validation. In order to create it, you must choose the `PyTorchExperiment`, which is basically just a subclass of the `AbstractExperiment` to provide a general setup for PyTorch modules. Running an experiment could look like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "from delira.training import PyTorchExperiment\n",
    "from delira.training.train_utils import create_optims_default_pytorch\n",
    "\n",
    "# Add model parameters to Parameter class\n",
    "params.fixed.model = {\"in_channels\": 1, \"n_outputs\": 10}\n",
    "\n",
    "experiment = PyTorchExperiment(params=params, \n",
    "                               model_cls=ClassificationNetworkBasePyTorch,\n",
    "                               name=\"TestExperiment\", \n",
    "                               save_path=\"./results\",\n",
    "                               optim_builder=create_optims_default_pytorch,\n",
    "                               gpu_ids=[0])\n",
    "\n",
    "experiment.run(manager_train, manager_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "An `Experiment` is the most abstract (and recommended) way to define, train and validate your network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "## Logging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "Previous class and function definitions used pythons's `logging` library. As extensions for this library `delira` provides a package (`delira.logging`) containing handlers to realize different logging methods. \n",
    "\n",
    "To use these handlers simply add them to your logger like this:\n",
    "```python\n",
    "logger.addHandler(logging.StreamHandler())\n",
    "```\n",
    "\n",
    "Nowadays, delira mainly relies on [trixi](https://github.com/MIC-DKFZ/trixi/) for logging and provides only a `MultiStreamHandler` and a `TrixiHandler`, which is a binding to `trixi`'s loggers and integrates them into the python `logging` module\n",
    "\n",
    "### `MultiStreamHandler`\n",
    "The `MultiStreamHandler` accepts an arbitrary number of streams during initialization and writes the message to all of it's streams during logging.\n",
    "\n",
    "### Logging with `Visdom` - The `trixi` Loggers\n",
    "[`Visdom`](https://github.com/facebookresearch/visdom) is a tool designed to visualize your logs. To use this tool you need to open a port on the machine you want to train on via `visdom -port YOUR_PORTNUMBER` Afterwards just add the handler of your choice to the logger. For more detailed information and customization have a look at [this](https://github.com/facebookresearch/visdom) website.\n",
    "\n",
    "Logging the scalar tensors containing `1`, `2`, `3`, `4` (at the beginning; will increase to show epochwise logging) with the corresponding keys `\"one\"`, `\"two\"`, `\"three\"`, `\"four\"` and two random images with the keys `\"prediction\"` and `\"groundtruth\"` would look like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {}
   },
   "outputs": [],
   "source": [
    "NUM_ITERS = 4\n",
    "\n",
    "# import logging handler and logging module\n",
    "from delira.logging import TrixiHandler\n",
    "from trixi.logger import PytorchVisdomLogger\n",
    "import logging\n",
    "\n",
    "# configure logging module (and root logger)\n",
    "logger_kwargs = {\n",
    "    'name': 'test_env', # name of loggin environment\n",
    "    'port': 9999 # visdom port to connect to\n",
    "}\n",
    "logger_cls = PytorchVisdomLogger\n",
    "\n",
    "# configure logging module (and root logger)\n",
    "logging.basicConfig(level=logging.INFO,\n",
    "                    handlers=[TrixiHandler(logger_cls, **logger_kwargs)])\n",
    "# derive logger from root logger\n",
    "# (don't do `logger = logging.Logger(\"...\")` since this will create a new\n",
    "# logger which is unrelated to the root logger\n",
    "logger = logging.getLogger(\"Test Logger\")\n",
    "\n",
    "# create dict containing the scalar numbers as torch.Tensor\n",
    "scalars = {\"one\": torch.Tensor([1]),\n",
    "           \"two\": torch.Tensor([2]),\n",
    "           \"three\": torch.Tensor([3]),\n",
    "           \"four\": torch.Tensor([4])}\n",
    "\n",
    "# create dict containing the images as torch.Tensor\n",
    "# pytorch awaits tensor dimensionality of \n",
    "# batchsize x image channels x height x width\n",
    "images = {\"prediction\": torch.rand(1, 3, 224, 224),\n",
    "          \"groundtruth\": torch.rand(1, 3, 224, 224)}\n",
    "\n",
    "# Simulate 4 Epochs\n",
    "for i in range(4*NUM_ITERS): \n",
    "    logger.info({\"image_grid\": {\"images\": images[\"prediction\"], \"name\": \"predictions\"}})\n",
    "    \n",
    "    for key, val_tensor in scalars.items():\n",
    "        logger.info({\"value\": {\"value\": val_tensor.item(), \"name\": key}})\n",
    "        scalars[key] += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "## More Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {}
   },
   "source": [
    "More Examples can be found in \n",
    "* [the classification example](classification_pytorch.ipynb, \"Classification\")\n",
    "* [the 2d segmentation example](segmentation_2d_pytorch.ipynb, \"Segmentation 2D\")\n",
    "* [the 3d segmentation example](segmentation_3d_pytorch.ipynb, \"Segmentation 3D\")\n",
    "* [the generative adversarial example](gan_pytorch.ipynb, \"GAN\")"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}