mantichora.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Mantichora](https://github.com/alphatwirl/mantichora)\n",
    "\n",
    "A simple interface to _multiprocessing_ and _threading_\n",
    "\n",
    "[Tai Sakuma](https://github.com/TaiSakuma)\n",
    "\n",
    "_updated on March 29, 2020_\n",
    "\n",
    "_Mantichora_ provides a simple interface to\n",
    "[_multiprocessing_](https://docs.python.org/3/library/multiprocessing.html) and\n",
    "[_threading_](https://docs.python.org/3/library/threading.html).\n",
    "\n",
    "```python\n",
    "from mantichora import mantichora\n",
    "\n",
    "with mantichora() as mcore:\n",
    "    mcore.run(func1)\n",
    "    mcore.run(func2)\n",
    "    mcore.run(func3)\n",
    "    mcore.run(func4)\n",
    "    results = mcore.returns()\n",
    "```\n",
    "\n",
    "You can simply give Mantichora as many functions as you need to run.\n",
    "Mantichora will run them concurrently in background processes by using\n",
    "_multiprocessing_ or in different threads by using _threading_ and give\n",
    "you the return values of the functions. The return values are sorted\n",
    "in the order of the functions you have originally given to Mantichora.\n",
    "Progress bars from [atpbar](https://github.com/alphatwirl/atpbar) can\n",
    "be used in the functions.\n",
    "\n",
    "The code in this package was originally developed in the sub-package\n",
    "[_concurrently_](https://github.com/alphatwirl/alphatwirl/tree/v0.23.2/alphatwirl/concurrently)\n",
    "of [_alphatwirl_](https://github.com/alphatwirl/alphatwirl).\n",
    "\n",
    "*****\n",
    "\n",
    "- **GitHub:** https://github.com/alphatwirl/mantichora\n",
    "- **PyPI:** https://pypi.org/project/mantichora/\n",
    "\n",
    "*****\n",
    "\n",
    "- [**Quick start**](#Quick-start)\n",
    "    - [Import libraries](#Import-libraries)\n",
    "    - [Define a task function](#Define-a-task-function)\n",
    "    - [Run tasks concurrently with Mantichora](#Run-tasks-concurrently-with-Mantichora)\n",
    "- [**Features**](#Features)\n",
    "    - [Multiprocessing or Threading](#Multiprocessing-or-Threading)\n",
    "    - [Without the `with` statement](#Without-the-with-statement)\n",
    "        - [`end()`](#end())\n",
    "        - [`terminate()`](#terminate())\n",
    "    - [Receive results as tasks finish](#Receive-results-as-tasks-finish)\n",
    "        - [`receive_one()`](#receive_one())\n",
    "        - [`receive_finished()`](#receive_finished())\n",
    "- [**License**](#License)\n",
    "- [**Contact**](#Contact)\n",
    "\n",
    "\n",
    "*****\n",
    "## Quick start\n",
    "\n",
    "I will show here how to use Mantichora by simple examples.\n",
    "\n",
    "### Import libraries\n",
    "\n",
    "We are going use two python standard libraries\n",
    "[time](https://docs.python.org/3/library/time.html) and\n",
    "[random](https://docs.python.org/3/library/random.html) in an example\n",
    "task function. In the example task function, we are also going to use\n",
    "[atpbar](https://github.com/alphatwirl/atpbar) for progress bars.\n",
    "Import these packages and `mantichora`.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time, random\n",
    "from atpbar import atpbar\n",
    "from mantichora import mantichora"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define a task function\n",
    "\n",
    "Let us define a simple task function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def task_loop(name, ret=None):\n",
    "    n = random.randint(1000, 10000)\n",
    "    for i in atpbar(range(n), name=name):\n",
    "        time.sleep(0.0001)\n",
    "    return ret"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The task in this function is to sleep for `0.0001` seconds as many\n",
    "times as the number randomly selected from between `1000` and\n",
    "`10000`. `atpbar` is used to show a progress bar. The function takes\n",
    "two arguments: `name`, the label on the progress bar, and `ret`, the\n",
    "return value of the function.\n",
    "\n",
    "**Note:** In the multiprocessing mode, the default mode of mantichora,\n",
    "task functions, their arguments, and their return values need to be\n",
    "[picklable](https://docs.python.org/3.8/library/pickle.html#what-can-be-pickled-and-unpickled).\n",
    "\n",
    "You can just try running this function without using Mantichora. The next cell won't return immediately. It will wait for the function to finish. You will see a progress bar if running interactively. (If you are reading a saved notebook, for exampe, on GitHub, you might just see the string `VBox()` or other text messages instead of the progress bar.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "480cb7316a2a47acb594fbbb5450f0df",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'result1'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "task_loop('task1', 'result1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run tasks concurrently with Mantichora\n",
    "\n",
    "Now, we run multiple tasks concurrently with Mantichora. Progress bars will be shown by `atpbar`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3c9966b7c32a45f2ae1dd162633e6fad",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "with mantichora(nworkers=3) as mcore:\n",
    "    mcore.run(task_loop, 'task', ret='result1')\n",
    "    mcore.run(task_loop, 'another task', ret='result2')\n",
    "    mcore.run(task_loop, 'still another task', ret='result3')\n",
    "    mcore.run(task_loop, 'yet another task', ret='result4')\n",
    "    mcore.run(task_loop, 'task again', ret='result5')\n",
    "    mcore.run(task_loop, 'more task', ret='result6')\n",
    "    results = mcore.returns()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above cell, `mantichora` is initialized with an\n",
    "optional argument `nworkers`. The `nworkers` specifies the number of\n",
    "the workers. It is `3` in the above example. The default is `4`. At\n",
    "most as many tasks as `nworkers` can run concurrently.\n",
    "\n",
    "The [`with`\n",
    "statement](https://docs.python.org/3/reference/compound_stmts.html#the-with-statement)\n",
    "is used in the example. This ensures that `mantichora` properly\n",
    "ends the workers.\n",
    "\n",
    "You can give task functions and their arguments to `mcore.run()`. You\n",
    "can call `mcore.run()` as many times as you need. In the above\n",
    "example, `mcore.run()` is called with the same task function with\n",
    "different arguments. You can also use a different function each time.\n",
    "`mcore.run()` returns immediately; it doesn't wait for the task to\n",
    "finish or even to start. In each call, `mcore.run()` only puts a task in\n",
    "a queue. The workers in background processes pick up tasks from the\n",
    "queue and run them.\n",
    "\n",
    "The `mcore.returns()` waits until all tasks finish and returns their\n",
    "return values, which are sorted in the order of the tasks you have\n",
    "originally given to `mcore.run()` regardless of the order\n",
    "in which the tasks have finished."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['result1', 'result2', 'result3', 'result4', 'result5', 'result6']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*****\n",
    "## Features\n",
    "\n",
    "#### Multiprocessing or Threading\n",
    "        \n",
    "*New in version 0.10.0*\n",
    "\n",
    "From version 0.10.0, you can choose to use\n",
    "[threading](https://docs.python.org/3/library/threading.html) instead\n",
    "of\n",
    "[multiprocessing](https://docs.python.org/3/library/multiprocessing.html)\n",
    "by setting the option `mode` to `threading` (the default is\n",
    "`multiprocessing`).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "99bf2cd0b5a34c478a64a1717e58bdc2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "with mantichora(mode='threading') as mcore:\n",
    "    mcore.run(task_loop, 'task', ret='result1')\n",
    "    mcore.run(task_loop, 'another task', ret='result2')\n",
    "    mcore.run(task_loop, 'still another task', ret='result3')\n",
    "    mcore.run(task_loop, 'yet another task', ret='result4')\n",
    "    mcore.run(task_loop, 'task again', ret='result5')\n",
    "    mcore.run(task_loop, 'more task', ret='result6')\n",
    "    results = mcore.returns()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*****\n",
    "\n",
    "### Without the `with` statement\n",
    "\n",
    "#### `end()`\n",
    "\n",
    "If you don't use the `with` statement, you need to call `end()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "041cf585c46c4ed99d65a48c80a3bcee",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "['result1', 'result2', 'result3', 'result4', 'result5', 'result6']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mcore = mantichora()\n",
    "\n",
    "mcore.run(task_loop, 'task', ret='result1')\n",
    "mcore.run(task_loop, 'another task', ret='result2')\n",
    "mcore.run(task_loop, 'still another task', ret='result3')\n",
    "mcore.run(task_loop, 'yet another task', ret='result4')\n",
    "mcore.run(task_loop, 'task again', ret='result5')\n",
    "mcore.run(task_loop, 'more task', ret='result6')\n",
    "\n",
    "results = mcore.returns()\n",
    "\n",
    "mcore.end()\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `terminate()`\n",
    "\n",
    "`mantichora` can be terminated with `terminate()`. After `terminate()` is called, `end()` still needs to be called. In the example below, `terminate()` is called after 0.5 seconds of sleep while some tasks are still running."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d091331503b84ee08e27ae8ec4f0cc8a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mcore = mantichora()\n",
    "\n",
    "mcore.run(task_loop, 'task', ret='result1')\n",
    "mcore.run(task_loop, 'another task', ret='result2')\n",
    "mcore.run(task_loop, 'still another task', ret='result3')\n",
    "mcore.run(task_loop, 'yet another task', ret='result4')\n",
    "mcore.run(task_loop, 'task again', ret='result5')\n",
    "mcore.run(task_loop, 'more task', ret='result6')\n",
    "\n",
    "time.sleep(0.5)\n",
    "\n",
    "mcore.terminate()\n",
    "mcore.end()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:**: In the threading mode, `terminate()` does not do anything. If you initialize mantichora in the threading mode, i.e., `mantichora(mode='threading')`, in the above cell, all tasks run until completion."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*****\n",
    "\n",
    "### Receive results as tasks finish\n",
    "\n",
    "Instead of waiting for all tasks to finish beofre receiving the reulsts, you can get results as tasks finish with the method `receive_one()` or `receive_receive()`.\n",
    "\n",
    "#### `receive_one()`\n",
    "\n",
    "The method `receive_one()` returns a pair of the run ID and return value of a task function. If no task has finished, `receive_one()` waits until one task finishes. `receive_one()` returns `None` if no tasks are outstanding. The method `run()` returns the run ID for the task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "143b2aee4cbe47228444400f53efbba7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "with mantichora() as mcore:\n",
    "    runids = [ ]\n",
    "    runids.append(mcore.run(task_loop, 'task1', ret='result1'))\n",
    "    runids.append(mcore.run(task_loop, 'task2', ret='result2'))\n",
    "    runids.append(mcore.run(task_loop, 'task3', ret='result3'))\n",
    "    runids.append(mcore.run(task_loop, 'task4', ret='result4'))\n",
    "    runids.append(mcore.run(task_loop, 'task5', ret='result5'))\n",
    "    runids.append(mcore.run(task_loop, 'task6', ret='result6'))\n",
    "\n",
    "    pairs = [ ]\n",
    "    for i in range(len(runids)):\n",
    "        pairs.append(mcore.receive_one())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `runid` is the list of the run IDs in the order of the tasks that have been given to `run()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 2, 3, 4, 5]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "runids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `pairs` are in the order in which the tasks have finished."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(3, 'result4'),\n",
       " (1, 'result2'),\n",
       " (5, 'result6'),\n",
       " (0, 'result1'),\n",
       " (2, 'result3'),\n",
       " (4, 'result5')]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pairs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  `receive_finished()`\n",
    "\n",
    "The method `receive_finished()` returns a list of pairs of the run ID and the return value of finished task functions. The method `receive_finished()` doesn't wait for a task to finish. It returns an empty list if no task has finished."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "032d38dab22a4069938a82436310628f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "with mantichora() as mcore:\n",
    "    runids = [ ]\n",
    "    runids.append(mcore.run(task_loop, 'task1', ret='result1'))\n",
    "    runids.append(mcore.run(task_loop, 'task2', ret='result2'))\n",
    "    runids.append(mcore.run(task_loop, 'task3', ret='result3'))\n",
    "    runids.append(mcore.run(task_loop, 'task4', ret='result4'))\n",
    "    runids.append(mcore.run(task_loop, 'task5', ret='result5'))\n",
    "    runids.append(mcore.run(task_loop, 'task6', ret='result6'))\n",
    "\n",
    "    pairs = [ ]\n",
    "    while len(pairs) < len(runids):\n",
    "        pairs.extend(mcore.receive_finished())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `runid` is again the list of the run IDs in the order of the tasks that have been given to `run()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 2, 3, 4, 5]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "runids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `pairs` are also again in the order in which the tasks have finished."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(1, 'result2'),\n",
       " (0, 'result1'),\n",
       " (3, 'result4'),\n",
       " (4, 'result5'),\n",
       " (5, 'result6'),\n",
       " (2, 'result3')]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pairs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*****\n",
    "\n",
    "## License\n",
    "\n",
    "- Mantichora is licensed under the BSD license.\n",
    "\n",
    "*****\n",
    "\n",
    "## Contact\n",
    "\n",
    "- Tai Sakuma - tai.sakuma@gmail.com"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}