07-interfacing-with-other-languages/07.02-python-extension-modules.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python 扩展模块"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 简介"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "C Library | Interface | Python\n",
    "---|---|---\n",
    "`c header` <br> `c implementation` | Wrapper `C` $\\leftrightarrows$ `Python` <br> communication between `py + c` | `import fact` <br> `fact.fact(10)`\n",
    "\n",
    "**Python** 扩展模块将 `PyInt(10)` 转化为 `CInt(10)` 然后调用 `C` 程序中的 `fact()` 函数进行计算，再将返回的结果转换回 `PyInt`。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 产生一个扩展模块"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "假设我们有这样的一个头文件和程序："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing fact.h\n"
     ]
    }
   ],
   "source": [
    "%%file fact.h\n",
    "#ifndef FACT_H\n",
    "#define FACT_h\n",
    "int fact(int n);\n",
    "#endif"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing fact.c\n"
     ]
    }
   ],
   "source": [
    "%%file fact.c\n",
    "#include \"fact.h\"\n",
    "int fact(int n)\n",
    "{\n",
    "    if (n <= 1) return 1;\n",
    "    else return n * fact(n - 1);\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义包装函数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing fact_wrap.c\n"
     ]
    }
   ],
   "source": [
    "%%file fact_wrap.c\n",
    "\n",
    "/* Must include Python.h before any standard headers*/\n",
    "#include <Python.h>\n",
    "#include \"fact.h\"\n",
    "static PyObject* wrap_fact(PyObject *self, PyObject *args)\n",
    "{\n",
    "    /* Python->C data conversion */\n",
    "    int n, result;\n",
    "    // the string i here means there is only one integer\n",
    "    if (!PyArg_ParseTuple(args, \"i\", &n))\n",
    "        return NULL;\n",
    "    \n",
    "    /* C Function Call */\n",
    "    result = fact(n);\n",
    "    \n",
    "    /* C->Python data conversion */\n",
    "    return Py_BuildValue(\"i\", result);\n",
    "}\n",
    "\n",
    "/* Method table declaring the names of functions exposed to Python*/\n",
    "static PyMethodDef ExampleMethods[] = {\n",
    "    {\"fact\",  wrap_fact, METH_VARARGS, \"Calculate the factorial of n\"},\n",
    "    {NULL, NULL, 0, NULL}        /* Sentinel */\n",
    "};\n",
    "\n",
    "/* Module initialization function called at \"import example\"*/\n",
    "PyMODINIT_FUNC \n",
    "initexample(void)\n",
    "{\n",
    "    (void) Py_InitModule(\"example\", ExampleMethods);\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 手动编译扩展模块"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "手动使用 `gcc` 编译，`Windows` 下如果没有 `gcc`，可以通过 `conda` 进行安装：\n",
    "\n",
    "    conda install mingw4\n",
    "    \n",
    "`Window 64-bit` 下编译需要加上 `-DMS_WIN64` 的选项，`include` 和 `lib` 文件夹的路径对应于本地 **Python** 安装的环境："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!gcc -DMS_WIN64 -c fact.c fact_wrap.c -IC:\\Miniconda\\include"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!gcc -DMS_WIN64 -shared fact.o fact_wrap.o -LC:\\Miniconda\\libs -lpython27 -o example.pyd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Windows` 下最终生成的文件后缀为 `.pyd` ， `Unix` 下生成的文件后缀名为 `.so`。\n",
    "\n",
    "用法为：\n",
    "\n",
    "- `Windows 32-bit`\n",
    "```\n",
    "gcc -c fact.c fact_wrap.c -I<your python path>\\include\n",
    "gcc -shared fact.o fact_wrap.o -L<your python path>\\libs -lpython27 -o example.pyd\n",
    "```\n",
    "- `Unix`\n",
    "```\n",
    "gcc -c fact.c fact_wrap.c -I<your python path>\n",
    "gcc -shared fact.o fact_wrap.o -L<your python path>\\config -lpython27 -o example.so\n",
    "```\n",
    "\n",
    "编译完成后，我们就可以使用 `example` 这个模块了。\n",
    "\n",
    "导入生成的包："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['__doc__', '__file__', '__name__', '__package__', 'fact']\n"
     ]
    }
   ],
   "source": [
    "import example\n",
    "print dir(example)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用 `example` 中的函数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "factorial of 10: 3628800\n"
     ]
    }
   ],
   "source": [
    "print 'factorial of 10:', example.fact(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用 setup.py 进行编译"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "清理刚才生成的文件："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!rm -f example.pyd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "写入 `setup.py`："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing setup.py\n"
     ]
    }
   ],
   "source": [
    "%%file setup.py\n",
    "from distutils.core import setup, Extension\n",
    "\n",
    "ext = Extension(name='example', sources=['fact_wrap.c', 'fact.c'])\n",
    "\n",
    "setup(name='example', ext_modules=[ext])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用 `distutils` 中的函数，我们进行 `build` 和 `install`：\n",
    "\n",
    "    python setup.py build (--compiler=mingw64)\n",
    "    python setup.py install\n",
    "\n",
    "括号中的内容在 `windows` 中可能需要加上。\n",
    "\n",
    "这里我们使用 `build_ext --inplace` 选项将其安装在本地文件夹："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "running build_ext\n",
      "building 'example' extension\n",
      "creating build\n",
      "creating build\\temp.win-amd64-2.7\n",
      "creating build\\temp.win-amd64-2.7\\Release\n",
      "C:\\Miniconda\\Scripts\\gcc.bat -DMS_WIN64 -mdll -O -Wall -IC:\\Miniconda\\include -IC:\\Miniconda\\PC -c fact_wrap.c -o build\\temp.win-amd64-2.7\\Release\\fact_wrap.o\n",
      "C:\\Miniconda\\Scripts\\gcc.bat -DMS_WIN64 -mdll -O -Wall -IC:\\Miniconda\\include -IC:\\Miniconda\\PC -c fact.c -o build\\temp.win-amd64-2.7\\Release\\fact.o\n",
      "writing build\\temp.win-amd64-2.7\\Release\\example.def\n",
      "C:\\Miniconda\\Scripts\\gcc.bat -DMS_WIN64 -shared -s build\\temp.win-amd64-2.7\\Release\\fact_wrap.o build\\temp.win-amd64-2.7\\Release\\fact.o build\\temp.win-amd64-2.7\\Release\\example.def -LC:\\Miniconda\\libs -LC:\\Miniconda\\PCbuild\\amd64 -lpython27 -lmsvcr90 -o \"C:\\Users\\Jin\\Documents\\Git\\python-tutorial\\07. interfacing with other languages\\example.pyd\"\n"
     ]
    }
   ],
   "source": [
    "!python setup.py build_ext --inplace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用编译的模块"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "进行测试："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "factorial of 10: 3628800\n"
     ]
    }
   ],
   "source": [
    "import example\n",
    "\n",
    "print 'factorial of 10:', example.fact(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义 `Python` 函数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3628800\n",
      "3628800\n"
     ]
    }
   ],
   "source": [
    "def pyfact(n):\n",
    "    if n <= 1: return 1\n",
    "    return n * pyfact(n-1)\n",
    "\n",
    "print pyfact(10)\n",
    "print example.fact(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "时间测试："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The slowest run took 13.17 times longer than the fastest. This could mean that an intermediate result is being cached \n",
      "1000000 loops, best of 3: 213 ns per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit example.fact(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000000 loops, best of 3: 1.43 µs per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit pyfact(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果使用 `fact` 计算比较大的值： "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example.fact(100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "会出现溢出的结果，因为 `int` 表示的值有限，但是 `pyfact` 不会有这样的问题："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "93326215443944152681699238856266700490715968264381621468592963895217599993229915608941463976156518286253697920827223758251185210916864000000000000000000000000L"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pyfact(100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将生成的文件压缩到压缩文件中："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import zipfile\n",
    "\n",
    "f = zipfile.ZipFile('07-02-example.zip','w',zipfile.ZIP_DEFLATED)\n",
    "\n",
    "names = 'fact.o fact_wrap.c fact_wrap.o example.pyd setup.py'.split()\n",
    "for name in names:\n",
    "    f.write(name)\n",
    "\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "清理生成的文件："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!rm -f fact*.*\n",
    "!rm -f example.*\n",
    "!rm -f setup*.*\n",
    "!rm -rf build"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}