SAIS-2019.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vectorized Execution in SparkR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This nootebook demonstrates Arrow optimization with some small data (~10 MB) so that people can actually try out and refer when they run the benchmark in an actual cluster.\n",
    "\n",
    "**Note that** the performance improves far more greatly when the size of data is large. Given my benchmark with [500000 Records](http://eforexcel.com/wp/downloads-16-sample-csv-files-data-sets-for-testing/) dataset, I have observed up to around **1000% ~ 5000%** improvement.\n",
    "For more details, see [Databricks' blog](https://databricks.com/blog/2018/08/15/100x-faster-bridge-between-spark-and-r-with-user-defined-functions-on-databricks.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, enable R cell magic to execute R codes in Jupyter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import rpy2.rinterface\n",
    "%load_ext rpy2.ipython"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After that, prepare data to use. In this simple benchmark, [10000 Records](http://eforexcel.com/wp/downloads-16-sample-csv-files-data-sets-for-testing/) dataset is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import urllib.request\n",
    "from zipfile import ZipFile\n",
    "from io import BytesIO\n",
    "\n",
    "# `rownum` can be 10000, 50000, 100000, ....\n",
    "# See http://eforexcel.com/wp/downloads-16-sample-csv-files-data-sets-for-testing/\n",
    "rownum = 10000\n",
    "\n",
    "url = \"http://eforexcel.com/wp/wp-content/uploads/2017/07/%s-Records.zip\" % rownum\n",
    "ZipFile(BytesIO(urllib.request.urlopen(url).read())).extractall()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialize SparkR with enough memory and load libraries used for benchmarking. In this benchmark, it used 1 for shuffle and default parallelism to mimic the case when the large dataset is processed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "R[write to console]: \n",
      "Attaching package: ‘SparkR’\n",
      "\n",
      "\n",
      "R[write to console]: The following objects are masked from ‘package:stats’:\n",
      "\n",
      "    cov, filter, lag, na.omit, predict, sd, var, window\n",
      "\n",
      "\n",
      "R[write to console]: The following objects are masked from ‘package:base’:\n",
      "\n",
      "    as.data.frame, colnames, colnames<-, drop, endsWith, intersect,\n",
      "    rank, rbind, sample, startsWith, subset, summary, transform, union\n",
      "\n",
      "\n",
      "R[write to console]: Spark package found in SPARK_HOME: /home/jovyan/spark\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Launching java with spark-submit command /home/jovyan/spark/bin/spark-submit   sparkr-shell /tmp/Rtmpokvb6k/backend_portf2336d89cd39 \n"
     ]
    }
   ],
   "source": [
    "%%R\n",
    "library(SparkR, lib.loc = c(file.path(Sys.getenv(\"SPARK_HOME\"), \"R\", \"lib\")))\n",
    "# Let's limit core; otherwise, Jupyter might die on some heavy workload.\n",
    "sparkR.session(master = \"local[1]\")\n",
    "library(microbenchmark)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Prepare R DataFrame to test from the data downloaded above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9.9 Mb\n"
     ]
    }
   ],
   "source": [
    "%%R\n",
    "# `rownum` can be 10000, 50000, 100000, ....\n",
    "# See http://eforexcel.com/wp/downloads-16-sample-csv-files-data-sets-for-testing/\n",
    "rownum <- 10000\n",
    "df <- read.csv(paste0(rownum, \" Records.csv\"))\n",
    "print(object.size(df), units = \"MB\")\n",
    "# To load `createDataFrame` faster, let's turn on Arrow optimization\n",
    "sparkR.session(sparkConfig = list(spark.sql.execution.arrow.sparkr.enabled = \"true\"))\n",
    "spark_df <- cache(createDataFrame(df))\n",
    "num <- count(spark_df)  # trigger the count to make sure input DataFrame is cached."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Prepare the common function to use for the benchmarking."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "benchmark_arrow <- function(func) {\n",
    "    microbenchmark(\"No Arrow\" = {\n",
    "                     sparkR.session(sparkConfig = list(spark.sql.execution.arrow.sparkr.enabled = \"false\"))\n",
    "                     func()\n",
    "                   },\n",
    "                   \"Arrow\" = {\n",
    "                     sparkR.session(sparkConfig = list(spark.sql.execution.arrow.sparkr.enabled = \"true\"))\n",
    "                     func()\n",
    "                   }, times = 3L)\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## R DataFrame to Spark DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeAAAAHgCAMAAABKCk6nAAACylBMVEUAAAACAgIDAwMEBAQFBQUGBgYHBwcJCQkKCgoLCwsMDAwNDQ0ODg4PDw8QEBARERESEhITExMUFBQVFRUWFhYXFxcYGBgZGRkaGhobGxscHBwdHR0eHh4fHx8gICAhISEiIiIjIyMkJCQlJSUmJiYnJycoKCgpKSkqKiorKyssLCwtLS0uLi4vLy8wMDAxMTEzMzM0NDQ1NTU2NjY3Nzc4ODg5OTk6Ojo7Ozs8PDw9PT0+Pj4/Pz9AQEBBQUFCQkJDQ0NERERFRUVGRkZHR0dISEhJSUlKSkpLS0tMTExNTU1OTk5PT09QUFBRUVFSUlJTU1NUVFRVVVVWVlZXV1dYWFhZWVlaWlpbW1tcXFxdXV1eXl5fX19gYGBiYmJjY2NkZGRlZWVmZmZnZ2doaGhpaWlqampra2tsbGxtbW1ubm5wcHBxcXFycnJzc3N1dXV2dnZ3d3d4eHh5eXl6enp7e3t9fX1+fn5/f3+AgICBgYGCgoKDg4OEhISHh4eIiIiJiYmMjIyNjY2Ojo6Pj4+QkJCRkZGSkpKTk5OUlJSVlZWXl5eYmJiZmZmampqbm5ucnJydnZ2enp6fn5+hoaGioqKjo6OkpKSlpaWmpqanp6epqamqqqqrq6usrKytra2urq6vr6+wsLCxsbGysrKzs7O0tLS2tra3t7e4uLi5ubm6urq7u7u+vr6/v7/AwMDBwcHCwsLDw8PExMTFxcXGxsbHx8fIyMjJycnKysrLy8vMzMzNzc3Ozs7Pz8/Q0NDS0tLT09PU1NTV1dXW1tbX19fY2NjZ2dna2trb29vc3Nzd3d3e3t7f39/g4ODh4eHi4uLj4+Pk5OTl5eXm5ubn5+fo6Ojp6enq6urr6+vs7Ozt7e3u7u7v7+/w8PDx8fHy8vLz8/P09PT19fX29vb39/f4+Pj5+fn6+vr7+/v8/Pz9/f3+/v7///8+UWDbAAARb0lEQVR4nO3d/WNT1R3H8TjHQymloFYErAhSBJwOGR3Oh1GGSmWiDtSpY6Y+zTEFp1MciutQmbV14FTmFMQpUyZDZ7opyFPROUXRUVRc69AWtpa29PwPu03Sg96kJ/fknpz7vd98Xj8kkHvJve1bwjexyYkIYC0S9AlAbiEwcwjMHAIzh8DMITBzCMwcAjOHwMwhMHMIzBwCM4fAzCEwcwjMHAIzh8DMITBzCMwcAjOHwMwhMHMIzBwCM4fAzCEwcwjMHAIzh8DMITBzCMwcAjOHwMwhMHMIzBwCM4fAzCEwcwjMHAIzh8DMITBzCMwcAjOHwMwhMHMIzBwCM4fAzCEwcwjMHAIzh8DMITBzCMwcAjOHwMz5CNy8BghY25GrwKt/+FsI3lnv5yxwbfZ/FoyJIjBvCMwcAjOHwMwhMHMIzJz/wLFo+cTyaCzldgQmwXfgusJozaqaqsI69wYEJsF34OH18avXRrg3IDAJvgMPaI5f7S9wb0BgEnwHnlcRa+luqa+43L3BSOBXqzNbeIeHnbYaOJlQ8h24NTowEokUVLW6NxgJ/MHGzBYs9rDTRwZOJpQMPE3q2LV9V5r/J2XtIfrhNZYOFEoMngfXPmPpQKHkP/CG2qa2+659KuV2a4GbvrB0oFDyHfj+48eMW3jj9QNScmKKJsF34NGvNx31tJNzonuDtcBv77F0oFDyHbjf5+KYRiE+G+DekDHw64Z+6ih6q6E72p7xiw0h34FP+FD8wnmKtHuYe0PGwGcb+qmjpb82cz8Pzsn4xYaQ78A/Sfxvht9cKG9ZOyPupFkZ/uR5Ge/brs/nBH0GuWDqaVLrQfctt6S8tuWCwDbk7nkwApNgKvCf73LfgsAkmAq88nz3LQhMQoAP0RNOpuWkadl/sXQFGHhchJip2X+xdJkK3Hm7+5aMgacaen3ClMcv8PrFhompwG0pe+LfYBJ8B74hoQqBafId+Oirbu5xIwLT5Dvw6S/Er/AQTZTvwMvXxq868UIHTXipUkJgPQhMAgJLCKwHgUlAYAmB9SAwCQgsIbAeBCYBgSUE1oPAJCCwhMB6EJgEBJYQWA8Ck4DAEgLrQWASEFhCYD0ITAICSwisB4FJQGAJgfUgMAkILCGwHgQmAYElBNaDwCQYCbxzyeLNKTciMAm+Axe1i5f7z5rTb517AwKT4DtwpE1MrxZi5WT3BgQmwUTgke8I0ZyyZgMCk+A/8IrHRzUI0Vjk3oDAJPgOXOlYL8Sqc9wbEJgEU0+TDh5w34LAJOB5sITA6b23+m3nsutB9+0ITILvwM/3H/v1m7vwISxU+Q78jUfEJ+fN7UBgovyvXdgkRPtF3/8CgWnyv+pKz6qAhy7+7pE9NyVWC/wOPtKfAv9rNizoueycd2TPxsRqgZfOyfAnEdgG34EPJVal7PrQvQEP0STgebCEwCpYs4EorNkgIbAeBCYBgSUEVslizQYEtgFrNkgInBbWbKANazZICJwW1mygDWs2SAisB4FJQGAJgfUgMAkILCGwHgQmAYElBNaDwCQgsITAehCYBASWEFgPApOAwBIC60FgEhBYQmA9CEwCAksIrAeBSUBgCYH1IDAJCCwhsB4EJgGBJQTWg8AkILCEwHoQmAQElhBYDwKT4D9wLFo+sTwaS7kdgUnwHbiuMFqzqqaqsM69AYFJ8B14eH386rUR7g0ITIL/DyNtjl/tx6IcNPkOPK8i1tLdUl+RkhOBSfAduDU6MBKJFFS1ujcgMAkGniZ17Nq+qyP1ZgQmAc+DJQROD4tykIZFOSQETguLctCGRTkkBE4rdVGOJAQmIQeLcqyeHFdSkeFPIrANWJRDQmA9CEwCFuWQEFgFi3IQhYdoCYH1IDAJWJRDQmCVLF6qPHc3LTvmeP1iwyTARTlqrjdj7pWG7ujJjF9sCAW4KIcpD6+xdKBQCnBRDlMQWMVD4Mul69Lskf2iHKbUPmPpQKHkIXDxg71Kde7ZWuDPD1o6UCh5CDxT/m62zj1bCwwqAb7QYcpbH1g6kFd7tma2cZOHnVoMnIy3wE/EL1do3XP+Dlmrb8+s/BoPO71p4GS8BU784ztE657zN7AXd7xu6UBeAm/ZMnyL4+lRWveMKVqFVOARI742YsSIUZOf1bpnTNEqrzZZOpC3h+ibsrhnTNEkYIoORlOad3PlhIfAy+TvluvcM4YslTs2WTqQh8Aj9/Y6ReeeEViF0pDVX0r5mAYVTNEqlAJnCVO0CrEpOhuYoklgEBhTtAqDwOEcsghN0VlCYBUMWd5hilbxFLjx0QXXLHi0Ue+eMUWrUJqiG2YPveDWu2+9YOjsHTr3jCmaBA+By59pj/+6fW25zj1jilbBFO1dOIcsYlP0i/MvEZs3aN0zAqvQGrLEA+MeGi3emqp1z5iiVYgFPnGPKBNdRH8mC1O0irfAx7Q5gVuHa90zpmgSvAW+7OddZd0Lr9G6Z0zRKsSm6E/KB/cvmab3qIIhS4XaFN3dsG5Ht949I7AKsSErG5iiVYgFPrh8/mWOvvbauWTx5pQbMUWrEJuiLzrr/jpHuj2K2sXL/WfN6bfOvQFTNAneAhf0/Zck0iamVwuxcrJ7A6ZoFWJT9Dl9v8/NCTzyHSGa9ddNMiWcQxatKbpx2l01jrR7rHh8VIOzR5F7AwKrEBuyFg6Z3fMZHen2qHSsF2LVOe4NmKJViAUu+DDT/Rw84L4FU7QKsSn6lM/07xlTNAneAtd+b+ObDsWOWXxWpSmYolW8BS5JUOyID0LTQ2yKVsj+sypNsRr4rBlmTJhm5n7OujvDCXsIvKVJbElIt0fwn1VpdYqm9hnIOxdk2MFD4NInRGlCuj1SP6uyY3/cTfO0zjR7VqdojoHVUj+rct3cuFMyrZsUSkwDk/4gNKtTNNPA+CC0XiwDe/kgtCzWTTIFgVVMfRBaFusmmYIpWoXBB6FhilZh8DNZVnEM/KvW5G8OVPe9Y768Fs0x8D3Dr14Z2xFbefXwJX3vmC+vRXMMLL6orRw1aFRl7Rfp9siz16JZBlbCa9GByn3g4NdNwhSt4i1wYgmBu1d8lLpH8OsmWcU0cHTAxT+9eMD82QUvej80pmgbDAWufMW5eKVSrD/N+6ExZNlgKHBhl3PRNVh0F3s/NALbYCjwmcsOi8PLpohPNT4xGlO0DYYCN4w57ozjxr4p6ld6PzSmaBtMPU3q3PTc5i69Q2OKtsFU4G2Lrli0Xe/QmKJtMBT4qUHRpdFBq7UOjSHLBkOBy2LORWy81qER2AZDgYsOOReHiP5MFqZoFW+Bpy/pFJ13z9A6NKZoGwwFfnd88aTiU9/TOjSmaBtMTdFdbzz/BtWnSZiiVby8N0nSOjSGLBuMvDdJ0jo0AtuQDz9ViSlahUFgTNEqDAJbhcASpmgb8iEwhiwVBNaDwBLPKXrKGlqWXZ/hhBkEtjpFl0aIyfRdZhDYqtHFtAy+MsMJMwiMKVqFQWAMWSoIrAeBJZ5TNAJLPKdoBJZ4TtEILGGKtiEfAmPIUkFgPXkYOBYtn1gejaXcjinahtwHriuM1qyqqSpMWfcOU7QNuQ88vD5+9VrKO4cxRduQ+8ADmuNX+4Nb2g5TtIrvwPMqYi3dLfUVKTkxZNmQ+8Ct0YGRSKSgqtW9AYFtsPE0qWPX9l1plnnCFG1DPjwPxhStYipwgB8nbFXeBg7wsyoxRatgaTs9+Rc4+I8TRmCVHHyc8NrEsoknTs/4Z83AFK2Sg6XtkjBF25APT5OsQmAJU7QN1gJjabtgWAsc4NJ2CKzC4CEaU7QKg8CYolXwWrSevA2M16KDgdeiDcu/wHgtOlBhX9ruL7dnNnOOh502GTiZHvkXOLevRe/ZmNlzL3jY6d8GTqZH/gXuE6ZoGxDYMASWENgGBDYMgSUEtgGBDUNgCYFtQGDDEFhCYBsQ2DAElhDYBgQ2DIElBLYBgQ1DYAmBbUBgwxBYQmAbENgwBJYQ2AYENgyBJZ6Bz6im5RYENuuVoNeyc1P3Q2D2EDgYa/dYOhACB+OO1y0dCIGDgcDMLd5q6UAIHIxOWwdCYOYQOBiYopnDkMUcAjOHKZo5TNFghpHAO5cs3pxyIwKrhGeKLmoXL/efNaffOvcGBFYJz5AVaRPTq4VYOdm9AYFVQhV45DtCNAe3OGUohWeKjqx4fFSDEI1F7g0IrBKeKbrSsV6IVee4NyAwCaaeJh084L4FgVXCM0X3CYFVwjNkiVi0fGJ5NJZyOwKrhCdwXWG0ZlVNVWGdewMCq4Rnih5eH796bYR7AwKrhGeKHtAcv9qP58E0+Q48ryLW0t1SX3EkZ3JxytKZvk+OsfBM0a3RgZFIpKCq1b1hdW32Z8VfeIYsITp2bd/VkXozAquEKXAfEFglPFN0UurilAisEp4pOil1cUoEJgEP0cEIzxTdJwRWCd2Qlbo4JQKrhC5w6spnCKwSnim6z8UpEVglPFN0n4tTIjAJuVucEoFVwjNF97k4JQKrhG7ISoXAKgjMXHim6D4hsEp4pug+ITAJCByM8EzRfUJgFQxZzCEwc5iimcMUDWYgcDAwRTOHIYs5BGYOUzRzmKLBDAQOBqZo5jBkMYfAIVYzN7MJMzzstMnAySCweW37M2vysM/+LgMng8DMITBzCMwcAjOHwMwhMHMIzBwCM4fAzCEwcwjMXO4Cv3T6DDvGDh4aPgVTLH13yj7OVWBrHl4T9Blkwdr/LswAgXMEgb1DYB8QOEcQ2DsE9gGBcwSBvat7NugzyMKdqStqByIMgdvSLBhB3sHuoM8gIQyBwQcEZg6BmUNg5hCYOQRmDoGZQ2Dm6ASeWPiZEE9Pdt167pD2QM4mCzRPlVDgobelBv7wqOLkC9Gd8oIooqdKKPDSwn3xwO/NGDJ+dfLGe6bdWulclVR/c3ziIrHxsYuEGDNXiJE7gjzjryJ6qoQCP/OjBT2Bu05dfCg2aHvixjG1W4/+1PmuzWrvjl8kN+4uOvzJiSeI3UMOB3vOX0b0VCkF3j2o0Qn8RrHz6BZdFL+t/uhmUfaA813bIBIXvRtHbnv6uin/fGx2sKf8ZVRPlVJgcX2VE/j5Sc5vll4Rv61qlvPQd7rzDdspEhe9G6+6/6Ynb3tk/rIAz9eF6qmSCtxYcN9X/gb/b3BBScmQSIMoedP5Xc9F78YVsyftffHKk7YEfM5HkD1VUoHFgqHOv8Hjl3T8rXBbz01PFe/Zt2/f2QuPfNd6N747aIxoKS408RkXZpA9VVqB9w10puh3pxeVPRW/6fyFPZerSzrld01uPP7HQky+ILCTTUH2VOkEhpxAYOYQmDkEZg6BmUNg5hCYOQRmDoGZQ2DmEJg5BGYOgZlDYOYQmDkEZg6BmUNg5hCYOQT+ilI6P6dpCPfAEyOOSs+7P9GUw3MJBPvAj7W1pf8YJirvDssx9oF/H796v3ib+HjYK6Lk3vIJNxxKvkXs07nHlj4gxC+PLxq3OXnpPEQn3/tWUj1l7HwG/xHkSWCxYvx/Zy5ysl3Y2TlraeItYt3fXtT2r3Evbi1tEh/sTVw6gXvf+1ZyaUfHt/4Q7MmbwD7woGHDht3r/GL2pNPanbDrhXhpQuLdYQ2Du4R4aH7DsRudv9KJSydw7ztOSv4qxJ2Lgj59/9gHfmjv3r0tzi/+FFnhXJZsc1oOSbw7bF3/srKy0ZeI3505+Kqm5GXplt73jPW8OWHpDQGfvQHsAycfog+cHD3hP07YtUL8cULivSXbS3o/TrL5op8lL7/0NxiBQyE+RTsPvtfOFdfNdcKWN3125j2JwIfLbzvQ9Y/Nb/2989AVixKXPf8GJ98zhsDhEH8efLp43vnre2DMk6Jk+dih17Un3x326VUlQ6Zu2HRG4TE/2J+47Jmik+8ZQ+BQiofNKwjMHAIzl2eB8w8CM4fAzCEwcwjMHAIzh8DMITBzCMwcAjOHwMwhMHMIzBwCM4fAzCEwc/8Hw4UF6drtA5MAAAAASUVORK5CYII=\n"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%R\n",
    "func <- function() { createDataFrame(df) }\n",
    "boxplot(benchmark_arrow(func))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Spark DataFrame to R DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeAAAAHgCAMAAABKCk6nAAACylBMVEUAAAABAQECAgIDAwMEBAQFBQUGBgYHBwcJCQkKCgoLCwsMDAwNDQ0ODg4PDw8QEBARERESEhITExMUFBQVFRUWFhYXFxcYGBgZGRkaGhobGxscHBwdHR0eHh4fHx8gICAhISEiIiIjIyMkJCQlJSUmJiYnJycoKCgpKSkqKiorKyssLCwtLS0uLi4vLy8wMDAxMTEzMzM0NDQ1NTU2NjY3Nzc4ODg5OTk6Ojo7Ozs8PDw9PT0+Pj4/Pz9AQEBBQUFCQkJDQ0NERERFRUVGRkZHR0dISEhJSUlKSkpLS0tMTExNTU1OTk5PT09QUFBRUVFSUlJTU1NUVFRVVVVWVlZXV1dYWFhZWVlaWlpbW1tcXFxdXV1eXl5fX19gYGBhYWFiYmJjY2NkZGRlZWVmZmZnZ2doaGhpaWlqampra2tsbGxtbW1ubm5vb29wcHBxcXFycnJzc3N0dHR1dXV2dnZ3d3d4eHh6enp7e3t9fX1+fn5/f3+AgICBgYGCgoKDg4OEhISFhYWHh4eIiIiMjIyNjY2Ojo6Pj4+QkJCSkpKTk5OUlJSVlZWXl5eYmJiZmZmampqbm5ucnJydnZ2enp6fn5+goKChoaGioqKjo6OkpKSlpaWmpqanp6epqamqqqqrq6usrKytra2urq6vr6+wsLCxsbGysrKzs7O0tLS2tra4uLi5ubm6urq7u7u/v7/AwMDBwcHCwsLDw8PExMTFxcXGxsbHx8fIyMjJycnKysrLy8vMzMzNzc3Ozs7Pz8/Q0NDR0dHS0tLT09PU1NTV1dXW1tbX19fZ2dna2trb29vc3Nzd3d3e3t7f39/g4ODh4eHi4uLj4+Pk5OTl5eXm5ubn5+fo6Ojq6urr6+vs7Ozt7e3u7u7v7+/w8PDx8fHy8vLz8/P09PT19fX29vb39/f4+Pj5+fn6+vr7+/v8/Pz9/f3+/v7///+yCn4xAAAOeklEQVR4nO3d+2MU1RnG8W2rBZNAEtQlRowYJIgIItQIarlVqSFQa6O2qFgDWsVYJBVULEgLqIigYK22VoNYQ4tSraG2QLikohYQESKXYClXc9kl53/oZHczwAay52RmZ9598nx+mJCZ4exJvu7uWclsAoqgBfyeACUXA4NjYHAMDI6BwTEwOAYGx8DgGBgcA4NjYHAMDI6BwTEwOAYGx8DgGBgcA4NjYHAMDI6BwTEwOAYGx8DgGBgcA4NjYHAMDI6BwTEwOAYGx8DgGBgcA4NjYHAMDI6BwTEwOAYGx8DgGBgcA4NjYHAMDI6BwTEwOAYGx8DgGBgcA4NjYHAMDI6BwWkE3l7xmbUNP5f0uVASJA5c2aXPuVPDqp739ZSUONvAF9XeESVNDJyaEmfrWqdUw7jxhxg4JSXO1nuDtWmcMLLNmQeWkwArmhwGfqisZRu6vc2ZFT97ifw3/HOHgRuPRD6Ed9p7VoyJyL8t4d+l5Ct1Gvispt3R8b9LrnErcGhm/B4GFsGtwG1fJjGwCI4DPxA1mYFlchz4nIlTW/yCgWVyHHjQXyMf+BAtlOPAC1dEPoSejD/AwCLwZRI4BgbHwOCEB/5wfmLTH9c4aYMLk0lJwgN/8X5iZbM0TvrKhcmkJOGBdbyw3KMbSkkAgRe/5dENpSSAwHWHPLqhlAQQmNoDEPizXR7dUEoCCMxFVnsYGBxAYK6i2wMQmKvo9jgPXFVaeGVhaVWb/VxFi+A48JKM0kV/WDQ5Y0n8Aa6iRXAcOGdN5MM/c+MPcJElguPAXQ9EPhxMiz/AwCI4Dnz76KrDzYfXjG6Tk6toERwHPlJ6XiAQSJt8JP4AV9EiuPAyqWnbpm2nXsJWE73qaXSRo4mRO5LwOnhb9LrFovEdm5ExrqLbA3BtEhdZ7fHx2qQvN7hjxnyXBoL8sR4fr026caY77p3izjjTbk34xaYgH69NGpFwbG/970d+zyAZfLw2iYG94OO1SQzsBR//uZCBvcDANgY2w8AiMLCNgc0wsAgMbGNgMwwsAgPbGNgMA4vAwDYGNsPAIjCwjYHNMLAIDGxjYDMMLAID2xjYDAOLwMA2BjaTMPDg+2W5d2THv1i5fAzcNyDMtR3/YuXiPdjGe7AZPgeLwMA2BjaTMPBdY2QZ9UjHv1i5AN5G6W/rPbqhlOQ48Imlv1yt5gx/qM1l9rx8VATHgcsvndz74WGLrrsn/gADi+A4cPBztS1Qq/YE7T2xXy97ySjHk1PqVY3nzv6DNE5a6cJkUpLjwGnH1fFAvWrKiD/Ad7oTwXHgG6d8OLnvb4/OHx5/gIFFcBx4y+DMueuyAxeujT/AwCK48zLp+PZQm30MLALA62BqDwODY2BwDAyOgcExMDgGBsfA4BgYHAODY2BwDAyOgcExMDgGBsfA4BgYHAODY2BwDAyOgcExMDgGBsfA4BgYHAODY2BwDAyOgcExMDgGBsfA4BgYHAODY2BwDAyOgcE5Drz+S1U/a8iQ2Q3xBxhYBMeBL9+kpg5c9vKgafEHGFgEx4G/W6d67lZqT097T+hgxIMMLIHze/AqlbNfqQPd7T1vx96MdLTjyZFzjgP/PufVeddXVt4wJf4AH6JFcL6KXjn4W4HA+TO4yJLJjZdJDTv2nmEvA4vA18HgGBgcA4NjYHAMDI6BwTEwOAYGx8DgGBgcA4NjYHAMDI6BwTEwOAYGx8DgGBgcA4NjYHAMDI6BwTEwOAYGx8DgNALfYWtz/VF7GFgEjcBZz7XKMxmZgUXQCHyT/VmxycgMLAKfg8HpBX4tsl1qNDIDi6AXOPrkm2k0MgOLoBO4ujqn2vJmL6ORGVgEncC5ud/Jzc3tdc3bRiMzsAh6D9EPdmBkBhaBq2hwGoEX2J8tNBmZgUXQCHxxbavLTUZmYBE0Anex5ZqMzMAi8DkYHAODY2BwDAyOgcExMDitwLtfLru77OXdZiMzsAgagWuKs8c++tSjY7OLN5uMzMAiaAQufCv6XtANKwpNRmZgEfgcDE4z8KpJt6p1q41GZmAR9AI/2/f53uqTa41GZmAR9AJfsksVqDB/JisF6QU+v94KfCTHaGQGFkEv8G0zwgXN0+82GpmBRdALvLewW5fgsDqjkRlYBM1VdHPNys3NZiMzsAh8HQxOL/CxhZNus5z9vLz9bXYxsAh6gccNf2aJ5Uxn3Bzx3ZE3xx9gYBH0AqcdO+sZXa+fZ0l/cp69h79eVhK9wDduOesZO8cVf6FUcN/JPfz1spLoBd497MlFljOf807f2fWnBo7hQ7QIeoGnZxa3vEfHWU46NqPgPAYWSvM5eGf7o3y65Js2+xhYBL3Al39tPjIDi6AXePEP3t9iMRqZgUXQCxyMMhqZgUXg/6oEpxG4uk5VRxmNzMAiaATOe03lRRmNzMAi8CEaHN8IDRzfCA0c3wgNHN8IDRzfCA0cV9HgNALPOxL75Oh8k5EZWASNwHNy7lpWtblq2V05s01GZmARdB6iDy0u6pXeq2jxIaORGVgEPgeDY2BweoFnRjy19CuDkRlYBL3ApV0nPDyh66TitFX6IzOwCHqBiz6wNh8UqXev0h+ZgUXQC5wRtjbhbqo5S39kBhZBL/CQBSfUiQVD1X6Dd4xmYBH0AtfkX3j1hX22qDXL9EdmYBE0XyaF1r6zLmw2MgOLoBl4Y/md5ZvMRmZgEfQCv5FeOrc0vcJoZAYWQS9wQZW1qepnNDIDi6AXuHujtWnkz2SlIL3Ao2aHVOipMUYjM7AIeoG39ssakHXFdqORGVgEzVV0eH3ler5MSkU61ybZjEZmYBF0rk2yGY3MwCLwH/zBMTA4BgbHwOAYGBwDg2NgcAwMjoHBMTA4BgbHwOAYGBwDg2NgcAwMjoHBMTA4BgbHwOAYGBwDg2NgcAwMjoHBMTA4BgbHwOAYGBwDg2NgcAwMjoHBMTA4BgbHwOAYGBwDg2NgcAwMjoHBMTA4BgbHwO5bMCaxAcM1TvrIhckwsJnKl9wxdqZLA32cYMIMbCbvCnfkF7gzzmV3J5gwA5vJCwiT6LvMwGYGzpTlvp8nmDADmxnh9wTifFyW4AQGNsPANgb2AgO7jIFtDOwFBnYZA9sY2AsM7DIGtjGwFxjYZQxsY2AvMLDLGNjGwF5gYJcxsI2BvcDALmNgGwN7gYFdxsA2BvaCB4G3V3xmbcPPxe9nYC8kP3Bllz7nTg2r+jZnMrAXkh944Itq74iSplMCb10eUTReZ4KppvMF7lqnVMO48YdOnvnv6I/cT5ymM8FU0/kC995gbRonjGxzZsXihH83BXW+wA9FbiF0OwP7IvmBG49EPoR3xh9gYC/4+DqYgb3AwC5jYBsDe4GBXcbANgb2AgO7jIFtmIEHarx3ig6tN2HRUPirBBNmYH88/i+PboiB/TFrg0c3xMD+CHl1QwwMjoH9sWKXRzfEwP7gIgscA4PjKhocV9HkDgb2B1fR4LjIAsfA4LiKBsdVNLmDgf3BVTQ4LrLAMTA4rqLBcRVN7mBgf3AVDY6LLHAMDI6raHBcRZM7GNgfXEWD4yILHAOD4yoaHFfR5A4G9gdX0eC4yALHwOC4igbHVTS5g4H9wVU0OC6ywDFwCltUklj/MRonrXVhMgzsvvqDidVpnHMw7MJkGBgcA4NjYHAMDI6BwTEwOAYGx8DgGBgcA4NjYHDJC/zeIHd+9VNCfbplp560oR59dwr2JCuwZ15Y7vcMOsCzfy5MgIGThIH1MbADDJwkDKyPgR1g4CRhYH1L3vZ7Bh3wxDq/ZxCVCoHrm/yeQQcca/Z7BlGpEJgcYGBwDAyOgcExMDgGBsfA4BgYnJzAV2Z8rdSb18Tt/X5mgy+z6QCZUxUUOPuxtoF3fjsr9j+iQ/ZGKKFTFRR4bsa+SODtYzL7VcR2zhn2aJH1ITh/cL/oJnrwlXFK5ZcodfFmP2d8OqFTFRT4rXvKWgKHr5jVWJW+Kbozf/GGc/Zb37VbGpojm9jBHd1P7L3kIrUj84S/cz6V0KlKCrwjfbcVeH2W9ehWWh7Zt+acA6rgWeu7tlpFN60HL9745pSh/3ml2N8pn0rqVCUFVvdPtgJXDrA+mXtnZN/kW6yHvkHWN+xjFd20Hpz4zIOvP/bipAV+Tvh0UqcqKvDutN+cdg/+pltaMJgZqFHBLdZnLZvWg0uLB9Su+uml1T7P+SSxUxUVWJVlW8/B/WY3/SNjY8uuN7J27du374bpJ79rrQe3puerw1kZbrzHhTvETlVW4H3nWavoraO6F7wR2XXz9JZtRTBkf9fsgz3vVeqasf7NNp7YqcoJTEnBwOAYGBwDg2NgcAwMjoHBMTA4BgbHwOAYGBwDg2NgcAwMjoHBMTA4BgbHwOAYGBwDnyZPzs9pugQ98JUBS5H26a/VJXEuvoAP/Ep9/ZnfhknK1WFJBh/4j5EPn2dtVHt6fKCCTxf2f6AxdonY/pIL8p5V6tc9u/ddF9taD9Gxa9+C84f2mQTwH0EnCayW9jt+U7mV7Yeh0C1zo5eINV9XXv9l31Ub8urUF7XRrRW49dq34I+bmr73J38n7wb4wOk9evR42vpD8YCrGqyw7yr1Xv/o1WE13cJKPT+p5oL3rbt0dGsFbr3iJPh3pZ4o93v6zsEHfr62tvaw9Ye/BJZa2+BGq2Vm9OqwlV0KCgp636p+N6TbxLrYNq+69ZqxlosT5j7g8+xdAB849hB99LLSi/5rhV2h1J/7R68t2RRsfTvJA+MeiW1PuQczcEqIrKKtB9/7StSUEitsYd3XQ+ZEA58ofOxo+NN1n3wUaryzPLpteQ6OXTPGwKkh8jp4kKq07r5H819XwYV9sqc0xK4O2z8xmHnt6rVXZ5z/k4PRbcsqOnbNGAOnpEjYToWBwTEwuE4WuPNhYHAMDI6BwTEwOAYGx8DgGBgcA4NjYHAMDI6BwTEwOAYGx8DgGBjc/wFAUCOauNhycAAAAABJRU5ErkJggg==\n"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%R\n",
    "func <- function() { collect(spark_df) }\n",
    "boxplot(benchmark_arrow(func))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `dapply`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeAAAAHgCAMAAABKCk6nAAACvlBMVEUAAAACAgIDAwMEBAQFBQUGBgYHBwcJCQkKCgoLCwsMDAwNDQ0ODg4PDw8QEBARERESEhITExMUFBQVFRUWFhYXFxcYGBgZGRkaGhobGxscHBwdHR0eHh4fHx8gICAhISEiIiIjIyMkJCQlJSUmJiYnJycoKCgpKSkqKiorKyssLCwtLS0uLi4vLy8wMDAxMTEyMjIzMzM0NDQ1NTU2NjY3Nzc4ODg5OTk6Ojo7Ozs9PT0+Pj4/Pz9AQEBBQUFCQkJDQ0NERERFRUVGRkZHR0dISEhJSUlKSkpLS0tMTExNTU1OTk5PT09QUFBRUVFSUlJTU1NUVFRVVVVWVlZXV1dYWFhZWVlaWlpbW1tcXFxdXV1eXl5fX19gYGBiYmJjY2NkZGRlZWVmZmZnZ2doaGhpaWlqampra2tsbGxtbW1ubm5wcHBxcXFycnJzc3N1dXV2dnZ3d3d4eHh6enp7e3t9fX1+fn5/f3+AgICBgYGCgoKDg4OEhISFhYWHh4eIiIiMjIyNjY2Ojo6Pj4+QkJCRkZGSkpKTk5OUlJSVlZWXl5eYmJiZmZmampqbm5ucnJydnZ2enp6fn5+hoaGioqKjo6OkpKSlpaWmpqanp6epqamqqqqrq6usrKytra2urq6vr6+wsLCxsbGysrKzs7O0tLS2tra4uLi5ubm7u7u+vr6/v7/AwMDBwcHCwsLDw8PExMTFxcXGxsbHx8fIyMjJycnKysrLy8vMzMzNzc3Ozs7Pz8/Q0NDS0tLT09PU1NTV1dXW1tbX19fY2NjZ2dna2trb29vc3Nzd3d3e3t7f39/g4ODh4eHi4uLj4+Pk5OTl5eXm5ubn5+fo6Ojq6urr6+vs7Ozt7e3u7u7v7+/w8PDx8fHy8vLz8/P09PT19fX29vb39/f4+Pj5+fn6+vr7+/v8/Pz9/f3+/v7///8K2ZjRAAANtElEQVR4nO3d/WNVdQHH8ZUGjsEG6hUHLAQdAj4F5NIIYRkBE81AEw3b0NSWgqGCKBWoKIKCmeQDOA0Kk7SRJvIoZgkqCQoMI55qT5d9/4vO7t2ueAfe79k5u+e7D+/XD+eyc47f+919ex8O2znkGEjLiXoC6FgEFkdgcQQWR2BxBBZHYHEEFkdgcQQWR2BxBBZHYHEEFkdgcQQWR2BxBBZHYHEEFkdgcQQWR2BxBBZHYHEEFkdgcQQWR2BxBBZHYHEEFkdgcQQWR2BxBBZHYHEEFkdgcQQWR2BxBBZHYHEEFkdgcQQWR2BxBBZHYHEEFkdgcQQWR2BxBBZHYHEEFkdgcQEC71sBB1Q1dFTg5Tc8iehd/kGHBV7U/v8WoSknsDYCiyOwOAKLI7A4AosjsDgCiyOwOAKLczzwX+ZlNv0ei502hDCZTsnxwB+tyaxylsVOn4QwmU7J8cA2Hl+RpTvqlAQCL3oxS3fUKUUYeMeGcKx5PaSBdrX/m3VXhIFHznTLnde0/5t1V4SBR7V/7A7xn6ujnkFHIHAKgf3JGPiSW9xy8xXt/2bdFWHgc3Mcc2n7v1l3RRh46cjMhvTOrNcZFjtdYHFnkodbAsfB+DIEFkdgcQQWR2BxBBZHYHEEFkdgcQQWR2BxBBZHYHEEFkdgcQQWR2BxBBYXPHB1ecmQkvLqNusJ7ITAgRfnlS9ctrAib3H6BgI7IXDg3msTN28Wpm8gsBMCB+66L3GzPzd9A4GdEDjw5DHVB5sOrh1zXfoGAjshcOBD5afl5OTkVhxK30BgJ4RwmNSwbdO2Yy9KXFWacM7EQBNDODruOPjONi/aiEDwwNuXv+ct44+kryewEwIHXtll4Kl3xE1tmz0J7ITAgS98wuweNamBwI4KfhxcY0zdhKsOENhNgQP3b77EWP3EKwjspsCBb69sXjZOJrCbAgeuT/4NR3xH+gYCO4HjYHEEFkdgcQQWR2BxBBZHYHEEFkdgcQQWR2BxBBZHYHEEFkdgcQQWR2BxBBZHYHEEFkdgcQQWR2BxBBZHYHEEFkdgcQQWR2BxBBZHYHEEFkdgcQQWR2BxBBZHYHEEFkdgcQQWR2BxBBZHYHEEFkdgcQQWR2BxBBZHYHEEFkdgcQQWR2BxBBZHYHEEFkdgcQQWR2BxBBZHYHEEFkdgcQQWR2BxBBZHYHEEFkdgcQQWR2BxBBZHYHEEFkdgccEDV5eXDCkpr26znsBOCBx4cV75wmULK/IWp28gsBMCB+69NnHzZmH6BgI7IXDgrvsSN/tz0zcQ2AmBA08eU32w6eDaMW1yEtgJgQMfKj8tJycnt+JQ+gYCOyGEw6SGbZu2NRzzdVVpQr/RgSaGcFgEvi5lmp+ReQY7wSJwwSOtik68Y+PM9DUEdoJF4CtTX5WdeMfaNi/mBHZC4PfgW5MqCOwmu8DPJJZLjrfHKVPuaPZTArvJLnDyzTf/eHtc9MfEDS/RjrIJvH597/WeF/oeb48FVYmbxvvSNxDYCTaBCwu/WlhY2HfYS75GJrAT7F6ib2vHyAR2Aj/wF2cReH7qqwV+RiawEywC99nV6lw/IxPYCRaBu6S0+aH+lyGwE3gPFkdgcQQWR2BxBBZHYHFWgXc+VXlT5VM7/Y1MYCdYBN5S1nPsXbPvGtuzbLOfkQnsBIvAJS/WJf5cV1XiZ2QCO4H3YHGWgVdPvcase9XXyAR2gl3gh897rL9591JfIxPYCXaB+31sik38uL+TdUIEdoJd4NNrvcCHevsamcBOsAt87S/ixU3Tb/I1MoGdYBd4d0n3LrHLanyNTGAnWH6KbtqyanOTv5EJ7ASOg8XZBT6yYOq1Hl8jE9gJdoEnXP7QYo+vkQnsBLvAuUf8j0xgJ9gFHrnV/8gEdoJd4J2X3bfQ42tkAjvBLvD0/LLma3T4GpnATrB8D97hf2QCO8Eu8Lmf+R+ZwE6wC7zou2u2enyNTGAn2AWOJfkamcBO4K8qxVkEXl9j1if5GpnATrAIXPSMKUryNTKBncBLtLjAF0I7IQI7IfCF0E6IwE4IfCG0EyKwE7gQmjguhCaOT9HiLAL/uvXf2zg8z8/IBHaCReAHet+4tHpz9dIbe8/xMzKBnWDzEn1g0fi+3fqOX3TA18gEdgLvweIILM4u8MyE2Us+8TEygZ1gF7i868SfTew6tSx3tf3IBHaCXeDxr3mL18abVy6wH5nATrALnBf3FvHupqnAfmQCO8Eu8PD5R83R+SPMXh9XjCawE+wCbxlw5sVnDtxq1i61H5nATrA8TGp86/fr4v5GJrATLANvnHH9jE3+RiawE+wCP9+tfG55t+W+RiawE+wCF1d7i+pBvkYmsBPsAveo9xb1/E5WJ2QXePScRtM4u9TXyAR2gl3g9wcVDC04f7uvkQnsBMtP0fG3V77NYVJnZHNuUoqvkQnsBJtzk1J8jUxgJ/ADf3EEFkdgcQQWR2BxBBZHYHEEFkdgcQQWR2BxBBZHYHEEFkdgcaEEfmfOrHVtVhLYCYED96gzf+4y7uqvrUrfQGAnBA6cU2tGzzNm6bDUmqrShH6jA08OwYURuM8/jdmXm76BZ7ATggde8tu+W4zZ2SN9A4GdEDjweM8rxiwbmb6BwE4I6zDpyOH0NQR2AsfB4ggsjsDiCCyOwOIILI7A4ggsjsDiCCyOwOIILI7A4ggsjsDiCCyOwOIILI7A4ggsjsDiCCyOwOIILI7A4ggsjsDiCCyOwOIILI7A4ggsjsDiCCyOwOIILI7A4ggsjsDiCCyOwOIILI7A4ggsjsDiCBy++aWZDb3cYqc3QpgMgaNxz9+ydEcEjgaBxd2/OUt3RGBxBBZH4Gg8tyNLd0TgaPAhSxyBxfEpGuEgsDgCR4NP0eL4kCWOwOL4FI1wEFgcgaPBp2hHvfxkOMbODGmgTRkmTGB/hq8Ix6PLwhlnfmWGCRPYn1FRTyDNO5UZdiCwPwROIXA2EDhkBE4hcDYQOGQETiFwNhA4ZAROIXA2EDhkBE4hcDYQOGQETiFwNhA4ZAROIXA2EDhkBE4hcDYQOGQETiFwNhA4ZARO0Qw8LKRfhgzL7IoMEyawP0U5jsn0KBPYn4HnuKXPDRkmTGB/eA9OIXA2EDhkBE4hcDYQOGQETtEM/I2Qjl8ffDSccWZXZpgwgf1ZFU6X8E4ffSfDhAkcjVkbsnRHBI5GY7buiMDiCByNqo+zdEcEjgYXQhNH4E5s4aTMBpda7PRWCJMhcPhq92dWY7HP/ngIkyGwOAKLI7A4AosjsDgCiyOwOAKLCx54+/L3vGX8kfT1BHZC4MAruww89Y64qW2zJ4GdEDjwhU+Y3aMmNRwTuKo0od/owJNDcIEDd60xpm7CVQfa7PmrfqXZMbB7z84nd0SWHp3iTwMG7t/8y0X1E68I8HEsoMdXRHbX7Ze1HxdmkDnb7ZXNy8bJBPal8wSuP5S4iWfrn4lpi8ABRPe8tEfgAAjcQQhsj8ABdIbAi1+KegbtcO+6qGeQ1BkC1zZEPYN2ONIU9QySOkNgBEBgcQQWR2BxBBZHYHEEFkdgce4EHpL3mTEvDEtb+538ukhm0w5uTtWhwD3vbht4x1cKWv4iujG1cJSjU3Uo8Ny8PYnA20vzBy1vWfnAZXeN925i8y4ZlFwkNz49wZgBk4zpsznCCadxdKoOBX7xx5XNgePnz6qv7rYpuXLAog2n7PUetXF1TYlFy8YPexzd3e9s82H+0WjnfCxHp+pS4A+77fQCv13gvbqVz0isW3vKPlP8sPeovWqSi9aNfTa+MG3EP54ui3bKx3J1qi4FNrdUeIFXDvW+mHt9Yl3FOO+l7yLvAWu+3FvzonXjlIdue/buJ6bOj3LCX+TqVJ0KvDP3l194Bv+ve24slp+zxcS2el81L1o3Likbumv1j76+PuI5f87ZqToV2FT29N6DB81p+GvexuZVzxd8vGfPnm9P//xRa934frcB5mBBXhjXuAiHs1N1K/Ce07xP0e+P7lH8fGLV96Y3L5fHGlOPWmrjWTcbM2xsdLNN5+xU3QmMDkFgcQQWR2BxBBZHYHEEFkdgcQQWR2BxBBZHYHEEFkdgcQQWR2BxBBZHYHEEFkfgLyhy5/c0Q6IeeEiOZ7z17s/UdOBcIiEf+Ona2uNfhsmVs8M6mHzg3yVuPijYaD7t9ZqJPVgy+Nb6llPE9k46o+hhY+4/q8d561qW3kt0y7lvsXkjBk4V+J/gJAlslgz675UzvGzfb2wcNzd5iljTt2bU/uu81RuKasxHu5JLL3DruW+xHzQ0fPO5aCcfBvnA3Xr16vWg94eyoRfUeWFfMeZPg5Nnh23pHjfmsalbzljjPaWTSy9w6xknsdeNuXdG1NMPTj7wY7t27Tro/eEPOUu8ZWyj1zI/eXbYqi7FxcX9rzG/Gd59Sk3Lsmh96zljzScnzL014tmHQD5wy0v04XPKz/63F7bKmJcHJ88t2RRrvZzkvgk/b1ke8wwmcKeQ+BTtvfj+ZJKZNskLW1Lz2fAHkoGPltx9OP73de++0Vh//Yzksvk9uOWcMQJ3Donj4IvMSu/pe3jAsya2YGDPaXUtZ4ftnRLLv/TVty7OO/2H+5PL5k/RLeeMEbhTSoQ9qRBYHIHFnWSBTz4EFkdgcQQWR2BxBBZHYHEEFkdgcQQWR2BxBBZHYHEEFkdgcQQW93+SQ9xtkq0mGQAAAABJRU5ErkJggg==\n"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%R\n",
    "func <- function() { count(dapply(spark_df, function(rdf) { rdf }, schema(spark_df))) }\n",
    "boxplot(benchmark_arrow(func))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `gapply`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeAAAAHgCAMAAABKCk6nAAACylBMVEUAAAACAgIDAwMEBAQFBQUGBgYHBwcJCQkKCgoLCwsMDAwNDQ0ODg4PDw8QEBARERESEhITExMUFBQVFRUWFhYXFxcYGBgZGRkaGhobGxscHBwdHR0eHh4fHx8gICAhISEiIiIjIyMkJCQlJSUmJiYnJycoKCgpKSkqKiorKyssLCwtLS0uLi4vLy8wMDAxMTEyMjIzMzM0NDQ1NTU2NjY3Nzc4ODg5OTk6Ojo7Ozs8PDw9PT0+Pj4/Pz9AQEBBQUFCQkJDQ0NERERFRUVGRkZHR0dISEhJSUlKSkpLS0tMTExNTU1OTk5PT09QUFBRUVFSUlJTU1NUVFRVVVVWVlZXV1dYWFhZWVlaWlpbW1tcXFxdXV1eXl5fX19gYGBiYmJjY2NkZGRlZWVmZmZnZ2doaGhpaWlqampra2tsbGxtbW1ubm5wcHBxcXFycnJzc3N1dXV2dnZ3d3d4eHh5eXl6enp7e3t9fX1+fn5/f3+AgICBgYGCgoKDg4OEhISFhYWHh4eIiIiMjIyNjY2Ojo6Pj4+QkJCRkZGSkpKTk5OUlJSVlZWXl5eYmJiZmZmampqbm5ucnJydnZ2enp6fn5+hoaGioqKjo6OkpKSlpaWmpqanp6eoqKipqamqqqqrq6usrKytra2urq6vr6+wsLCxsbGysrK0tLS2tra4uLi5ubm7u7u8vLy+vr6/v7/AwMDBwcHCwsLDw8PExMTFxcXGxsbHx8fIyMjJycnKysrLy8vMzMzNzc3Ozs7Pz8/Q0NDR0dHS0tLT09PU1NTV1dXW1tbX19fY2NjZ2dna2trb29vc3Nzd3d3e3t7f39/g4ODh4eHi4uLj4+Pk5OTl5eXm5ubn5+fo6Ojq6urr6+vs7Ozt7e3u7u7v7+/w8PDx8fHy8vLz8/P09PT19fX29vb39/f4+Pj5+fn6+vr7+/v8/Pz9/f3+/v7///9Gu3DGAAAPjElEQVR4nO3d/0NV9R3H8dtKERDRigyVOU1MLdvURbVZCisTSathNWuuizUrV1r2xWa5pX1ZBE1b5ZqOsqXTbGXQShSVabXUzImlgjNBF1/l8z/scC+c7FyEz7nn3HPe98Xr8cMBzjme+4GncD8X7rknoAhawO8BUGwxMDgGBsfA4BgYHAODY2BwDAyOgcExMDgGBsfA4BgYHAODY2BwDAyOgcExMDgGBsfA4BgYHAODY2BwDAyOgcExMDgGBsfA4BgYHAODY2BwDAyOgcExMDgGBsfA4BgYHAODY2BwDAyOgcExMDgGBsfA4BgYHAODY2BwDAyOgcExMDgGBsfA4BgYHAODY2BwDAyOgcE5CFzzOgmwpilWgUt+8Ufy31WfxyxwUfT/llwTZGBsDAyOgcExMDgGBuc8cGkwa3RWsDRiPQOL4DhwcXKwcGVhQXKxdQMDi+A48MCy0JsP060bGFgEx4ETakJvjiVaNzCwCI4Dz8wurW2tLcu+xbqBgUVwHLgu2CcQCCQW1Fk3uBL4/aXdm/+wxk4VLgwmLrnwMKlpz/Y9p//J4l/h33LPut/RwMK+eKd78xZp7PSlC4OJSzF4HLw7/Heq3BuiG5FtL7zu0Q3FJZcCZxyJWHV/xL1yjBS94dENxSXHga8L6TXpOusGzwJXH/fohuKS84dJP3nKkPTYU9YNngWmrjgOvH9a3hdKpR2O2OBZ4E8PeHRDccmF++C3Riyu9zMwJ1ldcWOSdfKhzD4MLJQ7s+hPir+JWMdZtAix+3swZ9EiAASmrgAE5iy6KwCBOcnqCgOD8zHwTTnuGDveneNMmhv9JyuXj4Gvif7YMfH1DL9HEAsMbGJgexhYBAY2MbA9DCwCA5sY2B4GFoGBTQxsDwOLwMAmBraHgUVgYBMD28PAIvgY+EcLZbk/O/pPVi4fA48ICHN59J+sXD4GHpsvy4yro/9k5eJ9sIn3wfYwsAgMbGJgexhYBAY2MbA9DCwCA5sY2B4GFoGBTQxsT7eBJ/n9qyuLGfnRf7Jy+Rj4m2PCNET/ycoFcPLZ0VqPbiguAQTmSzh0BSAwTx/tCgODcyXwzsWLyiNWehb4k/0e3VBcchw4pUG923vqjF7rrBv4IiwiOA4cqFeTlyq1Ypx1A2fRIrgReNBnStVEXLOBs2gRnAde/srgSqWqUqwbOMkSwXHgXMMGpVZOtG5wJfB+t17S/ysXBhOX3HqYdPKE+W51RchtN0U/KtMmnYtyLNTYaYsLg4lLMXgc/H74eeTjI14DnnzgPPDGour63925KmI9HyaJ4Djw0xcMGzH/13clRFwliYFFcBx46EfVZ61WqmS0dQMDi+A4cK+v1blVSh1NsG5gYBEcB75wv3qkTql9A6wbGFgEx4HvCV8a+g/XWzcwsAhuPUyqO2ldw8AiAPw9mLrCwOAYGBwDg2NgcAwMjoHBMTA4BgbHwOAYGBwDg2NgcAwMjoHBMTA4BgbHwOAYGBwDg2NgcAwMjoHBMTA4BgbHwOAYGBwDg2NgcAwMjoHBMTA4BgbHwOAYGBwDg2NgcAwMjoHBMTA4BgbHwOAYGBwDg2NgcM4DlwazRmcFSyPWM7AIjgMXJwcLVxYWJBdbNzCwCI4DDywLvfkw3bqBgUVwHDihJvTmmH/XLqSuOA48M7u0trW2LDsiJwOL4DhwXbBPIBBILKizbmBgEVx4mNS0Z/uepsjVDCxCDB4Hr8kJGZId3YjIVbx2ITheuxAcr10IjtcuBMdrF4LjtQvB8dqF4Pj3YHAMDI6BwTEwOI3At5jm2DkyA4ugETj12Q4Zdo7MwCJoBL7W/CjPzpEZWATeB4PTC/xqaLnc1pEZWAS9wOE73362jszAIugE3rp14FbD6sG2jszAIugETk//Xnp6+uBxb9o6MgOLoPcjem4UR2ZgETiLBqcReJn50XN2jszAImgEHnSww0V2jszAImgE7m2KOMGsKwwsAu+DwTEwOAYGx8DgGBgcA4PTClz10rw75r1UZe/IDCyCRuDKvP5THnj8gSn983bYOTIDi6AROOuNhtD7DWuy7ByZgUXgfTA4zcDrZ9+oyjfaOjIDi6AX+JkRzw9VH19u68gMLIJe4CEHVKZq4XOy4pBe4HPrjcB1A20dmYFF0At880Mtma3z77B1ZAYWQS/woay+vdOurLZ1ZAYWQXMW3Vq5bkervSMzsAh8HAxOL/DJ52bfbLB1ZAYWQS/wtKueLjbYOjIDi6AXODHiNXS6x8Ai6AWeuMv+kRlYBL3AVVc+VmiwdWQGFkEv8Px+eW2v0WHryAwsguZ98H77R2ZgEfQCX3TU/pEZWAS9wEU/e2eXwdaRGVgEvcBpYbaOzMAi8FeV4DQCb61WW8NsHZmBRdAInPGqygizdWQGFoE/osHxhdDA8YXQwPGF0MDxhdDA8YXQwHEWDU4j8FMd134+sdTOkRlYBI3ATwy8fUXpjtIVtw9cbOfIDCyCzo/o40W5g5MG5xYdP9NeOxcvKo9YycAiOL4PTmlQ7/aeOqPXOusGBhbBceBAvZps3DevGGfdwMAi6AVeGPL48i872aNeDfpMqZpE6wYGFkEvcDBh+n3TE2bnJa6P3GP5K4MrlapKsW5gYBH0Aue+Zyzey1UbLonYI9ewQamVE60bGFgEvcDJLcaipa9qTT3TjidPmO+uyQkZMtmVAZIzeoHHLzulTi2boI7YeMVofgeLoBe4ctj5l50/fJcqWxG5y96ST41ly7PW9QwsgubDpObNb5W3dLrH2t7Dz7m3RdVHPKBiYBE0A29bcOuC7Z3ucemL6tA1+U0MLJRe4FVJwSXBpJLO9kioVqph2g3HGVgmvcCZpcaidGRnewytMBaN0ycxsEx6gVMajUVjp8/Jumde27J5JgPLpBd48uJm1fx4Tmd7NIb/WtwScf4hA4ugF3j3yNQxqRfvtXVkBhZBcxbdsmXtls4fJp0RA4ugc26SydaRGVgEnXOTTLaOzMAi8FmV4BgYHAODY2BwDAyOgcExMDgGBsfA4BgYHAODY2D3VVV07x/lGjvVujAYBnZfycLuZd2hsVMUr7MfgYH98fBHHt0QA/uDgcFtsnedwOgxMDgG9kdNs0c3xMD+eGSzRzfEwP7gJAscA4PjLJrcwcD+4CwaHGfR4DjJAsfA4DiLJncwsD84iwbHWTQ4TrLAMTA4zqLJHQzsD86iwXEWDY6TLHAMDI6zaHIHA/uDs2hwnEWD4yRLqJk57hie5c5xJhV2M2AGtucavwdgsXNeNzswsD0MbGJgLzCwyxjYxMBeYGCXMbCJgb3AwC5jYBMDe4GBXcbAJgb2AgO7jIFNDOwFBnYZA5sY2AsM7DIGNjGwFxjYZQxsYmAvMLDLGNjEwF5gYJcxsImBveBB4NJg1uisYGnEegb2QuwDFycHC1cWFiQXWzcwsBdiH3hgWejNh+nWDQzshdgHTqgJvTmWaN3AwF6IfeCZ2aW1rbVl2RE5GdgLsQ9cF+wTCAQSC+qsGxjYC148TGras31P02kfbwpfOnP8FI1/G3d6YmCrmvDFb5/8fRT/VrweGHhvyafGsuVZ6/qSou7/bfzpeYHX9h5+zr0tqj5iTwb2QuwDX/qiOnRNflNPCXyFxpXZvfTXud0M2Pnj4GqlGqbdcLyHBM4ICNPdYxXHgYdWGIvG6ZMY2B8xD3xP6E6geWYPCTzxmCwf3NfNgB0Hbgz/hqNlv3UDZuCeN8k6Iwb2AgO7jIFNDOwFBnYZA5sY2AsM7DIGNjGwFxjYZQxsYmAvMLDLGNjEwF5gYJcxsImBvcDALmNgEwN7gYFdxsAmBvYCA7uMgU2YgS/Nl+Xa+d0MmIHtOen3s+ysmroZMAP7Y80Bj26Igf0BcHFKBu4KA4NbVOHRDTGwP7y6ADgDo2Ngf3AWDY6TLHAMDI6zaHCcRZM7GNgfnEWD4yQLHAOD4ywaHGfR5A4G9gdn0eA4yQLHwHGsUOPprqNyNHba7MJgGNh99RrPdq3WeUpsiwuDYWBwDAyOgcExMDgGBsfA4BgYHAODY2BwDAyOgcHFLvDbY3O8Mbxv//iTOMGjr07mV7EK7JkXXvd7BFHw7M+F3WDgGGFgfQzsAAPHCAPrY2AHGDhGGFhf8Zt+jyAKj5b7PYKweAhc392r+Ul0stXvEYTFQ2BygIHBMTA4BgbHwOAYGBwDg2NgcHICj04+qtTqcZa1V/dr8GU0UZA5VEGB+z8YGXj/Wantv4huNhdCCR2qoMBLkg+HAu/N6TeypH3lE1c+kGu8SVv6w5HhRXjjy9OUGpav1KAdfo74u4QOVVDgN345ry1wy8WLGkuTtodXDiuqOPuI8VWb2tAaWrRv3Jdy6tCQC9W+fqf8HfPphA5VUuB9SVVG4C2pxk+34ILQurKza1TmM8ZXbaMKLzo2Dtq2es6Ef7+c5++QTyd1qJICq7sKjMBrxxgfLLk1tK5gqvGjb6zxBdupwouOjbOenvvagy/OXubjeC2kDlVU4KrE333nO/ibvolpaf0ClSptl/FR26Jj4/K8MQfX3/b9rT6P+VtihyoqsJrX37gPHrm46YPkbW2rVqUeOHz48E/nf/tV69i4O2mYqk1NduM1LtwhdqiyAh/uY8yid09OyVwVWnVd6MKMJWnN5lfN3HjBr5QaN8W/0VqJHaqcwBQTDAyOgcExMDgGBsfA4BgYHAODY2BwDAyOgcExMDgGBsfA4BgYHAODY2BwDAyOgcEx8HdkyHmepkvQA48OGHK1d3+1OoZj8QV84Jfr6zt/GSYpZ4fFGHzgP4fefJ66TX014D2V9mTWqLsb208RO5J/XsYzSv32gpQR5e1L40d0+7lvaUsnDJ8N8J+ghwRWy0f+79oFRrbrm5unLgmfItZ6xYL6/4xYX5FRrb44GF4agTvOfUu7qanpx3/xd/BugA+cNGDAgCeNd/LGXNJghN2g1NujwmeHVfZtUer52ZXnvWN8S4eXRuCOM07SNin16AK/h+8cfODnDx48WGu88/fAcmOZts1o2S98dti63pmZmUNvVH8a33dWdfsyY2vHOWNtJycsudvn0bsAPnD7j+gTPwhe+F8j7Bql/jYqfG7J9rSOl5Osmfab9uVp38EMHBdCs2jjh++d+WpOvhE2q/ro+CfCgU9lPXii5ZPyj//Z3HjrgvCy7T64/ZwxBo4PocfBY9Va49v3xLDXVNpzw/vPaWg/O+zIrLR+l2/cfFnyuT8/Fl62zaLbzxlj4LgUCtujMDA4BgbXwwL3PAwMjoHBMTA4BgbHwOAYGBwDg2NgcAwMjoHBMTA4BgbHwOAYGBwDg/s/Nimq4eMc8aIAAAAASUVORK5CYII=\n"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%R\n",
    "func <- function() {\n",
    "  count(gapply(spark_df,\n",
    "               \"Month_of_Joining\",\n",
    "               function(key, group) { group }, schema(spark_df)))\n",
    "}\n",
    "boxplot(benchmark_arrow(func))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}