JSON_Mem_Issues.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `JSON Data Ingestion Issues with Memory & Time`\n",
    "\n",
    "# <font color=red>Mr Fugu Data Science</font>\n",
    "\n",
    "# (◕‿◕✿)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `In General Considerations for parsing JSON or Data Manipulation:`\n",
    " + 1. ) reconsider doing loops with large data\n",
    " + 2. ) Json_normalize gets slow with big datasets\n",
    " + 3. ) *`Rethink`* **'Apply'** function for speed\n",
    "     + this is because you are not using vectorization but instead looping\n",
    "     + you are returning a DF or Series which inherently uses quite a bit of resources. \n",
    "\n",
    "https://shubhanshugupta.com/speed-up-apply-function-pandas-dataframe/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `Pandas Bottle Neck?`\n",
    "\n",
    "Consider this, when you use `Pandas`. Have you ever noticed that your data may:\n",
    "\n",
    "1. ) drastically eat up your memory and you may incur `an error` and suddenly crash \n",
    "\n",
    "2. ) data manipulations/computations `create copies` which are slowing you down\n",
    "\n",
    "3. ) `NOT parrallel computing` which can hinder you depending on task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import json\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# example: apply() function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# using np.vectorize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dask and Swifter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `CudDF`\n",
    "\n",
    "\n",
    "\n",
    "https://rapids.ai/\n",
    "\n",
    "https://github.com/rapidsai/cudf\n",
    "\n",
    "http://googlielmo.blogspot.com/2019/11/using-rapids-cudf-in-colab-notebook.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `cysimdjson`\n",
    "\n",
    "\n",
    "https://github.com/TeskaLabs/cysimdjson"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `SIMDJSON`\n",
    "\n",
    "https://simdjson.org/software/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `Glom:`\n",
    "https://enrico-alemani.medium.com/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`------------------------------------------`\n",
    "\n",
    "# <font color=red>Next Video: Speed up Pandas:</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `Swifter vs Pandas apply():`\n",
    "\n",
    "https://medium.com/@jmcarpenter2/swiftapply-automatically-efficient-pandas-apply-operations-50e1058909f9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "83.44080000000001"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "77.26+77.26*.08"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `Speed up Pandas:`\n",
    "\n",
    "https://alievmagomed.com/6-ways-to-significantly-speed-up-pandas-with-a-couple-lines-of-code/\n",
    "\n",
    "https://pandas.pydata.org/pandas-docs/stable/user_guide/enhancingperf.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `Citations:`\n",
    "\n",
    "https://ram-parameswaran22.medium.com/a-relatively-faster-approach-for-reading-json-lines-file-into-pandas-dataframe-90b57353fd38\n",
    "\n",
    "https://shubhanshugupta.com/speed-up-apply-function-pandas-dataframe/\n",
    "\n",
    "https://pythonspeed.com/articles/faster-json-library/\n",
    "\n",
    "https://pythonrepo.com/repo/TeskaLabs-cysimdjson-python-josn\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}