2.running-hive.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Initial definitions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up Hadoop envvars"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%env HADOOP_VERSION     2.9.2\n",
    "%env HADOOP_HOME hadoop-2.9.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating HDFS directories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!./$HADOOP_HOME/bin/hadoop fs -mkdir       /tmp\n",
    "!./$HADOOP_HOME/bin/hadoop fs -mkdir       /user/hive/\n",
    "!./$HADOOP_HOME/bin/hadoop fs -mkdir       /user/hive/warehouse\n",
    "!./$HADOOP_HOME/bin/hadoop fs -chmod g+w   /tmp\n",
    "!./$HADOOP_HOME/bin/hadoop fs -chmod g+w   /user/hive/warehouse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%env HIVE_VERSION     hive-2.3.5\n",
    "%env HIVE_HOME apache-hive-2.3.5-bin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preparing the Enviroment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Downloading Hive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget http://ftp.unicamp.br/pub/apache/hive/${HIVE_VERSION}/${HIVE_HOME}.tar.gz -q --show-progress"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extracting compressed files and removing .tar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tar -xvf ${HIVE_HOME}.tar.gz >/dev/null \n",
    "!rm       ${HIVE_HOME}.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Java Home envvar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%env JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64\n",
    "# !echo \"export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \" >> ~/.bashrc\n",
    "# !echo \"export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \" >> ~/.profile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting the Hive path envvar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!echo \"export HIVE_HOME=$(pwd)/$HIVE_HOME\" >> ~/.bashrc\n",
    "!echo \"export HIVE_HOME=$(pwd)/$HIVE_HOME\" >> ~/.profile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuring hive-env.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!echo \"export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64\" >> $(pwd)/${HIVE_HOME}/bin/hive-env.sh\n",
    "!echo \"export HADOOP_HOME=$(pwd)/$HADOOP_HOME\"             >> $(pwd)/${HIVE_HOME}/bin/hive-env.sh\n",
    "!echo \"export HIVE_HOME=$(pwd)/$HIVE_HOME\"                 >> $(pwd)/${HIVE_HOME}/bin/hive-env.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Running Hive CLI from a terminal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use the Hive command line interface (CLI) from the shell:\n",
    "\n",
    "```bash\n",
    "$HIVE_HOME/bin/hive\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!./$HIVE_HOME/bin/hive --debug --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Running HiveServer2 and Beeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating database to store the metadata\n",
    "\n",
    "Starting from Hive 2.1, we need to run the schematool command below as an initialization step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!./$HIVE_HOME/bin/schematool -dbType derby -initSchema"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running HiveServer2 and Beeline  from a terminal:\n",
    "\n",
    "```bash\n",
    "./$HIVE_HOME/bin/hiveserver2\n",
    "\n",
    "./$HIVE_HOME/bin/beeline -u jdbc:hive2://localhost:10000\n",
    "```\n",
    "\n",
    "Or to start Beeline and HiveServer2 in the same process for testing purpose, for a similar user experience to HiveCLI:\n",
    "\n",
    "```bash\n",
    "$HIVE_HOME/bin/beeline -u jdbc:hive2://\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!./$HIVE_HOME/bin/beeline --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running the HCatalog server from a terminal:\n",
    "\n",
    "```bash\n",
    "$HIVE_HOME/hcatalog/sbin/hcat_server.sh\n",
    "```\n",
    "\n",
    "## Using the HCatalog command line interface (CLI) from a terminal:\n",
    "```bash\n",
    "$HIVE_HOME/hcatalog/bin/hcat\n",
    "```\n",
    "\n",
    "To run the WebHCat server from the shell in Hive release 0.11.0 and later:\n",
    "```bash\n",
    "$HIVE_HOME/hcatalog/sbin/webhcat_server.sh\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!./$HIVE_HOME/hcatalog/sbin/hcat_server.sh start && $HIVE_HOME/hcatalog/sbin/webhcat_server.sh start"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc-autonumbering": true,
  "toc-showtags": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}