Agave Tutorial.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:f6f47f424cc731a7c208002f044ebf82a326e86b7395e1a2c702160e27a769de"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#**<font color=\"#1569C7\">Agave App Development Student Guide</font>**"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###**_<font color=\"#2C3539\">A Simple Tutorial To Make App Development Easier</font>_**"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "####**<font color=\"maroon\">Before you begin ensure the following criteria are met:</font>**\n",
      "1. Do you have an iPlant account and do you remember your iPlant username and password? If you do not have an account yet register for one here: https://user.iplantcollaborative.org/register/\n",
      "\n",
      "2. Do you have a TACC account? If not register for one here: https://portal.tacc.utexas.edu/web/tup/account-request\n",
      "\n",
      "3. Do you have an XSEDE account? If not register for one here: https://portal.xsede.org/\n",
      "4. Have you contacted iPlant support and requested to be added to the iPlant-Collabs project and provided SSH access to Stampede? If not then send a message to http://agaveapi.co/contact/ explictly requesting Stampede SSH access and membership in the iPlant-Collabs project. You need to already have an iPlant, TACC, and XSEDE account to receive access."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###**_<font color=\"#1569C7\">Ready?</font>_** <img src=\"http://troll.me/images/x-all-the-things/lets-do-it.jpg\" alt=\"Lets do it!\" width=\"300\" >"
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "1. Installing the Agave SDK to your Stampede instance:"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "First you need to SSH into your Stampede instance. The command is as follows except be sure to replace <font color=\" blue\">username</font> with your iPlant username. On Linux and OSX systems SSH is as easy as entering the following command in a terminal instance. For Windows you will require either an SSH client such as PUTTY http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html or install Cygwin https://www.cygwin.com/; a UNIX command-line emulator that will allow you to use Linux (or UNIX) style commands in Windows, including SSH. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ssh username@stampede.tacc.utexas.edu"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now that you are in your Stampede home directory you will need to actually install the Agave SDK to your Stampede instance. Don't worry if you don't understand all of the following commands, you don't need to change any variables, just execute each command. Remember, anything preceded by a # is just a comment, don't run those!  "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Check out the SDK from the iPlant's GitHub\n",
      "# Load an updated git module by typing:\n",
      "module load git\n",
      "# Clone the SDK repository (note the --recursive flag):\n",
      "git clone https://github.com/iPlantCollaborativeOpenSource/iplant-agave-sdk.git --recursive\n",
      "# Add SDK scripts to your PATH:\n",
      "echo \"export IPLANT_SDK_HOME=$PWD/iplant-agave-sdk\" >> ~/.profile\n",
      "echo \"PATH=\\$PATH:\\$IPLANT_SDK_HOME/scripts:\" >> ~/.profile\n",
      "echo \"export PATH=\\$PATH:\\$IPLANT_SDK_HOME/foundation-cli/bin\" >> ~/.profile\n",
      "# To re-init bash _type_:\n",
      "source ~/.profileNow "
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Thats it! You should now have the Agave SDK installed to your Stampede instance. Lets continue..."
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "2. Creating an OAuth2 client and getting a set of keys:"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The Agave API uses OAuth2 to manage authentication and authorization. This means that you will need to set up an OAuth client application associated with a set of API keys. Fortunately you only have to do this once. To do so simply run the following command. If you like you may change <font color=\"blue\">my_client</font> to a client name of your choosing and <font color=\"red\">\"Client used for app development\"</font> to a description of your choosing. This is not necessary however, and will work just fine as is. You will be prompted for your iPlant username and password. Enter them when prompted.\n",
      "\n",
      "Note: The -N flag allows you to specify a machine-readable name for your application; -D provides the description, and -S option stores your API keys for future use, so you will not need to manually enter them when you authenticate later."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "clients-create -S -v -N my_client -D \"Client used for app development\""
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You should receive output which looks similar to the following:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "{\n",
      "    \"_links\": {\n",
      "        \"self\": {\n",
      "            \"href\": \"https://agave.iplantc.org/clients/v2/my_client\"\n",
      "        },\n",
      "        \"subscriber\": {\n",
      "            \"href\": \"https://agave.iplantc.org/profiles/v2/vaughn\"\n",
      "        },\n",
      "        \"subscriptions\": {\n",
      "            \"href\": \"https://agave.iplantc.org/clients/v2/my_client/subscriptions/\"\n",
      "        }\n",
      "    },\n",
      "    \"callbackUrl\": \"\",\n",
      "    \"consumerKey\": \"fMYfhijQE8ajqcKaswlGH4D5sngh\",\n",
      "    \"consumerSecret\": \"8FZ6K9QwN1PY6bA9SOcyi4oaEkMa\",\n",
      "    \"description\": \"Client used for app development\",\n",
      "    \"name\": \"my_client\",\n",
      "    \"tier\": \"Unlimited\"\n",
      "}"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You are going to want to hang on to your consumerKey and consumerSecret for later use in interacting with the Agave API. A handy way to remember them is to assign them to a variable in command line, for example:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "key=fMYfhijQE8ajqcKaswlGH4D5sngh\n",
      "secret=8FZ6K9QwN1PY6bA9SOcyi4oaEkMa"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now to access the key and secret respectively run the following commands:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "echo $key\n",
      "echo $secret"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The output should look like this for echo $key:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "fMYfhijQE8ajqcKaswlGH4D5sngh"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "and like this for echo $secret:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "8FZ6K9QwN1PY6bA9SOcyi4oaEkMa"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Of course your key and secret values will be different, but you get the idea, pretty handy! If you ever forget your API keys you can always go to https://agave.iplantc.org/store/site/pages/subscriptions.jag , click subscriptions, log in with your iPlant credentials, and select your client from the drop down menu (if you used the default settings this will be <font color=\"blue\">my_client</font>). Your consumer key and secret should be displayed. Now we need to get an authorization token that allows your new client app to interact with the Agave API. "
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "3. Obtaining an OAuth 2 authentication token:"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Obtaining a OAuth 2 authentication token is easy! Simply run the following command in your stampede instance:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "auth-tokens-create -S -v"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You will need to enter your consumer secret, consumer key, iPlant username, and iPlant password in that order. Don't worry, the system will prompt you for each one, and actually show you exactly what you need to enter, so you can just copy it and paste in the terminal window. For example the prompt for the secret will look like:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "API secret [ZwfpGFiMDRg4ypHvSwZs3IdoZdIa]:"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "So all you have to do is copy that secret key contained in the bracket and paste it into the terminal and hit enter. \n",
      "\n",
      "<font><i>Hint: copy-paste shortcuts are different in terminal, instead of the usual ctrl-c to copy and ctrl-v to paste, you instead do ctrl-shift-c to copy and ctrl-shift-v to paste.</i></font>\n",
      "\n",
      "Once your have enterered all prompted information you should receive an output similar to the following:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "Token successfully refreshed and cached for 14400 seconds\n",
      "{\n",
      "    \"access_token\": \"e431846eb2c917a4c5796eb1cc2c6f\",\n",
      "    \"expires_in\": 14400,\n",
      "    \"refresh_token\": \"8212a515b26ebc1aec5d6e232bb455b\",\n",
      "    \"token_type\": \"bearer\"\n",
      "}"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Thats it! Nothing to remember here. If your token at some point in time expires, simply re-run the auth-tokens-create -S command. You'll only need to enter your iPlant password, as the other values will be automatically remembered."
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "4. Setting up iPlant development systems:"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "A \"system\" under Agave 2.0 is a combination of login credentials, information about available physical resources, policy descriptions, and configuration options that combine to create an abstraction of a physical computing system. This frees end users from needing to know specific details about the actual computing environment in order to move data to the system and run computing tasks on it.\n",
      "\n",
      "We will now set up private versions of all public systems that iPlant uses to power Discovery Environment applications. You only need to do this once, as the systems created here will be forever associated with your iPlant credentials (until you delete them from service).\n",
      "\n",
      "You are going to need the full path to your WORK directory on TACC systems. To find this out run the following command:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ssh stampede.tacc.utexas.edu 'echo $WORK'"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You are also going to need your TACC username and password (which may be different than your iPlant username and password). Once you have all this information ready, run the following command:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "iplant-systems-create"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You will be prompted for a series of information including iPlant username and password, TACC username and password, consumer secret and consumer key. You will also be prompted for a TACC project associated with this system, the project name you need to enter is <font color=\"blue\">iPlant-Collabs</font>. When prompted \"Do you have all the information required?\" you MUST enter Yes with a capital Y, it is case sensitive and will fail if a lowercase y is used. Strange but necessary. An example of what the process and expected output should look like is shown below:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "We are going to create and enroll personal copies of various HPC systems\n",
      "used by iPlant staff developers to build apps.\n",
      "The following assumes you have created an Agave client via client-create.\n",
      "We will now ask you to refresh your access token...\n",
      "\n",
      "API secret [6bA9SOcyi4o8FZ6K9QwN1PYaEkMa]: \n",
      "API key [jqcKaswlGH4D5snghfMYfhijQE8a]: \n",
      "API username [vaughn]: \n",
      "API password: \n",
      "Token successfully refreshed and cached for 13150 seconds\n",
      "ef5f619c98ca45c56d4bc2a29dbc723b\n",
      "\n",
      "We will now configuring TACC-managed systems and need some info from you\n",
      "    TACC username\n",
      "    TACC password\n",
      "    TACC Project name\n",
      "    The path to your $WORK directory on TACC systems\n",
      "\n",
      "Do you have all the information required? [Yes]: \n",
      "OK. Let's begin.\n",
      "Enter your TACC user account []: vaughn\n",
      "Confirmed: TACC user account is vaughn\n",
      "Enter your TACC account password []: \\n\n",
      "Enter a TACC project associated with this system [iPlant-Collabs]: \n",
      "Confirmed: TACC project is iPlant-Collabs\n",
      "Enter your TACC work directory []: /work/01374/vaughn\n",
      "Confirmed: TACC work directory is \n",
      "tacc-lonestar4-template\n",
      "Enrolling private system tacc-lonestar4-template\n",
      "Successfully added system lonestar4-04012014-1718-vaughn\n",
      "tacc-maverick-template\n",
      "Enrolling private system tacc-maverick-template\n",
      "Successfully added system maverick-04012014-1718-vaughn\n",
      "tacc-stampede-template\n",
      "Enrolling private system tacc-stampede-template\n",
      "Successfully added system stampede-04012014-1718-vaughn\n",
      "\n",
      "Here is a listing of your systems:\n",
      "lonestar4.tacc.teragrid.org\n",
      "lonestar4-04012014-1718-vaughn\n",
      "stampede.tacc.utexas.edu\n",
      "stampede-04012014-1718-vaughn\n",
      "maverick-04012014-1718-vaughn\n",
      "data.iplantcollaborative.org"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now you can run applications you develop on the systems to which you have been added! Awesome! In your current directory are some JSON files associated with the systems on which you have been enrolled. Unfortunately each of these includes your password in plain text!!! Not good! To fix that set your permissions so only you can read them by running the following command:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "chmod 600 *.json"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You are now ready to start developing an iPlant application to run on the Stampede Supercomputer!"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}