examples/job/get-file-from-job.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "_eLv7xE68GIo"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/Exabyte-io/api-examples/blob/dev/examples/job/get-file-from-job.ipynb\" target=\"_parent\">\n",
    "<img alt=\"Open in Google Colab\" src=\"https://user-images.githubusercontent.com/20477508/128780728-491fea90-9b23-495f-a091-11681150db37.jpeg\" width=\"150\" border=\"0\">\n",
    "</a>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "q1kKqnws8GIq"
   },
   "source": [
    "# Get-File-From-Job\n",
    "\n",
    "This example demonstrates how to use Mat3ra RESTful API to check for and acquire files from jobs which have been run. This example assumes that the user is already familiar with the [creation and submission of jobs](create_and_submit_jobs.ipynb) using our API.\n",
    "\n",
    "> <span style=\"color: orange\">**IMPORTANT NOTE**</span>: In order to run this example in full, an active Mat3ra.com account is required. Alternatively, Readers may substitute the workflow ID below with another one (an equivalent one for VASP, for example) and adjust extraction of the results (\"Viewing job files\" section). RESTful API credentials shall be updated in [settings](../../utils/settings.json).\n",
    "\n",
    "\n",
    "## Steps\n",
    "\n",
    "After working through this notebook, you will be able to:\n",
    "\n",
    "1. Import [the structure of Si](https://materialsproject.org/materials/mp-149/) from Materials Project\n",
    "2. Set up and run a single-point calculation using Quantum Espresso.\n",
    "3. List files currently in the job's directory\n",
    "4. Check metadata for every file (modification date, size, etc)\n",
    "5. Access file contents directly and print them to console\n",
    "6. Download files to your local machine\n",
    "\n",
    "## Pre-requisites\n",
    "\n",
    "The explanation below assumes that the reader is familiar with the concepts used in Mat3ra platform and RESTful API. We outline these below and direct the reader to the original sources of information:\n",
    "\n",
    "- [Generating RESTful API authentication parameters](../system/get_authentication_params.ipynb)\n",
    "- [Importing materials from materials project](../material/import_materials_from_materialsproject.ipynb)\n",
    "- [Creating and submitting jobs](../job/create_and_submit_job.ipynb)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "p1oEmy_N8GIr"
   },
   "source": [
    "# Complete Authorization Form and Initialize Settings\n",
    "\n",
    "This will also determine environment and set all environment variables. We determine if we are using Jupyter Notebooks or Google Colab to run this tutorial.\n",
    "\n",
    "If you are running this notebook from Google Colab, Colab takes ~1 min to execute the following cell.\n",
    "\n",
    "ACCOUNT_ID and AUTH_TOKEN - Authentication parameters needed for when making requests to [Mat3ra.com's API Endpoints](https://docs.mat3ra.com/rest-api/endpoints/).\n",
    "\n",
    "MATERIALS_PROJECT_API_KEY - Authentication parameter needed for when making requests to [Material Project's API](https://materialsproject.org/open)\n",
    "\n",
    "ORGANIZATION_ID - Authentication parameter needed for when working with collaborative accounts https://docs.mat3ra.com/collaboration/organizations/overview/\n",
    "\n",
    "> <span style=\"color: orange\">**NOTE**</span>: If you are running this notebook from Jupyter, the variables ACCOUNT_ID, AUTH_TOKEN, MATERIALS_PROJECT_API_KEY, and ORGANIZATION_ID should be set in the file [settings.json](../../utils/settings.json) if you need to use these variables. To obtain API token parameters, please see the following link to the documentation explaining how to get them: https://docs.mat3ra.com/accounts/ui/preferences/api/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "nXDYU_oP8GIr",
    "outputId": "aee909b7-1a48-43c2-81b7-3659e0fc9c79"
   },
   "outputs": [],
   "source": [
    "# @title Authorization Form\n",
    "ACCOUNT_ID = \"ACCOUNT_ID\"  # @param {type:\"string\"}\n",
    "AUTH_TOKEN = \"AUTH_TOKEN\"  # @param {type:\"string\"}\n",
    "MATERIALS_PROJECT_API_KEY = \"MATERIALS_PROJECT_API_KEY\"  # @param {type:\"string\"}\n",
    "ORGANIZATION_ID = \"ORGANIZATION_ID\"  # @param {type:\"string\"}\n",
    "\n",
    "import os\n",
    "import sys\n",
    "\n",
    "if \"COLAB_JUPYTER_IP\" in os.environ:\n",
    "    os.environ.update(\n",
    "        dict(\n",
    "            ACCOUNT_ID=ACCOUNT_ID,\n",
    "            AUTH_TOKEN=AUTH_TOKEN,\n",
    "            MATERIALS_PROJECT_API_KEY=MATERIALS_PROJECT_API_KEY,\n",
    "            ORGANIZATION_ID=ORGANIZATION_ID,\n",
    "        )\n",
    "    )\n",
    "\n",
    "    !GIT_BRANCH=\"dev\"; export GIT_BRANCH; curl -s \"https://raw.githubusercontent.com/Exabyte-io/api-examples/${GIT_BRANCH}/scripts/env.sh\" | bash\n",
    "\n",
    "if sys.platform == \"emscripten\":\n",
    "    apiConfig = data_from_host.get(\"apiConfig\")\n",
    "    os.environ.update(data_from_host.get(\"environ\", {}))\n",
    "    os.environ.update(\n",
    "        dict(\n",
    "            ACCOUNT_ID=apiConfig.get(\"accountId\"),\n",
    "            AUTH_TOKEN=apiConfig.get(\"authToken\"),\n",
    "            MATERIALS_PROJECT_API_KEY=MATERIALS_PROJECT_API_KEY,\n",
    "            ORGANIZATION_ID=apiConfig.get(\"organizationId\") or \"\",\n",
    "        )\n",
    "    )\n",
    "\n",
    "    import micropip\n",
    "\n",
    "    await micropip.install(\"mat3ra-api-examples\", deps=False)\n",
    "    await micropip.install(\"exabyte-api-client\")\n",
    "    from utils.jupyterlite import install_packages\n",
    "\n",
    "    await install_packages(\"\", \"../../config.yml\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "6Z9Cr0_M8GIs"
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "rG9HfjnR8GIs"
   },
   "outputs": [],
   "source": [
    "# Import settings file and utils file\n",
    "from utils.settings import ENDPOINT_ARGS, ACCOUNT_ID, MATERIALS_PROJECT_API_KEY\n",
    "from utils.generic import (\n",
    "    wait_for_jobs_to_finish,\n",
    "    get_property_by_subworkflow_and_unit_indicies,\n",
    "    dataframe_to_html,\n",
    "    display_JSON,\n",
    ")\n",
    "\n",
    "# Relevant functions from the API client\n",
    "from exabyte_api_client.endpoints.jobs import JobEndpoints\n",
    "from exabyte_api_client.endpoints.projects import ProjectEndpoints\n",
    "from exabyte_api_client.endpoints.materials import MaterialEndpoints\n",
    "from exabyte_api_client.endpoints.bank_workflows import BankWorkflowEndpoints\n",
    "from exabyte_api_client.endpoints.properties import PropertiesEndpoints"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "_zKrDX3q8GIt"
   },
   "source": [
    "### Create and submit the job\n",
    "\n",
    "For this job, we'll use the workflow located [here](https://platform.mat3ra.com/analytics/workflows/84DAjE9YyTFndx6z3).\n",
    "\n",
    "This workflow is a single-point total energy calculation using Density-Functional Energy as-implemented in Quantum Espresso version 5.4.0.\n",
    "\n",
    "The PBE functional is used in conjunction with an ultrasoft pseudopotential and a planewave basis set.\n",
    "\n",
    "The material we will investigate is elemental [Silicon](https://materialsproject.org/materials/mp-149/), as-is from Materials Project.\n",
    "\n",
    "> <span style=\"color: orange\">Note</span>: This cell uses our API to copy the unit cell of silicon from Materials Project into your account. It then copies a workflow to get the total energy of a system using Quantum Espresso to your account. Finally, a job is created using the Quantum Espresso workflow for the silicon unit cell, and the job is submitted to the cluster. For more information, please refer to our [run-simulation-and-extract-properties](./run-simulations-and-extract-properties.ipynb) notebook, located in this directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "le9f28c08GIt",
    "outputId": "27e9b94f-29dd-4742-9448-2e1bce79cb15"
   },
   "outputs": [],
   "source": [
    "# Get some account information\n",
    "project_endpoints = ProjectEndpoints(*ENDPOINT_ARGS)\n",
    "project_metadata = project_endpoints.list({\"isDefault\": True, \"owner._id\": ACCOUNT_ID})[0]\n",
    "project_id = project_metadata[\"_id\"]\n",
    "owner_id = project_metadata[\"owner\"][\"_id\"]\n",
    "\n",
    "# Get a workflow for the job from the bank, and copy it to our account\n",
    "bank_workflow_endpoints = BankWorkflowEndpoints(*ENDPOINT_ARGS)\n",
    "BANK_WORKFLOW_ID = \"84DAjE9YyTFndx6z3\"\n",
    "workflow_id = bank_workflow_endpoints.copy(BANK_WORKFLOW_ID, owner_id)[\"_id\"]\n",
    "\n",
    "# Get materials for the job\n",
    "material_endpoints = MaterialEndpoints(*ENDPOINT_ARGS)\n",
    "material_project_id = [\"mp-149\"]  # The importer expects a list\n",
    "materials = material_endpoints.import_from_materialsproject(MATERIALS_PROJECT_API_KEY, material_project_id, owner_id)\n",
    "\n",
    "# Create the job\n",
    "job_endpoints = JobEndpoints(*ENDPOINT_ARGS)\n",
    "job = job_endpoints.create_by_ids(\n",
    "    materials=materials, workflow_id=workflow_id, project_id=project_id, owner_id=owner_id, prefix=\"Test_Job_Output\"\n",
    ")[0]\n",
    "\n",
    "# Submit the job\n",
    "job_endpoints.submit(job[\"_id\"])\n",
    "wait_for_jobs_to_finish(job_endpoints, [job[\"_id\"]])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "kiCDWnVP8GIu"
   },
   "source": [
    "Monitor the jobs and print the status until they are all finished."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "XygP7P1m8GIv"
   },
   "source": [
    "## Viewing job files\n",
    "### Retreive a list of job files\n",
    "\n",
    "Here, we'll get a list of all files that belong to the job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "yijRe7qp8GIv",
    "outputId": "19b147fc-736e-41f0-bd0f-6f69e77b0adf"
   },
   "outputs": [],
   "source": [
    "files = job_endpoints.list_files(job[\"_id\"])\n",
    "paths = [file[\"key\"] for file in files]\n",
    "for path in paths:\n",
    "    if \"outdir\" not in path:\n",
    "        print(path)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "exnQTPwF8GIv"
   },
   "source": [
    "### Get metadata for the Output File\n",
    "The .out file is where Quantum Espresso shows its work and prints its results, so you most likely will want to view this files. Let's print out some of its metadata.\n",
    "\n",
    "You'll find that we get a lot of data describing the file and its providence. Brief explanations of each entry are:\n",
    "- Key - Path to the file on the cluster\n",
    "- size - Size of the file, in bytes.\n",
    "- Bucket - The name of the cluster which ran the job.\n",
    "- Region - Which server region was used to run the job.\n",
    "- Provider - The cluster provider for the compute resources (in our case, we used AWS).\n",
    "- lastModified - Unix timestamp representing when the file was last modified.\n",
    "- name - The filename.\n",
    "- signedUrl - This is a link which can be used to download the file for a short amount of time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "U2dq0A1e8GIw",
    "outputId": "7b4f8c36-6342-46fa-d02e-0825b44e6c64"
   },
   "outputs": [],
   "source": [
    "for file in files:\n",
    "    if file[\"name\"] == \"pw_scf.out\":\n",
    "        output_file_metadata = file\n",
    "display_JSON(output_file_metadata)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "PwC7wQZR8GIw"
   },
   "source": [
    "### Display file contents to console\n",
    "\n",
    "The signedUrl gives us a place to access the file and download it. Let's read it into memory, and print out the last few lines of our job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "OpdOMuDT8GIw",
    "outputId": "e4e2e6b1-8431-439a-8606-ae500534e601"
   },
   "outputs": [],
   "source": [
    "import urllib\n",
    "\n",
    "server_response = urllib.request.urlopen(output_file_metadata[\"signedUrl\"])\n",
    "output_file_bytes = server_response.read()\n",
    "\n",
    "# The server returns us a bytes-string. That's useful for things like binaries or other non-human-readable data, but this should be decoded if we're planning to write to console.\n",
    "# Because this is a human-readable text file, we'll decode it to UTF-8.\n",
    "output_file = output_file_bytes.decode(encoding=\"UTF-8\")\n",
    "\n",
    "# Tail the last 90 lines\n",
    "lines = output_file.split(\"\\n\")\n",
    "for line in lines[-90:]:\n",
    "    print(line)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "-z7VqM3K8GIx"
   },
   "source": [
    "### Save the input file and output file to disk.\n",
    "\n",
    "Now that we've verified the job is done, let's go ahead and save it and its input to disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "osy2JKpB8GIx"
   },
   "outputs": [],
   "source": [
    "# We've already got an output file, so let's grab the input file we sent to Quantum Espresso\n",
    "for file in files:\n",
    "    if \"pw_scf.in\" == file[\"name\"]:\n",
    "        input_file_metadata = file\n",
    "server_response = urllib.request.urlopen(input_file_metadata[\"signedUrl\"])\n",
    "input_file_bytes = server_response.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "COPinBrO8GIx"
   },
   "outputs": [],
   "source": [
    "# Let's write the input file to disk. Note that we get files as a bytes string from the server, which is convenient for binaries, images, and other non-human-readable data.\n",
    "# Although we could decode before writing to disk, we can just write it directly with the \"wb\" (write bytes) file mode.\n",
    "with open(input_file_metadata[\"name\"], \"wb\") as file_descriptor:\n",
    "    file_descriptor.write(input_file_bytes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "rzDcWyrL8GIx"
   },
   "outputs": [],
   "source": [
    "# Now, let's write our output file to the disk. Note that because we already decoded it, we can just use the 'w' file mode.\n",
    "with open(output_file_metadata[\"name\"], \"w\") as file_descriptor:\n",
    "    file_descriptor.write(output_file)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "get-file-from-job.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}