5_downloading_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0",
   "metadata": {},
   "source": [
    "# Downloading Data\n",
    "\n",
    "## Setting up and using Globus\n",
    "\n",
    "We've seen how to search and download the ASDF metadata files with Fido.\n",
    "However, the actual data files are distributed using [Globus](https://www.globus.org/data-transfer)\n",
    "For the next portion of the workshop you will need to be running Globus Connect Personal, so follow the installation instructions for your platform [here](https://www.globus.org/globus-connect-personal) if you haven't already.\n",
    "During the setup, you will need to login to Globus.\n",
    "For this you can use your login for your institution, or alternatively you can login with Google or ORCID.\n",
    "\n",
    "Once Globus is installed and set up, you will need to run Globus Connect Personal (GCP) as described on the installation page.\n",
    "You will need to do this every time you want to download data, either through the user tools or through the Globus web app.\n",
    "When you start GCP You may also want to define the location or locations on your computer which you want Globus to have access to.\n",
    "On Linux you can do this using the `-restrict-paths` command line argument, or by editing the config file.\n",
    "On Windows and Mac OS this option is in the \"Access\" tab of the configuration options.\n",
    "Globus will only be able to transfer files onto your machine in the specified paths.\n",
    "\n",
    "### The Globus web app\n",
    "\n",
    "Many of you will already be familiar with using the [Globus web app](https://app.globus.org/) to download data.\n",
    "If you are not, you should read through the [getting started docs here](https://docs.globus.org/how-to/get-started/).\n",
    "We will not be using the web app significantly for this workshop, and generally we don't recommend downloading data this way, since the user tools are better suited to navigating the quantities of data that DKIST provides.\n",
    "However, we will be going over how to use the web app now so that we can demonstrate some of the underlying concepts.\n",
    "\n",
    "**Endpoints** (also called **Collections** in the web app) are locations registered with Globus for data transfer.\n",
    "For example, you may want to define an endpoint for both your desktop machine in the office and your laptop, so that you can download data on each depending on where you're working.\n",
    "You would then be able to transfer data directly from one to the other using Globus.\n",
    "Many institutions will have their own Globus endpoints, such as a computing cluster, that you may have access to.\n",
    "DKIST has an endpoint called \"DKIST Data Transfer\", which is where DKIST data will be made available.\n",
    "\n",
    "**Paths** When working with paths in Globus it's worth remembering that the paths are as the Globus endpoint sees them, so might not be identical to how you refer to them on your local system.\n",
    "\n",
    "To start a data transfer from one endpoint to another, go to the \"File Manager\" tab of the web app.\n",
    "Here you will find a split screen - on either side you can select an endpoint in the \"Collection\" search box.\n",
    "(If you do not see the split screen, find and click the \"set two pane\" button in the top right hand corner of the page.)\n",
    "Select \"DKIST Data Transfer\" on the left hand side and the endpoint corresponding to your local machine on the right.\n",
    "Then you can navigate the file system on either machine (remembering that Globus will only have access to whichever local directories you've specified).\n",
    "\n",
    "Let's demonstrate a simple file transfer by grabbing the preview movie for a dataset.\n",
    "On the right hand side in your local endpoint, navigate to a suitable place to download the movie.\n",
    "Then on the left hand side navigate to `/data/pid_1_123/BEOGN/`.\n",
    "We will use this dataset for this and some other examples later in this session.\n",
    "You should see a list of the files available in this dataset, mostly the data stored in `.fits` format.\n",
    "Select the preview movie, `BEOGN.mp4`, by clicking the checkbox next to it, then click the \"Start\" button above the file list to begin the download.\n",
    "\n",
    "![](./globus1.png)\n",
    "\n",
    "You can check the progress of your transfer by going to the \"Activity\" tab, which shows both active and previous transfers.\n",
    "Various useful information is displayed here but for now the most important is whether the transfer task has failed or succeeded.\n",
    "\n",
    "![](./globus2.png)\n",
    "\n",
    "In either case Globus will also send an email to your registered email address when the task finishes.\n",
    "Of course in this trivial example this is unneccessary, but if you're transfering a whole large dataset it will likely take some time to download and it may be useful to be notified when it's complete.\n",
    "You do not need to leave the web app open for the transfer to continue, but remember that you do need to have GCP running - so if you stop it then your data download will stop as well.\n",
    "\n",
    "If you try transfering the same file again to the same location, you will find that the task completes successfully but the file is not actually transferred.\n",
    "This is to save download time and avoid duplication.\n",
    "\n",
    "## Dataset and downloading\n",
    "\n",
    "Now that we've been over the basic concepts of how data downloads work in Globus, let's see how to do it with the user tools.\n",
    "For this section we don't recommend that you run the download commands as we go through the workshop unless you're willing to wait for them to complete, which may take some time.\n",
    "First let's reload the VBI dataset we were using before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1",
   "metadata": {
    "tags": [
     "keep-inputs"
    ]
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "2",
   "metadata": {},
   "source": [
    "As we saw earlier, we can use the `files` attribute to access information about the number and names of files in the dataset even before downloading any."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "4",
   "metadata": {},
   "source": [
    "The `files` attribute has a `download()` method that we will use for downloading the data.\n",
    "In order to speed up this demonstration a bit, we will just download the first file.\n",
    "To do this we can slice the dataset so that we're only accessing the portion of the data saved in the first file, paying attention to the chunking information in the `Dataset`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "7",
   "metadata": {},
   "source": [
    "The default download directory used by `donwload()` is in the same folder as the ASDF file we loaded, so in this case `~/sunpy/data/VBI/BLKGA`.\n",
    "Since the `download()` method set up the transfer through globus, you can check on the status of your download in the activity tab of the web app as we saw earlier.\n",
    "\n",
    "We can change the download location of the files using the `path` argument.\n",
    "But remember that whatever path you specify must be accessible by Globus Connect Personal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8",
   "metadata": {
    "tags": [
     "skip-execution"
    ]
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "9",
   "metadata": {},
   "source": [
    "The `path` keyword will replace placeholders in the path in the same way as we saw with `Fido.fetch()`.\n",
    "Any key in the dataset inventory (`ds.meta['inventory']`) can be used for this.\n",
    "So for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10",
   "metadata": {
    "tags": [
     "skip-execution"
    ]
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "11",
   "metadata": {},
   "source": [
    "would save the file to `~/sunpy/data/BLKGA/VBI_2022_06_02T17_22_50_173_00486136_I_BLKGA_L1.fits`.\n",
    "\n",
    "If we know that we will want to download an entire dataset, this can be done in the same way but using the full dataset object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12",
   "metadata": {
    "tags": [
     "skip-execution"
    ]
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "13",
   "metadata": {},
   "source": [
    "Alternatively, the user tools offer another function which can also be used to download a full dataset.\n",
    "The `transfer_complete_datasets()` function can take a Fido search results object and download full datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14",
   "metadata": {
    "tags": [
     "skip-execution"
    ]
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "15",
   "metadata": {},
   "source": [
    "Notice that we have to specify `results[\"dkist\"]` here, because `transfer_complete_datasets` only works for DKIST datasets, not any other kind of result that Fido might return.\n",
    "\n",
    "We can also download many datasets at once.\n",
    "For example if we have a proposal ID that we want to download all the data for we could run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16",
   "metadata": {
    "tags": [
     "skip-execution"
    ]
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "17",
   "metadata": {},
   "source": [
    "This will iterate over the results and download each dataset in turn, with a progress bar for each.\n",
    "\n",
    "Of course, if this is a dataset you already know you will want to download all of - for example if it's your own observation - then you may not need to find it through Fido first.\n",
    "Fortunately, `transfer_complete_datasets()`, also lets you specify a dataset or datasets for which to download all files, by passing the dataset IDs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18",
   "metadata": {
    "tags": [
     "skip-execution"
    ]
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "19",
   "metadata": {},
   "source": [
    "Both `transfer_complete_datasets()` and `ds.files.download()` also allow you to specify remote endpoints using the `destination_endpoint` keyword.\n",
    "\n",
    "Normally both of these functions will block the terminal while the download is active.\n",
    "If you want to download a lot of data this is probably not useful, so you can turn this functionality off by passing `wait=False`.\n",
    "This will set up the transfer in Globus but then return from the function.\n",
    "Of course, be cautious with this approach if the next step of your code depends on the data being present.\n",
    "Setting `wait=False` will also skip the wait at the end of each dataset if downloading more than one, so all the transfers will be set up on Globus and then the function will return.\n",
    "\n",
    "To demonstrate this, let's grab some data for the next session, which will be on visualisation.\n",
    "We will want a multi-dimensional dataset, so let's use the VISP one we looked at in the last session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20",
   "metadata": {
    "tags": [
     "skip-execution"
    ]
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "21",
   "metadata": {},
   "source": [
    "Here we're setting the data to download, but notice that the function has returned almost immediately, so we can close the notebook and continue to the next session without interrupting it."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}