examples/Getting_started.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting started with hepdata_lib\n",
    "\n",
    "The following instructions and examples should get you started to get your analysis into [HEPData](https://hepdata.net) using `hepdata_lib`. Please also refer to the [documentation](http://hepdata-lib.readthedocs.io/). While you can also run `hepdata_lib` on your local computer, you can use the [binder](https://mybinder.org/) or [SWAN](http://swan.cern.ch/) services in the browser. Mind that SWAN is only available for people with a CERN account."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## General setup\n",
    "\n",
    "To make sure things are working and `hepdata_lib` is available, run the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Welcome to JupyROOT 6.30/04\n",
      "hepdata_lib version 0.15.0\n"
     ]
    }
   ],
   "source": [
    "import hepdata_lib\n",
    "print(\"hepdata_lib version\", hepdata_lib.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating your HEPData submission\n",
    "\n",
    "The `Submission` object represents the whole HEPData entry and thus carries the top-level meta data that is equally valid for all the tables and variables you may want to enter. The object is also used to create the physical submission files you will upload to the HEPData web interface.\n",
    "\n",
    "When using `hepdata_lib` to make an entry, you always need to create a `Submission` object. Let's do that now, and then add data to it step by step:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from hepdata_lib import Submission\n",
    "submission = Submission()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In general, a `Submission` should contain details on the actual analysis such as it's abstract as well as links to the actual publication. The abstract should be in a plain text file. For `inspire` there's a special `record_id`, while for links to `arXiv` etc. one should use plain hyperlinks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "submission.read_abstract(\"example_inputs/abstract.txt\")\n",
    "submission.add_link(\"Webpage with all figures and tables\", \"https://cms-results.web.cern.ch/cms-results/public-results/publications/B2G-16-029/\")\n",
    "submission.add_link(\"arXiv\", \"http://arxiv.org/abs/arXiv:1802.09407\")\n",
    "submission.add_record_id(1657397, \"inspire\")\n",
    "submission.add_additional_resource(\"Original abstract file\", \"example_inputs/abstract.txt\", copy_file=True)  # for illustration, probably not useful"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding a table/figure\n",
    "\n",
    "In HEPData, figures and table will both be `Table` objects. The example here shows reading a plain text file containing the signal effiency times acceptance as a function of resonance mass for different signal models. The file has been uploaded to the `example_files` directory. For your submission, create a new directory, e.g. using the analysis identifier.\n",
    "\n",
    "Let's have a look at the file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mass     BulkG             Wprime\n",
      "-------  -------           -------\n",
      "1000.0   0.4651            0.45136\n",
      "1200.0   0.50336           0.5109\n",
      "1400.0   0.5126            0.54016\n",
      "1600.0   0.52474           0.5513\n",
      "1800.0   0.531             0.56724\n",
      "2000.0   0.5391            0.5728\n",
      "2500.0   0.549430894308    0.585602409637\n",
      "3000.0   0.55378           0.58952\n"
     ]
    }
   ],
   "source": [
    "!head example_inputs/effacc_signal.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first column is the mass value, the other columns contain the efficiency times acceptance values.\n",
    "\n",
    "Let's create the table/figure. First, we need to give it a name, which is usually just the identifier in the paper, here \"Figure 1\". The table also needs a description, which is usually the caption. You also need to describe the location, i.e. where to find it in the publication:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from hepdata_lib import Table\n",
    "table = Table(\"Additional Figure 1\")\n",
    "table.description = \"Signal selection efficiency times acceptance as a function of resonance mass for a spin-2 bulk graviton decaying to WW and a spin-1 W' decaying to WZ.\"\n",
    "table.location = \"Data from additional Figure 1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need to provide more information on what is actually shown, which is done via `keywords`. The ones that are available can be taken from the documentation:\n",
    "- [Observables](https://hepdata-submission.readthedocs.io/en/latest/keywords/observables.html)\n",
    "- [Phrases](https://hepdata-submission.readthedocs.io/en/latest/keywords/phrases.html)\n",
    "- [Particles](https://hepdata-submission.readthedocs.io/en/latest/keywords/partlist.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "table.keywords[\"observables\"] = [\"ACC\", \"EFF\"]\n",
    "table.keywords[\"reactions\"] = [\"P P --> GRAVITON --> W+ W-\", \"P P --> WPRIME --> W+/W- Z0\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's read in the file. For this purpose, `numpy` is very handy. Since the first two rows are the header, we skip them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "data = np.loadtxt(\"example_inputs/effacc_signal.txt\", skiprows=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`numpy` stores the content as arrays. You can actually see that the entry that was labelled as `NaN` is correctly read in:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1.00000000e+03 4.65100000e-01 4.51360000e-01]\n",
      " [1.20000000e+03 5.03360000e-01 5.10900000e-01]\n",
      " [1.40000000e+03 5.12600000e-01 5.40160000e-01]\n",
      " [1.60000000e+03 5.24740000e-01 5.51300000e-01]\n",
      " [1.80000000e+03 5.31000000e-01 5.67240000e-01]\n",
      " [2.00000000e+03 5.39100000e-01 5.72800000e-01]\n",
      " [2.50000000e+03 5.49430894e-01 5.85602410e-01]\n",
      " [3.00000000e+03 5.53780000e-01 5.89520000e-01]\n",
      " [3.50000000e+03 5.62160000e-01 6.03240000e-01]\n",
      " [4.00000000e+03 5.64538153e-01            nan]\n",
      " [4.50000000e+03 5.66820000e-01 5.99780000e-01]]\n"
     ]
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now use this for our `Variable` definitions. The x-axis is usually the independent variable (`is_independent=True`), whereas the other ones are dependent (i.e. a function of the former). You also need to declare whether the variable is binned or not as well as the units. Similar as for the `keywords` used above, it is again important to provide additional information that can be found via the HEPData web interface using the observables and particles linked above. The values assigned are just slices of the `data` array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from hepdata_lib import Variable\n",
    "d = Variable(\"Resonance mass\", is_independent=True, is_binned=False, units=\"GeV\")\n",
    "d.values = data[:,0]\n",
    "\n",
    "BulkG = Variable(\"Efficiency times acceptance\", is_independent=False, is_binned=False, units=\"\")\n",
    "BulkG.values = data[:,1]\n",
    "BulkG.add_qualifier(\"Efficiency times acceptance\", \"Bulk graviton --> WW\")\n",
    "BulkG.add_qualifier(\"SQRT(S)\", 13, \"TeV\")\n",
    "\n",
    "Wprime = Variable(\"Efficiency times acceptance\", is_independent=False, is_binned=False, units=\"\")\n",
    "Wprime.values = data[:,2]\n",
    "Wprime.add_qualifier(\"Efficiency times acceptance\", \"Wprime --> WZ\")\n",
    "Wprime.add_qualifier(\"SQRT(S)\", 13, \"TeV\")\n",
    "\n",
    "table.add_variable(d)\n",
    "table.add_variable(BulkG)\n",
    "table.add_variable(Wprime)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In case of a plot, you should also add the original figure itself. `hepdata_lib` will take care of creating the thumbnail as well. Just add the figure as below.\n",
    "\n",
    "*WARNING*: This needs `ImageMagick` to be installed (this is the case when running on Binder and SWAN with LCG_94 or later). Executing the following line will fail if it is missing. In this case, comment out this line and restart from the top."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "table.add_image(\"example_inputs/signalEffVsMass.pdf\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want, the original data file can be attached to the table as an additional resource file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "table.add_additional_resource(\"Original data file\", \"example_inputs/effacc_signal.txt\", copy_file=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is all that's needed for the table/figure. We still need to add it to the submission:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "submission.add_table(table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you've added all tables/figures and the general submission details, you should add a few more keywords to all tables for better identification and searchability, e.g. the centre-of-mass energy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "for table in submission.tables:\n",
    "    table.keywords[\"cmenergies\"] = [13000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it's time to create the submission for the upload. Here, we choose `example_output` as output directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "outdir = \"example_output\"\n",
    "submission.create_files(outdir, remove_old=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the working directory, you will now find a `submission.tar.gz` file, which you can use for uploading to your HEPData sandbox:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "submission.tar.gz\n"
     ]
    }
   ],
   "source": [
    "!ls submission.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the `example_output` directory will contain the generated `yaml` files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "abstract.txt              effacc_signal.txt         submission.yaml\n",
      "additional_figure_1.yaml  signalEffVsMass.png       thumb_signalEffVsMass.png\n"
     ]
    }
   ],
   "source": [
    "!ls example_output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---\n",
      "additional_resources:\n",
      "- description: Created with hepdata_lib 0.15.0\n",
      "  location: https://doi.org/10.5281/zenodo.1217998\n",
      "- description: Webpage with all figures and tables\n",
      "  location: https://cms-results.web.cern.ch/cms-results/public-results/publications/B2G-16-029/\n",
      "- description: arXiv\n",
      "  location: http://arxiv.org/abs/arXiv:1802.09407\n",
      "- description: Original abstract file\n",
      "  location: abstract.txt\n",
      "comment: A search for a new heavy particle decaying to a pair of vector bosons (WW\n",
      "  or WZ) is presented using data from the CMS detector corresponding to an integrated\n",
      "  luminosity of $35.9~\\mathrm{fb}^{-1}$ collected in proton-proton collisions at a\n",
      "  centre-of-mass energy of 13~TeV in 2016. One of the bosons is required to be a W\n",
      "  boson decaying to $e\\nu$ or $mu\\nu$, while the other boson is required to be reconstructed\n",
      "  as a single massive jet with substructure compatible with that of a highly-energetic\n",
      "  quark pair from a W or Z boson decay. The search is performed in the resonance mass\n",
      "  range between 1.0 and 4.5~TeV.  The largest deviation from the background-only hypothesis\n",
      "  is observed for a mass near 1.4~TeV and corresponds to a local significance of 2.5\n",
      "  standard deviations. The result is interpreted as an upper bound on the resonance\n",
      "  production cross section. Comparing the excluded cross section values and the expectations\n",
      "  from theoretical calculations in the bulk graviton and heavy vector triplet models,\n",
      "  spin-2 WW resonances with mass smaller than 1.07~TeV and spin-1 WZ resonances lighter\n",
      "  than 3.05~TeV, respectively, are excluded at 95\\% confidence level.\n",
      "data_license:\n",
      "  description: CC0 enables reusers to distribute, remix, adapt, and build upon the\n",
      "    material in any medium or format, with no conditions.\n",
      "  name: CC0\n",
      "  url: https://creativecommons.org/publicdomain/zero/1.0/\n",
      "record_ids:\n",
      "- id: 1657397\n",
      "  type: inspire\n",
      "---\n",
      "additional_resources:\n",
      "- description: Original data file\n",
      "  location: effacc_signal.txt\n",
      "- description: Image file\n",
      "  location: signalEffVsMass.png\n",
      "- description: Thumbnail image file\n",
      "  location: thumb_signalEffVsMass.png\n",
      "data_file: additional_figure_1.yaml\n",
      "description: Signal selection efficiency times acceptance as a function of resonance\n",
      "  mass for a spin-2 bulk graviton decaying to WW and a spin-1 W' decaying to WZ.\n",
      "keywords:\n",
      "- name: observables\n",
      "  values:\n",
      "  - ACC\n",
      "  - EFF\n",
      "- name: reactions\n",
      "  values:\n",
      "  - P P --> GRAVITON --> W+ W-\n",
      "  - P P --> WPRIME --> W+/W- Z0\n",
      "- name: cmenergies\n",
      "  values:\n",
      "  - 13000\n",
      "location: Data from additional Figure 1\n",
      "name: Additional Figure 1\n"
     ]
    }
   ],
   "source": [
    "!cat example_output/submission.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dependent_variables:\n",
      "- header:\n",
      "    name: Efficiency times acceptance\n",
      "  qualifiers:\n",
      "  - name: Efficiency times acceptance\n",
      "    value: Bulk graviton --> WW\n",
      "  - name: SQRT(S)\n",
      "    units: TeV\n",
      "    value: 13\n",
      "  values:\n",
      "  - value: 0.4651\n",
      "  - value: 0.50336\n",
      "  - value: 0.5126\n",
      "  - value: 0.52474\n",
      "  - value: 0.531\n",
      "  - value: 0.5391\n",
      "  - value: 0.54943\n",
      "  - value: 0.55378\n",
      "  - value: 0.56216\n",
      "  - value: 0.56454\n",
      "  - value: 0.56682\n",
      "- header:\n",
      "    name: Efficiency times acceptance\n",
      "  qualifiers:\n",
      "  - name: Efficiency times acceptance\n",
      "    value: Wprime --> WZ\n",
      "  - name: SQRT(S)\n",
      "    units: TeV\n",
      "    value: 13\n",
      "  values:\n",
      "  - value: 0.45136\n",
      "  - value: 0.5109\n",
      "  - value: 0.54016\n",
      "  - value: 0.5513\n",
      "  - value: 0.56724\n",
      "  - value: 0.5728\n",
      "  - value: 0.5856\n",
      "  - value: 0.58952\n",
      "  - value: 0.60324\n",
      "  - value: .nan\n",
      "  - value: 0.59978\n",
      "independent_variables:\n",
      "- header:\n",
      "    name: Resonance mass\n",
      "    units: GeV\n",
      "  values:\n",
      "  - value: 1000.0\n",
      "  - value: 1200.0\n",
      "  - value: 1400.0\n",
      "  - value: 1600.0\n",
      "  - value: 1800.0\n",
      "  - value: 2000.0\n",
      "  - value: 2500.0\n",
      "  - value: 3000.0\n",
      "  - value: 3500.0\n",
      "  - value: 4000.0\n",
      "  - value: 4500.0\n"
     ]
    }
   ],
   "source": [
    "!cat example_output/additional_figure_1.yaml"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}