Tutorial-ETK_thorn-MaxwellEvol.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<script async src=\"https://www.googletagmanager.com/gtag/js?id=UA-59152712-8\"></script>\n",
    "<script>\n",
    "  window.dataLayer = window.dataLayer || [];\n",
    "  function gtag(){dataLayer.push(arguments);}\n",
    "  gtag('js', new Date());\n",
    "\n",
    "  gtag('config', 'UA-59152712-8');\n",
    "</script>\n",
    "\n",
    "# MaxwellEvol: Solving Maxwell's Equations in the Einstein Toolkit with the Method of Lines\n",
    "\n",
    "## Author: Zach Etienne\n",
    "### Formatting improvements courtesy Brandon Clark\n",
    "\n",
    "### NRPy+ Source Code for this module: [Maxwell/MaxwellCartesian_Evol.py](../edit/Maxwell/MaxwellCartesian_Evol.py) [\\[tutorial\\]](Tutorial-MaxwellCartesian.ipynb) Constructs Maxwell's equations and initial data as SymPy expressions\n",
    "\n",
    "## Introduction:\n",
    "This module focuses on using the equations developed in the [Tutorial-MaxwellCartesian](Tutorial-MaxwellCartesian.ipynb) tutorial notebook to build an Einstein Toolkit (ETK) thorn to solve Maxwell's equations in Cartesian coordinates. This tutorial will focus on implementing the time evolution aspects; the next will construct the thorn that will set up the initial data to be evolved.\n",
    "\n",
    "When interfaced properly with the ETK, this module will propagate the initial data for $E_i$, $A_i$, and $\\psi$ ( and $\\Gamma$, if we so choose), defined in the next tutorial, forward in time by integrating the equations for $\\partial_t E_i$, $\\partial_t A_i$ and $\\partial_t \\psi$ (and possibly $\\partial_t \\Gamma$) subject to spatial boundary conditions. The time evolution itself is handled by the $\\text{MoL}$ (Method of Lines) thorn in the $\\text{CactusNumerical}$ arrangement, and the boundary conditions by the $\\text{Boundary}$ thorn in the $\\text{CactusBase}$ arrangement. \n",
    "\n",
    "Similar to the Maxwell initial data module, we will construct the WaveToyNRPy module in two steps.\n",
    "\n",
    "1. Call on NRPy+ to convert the SymPy expressions for the evolution equations into one C-code kernel.\n",
    "1. Write the C code and linkages to the Einstein Toolkit infrastructure (i.e., the .ccl files) to complete this Einstein Toolkit module."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc'></a>\n",
    "\n",
    "# Table of Contents\n",
    "$$\\label{toc}$$\n",
    "\n",
    "1. [Step 1](#initializenrpy): Initialize core Python/NRPy+ modules\n",
    "1. [Step 2](#etk): Interfacing with the Einstein Toolkit\n",
    "    1. [Step 2.a](#etkc): Constructing the Einstein Toolkit C-code calling functions that include the C code kernels\n",
    "    1. [Step 2.b](#cclfiles): CCL files - Define how this module interacts and interfaces with the larger Einstein Toolkit infrastructure\n",
    "    1. [Step 2.c](#etk_list): Add the C file to Einstein Toolkit compilation list\n",
    "1. [Step 3](#code_validation): Code Validation (**To be performed**)\n",
    "1. [Step 4](#latex_pdf_output): Output this notebook to $\\LaTeX$-formatted PDF file\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='initializenrpy'></a>\n",
    "\n",
    "# Step 1: Initialize core Python/NRPy+ modules \\[Back to [top](#toc)\\]\n",
    "$$\\label{initializenrpy}$$\n",
    "\n",
    "Let's start by importing all the needed modules from Python/NRPy+:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Warning: System I is less stable!\n"
     ]
    }
   ],
   "source": [
    "# Step 1a: Import needed NRPy+ core modules:\n",
    "from outputC import lhrh         # NRPy+: Core C code output module\n",
    "import finite_difference as fin  # NRPy+: Finite difference C code generation module\n",
    "import loop as lp                # NRPy+: Generate C code loops\n",
    "import NRPy_param_funcs as par   # NRPy+: Parameter interface\n",
    "import indexedexp as ixp         # NRPy+: Symbolic indexed expression (e.g., tensors, vectors, etc.) support\n",
    "import grid as gri               # NRPy+: Functions having to do with numerical grids\n",
    "import sympy as sp               # SymPy: The Python computer algebra package upon which NRPy+ depends\n",
    "\n",
    "# Step 1b: This is an Einstein Toolkit (ETK) thorn. Here we\n",
    "#          tell NRPy+ that gridfunction memory access will\n",
    "#          therefore be in the \"ETK\" style.\n",
    "par.set_parval_from_str(\"grid::GridFuncMemAccess\",\"ETK\")\n",
    "#Set the spatial dimension parameter to 3.\n",
    "par.set_parval_from_str(\"grid::DIM\", 3)\n",
    "DIM = par.parval_from_str(\"grid::DIM\")\n",
    "\n",
    "# Step 1c: Call the MaxwellCartesian_Evol() function from within the\n",
    "#          Maxwell/MaxwellCartesian_Evol.py module.\n",
    "import Maxwell.MaxwellCartesian_Evol as mwrhs\n",
    "par.set_parval_from_str(\"System_to_use\",\"System_I\")\n",
    "mwrhs.MaxwellCartesian_Evol()\n",
    "par.set_parval_from_str(\"finite_difference::FD_CENTDERIVS_ORDER\",2)\n",
    "\n",
    "# Step 2: Register gridfunctions so they can be written to by NRPy.\n",
    "# System I:\n",
    "AIrhsD = ixp.register_gridfunctions_for_single_rank1(\"EVOL\",\"AIrhsD\")\n",
    "EIrhsD = ixp.register_gridfunctions_for_single_rank1(\"EVOL\",\"EIrhsD\")\n",
    "psiIrhs = gri.register_gridfunctions(\"EVOL\",\"psiIrhs\")\n",
    "\n",
    "# Step 3: Set the rhs gridfunctions to their variables\n",
    "#         defined by MaxwellEvol().\n",
    "for i in range(DIM):\n",
    "    AIrhsD[i] = mwrhs.ArhsD[i]\n",
    "    EIrhsD[i] = mwrhs.ErhsD[i]\n",
    "psiIrhs = mwrhs.psi_rhs\n",
    "Maxwell_Evol_to_printI = [\\\n",
    "                         lhrh(lhs=gri.gfaccess(\"out_gfs\",\"AIrhsD0\"),rhs=AIrhsD[0]),\\\n",
    "                         lhrh(lhs=gri.gfaccess(\"out_gfs\",\"AIrhsD1\"),rhs=AIrhsD[1]),\\\n",
    "                         lhrh(lhs=gri.gfaccess(\"out_gfs\",\"AIrhsD2\"),rhs=AIrhsD[2]),\\\n",
    "                         lhrh(lhs=gri.gfaccess(\"out_gfs\",\"EIrhsD0\"),rhs=EIrhsD[0]),\\\n",
    "                         lhrh(lhs=gri.gfaccess(\"out_gfs\",\"EIrhsD1\"),rhs=EIrhsD[1]),\\\n",
    "                         lhrh(lhs=gri.gfaccess(\"out_gfs\",\"EIrhsD2\"),rhs=EIrhsD[2]),\\\n",
    "                         lhrh(lhs=gri.gfaccess(\"out_gfs\",\"psiIrhs\"),rhs=psiIrhs)]\n",
    "# Set outCverbose=False to avoid enormous file output.\n",
    "Maxwell_Evol_CcodeKernelI = fin.FD_outputC(\"returnstring\",Maxwell_Evol_to_printI,\n",
    "                                           params=\"outCverbose=False\").replace(\"AD\",\"AID\")\\\n",
    "                                                                      .replace(\"ED\",\"EID\")\\\n",
    "                                                                      .replace(\"psi\",\"psiI\")\n",
    "# For debugging only:\n",
    "#fin.FD_outputC(\"stdout\",Maxwell_Evol_to_printI)\n",
    "\n",
    "gri.glb_gridfcs_list = []\n",
    "# Step 1c: Call the MaxwellCartesian_Evol() function from within the\n",
    "#          Maxwell/MaxwellCartesian_Evol.py module.\n",
    "par.set_parval_from_str(\"System_to_use\",\"System_II\")\n",
    "mwrhs.MaxwellCartesian_Evol()\n",
    "par.set_parval_from_str(\"finite_difference::FD_CENTDERIVS_ORDER\",2)\n",
    "\n",
    "# Step 2: Register gridfunctions so they can be written to by NRPy.\n",
    "# System II:\n",
    "AIIrhsD = ixp.register_gridfunctions_for_single_rank1(\"EVOL\",\"AIIrhsD\")\n",
    "EIIrhsD = ixp.register_gridfunctions_for_single_rank1(\"EVOL\",\"EIIrhsD\")\n",
    "psiIIrhs = gri.register_gridfunctions(\"EVOL\",\"psiIIrhs\")\n",
    "Gammarhs = gri.register_gridfunctions(\"EVOL\",\"Gammarhs\")\n",
    "# Step 3: Set the rhs gridfunctions to their variables\n",
    "#         defined by MaxwellEvol().\n",
    "\n",
    "for i in range(DIM):\n",
    "    AIIrhsD[i] = mwrhs.ArhsD[i]\n",
    "    EIIrhsD[i] = mwrhs.ErhsD[i]\n",
    "psiIIrhs = mwrhs.psi_rhs\n",
    "Gammarhs = mwrhs.Gamma_rhs\n",
    "\n",
    "# Step 4: Create the C code output kernel.\n",
    "Maxwell_Evol_to_printII = [\\\n",
    "                          lhrh(lhs=gri.gfaccess(\"out_gfs\",\"AIIrhsD0\"),rhs=AIIrhsD[0]),\\\n",
    "                          lhrh(lhs=gri.gfaccess(\"out_gfs\",\"AIIrhsD1\"),rhs=AIIrhsD[1]),\\\n",
    "                          lhrh(lhs=gri.gfaccess(\"out_gfs\",\"AIIrhsD2\"),rhs=AIIrhsD[2]),\\\n",
    "                          lhrh(lhs=gri.gfaccess(\"out_gfs\",\"EIIrhsD0\"),rhs=EIIrhsD[0]),\\\n",
    "                          lhrh(lhs=gri.gfaccess(\"out_gfs\",\"EIIrhsD1\"),rhs=EIIrhsD[1]),\\\n",
    "                          lhrh(lhs=gri.gfaccess(\"out_gfs\",\"EIIrhsD2\"),rhs=EIIrhsD[2]),\\\n",
    "                          lhrh(lhs=gri.gfaccess(\"out_gfs\",\"psiIIrhs\"),rhs=psiIIrhs),\\\n",
    "                          lhrh(lhs=gri.gfaccess(\"out_gfs\",\"Gammarhs\"),rhs=Gammarhs)]\n",
    "Maxwell_Evol_CcodeKernelII = fin.FD_outputC(\"returnstring\",Maxwell_Evol_to_printII).replace(\"AD\",\"AIID\")\\\n",
    "                                                                                   .replace(\"ED\",\"EIID\")\\\n",
    "                                                                                   .replace(\"psiGF\",\"psiIIGF\")\n",
    "\n",
    "Maxwell_Evol_looped = lp.loop([\"i2\",\"i1\",\"i0\"],[\"cctk_nghostzones[2]\",\"cctk_nghostzones[1]\",\"cctk_nghostzones[0]\"],\\\n",
    "                             [\"cctk_lsh[2]-cctk_nghostzones[2]\",\"cctk_lsh[1]-cctk_nghostzones[1]\",\\\n",
    "                              \"cctk_lsh[0]-cctk_nghostzones[0]\"],\\\n",
    "                             [\"1\",\"1\",\"1\"],[\"#pragma omp parallel for\",\"\",\"\"],\"\",\\\n",
    "                             (Maxwell_Evol_CcodeKernelI+Maxwell_Evol_CcodeKernelII).replace(\"time\",\"cctk_time\"))\n",
    "\n",
    "# Step 5: Create directories for the thorn if they don't exist.\n",
    "!mkdir MaxwellEvol     2>/dev/null # 2>/dev/null: Don't throw an error if the directory already exists.\n",
    "!mkdir MaxwellEvol/src 2>/dev/null # 2>/dev/null: Don't throw an error if the directory already exists.\n",
    "\n",
    "# Step 6: Write the C code kernel to file.\n",
    "with open(\"MaxwellEvol/src/Maxwell_Evol.h\", \"w\") as file:\n",
    "    file.write(str(Maxwell_Evol_looped))\n",
    "\n",
    "with open(\"MaxwellEvol/src/NRPy_params.h\", \"w\") as file:\n",
    "    file.write(\"#define FD_CENTDERIVS_ORDER \"+str(par.parval_from_str(\"finite_difference::FD_CENTDERIVS_ORDER\"))+\"\\n\")\n",
    "\n",
    "# Step 7: Create a C code kernel to evaluate the constraint violation\n",
    "Cviolation_CcodeKernel = fin.FD_outputC(\"returnstring\",[lhrh(lhs=gri.gfaccess(\"out_gfs\",\"Cviolation\"),rhs=mwrhs.Cviolation)])\n",
    "Cviolation_CcodeKernel_geminated = (Cviolation_CcodeKernel.replace(\"ED\",\"EID\").replace(\"Cviolation\",\"CviolationI\") + \\\n",
    "                                Cviolation_CcodeKernel.replace(\"ED\",\"EIID\").replace(\"Cviolation\",\"CviolationII\"))\n",
    "Cviolation_CcodeKernel_looped = lp.loop([\"i2\",\"i1\",\"i0\"],[\"1\",\"1\",\"1\"],[\"cctk_lsh[2]-1\",\"cctk_lsh[1]-1\",\"cctk_lsh[0]-1\"],\\\n",
    "                                   [\"1\",\"1\",\"1\"],[\"#pragma omp parallel for\",\"\",\"\"],\"\",\\\n",
    "                                   Cviolation_CcodeKernel_geminated.replace(\"time\",\"cctk_time\"))\n",
    "with open(\"MaxwellEvol/src/Constraint_violation.h\", \"w\") as file:\n",
    "    file.write(str(Cviolation_CcodeKernel_looped))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='etk'></a>\n",
    "\n",
    "# Step 2: Interfacing with the Einstein Toolkit \\[Back to [top](#toc)\\]\n",
    "$$\\label{etk}$$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='etkc'></a>\n",
    "\n",
    "## Step 2.a: Constructing the Einstein Toolkit C-code calling functions that include the C code kernels \\[Back to [top](#toc)\\]\n",
    "$$\\label{etkc}$$\n",
    "\n",
    "Now that we have generated the C code kernel `ScalarWave_RHSs.h` and the parameters file `NRPy_params.h`, we will need to write C code to make use of these files. To do this, we can simply follow the example within the [IDScalarWaveNRPy tutorial notebook](Tutorial-ETK_thorn-IDScalarWaveNRPy.ipynb). Functions defined by these files will be called by the Einstein Toolkit scheduler (specified in schedule.ccl below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing MaxwellEvol/src/MaxwellEvol.c\n"
     ]
    }
   ],
   "source": [
    "%%writefile MaxwellEvol/src/MaxwellEvol.c\n",
    "#include <math.h>\n",
    "#include <stdio.h>\n",
    "\n",
    "#include \"cctk.h\"\n",
    "#include \"cctk_Arguments.h\"\n",
    "#include \"cctk_Parameters.h\"\n",
    "#include \"Symmetry.h\"\n",
    "\n",
    "#include \"NRPy_params.h\"\n",
    "\n",
    "void MaxwellEvol_calc_constraint_violation(CCTK_ARGUMENTS) {\n",
    "  DECLARE_CCTK_ARGUMENTS;\n",
    "  DECLARE_CCTK_PARAMETERS;\n",
    "\n",
    "  const CCTK_REAL invdx0 = 1.0 / (CCTK_DELTA_SPACE(0));\n",
    "  const CCTK_REAL invdx1 = 1.0 / (CCTK_DELTA_SPACE(1));\n",
    "  const CCTK_REAL invdx2 = 1.0 / (CCTK_DELTA_SPACE(2));\n",
    "  const CCTK_REAL *gammaDD00GF = gxx;\n",
    "  const CCTK_REAL *gammaDD01GF = gxy;\n",
    "  const CCTK_REAL *gammaDD02GF = gxz;\n",
    "  const CCTK_REAL *gammaDD11GF = gyy;\n",
    "  const CCTK_REAL *gammaDD12GF = gyz;\n",
    "  const CCTK_REAL *gammaDD22GF = gzz;\n",
    "\n",
    "    #include \"Constraint_violation.h\"\n",
    "}\n",
    "\n",
    "void MaxwellEvol_set_rhs(CCTK_ARGUMENTS) {\n",
    "  DECLARE_CCTK_ARGUMENTS;\n",
    "  DECLARE_CCTK_PARAMETERS;\n",
    "\n",
    "  const CCTK_REAL invdx0 = 1.0 / (CCTK_DELTA_SPACE(0));\n",
    "  const CCTK_REAL invdx1 = 1.0 / (CCTK_DELTA_SPACE(1));\n",
    "  const CCTK_REAL invdx2 = 1.0 / (CCTK_DELTA_SPACE(2));\n",
    "  const CCTK_REAL *gammaDD00GF = gxx;\n",
    "  const CCTK_REAL *gammaDD01GF = gxy;\n",
    "  const CCTK_REAL *gammaDD02GF = gxz;\n",
    "  const CCTK_REAL *gammaDD11GF = gyy;\n",
    "  const CCTK_REAL *gammaDD12GF = gyz;\n",
    "  const CCTK_REAL *gammaDD22GF = gzz;\n",
    "\n",
    "#include \"Maxwell_Evol.h\"\n",
    "}\n",
    "\n",
    "/* Boundary Condition code adapted from WaveToyC thorn in ETK, implementing built-in\n",
    " * ETK BC functionality\n",
    " */\n",
    "void MaxwellEvol_SelectBCs(CCTK_ARGUMENTS)\n",
    "{\n",
    "  DECLARE_CCTK_ARGUMENTS;\n",
    "  DECLARE_CCTK_PARAMETERS;\n",
    "\n",
    "  const char *bctype;\n",
    "\n",
    "\n",
    "  bctype = NULL;\n",
    "  if (CCTK_EQUALS(bound,\"flat\") || CCTK_EQUALS(bound,\"static\") ||\n",
    "      CCTK_EQUALS(bound,\"radiation\") || CCTK_EQUALS(bound,\"robin\") ||\n",
    "      CCTK_EQUALS(bound,\"none\"))\n",
    "  {\n",
    "    bctype = bound;\n",
    "  }\n",
    "  else if (CCTK_EQUALS(bound,\"zero\"))\n",
    "  {\n",
    "    bctype = \"scalar\";\n",
    "  }\n",
    "\n",
    "  /* Uses all default arguments, so invalid table handle -1 can be passed */\n",
    "  if (bctype && Boundary_SelectVarForBC (cctkGH, CCTK_ALL_FACES, 1, -1,\n",
    "                                         \"MaxwellEvol::AID0GF\", bctype) < 0)\n",
    "  {\n",
    "    CCTK_WARN (0, \"MaxwellEvol_Boundaries: Error selecting boundary condition\");\n",
    "  }\n",
    "  if (bctype && Boundary_SelectVarForBC (cctkGH, CCTK_ALL_FACES, 1, -1,\n",
    "                                         \"MaxwellEvol::AID1GF\", bctype) < 0)\n",
    "  {\n",
    "    CCTK_WARN (0, \"MaxwellEvol_Boundaries: Error selecting boundary condition\");\n",
    "  }\n",
    "  if (bctype && Boundary_SelectVarForBC (cctkGH, CCTK_ALL_FACES, 1, -1,\n",
    "                                         \"MaxwellEvol::AID2GF\", bctype) < 0)\n",
    "  {\n",
    "    CCTK_WARN (0, \"MaxwellEvol_Boundaries: Error selecting boundary condition\");\n",
    "  }\n",
    "  if (bctype && Boundary_SelectVarForBC (cctkGH, CCTK_ALL_FACES, 1, -1,\n",
    "                                         \"MaxwellEvol::EID0GF\", bctype) < 0)\n",
    "  {\n",
    "    CCTK_WARN (0, \"MaxwellEvol_Boundaries: Error selecting boundary condition\");\n",
    "  }\n",
    "  if (bctype && Boundary_SelectVarForBC (cctkGH, CCTK_ALL_FACES, 1, -1,\n",
    "                                         \"MaxwellEvol::EID1GF\", bctype) < 0)\n",
    "  {\n",
    "    CCTK_WARN (0, \"MaxwellEvol_Boundaries: Error selecting boundary condition\");\n",
    "  }\n",
    "  if (bctype && Boundary_SelectVarForBC (cctkGH, CCTK_ALL_FACES, 1, -1,\n",
    "                                         \"MaxwellEvol::EID2GF\", bctype) < 0)\n",
    "  {\n",
    "    CCTK_WARN (0, \"MaxwellEvol_Boundaries: Error selecting boundary condition\");\n",
    "  }\n",
    "  if (bctype && Boundary_SelectVarForBC (cctkGH, CCTK_ALL_FACES, 1, -1,\n",
    "                                         \"MaxwellEvol::psiIGF\", bctype) < 0)\n",
    "  {\n",
    "    CCTK_WARN (0, \"MaxwellEvol_Boundaries: Error selecting boundary condition\");\n",
    "  }\n",
    "  if (bctype && Boundary_SelectVarForBC (cctkGH, CCTK_ALL_FACES, 1, -1,\n",
    "                                         \"MaxwellEvol::AIID0GF\", bctype) < 0)\n",
    "  {\n",
    "    CCTK_WARN (0, \"MaxwellEvol_Boundaries: Error selecting boundary condition\");\n",
    "  }\n",
    "  if (bctype && Boundary_SelectVarForBC (cctkGH, CCTK_ALL_FACES, 1, -1,\n",
    "                                         \"MaxwellEvol::AIID1GF\", bctype) < 0)\n",
    "  {\n",
    "    CCTK_WARN (0, \"MaxwellEvol_Boundaries: Error selecting boundary condition\");\n",
    "  }\n",
    "  if (bctype && Boundary_SelectVarForBC (cctkGH, CCTK_ALL_FACES, 1, -1,\n",
    "                                         \"MaxwellEvol::AIID2GF\", bctype) < 0)\n",
    "  {\n",
    "    CCTK_WARN (0, \"MaxwellEvol_Boundaries: Error selecting boundary condition\");\n",
    "  }\n",
    "  if (bctype && Boundary_SelectVarForBC (cctkGH, CCTK_ALL_FACES, 1, -1,\n",
    "                                         \"MaxwellEvol::EIID0GF\", bctype) < 0)\n",
    "  {\n",
    "    CCTK_WARN (0, \"MaxwellEvol_Boundaries: Error selecting boundary condition\");\n",
    "  }\n",
    "  if (bctype && Boundary_SelectVarForBC (cctkGH, CCTK_ALL_FACES, 1, -1,\n",
    "                                         \"MaxwellEvol::EIID1GF\", bctype) < 0)\n",
    "  {\n",
    "    CCTK_WARN (0, \"MaxwellEvol_Boundaries: Error selecting boundary condition\");\n",
    "  }\n",
    "  if (bctype && Boundary_SelectVarForBC (cctkGH, CCTK_ALL_FACES, 1, -1,\n",
    "                                         \"MaxwellEvol::EIID2GF\", bctype) < 0)\n",
    "  {\n",
    "    CCTK_WARN (0, \"MaxwellEvol_Boundaries: Error selecting boundary condition\");\n",
    "  }\n",
    "  if (bctype && Boundary_SelectVarForBC (cctkGH, CCTK_ALL_FACES, 1, -1,\n",
    "                                         \"MaxwellEvol::psiIIGF\", bctype) < 0)\n",
    "  {\n",
    "    CCTK_WARN (0, \"MaxwellEvol_Boundaries: Error selecting boundary condition\");\n",
    "  }\n",
    "  if (bctype && Boundary_SelectVarForBC (cctkGH, CCTK_ALL_FACES, 1, -1,\n",
    "                                         \"MaxwellEvol::GammaGF\", bctype) < 0)\n",
    "  {\n",
    "    CCTK_WARN (0, \"MaxwellEvol_Boundaries: Error selecting boundary condition\");\n",
    "  }\n",
    "}\n",
    "\n",
    "void MaxwellEvol_InitSymBound(CCTK_ARGUMENTS)\n",
    "{\n",
    "  DECLARE_CCTK_ARGUMENTS;\n",
    "\n",
    "  int sym[3];\n",
    "\n",
    "  sym[0] = 1;\n",
    "  sym[1] = 1;\n",
    "  sym[2] = 1;\n",
    "\n",
    "  SetCartSymVN(cctkGH, sym,\"MaxwellEvol::AID0GF\");\n",
    "  SetCartSymVN(cctkGH, sym,\"MaxwellEvol::AID1GF\");\n",
    "  SetCartSymVN(cctkGH, sym,\"MaxwellEvol::AID2GF\");\n",
    "  SetCartSymVN(cctkGH, sym,\"MaxwellEvol::EID0GF\");\n",
    "  SetCartSymVN(cctkGH, sym,\"MaxwellEvol::EID1GF\");\n",
    "  SetCartSymVN(cctkGH, sym,\"MaxwellEvol::EID2GF\");\n",
    "  SetCartSymVN(cctkGH, sym,\"MaxwellEvol::psiIGF\");\n",
    "  SetCartSymVN(cctkGH, sym,\"MaxwellEvol::CviolationIGF\");\n",
    "  SetCartSymVN(cctkGH, sym,\"MaxwellEvol::AIID0GF\");\n",
    "  SetCartSymVN(cctkGH, sym,\"MaxwellEvol::AIID1GF\");\n",
    "  SetCartSymVN(cctkGH, sym,\"MaxwellEvol::AIID2GF\");\n",
    "  SetCartSymVN(cctkGH, sym,\"MaxwellEvol::EIID0GF\");\n",
    "  SetCartSymVN(cctkGH, sym,\"MaxwellEvol::EIID1GF\");\n",
    "  SetCartSymVN(cctkGH, sym,\"MaxwellEvol::EIID2GF\");\n",
    "  SetCartSymVN(cctkGH, sym,\"MaxwellEvol::psiIIGF\");\n",
    "  SetCartSymVN(cctkGH, sym,\"MaxwellEvol::GammaGF\");\n",
    "  SetCartSymVN(cctkGH, sym,\"MaxwellEvol::CviolationIIGF\");\n",
    "\n",
    "  return;\n",
    "}\n",
    "\n",
    "void MaxwellEvol_RegisterVars(CCTK_ARGUMENTS)\n",
    "{\n",
    "  DECLARE_CCTK_ARGUMENTS;\n",
    "  DECLARE_CCTK_PARAMETERS;\n",
    "\n",
    "  CCTK_INT ierr CCTK_ATTRIBUTE_UNUSED = 0;\n",
    "  /* Register all the evolved grid functions with MoL */\n",
    "  ierr += MoLRegisterEvolved(CCTK_VarIndex(\"MaxwellEvol::AID0GF\"),  CCTK_VarIndex(\"MaxwellEvol::AIrhsD0GF\"));\n",
    "  ierr += MoLRegisterEvolved(CCTK_VarIndex(\"MaxwellEvol::AID1GF\"),  CCTK_VarIndex(\"MaxwellEvol::AIrhsD1GF\"));\n",
    "  ierr += MoLRegisterEvolved(CCTK_VarIndex(\"MaxwellEvol::AID2GF\"),  CCTK_VarIndex(\"MaxwellEvol::AIrhsD2GF\"));\n",
    "  ierr += MoLRegisterEvolved(CCTK_VarIndex(\"MaxwellEvol::EID0GF\"),  CCTK_VarIndex(\"MaxwellEvol::EIrhsD0GF\"));\n",
    "  ierr += MoLRegisterEvolved(CCTK_VarIndex(\"MaxwellEvol::EID1GF\"),  CCTK_VarIndex(\"MaxwellEvol::EIrhsD1GF\"));\n",
    "  ierr += MoLRegisterEvolved(CCTK_VarIndex(\"MaxwellEvol::EID2GF\"),  CCTK_VarIndex(\"MaxwellEvol::EIrhsD2GF\"));\n",
    "  ierr += MoLRegisterEvolved(CCTK_VarIndex(\"MaxwellEvol::psiIGF\"),  CCTK_VarIndex(\"MaxwellEvol::psiIrhsGF\"));\n",
    "  ierr += MoLRegisterEvolved(CCTK_VarIndex(\"MaxwellEvol::AIID0GF\"),  CCTK_VarIndex(\"MaxwellEvol::AIIrhsD0GF\"));\n",
    "  ierr += MoLRegisterEvolved(CCTK_VarIndex(\"MaxwellEvol::AIID1GF\"),  CCTK_VarIndex(\"MaxwellEvol::AIIrhsD1GF\"));\n",
    "  ierr += MoLRegisterEvolved(CCTK_VarIndex(\"MaxwellEvol::AIID2GF\"),  CCTK_VarIndex(\"MaxwellEvol::AIIrhsD2GF\"));\n",
    "  ierr += MoLRegisterEvolved(CCTK_VarIndex(\"MaxwellEvol::EIID0GF\"),  CCTK_VarIndex(\"MaxwellEvol::EIIrhsD0GF\"));\n",
    "  ierr += MoLRegisterEvolved(CCTK_VarIndex(\"MaxwellEvol::EIID1GF\"),  CCTK_VarIndex(\"MaxwellEvol::EIIrhsD1GF\"));\n",
    "  ierr += MoLRegisterEvolved(CCTK_VarIndex(\"MaxwellEvol::EIID2GF\"),  CCTK_VarIndex(\"MaxwellEvol::EIIrhsD2GF\"));\n",
    "  ierr += MoLRegisterEvolved(CCTK_VarIndex(\"MaxwellEvol::psiIIGF\"),  CCTK_VarIndex(\"MaxwellEvol::psiIIrhsGF\"));\n",
    "  /* Register all the evolved Array functions with MoL */\n",
    "  return;\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='cclfiles'></a>\n",
    "\n",
    "## Step 2.b: CCL files - Define how this module interacts and interfaces with the larger Einstein Toolkit infrastructure \\[Back to [top](#toc)\\]\n",
    "$$\\label{cclfiles}$$\n",
    "\n",
    "Writing a module (\"thorn\") within the Einstein Toolkit requires that three \"ccl\" files be constructed, all in the root directory of the thorn:\n",
    "\n",
    "1. `interface.ccl`: defines the gridfunction groups needed, and provides keywords denoting what this thorn provides and what it should inherit from other thorns. This file governs the interaction between this thorn and others; more information can be found in the [official Einstein Toolkit documentation](http://einsteintoolkit.org/usersguide/UsersGuidech12.html#x17-178000D2.2). \n",
    "With \"implements\", we give our thorn its unique name. By \"inheriting\" other thorns, we tell the Toolkit that we will rely on variables that exist within those functions. Then, we tell the toolkit that we want the gridfunctions $A_i$, $E_i$, $\\psi$, and $\\Gamma$ to be visible to other thorns by using the keyword \"public\". "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing MaxwellEvol/interface.ccl\n"
     ]
    }
   ],
   "source": [
    "%%writefile MaxwellEvol/interface.ccl\n",
    "implements: MaxwellEvol\n",
    "\n",
    "inherits: admbase Boundary grid\n",
    "\n",
    "USES INCLUDE: loopcontrol.h\n",
    "USES INCLUDE: Symmetry.h\n",
    "USES INCLUDE: Boundary.h\n",
    "\n",
    "CCTK_INT FUNCTION MoLRegisterEvolved(CCTK_INT IN EvolvedIndex, CCTK_INT IN RHSIndex)\n",
    "USES FUNCTION MoLRegisterEvolved\n",
    "\n",
    "CCTK_INT FUNCTION GetBoundarySpecification(CCTK_INT IN size, CCTK_INT OUT ARRAY nboundaryzones, CCTK_INT OUT ARRAY is_internal, CCTK_INT OUT ARRAY is_staggered, CCTK_INT OUT ARRAY shiftout)\n",
    "USES FUNCTION GetBoundarySpecification\n",
    "\n",
    "CCTK_INT FUNCTION SymmetryTableHandleForGrid(CCTK_POINTER_TO_CONST IN cctkGH)\n",
    "USES FUNCTION SymmetryTableHandleForGrid\n",
    "\n",
    "CCTK_INT FUNCTION Boundary_SelectGroupForBC(CCTK_POINTER_TO_CONST IN GH, CCTK_INT IN faces, CCTK_INT IN boundary_width, CCTK_INT IN table_handle, CCTK_STRING IN group_name, CCTK_STRING IN bc_name)\n",
    "USES FUNCTION Boundary_SelectGroupForBC\n",
    "\n",
    "CCTK_INT FUNCTION Boundary_SelectVarForBC(CCTK_POINTER_TO_CONST IN GH, CCTK_INT IN faces, CCTK_INT IN boundary_width, CCTK_INT IN table_handle, CCTK_STRING IN var_name, CCTK_STRING IN bc_name)\n",
    "USES FUNCTION Boundary_SelectVarForBC\n",
    "\n",
    "public:\n",
    "cctk_real system_I_rhs type = GF Timelevels=3 tags='tensortypealias=\"Scalar\"'\n",
    "{\n",
    "  AIrhsD0GF,AIrhsD1GF,AIrhsD2GF,EIrhsD0GF,EIrhsD1GF,EIrhsD2GF,psiIrhsGF\n",
    "} \"The evolved scalar fields\"\n",
    "\n",
    "public:\n",
    "cctk_real system_II_rhs type = GF Timelevels=3 tags='tensortypealias=\"Scalar\"'\n",
    "{\n",
    "  AIIrhsD0GF,AIIrhsD1GF,AIIrhsD2GF,EIIrhsD0GF,EIIrhsD1GF,EIIrhsD2GF,psiIIrhsGF,GammarhsGF\n",
    "} \"The evolved scalar fields\"\n",
    "\n",
    "public:\n",
    "cctk_real system_I type = GF Timelevels=3 tags='tensortypealias=\"Scalar\"'\n",
    "{\n",
    "  AID0GF,AID1GF,AID2GF,EID0GF,EID1GF,EID2GF,psiIGF\n",
    "} \"The evolved scalar fields\"\n",
    "\n",
    "public:\n",
    "cctk_real system_II type = GF Timelevels=3 tags='tensortypealias=\"Scalar\"'\n",
    "{\n",
    "  AIID0GF,AIID1GF,AIID2GF,EIID0GF,EIID1GF,EIID2GF,psiIIGF,GammaGF\n",
    "} \"The evolved scalar fields\"\n",
    "\n",
    "public:\n",
    "cctk_real system_I_constraint_violation type = GF Timelevels=3 tags='tensortypealias=\"Scalar\"'\n",
    "{\n",
    "  CviolationIGF\n",
    "} \"The constraint violation for system I\"\n",
    "\n",
    "public:\n",
    "cctk_real system_II_constraint_violation type = GF Timelevels=3 tags='tensortypealias=\"Scalar\"'\n",
    "{\n",
    "  CviolationIIGF\n",
    "} \"The constraint violation for system II\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. `param.ccl`: specifies free parameters within the thorn, enabling them to be set at runtime. It is required to provide allowed ranges and default values for each parameter. More information on this file's syntax can be found in the [official Einstein Toolkit documentation](http://einsteintoolkit.org/usersguide/UsersGuidech12.html#x17-183000D2.3). A number of parameters are defined, and more parameters can be easily added in later versions. We also set the number of timelevels we will store in memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing MaxwellEvol/param.ccl\n"
     ]
    }
   ],
   "source": [
    "%%writefile MaxwellEvol/param.ccl\n",
    "shares: MethodOfLines\n",
    "\n",
    "USES CCTK_INT MoL_Num_Evolved_Vars\n",
    "USES CCTK_INT MoL_Num_ArrayEvolved_Vars\n",
    "\n",
    "restricted:\n",
    "CCTK_INT MaxwellEvol_MaxNumEvolvedVars \"Number of evolved variables used by this thorn\" ACCUMULATOR-BASE=MethodofLines::MoL_Num_Evolved_Vars STEERABLE=RECOVER\n",
    "{\n",
    "  15:15 :: \"Number of evolved variables used by this thorn\"\n",
    "} 15\n",
    "\n",
    "restricted:\n",
    "CCTK_INT SimpleWave_MaxNumArrayEvolvedVars \"Number of Array evolved variables used by this thorn\" ACCUMULATOR-BASE=MethodofLines::MoL_Num_ArrayEvolved_Vars STEERABLE=RECOVER\n",
    "{\n",
    "  0:0 :: \"Number of Array evolved variables used by this thorn\"\n",
    "} 0\n",
    "\n",
    "restricted:\n",
    "KEYWORD bound \"Type of boundary condition to use\"\n",
    "{\n",
    "  \"flat\"      :: \"Flat (von Neumann, n grad phi = 0) boundary condition\"\n",
    "  \"static\"    :: \"Static (Dirichlet, dphi/dt=0) boundary condition\"\n",
    "  \"radiation\" :: \"Radiation boundary condition\"\n",
    "  \"robin\"     :: \"Robin (phi(r) = C/r) boundary condition\"\n",
    "  \"zero\"      :: \"Zero (Dirichlet, phi=0) boundary condition\"\n",
    "  \"none\"      :: \"Apply no boundary condition\"\n",
    "} \"static\"\n",
    "\n",
    "restricted:\n",
    "CCTK_INT timelevels \"Number of active timelevels\" STEERABLE=RECOVER\n",
    "{\n",
    "  0:3 :: \"\"\n",
    "} 3\n",
    "\n",
    "restricted:\n",
    "CCTK_REAL amp \"The amplitude of the wavepacket\"\n",
    "{\n",
    " *:* :: \"The amplitude of the wavepacket\"\n",
    "} 1.0\n",
    "\n",
    "restricted:\n",
    "CCTK_REAL lam \"The size lambda of the wavepacket\"\n",
    "{\n",
    " *:* :: \"The size lambda of the wavepacket\"\n",
    "} 1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. `schedule.ccl`: allocates storage for gridfunctions, defines how the thorn's functions should be scheduled in a broader simulation, and specifies the regions of memory written to or read from gridfunctions. `schedule.ccl`'s official documentation may be found [here](http://einsteintoolkit.org/usersguide/UsersGuidech12.html#x17-186000D2.4). \n",
    "\n",
    "We first assign storage for both scalar gridfunctions, and then specify the standardized ETK \"scheduling bins\" in which we want each of our thorn's functions to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing MaxwellEvol/schedule.ccl\n"
     ]
    }
   ],
   "source": [
    "%%writefile MaxwellEvol/schedule.ccl\n",
    "STORAGE: system_I_rhs[timelevels]\n",
    "STORAGE: system_I[timelevels]\n",
    "STORAGE: system_II_rhs[timelevels]\n",
    "STORAGE: system_II[timelevels]\n",
    "STORAGE: system_I_constraint_violation[timelevels]\n",
    "STORAGE: system_II_constraint_violation[timelevels]\n",
    "\n",
    "schedule MaxwellEvol_InitSymBound at BASEGRID\n",
    "{\n",
    "  LANG: C\n",
    "  OPTIONS: global\n",
    "} \"Schedule symmetries\"\n",
    "\n",
    "schedule MaxwellEvol_calc_constraint_violation IN CCTK_ANALYSIS\n",
    "{\n",
    "  LANG: C\n",
    "  READS: admbase::gxx(Everywhere)\n",
    "  READS: admbase::gxy(Everywhere)\n",
    "  READS: admbase::gxz(Everywhere)\n",
    "  READS: admbase::gyy(Everywhere)\n",
    "  READS: admbase::gyz(Everywhere)\n",
    "  READS: admbase::gzz(Everywhere)\n",
    "  READS: EID0GF(Everywhere)\n",
    "  READS: EID1GF(Everywhere)\n",
    "  READS: EID2GF(Everywhere)\n",
    "  READS: EIID0GF(Everywhere)\n",
    "  READS: EIID1GF(Everywhere)\n",
    "  READS: EIID2GF(Everywhere)\n",
    "  WRITES: CviolationIGF(Interior)\n",
    "  WRITES: CviolationIIGF(Interior)\n",
    "}\"Calculate the contraint violation of the simulation\"\n",
    "\n",
    "schedule MaxwellEvol_set_rhs as WaveToy_Evolution IN MoL_CalcRHS\n",
    "{\n",
    "  LANG: C\n",
    "  READS: admbase::gxx(Everywhere)\n",
    "  READS: admbase::gxy(Everywhere)\n",
    "  READS: admbase::gxz(Everywhere)\n",
    "  READS: admbase::gyy(Everywhere)\n",
    "  READS: admbase::gyz(Everywhere)\n",
    "  READS: admbase::gzz(Everywhere)\n",
    "  READS: AID0GF(Everywhere)\n",
    "  READS: AID1GF(Everywhere)\n",
    "  READS: AID2GF(Everywhere)\n",
    "  READS: EID0GF(Everywhere)\n",
    "  READS: EID1GF(Everywhere)\n",
    "  READS: EID2GF(Everywhere)\n",
    "  READS: psiIGF(Everywhere)\n",
    "  READS: AIID0GF(Everywhere)\n",
    "  READS: AIID1GF(Everywhere)\n",
    "  READS: AIID2GF(Everywhere)\n",
    "  READS: EIID0GF(Everywhere)\n",
    "  READS: EIID1GF(Everywhere)\n",
    "  READS: EIID2GF(Everywhere)\n",
    "  READS: psiIIGF(Everywhere)\n",
    "  READS: GammaGF(Everywhere)\n",
    "  WRITES: AIrhsD0GF(Interior)\n",
    "  WRITES: AIrhsD1GF(Interior)\n",
    "  WRITES: AIrhsD2GF(Interior)\n",
    "  WRITES: EIrhsD0GF(Interior)\n",
    "  WRITES: EIrhsD1GF(Interior)\n",
    "  WRITES: EIrhsD2GF(Interior)\n",
    "  WRITES: psiIrhsGF(Interior)\n",
    "  WRITES: AIIrhsD0GF(Interior)\n",
    "  WRITES: AIIrhsD1GF(Interior)\n",
    "  WRITES: AIIrhsD2GF(Interior)\n",
    "  WRITES: EIIrhsD0GF(Interior)\n",
    "  WRITES: EIIrhsD1GF(Interior)\n",
    "  WRITES: EIIrhsD2GF(Interior)\n",
    "  WRITES: psiIIrhsGF(Interior)\n",
    "  WRITES: GammarhsGF(Interior)\n",
    "} \"Evolution of Maxwell's equations\"\n",
    "\n",
    "schedule MaxwellEvol_SelectBCs in MoL_PostStep\n",
    "{\n",
    "  LANG: C\n",
    "  OPTIONS: level\n",
    "  SYNC: system_I\n",
    "  SYNC: system_II\n",
    "} \"Boundaries of Maxwell's equations\"\n",
    "\n",
    "schedule GROUP ApplyBCs as MaxwellEvol_ApplyBCs in MoL_PostStep after MaxwellEvol_SelectBCs\n",
    "{\n",
    "} \"Apply boundary conditions\"\n",
    "\n",
    "\n",
    "schedule GROUP ApplyBCs as MaxwellEvol_ApplyBCs at POSTRESTRICT\n",
    "{\n",
    "} \"Apply boundary conditions\"\n",
    "\n",
    "schedule MaxwellEvol_RegisterVars in MoL_Register\n",
    "{\n",
    "  LANG: C\n",
    "  OPTIONS: meta\n",
    "} \"Register Variables for MoL\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='etk_list'></a>\n",
    "\n",
    "## Step 2.c: Add the C file to Einstein Toolkit compilation list \\[Back to [top](#toc)\\]\n",
    "$$\\label{etk_list}$$\n",
    "\n",
    "We will also need `make.code.defn`, which indicates the list of files that need to be compiled. This thorn only has the one C file to compile."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing MaxwellEvol/src/make.code.defn\n"
     ]
    }
   ],
   "source": [
    "%%writefile MaxwellEvol/src/make.code.defn\n",
    "SRCS = MaxwellEvol.c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='code_validation'></a>\n",
    "\n",
    "# Step 3: Code Validation (To be performed) \\[Back to [top](#toc)\\]\n",
    "$$\\label{code_validation}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='latex_pdf_output'></a>\n",
    "\n",
    "# Step 4: Output this notebook to $\\LaTeX$-formatted PDF file \\[Back to [top](#toc)\\]\n",
    "$$\\label{latex_pdf_output}$$\n",
    "\n",
    "The following code cell converts this Jupyter notebook into a proper, clickable $\\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename\n",
    "[Tutorial-ETK_thorn-MaxwellEvol.pdf](Tutorial-ETK_thorn-MaxwellEvol.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created Tutorial-ETK_thorn-MaxwellEvol.tex, and compiled LaTeX file to PDF\n",
      "    file Tutorial-ETK_thorn-MaxwellEvol.pdf\n"
     ]
    }
   ],
   "source": [
    "import cmdline_helper as cmd    # NRPy+: Multi-platform Python command-line interface\n",
    "cmd.output_Jupyter_notebook_to_LaTeXed_PDF(\"Tutorial-ETK_thorn-MaxwellEvol\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}