01_actions.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|default_exp actions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GitHub Actions details\n",
    "\n",
    "> Functionality for helping to create GitHub Actions workflows in Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "from fastcore.all import *\n",
    "from ghapi.core import *\n",
    "from ghapi.templates import *\n",
    "\n",
    "import textwrap\n",
    "from contextlib import contextmanager\n",
    "from enum import Enum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|hide\n",
    "from nbdev import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Workflow setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In your GitHub Actions workflow, include the following in your `run` step:\n",
    "\n",
    "```bash\n",
    "env:\n",
    "  CONTEXT_GITHUB: ${{ toJson(github) }}\n",
    "```\n",
    "\n",
    "This stores the full [github context](https://docs.github.com/actions/reference/context-and-expression-syntax-for-github-actions#github-context), which includes information such as the name of the current workflow being run, the GitHub access token, and so forth.\n",
    "\n",
    "As well as the `github` context, you can do that same thing for any of the other GitHub Actions contexts, which are:\n",
    "\n",
    "`github` `env` `job` `steps` `runner` `secrets` `strategy` `matrix` `needs`\n",
    "\n",
    "For instance, for the [needs](https://docs.github.com/actions/reference/workflow-syntax-for-github-actions#jobsjob_idneeds) context, information about previous jobs specified in your `needs` clause, add this underneath your `CONTEXT_GITHUB` line:\n",
    "\n",
    "```bash\n",
    "  CONTEXT_NEEDS: ${{ toJson(needs) }}\n",
    "```\n",
    "\n",
    "Note that here's no harm having entries that are not used -- GitHub Actions will set them to an empty dictionary by default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "# So we can run this outside of GitHub actions too, read from file if needed\n",
    "for a,b in (('CONTEXT_GITHUB',context_example), ('CONTEXT_NEEDS',needs_example), ('GITHUB_REPOSITORY','octocat/Hello-World')):\n",
    "    if a not in os.environ: os.environ[a] = b\n",
    "\n",
    "contexts = 'github', 'env', 'job', 'steps', 'runner', 'secrets', 'strategy', 'matrix', 'needs'\n",
    "for context in contexts:\n",
    "    globals()[f'context_{context}'] = dict2obj(loads(os.getenv(f\"CONTEXT_{context.upper()}\", \"{}\")))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "_all_ = ['context_github', 'context_env', 'context_job', 'context_steps', 'context_runner', 'context_secrets', 'context_strategy', 'context_matrix', 'context_needs']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GitHub also adds a number of [`GITHUB_*` environment variables](https://docs.github.com/actions/reference/environment-variables#default-environment-variables) to all runners. These are available through the `env_github` `AttrDict`, with the `GITHUB_` prefix removed, and remainder converted to lowercase. For instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "env_github = dict2obj({k[7:].lower():v for k,v in os.environ.items() if k.startswith('GITHUB_')})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'octocat/Hello-World'"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env_github.repository"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "def user_repo():\n",
    "    \"List of `user,repo` from `env_github.repository\"\n",
    "    return env_github.repository.split('/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['octocat', 'Hello-World']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_repo()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The possible events are available in the `Event` `enum`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'page_build','content_reference','repository_import','create','workflow_run','delete','organization','sponsorship','project_column','push','context','milestone','project_card','project','package','pull_request','repository_dispatch','team_add','workflow_dispatch','member','meta','code_scanning_alert','public','needs','check_run','security_advisory','pull_request_review_comment','org_block','commit_comment','watch','marketplace_purchase','star','installation_repositories','check_suite','github_app_authorization','team','status','repository_vulnerability_alert','pull_request_review','label','installation','release','issues','repository','gollum','membership','deployment','deploy_key','issue_comment','ping','deployment_status','fork'\n"
     ]
    }
   ],
   "source": [
    "#|hide\n",
    "print(','.join(repr(o) for o in Path('examples/').ls(file_exts=['.json']).attrgot('stem')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "Event = str_enum('Event',\n",
    "    'page_build','content_reference','repository_import','create','workflow_run','delete','organization','sponsorship',\n",
    "    'project_column','push','context','milestone','project_card','project','package','pull_request','repository_dispatch',\n",
    "    'team_add','workflow_dispatch','member','meta','code_scanning_alert','public','needs','check_run','security_advisory',\n",
    "    'pull_request_review_comment','org_block','commit_comment','watch','marketplace_purchase','star','installation_repositories',\n",
    "    'check_suite','github_app_authorization','team','status','repository_vulnerability_alert','pull_request_review','label',\n",
    "    'installation','release','issues','repository','gollum','membership','deployment','deploy_key','issue_comment','ping',\n",
    "    'deployment_status','fork','schedule')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'page_build, content_reference, repository_import, create, workflow_run, delete, organization, sponsorship, project_column, push, context, milestone, project_card, project, package, pull_request, repository_dispatch, team_add, workflow_dispatch, member, meta, code_scanning_alert, public, needs, check_run, security_advisory, pull_request_review_comment, org_block, commit_comment, watch, marketplace_purchase, star, installation_repositories, check_suite, github_app_authorization, team, status, repository_vulnerability_alert, pull_request_review, label, installation, release, issues, repository, gollum, membership, deployment, deploy_key, issue_comment, ping, deployment_status, fork, schedule'"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "', '.join(Event)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "def _create_file(path:Path, fname:str, contents):\n",
    "    if contents and not (path/fname).exists(): (path/fname).write_text(contents)\n",
    "\n",
    "def _replace(s:str, find, repl, i:int=0, suf:str=''):\n",
    "    return s.replace(find, textwrap.indent(repl, ' '*i)+suf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "def create_workflow_files(fname:str, workflow:str, build_script:str, prebuild:bool=False):\n",
    "    \"Create workflow and script files in suitable places in `github` folder\"\n",
    "    if not os.path.exists('.git'): return print('This does not appear to be the root of a git repo')\n",
    "    wf_path  = Path('.github/workflows')\n",
    "    scr_path = Path('.github/scripts')\n",
    "    wf_path .mkdir(parents=True, exist_ok=True)\n",
    "    scr_path.mkdir(parents=True, exist_ok=True)\n",
    "    _create_file(wf_path, f'{fname}.yml', workflow)\n",
    "    _create_file(scr_path, f'build-{fname}.py', build_script)\n",
    "    if prebuild: _create_file(scr_path, f'prebuild-{fname}.py', build_script)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "def fill_workflow_templates(name:str, event, run, context, script, opersys='ubuntu', prebuild=False):\n",
    "    \"Function to create a simple Ubuntu workflow that calls a Python `ghapi` script\"\n",
    "    c = wf_tmpl\n",
    "    if event=='workflow_dispatch:': event=''\n",
    "    needs = '    needs: [prebuild]' if prebuild else None\n",
    "    for find,repl,i in (('NAME',name,0), ('EVENT',event,2), ('RUN',run,8), ('CONTEXTS',context,8),\n",
    "                       ('OPERSYS',f'[{opersys}]',0), ('NEEDS',needs,0), ('PREBUILD',pre_tmpl if prebuild else '',2)):\n",
    "        c = _replace(c, f'${find}', str(repl), i)\n",
    "    create_workflow_files(name, c, script, prebuild=prebuild)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`event` is the event to trigger on. `run` is the shell lines to run before running the script, such as a `pip install` step. `context` are the env var context lines to include in the `env:` section of the workflow, normally created with `env_contexts`. `opersys` can be a string containing a comma-separated list of operating systems, e.g. `macos, ubuntu, windows`, which will be used to create a parallel matrix build.\n",
    "\n",
    "The `prebuild` bool tells `ghapi` to include a prebuild job, which contains the following workflow:\n",
    "\n",
    "```bash\n",
    "runs-on: ubuntu-latest\n",
    "outputs:\n",
    "  out: ${{ toJson(steps) }}\n",
    "steps:\n",
    "- uses: actions/checkout@v1\n",
    "- uses: actions/setup-python@v2\n",
    "  with: {python-version: '3.8'}\n",
    "- name: Create release\n",
    "  id: step1\n",
    "  env:\n",
    "    CONTEXT_GITHUB: ${{ toJson(github) }}\n",
    "  run: |\n",
    "    pip install -q ghapi\n",
    "    python .github/scripts/prebuild.py\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "def env_contexts(contexts):\n",
    "    \"Create a suitable `env:` line for a workflow to make a context available in the environment\"\n",
    "    contexts = uniqueify(['github'] + listify(contexts))\n",
    "    return \"\\n\".join(\"CONTEXT_\" + o.upper() + \": ${{ toJson(\" + o.lower() + \") }}\" for o in contexts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "def_pipinst = 'pip install -Uq ghapi'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "def create_workflow(name:str, event:Event, contexts:list=None, opersys='ubuntu', prebuild=False):\n",
    "    \"Function to create a simple Ubuntu workflow that calls a Python `ghapi` script\"\n",
    "    script = \"from fastcore.all import *\\nfrom ghapi import *\"\n",
    "    fill_workflow_templates(name, f'{event}:', def_pipinst, env_contexts(contexts),\n",
    "                            script=script, opersys=opersys, prebuild=prebuild)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_workflow('test', Event.release)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create a basic `ghapi` workflow, call `create_workflow`, passing in the event that you wish to respond to, and a name for your workflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "@call_parse\n",
    "def gh_create_workflow(\n",
    "    name:str,  # Name of the workflow file\n",
    "    event:str,  # Event to listen for\n",
    "    contexts:str=''  # Space-delimited extra contexts to include in `env` in addition to github\n",
    "):\n",
    "    \"Supports `gh-create-workflow`, a CLI wrapper for `create_workflow`.\"\n",
    "    create_workflow(name, Event[event], contexts.split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accessing contexts from Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The information from these variables are provided by `context_github`, `context_needs`, and so forth for each named context. These variables are `AttrDict` objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#26) ['token','job','ref','sha','repository','repository_owner','repositoryUrl','run_id','run_number','retention_days'...]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L(context_github)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'refs/heads/master'"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "context_github.ref"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you use our recommended workflow template, you will have this included in your prebuild step (if you have any):\n",
    "\n",
    "```bash\n",
    "outputs:\n",
    "  out: ${{ toJson(steps) }}\n",
    "```\n",
    "\n",
    "You can access this content as a dictionary like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'step1': {'outputs': {'tag': 'v0.79.0'},\n",
       "  'outcome': 'success',\n",
       "  'conclusion': 'success'}}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loads(nested_idx(context_needs, \"prebuild\", \"outputs\", \"out\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "_example_url = 'https://raw.githubusercontent.com/fastai/ghapi/master/examples/{}.json'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "def example_payload(event):\n",
    "    \"Get an example of a JSON payload for `event`\"\n",
    "    return dict2obj(urljson(_example_url.format(event)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Workflow helper functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "def github_token():\n",
    "    \"Get GitHub token from `GITHUB_TOKEN` env var if available, or from `github` context\"\n",
    "    return os.getenv('GITHUB_TOKEN', context_github.get('token', None))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "def actions_output(name, value):\n",
    "    \"Print the special GitHub Actions `::set-output` line for `name::value`\"\n",
    "    print(f\"::set-output name={name}::{value}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Details in the [GitHub Documentation for `set-output`](https://docs.github.com/actions/reference/workflow-commands-for-github-actions#setting-an-output-parameter)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "def actions_debug(message):\n",
    "    \"Print the special `::debug` line for `message`\"\n",
    "    print(f\"::debug::{message}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Details in the [GitHub Documentation for `debug`](https://docs.github.com/actions/reference/workflow-commands-for-github-actions#setting-a-debug-message). Note that you must create a secret named `ACTIONS_STEP_DEBUG` with the value true to see the debug messages set by this command in the log."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "def actions_warn(message, details=''):\n",
    "    \"Print the special `::warning` line for `message`\"\n",
    "    print(f\"::warning {details}::{message}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Details in the [GitHub Documentation for `warning`](https://docs.github.com/actions/reference/workflow-commands-for-github-actions#setting-a-warning-message). For the optional `details`, you can provide comma-delimited file, line, and column information, e.g.: `file=app.js,line=1,col=5`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "def actions_error(message, details=''):\n",
    "    \"Print the special `::error` line for `message`\"\n",
    "    print(f\"::error {details}::{message}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Details in the [GitHub Documentation for `error`](https://docs.github.com/actions/reference/workflow-commands-for-github-actions#setting-an-error-message). For the optional `details`, you can provide comma-delimited file, line, and column information, e.g.: `file=app.js,line=1,col=5`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "@contextmanager\n",
    "def actions_group(title):\n",
    "    \"Context manager to print the special `::group`/`::endgroup` lines for `title`\"\n",
    "    print(f\"::group::{title}\")\n",
    "    yield\n",
    "    print(f\"::endgroup::\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "::group::Adding numbers...\n",
      "2\n",
      "::endgroup::\n"
     ]
    }
   ],
   "source": [
    "with actions_group('Adding numbers...'): print(1+1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Details in the GitHub Documentation for [grouping log lines](https://docs.github.com/actions/reference/workflow-commands-for-github-actions#grouping-log-lines)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "def actions_mask(value):\n",
    "    \"Print the special `::add-mask` line for `value`\"\n",
    "    print(f\"::add-mask::{value}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Details in the [GitHub Documentation for `add-mask`](https://docs.github.com/actions/reference/workflow-commands-for-github-actions#masking-a-value-in-log)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#|export\n",
    "def set_git_user(api=None):\n",
    "    \"Set git user name/email to authenticated user (if `api`) or GitHub Actions bot (otherwise)\"\n",
    "    if api:\n",
    "        user  = api.users.get_authenticated().name\n",
    "        email = first(api.users.list_emails_for_authenticated(), attrgetter('primary')).email\n",
    "    else:\n",
    "        user  = 'github-actions[bot]'\n",
    "        email = 'github-actions[bot]@users.noreply.github.com'\n",
    "    run(f'git config --global user.email \"{email}\"')\n",
    "    run(f'git config --global user.name  \"{user}\"')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When pushing to git from a workflow, you'll need to set your username and email address. You can set them to the `GhApi` authenticated user's details by passing `api`. Otherwise, `github-actions[bot]` and `github-actions[bot]@users.noreply.github.com` will be used, which will make a push appear to be from \"GitHub Actions\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Export -"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Converted 00_core.ipynb.\n",
      "Converted 01_actions.ipynb.\n",
      "Converted 02_auth.ipynb.\n",
      "Converted 03_page.ipynb.\n",
      "Converted 04_event.ipynb.\n",
      "Converted 10_cli.ipynb.\n",
      "Converted 50_fullapi.ipynb.\n",
      "Converted 80_tutorial_actions.ipynb.\n",
      "Converted 90_build_lib.ipynb.\n",
      "Converted ghapi demo.ipynb.\n",
      "Converted index.ipynb.\n"
     ]
    }
   ],
   "source": [
    "#|hide\n",
    "from nbdev import nbdev_export\n",
    "nbdev_export()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}