Skip to content

Commit

Permalink
fixed and clarified coverage section
Browse files Browse the repository at this point in the history
  • Loading branch information
@ttimbers
ttimbers committed Mar 14, 2024
1 parent fb2998f commit e539c45
Show file tree
Hide file tree
Showing 13 changed files with 909 additions and 2,095 deletions.
207 changes: 155 additions & 52 deletions docs/_sources/materials/lectures/06-intro-to-testing-code.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1101,26 +1101,23 @@
"\n",
"\n",
"```\n",
"my_function <- function(x) {\n",
" # Branch 1\n",
" if (condition_met) {\n",
" y = function_a(x)\n",
" z = function_b(y)\n",
" }\n",
" # Branch 2\n",
" else { \n",
" y = function_b(x)\n",
" z = function_c(y)\n",
" }\n",
" z\n",
"}\n",
"```\n",
"\n",
"\n",
"```\n",
"test_that(\"ponies are actually unicorns\", {\n",
" expect_equal(my_function(\"pony\"), (\"Actually a unicorn\"))\n",
"})\n",
"1 | my_function <- function(x) {\n",
"2 | # Branch 1\n",
"3 | if (x == \"pony\") {\n",
"4 | y = function_a(x)\n",
"5 | z = function_b(y)\n",
"6 | }\n",
"7 | # Branch 2\n",
"8 | else { \n",
"9 | y = function_b(x)\n",
"10 | z = function_c(y)\n",
"11 | }\n",
"12 | z\n",
"13 | }\n",
"14 |\n",
"15 | test_that(\"ponies are actually unicorns\", {\n",
"16 | expect_equal(my_function(\"pony\"), (\"Actually a unicorn\"))\n",
"17 | })\n",
"```\n",
"\n",
"*Note: function definitions are not counted as lines when calculating coverage*"
Expand All @@ -1132,6 +1129,10 @@
"source": [
"#### Using branch as the coverage metric:\n",
"\n",
"Branch 1 (lines 3-5) is covered by the test, because `if (x == \"pony\")` evaluates to true. \n",
"Therefore we have one branch covered, and one branch uncovered (lines 8-10),\n",
"resulting in 50% branch coverage when we plug these numbers into our formula.\n",
"\n",
"$Coverage = \\frac{covered}{(covered + uncovered)} * 100$\n",
"\n",
"$Coverage = \\frac{1}{(1 + 1)} * 100$\n",
Expand All @@ -1140,11 +1141,16 @@
"\n",
"#### Using line as the coverage metric:\n",
"\n",
"Lines 3-5 and 12 are executed, and lines 8-10 are not (function definitions are not typically counted in line coverage).\n",
"There fore we have 4 lines covered, and 3 lines uncovered, resulting in 57% line coverage when we plug these numbers into our formula.\n",
"\n",
"$Coverage = \\frac{covered}{(covered + uncovered)} * 100$\n",
"\n",
"$Coverage = \\frac{4}{(4 + 4)} * 100$\n",
"$Coverage = \\frac{4}{(4 + 3)} * 100$\n",
"\n",
"$Coverage = 50\\%$"
"$Coverage = 57\\%$\n",
"\n",
"In this case, both metrics give us relatively similar estimates of code coverage."
]
},
{
Expand All @@ -1156,33 +1162,31 @@
"Let's alter our function and re-calculate line and branch coverage:\n",
"\n",
"```\n",
"my_function <- function(x) {\n",
" # Branch 1\n",
" if (condition_met) {\n",
" y = function_a(x)\n",
" z = function_b(y)\n",
" print(z)\n",
" print(\"some important message\")\n",
" print(\"another important message\")\n",
" print(\"a less important message\")\n",
" print(\"just makin' stuff up here...\")\n",
" print(\"out of things to say...\")\n",
" print(\"how creative can I be...\")\n",
" print(\"I guess not very...\")\n",
" }\n",
" # Branch 2\n",
" else { \n",
" y = function_b(x)\n",
" z = function_c(y)\n",
" }\n",
" z\n",
"}\n",
"```\n",
"\n",
"```\n",
"test_that(\"ponies are actually unicorns\", {\n",
" expect_equal(my_function(\"pony\"), (\"Actually a unicorn\"))\n",
"})\n",
"1 | my_function <- function(x) {\n",
"2 | # Branch 1\n",
"3 | if (x == \"pony\") {\n",
"4 | y = function_a(x)\n",
"5 | z = function_b(y)\n",
"6 | print(z)\n",
"7 | print(\"some important message\")\n",
"8 | print(\"another important message\")\n",
"9 | print(\"a less important message\")\n",
"10 | print(\"just makin' stuff up here...\")\n",
"11 | print(\"out of things to say...\")\n",
"12 | print(\"how creative can I be...\")\n",
"13 | print(\"I guess not very...\")\n",
"14 | }\n",
"15 | # Branch 2\n",
"16 | else { \n",
"17 | y = function_b(x)\n",
"18 | z = function_c(y)\n",
"19 | }\n",
"20 | z\n",
"21 | }\n",
"22 |\n",
"23 | test_that(\"ponies are actually unicorns\", {\n",
"24 | expect_equal(my_function(\"pony\"), (\"Actually a unicorn\"))\n",
"25 | })\n",
"```"
]
},
Expand All @@ -1192,6 +1196,10 @@
"source": [
"#### Using branch as the coverage metric:\n",
"\n",
"Branch 1 (lines 3-13) is covered by the test, because `if (x == \"pony\")` evaluates to true. \n",
"Therefore we have one branch covered, and one branch uncovered (lines 16-18),\n",
"resulting in 50% branch coverage when we plug these numbers into our formula.\n",
"\n",
"$Coverage = \\frac{covered}{(covered + uncovered)} * 100$\n",
"\n",
"$Coverage = \\frac{1}{(1 + 1)} * 100$\n",
Expand All @@ -1200,13 +1208,16 @@
"\n",
"#### Using line as the coverage metric:\n",
"\n",
"Lines 3-13 and 20 are executed, and lines 16-18 are not (function definitions are not typically counted in line coverage).\n",
"There fore we have 12 lines covered, and 3 lines uncovered, resulting in 57% line coverage when we plug these numbers into our formula.\n",
"\n",
"$Coverage = \\frac{covered}{(covered + uncovered)} * 100$\n",
"\n",
"$Coverage = \\frac{12}{(12 + 4)} * 100$\n",
"$Coverage = \\frac{12}{(12 + 3)} * 100$\n",
"\n",
"$Coverage = 75\\%$\n",
"$Coverage = 80\\%$\n",
"\n",
"🤯\n",
"In this case, the different metrics give us very different estimates of code coverage! 🤯\n",
"\n",
"### Take home message:\n",
"\n",
Expand Down Expand Up @@ -1236,6 +1247,98 @@
"> Why has this not been implemented? It has been in an now unsupported package (see [here](https://github.com/MangoTheCat/testCoverage)), but its implementation was too complicated for others to understand. Automating the calculation of branch coverage is non-trivial, and this is a perfect demonstration of that."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Calculating coverage in Python\n",
"\n",
"We use the plugin tool [`pytest-cov`](https://github.com/pytest-dev/pytest-cov) to do this. \n",
"\n",
"Install as a package via conda:\n",
"```\n",
"conda install pytest-cov\n",
"```\n",
"\n",
"### Calculating coverage in Python\n",
"\n",
"To calculate line coverage and print it to the terminal:\n",
"```\n",
"pytest --cov=<directory> \n",
"```\n",
"\n",
"To calculate line coverage and print it to the terminal:\n",
"```\n",
"pytest --cov-branch --cov=<directory>\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### How does `coverage` in Python actually count line coverage?\n",
"\n",
"- the output from `poetry run pytest --cov=src` gives a table that looks like this:\n",
"\n",
"```\n",
"---------- coverage: platform darwin, python 3.7.6-final-0 -----------\n",
"Name Stmts Miss Cover\n",
"-----------------------------------------\n",
"big_abs/big_abs.py 8 2 75%\n",
"-----------------------------------------\n",
"TOTAL 9 2 78%\n",
"```\n",
"\n",
"In the column labelled as \"Stmts\", coverage is calculating all possible line jumps that could have been executed (these line jumps are sometimes called arcs). This is essentially covered + uncovered lines of code.\n",
"\n",
"> Note - this leads coverage to count two statements on one line that are separated by a \";\" (e.g., print(\"hello\"); print(\"there\")) as one statement, as well as calculating a single statement that is spread across two lines as one statement. \n",
"\n",
"In the column labelled as \"Miss\", this is the number of line jumps not executed by the tests. \n",
"So our covered lines of code is \"Stmts\" - \"Miss\".\n",
"\n",
"The coverage percentage in this scenario is calculated by:\n",
"$$Coverage = \\frac{(Stmts - Miss)}{Stmts}$$\n",
"$$Coverage = \\frac{8 - 2}{8} * 100 = 75\\%$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### How does `coverage` in Python actually branch coverage?\n",
"\n",
"- the output from `poetry run pytest --cov-branch --cov=src` gives a table that looks like this:\n",
"\n",
"```\n",
"---------- coverage: platform darwin, python 3.7.6-final-0 -----------\n",
"Name Stmts Miss Branch BrPart Cover\n",
"-------------------------------------------------------\n",
"big_abs/big_abs.py 8 2 6 3 64%\n",
"-------------------------------------------------------\n",
"TOTAL 9 2 6 3 67%\n",
"```\n",
"\n",
"In the column labelled as \"Branch\", coverage is actually counting the number of possible jumps from branch points.\n",
"This is essentially covered + uncovered branches of code.\n",
"\n",
"> Note: because coverage is using line jumps to count branches, each `if` inherently has an `else` even if its not explicitly written in the code.\n",
"\n",
"In the column labelled as \"BrPart\", this is the number of of possible jumps from branch points executed by the tests.\n",
"This is essentially our covered branches of code.\n",
"\n",
"The branch coverage percentage in this tool is calculated by:\n",
"\n",
"$$Coverage = \\frac{(Stmts\\:executed + BrPart)}{(Stmts + Branch)}$$\n",
"\n",
"$$Coverage = \\frac{((Stmts - Miss) + BrPart)}{(Stmts + Branch)}$$\n",
"\n",
"> Note: You can see this formula actually includes both line and branch coverage in this calculation.\n",
"\n",
"So for `big_abs/big_abs.py` 64% was calculated from:\n",
"$$Coverage = \\frac{((8 - 2) + 3)}{(8 + 6)} * 100 = 64\\%$$"
]
},
{
"cell_type": "markdown",
"metadata": {
Expand Down

0 comments on commit e539c45

Please sign in to comment.