Handling optionality of test cases that require additions to the description.md

[SaschaMann]

Problem

In #1674 the notion that every test case is optional was introduced. However, some optional test cases may require additions to the description.md file. There may also be situations where one will choose between two contradicting test cases of which both require different instructions.

Tracks could add these descriptions in the track-specific README inserts but these explanations are likely not track-specific and would benefit from being synced across repos.

Example

#1736 introduces a new test case for the rational-numbers exercise that introduces tests for the ordering of rational numbers. The ordering relation is not trivial and needs to be explained to students who are not familiar with rational numbers:

diff --git a/exercises/rational-numbers/description.md b/exercises/rational-numbers/description.md
index 55ebaeb4b..39f9ed475 100644
--- a/exercises/rational-numbers/description.md
+++ b/exercises/rational-numbers/description.md
@@ -18,9 +18,12 @@ Exponentiation of a rational number `r = a/b` to a real (floating-point) number
 
 Exponentiation of a real number `x` to a rational number `r = a/b` is `x^(a/b) = root(x^a, b)`, where `root(p, q)` is the `q`th root of `p`.
 
+Ordering of rational numbers: `a/b < c/d` if and only if `a * sign(b) * |d| < |b| * c * sign(d)`.
+
 Implement the following operations:
  - addition, subtraction, multiplication and division of two rational numbers,
  - absolute value, exponentiation of a given rational number to an integer power, exponentiation of a given rational number to a real (floating-point) power, exponentiation of a real number to a rational number.
+ - the ordering of rational numbers.
 
 Your implementation of rational numbers should always be reduced to lowest terms. For example, `4/4` should reduce to `1/1`, `30/60` should reduce to `1/2`, `12/8` should reduce to `3/2`, etc. To reduce a rational number `r = a/b`, divide `a` and `b` by the greatest common divisor (gcd) of `a` and `b`. So, for example, `gcd(12, 8) = 4`, so `r = 12/8` can be reduced to `(12/4)/(8/4) = 3/2`.

Tracks that choose to not implement this test would be left behind with an irrelevant stub in the description.

[SaschaMann]

Potential solution

This is the first potential solution that came to my mind. I don’t like it too much.

Introduce a new optional key to canonical-data.json that contains description.md inserts:

{
    "uuid": "bd1eab16-7603-4a77-ae4e-23c930fe39cd",
    "description": "Ordering",
    "description_inserts": [
        "Ordering of rational numbers: `a/b < c/d` if and only if `a * sign(b) * |d| < |b| * c * sign(d)`.\n",
        " - the ordering of rational numbers."
    ],
    "property": "<",
    "comments": ["..."]
}

configlet generate could insert these depending on tests.toml. The changes to description.md would then look something like this:

diff --git a/exercises/rational-numbers/description.md b/exercises/rational-numbers/description.md
index 55ebaeb4b..39f9ed475 100644
--- a/exercises/rational-numbers/description.md
+++ b/exercises/rational-numbers/description.md
@@ -18,9 +18,12 @@ Exponentiation of a rational number `r = a/b` to a real (floating-point) number
 
 Exponentiation of a real number `x` to a rational number `r = a/b` is `x^(a/b) = root(x^a, b)`, where `root(p, q)` is the `q`th root of `p`.
 
+<!-- bd1eab16-7603-4a77-ae4e-23c930fe39cd -->

 Implement the following operations:
  - addition, subtraction, multiplication and division of two rational numbers,
  - absolute value, exponentiation of a given rational number to an integer power, exponentiation of a given rational number to a real (floating-point) power, exponentiation of a real number to a rational number.
+<!-- bd1eab16-7603-4a77-ae4e-23c930fe39cd -->
 
 Your implementation of rational numbers should always be reduced to lowest terms. For example, `4/4` should reduce to `1/1`, `30/60` should reduce to `1/2`, `12/8` should reduce to `3/2`, etc. To reduce a rational number `r = a/b`, divide `a` and `b` by the greatest common divisor (gcd) of `a` and `b`. So, for example, `gcd(12, 8) = 4`, so `r = 12/8` can be reduced to `(12/4)/(8/4) = 3/2`.

We’d likely have to introduce UUIDs and description_inserts for groups of test cases as well because there doesn’t need to be one insert per addition test but instead one for all additions, e.g.

{
    "description": "Addition",
    "uuid": "de6b3f52-a3ea-4761-ae6a-d1e9b48c85ea",
    "description_inserts": [
        "The sum of two rational numbers `r₁ = a₁/b₁` and `r₂ = a₂/b₂` is `r₁ + r₂ = a₁/b₁ + a₂/b₂ = (a₁ * b₂ + a₂ * b₁) / (b₁ * b₂)`."
    ],
    "cases": ["..."]
}

Downsides

• This raises the question which tests are »truly« optional and which are key to the exercise again. This is an issue that lead to problems in the past and was meant to be fixed by making all tests optional. To demonstrate why this would come up, consider what rational-numbers may look like if it was added in the future. One would have to decide if only constructing a rational number would be a core test, with all operations being optional, or whether addition and multiplication are also core etc. Applying the same consistency that exists for test cases, we may end up with a description.md that looks like this:

  A rational number is defined as the quotient of two integers `a` and `b`, called the numerator and denominator, respectively, where `b != 0`.
  
  <!-- Absolute value --><!-- b1d45c79-870a-48e3-a4af-4b04c21f6c0a -->
  
  <!-- Addition --><!-- de6b3f52-a3ea-4761-ae6a-d1e9b48c85ea -->
  
  <!-- Subtraction --><!-- ceb12c67-a9f4-4042-ba0e-d7703338e013 -->
  
  <!-- Multiplication --><!-- a95fee15-04e6-4c1b-ad05-65a660227ecd -->
  
  [...]
  
  Implement the following operations:
  - <!-- Absolute value --><!-- b1d45c79-870a-48e3-a4af-4b04c21f6c0a -->
  - <!-- Addition --><!-- de6b3f52-a3ea-4761-ae6a-d1e9b48c85ea -->
  - <!-- Subtraction --><!-- ceb12c67-a9f4-4042-ba0e-d7703338e013 -->
  - <!-- Multiplication --><!-- a95fee15-04e6-4c1b-ad05-65a660227ecd -->

• The definition and test cases of an exercise are no longer cleanly separated.

• Editing description.md files will require mentally stitching together snippets rather than editing a single text.

Granted, this is an extreme example because most exercises are not made up of cleanly separated properties.

[iHiD]

How about if the test could be marked as a bonus test (a key with a description similar to what you're suggesting). We can then have a specific area for inserts into the README in a subtitled "Bonus" section? We could do nice things in the UI, like only show them when the other tests are completed.

Tracks can then choose whether to keep a test as a "bonus" test or core themselves when they implement it (by manually overriding the README if they don't want it to be bonus). Any new tests that require a description to be altered then either need strong maintainer-consensus, or they get marked as "bonus".

[SaschaMann]

How about if the test could be marked as a bonus test (a key with a description similar to what you're suggesting).

We had that before! #1518 #1677 #1492

AFAIR we got rid of it because this whole question if a test case is key to an exercise or not leads to endless debates, and one of the goals of the redesign was to minimise any potential for conflicts or debates. That's why all cases are optional and immutable. (@ErikSchierboom Please correct me if I'm wrong, I remember we discussed this in great detail before but the Slack paywall ate it.)

---

We could do nice things in the UI, like only show them when the other tests are completed.

It's besides the point of this issue but having proper support for bonus tasks in the UI would be quite nice regardless. Right now we kinda abuse hints.md for that in some exercises on the Julia track.

[ErikSchierboom]

AFAIR we got rid of it because this whole question if a test case is key to an exercise or not leads to endless debates, and one of the goals of the redesign was to minimise any potential for conflicts or debates.

This is correct. Things like this tended to result in endless (polarizing) discussions so I'd like to avoid this where possible. What if we would allow exercise-specific README templates? I think we currently support overriding the entire README, but do we allow templating in that document? If not, we could consider having something like a README.md.tpl file to go with an exercise and that template could then extend the basic README with the optional tests. It still depends on identifying which tests are optional though.

[ErikSchierboom]

I think the above mentioned solution is not ideal, as it mixes responsibilities a bit too much I feel (canonical data now contains exercise description text).
I do agree that there is value in being able to customize the description more than what we currently allow for.

[SaschaMann]

I think the above mentioned solution is not ideal, as it mixes responsibilities a bit too much I feel (canonical data now contains exercise description text).

We could come up with a syntax to include it in the description itself, too. Something like

A rational number is defined as the quotient of two integers `a` and `b`, called the numerator and denominator, respectively, where `b != 0`.

<!-- de6b3f52-a3ea-4761-ae6a-d1e9b48c85ea : The sum of two rational numbers `r₁ = a₁/b₁` and `r₂ = a₂/b₂` is `r₁ + r₂ = a₁/b₁ + a₂/b₂ = (a₁ * b₂ + a₂ * b₁) / (b₁ * b₂)`. -->

That works around 2/3 downsides mentioned above and only leaves the optionality question. Though as I said that may only apply to outlier exercises.

[SaschaMann]

Or in case of #1936:

[...]

Write code that shows:

- how many grains were on a given square, and
<!-- 6eb07385-3659-4b45-a6be-9dc474222750 : - the total number of grains on the chessboard after placing grains on every square. -->
<!-- d18dfa5d-cbcb-4b73-af95-048af063933f : - the total number of grains on the chessboard after placing grains on the first `n` squares. -->

Where d18dfa5d-cbcb-4b73-af95-048af063933f would be the UUID of the test group that contains the cases for calculating the total number of grains up to the first n squares.

[ErikSchierboom]

It's a possibility, but it would have some downsides:

• Harder to read for prob-specs maintainers
• configlet needs to do custom parsing

I'll have a think

[SaschaMann]

configlet needs to do custom parsing

Just to be clear, the syntax in my examples is meant to demonstrate the underlying idea. It could use a template engine or whatever works best.