Proposed meta-check API

[jackfirth]

As mentioned in #14, it's very difficult to test custom RackUnit checks. This issue proposes some additional checks for RackUnit to improve the situation

The checks

• (check-fail (lambda () (check-equal? 1 2))) - asserts that the given thunk raises a single check failure
• (check-fail/info (make-actual-info 1) (lambda () (check-equal? 1 2))) - asserts that the given thunk raises a single check failure containing the given check info
• (check-fail/message #rx"foo" (lambda () (check-equal? 1 2))) - asserts that the given thunk raises a single check failure whose fail-check message matches the given regexp (a predicate could also be provided, to match the behavior of check-exn)

These checks would need to ensure the checks to be tested don't increment the test failure counter, as well as ensuring they don't print their typical output. This will likely require some modification of define-check to allow selectively disabling the test failure counter.

Alternatives considered

• Keyword arguments in check-fail for asserting the info and message. This might be nicer to read, but then one check expression is asserting multiple things. This conflicts with the typical philosophy of "one check = one assertion". Plus, keyword arguments aren't currently supported in custom checks (see Support keyword, rest, and default arguments in custom checks #17).
• Taking expressions instead of thunks. Existing checks like check-exn take thunks, and define-check doesn't allow delaying evaluation.
• A check-pass form. There's no point; just use the check directly.
• A check for making assertions about the entire info stack instead of a single info value, or for checking an info value matches a particular predicate. The info stack is intended to be read by humans, not programmatically manipulated, so its structure is not something that should be rigorously enforced.

[bennn]

Why not just one check-fail function?
(check-fail check-predicate thunk)

• if check-predicate is a regular expression, match against the fail-check message
• if check-predicate is a procedure, call it with the check failure?

[jackfirth]

It seems awkward for cases where I just want to assert that a check fails and don't want to do anything with infos or the failure message. What about if the /info and /message variants were a single function (check-fail* check-predicate thunk) that did what you describe? Then it's still short and easy to read tests for simple failure assertions.

It's cumbersome to assert a particular info is added with a predicate, so I'd still like something like check-fail/info.

[bennn]

I'd still prefer just having (check-fail pred thunk) because (1) I like that API, and wouldn't mind (check-fail values thunk) (2) I like the symmetry with check-exn (3) I think it will be useful to support arbitrary predicates.

Maybe (3) is not such a big deal? Suppose you added /info and /message and check-fail*. Do you have any important uses now for check-fail* with a predicate?

(I've never actually written a define-check, or thought hard about testing one)

[AlexKnauth]

(Re: symmetry with check-exn) Using check-exn can be cumbersome too. I can imagine for example check-fail/info being useful over check-fail* for two reasons:

(1) A check-info value is shorter and makes less noise than a predicate. It's more readable in the source code.

(2) A check-info value is transparent and can be incredibly useful to see in the failure message. A predicate can specify arbitrary things, but it doesn't give any useful information beyond #<procedure:...th/to/file.rkt:3:0> returned #false. A check-info value is more readable in the failure message.

[samth]

Just a suggestion: the /info and /message variants seem like they'd be better accomplished with a keyword argument, but I haven't followed the discussion super-closely so that may not work for other reasons.

[jackfirth]

Proposal v2

• (check-fail <predicate/regexp> <check-thunk>) - provides the least surprise for folks used to check-exn.
• (check-fail* <check-thunk>) - unlike exception-throwing-thunks, it's actually useful to check that a check fails without using a predicate or caring about the message
• (check-fail/info <info-expr> <check-thunk>) - a shortcut over check-fail with better info and failure information, differentiating between the check failure having the right info key with the wrong value and the check failure not having the info key at all

Note that check-info values aren't transparent, they're opaque. That should probably be changed.

[bennn]

What about (check-fail (or/c regexp? check-info? predicate?) thunk?), and then remove the check-fail/info case?

(For this to work, (check-info? (string-info "")) should return #true. I'm not sure if there are other issues.)

[jackfirth]

I dislike that the same check can do so many different things depending on the type of argument, but it's weird to have the regexp? case included while having a separate check for the check-info? case. If we're going to be consistent with check-exn we might as well go all the way and do it that way @bennn.

I don't think I understand what you mean about (check-info? (string-info "")). Wouldn't someone put the (string-info "") value in a check-info struct first by using make-check-info?

[bennn]

I don't think I understand what you mean about (check-info? (string-info "")). Wouldn't someone put the (string-info "") value in a check-info struct first by using make-check-info?

I'm the one who didn't understand --- I thought it'd be okay to write (check-fail... (string-info "") <thunk>).

Do what you think is best about check-fail/info, this is just my 2c