Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: introduce flags for line and form coverage thresholds #343

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

feat: introduce flags for line and form coverage thresholds #343

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
162112b
Introduce flags for line and form coverage thresholds
Feb 16, 2024
00fd2c7
add precondition
krikitti Mar 1, 2024
70445fe
add unit test for coverage-under? fn
Mar 1, 2024
3871a34
update readme with use case on different threshold flags
Mar 1, 2024
07ddc3a
formatting cloverage-test
Mar 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
11 changes: 6 additions & 5 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,18 +7,19 @@ provide a short summary description of the particular changes of the patch.
Include the issue number (#xxx) which will link back to the originating issue
in github. Commentary on the change should appear as a nested, unordered list. -->

## 1.2.5 (WIP)
## 1.2.5

Nothing yet!
- Features
- Option to set thresholds separately for line and form coverage

## 1.2.4

- Make validation errors in the project settings fail Cloverage runs
- Fixes [#331](https://github.com/cloverage/cloverage/issues/331)
- Upgrade various dependencies
- Fixes [218](https://github.com/cloverage/cloverage/issues/218)
- Fixes [319](https://github.com/cloverage/cloverage/issues/319)
- Fixes [334](https://github.com/cloverage/cloverage/issues/334)
- Fixes [218](https://github.com/cloverage/cloverage/issues/218)
- Fixes [319](https://github.com/cloverage/cloverage/issues/319)
- Fixes [334](https://github.com/cloverage/cloverage/issues/334)

Also some more development niceties, like improvements to CI (including a
proper CircleCI test matrix).
Expand Down
61 changes: 32 additions & 29 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,37 +50,40 @@ project. Command line arguments can still be used and will be merged
in. List options are merged by concatenation, for other options the
project value is used.

Note about the three different threshold flags: sometimes, line coverage is high while form coverage isn't as high, so setting a high line coverage requirement can help maintaining an existing standard.

Available options and command-line arguments:
```
Project Switches Default Desc
------- -------- ------- ----
:output -o, --output target/coverage Output directory.
:text? --no-text, --text false Produce a text report.
:html? --no-html, --html true Produce an HTML report.
:emma-xml? --no-emma-xml, --emma-xml false Produce an EMMA XML report. [emma.sourceforge.net]
:lcov? --no-lcov, --lcov false Produce a lcov/gcov report.
:codecov? --no-codecov, --codecov false Generate a JSON report for Codecov.io
:coveralls? --no-coveralls, --coveralls false Send a JSON report to Coveralls if on a CI server
:junit? --no-junit, --junit false Output test results as junit xml file. Supported in :clojure.test runner
:raw? --no-raw, --raw false Output raw coverage data (for debugging).
:summary? --no-summary, --summary true Prints a summary
:fail-threshold --fail-threshold 0 Sets the percentage threshold at which cloverage will abort the build. Default: 0%
:low-watermark --low-watermark 50 Sets the low watermark percentage (valid values 0..100). Default: 50%
:high-watermark --high-watermark 80 Sets the high watermark percentage (valid values 0..100). Default: 80%
:debug? -d, --no-debug, --debug false Output debugging information to stdout.
:runner -r, --runner :clojure.test Specify which test runner to use. Built-in runners are `clojure.test`, `midje` and `eftest`.
:runner-opts (not allowed as a cli arg) {} Configure specified runner with any options map.
:nop? --no-nop, --nop false Instrument with noops.
:ns-regex -n, --ns-regex [] Regex for instrumented namespaces (can be repeated).
:ns-exclude-regex -e, --ns-exclude-regex [] Regex for namespaces not to be instrumented (can be repeated).
:exclude-call --exclude-call [] Name of fn/macro whose call sites are not to be instrumented (can be repeated).
:test-ns-regex -t, --test-ns-regex [] Regex for test namespaces (can be repeated).
:src-ns-path -p, --src-ns-path [] Path (string) to directory containing source code namespaces (can be repeated).
:test-ns-path -s, --test-ns-path [] Path (string) to directory containing test namespaces (can be repeated).
:extra-test-ns -x, --extra-test-ns [] Additional test namespace (string) to add (can be repeated).
:custom-report -c, --custom-report Load and run a custom report writer. Should be a namespaced symbol. The function is passed
project-options args-map output-directory forms
:help? -h, --no-help, --help false Show help.
Project Switches Default Desc
------- -------- ------- ----
:output -o, --output target/coverage Output directory.
:text? --no-text, --text false Produce a text report.
:html? --no-html, --html true Produce an HTML report.
:emma-xml? --no-emma-xml, --emma-xml false Produce an EMMA XML report. [emma.sourceforge.net]
:lcov? --no-lcov, --lcov false Produce a lcov/gcov report.
:codecov? --no-codecov, --codecov false Generate a JSON report for Codecov.io
:coveralls? --no-coveralls, --coveralls false Send a JSON report to Coveralls if on a CI server
:junit? --no-junit, --junit false Output test results as junit xml file. Supported in :clojure.test runner
:raw? --no-raw, --raw false Output raw coverage data (for debugging).
:summary? --no-summary, --summary true Prints a summary
:fail-threshold --fail-threshold 0 Sets the percentage threshold for both line and form coverage at which cloverage will abort the build. Default: 0%
:line-fail-threshold --line-fail-threshold 0 Sets the percentage threshold for line coverage at which cloverage will abort the build. Ignored if --fail-threshold is non-zero. Default: 0%
:form-fail-threshold --form-fail-threshold 0 Sets the percentage threshold for form coverage at which cloverage will abort the build. Ignored if --fail-threshold is non-zero. Default: 0%
:low-watermark --low-watermark 50 Sets the low watermark percentage (valid values 0..100). Default: 50%
:high-watermark --high-watermark 80 Sets the high watermark percentage (valid values 0..100). Default: 80%
:debug? -d, --no-debug, --debug false Output debugging information to stdout.
:runner -r, --runner :clojure.test Specify which test runner to use. Built-in runners are `clojure.test`, `midje` and `eftest`.
:runner-opts (not allowed as a cli arg) {} Configure specified runner with any options map.
:nop? --no-nop, --nop false Instrument with noops.
:ns-regex -n, --ns-regex [] Regex for instrumented namespaces (can be repeated).
:ns-exclude-regex -e, --ns-exclude-regex [] Regex for namespaces not to be instrumented (can be repeated).
:exclude-call --exclude-call [] Name of fn/macro whose call sites are not to be instrumented (can be repeated).
:test-ns-regex -t, --test-ns-regex [] Regex for test namespaces (can be repeated).
:src-ns-path -p, --src-ns-path [] Path (string) to directory containing source code namespaces (can be repeated).
:test-ns-path -s, --test-ns-path [] Path (string) to directory containing test namespaces (can be repeated).
:extra-test-ns -x, --extra-test-ns [] Additional test namespace (string) to add (can be repeated).
:custom-report -c, --custom-report Load and run a custom report writer. Should be a namespaced symbol. The function is passed project-options args-map output-directory forms
:help? -h, --no-help, --help false Show help.
```

### mvn
Expand Down
66 changes: 39 additions & 27 deletions cloverage/src/cloverage/args.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,32 +19,34 @@
(every? symbol? coll))

(def valid
{:text? boolean?
:html? boolean?
:raw? boolean?
:emma-xml? boolean?
:junit? boolean?
:lcov? boolean?
:codecov? boolean?
:coveralls? boolean?
:summary? boolean?
:colorize? boolean?
:fail-threshold integer?
:low-watermark integer?
:high-watermark integer?
:debug? boolean?
:nop? boolean?
:extra-test-ns regexes-or-strings?
:help? boolean?
:ns-regex regexes-or-strings?
:test-ns-regex regexes-or-strings?
:ns-exclude-regex regexes-or-strings?
:exclude-call symbols?
:src-ns-path regexes-or-strings?
:runner keyword?
:runner-opts map?
:test-ns-path regexes-or-strings?
:custom-report symbol?})
{:text? boolean?
:html? boolean?
:raw? boolean?
:emma-xml? boolean?
:junit? boolean?
:lcov? boolean?
:codecov? boolean?
:coveralls? boolean?
:summary? boolean?
:colorize? boolean?
:fail-threshold integer?
:line-fail-threshold integer?
:form-fail-threshold integer?
:low-watermark integer?
:high-watermark integer?
:debug? boolean?
:nop? boolean?
:extra-test-ns regexes-or-strings?
:help? boolean?
:ns-regex regexes-or-strings?
:test-ns-regex regexes-or-strings?
:ns-exclude-regex regexes-or-strings?
:exclude-call symbols?
:src-ns-path regexes-or-strings?
:runner keyword?
:runner-opts map?
:test-ns-path regexes-or-strings?
:custom-report symbol?})

(defn- fn-sym [^Object f]
(let [[_ f-ns f-n] (re-matches #"(.*)\$(.*?)(__[0-9]+)?" (.. f getClass getName))]
Expand Down Expand Up @@ -140,7 +142,17 @@
["--[no-]colorize"
"Adds ANSI color to the summary" :default true]
["--fail-threshold"
"Sets the percentage threshold at which cloverage will abort the build. Default: 0%"
"Sets the percentage threshold for both line and form coverage at which cloverage will abort the build. Default: 0%"
:default 0
:parse-fn #(Integer/parseInt %)]
["--line-fail-threshold"
"Sets the percentage threshold for line coverage at which cloverage will abort the build.
Ignored if --fail-threshold is non-zero. Default: 0%"
:default 0
:parse-fn #(Integer/parseInt %)]
["--form-fail-threshold"
"Sets the percentage threshold for form coverage at which cloverage will abort the build.
Ignored if --fail-threshold is non-zero. Default: 0%"
:default 0
:parse-fn #(Integer/parseInt %)]
["--low-watermark"
Expand Down
31 changes: 22 additions & 9 deletions cloverage/src/cloverage/coverage.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -288,16 +288,26 @@
:output output
:forms forms})))

(defn- coverage-under? [forms failure-threshold]
(when (pos? failure-threshold)
(let [pct-covered (apply min (vals (rep/total-stats forms)))
failed? (< pct-covered failure-threshold)]
(when failed?
(println "Failing build as coverage is below threshold of" failure-threshold "%"))
failed?)))
(defn- coverage-under? [forms failure-threshold line-failure-threshold form-failure-threshold]
krikitti marked this conversation as resolved.
Show resolved Hide resolved
{:pre [failure-threshold line-failure-threshold form-failure-threshold]}
(let [{:keys [percent-lines-covered percent-forms-covered]} (rep/total-stats forms)]
krikitti marked this conversation as resolved.
Show resolved Hide resolved
(if (pos? failure-threshold)
(let [pct-covered (min percent-lines-covered percent-forms-covered)
failed? (< pct-covered failure-threshold)]
(when failed?
(println "Failing build as coverage is below threshold of" failure-threshold "%"))
failed?)
(when (or (pos? line-failure-threshold) (pos? form-failure-threshold))
(let [line-failed? (< percent-lines-covered line-failure-threshold)
form-failed? (< percent-forms-covered form-failure-threshold)]
(when line-failed?
(println "Failing build as line coverage is below threshold of" line-failure-threshold "%"))
(when form-failed?
(println "Failing build as form coverage is below threshold of" form-failure-threshold "%"))
(or line-failed? form-failed?))))))

(defn run-main
[[{:keys [debug? fail-threshold help?], :as opts} add-nses help] project-opts]
[[{:keys [debug? fail-threshold line-fail-threshold form-fail-threshold help?], :as opts} add-nses help] project-opts]
(binding [*ns* (find-ns 'cloverage.coverage)
debug/*debug* debug?]
(if help?
Expand All @@ -308,7 +318,10 @@
exit-code (cond
(not test-result) -1
(> num-errors 128) -2
(coverage-under? forms fail-threshold) -3
(coverage-under? forms
fail-threshold
line-fail-threshold
form-fail-threshold) -3
:else num-errors)]
(report-results opts project-opts forms)
(if *exit-after-test*
Expand Down
24 changes: 24 additions & 0 deletions cloverage/test/cloverage/coverage_test.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
[clojure.test :as t]
[cloverage.coverage :as cov]
[cloverage.instrument :as inst]
[cloverage.report :as rep]
[cloverage.source :as src]
[riddley.walk :as rw]
[clojure.java.io :as io]))
Expand Down Expand Up @@ -449,3 +450,26 @@
require (constantly nil)]
(t/is (= {:errors 3}
((cov/runner-fn (merge opts runner-opts)) []))))))))

(t/deftest test-coverage-under?
krikitti marked this conversation as resolved.
Show resolved Hide resolved
(t/testing "check that the different thresholds work properly"
(let [coverage-under? #'cov/coverage-under?
forms {}]
(with-redefs [rep/total-stats (constantly {:percent-lines-covered 97
:percent-forms-covered 75})]
(t/are [result fail-threshold line-fail-threshold form-fail-threshold]
(= result (coverage-under? forms fail-threshold line-fail-threshold form-fail-threshold))
; Non-zero fail-threshold
true 100 0 0 ; line and form coverage both under fail-threshold
true 90 0 0 ; line coverage is under fail-threshold, form coverage is above fail-threshold
false 70 0 0 ; line and form coverage both above fail-threshold
false 70 100 0 ; line-fail-threshold ignored because fail-threshold is non-zero
false 70 0 100 ; form-fail-threshold ignored because fail-threshold is non-zero
false 70 100 100 ; line- and form-fail-threshold ignored because fail-threshold is non-zero

; fail-threshold is 0
nil 0 0 0
false 0 90 70 ; line coverage is above line-fail-threshold and form coverage is above form-fail-threshold
true 0 100 70 ; line coverage is under line-fail-threshold, form coverage is above form-fail-threshold
true 0 90 90 ; line coverage is above line-fail-threshold, form coverage is under form-fail-threshold
true 0 100 100))))) ; line coverage is under line-fail-threshold and form coverage is under form-fail-threshold
2 changes: 2 additions & 0 deletions cloverage/tests.edn
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@
:lcov? false,
:high-watermark 80,
:fail-threshold 0,
:line-fail-threshold 0,
:form-fail-threshold 0,
:output "target/coverage",
:low-watermark 50,
:ns-regex [],
Expand Down