Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add test tags as cmake label #2690

Open
wants to merge 4 commits into
base: devel
Choose a base branch
from
+25 −9
Open

Add test tags as cmake label #2690

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0645765
Add test tags as cmake label
uyha May 21, 2023
c541a31
Complete adding tags as labels to tests using the JSON reporter
uyha Oct 31, 2023
41ecdb6
Remove custom escape logic
uyha Nov 15, 2023
3cf405b
Quote variable dereference properly
uyha Nov 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
34 changes: 25 additions & 9 deletions extras/CatchAddTests.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ function(catch_discover_tests_impl)
endif()

execute_process(
COMMAND ${_TEST_EXECUTOR} "${_TEST_EXECUTABLE}" ${spec} --list-tests --verbosity quiet
COMMAND ${_TEST_EXECUTOR} "${_TEST_EXECUTABLE}" ${spec} --list-tests --reporter JSON
OUTPUT_VARIABLE output
RESULT_VARIABLE result
WORKING_DIRECTORY "${_TEST_WORKING_DIR}"
Expand All @@ -74,12 +74,6 @@ function(catch_discover_tests_impl)
)
endif()

# Make sure to escape ; (semicolons) in test names first, because
# that'd break the foreach loop for "Parse output" later and create
# wrongly splitted and thus failing test cases (false positives)
string(REPLACE ";" "\;" output "${output}")
string(REPLACE "\n" ";" output "${output}")

# Prepare reporter
if(reporter)
set(reporter_arg "--reporter ${reporter}")
Expand Down Expand Up @@ -121,9 +115,15 @@ function(catch_discover_tests_impl)
endforeach()
endif()

string(JSON listings GET "${output}" "listings")
string(JSON tests GET "${listings}" "tests")
string(JSON tests_length LENGTH "${tests}")
# CMake foreach loop is inclusive
math(EXPR test_end "${tests_length} - 1")
# Parse output
foreach(line ${output})
set(test "${line}")
foreach(index RANGE "${test_end}")
string(JSON test_spec GET "${tests}" "${index}")
string(JSON test GET "${test_spec}" "name")
# Escape characters in test case names that would be parsed by Catch2
# Note that the \ escaping must happen FIRST! Do not change the order.
set(test_name "${test}")
Expand Down Expand Up @@ -161,6 +161,22 @@ function(catch_discover_tests_impl)
endif()

list(APPEND tests "${prefix}${test}${suffix}")

string(JSON tags GET "${test_spec}" "tags")
string(JSON tags_length LENGTH "${tags}")

if("${tags_length}" GREATER 0)
math(EXPR tag_end "${tags_length} - 1")

foreach(tag_index RANGE "${tag_end}")
string(JSON tag GET "${tags}" "${tag_index}")
add_command(set_tests_properties
"${prefix}${test}${suffix}"
PROPERTIES
LABELS "${tags}"
)
endforeach()
Comment on lines +171 to +178
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure why, but this does not do what it is supposed to. The resulting ctest script looks like this:

add_test( [==[And now a test case with weird tags.]==] /home/xarn/builds/Catch2/uyha-json-reporter/tests/ctest-registration-test/tests [==[And now a test case with weird tags.]==]  )
set_tests_properties( [==[And now a test case with weird tags.]==] PROPERTIES WORKING_DIRECTORY /home/xarn/builds/Catch2/uyha-json-reporter/tests/ctest-registration-test)
set_tests_properties( [==[And now a test case with weird tags.]==] PROPERTIES LABELS [==[[ "foo,bar", "tl;dr", "tl;dw" ]]==])
set_tests_properties( [==[And now a test case with weird tags.]==] PROPERTIES LABELS [==[[ "foo,bar", "tl;dr", "tl;dw" ]]==])
set_tests_properties( [==[And now a test case with weird tags.]==] PROPERTIES LABELS [==[[ "foo,bar", "tl;dr", "tl;dw" ]]==])

Which has the obvious issue of setting the same set of labels multiple times, but also it should set each tag as its own label, rather than the amalgamation of all tags as one big label.

Also check #1658 for proper handling of labels -> they need to be appended, rather than just set.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason the amalgamation is there is LABELS "${tags}", rather than LABELS "${tag}".

endif()
endforeach()

# Create a list of all discovered tests, which users may use to e.g. set
Expand Down