some tests failing in the multithread mode

[xambroz]

Hello,
these test jobs were failing when running in multithread mode

$ ctest -j4
	 39 - API/OVAL/unittests/test_object_component_type.sh (Failed)
	 44 - API/OVAL/unittests/test_state_check_existence.sh (Failed)
	 62 - API/XCCDF/applicability/test_remediate_fix_notapplicable.sh (Failed)
	 63 - API/XCCDF/applicability/test_remediate_fix_processing.sh (Failed)
	 77 - API/XCCDF/unittests/all_python.sh (Failed)
	 87 - API/XCCDF/unittests/test_fix_instance.sh (Failed)
	100 - API/XCCDF/unittests/test_xccdf_check_processing_complex_priority.sh (Failed)
	111 - API/XCCDF/unittests/test_deriving_xccdf_result_from_oval.sh (Failed)
	114 - API/XCCDF/unittests/test_deriving_xccdf_result_from_oval_multicheck.sh (Failed)
	115 - API/XCCDF/unittests/test_multiple_oval_files_with_same_basename.sh (Failed)
	132 - API/XCCDF/unittests/test_remediation_simple.sh (Failed)
	136 - API/XCCDF/unittests/test_remediation_bad_fix.sh (Failed)
	141 - API/XCCDF/unittests/test_remediation_subs_value_without_selector.sh (Failed)
	142 - API/XCCDF/unittests/test_remediation_subs_value_title.sh (Failed)
	143 - API/XCCDF/unittests/test_remediation_amp_escaping.sh (Failed)
	144 - API/XCCDF/unittests/test_remediation_xml_comments.sh (Failed)
	145 - API/XCCDF/unittests/test_remediation_cdata.sh (Failed)
	146 - API/XCCDF/unittests/test_remediation_subs_unresolved.sh (Failed)
	149 - API/XCCDF/unittests/test_remediate_simple.sh (Failed)
	166 - API/XCCDF/variable_instance/all.sh (Failed)
	192 - probes/file/test_probes_file.sh (Failed)
	193 - probes/file/test_probes_file_behaviour.sh (Failed)
	198 - probes/filehash58/rhbz1959570_segfault.sh (Failed)
	261 - probes/yamlfilecontent/test_probes_yamlfilecontent_types.sh (Failed)

Running the same without multithreading fixes the issue:

$ ctest --rerun-failed --output-on-failure -E sce-community-content_rule_services_obsolete-disable_nfs_exports
Test project /home/mambroz/rpmbuild/BUILD/openscap-1.3.9/redhat-linux-build
      Start  62: API/XCCDF/applicability/test_remediate_fix_notapplicable.sh
 1/11 Test  #62: API/XCCDF/applicability/test_remediate_fix_notapplicable.sh ...   Passed    3.67 sec
      Start  63: API/XCCDF/applicability/test_remediate_fix_processing.sh
 2/11 Test  #63: API/XCCDF/applicability/test_remediate_fix_processing.sh ......   Passed    4.98 sec
      Start  77: API/XCCDF/unittests/all_python.sh
 3/11 Test  #77: API/XCCDF/unittests/all_python.sh .............................   Passed    3.10 sec
      Start  87: API/XCCDF/unittests/test_fix_instance.sh
 4/11 Test  #87: API/XCCDF/unittests/test_fix_instance.sh ......................   Passed    1.75 sec
      Start 111: API/XCCDF/unittests/test_deriving_xccdf_result_from_oval.sh
 5/11 Test #111: API/XCCDF/unittests/test_deriving_xccdf_result_from_oval.sh ...   Passed    1.52 sec
      Start 143: API/XCCDF/unittests/test_remediation_amp_escaping.sh
 6/11 Test #143: API/XCCDF/unittests/test_remediation_amp_escaping.sh ..........   Passed    2.54 sec
      Start 144: API/XCCDF/unittests/test_remediation_xml_comments.sh
 7/11 Test #144: API/XCCDF/unittests/test_remediation_xml_comments.sh ..........   Passed    2.56 sec
      Start 145: API/XCCDF/unittests/test_remediation_cdata.sh
 8/11 Test #145: API/XCCDF/unittests/test_remediation_cdata.sh .................   Passed    2.67 sec
      Start 146: API/XCCDF/unittests/test_remediation_subs_unresolved.sh
 9/11 Test #146: API/XCCDF/unittests/test_remediation_subs_unresolved.sh .......   Passed    2.74 sec
      Start 149: API/XCCDF/unittests/test_remediate_simple.sh
10/11 Test #149: API/XCCDF/unittests/test_remediate_simple.sh ..................   Passed    8.39 sec
      Start 166: API/XCCDF/variable_instance/all.sh
11/11 Test #166: API/XCCDF/variable_instance/all.sh ............................   Passed   28.84 sec

100% tests passed, 0 tests failed out of 11

Total Test time (real) =  62.80 sec

[evgenyz]

That's a known issue of the test suite. It does not support concurrent invocation.

[havardAasen]

@evgenyz

That's a known issue of the test suite. It does not support concurrent invocation.

Do you know if this is documented anywhere? I looked in Run the tests section in docs/developer/developer.adoc but couldn't find anything.