dataProvider and assertEquals throws unhandled exception in some cases

[N3XT0R]

Q	A
PHPUnit version	6.5.14
PHP version	PHP 7.0.33-0ubuntu0.16.04.2 (cli) ( NTS )
Installation Method	Composer

Hello Guys,
i have a strange problem within a test-class.

At first to explain how my testcases will be loaded:

i have a bootstrap file, where i have initialized a PSR-0 autoloader for loading the test classes.
Also i have extended the phpunit TestCase class for testing my own framework. At my parent-class i initialize a configuration class, where defines for different cases will be done, they are required to load classes through the application, for that i write tests.

All in all works fine, but when using a dataProvider with a class that requires another class i have exceptions that will not be handled by phpunit.

PHPUnit\Framework\Exception: a:4:{s:10:"testResult";N;s:13:"numAssertions";i:1;s:6:"result";O:28:"PHPUnit\Framework\TestResult":30:{s:9:"*passed";a:0:{}s:9:"*errors";a:0:{}s:11:"*failures";a:1:{i:0;O:29:"PHPUnit\Framework\TestFailure":3:{s:39:"PHPUnit\Framework\TestFailuretestName";s:88:"UserValidationRuleHelperTest::testIsHrNumberRequired with data set "7" ('swe', 16, true)";s:13:"*failedTest";N;s:18:"*thrownException";O:44:"PHPUnit\Framework\ExpectationFailedException":7:{s:20:"*comparisonFailure";O:46:"SebastianBergmann\Comparator\ComparisonFailure":13:{s:11:"*expected";b:1;s:9:"*actual";b:0;s:19:"*expectedAsString";s:0:"";s:17:"*actualAsString";s:0:"";s:12:"*identical";N;s:10:"*message";s:50:"Failed asserting that false matches expected true.";s:17:"Exceptionstring";s:0:"";s:7:"*code";i:0;s:7:"*file";s:111:"/data/srv/xx/www/htdocs/projectFolder/intern/tests/vendor/sebastian/comparator/src/ScalarComparator.php";s:7:"*line";i:77;s:16:"Exceptiontrace";a:11:{i:0;a:6:{s:4:"file";s:118:"/data/srv/xx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/Constraint/IsEqual.php";s:4:"line";i:129;s:8:"function";s:12:"assertEquals";s:5:"class";s:45:"SebastianBergmann\Comparator\ScalarComparator";s:4:"type";s:2:"->";s:4:"args";a:5:{i:0;b:1;i:1;b:0;i:2;d:0;i:3;b:0;i:4;b:0;}}i:1;a:6:{s:4:"file";s:106:"/data/srv/xx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/Assert.php";s:4:"line";i:2116;s:8:"function";s:8:"evaluate";s:5:"class";s:36:"PHPUnit\Framework\Constraint\IsEqual";s:4:"type";s:2:"->";s:4:"args";a:2:{i:0;b:0;i:1;s:30:"generalCountry swe is required";}}i:2;a:6:{s:4:"file";s:106:"/data/srv/xx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/Assert.php";s:4:"line";i:541;s:8:"function";s:10:"assertThat";s:5:"class";s:24:"PHPUnit\Framework\Assert";s:4:"type";s:2:"::";s:4:"args";a:3:{i:0;b:0;i:1;O:36:"PHPUnit\Framework\Constraint\IsEqual":6:{s:8:"*value";b:1;s:8:"*delta";d:0;s:11:"*maxDepth";i:10;s:15:"*canonicalize";b:0;s:13:"*ignoreCase";b:0;s:11:"*exporter";O:35:"SebastianBergmann\Exporter\Exporter":0:{}}i:2;s:30:"generalCountry swe is required";}}i:3;a:6:{s:4:"file";s:123:"/data/srv/xx/www/htdocs/projectFolder/intern/tests/FrameworkTest/ApplicationHelper/UserValidationRuleHelperTest.php";s:4:"line";i:301;s:8:"function";s:12:"assertEquals";s:5:"class";s:24:"PHPUnit\Framework\Assert";s:4:"type";s:2:"::";s:4:"args";a:3:{i:0;b:1;i:1;b:0;i:2;s:30:"generalCountry swe is required";}}i:4;a:4:{s:8:"function";s:22:"testIsHrNumberRequired";s:5:"class";s:28:"UserValidationRuleHelperTest";s:4:"type";s:2:"->";s:4:"args";a:3:{i:0;s:3:"swe";i:1;i:16;i:2;b:1;}}i:5;a:6:{s:4:"file";s:108:"/data/srv/xx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/TestCase.php";s:4:"line";i:1071;s:8:"function";s:10:"invokeArgs";s:5:"class";s:16:"ReflectionMethod";s:4:"type";s:2:"->";s:4:"args";a:2:{i:0;O:28:"UserValidationRuleHelperTest":49:{s:9:"*helper";O:24:"UserValidationRuleHelper":5:{s:16:"*regexPatterns";a:9:{s:13:"germanZipCode";s:7:"^\d{5}$";s:11:"phoneNumber";s:17:"^[ 0-9+-\\(\\)]+$";s:9:"faxNumber";s:17:"^[ 0-9+-\\(\\)]+$";s:17:"mobilePhoneNumber";s:17:"^[ 0-9+-\\(\\)]+$";s:8:"userName";s:6:"\w{6,}";s:11:"houseNumber";s:62:"([\d]+[a-zA-Z]{0,1}|[\d]+[a-zA-Z]{0,1}-[\d]+[a-zA-Z]{0,1}){1}$";s:18:"germanDateDDMMYYYY";s:58:"(0[1-9]|[12][0-9]|3[01])[\.](0[1-9]|1[012])[\.](19|20)\d\d";s:14:"germanTimeHHMM";s:35:"(0[0-9]|1[0-9]|2[0-3])(:[0-5][0-9])";s:28:"germanTimeNewsletterDispatch";s:25:"(09|1[0-8]):(00|15|30|45)";}s:23:"*mailAddressValidator";N;s:13:"*validation";a:0:{}s:17:"*dataTypeMaster";N;s:9:"*object";N;}s:9:"*tables";N;s:13:"*testDbName";s:20:"au_projectFolder_test";s:11:"*auConfig";O:22:"Mock_AuConfig_8e738787":3:{s:50:"Mock_AuConfig_8e738787__phpunit_invocationMocker";N;s:48:"Mock_AuConfig_8e738787__phpunit_originalObject";N;s:46:"Mock_AuConfig_8e738787__phpunit_configurable";a:1:{i:0;s:16:"initcustomconfig";}}s:22:"*initializeFramework";b:1;s:27:"*disableConfigConstructor";b:0;s:13:"*isCiServer";b:0;s:16:"*backupGlobals";N;s:25:"*backupGlobalsBlacklist";a:0:{}s:25:"*backupStaticAttributes";N;s:34:"*backupStaticAttributesBlacklist";a:0:{}s:27:"*runTestInSeparateProcess";N;s:53:"PHPUnit\Framework\TestCaserunClassInSeparateProcess";N;s:22:"*preserveGlobalState";b:1;s:39:"PHPUnit\Framework\TestCaseinIsolation";b:1;s:32:"PHPUnit\Framework\TestCasedata";a:3:{i:0;s:3:"swe";i:1;i:16;i:2;b:1;}s:36:"PHPUnit\Framework\TestCasedataName";s:1:"7";s:43:"PHPUnit\Framework\TestCaseuseErrorHandler";N;s:45:"PHPUnit\Framework\TestCaseexpectedException";N;s:52:"PHPUnit\Framework\TestCaseexpectedExceptionMessage";N;s:58:"PHPUnit\Framework\TestCaseexpectedExceptionMessageRegExp";N;s:49:"PHPUnit\Framework\TestCaseexpectedExceptionCode";N;s:32:"PHPUnit\Framework\TestCasename";s:22:"testIsHrNumberRequired";s:40:"PHPUnit\Framework\TestCasedependencies";a:0:{}s:43:"PHPUnit\Framework\TestCasedependencyInput";a:0:{}s:39:"PHPUnit\Framework\TestCaseiniSettings";a:0:{}s:34:"PHPUnit\Framework\TestCaselocale";a:0:{}s:39:"PHPUnit\Framework\TestCasemockObjects";a:0:{}s:47:"PHPUnit\Framework\TestCasemockObjectGenerator";N;s:34:"PHPUnit\Framework\TestCasestatus";i:3;s:41:"PHPUnit\Framework\TestCasestatusMessage";s:81:"generalCountry swe is required
Failed asserting that false matches expected true.";s:41:"PHPUnit\Framework\TestCasenumAssertions";i:1;s:34:"PHPUnit\Framework\TestCaseresult";N;s:38:"PHPUnit\Framework\TestCasetestResult";N;s:34:"PHPUnit\Framework\TestCaseoutput";s:0:"";s:47:"PHPUnit\Framework\TestCaseoutputExpectedRegex";N;s:48:"PHPUnit\Framework\TestCaseoutputExpectedString";N;s:42:"PHPUnit\Framework\TestCaseoutputCallback";b:0;s:49:"PHPUnit\Framework\TestCaseoutputBufferingActive";b:0;s:48:"PHPUnit\Framework\TestCaseoutputBufferingLevel";i:0;s:36:"PHPUnit\Framework\TestCasesnapshot";N;s:35:"PHPUnit\Framework\TestCaseprophet";N;s:61:"PHPUnit\Framework\TestCasebeStrictAboutChangesToGlobalState";b:0;s:75:"PHPUnit\Framework\TestCaseregisterMockObjectsFromTestArgumentsRecursively";b:0;s:36:"PHPUnit\Framework\TestCasewarnings";a:0:{}s:34:"PHPUnit\Framework\TestCasegroups";a:0:{}s:52:"PHPUnit\Framework\TestCasedoesNotPerformAssertions";b:0;s:45:"PHPUnit\Framework\TestCasecustomComparators";a:0:{}s:17:"*databaseTester";N;}i:1;a:3:{i:0;s:3:"swe";i:1;i:16;i:2;b:1;}}}i:6;a:6:{s:4:"file";s:108:"/data/srv/xx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/TestCase.php";s:4:"line";i:939;s:8:"function";s:7:"runTest";s:5:"class";s:26:"PHPUnit\Framework\TestCase";s:4:"type";s:2:"->";s:4:"args";a:0:{}}i:7;a:6:{s:4:"file";s:110:"/data/srv/xx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/TestResult.php";s:4:"line";i:698;s:8:"function";s:7:"runBare";s:5:"class";s:26:"PHPUnit\Framework\TestCase";s:4:"type";s:2:"->";s:4:"args";a:0:{}}i:8;a:6:{s:4:"file";s:108:"/data/srv/xx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/TestCase.php";s:4:"line";i:894;s:8:"function";s:3:"run";s:5:"class";s:28:"PHPUnit\Framework\TestResult";s:4:"type";s:2:"->";s:4:"args";a:1:{i:0;r:86;}}i:9;a:6:{s:4:"file";s:1:"-";s:4:"line";i:384;s:8:"function";s:3:"run";s:5:"class";s:26:"PHPUnit\Framework\TestCase";s:4:"type";s:2:"->";s:4:"args";a:1:{i:0;r:4;}}i:10;a:4:{s:4:"file";s:1:"-";s:4:"line";i:526;s:8:"function";s:27:"__phpunit_run_isolated_test";s:4:"args";a:0:{}}}s:19:"Exceptionprevious";N;s:14:"xdebug_message";s:2193:"
SebastianBergmann\Comparator\ComparisonFailure: Failed asserting that false matches expected true. in /data/srv/x/www/htdocs/x/intern/tests/vendor/sebastian/comparator/src/ScalarComparator.php on line 77

Call Stack:
    0.0015     532256   1. {main}() -:0
    0.0202    2331440   2. __phpunit_run_isolated_test() -:526
    0.0212    2426528   3. PHPUnit\Framework\TestCase->run(???) -:384
    0.0226    2588920   4. PHPUnit\Framework\TestResult->run(???) /data/srv/x/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/TestCase.php:894
    0.0231    2611248   5. PHPUnit\Framework\TestCase->runBare() /data/srv/x/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/TestResult.php:698
    0.1302    6503432   6. PHPUnit\Framework\TestCase->runTest() /data/srv/x/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/TestCase.php:939
    0.1302    6504048   7. ReflectionMethod->invokeArgs(???, ???) /data/srv/xxxx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/TestCase.php:1071
    0.1302    6504096   8. UserValidationRuleHelperTest->testIsHrNumberRequired(???, ???, ???) /data/srv/xxxx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/TestCase.php:1071
    0.1324    6545216   9. PHPUnit\Framework\Assert::assertEquals(???, ???, ???, ???, ???, ???, ???) /data/srv/xxxx/www/htdocs/projectFolder/intern/tests/FrameworkTest/ApplicationHelper/UserValidationRuleHelperTest.php:301
    0.1333    6611312  10. PHPUnit\Framework\Assert::assertThat(???, ???, ???) /data/srv/xxxx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/Assert.php:541
    0.1333    6611312  11. PHPUnit\Framework\Constraint\IsEqual->evaluate(???, ???, ???) /data/srv/xxxx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/Assert.php:2116
    0.1358    6793768  12. SebastianBergmann\Comparator\ScalarComparator->assertEquals(???, ???, ???, ???, ???) /data/srv/xxxx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/Constraint/IsEqual.php:129
";}s:20:"*serializableTrace";a:10:{i:0;a:5:{s:4:"file";s:106:"/data/srv/xxxx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/Assert.php";s:4:"line";i:2116;s:8:"function";s:8:"evaluate";s:5:"class";s:36:"PHPUnit\Framework\Constraint\IsEqual";s:4:"type";s:2:"->";}i:1;a:5:{s:4:"file";s:106:"/data/srv/xxxx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/Assert.php";s:4:"line";i:541;s:8:"function";s:10:"assertThat";s:5:"class";s:24:"PHPUnit\Framework\Assert";s:4:"type";s:2:"::";}i:2;a:5:{s:4:"file";s:123:"/data/srv/xxxx/www/htdocs/projectFolder/intern/tests/FrameworkTest/ApplicationHelper/UserValidationRuleHelperTest.php";s:4:"line";i:301;s:8:"function";s:12:"assertEquals";s:5:"class";s:24:"PHPUnit\Framework\Assert";s:4:"type";s:2:"::";}i:3;a:3:{s:8:"function";s:22:"testIsHrNumberRequired";s:5:"class";s:28:"UserValidationRuleHelperTest";s:4:"type";s:2:"->";}i:4;a:5:{s:4:"file";s:108:"/data/srv/xxxx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/TestCase.php";s:4:"line";i:1071;s:8:"function";s:10:"invokeArgs";s:5:"class";s:16:"ReflectionMethod";s:4:"type";s:2:"->";}i:5;a:5:{s:4:"file";s:108:"/data/srv/xxxx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/TestCase.php";s:4:"line";i:939;s:8:"function";s:7:"runTest";s:5:"class";s:26:"PHPUnit\Framework\TestCase";s:4:"type";s:2:"->";}i:6;a:5:{s:4:"file";s:110:"/data/srv/xxxx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/TestResult.php";s:4:"line";i:698;s:8:"function";s:7:"runBare";s:5:"class";s:26:"PHPUnit\Framework\TestCase";s:4:"type";s:2:"->";}i:7;a:5:{s:4:"file";s:108:"/data/srv/xxxx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/TestCase.php";s:4:"line";i:894;s:8:"function";s:3:"run";s:5:"class";s:28:"PHPUnit\Framework\TestResult";s:4:"type";s:2:"->";}i:8;a:5:{s:4:"file";s:1:"-";s:4:"line";i:384;s:8:"function";s:3:"run";s:5:"class";s:26:"PHPUnit\Framework\TestCase";s:4:"type";s:2:"->";}i:9;a:3:{s:4:"file";s:1:"-";s:4:"line";i:526;s:8:"function";s:27:"__phpunit_run_isolated_test";}}s:10:"*message";s:81:"generalCountry swe is required
Failed asserting that false matches expected true.";s:7:"*code";i:0;s:7:"*file";s:118:"/data/srv/xxxx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/Constraint/IsEqual.php";s:7:"*line";i:136;s:14:"xdebug_message";s:1002:"
PHPUnit\Framework\ExpectationFailedException: generalCountry swe is required
Failed asserting that false matches expected true. in /data/srv/xxxx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/Constraint/IsEqual.php on line 136

Caused by
ErrorException: Use of undefined constant CLASS_PATH - assumed 'CLASS_PATH' in /data/srv/xxx/www/htdocs/projectFolder/php/class/applicationHelper/UserValidationRuleHelper.php:3
Stack trace:
#0 /data/srv/xxx/www/htdocs/projectFolder/php/class/applicationHelper/UserValidationRuleHelper.php(3): PHPUnit\Util\PHP\AbstractPhpProcess->PHPUnit\Util\PHP\{closure}(8, 'Use of undefine...', '/data/srv/alpha...', 3, Array)
#1 /data/srv/xxx/www/htdocs/projectFolder/intern/tests/bootstrap.php(44): require_once('/data/srv/alpha...')
#2 [internal function]: bootstrap->autoload('UserValidationR...')
#3 [internal function]: spl_autoload_call('UserValidationR...')
#4 /data/srv/xxx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Util/PHP/AbstractPhpProcess.php(301): unserialize('a:4:{s:10:"test...')
#5 /data/srv/xxx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Util/PHP/AbstractPhpProcess.php(205): PHPUnit\Util\PHP\AbstractPhpProcess->processChildResult(Object(UserValidationRuleHelperTest), Object(PHPUnit\Framework\TestResult), 'a:4:{s:10:"test...', '')
#6 /data/srv/xxx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/TestCase.php(892): PHPUnit\Util\PHP\AbstractPhpProcess->runTestJob('<?php\nuse PHPUn...', Object(UserValidationRuleHelperTest), Object(PHPUnit\Framework\TestResult))
#7 /data/srv/xxx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/TestSuite.php(755): PHPUnit\Framework\TestCase->run(Object(PHPUnit\Framework\TestResult))
#8 /data/srv/xxx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/Framework/TestSuite.php(755): PHPUnit\Framework\TestSuite->run(Object(PHPUnit\Framework\TestResult))
#9 /data/srv/xxx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/TextUI/TestRunner.php(545): PHPUnit\Framework\TestSuite->run(Object(PHPUnit\Framework\TestResult))
#10 /data/srv/xxx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/TextUI/Command.php(195): PHPUnit\TextUI\TestRunner->doRun(Object(PHPUnit\Framework\TestSuite), Array, true)
#11 /data/srv/xxx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/src/TextUI/Command.php(148): PHPUnit\TextUI\Command->run(Array, true)
#12 /data/srv/xxx/www/htdocs/projectFolder/intern/tests/vendor/phpunit/phpunit/phpunit(53): PHPUnit\TextUI\Command::main()
#13 {main}

i had debugged that shit for a while without any useful result.

public function hrNumberRequirementsProvider(): array {
		return [
			//Hauptland, countryId, expected Result
			['deu', 0, false], //kein Pflichtfeld
			['deu', 1, false], //Kein Pflichtfeld
			['deu', 13, true], //Pflichtfeld - 13 = Polen
			['deu', 16, false], // Kein Pflichtfeld - 16 = Schweiz
			['deu', 11, false], // Kein Pflichtfeld - 11 = Norwegen
			['swe', 31, false], // Kein Pflichtfeld - 31 = Schweden
			['swe', 13, true], //Pflichtfeld - 13 = Polen
			['swe', 16, true], //Kein Pflichtfeld - 16 = Schweiz
		];
	}

	/**
	 * @param string $generalCountry
	 * @param int $countryId
	 * @param bool $expectedResult
	 * @throws
	 * @dataProvider hrNumberRequirementsProvider
	 */
	public function testIsHrNumberRequired(string $generalCountry, int $countryId, bool $expectedResult) {
		$result = $this->helper->isHrNumberRequired($generalCountry, $countryId);
		if (defined('CLASS_PATH')) { //<-- this line i just had added for debugging reasons. 
			$this->assertEquals($expectedResult, $result, 'generalCountry '.$generalCountry.' is '.(($expectedResult === true) ? 'required' : 'not required'));
		}
	}

do anyone know a solutation for this problem? Or could it be a bug ?

[epdenouden]

i had debugged that shit for a while without any useful result.

[epdenouden]

Since the older version 6 is no longer officially supported @sebastianbergmann will most likely close this issue, bug or not, as there will be no more patches the older version. Unless of course it also appears in >=7.

@N3XT0R myself I no longer use PHPunit version 6 as I'm working on new features including upgraded data providers which might help you with your problem, should you choose to adopt v8 later this year. None of this helps you right now, I know, and I understand your frustration with the non-obvious sequencing of data providers.

At first glance:

1. the error shown is caused by an exception re: the syntax for using the global constant \CLASS_PATH; if I were to debug this for a client, I would make sure I to understand why my code even ends up where (or when) it's not available. Are includes out of sequence? Is something missing from the autoloader after all?
2. the data providers are executed in a static context long before the tests are run, even if you extend TestCase to facilitate testing your own application framework

If you want to see this order of events: add some dummy logging or output statements to your extended TestCase and the data providers. You'll immediately see from the order the events are logged that current data providers behave in a way that is not immediately obvious.

Hope this helps you a step further and do not hesitate to comment if you have more questions.

[epdenouden]

@N3XT0R closing this as we haven't heard back, so hopefully you have found and fixed the test issues.

Or maybe the current Euro heatwave has melted all Pflichtfelder and you're having a nice großes Radler im Stadtpark.