How to Create a FAT Project That Runs a Maven Based TCK Using Surefire and Arquillian
Quickstart
Dynamically Download A TCK From GIT As Part of the TestRun
Java 2 Security
Excluding Tests
Debugging
Useful Environment Variables
Jakarta
- Copy project com.ibm.ws.microprofile.config_fat_tck from integration
- Replace all the references to 'config' with your own project's equivalent. If you have a look at the PR for the first copy/paste of this approach by someone else, at https://github.com/OpenLiberty/open-liberty/pull/1725, there is a comment in each place you would have to change for new TCK after copy/pasting the com.ibm.ws.microprofile.config_fat_tck directory under dev.
- Replace the publish/tckRunner folder with the top level folder from your tckRunner project
- If you add any jars into the test wars using an Arquillian ApplicationArchiveProcessor put these jars in the projects's top level build.gradle file and reference them as per https://github.com/OpenLiberty/open-liberty/pull/1400/files#diff-324e430492dcf6ce548918c8407a4e7f with a path like '../../../lib/hamcrest-all-1.3.jar'
- Edit the publish/tckRunner/tck/pom.xml and replace all the systemPath element contents that refer to a specific versions of jar files as these can vary from one version of Liberty to the next, for example: '/libertyGit/open-liberty/dev/build.image/wlp/lib/com.ibm.ws.microprofile.config_1.0.18.jar' is brittle and will break when version 19 comes along. Replace it with a symbolic equivalent. e.g.:
<systemPath>${/libertyGit/open-liberty/dev/build.image/wlp/lib/com.ibm.ws.microprofile.config_1.0.18.jar</systemPath>
...won't work when version "18" becomes "19" so replace it with the equivalent:
<systemPath>${com.ibm.ws.microprofile.config.1.0.}</systemPath>
The system uses regex but in the systemPath "." is literal and will match ".", "DOT" will become a regex "." and "STAR" becomes ".*" (as does "DOTSTAR"). Underscores match literally. Other regex chars that are not part of valid environment variables (^-[] etc.) are not supported.
String have ".jar" added automatically if necessary and are bookended with ".*" in order to act like a regex 'contains'.
See method jarPathInDir in MvnUtils.jar for what strings with 'match' jars from Liberty.
The test front end's call to MvnUtils.init(server) will create an environment variable like ${com.ibm.ws.microprofile.config} with the value of the actual file path "/libertyGit/open-liberty/dev/build.image/wlp/lib/com.ibm.ws.microprofile.config_1.0.18.jar" that can be used in the pom.xml. The prefix of the absolute path location comes from the Simplicity testcase LibertyServer object's server install location that is used in the test case. The pom.xml is scanned to find the list of symbolic systemPath environment variables that need binding to real versioned jar paths for that test's server.
If you look at the arquillian.xml file you will see that the arquillian server configuration is set up in a similar fashion using environment variables. These are all set from the Simplicity framework's LibertyServer that is used in the JUnit testcase front end.
When running a tckFat in your laptop you need to have maven isntalled and on the path and you should include IBM's artifactory in the set of places your local mvn will look at (an alternative is to get maven to put all the needed jars in your local cache maven repo).
Update your maven settings ~/.m2/settings.xml to add artifactory... example settings.xml
The TCK's TestNg tests all output report data in XML format. These raw XML files are copied into the same place and format that the Simplicity Tests' raw reports go to BEFORE the html report is created. Thus the final Simplicity HTML report will include all the TCK tests individually reported and TCK and LibertyFat tests can be freely mixed in the same FAT bucket if desired (for example testing non TCK issues like ranges) and will all appear in one HTML report directory and index.html summary in the usual places.
A build will be failed if any TCK test fails.
Arquillian does org.jboss.arquillian.protocol.servlet.runner.SecurityActions.getThreadContextClassLoader(SecurityActions.java:61) and some tests do things like System.getProperties(). Without appropriate Java 2 security permissions you will get an error when a build is done with Java2 security switched on.
A server.xml element can ge used to grant permissions. For example:
<javaPermission className="java.security.AllPermission" name="*" actions="*" />
The automated builds will test with Java 2 security if the following property is set:
global.java2.sec=true
When running locally, i.e.:
cd ..repo/dev
./gradlew <project>:buildandrun
For low level detailed logs have a look at:
${project}/build/libs/autoFVT/results/junit.html/index.html
e.g.:
file:///libertyGit/open-liberty/dev/com.ibm.ws.microprofile.config_fat_tck/build/libs/autoFVT/results/junit.html/index.html
and
${project}/build/libs/autoFVT/results/mvnOutput_TCK for the mvn stdout
and
${project}/build/libs/autoFVT/results/junit/TEST-org.eclipse.microprofile.config.tck.FATSuite.xml
For a Liberty personal build on the EBC. Go to the Downloads tab; FAT Logs; File Name: fatoutput-myProject.zip -> myProject/results/... e.g. fatoutput-com.ibm.ws.microprofile.faulttolerance_fat_tck.zip Unzip then see {results/mvnOutput_TCK, results/junit.html/index.html, results/junit/TESTS-TestSuites.xml}
There are about 4 places/processes that one can catch with Eclipse:
https://github.com/OpenLiberty/open-liberty/wiki/FAT-tests#debugging-a-fat
./gradlew com.ibm.ws.microprofile.config_fat_tck:buildandrun -Ddebug.framework
Editing the MvnUtils.java code to run mvnDebug instead of mvn
mvnDebug -DforkCount=0 test
mvn -Dmaven.surefire.debug="-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8014" test
arquillian.xml
<property name="javaVmArguments">-agentlib:jdwp=transport=dt_socket,address=8787,server=y,suspend=n</property>
Either run the server in debug mode and make sure you have.
<property name="allowConnectingToRunningServer">true</property>
in arquillian.xml
Unfortunately, I have not been able to find out how to debug the Containers deploy method.
runTests:
[echo] Base dir is: /libertyGit/open-liberty/dev/com.ibm.ws.microprofile.config_fat_tck/build/libs/autoFVT
[echo] Liberty dir is: ${liberty.dir}
[echo] libertyInstallPath is: ../../../../build.image/wlp
[echo] liberty.location is: ../../../../build.image/wlp
[echo] install.location is: /libertyGit/open-liberty/dev/com.ibm.ws.microprofile.config_fat_tck/build/libs/autoFVT/
Notes:
If you have a look at, for example,
file:///libertyGit/open-liberty/dev/com.ibm.ws.microprofile.faulttolerance_fat_tck/build/libs/autoFVT/results/tck/surefire-reports/index.html
or for a remote build
.../autoFVT/results/tck/surefire-reports/index.html#
and then click on the name of the xml file in the top left
microprofile-faulttolerance-TCK
Info
tck-suite.xml
Then the right hand pane will display all the expanded wildcards from the file, for example:
<suite name="microprofile-faulttolerance-TCK" verbose="2" configfailurepolicy="continue" >
<test name="microprofile-faulttolerance 1.0 TCK">
<packages>
<package name="org.eclipse.microprofile.fault.tolerance.tck.*"></package>
<package name="org.eclipse.microprofile.*"></package>
</packages>
</test>
</suite>
becomes
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite configfailurepolicy="continue" guice-stage="DEVELOPMENT" verbose="0" name="microprofile-faulttolerance-TCK">
<test name="microprofile-faulttolerance 1.0 TCK">
<packages>
<package name="org.eclipse.microprofile.fault.tolerance.tck.*"/>
<package name="org.eclipse.microprofile.*"/>
</packages>
<classes>
<class name="org.eclipse.microprofile.config.inject.ConfigProperty"/>
<class name="org.eclipse.microprofile.config.ConfigProvider"/>
<class name="org.eclipse.microprofile.config.Config"/>
<class name="org.eclipse.microprofile.config.spi.ConfigSource"/>
...
For deployment problems one often wishes to include the whole TCK package(s) but not include anything from a (set of) classes. If you specify individual classes, then you can include and exclude individual methods ( See http://testng.org/testng-1.0.dtd ) with:
<suite name="bulkhead-suite" verbose="2" configfailurepolicy="continue" >
<test name="bulkhead-suite">
<classes>
<class name="org.eclipse.microprofile.fault.tolerance.tck.bulkhead.BulkheadSyncRetryTest">
<methods>
<include name="testBulkheadRetryClassSynchronous55" />
But listing all classes is a lot of work and brittle.
There is very useful support for beanshell in the testng.xml files which is very rarely referenced but works. This can be used to 'parameterize' to some extent the testng.xml which (unlike the pom.xml and arquillian.xml) cannot embed environment variables.
<suite name="microprofile-config-1.1-tck" verbose="2"
configfailurepolicy="continue">
<test name="tck-package-org.eclipse.microprofile.config.tck">
<packages>
<method-selectors>
<method-selector>
<script language="beanshell">
<![CDATA[
!method.getDeclaringClass().getSimpleName().startsWith("CDIPlainInjectionTest") &&
!method.getDeclaringClass().getSimpleName().startsWith("ConfigProviderTest")
]]>
</script>
</method-selector>
</method-selectors>
<package name="org.eclipse.microprofile.config.tck.*"></package>
</packages>
</test>
</suite>
Note, if you use this to cherry pick particular methods such as:
method.getName().contains("testBooleanLookupProgrammatically") || method.getName().contains("testEnvironmentConfigSource")
then you are likely to get an Arquillian error,
No TestRunnerAdaptor found, @BeforeSuite has not been called java.lang.IllegalStateException: No TestRunnerAdaptor found, @BeforeSuite has not been called
the trick here is to add in:
|| method.getName().contains("arquillian")
so that the Arquillian utility methods, e.g. beforeTest and afterTest can be run.
One can also look at the OS:
<method-selectors>
<method-selector>
<script language="beanshell">
<![CDATA[
String os = System.getProperty("os.name");
if (os.contains("Windows")){
return !method.getName().startsWith("testEnvironmentConfigSource");
}else{
return true;
}
]]>
</script>
</method-selector>
</method-selectors>
You can control the 'Mode' by annotating the test method with
@Mode(TestMode.QUARANTINE) -- to not fail builds on TCK test faulure
@Mode(TestMode.FULL) -- for 'long running' tests (> 5 minutes)
@Mode(TestMode.LITE) -- for all builds
If you are pulling 'SNAPSHOT' TCK builds with 3rd parties committing tests before they are shown to pass on Libery consider setting such a TCK FAT bucket as @Mode(TestMode.QUARANTINE) so that Libery builds are not failed due to test instability.
See http://was.pok.ibm.com/xwiki/bin/view/Liberty/Test-FAT#HLitevsFullvsQuarantinemodes for more details.
You can prototype the TCK directly with a command line that looks like the one the java JUnit testcase generates. For example:
cd ...fat_tck/publish/tckRunner/tck
mvn clean test -Dwlp=/libertyGit/open-liberty/dev/build.image/wlp -Dtck_server=FATServer -Dtck_port=8010 -DtargetDirectory=/libertyGit/open-liberty/dev/com.ibm.ws.microprofile.faulttolerance_fat_tck/build/libs/autoFVT/results/tck -Dcom.ibm.websphere.org.eclipse.microprofile.faulttolerance=/libertyGit/open-liberty/dev/build.image/wlp/dev/api/stable/com.ibm.websphere.org.eclipse.microprofile.faulttolerance.1.0_1.0.18.jar -Dcom.ibm.ws.microprofile.faulttolerance=/libertyGit/open-liberty/dev/build.image/wlp/lib/com.ibm.ws.microprofile.faulttolerance_1.0.18.jar -Dcom.ibm.ws.microprofile.config=/libertyGit/open-liberty/dev/build.image/wlp/lib/com.ibm.ws.microprofile.config_1.0.18.jar -Dcom.ibm.ws.microprofile.config.cdi=/libertyGit/open-liberty/dev/build.image/wlp/lib/com.ibm.ws.microprofile.config.cdi_1.0.18.jar -Dcom.ibm.ws.microprofile.faulttolerance.cdi=/libertyGit/open-liberty/dev/build.image/wlp/lib/com.ibm.ws.microprofile.faulttolerance.cdi_1.0.18.jar -Dcom.ibm.websphere.org.eclipse.microprofile.config=/libertyGit/open-liberty/dev/build.image/wlp/dev/api/stable/com.ibm.websphere.org.eclipse.microprofile.config.1.1_1.2.18.jar -DsuiteXmlFile=tck-suite.xml
The PR https://github.com/OpenLiberty/open-liberty/pull/2818 creates a FAT bucket that firstly downloads a branch of the TCK from github before run
It can be run with
./gradlew -Dfat.test.mode=experimental com.ibm.ws.microprofile.faulttolerance_git_fat_tck:buildandrun
./gradlew -Dfat.test.mode=experimental -Dbranch=emily_ft com.ibm.ws.microprofile.faulttolerance_git_fat_tck:buildandrun
Note that you need to set the test mode to experimental. This is to protect from an external party from adding a test that does not pass to the release branch and then that test causing Liberty builds to fail.
The command line can take the following parameters:
-Dbranch=emily_ft // This is the branch that gets downloaded
-Dimpl.version= // This is poked into the regex that looks for Jars
// like "com.ibm.websphere.org.eclipse.microprofile.[impl.version]"
// to find com.ibm.websphere.org.eclipse.microprofile.faulttolerance.1.0_1.0.20.jar
-Dapi.version= // This can be used to override the tckRunner's pom.xml api dependancy version
-Dtck.version= // This can be used to override the tckRunner's pom.xml tck dependancy version
-DsuiteXmlFile= // This can be used for custom test suite files like a standard TCK runner.
For JakartaEE9 doc see https://github.com/OpenLiberty/liberty-arquillian/blob/master/liberty-managed/JakartaEE9_README.md
Currently due to a limitations in the FeatureDependencyProcessor that checks server.xml files we don't dynamically change the feature in server.xml.
The method to run a prototype or unreleased version of a TCK is as follows:
1.Get hold of the TCK by cloning the git repo that contains it. 2.Modify your tests to fit your requirements 3.Optionally change the version in -project-/tck/pom.xml (you might need to add a version element if the pom references a parent as by default versions can be omitted in that case) 4. mvn clean install to get the TCK test into your local repo. (mvn clean install | grep -i Installing is useful) 5. Go to the open libery FAT as TCK project edit -project-/publish/tckRunner/tck/pom.xml 6. Search for the equivalent of:
<dependency>
<groupId>org.eclipse.microprofile.config</groupId>
<artifactId>microprofile-config-tck</artifactId>
<version>${microprofile.config.version}</version>
</dependency>
and edit the version number to match what you did in step 3 above.
7.This will probably work if you have not changed the package names but you can check the tck-suite.xml file in -project-/publish/tckRunner/tck/ to check it will reference your new classes. 8.Run the FAT TCK as normal.