TwxUnit is an Apache 2.0-licensed unit testing framework for PTC ThingWorx platform, compatible with JUnit semantics and existing tooling for JUnit 2+. It is the first such solution that we are aware of, and as such it is actively developed. It allows you to do this:
// -- QueryTestSuite.TestWithFilters()
// Tests that QueryUtility.Query() returns exactly three rows
// The rows are added in me.Before() and deleted in me.After()
var things = Things["QueryUtility"].Query({ filters: { tag: "testdata" } });
assertNotNull(things, "Query returned null");
assertEquals(3, things.rows.count, "Count is not 3");or this:
// -- PermissionsTestSuite.TestSystemRepository()
// Tests that SystemRepository is visible to all users, but only System can list files
assertHasNoPermissions("Users", function() {
// Everything here is executed under an anonymous member of Users group
assertNotNull(Things["SystemRepository"], "All users should see SystemRepository");
Things["SystemRepository"].GetFileListingWithLinks(); // Expecting an error here
}, "Non-admin users can get the list of files");
assertHasPermissions("System", function() {
// Everything here is executed under System user
Things["SystemRepository"].GetFileListingWithLinks(); // Expecting NO error here
}, "System user can't get the list of files");or even this:
// -- ImporterTestSuite.TestPurging()
// Mocks SystemRepository to test data purging
runWithMocks(function() {
// All references to SystemRepository here (including nested calls) will be replaced
// by MockSystemRepository thing, which defines a dummy set of services
Things["MockSystemRepository"].InitializeWithTestData();
Things["DataPurger"].DeleteAllFilesFromSystemRepository();
assertEquals(
Things["MockSystemRepository"].GetFiles().rows.length, 0,
"Some files were not deleted");
}, {"SystemRepository": "MockSystemRepository"});(see more examples below)
It is originally developed and actively maintained by Vilia as part of our effort towards open industrialization of ThingWorx development.
Lack of static typing in ThingWorx provides great productivity benefits for small projects, but becomes somewhat of an issue as soon as the code base grows beyond some limit. Unit testing is essential to addressing this problem. Unfortunately PTC hasn't included any unit testing facility into the core platform (yet). This project is here to try filling this gap.
- Simplify learning curve by replicating JUnit 4 semantics and familiar naming convention
- Allow executing tests entirely from within ThingWorx, without doing HTTP calls, and thus avoid storing any credentials outside of the platform.
- (Re-)use existing JUnit tooling for executing tests from outside of ThingWorx (e.g. from a CI pipeline)
- Avoid complex trickery, be reasonably lightweight, predictable and simple for an average ThingWorx developer to understand inner workings
- Allow executing specific tests in the context of arbitrary ThingWorx users
- Small and light codebase, trivial to install and remove, no external dependencies
TwxUnit was tested against ThingWorx 7.4, 8.4, 8.5, 9.0 and 9.3 although it should work with other versions just as well.
On Java side (optional) it requires Java 1.6 and JUnit 2, 3, 4 or 5 with junit-vintage-engine
on the classpath. We compile and test TwxUnit against JUnit 4.12, and there are no
external runtime dependencies.
TwxUnit can be executed either directly from ThingWorx, or as a JUnit4 test suite.
In the latter case the custom runner will do all necessary remote introspection for
you. Be aware that this scenario is not fully supported yet, so the only way to run
test suites today is via TwxUnit thing in ThingWorx Composer (see below).
Check out this project or simply download the ZIP file with extension package (the latest version is twxunit-ext-2.4.1.zip)
In ThingWorx Composer go to Import/Export > IMPORT > Extension, choose the ZIP file and click Import. That's all, you can start using TwxUnit now (see examples below).
Here is the complete list of all entities, which make up TwxUnit extension:
AssertFunctionsScript function library withassert*functions;TwxUnit(Project): A Project marker for all entities related to TwxUnit;TwxUnit(Thing): The default test runner;TwxUnit(Mashup): A Mashup, which alows to execute and monitor test runs;HasTestCases: A Thing Shape to mark test simple suites;HasTestSuite: A Thing Shape for configurable test suites;TestDefinition: A Data Shape representing test suites / cases tree;TestExecution: A Data Shape describing test run results (both synchronous and async);TwxUnitExecutor: Thing Template for test runners, this is where most of the "business logic" resides.
TODO: Currently work in progress.
To uninstall TwxUnit one needs to delete all test suite things and then uninstall twx-unit extension.
Just import a newer version of the extension and restart Tomcat. We follow semantic versioning to ensure that minor-version upgrades do not break your test cases.
TwxUnit is conceptually similar to JUnit in that test suites are things (/ object instances) and test cases are
services (/ methods). Test suites are executed by calling Run({ testSuite, [defaultRunAs], [async] })
on a thing with TwxUnitExecutor base template (TwxUnit by default).
Test executor parses a tree of test suites and generates a complete test plan. It is also responsible for executing this plan and collecting its results. You can abort the execution and reset collected statistics at any moment. See more details in the "TwxUnit thing and Web UI" section below.
Test cases (and test suites) go through the following lifecycle states:
+---------+
+-->| Success |
| +---------+
|
+-----------+ +-----------+ | +---------+
| Scheduled |--+-->| Executing |--+-->| Failure |
+-----------+ | +-----------+ | +---------+
| |
| | +---------+
| +-->| Timeout |
| | +---------+
| |
| | +---------+
+------------------+-->| Aborted |
+---------+
In ThingWorx Composer create a GenericThing called HelloWorldTest, implementing
thing shape HasTestCases. Add a service TestSimpleAssertion, keeping its Inputs and Outputs
empty:
var testedValue = ["Hello", "Earth"];
assertEquals("Hello, World", testedValue.join(', '));Save HelloWorldTest, then find TwxUnit thing and execute service Run, specifying HelloWorldTest as testSuite.
Leave default values for the remaining parameters. Expected results:
| id | testSuite | testCase | description | result | state | runAs | start | end | duration |
|---|---|---|---|---|---|---|---|---|---|
| HelloWorldTest > TestSimpleAssertion | HelloWorldTest | TestSimpleAssertion | Execution error in service script [TestSimpleAssertion] :: Assertion failed, values should be equal: Hello, World != Hello, Earth | Failure | 2020-04-18 22:20:27.548 | 2020-04-18 22:20:27.568 | 20 |
Note: If you specify any inputs or outputs for your Test* services, TwxUnit will fail to execute the test suite.
Try to replace "Earth" with "World" and confirm successful execution.
TwxUnit supports most of classic JUnit assertion methods,
exposed via AssertFunctions script function library:
assertTrue(testValue, [description]);
assertTrue(5 == "5"); // OK
assertTrue(5 === "5"); // Exception 'Assertion failed, value should be true'
assertTrue(5 === "5", '5 !== "5"'); // Exception '5 !== "5"'
assertFalse(testValue, [description]);
assertFalse(5 == "5"); // Exception 'Assertion failed, value should be false'
assertFalse(5 === "5"); // OK
assertEquals(expectedValue, testedValue, [description]);
assertEquals(3, 3.0); // OK
assertEquals(3, 3.1); // Exception 'Assertion failed, values should be equal: 3.0 != 3.1'
assertNotEquals(expectedValue, testedValue, [description]);
assertNotEquals(3, 3.0); // Exception 'Assertion failed, values should not be equal: 3 == 3'
assertNotEquals(3, 3.1); // OK
assertNull(testValue, [description]);
assertNull(Things["Nessie"]); // OK
assertNull(Things["SystemRepository"]); // Exception 'Assertion failed, value should be null: com.thingworx.things.repository.FileRepositoryThing...'
assertNotNull(testValue, [description]);
assertNotNull(Things["Nessie"]); // Exception 'Assertion failed, value should not be null'
assertNotNull(Things["SystemRepository"]); // OK
assertHasPermissions(principal, function, [description]);
assertHasPermissions( // OK
"Administrator", // NB: Administrator is a user
function() {
Subsystems["PlatformSubsystem"].GetPlatformStats();
},
"Administrator can't get platform stats"
);
assertHasPermissions( // Exception 'Assertion failed, principal Users should have permissions: Not authorized for ServiceInvoke on GetPlatformStats in PlatformSubsystem'
"Users", // NB: Users is a group
function() {
Subsystems["PlatformSubsystem"].GetPlatformStats();
}
);
assertHasNoPermissions(principal, function, [description]);
assertHasNoPermissions( // Exception 'Assertion failed, principal Administrator should not have permissions'
"Administrator",
function() {
Subsystems["PlatformSubsystem"].GetPlatformStats();
}
);
assertHasNoPermissions( // OK
"Users",
function() {
Subsystems["PlatformSubsystem"].GetPlatformStats();
}
);Functions assertHasPermissions / assertHasNoPermissions execute their "body" under the specified principal
and System user, following standard ThingWorx semantics. It means that if System has access to execute some
service or write a property, then you won't be able to check that some other user has no such access. For
example, the second assertion in the test below will always fail, regardless of how you configure permissions:
assertHasPermissions("System", function() { me.SomeService(); });
// If we get here, then System has the right to execute SomeService(). It means
// that this test for no permissions will always fail, because SomeService() is
// called with "onion" security context, which includes both: Users and System.
assertHasNoPermissions("Users", function() { me.SomeService(); });assertHasPermissions / assertHasNoPermissions also override runAs parameter (see below).
Test suites provide test grouping functionality. TwxUnit supports two types of test suites -- "simple"
and "advanced", which require using HasTestCases and HasTestSuite thing shapes, respectively. Both of
those thing shapes define optionally overridable services Before() and After(), which wrap the execution
of a test suite (see below), while HasTestSuite also adds testSuite property, which allows to configure
test cases precisely.
Test suite things can use any base thing template, e.g. GenericThing.
The services in simple test suites (implementing HasTestCases) are detected and executed automatically
based on their names. A service is considered test case when its name begins with Test, it returns NOTHING
and takes no parameters. All such services are executed in alphabetical order. A simple test suite cannot
refer to other test suites.
Advanced test suites allow to specify exact names of test cases / services, their order and the principal,
which should be used to execute them. Such test suites can refer to other test suites (complete or separate
services). This is configure via testSuite property, which is a persistable INFOTABLE with three fields:
| testSuite | testCase | runAs |
|---|---|---|
Thing name (optional, me.name by default) |
Service name (if empty, then complete testSuite is executed) |
Principal name (current user, if empty) |
The use of those parameters is self-explanatory and rather intuitive. For example, TwxUnit will inherit
runAs configuration, it will execute Before() and After() in the right order and prevent circular
dependencies between test suites.
IMPORTANT: Make sure that runAs users have execution rights for the corresponding test case services to
avoid Not authorized for ServiceInvoke runtime errors.
Here's a comprehensive example of test suite configuration:
+===================================================+
| RootTestSuite: HasTestSuite | +=====================================+
+===================================================+ +---|> | QueryTestSuite: HasTestCases |
| | | +=====================================+
| - testCases: | | | |
| +================+==========+===============+ | | | - Before(): { ...log "3"... } |
| | testSuite | testCase | runAs | | | | - TestSimpleQuery() |
| +================+==========+===============+ | | | - TestComplexQuery() |
| | QueryTestSuite | | |---------+ | - CheckResults({ data }): BOOLEAN |
| +----------------+----------+---------------+ | | - After(): { ...log "4"...} |
| | AlertTestSuite | | System |---------+ | |
| +----------------+----------+---------------+ | | +-------------------------------------+
| | | MiscTest | Administrator | | |
| +----------------+----------+---------------+ | | +=====================================================+
| | +---|> | AlertTestSuite: HasTestSuite |
| - Before(): { ...log "1"... } | +=====================================================+
| - MiscTest() | | |
| - After(): { ...log "2"... } | | - testCases: |
| | | +================+==================+=========+ |
+---------------------------------------------------+ | | testSuite | testCase | runAs | |
| +================+==================+=========+ | +==================================+
| | CommonTests | AlwaysRunMe | Bob |------+ | CommonTests: HasTestCases |
| +----------------+------------------+---------+ | | +==================================+
| | | TestNotification | | | | | |
| +----------------+------------------+---------+ | | | - Before(): { ...log "5"... } |
| | | TestAlert | Bob | | +-------|> AlwaysRunMe() |
| +----------------+------------------+---------+ | | - TestSomething() |
| | | |
| - TestNotification() | +----------------------------------+
| - TestAlert() |
| |
+-----------------------------------------------------+
If you try to execute RootTestSuite via TwxUnit.Run(), specifying Alice as runAs user, it will follow
this sequence:
RootTestSuite.Before, executed underAlice(andSystem, which is also the case for all other services)QueryTestSuite.Before, executed underAliceQueryTestSuite.TestComplexQuery, executed underAliceQueryTestSuite.TestSimpleQuery, executed underAliceQueryTestSuite.After, executed underAliceCommonTests.Before, executed underBobCommonTests.AlwaysRunMe, executed underBobAlertTestSuite.TestNotification, executed underSystemAlertTestSuite.TestAlert, executed underBobRootTestSuite.MiscTest, executed underAdministratorRootTestSuite.After, executed underAlice
BUG: Currently RootTestSuite.MiscTest (i.e. a local test case) is executed at point (3), which is
incorrect.
See an example above. If Before fails with an exception, the rest of the execution is canceled. If After
fails, the execution continues. After works like finally, it will be always executed if Before executed
successfully. Default implementations of Before and After are empty.
BUG: On ThingWorx 8.5 (and probably 9.x, too) After is not executed if the script times out, or when the
execution is aborted via Abort() service.
To reduce side effects and performance degradation, it is often useful to replace "external" dependencies with lightweight mocks. For example, if you are testing the code, which generates a report and sends an email, you probably want to avoid sending the actual email, but test the report generating logic instead. Some of the typical mock candidates are:
- File repositories
- Streams and data tables
- External systems and connectors
- FileTransferSubsystem, AuditSubsystem, etc.
TwxUnit allows you to temporarily replace an entity with another one in runtime. A typical way to mock a file
repository would be to create a GenericThing (e.g. MockSystemRepository) and define the services, which mimic
the real repo, but without accessing the actual filesystem. For example, you can define a service like
GetFileListingWithLinks, which returns a pre-defined list of "files", or the value of a property like me.files.
Then you need to specify the mapping between original and mock entity names and use runWithMocks script:
runWithMocks(function() {
let files = Things["SystemRepository"].GetFileListingWithLinks({ path: '/' });
// This becomes: Things["MockSystemRepository"].GetFileListingWithLinks({ path: '/' });
// Also, if you call a service, which calls another service, which uses SystemRepository,
// it also gets replaced with MockSystemRepository.
}, {"SystemRepository": "MockSystemRepository"});This entity replacement has system-wide scope, so you may have undesirable side effects if you try to run several tests in parallel. Avoid doing it.
WARNING: If your test times out, it will leave the platform in inconsistent state, with the entities being replaced permanently. The only way to fix it is to restart Tomcat. Make sure your services do not time out.
Regardless of ThingWorx version, network infrastructure, Tomcat and ThingWorx runtime all impose some timeouts on HTTP requests, which (by default) prevent you from receiving results of long-running services.
In addition to that, ThingWorx 8.5 introduced service execution timeouts (30 seconds by default), which
will terminate tests abruptly (After will not be executed in such case, just like finally blocks in
your services). For large or slow test suites you should strongly consider executing TwxUnit.Run with
async flag set to true. Unfortunately, even this won't prevent it from failing if a single test case
takes longer than 30 seconds to execute.
In async mode TwxUnit executes all tests in a separate thread. Currently it still spawns one long-running
thread (that should be killed by the platform), and we are working on a solution to this problem, which
either uses ThingWorx Timer things, or just alternates between two threads spawning one enother.
TODO: Explain getting results in async mode
Each test case, Before and After is executed within its own transaction context, which ensures that you
can expect realistic side effects from the platform.
TwxUnit thing is a default test executor (it uses TwxUnitExecutor thing template). TwxUnitExecutor
extends RemoteThing to simplify integration with other unit testing frameworks, such as JUnit (see below).
Test execution is controlled via the following
-
Run actually executes test suites, and takes the following parameters:
testSuite: THINGNAMEis the name of a thing, which implements eitherHasTestCasesorHasTestSuite(see above);defaultRunAs: USERNAME(optional): is the user or group name and defaults to the current user;async: BOOLEAN(optional) defines whether the tests should run in a background thread and defaults tofalse;
It returns an
INFOTABLEwith the summary of execution results (ifasync == false) or with the test plan, which it has just started executing in the background (ifasync == true). -
PreviewExecutionPlan, which returns an
INFOTABLEwith an execution plan, that will be used once theRunservice is executed. Parameters are the same as forRunservice. For the above example withRootTestSuiteexecuted underAliceit returns the following:
| testSuite | testCase | runAs |
|---|---|---|
| RootTestSuite | MiscTest | Administrator |
| QueryTestSuite | TestComplexQuery | Alice |
| QueryTestSuite | TestSimpleQuery | Alice |
| CommonTests | AlwaysRunMe | Bob |
| AlertTestSuite | TestNotification | System |
| AlertTestSuite | TestAlert | Bob |
-
Abort to stop an execution, regardless of whether it's sync or async. All test cases, which are scheduled for execution after the current one will automatically complete with
Abortedstate. This service takes no parameters and returnsNOTHING. It may take a while for the actual execution to complete, since it won't abort the current test case, just cancel further executions.BUG: This service should run all necessary
Afterservices in the right (stack) order, but currently it doesn't. -
Reset simply clears the last execution results log and resets the executor to its initial state. It can be used to reset the executor if it gets stuck in "executing" state for some reason (usually due to bugs in TwxUnit).
All properties work similarly in sync and async modes to simplify infrastructure implementation. Neither of them is logged or persisted.
-
countExecuted, countRemaining, countTotal: Essentially represent a progress bar.
-
execution is an
INFOTABLE<TestExecution>, which is updated before and after each test case is executed. Although this property is not read-only, any changes to it will be overwritten by the executor. This property should be monitored to get the detailed log of the execution. -
isExecuting is a boolean value, which should be used as a lock, preventing concurrent execution attempts.
TwxUnit mashup exposes TwxUnit services and properties via a simple web UI.
TODO: Implement and describe.
TwxUnit theoretically supports concurrent test execution, although this has never been properly tested.
To do this, you would have to create several test executors and call their Run services at the same time.
Each TwxUnitExecutor thing is self-contained and independent.
TwxUnit executors can be controlled remotely via ThingWorx Always On protocol. We're working on a JUnit test runner, which does it transparently and thus allows to use JUnit infrastructure for running TwxUnit tests, collecting statistics, etc.
TODO: Work in progress, see /junit-runner sub-project.
- Test suites are things ending with "Tests", e.g.
PerformanceTests,UserManagementTests,AlertTests; - Test case names should be relatively short, with longer explanations in service description;
- Numeric prefixes can be used to control test case order, e.g.
Test01CreateThing,Test02ModifyThing; - Fill descriptions for test suites and test cases;
- Use the last (optional) description parameter of
assert*functions; - By default, one should prefer simple test suites (implementing
HasTestCases) to advanced ones (HasTestSuite); - Test suites should be tagged with some model tag, or be part of some Tests project to make sure they don't accidentally become part of production codebase;
- Tag all test data generated in
Beforeto simplify cleanup inAfter;
- Make sure that neither test cases nor TwxUnit extension become part of production codebase;
- Use PASSWORD properties to encrypt credentials when executing tests;
- Create some Testers group to make sure all TwxUnit tests are executed by non-Admin user;
TODO: Expand this section.
ThingWorx doesn't allow to hide exceptions, always appending them to the Script log
no matter what, therefore you will find errors in ScriptLog for every failed assert.
TODO: Validate it for 9.x.
TwxUnit is built using Maven. It relies on the ThingWorx Extension SDK installed in a local Maven repository.
The building preparation process is therefore a bit involved due to the fact that ThingWorx Extension SDK is not available in public Maven repositories. Luckily you need to configure it only once.
- Download ThingWorx Extension SDK 9.0. The version is important.
- Unzip it into
extension/ext-sdkdirectory. - Extract
thingworx-platform-common-xxx.jarfile from your ThingWorx installation and copy it into the sameextension/ext-sdkdirectory. The specific version is not important here, you can take the file from ThingWorx 9.0 or 9.3, for example. This step is not recommended by official extension guide(s), but is required to accessThingWorxEntityManagerfacility, not exposed via the SDK. - Go to
extensionsdirectory and execute./init-maven-repo.shbash script. You can use Git Bash for that. This script will create a new local Maven repo inextension/libdirectory add will all all JAR files fromext-sdkto this repo.
Once the last step executes successfully, you can check that you have a lib/twx directory, which contains a
bunch of subdirectories whose names corrspond to the individual SDK files.
Once you "installed" the SDK in a local Maven repo, building TwxUnit is trivial -- just execute mvn package.
You will find the zipped extension package in the target directory.
Pull requests are welcome!
It would be great for TwxUnit to test itself. Unfortunately, it is not yet the case, but this
is on our priorities list. The approach we envision includes using two test executors and test
cases that verify results of one of their Run services.
In addition to that we need to have a JUnit test suite for the JUnit test runner. This test suite will assume a ThingWorx instance running side-by-side with JUnit test suite.
TODO: Describe the test suites.
Collected from misc. TODOs in no specific order:
- Add support for setting remote properties
- Add support for remote things emulation
- Add support for testing subscriptions
- Localizable failure messages
- Fix function signatures
- Publish TwxUnit mashup
- Publish JUnit test runner
- Fix bugs
- Run unit tests in alternating threads to avoid platform timeouts
- Write-up on security
- Test suite for TwxUnit and its JUnit test runner
- Add few examples of CI pipeline configuration