Programatic interface for TCases

[grzegorzgrzegorz]

Hi,

I use TCases from time to time. I think it is great concept and great tool. The biggest problem for me is input space creation. While it was great idea to change XML to JSON it is still problematic to create JSON file input space for me.
My real life scenario is:

• I am writing some code and I create many tests at the same time (behaviour unit tests)
• at some point I have some area which I want to cover very carefully with tests - either these are acceptance tests for some simple tool (so all combinations can be covered - I just want to get the clear set of testcases) or there is some crucial area which I want to have pair-wisely covered with testcases

There are drawbacks at this point:

• my mind is set to coding mode and it is troublesome to quickly create JSON file at that point
• JSON file is not consistent with what I create: I have code which describes how application work, I have tests (written as code) which describe desired behaviour of the application and I should have code to describe input space
• very often I want to change input space - sometimes rewrite from scratch for some reason - having to recreate huge JSON file is always an awful perspective

I think there should exist "input space as code" approach which would have other advantages as well:

• less lines used to describe input space (json files can have hundreds of lines because of formatting)
• more clear and easier to understand at one glance
• easy comparison with other versions of input space as code
• draw more industry attention which is used to "idea-as-code" approach

Did you ever consider such interface? Here is the example of similar project which tries to implement this idea:
https://github.com/RetailMeNot/pairwise
What do you think about it ?

(I just realized that maybe this should go to discussion section - I am sorry if I make some noise here.)

[kerrykimbrough]

Hi, @grzegorzgrzegorz

You make some very good points. I agree that creating and maintaining an input model can be a difficult job.

Tcases is actually designed to make it easy to do everything from code. Take a look at all of the *Builder classes in tcases-lib. For example, the SystemInputDefBuilder. It's easy to imagine a program that creates an input model and that is modularized to better represent the structure of your specific SUT.

Supposing you had a program that built a SystemInputDef, what would you want it to do next?

[kerrykimbrough]

Hi, @grzegorzgrzegorz

Have you had any more thoughts about the kind of "input space as code" you'd like to see?

[grzegorzgrzegorz]

Hi, sorry for late answer.

I think there is a need for some kind of easy language interface to TCases. And so there are few options as with any other DSL related area.
I am thinking of these (from hardest to easiest):

• ANTLR - to create custom DSL (this seems to be the hardest option I guess - I had problems creating even simple DSL in the past)
• adapt some existing DSL solutions like Gherkin or other stuff (I didn't use it for a long time, but I remember you can create custom glue code there which is a proxy between DSL words and Java or any other supported language)
• create DSL by using pure Groovy or Java (this seems to be simple solution and doesn't require any extra dependencies)

To give any example I created some Java code:

    Property empty = Property("empty").get();
    Property singleChar = Property("singleChar").get();
    Property quoted = Property("quoted").get();

    Variable size = Variable("size")
            .setValue("empty", empty)
            .setValue("singleChar", singleChar)
            .setValue("manyChars")
            .get();

    Variable quoted = Variable("quoted")
            .setValue("yes", quoted)
            .setValue("no")
            .setValue("unterminated", Failure) // predefined Failure property
            .get();

    Variable blanks = Variable("blanks")
            .setValue("none")
            .setValue("one")
            .setValue("many")
            .get();

    VariableSet members = VariableSet("members")
            .setVariable(size)
            .setVariable(quoted)
            .setVariable(blanks)
            .get();

    VariableSet pattern = VariableSet("pattern")
            .setVariableSet(members)
            .get();

    InputType arg = InputType("arg")
            .setVariableSet(pattern)
            .get();

    Function find = Function("find").setInputType(arg).get();
    System examples = System("Examples").setFunction(find).get();

We could use builder pattern to do the job. The obvious advantage I can see here is that I can start with variables first adding higher hierarchy items later as needed. Another advantage is every Java developer feels comfortable with this. It can be easily incorporated into Java and Groovy projects. I can also easily highlight and cycle through all variables and other specific entities which is not the case when using XML/JSON.

With Groovy it is possible to do similar things with less typing and also do some extra tricks to make it even more custom like here:
https://objectcomputing.com/resources/publications/sett/november-2016-the-power-of-groovy-dsls

General rule comes to my mind: it should be as simple as possible in terms of code complexity as I think we cannot write any tests for this. So the same rule should be applied as for unit tests: it should be DAMP not DRY https://enterprisecraftsmanship.com/posts/dry-damp-unit-tests/. That is the reason in the example I repeat setVariable and other methods multiple times.

We are in Java world only but this is where I am now. Users from other realms would be forced to use JVM to generate testcases programatically.

Also. maybe I am missing some other obvious options here.

We are speaking about input only here - I think the output should also be available in cleaner format like set of unit tests which would consist of variables values only or be enriched with annotations. Similar to your documentation examples but without need of using xml transformations. I have some experience with generating testcases - this is much easier when using domain language than raw code. Anyway output is different thing to discuss I think.

Let's focus on input for now.

What do you think about all these ?

[kerrykimbrough]

Looking at your example, it seems quite similar to the existing builder paradigm. If it seems simpler, it is partly because it doesn't demonstrate all of the elements that go into an input model. That's especially true now with the introduction of schemas for variable and value definitions.

On the other hand, this does suggest way to make it easier to use the builders. See PR #270 for a sample implementation. What do you think?

[kerrykimbrough]

Yet another simplification: allow subcomponents to be added in the form of builder instances. Less "build-y"!

[grzegorzgrzegorz]

Yes, it looks really good I think. It is great you can show working example so fast.
I guess we could already start the topic of output then.

I would like to show you how I was generating test cases in the past, using XML version of TCases. I wrote this few years ago:
https://forseti79.e-kei.pl/wordpress/more-about-the-coverage-lets-get-it-right-and-fast-in-practice/
This is simplified example. In reality I had to write much more code to be able to generate testcases at my job. Still, it was possible to create many Selenium based testcases which were testing application which was actually testing application itself with GUI for some sophisticated system company was selling. That is why it was critical this application was working properly and didn't have any defects customer could find. What I am pointing out here is having many good testcases with perfect coverage for the system is not enough if we are not able to translate it into executable testcases automatically.

This would be game changer if it was built into TCases: automatic testcase generation basing on HAS properties. Or maybe some other way which would produce the same result (I remember it was like mixing up 2 completly different things in the same file - variables with values and executable code in the same place - which was making things unclear and hard to read and maintain). I think this is harder topic in general.

What do you think about automatic testcase generation for TCases?

[kerrykimbrough]

Agreed, automatic test generation is the ultimate goal. I've thought about it lot, but it is a complicated problem.

I was able to make substantial progress in a very specific domain with Tcases for OpenAPI. Starting with an OpenAPI definition, TFOA first generates a Tcases input model using special annotations, much as you did for your Se tests. From resulting test case definitions, it was possible to translate these annotations into concrete input values. Then, because the form of an API test execution is fairly well-defined, it was possible to automatically create an executable test.

One thing I learned from this was the power of JSON Schema to describe an abstract input value class. In the latest version, I've pulled that approach back up into the core test case generation framework. As a result, using schemas, Tcases can generate test definitions that contain actual input values, not just abstract "names". That's a step closer to automatic test generation, but a large gap remains.

The first problem is how to serve the broad ranges of requirements for how an executable test is formed and applied to a SUT. Tcases can supply the basic information of a test case, but there needs to be some kind of adapter that can translate that info into the proper form. "Has" annotations can help, but I think your experience confirms that annotations are a cumbersome vehicle for defining the test code. Maybe there's a way to apply template technology like Freemarker.

An even harder problem is how to express in advance the expected results for an automatically-generated collection of input values.
This may have to remain the last manual part of the process.

[grzegorzgrzegorz]

From my experience it is possible to set expected results. For most of the variables there is expected result there. So I had GIVENs and THENs in each variable. There were also WHENs variables which existed only for the purpose of connecting GIVEN and THEN in executable test cases (they were present in each testcase which was generated by TCases).

I was thinking about how to build this bridge between model and executable code. In my opinion it could happen after test cases are created - this would allow not to pollute the input model with some non related stuff.

My idea is that maybe it would be possible to deserialize generated testcases into classes which would contain information on how to generate executable testcase. I created small example basing on icecream example from TCases documentation:
https://github.com/grzegorzgrzegorz/testcase-generation

There is part of variables from the first generated testcase manually instantiated inside main/src/main/java/main/Start.java.
If TCases could only provide Cone and Flavor class instances just like they are created manually there in Start.java it would be possible to create executable testcases as shown there.
Of course this requires extra work but so it is required for the input model. The more variables exist, the more classes should be written.
The problem to be tested is always custom: you need to invest some effort in creating the input model and so you should invest some in creating proper automatic testcase creation mechanism.
I am not sure at this point if these classes should exist only for the purpose of DSL or maybe it would be enough just to update production classes with whenDsl and thenDsl.
I was thinking about Gson deserialization of generated testcases in json format into instances as shown in the example repository. I don't have much experience with deserilization but to me it looks promising. What do you think about it ?

[kerrykimbrough]

Interesting approach. I'm not yet convinced about how to automatically generate expected results. The generator would have know in advance, given any random combination of input values, how to calculate the expected result. In effect, the generator would have to be the SUT!

I see the main issues a bit differently. The Tcases API already provides a data model for the test case values, i.e. SystemTestDef, etc. What's missing is a model of the executable test program to be generated. This actually has two parts. First, there is the overall form of the test code, which depends on the test framework or harness. For example, for JUnit, a Java class with @Test methods, etc. Then, there is form of the code that knows how to apply inputs to the SUT and how to extract the results. This depends on the SUT and perhaps on some additional SUT-specific test harness. For example, for a Web service, the test has to form an HTTP message and send it to a server, perhaps using something like rest-assured or HttpClient.

I could definitely see how you could model the test program and fill it will test case data using a templating engine.

[grzegorzgrzegorz]

Thanks for feedback. Yes, I guess we have a bit different point of view. For sure I agree there is a threat to create mechanism which would require another layer of tests which wouldn't make much sense.
When I was generating testcases with TCases, the solution which was working in my case was adding expected results to most of the variables. I guess I will try this approach to some test program as you say or maybe even to some class to model and generate unit tests. I cannot see any obvious blockers for this right now but it is true I need to write model, generate json output, deserialize with Gson and only then can I see if it is really working... I hope to find some time to try it soon.

[kerrykimbrough]

Good luck and stay in touch!

[grzegorzgrzegorz]

Hi, I finally made working example of generating testcases basing on deserialization of json testcases here:
https://github.com/grzegorzgrzegorz/testcase-generation.
There is a readme there with details. So as I said before, I did it in the past using the same concept of GIVEN/THENs produced by each variable. Just the technical base is now different and instead of processing XMLs I deserialize testcases. The big advantage is that code generation is completely separated from model and json testcases now.