unittest location

[stdweird]

We need a subdirectory to put unittests in that will not be packaged/part of the release

[stdweird]

@jrha any suggestions?

[jouvin]

@stdweird what is the use case? What do you want to put in such a directory? Based on my experience unit testing templates doesn't really make sense. The approach I tried to push so far is to have a set of examples that you compile. This is what is done by create-vanilla-SCDB.sh in scdb repo. The main test that can be done is to check that you can compile successfully the various profiles as in most changes, you expect some modification in the XML/Json generated.

Another use case is when you do template refactoring. In this particular situation you generally want to ensure that the resulting profile is the same. Again, unit testing the template that you modified doesn't really make sense because you may need to define various things before callng the template. What I have been personnally doing for 10 years that I am doing template development (with a lot of refactoring!) is using the SCDB-provided compare_profiles script (it is not linked in SCDB in any way, except the default path for finding profiles) that list the profiles that have been modified (based on a reference version that I make by copying the current profiles before compiling the modification) and with -v display a text diff of the modified profiles. This way you can easily spot problems, in particular if you don't expect any modification in the profile. I am typically doing this with all the GRIF profiles (# 1000), providing a good chance to spot any issue.

Is your need different?

[stdweird]

See e.g. #102
I started cleaning up the pan/functions, and than it is better for some simpler tests to make really basic stuff still works (in this case, the examples in the annotation).
obviously, more complex test cases like the SCDB based build tests cover more realistic cases and can only make the testing more complete.

[jouvin]

I was about to comment in #102 that I am not at all convinced by the value of these tests. And their value is really limited to functions, I don't see any other use cases really possible. If this is really adopted, not that we also need to add get_template_library script in the release repo that is (should be) the real way to obtain the template library (relying on tar file is painful as in some directories, you need to check out several production branches). It is certainly not a major change in get_template_library but we should not forget to do it.

[stdweird]

it allows you to test functions individually and thus catch problems very early. imagine a subtle issue in npush, it will be show up in the SCDB test, but tracking it down to the issue in npush might take a while.
it's complementary, not meant as a replcement.
the other case this allows is testing error conditions, but i haven't come up with a proper framework for that.

[jrha]

@jouvin are you really trying to suggest that unit testing Pan code would not valuable?

[jouvin]

Not exactly... I'm discussing if setting up this separate test framework is worth the effort. I have the feeling that we could do it with the "standard" template library testing tool (that I'd like to see run at each PR!), even adding some "proto-profiles" doing unit testing of the functions rather than compiling a full profile. I may come with a proposal around this, if you think it can be a good approach.